How to call anther webform by using javascript

Kumar, why do you write code to open a popup window in business logic layer.
Purpose of Business Logic Layer is different. Be clear with what is 3-tier/n-tier architecture is about. At the end of it all, you must be able to appreciate why a tier (or Layer) is required and what each one does specifically.

Coming to you question, I assume that you are required to open modal/nonmodal popup window. Since this is purely a UI thing, you will not be required to go to Business logic layer and open it from there. You can very well do that in UI itself.

In Main.aspx.cs, in the button click event write some thing like below.

void Button_Click(object sender, EventArgs e)
{
   OpenNewWindow("Sub.aspx");
}

public void OpenNewWindow(string url)
{
   ClientScript.RegisterStartupScript(this.GetType(), "SubWindow", String.Format ("<script>window.open('{0}');</script>", url));
}

This shoould do for you. Else, try out for different ways of opening a modal dialog in asp.net.

To implement in javascript please use the code below:
Declare your button in your .aspx page,

<asp:Button runat="server" ID="BtnRedirect" Text="Open Popup"  OnClientClick="return OpenWin();"/>

And use the script below to open the Sub.apsx in dialog. You will be able to work with your parent page only after closing Sub.aspx

<script language="javascript" type="text/javascript">
 function OpenWin() {
    window.showModalDialog('Sub.aspx');
    return false;
 }
</script>

Add one button on main page and double click on button and write below code
in that button click call your method in business class and then use

HttpContext.Current.Response.Redirect("sub.aspx");

then it will redirects to that page