Re: Frames in HTTPWebRequest - C# Discussion Boards

10-Dec-04 7:08

I am accessing a web site in C# using HTTPWebRequest. The site uses frames. I POST a request but the response is only a thousand bytes that lists the frames to be included in the page. How do I use the response from my HTTPWebRequest to access the second frame that contains the data I am interested in retrieving?

Here is the response I get from my POST:

<title>MSU Schedule Of Courses

To view this page, your browser must support
frames.

Heath Stewart10-Dec-04 12:41

10-Dec-04 12:41

The same way that browsers do - you need to parse the text and find the frames. Make sure you keep a reference to the page's URL so that you can resolve relative URLs like so:

Uri newUrl = new Uri(oldUrl, "HeadingBlank.asp");

If you want a handy class to help parse HTML (a grammar defined by SGML), download and reference the SgmlReader[^] on GotDotNet.

This posting is provided "AS IS" with no warranties, and confers no rights.

Software Design Engineer
Developer Division Sustained Engineering
Microsoft

[My Articles] [My Blog]

JeromeKJerome10-Dec-04 14:42

10-Dec-04 14:42

Heath, I think you lost me on that one. All I get back from my C# HTTPWebRequest is 1k bytes. If I use IE for the same POST and use proxy sniffer to monitor it, I get 40k back. Obviously my browser is doing something more than my C# is doing.

You suggest I parse the text, but the text is 1k and no longer. Here is the C# that gets the response:

// read response stream and dump to string
Stream streamResponse = webResponse.GetResponseStream();
StreamReader streamRead = new StreamReader(streamResponse);

output = streamRead.ReadToEnd();

streamRead.Close();
streamResponse.Close();

Should I just leave the stream reader open and try to read from it again? I have to think that the web server is sending all the frames in one response with some kind of delimiter between frames. Does my ReadToEnd just stop prematurely? The 1K I actually get tells me what frames are coming, there are four. Should I just try ReadToEnd four times?

Heath Stewart10-Dec-04 20:24

10-Dec-04 20:24

First of all - never use ReadToEnd if you don't know what to expect. 40K may not be a big buffer, but lets say you get back a 1 or 2MB buffer, or even more. You'll read that all into memory? I would hope not.

A stream is just that - a stream. Read-in the stream in blocks as it comes to you.

As far as the problem, it's hard to say if you don't post the HTML (which I thought was the frames page). When you're browser reads a file with frames in it, it parses the location of each frame then requests those files. The same is true for images, script, and other links. Everything you see in your browser - from pages to images to scripts - are separate requests and separate downloads. This is a basic principal of HTTP. You request one file, you get one file.

So, once again you parse that file with the frames to get the other document locations, then make a request for those.

If you're not getting the whole frameset content, then something else is wrong. It's possible that the service is using a browser capacity (browsercap) file and not sending you frames because you don't have a user-agent (an HTTP header that identifies the browser). To get around that and look like Internet Explorer, before you make your request set your HttpWebRequest.UserAgent property to the follow (IE6 on WinXP):

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;)

To the server, you look like Internet Explorer and it would give you a frameset page instead of something else. That's assuming that's what's happening.

This posting is provided "AS IS" with no warranties, and confers no rights.

Software Design Engineer
Developer Division Sustained Engineering
Microsoft

[My Articles] [My Blog]

JeromeKJerome11-Dec-04 2:24

11-Dec-04 2:24

Well, that 1k I'm getting IS the frameset page. I showed it on my first post, but here it is again:

<title>MSU Schedule Of Courses

To view this page, your browser must support
frames.

Here is the request header:

1 POST /ScheduleBook/schedule.asp HTTP/1.1
2 Content-Type: application/x-www-form-urlencoded
3 Connection: close
4 Accept-Language: en-us,en;q=0.5
5 Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
6 Referer: http://ntweb8.ais.msu.edu/ScheduleBook/selection.asp
7 User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.3) Gecko/20040910
8 Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,image/jpeg,image/gif;q=0.2,*/*;q=0.1
9 Content-Length: 248
10 Expect: 100-continue
11 Host: ntweb8.ais.msu.edu

Here is the response header:

1 HTTP/1.1 200 OK
2 Server: Microsoft-IIS/5.0
3 Date: Fri, 10 Dec 2004 17:34:28 GMT
4 X-Powered-By: ASP.NET
5 Connection: close
6 Content-Length: 1043
7 Content-Type: text/html
8 Set-Cookie: ASPSESSIONIDQCQCQDBS=OEFOLADCEDMDCKNNFEEJJBOJ; path=/
9 Cache-control: private

As you can see, I am setting the user agent to "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.3) Gecko/20040910."

I see from your comments that if the length is large I should pay attention to the content-length and don't use ReadToEnd. So for this particular application, where I expect to get about 40k each request, I could check the length and halt if the length is over say, 100k. This would not require me to learn how to use something other than ReadToEnd, since I am comfortable with that and don't have the lead time to learn more new tricks than necessary right now.

I am only interested in the frame named "Courses." This application is very specific, I will always only want the frame named "Courses," so I don't even need to parse the frames page unless the name of the "Courses" frame changes.

In your first post to my question, you said "...you can resolve relative URLs like so: Uri newUrl = new Uri(oldUrl, 'HeadingBlank.asp')..." My HTTPWebRequest POSTS is constructed using this statement based on UriBuilder:

uBuild2.Path = "ScheduleBook/schedule.asp";
Uri location = uBuild2.Uri;

So can I infer that on my POST for the "Courses" frame, I should simply reconstruct my Uri using this statement?:

uBuild2.Path = "ScheduleBook/intro.asp";

If so, how does the server know what page to return? From the cookie? From the postData?

If these assumptions are correct, then my process would be two-step: request frames page and cookie, then request "Courses" page (intro.asp). Can I short-circuit this process and just repeatedly request subsequent courses pages?

BTW, I am using Proxy Sniffer from David Fisher ( http://www.d-fischer.com) to compare my browser requests with my C# requests. It doesn't show that my browser is making multiple requests to get these frames, although it does show my browser making multiple requests to get other things like images.

Dave Kreskowiak11-Dec-04 4:35

Dave Kreskowiak

11-Dec-04 4:35

JeromeKJerome wrote:
uBuild2.Path = "ScheduleBook/intro.asp";

If so, how does the server know what page to return? From the cookie? From the postData?

You just told it which page to return! Like Heath said, one file requested, one file returned. It's up to your code to parse out the filenames of the pages that make up the frameset. The first request will get back the 1K HTML you've been getting. Your code must then search through that returned data and find the <frame> tags, pull out the filename from the SRC parameter of the tag, then generate a new request using that filename. Do this for each frame tag... That's how a web browser does it.

JeromeKJerome wrote:
Can I short-circuit this process and just repeatedly request subsequent courses pages?

Yes! The intro.asp file is just it's own seperate HTML page. You can request just that page and it'll get returned to you just as if the browser requested it.

JeromeKJerome wrote:
BTW, I am using Proxy Sniffer from David Fisher ( http://www.d-fischer.com) to compare my browser requests with my C# requests. It doesn't show that my browser is making multiple requests to get these frames, although it does show my browser making multiple requests to get other things like images.

Your browser IS making seperate requests to get the pages specified in the frameset. It's the ONLY way it's going to get them. The Proxy Sniffer is either missing them, or you're misinterpreting what it's showing you, or it's entirely possible that the page requests aren't being generated because the browser is getting the content from it's cache.

Think of it this way... in your case, web servers are stupid. They only know how to return the file that was requested. They DON'T look into the contents of the file and pick out other files to return along with the requested file. They won't send any images, data, CSS sheets, or javascript files back with the requested page. It's up to the browser to interpret the page then make additional requests for each image, each CSS page, each script file, everything.

When you click on a <reply> link here at Code Project, your browser can make upwards of 35 requests to get all of the content that you see in this simple page!

RageInTheMachine9532
"...a pungent, ghastly, stinky piece of cheese!" -- The Roaming Gnome

Heath Stewart11-Dec-04 21:51

11-Dec-04 21:51

Yes, and as I said the first time you need to parse the frameset page and get the frame's src attribute, then create a new Uri using the frameset page's URL plus the src attribute to create an absolute URL. This is what your browser does. It does not fetch the entire page that you see in one request: it requests the page you requested (the frameset page), then requests the pages that it references, plus all the images, scripts, and other links that those pages reference. That's how HTTP works. That 40K of data is over several requests if you look carefully.

I showed you how to construct an absolute URL in my first reply. You must make a request using the absolute URL including the scheme and host, otherwise the request doesn't know what server to connect to. This is why you use the frameset page's URL and use the <frame src/> attribute to construct an absolute URL (so "http://www.domain.com/default.htm" as the frameset, plus a src attribute within that page of "pages/main.htm" would resolve as "http://www.domain.com/pages/main.htm"). If you use the Uri structure correctly (read the .NET Framework SDK for details) it will do this for you.

Again, this is exactly what your browser does in order to display all the frames and their images and other linked files - each made with a separate request.

This posting is provided "AS IS" with no warranties, and confers no rights.

Software Design Engineer
Developer Division Sustained Engineering
Microsoft

[My Articles] [My Blog]

JeromeKJerome14-Dec-04 6:37

14-Dec-04 6:37

Thanks, Heath. The problem is solved and I learned a lot about how to set up a structure to handles frames in my C# application. Your postings are top-notch!

Help in Delimiter!

ahjiefreak10-Dec-04 5:25

10-Dec-04 5:25

Hi,

I actually facing problem in coding delimiter algorithm especially when user key in their entries in a rich textbox, the space that the user key in will be treated as delimiter ";".In other words, the space that user key-in after entering a word will be converted(interpreted) to ";" as delimiter symbol before user key in next word in the same line of text box as the second item.

Please give me some guidance or any available reference/samples available algorithm on this in C# code.I need to clarify my doubts on this as soon as possible.

Thanks..

Alexander Kent10-Dec-04 17:00

Alexander Kent

10-Dec-04 17:00

Maybe you mean something like this:

string delimiter = ";";
textBox1.Text = textBox1.Text.Replace(" ", delimiter);

Confused | :confused:

~Alexander Kent

Help in Delimiter!

ahjiefreak10-Dec-04 5:23

10-Dec-04 5:23

Hi,

I actually facing problem in coding delimiter algorithm especially when user key in their entries in a rich textbox, the space that the user key in will be treated as delimiter (;).In other words, the space that user key-in after entering a word will be converted(interpreted) to ";" as delimiter symbol before user key in next word in the same line of text box as the second item.

Please give me some guidance or any available reference/samples available algorithm on this in C# code.I need to clarify my doubts on this as soon as possible.

Thanks..

Alexander Kent10-Dec-04 17:00

Alexander Kent

10-Dec-04 17:00

Maybe you mean something like this:

string delimiter = ";";
textBox1.Text = textBox1.Text.Replace(" ", delimiter);

Confused | :confused:

~Alexander Kent

ahjiefreak11-Dec-04 2:11

11-Dec-04 2:11

The above line provided does make sense..but how am i going to check if user key in space (" ") in the particular text box, the ";" will be replace onto it and the pointer for next word is after the symbol ";"?

Thanks in advance,
Lee

Alexander Kent11-Dec-04 13:28

Alexander Kent

11-Dec-04 13:28

You can try and use the TextBox SelectionStart property – something like:

In the KeyDown event

if (e.KeyCode == Keys.Space)
{
// Cancel the event
e.Handled = true;

// Use your delimiter
textBox1.Text += ";";

// Put the cursor after the newly appended text.
textBox1.SelectionStart = textBox1.Text.Length;
}

~Alexander Kent

ahjiefreak16-Dec-04 4:48

ILDASM and Memory. Some confusion...

16-Dec-04 4:48

Yea..I have tried and it only works in C# form but what I face the problem of delimiter lies in Web Application Form in C# Project Folder..(fyi, I am trying to put my C# application into Web Based)

I have tried to code the particular text box for the delimiter but it doesnt have SelectionStart function in Web Application.Thus, I used the previous code as below:-

private void TextBox3_TextChanged(object sender, System.EventArgs e){

string delimiter =";";

if (TextBox3.Text.EndsWith(" "))
{
TextBox3.Text += ";";
TextBox3.Text=TextBox3.Text.Repla ce(" ", delimiter);

}

}

With this code, I face another problem which is when i enter the input then";" doesnt come up automatically..it only comes up occasionally especially after I let the Validation Error appear in other textbox.

Please help me to clarify this doubt..I am stuck nowhere.

Thanks,
Lee

...---...10-Dec-04 5:23

...---...

10-Dec-04 5:23

I'm confused.
If something is popped form the stack and stored in the 'local variable', where is it?

It's not on the stack?

Confused | :confused:

What's distinguishing between the following?

1. The registers :

2. The Heap.

3. The 'stored local variable'.

4.As for the three above, how do I know 'what is where'??

See code below as an example...

//allocating and deallocating multiple variables on the stack
.ldstr "Hello World" //push this string on stack
ldstr "Goodbye World" //push second string on stack
stloc.0 //pop first string from the stack and store it in the local variable 0. // Huh? Where is it now?

leppie10-Dec-04 5:58

leppie

10-Dec-04 5:58

SkunkedWorks wrote:
.ldstr "Hello World" //push this string on stack
ldstr "Goodbye World" //push second string on stack
stloc.0 //pop first string from the stack and store it in the local variable 0. // Huh? Where is it now?

it pops the top entry, iow the second string. if you dont understand this, go do some reading on stack behaviour. that value will be stored in the 1st local variable then, and only the 1st string will be left on the stack.

^{top secret
Download xacc-ide 0.0.3 now!
See some screenshots}

...---...10-Dec-04 6:39

...---...

10-Dec-04 6:39

leppie,
Thanks for your help.

1. WHERE IS the 1st local variable?

2.Where does one read about stack behavior?

Alex Korchemniy10-Dec-04 8:41

Alex Korchemniy

10-Dec-04 8:41

Stack is the memory location where local variables are stored and other things that dont consume a lot of memory. This is done because there are not enough CPU registered to store everything.

You can read about CPU's, stacks, heaps, whatnot in a various assembly references. First thing that comes to mind is Art of Assembly, but I'm not sure if its a free resource anymore. Search in google and you should find a lot of stuff Wink | ;)

This posting is provided "AS IS" with no warranties, and confers no rights.

Alex Korchemniy

Heath Stewart10-Dec-04 12:46

10-Dec-04 12:46

Alex Korchemniy wrote:
Stack is the memory location where local variables are stored

That's not exactly true. What stored on the execution stack are pointers to the variables, whether they be parameters, local variables or fields.

So, to answer the original question the local variable is at a certain address apart from the execution stack. When you use stloc.* it pops the variable referenced on the stack and assigns that reference to the applicable local variable.

The best way to learn about this stuff is to learn about computer architecture - specifically about memory addressing.

This posting is provided "AS IS" with no warranties, and confers no rights.

Software Design Engineer
Developer Division Sustained Engineering
Microsoft

[My Articles] [My Blog]

Alex Korchemniy10-Dec-04 13:28

Alex Korchemniy

10-Dec-04 13:28

> That's not exactly true.

Well its a simplistic answer.