WebRequest Times out

Question

0.00/5 (No votes)

See more:

I have following code to read a webpage which works fine. Except whenever same url is being called twice second time WebRequest times out.
for example first time http:www.yahoo.com/main.html will work fine
second time http://www.yahoo.com/test.html it will time out.
Anyone knows how to fix this? thanks

C#

public static string ReadWebPage(string url)
        {
            string webPage;
            WebRequest request = WebRequest.Create(url);
            using (Stream stream = request.GetResponse().GetResponseStream())
            {
                StreamReader sr = new StreamReader(stream);
                webPage = sr.ReadToEnd();
                sr.Close();
            }
            return webPage;
        }

Posted 30-Jul-12 22:36pm

EricThe

Updated 30-Jul-12 22:43pm

Kenneth Haugland

v2

Add a Solution

Comments

Kenneth Haugland 31-Jul-12 4:44am

cold be that the server does deny you multiple searches within a given time frame...

I.explore.code 31-Jul-12 5:09am

that would be my guess too Ken. Modern websites, including Google, deny further requests once they detect that they are being crawled by bots. Since they have a threshold as to how many requests per second or whatever can humans make and what value would be representative of automated requests like the one above. May be the OP can try sending requests every 10 seconds or so rather than doing a rapid fire. This can still fail because sites like Google have pretty smart firewalls and they will still detect automated requests.

Rob Philpott 31-Jul-12 5:10am

Yeah, odd. Looks fine to me, in fact I've just tried it to access news.bbc.co.uk twice and it worked fine. Only thing I can think is might have something to do with keep-alives - but I doubt that.

2 solutions

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Aarti Meswania · Answer 1 · 2012-07-31T00:47:00

Solution 2

http://www.yahoo.com/test.html[^]

this page is not found when open it directly in browser., it is redirecting you on something different yahoo's page. so, it can't process two requests.(one page is causing another page to load)

Happy coding!
:)

Posted 31-Jul-12 0:47am

Aarti Meswania

Rahul Chitte · Answer 2 · 2012-07-30T23:14:00

Solution 1

Use garbage collection to dispose your WebRequest object which is causing issue.
Add GC.Collect() at end of your method call.

Thanks and regards,
Rahul Chitte

Posted 30-Jul-12 23:14pm

Rahul Chitte

Comments

I.explore.code 31-Jul-12 6:03am

As far as i understand, this is never a recommended practice since you cannot tell (atleast in managed environment) when the objects will be eligible for GC. The collector does its job tuning itself based on the generation of objects. Calling GC.Collect() is not like defining a destructor in C/C++ unmanaged environment. The problem OP is facing is more about how the web works rather than GC.