Read web page source code in a web page by using .NET C#

Question

0.00/5 (No votes)

See more:

Hi,

I try to read a web page as follow in code, it works. The problem is that the web page has two section, main body and answer pane, The following code is only report the main body source code, and it has a error message "Your browser doesn't support iframes" in the answer pane section. However, in the activated web page, click on the answer pane section, and manually review source. The source code report is correct. In C#, How can I control the webpage and focus on the answer pane section before call webClent DownloadString() to get right source text.

It is appreciated for your reply and time.

RQ

using System.Windows.Forms;
using System.Net
...
System.Net.WebClient wc = new System.Net.WebClient();
string webData = wc.DownloadString("http://start.csail.mit.edu/answer.php?query=Who+is+the+41th+president+in+USA");

What I have tried:

I search internet, and try different code, but no luck.

Posted 6-Jul-16 6:25am

RQ yang

Updated 8-Jul-16 2:18am

Add a Solution

Comments

Beginner Luck 7-Jul-16 2:23am

use htmlagilitypack. just google htmlagilitypack

2 solutions

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Ehsan Skardu · Answer 1 · 2016-07-08T01:30:00

Try this

C#

private static string ReadsourceCode(string Url)
{
string data="";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(Url);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();

if (response.StatusCode == HttpStatusCode.OK)
{
  Stream receiveStream = response.GetResponseStream();
  StreamReader readStream = null;

  if (response.CharacterSet == null)
  {
     readStream = new StreamReader(receiveStream);
  }
  else
  {
     readStream = new StreamReader(receiveStream, Encoding.GetEncoding(response.CharacterSet));
  }

  data = readStream.ReadToEnd();

  response.Close();
  readStream.Close();
}
return data; 
}

call like this

C#

var source =ReadsourceCode("http://start.csail.mit.edu/answer.php?query=Who+is+the+41th+president+in+USA");

F-ES Sitecore · Answer 2 · 2016-07-08T02:18:00

An iframe is shown as a page within a page by your browser, it is a visual trick to make two pages look like one. You'll have to parse the html of the page you've downloaded to find the iframe, then read the "src" element (you can do all this using agility pack, or just plain text manipulation or regex). You then need to issue a second request to the page in src just as you did your initial page, you'll probably have to append the domain name to it (http://start.csail.mit.edu/justanswer.php?query=....).

Read web page source code in a web page by using .NET C#

2 solutions

Solution 1

Solution 2

Add your solution here

Preview 0