|
WebRequest and WebResponse has issues
I wrote a C# program that uses WebRequest and WebResponse to perform a simple web crawler. I discovered something about web sites. Web browsers such as IE and FireFox offer the capacity to view the HTML source code. But it seems that html code that is sent to the browser is one thing and what the browser interprets and displays is something else. For example, if you run a google search in IE and run the same google search in FireFox, the content that you can see when you view the source in IE will NOT have the hyperlinks and content from the search results, but you can see the html hyperlinks and content from the search results when you view the source in FireFox. So my question is this. How do you specialise the WebRequest and WebResponse to show the content after it is processed by the browser instead of before?
|
|
|
|
|
Xarzu wrote: WebRequest and WebResponse has issues
No, really they dont. What you go on to describe has nothing to do with the way you make an http, ftp or any other sort of request/response using WebRequest and WebResponse.
Makers of browsers are free to parse and change the raw content they get from a web server in any way they see fit, what you are doing is making the raw request and getting back the raw response. Using the classes you mention, you will have no way of showing the "content after it is processed by the browser". The only way to do that is get the browser (or an embedded web browser control) to make the request for you.
I suspect the real problem you're facing is badly formatted, or even invalid HTML. In which case this is a known problem with many solutions - the most common of which is probably the HTML Agility pack[^]
|
|
|
|
|
Just to be clear, my goal is to programatically get search results from a google search.
After looking closely at the different html source from IE and firefox and seeing the point at which they start to differ, I can safely conclude that what is happening is that firefox is showing the html prior to the browser running the javascript and IE is showing the html that results after javascript has been processed by the browser.
I think maybe I need to format what I sent to the WebRequest.Create method, but the question I know have is this. How do I send to this method the desire that I want to create the WebRequest such that I set the visibility to hidden. I believe that therein might lie the key to getting this done. It seems that the javascript processes the html in IE because the visibility is not hidden.
|
|
|
|
|
Quote: after it is processed by the browser instead of before?
And what do you mean by that? It is not hard really to differentiate html across browsers. Google is known from that. To get html for FireFox, IE, Android etc. just change UserAgent HTTP header:
http://tools.ietf.org/html/rfc2616#section-14.43[^]
No more Mister Nice Guy... >: |
|
|
|
|
|
Agree 100%.
This can be accomplished by checking the Request.UserAgent in ASP.NET and the sending different output. Easy way to optimise your code for each browser and not to have to send everything down the wire.
|
|
|
|
|
my goal is to programatically get search results from a google search.
Using WebRequest and WebResponse yelds results like I see in IE where the (if I look at the processed HTML source) javascript manages to hide the search engine results from google after the browser processes the javascript.
The source code from Firefox is before html is processed by the browser and the javascript manages to hide the content.
How do I programatically make web requests to be like it is in Firefox?
|
|
|
|
|
I don't get it. If you wan't to get html for FF just use FF User-Agent.
If you want to have HTML after it is swallowed by JS then I have 2 questions: why you need this anyway? why do you think it is even doable in C#, it's JS code how are you imagine to execute JS code by c# program?
No more Mister Nice Guy... >: |
|
|
|
|
|
the source display in IE differs from FireFox on the exact same page.
If you use google in these two browsers, IE manages to have the resulting content -- the search result -- hidden in the html source.
|
|
|
|
|
As I wrote before Google feeds different browsers with different html.
Just choose one that suits you best.
No more Mister Nice Guy... >: |
|
|
|
|
|
Hi
I found here in an article, a ready function which returns the square root of biginteger, but I can't understand how it works. Can someone please explain me the algorithm and why it works:
public static BigInteger IntegerSqrt(BigInteger n)
{
BigInteger oldValue = new BigInteger(0);
BigInteger newValue;
newValue=n;
while (BigInteger.Abs(newValue - oldValue) >=1)
{
oldValue = newValue;
newValue = (oldValue + (n / oldValue)) /2;
}
return newValue;
}
thanks
|
|
|
|
|
|
In fact, this implementation doesn't always work!
try IntegerSqrt(3)
oldValue and newValue will alternate between 1 and 2
Changing the while condition from >= to > will enable the loop to terminate.
|
|
|
|
|
Maybe its mistake declaring root of number as int since roots are usually fractional numbers.
No more Mister Nice Guy... >: |
|
|
|
|
|
No, they're never fractional numbers (unless you'd count n/1). A square root is either an integer or an irrational number.
|
|
|
|
|
True, but I think he meant that they are usually non-integers.
|
|
|
|
|
Perhaps. I didn't vote him down, by the way.
|
|
|
|
|
|
Um... Maybe I am missing something (English is not my native language but if I am understand correctly square root is this http://en.wikipedia.org/wiki/Square_root#As_decimal_expansions[^]) How is this a usually integer? Irrational number you get only if you take sqrt of negative number. You can get sqrt of fractional to. And how is this an int?
People which voted me down before plaese forgive me if I get something wrong cause english related to math is not my strong point.
No more Mister Nice Guy... >: |
|
|
|
|
|
The square root of a perfect square is an integer.
The square root of anything else is an irrational number[^].
n.podbielski wrote: Irrational number you get only if you take sqrt of negative number.
That's imaginary. Also irrational, usually.
|
|
|
|
|
Thats what I meant. Thanks for clarifying this to me 
No more Mister Nice Guy... >: |
|
|
|
|
|
Hi,
how set request value for WSDL choice element - see example:
ws_test.AType my_ws = new AType();
my_ws.AData = new ADataType();
my_ws.AData.Item ??
namespace ws_test
{
[Serializable]
[DebuggerStepThrough]
[XmlType(AnonymousType = true, Namespace = "urn:cz:aaa:bbb:schemas v1")]
[GeneratedCode("System.Xml", "4.0.30319.1")]
[DesignerCategory("code")]
public class ATypeReq
{
public ATypeReq();
public ADataType AData { get; set; }
}
[Serializable]
[DesignerCategory("code")]
[GeneratedCode("System.Xml", "4.0.30319.1")]
[DebuggerStepThrough]
[XmlType(Namespace = "urn:cz:aaa:bbb:schemas:xxx:v2")]
public class ADataType
{
public ADataType();
[XmlElement("AElement", typeof(ADataTypeAA))]
[XmlElement("BElement", typeof(ADataTypeBB))]
public AStavAbstractType Item { get; set; }
}
}
<xs:complextype name="ADataType">
<xs:annotation>
<xs:documentation xmlns:xml="http://www.w3.org/XML/1998/namespace" xml:lang="cs">
<xs:choice>
<xs:element name="AElement">
<xs:complextype>
<xs:complexcontent mixed="false">
<xs:extension base="edi:AStavAbstractType">
<xs:sequence>
<xs:element minoccurs="0" name="AA" type="edi:AAStavType">
<xs:element name="BElement">
<xs:complextype>
<xs:complexcontent mixed="false">
<xs:extension base="edi:AStavAbstractType">
<xs:sequence>
<xs:element name="BB" type="edi:BBStavType">
Thanks,
Marián
|
|
|
|
|
Does anyone know the best way to migrate all my groups screwturn wiki content(pages) over to Sharepoint's wiki?
Currently I can use the Sharepoint Client API to pragmatically create a wiki page in sharepoint, so I can read in a screwturn page and save it as a Sharepoint page. This works, but I have to do the formatting (not done yet) and that seems like its going to be a real pain in the ass. Can anyone give some advice on either formatting or a better way to migrate, maybe with some tool or something?
|
|
|
|
|
Hi All,
i need assistance with reading an excel file, writing it to a datagrid (this i already have in my code) and then for each line in the datagrid i need the following.
if a cell value is xxx in column y then do....
for example:
i have a column called "vehicle/pedestrian" now i want the following:
for each line in the datagrid if cell value in this column (vehicle......) then do.......
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Text;
using System.Windows.Forms;
using System.Data.OleDb;
using System.IO;
using System.Runtime.InteropServices;
namespace Datagridtest
{
public partial class Form1 : Form
{
private void BrowseBtn_Click(object sender, EventArgs e)
{
FilePathBrowse.Filter = "SIT Files (*.xls) | *.Xls";
if (FilePathBrowse.ShowDialog() == DialogResult.OK)
{
if (FilePathBrowse.FileName != "")
FilePath.Text = FilePathBrowse.FileName;
}
}
private DataTable dt;
public static string BuildExcelConnectionString(string Filename, bool FirstRowContainsHeaders)
{
return string.Format("Provider=Microsoft.Jet.OLEDB.4.0;Data Source='{0}';Extended Properties=\"Excel 8.0;HDR={1};\"",
Filename.Replace("'", "''"),
FirstRowContainsHeaders ? "Yes" : "No");
}
public Form1()
{
InitializeComponent();
}
private void button1_Click(object sender, EventArgs e)
{
string fileName = FilePath.Text;
if (!File.Exists(fileName))
{
MessageBox.Show("Cannot find file");
return;
}
string connStr = BuildExcelConnectionString(fileName, true);
using (OleDbConnection conn = new OleDbConnection(connStr))
{
conn.Open();
using (OleDbCommand cmd = new OleDbCommand("Select * From [Sheet1$]", conn))
{
using (OleDbDataReader dr = cmd.ExecuteReader())
{
if (dt != null)
dt.Dispose();
dt = new DataTable();
dt.Load(dr);
}
}
}
dataGridView1.DataSource = dt;
}
}
}
the excel i have is something like this
point Long Lat FOV [pedestrian / vehicle] [single / multiple] Target Location LOS/Detection Freetext
1 -73.882644 40.692344 5 Vehicle Multi TGT Road MIN DET Range Your Fee text here
2 -73.882903 40.69259 2 Pedestrian Single TGT sidewalk DET from PREV to this PT Your Fee text here
thanks alot.
Jonathan
|
|
|
|
|
If you use a OleDbDataAdapter[^] to read the worksheet you can get it to populate the datagridview automatically for you, something like:
OleDbCommand cmd = new OleDbCommand(string.Format("Select * from [Sheet1$]");
cmd.Connection = oleDbConnection;
OleDbDataAdapter adpt = new OleDbDataAdapter(cmd);
DataSet ds = new DataSet();
adpt.Fill(ds, "[Sheet1$]");
if (ds.Tables.Count == 1)
{
dgView.DataSource = ds.Tables[0];
}
One of these days I'm going to think of a really clever signature.
|
|
|
|
|
Hi
thanks for the prompt reply but i didnt understand it.
my code is currently working with reading the excel file and putting it in the datagrid.
what i am missing is that for each line that contains a certain value in a cell in a certain colum todo something.
with the code you posted how can it be done?
thanks
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
