Click here to Skip to main content
15,890,882 members
Home / Discussions / C#
   

C#

 
GeneralRe: Releasing Memory Pin
Dan Neely29-Sep-05 2:10
Dan Neely29-Sep-05 2:10 
QuestionCopy a Row from one DataView to another? Pin
thomasa28-Sep-05 2:38
thomasa28-Sep-05 2:38 
AnswerRe: Copy a Row from one DataView to another? Pin
Wjousts28-Sep-05 9:24
Wjousts28-Sep-05 9:24 
QuestionObfuscator - Crytografy code Pin
horacyjr28-Sep-05 1:47
horacyjr28-Sep-05 1:47 
AnswerRe: Obfuscator - Crytografy code Pin
Colin Angus Mackay28-Sep-05 2:33
Colin Angus Mackay28-Sep-05 2:33 
GeneralRe: Obfuscator - Crytografy code Pin
horacyjr28-Sep-05 3:27
horacyjr28-Sep-05 3:27 
QuestionModify app.config Pin
Talktorajeev28-Sep-05 1:47
Talktorajeev28-Sep-05 1:47 
QuestionHelp needed with cookies and WebRequest Pin
nzmike28-Sep-05 1:13
nzmike28-Sep-05 1:13 
Hi all,

I have to do some web scraping from a client's web site as they can't give me a direct feed of data (for various reasons too long and borign to explain here!) so I thought I'd use the WebRequest class which I've used before with success.

However, the data I'm scraping is on pages linked 1 or 2 levels down and they require cookies which WebRequest doesn't seem to handle natively. I added a CookieCollection object to my code but when I call my FetchHTML method (below) for a lower level page (ie: linked from the top page) I just get back HTML containing a message saying that cookies have to be turned on. I think I basically just don't really understand what I'm meant to do with the CookiesColelction between requests so if anyone can look at my method below and tell me where I'm mesing up it would be deeply appreciated!

To recap - it loads the first level page in but any successive pages I get HTML back containing the "cookies are not turned on" message rather than the page I want.

Any help would be great!!

Mike

private string FetchHTMLPage(string strURL)
{
//Create a new UTF8 encoding object
System.Text.UTF8Encoding objUTF8 = new System.Text.UTF8Encoding();

request = (HttpWebRequest)WebRequest.Create(strURL);
request.CookieContainer = new CookieContainer();

response = (HttpWebResponse) request.GetResponse();
response.Cookies = request.CookieContainer.GetCookies(request.RequestUri);

Stream resp = response.GetResponseStream();

System.Text.Encoding encode = System.Text.Encoding.GetEncoding("utf-8");

// Pipe the stream to a stream reader with the required encoding format.
StreamReader sr = new StreamReader( resp , encode );
strHTML = sr.ReadToEnd();

return strHTML;
}


AnswerRe: Help needed with cookies and WebRequest Pin
Guffa28-Sep-05 2:18
Guffa28-Sep-05 2:18 
GeneralRe: Help needed with cookies and WebRequest Pin
nzmike28-Sep-05 17:15
nzmike28-Sep-05 17:15 
GeneralRe: Help needed with cookies and WebRequest Pin
Guffa29-Sep-05 22:06
Guffa29-Sep-05 22:06 
QuestionRemoving PropertyGrid border Pin
Rob Cooper28-Sep-05 0:35
Rob Cooper28-Sep-05 0:35 
QuestionNightmare!!! Pin
sean_mufc28-Sep-05 0:12
sean_mufc28-Sep-05 0:12 
AnswerRe: Nightmare!!! Pin
Russell Jones28-Sep-05 4:19
Russell Jones28-Sep-05 4:19 
AnswerRe: Nightmare!!! Pin
Russell Jones28-Sep-05 4:20
Russell Jones28-Sep-05 4:20 
QuestionSend MouseWeel event to another control Pin
nedel27-Sep-05 22:33
nedel27-Sep-05 22:33 
QuestionSaving RTF Pin
Greeky27-Sep-05 21:40
Greeky27-Sep-05 21:40 
AnswerRe: Saving RTF Pin
Dave Kreskowiak28-Sep-05 1:33
mveDave Kreskowiak28-Sep-05 1:33 
QuestionDraw a UML inherit shape use with GDI+ Pin
dreamwinter27-Sep-05 21:39
dreamwinter27-Sep-05 21:39 
AnswerRe: Draw a UML inherit shape use with GDI+ Pin
Christian Graus28-Sep-05 1:54
protectorChristian Graus28-Sep-05 1:54 
GeneralRe: Draw a UML inherit shape use with GDI+ Pin
dreamwinter29-Sep-05 15:22
dreamwinter29-Sep-05 15:22 
GeneralRe: Draw a UML inherit shape use with GDI+ Pin
dreamwinter29-Sep-05 15:32
dreamwinter29-Sep-05 15:32 
QuestionADO.NET Transaction question Pin
seee sharp27-Sep-05 20:24
seee sharp27-Sep-05 20:24 
QuestionError While opening a Dot net Project Pin
vinidimple27-Sep-05 19:36
vinidimple27-Sep-05 19:36 
AnswerRe: Error While opening a Dot net Project Pin
sean_mufc28-Sep-05 0:17
sean_mufc28-Sep-05 0:17 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.