|
I need to download HTML pages from a website. These pages require username/password access which I DO have. The problem is, the website sets a persistent cookie in my Cookies folder and it's encrypted, so I can't read it's content.
How can I automate the download of these pages but not through internet explorer? In other words, how can I make WebClient/HttpWebRequest read the persistent cookie in my local folder?
If I can't do that, then how do I do it?!
Thanks.
Sammy
"A good friend, is like a good book: the inside is better than the cover..."
|
|
|
|
|
First of all, the cookie should NOT have the password contained within it. This is very insecure. The web site should authenticate the credentials in whatever way is fit (say, BASIC authentication of HTTPS) and return a ticket that signifies that the credentials have been authenticated. This is the underlying concept of authentication.
Second, if you have the username and password, you could make the call to the web server to get the ticket and that should return you the cookie. In order to accept the cookie, though, your HttpWebRequest.CookieContainer must be initialized:
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("http://www.codeproject.com");
request.CookieContainer = new CookieContainer();
HttpWebResponse response = (HttpWebResponse)request.GetResponse();This example if very basic, but you can find more information in the documentation for HttpWebRequest.CookieContainer.
Additionally, you can also pass the username and password to the initial request to avoid making several requests. You'll still need the initialized CookieContainer to store the response cookies (and use that over for every request and response), but you can put the username, password, and domain in a NetworkCredential and assign that to the HttpWebRequest.Credentials property before getting the response (which commits the request). See the documentation for the HttpWebRequest.Credentials property in the .NET Framework SDK for more information.
Microsoft MVP, Visual C#
My Articles
|
|
|
|
|
I'm going nuts 
Nothing is working. The URL I'm trying to get is the following:
http://search-completed.ebay.com/search/search.dll?GetResult&sosortorder=2&sacategory=29278
It retrieves certain completed auctions from eBay.
This URL however, results in a page that says: you have to sign in. I get the source of this page, I see that it gets a posting (username/password) from you, and redirects you to the original page you were requesting. I post this data, it gives me a Redirect Error, although I have HttpWebRequest.AllowAutoRedirect set to true. I also tried KeepAlive = true/false. No luck.
So, I try posting my username/password to http://signin.ebay.com/aw-cgi/eBayISAPI.dll which is the sign in page, and getting the cookies returned. It gives me a Cookie Error page, where it says my browser cannot accept cookies, although I did set CookieContainer of my HttpWebRequest.
What can I do?!!! If anyone wants to try:
username = booksbysammy
password = stupiddog
Thank you very much.
Sammy
"A good friend, is like a good book: the inside is better than the cover..."
|
|
|
|
|
Okay, there's a few things I have to mention. When using form-based authentication, the HttpWebRequest.Credentials won't suffice. You either have to POST or GET data to the form action as you've discovered. I just wanted to mention this to be clear.
The problem looks like it doesn't recognize your user agent (the browser name and info). Fortunately, you can trick into thinking another browser is making a request (i.e., one that supports cookies) by the HttpWebRequest.UserAgent to something like "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)" and it will think Internet Explorer 6.0 on Windows XP is making the request. This error might be making such assumptions because it's quicker than setting a cookie and checking for it in the next request (which requires making a request and getting a response twice). It's worth a try.
Microsoft MVP, Visual C#
My Articles
|
|
|
|
|
Heath Stewart wrote:
When using form-based authentication, the HttpWebRequest.Credentials won't suffice.
I'm sorry, I don't get you there. I don't use HttpWebRequest.Credentials at all in my code. Am I doing something wrong?
Heath Stewart wrote:
Fortunately, you can trick into thinking another browser is making a request (i.e., one that supports cookies) by the HttpWebRequest.UserAgent to something like "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"...
I already did this, no luck
Sammy
"A good friend, is like a good book: the inside is better than the cover..."
|
|
|
|
|
I was saying that when you use form-based authentication (like putting your credentials in a form online) the Credentials property won't work to pass your username and password. That only works for authentication protocols like BASIC or NTLM (and this is extensible, too).
Did you set the UserAgent before making the request to the ISAPI extension DLL on eBay to authenticate your credentials? Also, you should'be set an instance of the CookieContainer on the HttpWebRequest.CookieContainer property, too, but I believe you said you already did this.
Are you saving the request the web server returns? What exactly is the error message?
Microsoft MVP, Visual C#
My Articles
|
|
|
|
|
HttpWebRequest req = (HttpWebRequest)WebRequest.Create("http://signin.ebay.com/aw-cgi/eBayISAPI.dll");<br />
req.AllowAutoRedirect = true;<br />
req.CookieContainer = new CookieContainer();<br />
req.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)";<br />
req.ContentType = "application/x-www-form-urlencoded";<br />
req.Method = "POST";<br />
byte[] PostData = System.Text.Encoding.ASCII.GetBytes("MfcISAPICommand=SignInWelcome&siteid=0&co_partnerId=2&UsingSSL=0&ru=&pp=&pa1=&pa2=&pa3=&i1=-1&pageType=559&userid=booksbysammy&pass=stupiddog&keepMeSignInOption=1");<br />
req.ContentLength = PostData.Length;<br />
Stream tempStream = req.GetRequestStream();<br />
tempStream.Write(PostData, 0, PostData.Length);<br />
tempStream.Close();<br />
HttpWebResponse resp = (HttpWebResponse)req.GetResponse();<br />
Stream strm = resp.GetResponseStream();<br />
StreamReader sr = new StreamReader(strm);<br />
textBox1.Text = sr.ReadToEnd();<br />
strm.Close();<br />
sr.Close();
(By the way, are the last two lines valid, or should I also add strm = null; and sr = null; ?
Heath Stewart wrote:
What exactly is the error message?
Well, it's not really an error message. The point is, after executing the following code, textBox1.Text contains the HTML source of an eBay page the says that my browser has rejected cookies, and there's a "Cookie Error" according to eBay. When I post these settings in Internet Explorer, that's not what I get of course!
By the way, there's one field I'm not adding to post data, which says <input type="submit" value="Sign In >">. As you can see, the submit button does not have a name assigned to it. I don't suppose that's the problem?!
Sammy
"A good friend, is like a good book: the inside is better than the cover..."
|
|
|
|
|
Actually, "application/x-www-form-urlencoded" uses the UTF-8 encoding, so use Encoding.UTF8 to get the bytes instead and see if that helps. Since you're only using ASCII characters, though, it might not. It's worth a try, anyway.
No, you don't have to set the object to null. Calling Close closes the stream (closes the native handle). The GC will clean it up once it goes out of scope and either the application is idle or garbage collecting is necessary.
Including that submit value shouldn't be necessary.
I'm really not sure what else to tell you. It should be working according to the documentation and from experience.
The only other thing I can think of is that there might be another cookie (like a session ID) that eBay is expecting. Log out of eBay using Internet Explorer and look at the Cookies directory for a file named username@ebay.com.txt or something like that. Open it in notepad and see what values it has (if any).
Microsoft MVP, Visual C#
My Articles
|
|
|
|
|
Heath Stewart wrote:
I'm really not sure what else to tell you. It should be working according to the documentation and from experience.
I really appreciate you taking all this time to reply. And, as I said before, you are my hero  Well, your wife should really say that, so it should be inappropriate to come out of me 
Heath Stewart wrote:
The only other thing I can think of is that there might be another cookie (like a session ID) that eBay is expecting. Log out of eBay using Internet Explorer and look at the Cookies directory for a file named username@ebay.com.txt or something like that. Open it in notepad and see what values it has (if any).
Okay, I thought about that. Before logging in, here's sammy@ebay[1].txt from my c:\document and settings\...\cookies folder.
lucky9
518210
ebay.com/
1600
2767397504
29984326
4106446384
29616796
*
nonsession
AQAAAAQAAACkAAAANwAAAIjBH0AI9QBCMUA1Njc0MDM3OTA7ICQyJE1vemlsbGEvJFRiLnNZVTJuM
jloVE4zT0M1VnhCZjFAAAAAOAAAAIjBH0AI9QBCMWJvb2tzYnlzYW1teSAkMiRNb3ppbGxhLyRvTl
kwQzlkNFdtcFpjLlI2Q2F4WG8vnQAAABEAAACIwR9ACPUAQjAwMDAwMDAwMKoAAAAKAAAAiMEfQAj
1AEIwMA**k
ebay.com/
1536
1819020288
29690222
4098476384
29616796
*
ns1
AQAAAAEAAAClAAAAOAAAAIjBH0AI9QBCMTEwODMyMDg4My8wOyAkMiRNb3ppbGxhLyRoVGtncEoyc
E5DMHlVVlRyR3ZSR1Muf
ebay.com/
1536
1819020288
29690222
4098476384
29616796
*
reg
%5EflagReg%3D1%5E
ebay.com/
1600
2767397504
29984326
4106286384
29616796
*
CFID
47490547
ebay.com/
1536
3546759168
32088942
1680224000
29616625
*
CFMAGIC
47490547%3A62005554
ebay.com/
1536
3546759168
32088942
1680384000
29616625
*
CFTOKEN
62005554
ebay.com/
1536
3546759168
32088942
1680694000
29616625
*
After logging in, it was deleted, and replaced with sammy@ebay[2].txt which reads:
lucky9
518210
ebay.com/
1600
3287397504
29984326
329909088
29616797
*
nonsession
AQAAAAYAAABAAAAAOAAAAIjBH0AI9QBCMWJvb2tzYnlzYW1teSAkMiRNb3ppbGxhLyRvTlkwQzlkN
FdtcFpjLlI2Q2F4WG8vpAAAADwAAAApEyBAqUYBQjFANTY3NDAzNzkwOyAkMiRNb3ppbGxhLyRUYi
5zWVUybjI5aFROM09DNVZ4QmYxAM3Nzc2qAAAACgAAACwTIECsRgFCMDCdAAAAEQAAACwTIECsRgF
CMDAwMDAwMDAwEAAAAD0AAAAsEyBArEYBQjFib29rc2J5c2FtbXkgJDIkTW96aWxsYS8kb05ZMEM5
ZDRXbXBaYy5SNkNheFhvLwDNzc3NnAAAAGkAAAAsEyBArEYBQjFuWStzSFoyUHJCbWRqNndWblkrc
0VaMlByQTJkajZ3RmtJcWlDcENLcVErZGo2eDluWStzZVE9PSAkMiRNb3ppbGxhLyRaMHBkWUhxSV
FTNUFzLnFmNFJlSzYuAM3Nzc0*l
ebay.com/
1536
2339020288
29690222
323039088
29616797
*
ns1
AQAAAAEAAAClAAAAPQAAACkTIECpRgFCMTEwODMyMDg4My8wOyAkMiRNb3ppbGxhLyRoVGtncEoycE5DMHlVVlRyR3ZSR1MuAM3Nzc0*j
ebay.com/
1536
2339020288
29690222
323189088
29616797
*
reg
%5EflagReg%3D1%5E
ebay.com/
1600
3287397504
29984326
329599088
29616797
*
CFID
47490547
ebay.com/
1536
3546759168
32088942
1680224000
29616625
*
CFMAGIC
47490547%3A62005554
ebay.com/
1536
3546759168
32088942
1680384000
29616625
*
CFTOKEN
62005554
ebay.com/
1536
3546759168
32088942
1680694000
29616625
*
Since I'm stupid, I hope this means anything to you!!!
Sammy
"A good friend, is like a good book: the inside is better than the cover..."
|
|
|
|
|
Okay, I didn't want to make the page look ugly, so I modified those lines of gibberish (inserted line breaks). So they're actually on one line originally.
A question out of topic please: Do you use IE to read these threads and reply to them? It seems to me that you should have some sort of client app that reads new threads off Code Project and lets you post directly?
Thanks again.
Sammy
"A good friend, is like a good book: the inside is better than the cover..."
|
|
|
|
|
I figured they were wrapped.
I do use IE, yes. A better way than a separate program is to use an NNTP gateway (newsgroups) since these threads follow the same behavior already. Then you could use any NNTP client. Such a task isn't the easiest thing to do, though.
Microsoft MVP, Visual C#
My Articles
|
|
|
|
|
Hi,
I'm developing an Explorer bar using VS.NET 2003 (C#) on Windows XP. For
some time, I've noticed that I don't have filenames and line numbers
appearing in my exceptions' stack traces. On the other hand, debugging with
breakpoints, etc. seems to work fine in VS.NET 2003. I can't understand why
this isn't working -- and I think it used to work, perhaps under VS.NET
2002, but maybe not. The information doesn't appear in exceptions when
running under VS.NET or when not.
I am running my Explorer bar (which is a class library, not an application)
on the same machine (and user account) which built the library. The PDB
file is accessible to me, and its location does match the location in the
assembly. I am making Debug builds, and I do have "Generate debugging
information" enabled. As far as I can tell, the PDB and DLL files are
equally fresh and the same version?
Any tips? Is this a known bug? It's driving me batty trying to locate an
exception in code blocks without any help from line numbers.
Thanks!
Arun
|
|
|
|
|
When you debug your shell extension, are you debugging a new instance of the shell (explorer.exe)? If the release assembly was registered and the shell invoked it it will continue to use it until the reference count is zero. The only way to do this is kill your shell or be sure to start a new shell when debugging. So long as you're on the same machine, the PE/COFF header (not the assembly) will point to the path of the PDB for debug builds). Since you're sure that it's pointing to the right place, I figured the release version might be registered and running currently.
Otherwise, I can't think of any reason besides your debugging environment being hosed. If you're still getting runtime exceptions (which shouldn't happen in production code, of coruse) that prompt to start a debugger, I doubt this is the case.
If none of the above are the case, when debugging your app, look at the Output window. Do you see where it loads your assembly? Does it say "Symbols loaded" or something similar, or that they can't be found?
Microsoft MVP, Visual C#
My Articles
|
|
|
|
|
Thanks for responding. I do see "Symbols loaded" for my assembly when debugging my assembly, but even still, the line numbers don't appear in the exception's StackTrace.
I'm pretty sure that I'm using the Debug build, as I make a point of not popping up exception-laden message boxes in the release builds.
I even tried rebooting my computer yesterday after building and installing a new Debug release of my browser bar.... just in case.
I started a new solution for a Windows application and was able to get line numbers in the stack trace, so at least I know that the problem is limited to my solution.
|
|
|
|
|
If you took many of the defaults when installing VS.NET (or have installed the Platform SDK), you should have a utility called dumpbin.exe. If you do, type dumpbin.exe /pdbpath <dll path> and see where it thinks the PDB file is located. If it uses a relative path (and I think the PATH directories can contain it, too...don't remember off the top of my head), try moving the PDB file into the same path as the executable (or try another PATH directory). Also make sure that your assembly is registered using regasm.exe /codebase (registering this from the project options uses this switch too).
Microsoft MVP, Visual C#
My Articles
|
|
|
|
|
Okay, there's the ticket!
dumpbin.exe was telling me that the PDB path was relative (.\). So I copied my PDB file into the GAC, right next to my DLL, and it worked. I didn't really expect that to work, but I guess the GAC is more lenient than it seems.
What does the /codebase switch get me? I'm starting to question my current practices, even though they've been working.
I install my assembly like so:
<br />
gacutil /nologo /if VisIT.dll<br />
regasm VisIT.dll<br />
According to C# in a Nutshell, I shouldn't use /codebase if the assembly is in the GAC or if I plan to add it to the GAC. I wonder if I need it in the GAC. I thought I did -- or that it seemed like a good idea -- since this is for a shell extension.
Thanks,
Arun
|
|
|
|
|
The /codebase switch tells regasm.exe to register the full path to the assembly in the registry. Since your assembly is in the GAC, you really don't need this (and I would recommend against it).
IMO, you should refrain from testing assemblies in the GAC. Until you get your explorer bar working, just select the option in your project settings to register it (again, this uses the /codebase switch or at least duplicates the behavior) and test against that until you're ready to deploy it with an installation or something. Sure gets rid of problems like this and keeps your GAC clean, especially if you're using automatic versioning (using the asterisk in the AssemblyVersionAttribute), which I've learned causes more problems than not (especially large solutions where you update and redirect assembly bindings regularily).
Microsoft MVP, Visual C#
My Articles
|
|
|
|
|
Okay, I'll try to avoid the GAC for now, even though it's been okay for me for the past year.  For a short while, I was trying out automatic versioning, but then I turned it off because my installer used a hardcoded assembly version number, and I never got around to finding a dynamic workaround.
Thanks for your helpful persistence.
|
|
|
|
|
If you're using the Windows Installer project in VS.NET, add the output of your project to the installer and it is updated whenever you recompile your C# project (it becomes a dependency of the installer project).
I'm not saying the GAC doesn't work, I'm just saying that it can lead to problems until you're done testing. You've already had one. If you use automatic versioning you'll have lots of assemblies (differing only by version) in your GAC. It doesn't hurt anything, but it sure clutters up the GAC. It's just a recommendation that's not uncommon when debugging applications, especially in larger teams.
Microsoft MVP, Visual C#
My Articles
|
|
|
|
|
I have a WinForm application which appears fine on my computer and several others, but I have found one computer on which some of the controls appears "cut-off".
I have all the controls on the application anchored so that when i view the application on my computer, it looks fine no matter how I resize it. It seems as though on this one other computer that the form is not following the anchoring I had set up. The strange part is, that this only happens on one tab of the application. (There are several other pages in the TabControl set up in the same manner which display fine on all computers). Also, when this "problem computer" is used to view the application via RAdmin on another computer, it appears as it is supposed to.
Any ideas on what could be going on here???
|
|
|
|
|
~michL~ wrote:
Any ideas on what could be going on here???
Screen resolution.
Since .NET uses pixels and pixels display differently on different resolutions, the layout of the form can't truly be garaunteed. This is why some languages like VB6 (and below) use twips, points, and other logical units. The anchoring won't matter in this case because the initial positions and sizes of the controls are correct in relation to pixels. It's pretty hard to explain.
I've actually never seen that big of a problem before. What resolution are you using? What resolution is the other machine using?
If necessary, you can always P/Invoke the GetDeviceCaps native function (for which the SystemInformationClass does, but doesn't support everything) and get the number of pixels per whatever logical unit you like. The reason it works when you use RDP (termainal services) is because you can specify a resolution in which to work when you connect. The graphics card on the target machine should support your target resolution, so you might want to try using that resolution locally on that machine.
Microsoft MVP, Visual C#
My Articles
|
|
|
|
|
Unfortunately, all resolutions on that particular computer display incorrectly so it could not be fixed that easily...meanwhile, on other computers all resolutions do display correctly. I had been using 1280x1024, while the other computer had a res of 1024x768, but like I said, even when changing the resolution of the other computer to 1280x1024, the problem was not fixed.
And since the resolution doesn't solve the problem, it seems that GetDeviceCaps would not solve it either.
|
|
|
|
|
There's more to calculating pixels per logical unit. What is the dpi of the monitor? Is the right driver configured for the display adapter and the monitor itself? These all have a major impact on how things are displayed on your monitor. There's plenty of information about this in the Platform SDK.
Microsoft MVP, Visual C#
My Articles
|
|
|
|
|
I have a Windows form in which I have 5 Button controls, named B1, B2,
B3 .... B5. I want to programatically iterate through these controls,
where I have to work on their properties. Initially all Buttons need
to be invisible. But then iteratively, I want them to be visible. A long handed version is as follows:
while (dr.Read()) {
B1.Visible=true;
B2.Visible=true;
B3.Visible=true;
B4.Visible=true;
B5.Visible=true;
}
I want the following:
while (dr.Read()) {
int i=1;
Bi.Visible=true; //this is just my intent, not an actual statement
i=i+1;
}
where dr is a SqlDataReader. I want i to iterate from 1 through 5, and
then all Buttons are visible. I tried concatenating string to form B1,
B2 etc. programatically, but that does not work.
Help appreciated.
|
|
|
|
|
There are many ways you can do this, such as keeping an array or collection of specific controls you're interested in, or checking the type of the Control in the Controls property of the container Control (like a Form, UserControl, Panel, etc.) like so:
foreach (Control c in Controls)
{
if (c is Button)
{
Button b = (Button)c;
b.Visible = true;
}
}If you only want specific names, make sure the Name property of the Button is assigned and compare that with "B1" through "B5" to make sure it's a Button you want to control.
Microsoft MVP, Visual C#
My Articles
|
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
