I'm building a Windows application that will connect to a website using private proxies and do stuff on that site. I want to use the new HTTPClient to do that. I have the next scenario for the login part:
I GET the login page. The login page sets a cookie (I need to store that).
In the code of the login page, I find some dynamic parameters that I scrape using Regex.
I create the POST url using the parameters that I scraped and a user and a password.
I use the cookie that the login page stored and I POST to the website (the created URL).
I receive the content of the response and if the login is done correctly, I see in the code a redirect url (the http web request will not follow this redirect!) and a cookie is stored.
I scrape the redirect url and I use the cookies stored with the POST and do a GET request to the redirect url. Another final cookie is stored (overwritten the last one).
I have tried to do this with httpwebrequest but after the POST, I get the response and when I load that response in a browser, I see "your browser has the cookies functionality turned off" so I obviously do not know how to store my cookies and use them with the next request, then store the new cookies and use them with the last request ...
Now I try to do all over again with HTTPClient... can anyone help me with an example using proxy (with proxy user and proxy password) for all 3 requests and with cookie functionality ??
Please HELP!!!
EDIT: After some sleep and more research, I found out why I could not find any example for httpclient in VB.NET: httpclient is implemented only in C#, C++ and JS... so please help me with httpwebrequest (my initial approach).
EDIT:
Strange - In C#, I have HTTPClient by default, but now I managed to install it for VB.NET using NuGet ... so the question remains: Does anyone know how to use it effectively?
This is my code - the version that returns: "cookies functionality disabled, please enable cookies".
Imports System.Net
Imports System.Text
Imports System.IO
Imports System.Runtime.Serialization.Formatters.Binary
Imports System.Text.RegularExpressions
Imports System.Web
Namespace Functions
Public Class Functions
Public xuser As String
Public xpassword As String
Public xproxy As String
Public xport As String
Public xproxypassword As String
Public xproxyuser As String
Public xphonenumber As String
Public logincookies As CookieContainer
Dim UA As String = "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:22.0) _
Gecko/20100101 Firefox/22.0"
Public Function login(ByVal user As String, ByVal password As String, _
ByVal proxy As String, ByVal port As String, ByVal proxyuser As String, _
ByVal proxypassword As String, ByVal phonenumber As String)
xuser = user
xpassword = password
xproxy = proxy
xport = port
xproxypassword = proxypassword
xproxyuser = proxyuser
xphonenumber = phonenumber
logincookies = New CookieContainer
Dim loginpage As String = "https://theloginpageURL"
Dim request As HttpWebRequest = _
DirectCast(WebRequest.Create(loginpage), HttpWebRequest)
request.UserAgent = UA
request.Referer = "http://thepagethatreferedmeURL"
request.KeepAlive = True
request.CookieContainer = New CookieContainer()
request.CookieContainer = logincookies
Dim response As HttpWebResponse = _
DirectCast(request.GetResponse(), HttpWebResponse)
logincookies.Add(response.Cookies)
Dim reader As New StreamReader(response.GetResponseStream())
Dim loginpagecontent As String = reader.ReadToEnd
Dim pageGetLogin As String = loginpagecontent
Dim param1 As String = "this is a parameter value"
Dim param2 As Object
Dim param2fin As String
param2 = Regex.Match(pageGetLogin, _
"(?<=param2""\s* value="")[\w\W]*?(?="")")
param2fin = param2.ToString
Dim param3 As String = "this is a parameter value"
Dim param4 As String = "this is a parameter value"
Dim encoding As New UTF8Encoding
Dim postLoginRequest As HttpWebRequest = _
DirectCast(WebRequest.Create
("https://website.com/loginservice?param1=" _
& param1 & "¶m2=" & param2fin & "¶m3=" & param3 & "¶m4=" _
& param4 & "&Email=" & Email & "&Passwd=" & Passwd & "&signIn=" _
& signIn), HttpWebRequest)
postLoginRequest.Method = "POST"
postLoginRequest.KeepAlive = True
postLoginRequest.AllowAutoRedirect = True
postLoginRequest.MaximumAutomaticRedirections = 50
postLoginRequest.ContentType = _
"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"
postLoginRequest.Referer = "https://theloginpageURL"
postLoginRequest.UserAgent = UA
postLoginRequest.ContentLength = 0
postLoginRequest.AutomaticDecompression = DecompressionMethods.GZip
postLoginRequest.CookieContainer = New CookieContainer
postLoginRequest.CookieContainer = logincookies
Dim postLoginResponse As HttpWebResponse
postLoginResponse = _
DirectCast(postLoginRequest.GetResponse(), HttpWebResponse)
logincookies.Add(postLoginResponse.Cookies)
Dim postLoginReader _
As New StreamReader(postLoginResponse.GetResponseStream())
Dim loginResponsePageContent As String = postLoginReader.ReadToEnd
MessageBox.Show(param1 & "--" & param2 & "--" & user _
& "--" & xport & "--" & xproxyuser & "--" & xproxypassword)
Dim filename As String = "d:/" & "cookie.dat"
If (File.Exists(filename)) Then
File.Delete(filename)
End If
Dim stream As FileStream = File.Create(filename)
Dim formatter As New BinaryFormatter()
formatter.Serialize(stream, response.Cookies)
stream.Close()
Return loginResponsePageContent
End Function
End Class
End Namespace
I have replaced all the URLs in the code and most of the commented code that I used in later trials. I removed the parameters also and I replaced them with similar ones (also I formatted the post url to match the real one in aspect).
I call this class from a button on a form and I supply the response variable to a web browser component just to see how the response looks (I'm too tired to read the code in a text editor by now)).
The cookie that is saved is the first cookie that the login page set so I know that the first cookie is saved in the container.
The
GET
part is working as it should and I receive the response and I parse it and obtain all the parameters, I can verify that I have them by showing them in a
messagebox.show()
The
POST
it seem to be ok but I receive the "cookies disabled" message on the response html...
Please show me where I'm doing things wrong and how to fix it...
I can't post the real URLs and the entire code... I think an experienced programmer will spot the mistakes from this code.
Thank You!!
PS - This code does not have the next
GET
as I can't receive a good response from the
POST
so I can't find the redirect URL and make the next
GET
request.