Having scraped hundreds of sites, I might disagree with your tactic on trying to bot the page controls. Instead, you can manually construct your POST request and populate the content/payload appropriately. For example, in Google Chrome, visit the page you would launch your POST request from. On that page, right-click and select "inspect". Then click on the "Network Tab" to see all the requests. Perform the POST manually and then click on that request in the resultant list of network activity. Click the "Headers" tab on the right to see all headers. On a request of type POST, you will see a section called "Form Data". That is the POST content that you want capture. You would paste that as a template into your own POST request. So for example to insert your own date into the content, your routine might look something like:
public static HttpWebRequest CreatePOSTRequest(string url, DateTime someDate)
{
HttpWebRequest request = (HttpWebRequest)WebRequest.Create("someURL");
request.Method = "POST";
string payload = "somepastedcontent;somedatevalue=" +
someDate.ToString() +
";somemoretemplatestuff";
byte[] byteArray = System.Text.Encoding.UTF8.GetBytes(payload);
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = byteArray.Length;
using (System.IO.Stream dataStream = request.GetRequestStream())
{
dataStream.Write(byteArray, 0, byteArray.Length);
}
return request;
}
Note, you would need to set the above request.ContentType to your target's content type listed in the Headers. Then to execute the above web request you do something like:
string responseText;
WebResponse response = (HttpWebResponse)request.GetResponse();
using (Stream stream = response.GetResponseStream()) {
using (StreamReader reader = new StreamReader(response.GetResponseStream(),
System.Text.ASCIIEncoding.UTF8))
{
responseText = _ConditionResponse(reader.ReadToEnd());
}
}
I would be happy to provide specific help if you provide the page you want to scrape. Otherwise good luck!