Web/HTTP Request and Automation Libraries

• Download JavaScript source - 3.87 KB
• Download VBScript source - 6.47 KB

Introduction

While working on web/HTTP automation a while back, I noticed that there weren't any available solutions for making HTTP requests with support for session state persistence in VBScript and .NET. You can use cookies in .NET's web request or web client classes, but to use them to support session state, you would have to write your own wrapper interface or library. VBScript can use Microsoft's version of XmlHttpRequest, but that doesn't come with built in support for cookies, so you have to write your own interface or library. I did find a commercial version for VBScript that did support cookies and session state persistence (see the Resources section). JavaScript can have support for session state persistence, but you have to code it yourself or use one of the many available AJAX libraries. I personally find the AJAX stuff and using XmlHttpRequest too much of a hassle when all you really want to do is make a simple HTTP request. So, I wrote a simplified library for doing so (it's not meant for AJAX or asynchronous requests though), based on some code for ASP.NET I found on the web (see the Resources section).

This article presents simple and good libraries for making HTTP requests with support for session state persistence for .NET, VBScript, and JavaScript. Some options and resources are given for C/C++ and Java as well. The article is broken down into sections describing the library for each language/platform:

• .NET
• VBScript
• JavaScript
• C/C++
• Java

Note: These libraries and the accompanying files are provided "as is", with limited support through this article's forum.

Background

This is a follow up to my article Web/HTTP Automation with Perl. At the time, the immediate solution was to use Perl, since that had support for session state and I knew Perl. Well now, people can have more choices.

.NET

I shall be humble to say that it seemed to be quite a challenge to develop a wrapper library to the .NET network/web/HTTP request classes, etc. And I never found time to work on this. However, on a recent web search, I found a solution that seems to offer what I intended. Rather than reinvent the wheel, I'll reference the solution for you to explore. The solution includes source code but does not offer a precompiled binary of the library code. You can either compile one yourself or include the code file in your project. The file of interest is wwHTTP.cs.

• WestWind Technologies White Paper: Retrieving HTTP content in .NET.

VBScript

The library (httpLib.vbs) provides an easy wrapper to Microsoft's implementation of XmlHttpRequest. With it, you don't need to know the details of XmlHttpRequest. All you need to know is how to make a GET and a POST request. The library provides two methods:

httpGet(url, ByRef cookie, debug)
httpPost(url, data, ByRef cookie, debug)

The two methods are pretty simple to use. Just pass the URL, and for POST requests, the data as well. The data parameter follows the GET/POST data format of var1=value1&var2=value2. For cookie and session state support, you can supply a cookie (by reference). The cookie is simply a text string that follows the HTTP RFC cookie format. For session state support, you would pass the first request an empty cookie. The HTTP response will return a valid cookie which you can resend for subsequent requests to establish session state. If you don't need cookies or session state support, send an empty cookie or "". The debug parameter is a boolean for whether you want the HTTP headers from the response. If set to true, it will return the headers along with the response content, with headers coming first. If set to false, only the response content is returned.

The library works with ASP, Windows Scripting Host, and within Internet Explorer via HTA applications or an HTML webpage. All you need to do is reference the library with a script tag.

For simplicity, I've ported the web automation parsing script from my Perl article to VBScript. So you can use it to run the script as follows:

cscript ParseHttpTrace.vbs <pathToTraceFile> <optionalPathForOutputScript>

For details on how to use this script, refer to the Perl article mentioned in the Background section, as this script does the same thing as that article but generates code for VBScript instead.

JavaScript

The library (xmlHttpLib.js) provides an easy wrapper to XmlHttpRequest. With it, you don't need to know the details of XmlHttpRequest. All you need to know is how to make a GET and a POST request. The library provides two methods:

httpGet(url, cookie, respType)
httpPost(url, data, cookie, respType)

The two methods are pretty simple to use. Just pass the URL, and for POST requests, the data as well. The data parameter follows the GET/POST data format of var1=value1&var2=value2. For cookie and session state support, you can supply a cookie. The cookie is simply a text string that follows the HTTP RFC cookie format. For session state support, you would pass the first request an empty cookie. The HTTP response will return a valid cookie which you can resend for subsequent requests to establish session state. If you don't need cookies or session state support, send an empty cookie or "". The respType parameter determines the type of content returned from the response. Supported and valid values for this parameter are: "text", "xml", and "json". When using XML or JSON, the returned content will be an XML DOM or JSON object, available for immediate use.

The HTTP response is returned in a custom object defined as follows:

//Constructor for HTTP response object
function httpResp(sessionCookie, httpHeaders, content, status)
{
  this.cookie = sessionCookie;
  this.headers = httpHeaders;
  this.content = content;    //can be text, XML, or JSON
  this.statusCode = status;
}

The response parameters can then be accessed as follows:

myResp.cookie //can get/set this cookie value
myResp.content //can get/set this value for "text" return type
myResp.content.jsonObjMemberName //access JSON obj member
myResp.content.xmlDomMethodOrProperty //access result via XML DOM

Rather than go into detail on how to support cookies, session state persistence, and JSON here in the article, refer to the sample script provided (sampleJsUa.wsf) that gives you a demo of how those features work. The script is a Windows Scripting Host file though, so you can only run it on Windows. It is best you run it from the command line with cscript as well since it outputs a lot of text, you don't want to get a ton of popup message boxes using wscript.

For simplicity and cross-platform support, the library includes three additional methods:

xmlHttp()
print(txtStr)
xmlObj(text)

The first method will return the XmlHttpRequest object appropriate for your platform (Windows/IE or Mozilla/Firefox). The second method will output or print text as appropriate for the platform (echo to the command line or display a popup message box for Windows Scripting Host, send an alert() to the browser, or write to the HTTP response for ASP). The third method will return the XML DOM object appropriate for your platform (Windows/IE or Mozilla/Firefox). Therefore, so to speak, the library works with ASP, Windows Scripting Host, Internet Explorer HTA applications, or HTML web pages via Internet Explorer or any Mozilla-based browser. All you need to do is reference the library with a script tag.

Note: This library is not an AJAX library. It's only for making HTTP requests.

C/C++

I am not a C/C++ library developer, so I don't offer a C/C++ version here, but I do have some information that can help you make web/HTTP requests or develop your own library. You can try WinHTTP from Microsoft. It provides a C/C++ API and appears to be very similar to XmlHttpRequest. It also offers a COM object interface, but in that case, you could try XmlHttpRequest as well. You can build your own wrapper using the API or COM interface modeled after my VBScript and JavaScript libraries here. I may, at some point in the future, attempt to develop a C/C++ library for this, but no guarantees.

• Windows HTTP Services (WinHTTP) MSDN documentation

Java

Like .NET, Java has built-in network/HTTP web request and response classes. And like .NET, it also doesn't offer a simple class interface to make HTTP requests with managed cookie support. I shall be humble to say that it seemed to be quite a challenge to develop a wrapper library to the Java network/web/HTTP request classes, etc. And I never found time to work on this. However, on a recent web search, I found a solution that seems to offer what I intended. Rather than reinvent the wheel, I'll reference the solution for you to explore: HTTPClient (for Java).

Resources

• This is a follow up to my article: Web/HTTP Automation with Perl.
• The following blog is where I obtained the original source code for parsing cookies out of an HTTP request (which is the basis for the development of these libraries): http://blog.netnerds.net/2006/04/asp-sustain-remote-cookie-sessions-in-an-asp-script-using-vbscript/
• A commercial HTTP request library for VBScript: http://www.tele-pro.co.uk/scripts/misc/httpsess.htm.
• .NET implementation of a good web/HTTP request library: WestWind Technologies White Paper: Retrieving HTTP content in .NET.
• C/C++ option for making web/HTTP requests: Windows HTTP Services (WinHTTP) MSDN documentation.
• Java implementation of a good web/HTTP request library: HTTPClient (for Java).

History

• July 14, 2007 - Initial post of article with initial release of VBScript and JavaScript libraries.
• October 27, 2007 - Revised JavaScript library and updated article for that.
• November 21, 2008 - Article update for solutions in .NET, Java, and C/C++.
• January 8, 2010 - Article update for Java solution.