Download your own free SOAPSonar

Introduction

A Web service is a piece of software that provides interoperability between applications running on disparate platforms. Bound to a public URL, a Web service supports a set of functions described through a public contract. Callers connect to the Web service’s endpoint, and invoke any of the contracted functions. This simple mechanism can work across disparate platforms only if common standards are employed to define the exposed contract, format, and layout of the messages being exchanged.

A number of industry standards have been developed over the past few years to favor Web services interoperability and take it to the next level. These standards address several aspects of Web services and Web services development, including security, reliability, and interoperability.

The idea of Web services, and the first early implementations, date back to late 90s. Today, Web services are definitely part of our everyday life and activity. Web services, for example, enable thousands of people to interact with popular Web sites (Google, Amazon, MapPoint, PayPal, eBay to name just a few) and extract information without even bothering a browser-guided visit.

The bottom line is that when you publish a Web service, like it or not, you may also put the heart of your system at the mercy of any people out there with access to the Internet. Two opposite forces stretch you.

On one end, the force of business pushes you to keep it simple for potential customers. You don’t want to bother potential customers with checks and crosschecks, boundaries, warnings, and limitations of any sort. You want the maximum reach to increase your business, and have investments made on software and Web services pay off quickly and effectively.

On the other end, the force of security suggests you stay on the safe side and raise the bar higher and higher not to compromise the core business and the core information system.

How would you deal with these two opposite but roughly equally strong forces? Should you simply further one over the other? That would be too easy. You can’t afford a binary, black-or-white choice. Both forces are essential to any company; especially once the company has matured the decision to go live with a set of well-done Web services.

Well-done! That’s the key word.

What’s Critical with Web Services?

A well-done Web service is primarily a Web service that works and delivers exactly what its WSDL contract promises to. A Web service is typically a façade that collects messages from callers and forwards to a back-end infrastructure and, ultimately, to the heart of the company’s information system. A Web service represents the face of the company exposed to the outside world.

Hacking the Web service might result in some damage to the company. Reaching a vast audience might result in some success for the company. The quality and performance of the Web service might influence the perception of the company in the real world.

Business-wise speaking, and aside from raw functionality, there are four fundamental parameters to evaluate a real-world Web service: vulnerability assessment, compliance to standards, quality assurance (QA), and overall performance. You address them at development time by applying best practices and using software development kits to their maximum. How do you measure them against the real-world?

Should you first deploy the service, wait for users to invoke that service, and then evaluate performance data? Should you first deploy the service and then wait for malicious attackers to penetrate to realize that, well, the security bar was not high enough?

Sure, you can do this, if you want. But it wouldn’t be the best strategy, to say the least.

On a totally different note, what’s a sonar? Simply put, it is an instrument based on underwater sound propagation that is used in navigation to detect icebergs, other vessels, and obstacles, in general. For example, submarines use a sonar to detect other submarines and submerged obstacles on their way. A sonar discovers obstacles by emitting a pulse of sound and listening to the reflections of the pulse. A sonar measures the distance to an object looking at the time between the emission and the return of the pulse. To use a software figure, the sonar pings an object to guarantee a proper distance, avoid collisions, and keep the crew and the machine safe.

Is there any special sonar out there to help protect Web services and, more importantly, the information system behind a Web service? Is there a piece of software that can ping a Web service to ensure quality, avoid penetration attacks, and keep the backend system safe?

Sure, there is. And its name is SOAPSonar™.

What’s SOAPSonar™, Anyway?

SOAPSonar™ is a testing and analysis tool specifically designed for enterprise Web Services. Put to use in a matter of minutes, SOAPSonar™ doesn’t require any modification to code or network topology, is completely unobtrusive, and enables various teams (development, QA, IT) to assess vulnerability, performance, compliance to standards, and functional capabilities of Web Services, before and after deployment.

SOAPSonar™ comes in two distinct editions—Personal and Enterprise.

SOAPSonar™ Personal Edition

The Personal Edition is free, and provides a set of capabilities limited to loading and testing the WSDL, functional QA regression, performance testing, and graphical reporting.

With the Personal Edition, you can load the WSDL of a local or remote Web service but not if more than Basic Authentication is required. If the site hosting the Web service implements advanced authentication features (i.e., X509 certificates), then the Enterprise Edition is necessary.

SOAPSonar™ parses the WSDL, and generates an easy to navigate list of the operation available. For users, SOAPSonar™ impersonates a rich client. You can use it to send SOAP request messages to the target Web service and capture responses. It is important to note that no code, or coding skills, is required for all this to work.

SOAPSonar™ Enterprise Edition

The Enterprise Edition provides a slew of additional features, primarily in the area of authentication (SAML, X509, Kerberos), integrated Public Key Infrastructure (PKI), vulnerability discovery, risk assessment and mediation, and design-time and run-time compliance testing.

The Enterprise Edition also supports SOAP with attachments and unlimited loading agents to effectively test performance and scalability.

Overall, SOAPSonar™ is a cutting-edge Web services testing and diagnostics product built on the Microsoft .NET platform. It provides functional regression tests, performance profiles, compliance reports, vulnerability assessment, and identity management tests. More importantly, it provides a no-code solution to Web services testing and diagnostics.

If you’re looking for a resource that fully compares the two editions of SOAPSonar™, pay a visit to http://www.crosschecknet.com/products/compare.php.

Getting Started with SOAPSonar™

Figure 1 shows the user interface of SOAPSonar™ Enterprise Edition immediately after startup. From the user interface, you can easily form an idea about how it works.

Figure 1—SOAPSonar™’s main menu

As your first step, you select a Web service and ask the software to capture its Web Service Description Language (WSDL). As mentioned, you can load the WSDL for a local or remote Web service. If you download the WSDL on the local machine, you can then work with SOAPSonar™ offline. Note, though, that the things you can do without being connected are not so many. For example, you can run a WS-I compliance analysis but can’t run test cases against the Web service.

In Figure 1, SOAPSonar™ is set to work on a sample Web service named PingSvc. You see a link to the list of failed assertions—no failed assertions in this case. At the bottom of the panel, another link points you to the list of Web methods exposed by the service. By clicking on each of them, you can send a message and test the behavior and the results of the method invocation.

Just QA is the default working mode of SOAPSonar™ and the first step to accomplish when you begin testing Web services.

QA-Testing a Web Service

You can either type the URL of the Web service’s WSDL in the tool’s address bar or browse to locate it. The Browse button is the second button right of the address bar. A dialog box will prompt you if you make an attempt to access a restricted site protected with Basic Authentication. If you know that the Web service you’re going to access requires credentials, you can also click on the third button in the WSDL toolbar for the authentication options. You’ll be displayed the dialog box below:

Figure 2—Specifying credentials for accessing a WSDL

When you click to set a certificate, SOAPSonar™ will check for a smart card reader and a certificate. You can locate and view your certificates through the PKI Management window (see Figure 3). The window appears by clicking the Tools|PKI Management menu.

Figure 3—The PKI Management dialog box

If the WSDL is successfully located and loaded, the contents of the WSDL document flow into the project tree and provide you with a graphical representation. As an example, let’s download a real-world Web service—the Amazon Web service. Figure 4 shows the WSDL once hosted in the SOAPSonar™’s project tree.

Figure 4—QA-testing Amazon’s Web Service

Two child nodes appear under the WSDL node: Documents and WSDL Services. The former node is a container for schema files; the latter lists the services described in the document. For Amazon’s search service, you see listed all possible methods. Let’s focus on the author-search method. The method sends a request for all books written by a given author.

Each method may be bound to a number of tests. By default, SOAPSonar™ creates a sample test case per method. You can create custom tests by right-clicking anywhere in the tree-view. A test case looks like the dialog box in Figure 5.

Figure 5—Configuring a test-case

SOAPSonar™ provides a parameter list where you can specify proper values. For example, to set up a request that returns the first set of books for “Dino Esposito”, you proceed as in Figure 5. Note that you must indicate your own developer token as the devtag field. You get a developer token after you register your own copy of the Amazon Web service toolkit (including the WSDL). Note also that the Amazon Web service allows for two levels of search—heavy and lite. They differ in the quantity of information returned. For example, as far as books are concerned, a lite search doesn’t include reviews and sales rank.

As you see in Figure 5, a couple of little icons appear right of the text box where you insert test parameters. The first icon indicates the so-called automatic attack function (AAF) you want to associate with the parameter. The second icon indicates the response variable, if any, to use to determine the value of the parameter for the test. As you’ll see in a moment, a response variable is automatically set with the results of a previous test.

AAFs are user-defined functions that let you dynamically generate values for a test parameter. AAFs serve a double role. On one end, they let you test the quality of the service; on the other hand, they can be used in vulnerability mode to generate a potentially harmful sequence of data to verify the level of security in the service. There are a number of supported AAF functions provided to not only include text based inputs to the test parameters, but also intelligently alter the XML structure of the SOAP request.

Figure 6—Using AAF functions

You compose AAF functions by using a number of predefined operators, as those listed in Figure 6 and commented in the table. The return value of an AAF is a value for a Web service method parameter. However, an AAF incorporates some logic to automatically generate a series of values, thus making the test more significant.

Function	Parameters	Description
`VALUES`	A\|B\|C	Generates the sequence as specified by the \|-separated string
`NUMBER`	FROM, TO, STEP	Generates numbers in the specified interval with the given step
`BUFFER`	CHAR, FROM, TO, STEP	Generates a buffer by repeating the character the specified number of times
`OCCURS`	VALUE, FROM, TO, STEP	Changes the number of times a given node appears in the SOAP request
`NEWELEMENT`	NAME, VALUE, FROM, TO, STEP	Adds a new element
`NEWNESTED`	NAME, VALUE, FROM, TO, STEP	Adds a new nested element
`NEWATTRIB`	NAME, VALUE, FROM, TO, STEP	Adds a new attribute

These functions are combined together to form a number of meaningful test cases. For example, the AAFs in the XSD Parameter group provide type checking capabilities. Custom AAFs can be created to address specific needs. For example, the following function sets the Page parameter in Figure 5 to the sequence 1 through 4.

NUMBER(1,4,1)

It reads: generates numbers starting in the range 1 to 4 with a step of 1. Once you add this function to a test case and run it, a request is sent to the server for each of the values in the range.

Request details and test parameters can be saved locally in a project file, and recalled at any time. Each request can be associated with authentication settings copied to the clipboard, loaded from an XML file, and examined and tweaked in its raw XML format. You do this, and even more, through the buttons in the vertical toolbar left of the request/response pane.

SOAPSonar™, though, provides a greater level of flexibility as far as test management is concerned. By clicking on the Test Case Management tab, you can group test cases and run them in a single batch. To create a group of tests, you simply drag test cases from the project onto the Test Groups panel.

Figure 7—The Test Group Panel

The results will be automatically logged to an XML file in a target directory of choice. Figure 8 shows the progress of the operation.

Figure 8—Running a batch of test cases

You can set the target directory at will. The result file contains information about the HTTP request, any served response, and success criteria. SOAPSonar™ can finally parse the results into a printable report. To examine the results, you turn the tool into the Logging and Reporting mode (far right button in the main toolbar) and select the test case group of choice. The gross amount of information is split in four views. Select the view you like through the drop-down list in the figure.

Figure 9—Examining the results of a test case group

Running Performance Tests

The user interface of the Test Groups panel changes slightly if you run SOAPSonar™ in Performance mode. As mentioned, you set the working mode by checking the corresponding item in the Mode menu. The default working mode is QA.

In performance mode, SOAPSonar™ allows you to set parameters that measure the performance of the Web service in real-world scenarios. In particular, it provides for virtual client loading, and simulates concurrent access to the service. Each loading agent can represent up to 50 concurrent virtual clients.

Tests can be run for a duration of time or over a given number of iterations. The choice is expressed by selecting either the duration mode or the iteration mode. Iteration mode means that each request in the test will be repeated for the specified number of times. In duration mode, the test will run for no longer than the set duration.

Figure 10—Configuring for a performance test

In Figure 10, the test is being run for 20 seconds, simulating 20 concurrent clients. A throttle setting is also available. It serves to control the desired throughput on a given time basis. For example, in Figure 10, the test is configured to place no more than one request per second.

What data is provided at the end of any performance test?

The test captures throughput and Transaction Per Second (TPS) statistics, and serves them in two views—log and report. The logging and reporting engine is the same that produces the results in Figure 10.

Finally, you can ask the system to trace errors to diagnose problems with the test and/or the service. If you turn tracing on, be aware that performance statistics will be skewed due to the overhead of request and response diagnostic logging.

Examining the Response

You can send individual test requests by filling the fields and clicking the Send Request button. The response is promptly displayed in the Response tab, as shown in Figure 8. While statistics are important, the first result that catches your eye is inevitably the response of the request. To see what’s been returned from the service, you just click the Response tab and take a look at the raw SOAP packet. (See Figure 11.)

Figure 11—Response for a Web service test request

The whole response can be split in two parts—XML and attachments. You can see both separately by clicking at the bottom of the Response tab. As Figure 11 clearly shows, there’s a third option that I haven’t mentioned yet—response variables.

Response variables address a particular test scenario where chains of test cases are being built. Imagine you need to use the results of a previous test case in a new test case. How would you do that? Response variables are dynamic placeholders which allow you to extract information from the HTTP header and the body of the response. Response variables associated with a given test case are filled at the end of a test, and become valid input for the next test.

You first define a response variable and then add it to one or more test cases. To identify a response variable, you select the Response Variable tab in Figure 11 and pick up the value you’re interested in. You drag the value onto the pane of selected variables at the bottom of the window.

Figure 12—Creating a response variable

Next, when populating a test case with input parameters, you can click one of the two icons I mentioned earlier commenting on Figure 5. As the figure below shows, by clicking on the response variable icon, you get a list of the available variables and bind the field to the contents of the variable.

Figure 13—Associating a response variable with a test case request

The response generated by each test case response is evaluated against the defined success criteria in order to determine whether the test passed or failed. To interpret the Web service response, SOAPSonar™ employs a patent pending technology named the Automatic Vulnerability Engine (AVD).

The success of a test depends on the success criteria set. Success criteria are a set of discovery operations commanded on the response packet. The available functions are listed below:

Check HTTP error code
Scan HTTP headers for a string (both case sensitive and insensitive)
Scan HTTP headers for a regular expression
Check SOAP fault
Scan SOAP packet for a string (both case sensitive and insensitive)
Scan SOAP packet for a regular expression
Scan SOAP packet for an XPath expression

Using these functions and the corresponding evaluation modes (Match All, Match Any, Match None), you build expressions that determine whether a given test passed or failed.

Testing for Compliance

When you load the WSDL of a Web service into the project, a WS-I Basic Profile scan is performed on the WSDL if SOAPSonar™ works in Compliance mode. Results of the analysis are shown on the screen with a link which will bring you to a more detailed view, like below:

Figure 14—Results of a WS-I compliance test on the Amazon Web service

The analysis tool compares the WSDL of the Web service against the WS-I Basic Profile standard and takes note of any assertion violations. The list of violations, if any, is displayed in the user interface. By clicking on an assertion violation, you highlight the portion of the WSDL file that caused the failure.

In Figure 9, the declaration of array types used by methods doesn’t conform to the standard. A link in the user interface point you directly to the source, that is the Basic Profile 1.0 Test Assertions used to generate the output. Another link in the user interface points you to a full WS-I Report which can be viewed in your client Web browser.

Some assertions may not highlight if they are related to schema violations. Note also that in the case of compound WSDL documents that import other WSDL documents, assertions may not be highlighted if the violation occurred in the nested document.

Checking Vulnerabilities

When set to work in vulnerability mode, SOAPSonar™ allows you to associate each test request with a set of attacks. At the end of the test, by examining the responses, you can make your assessment about existing vulnerabilities and information leakage. Figure 15 shows how to attach vulnerability tests to individual requests.

Figure 15—Defining vulnerability tests

You can choose from three different vulnerability groups: SOAP, XSD Parameters, and SOAP Structure attacks.

The first group inserts additional elements (DTD and processing instructions) to the request to see if the Web service can process them. It can also prepare ad hoc requests with improper SOAP schema, large attribute names, and format strings.

The second group tests the reaction of the Web service when the data of the wrong type is passed to a given parameter and when the type of a parameter is changed to all possible XSD types.

Finally, the third group changes the structure of the request by reversing the elements that indicate parameters, adding or removing parameters, and using null and large values.

In any case, SOAPSonar™ dynamically parses the XSDs specific to the WSDL and determines how to change the request to simulate the attack. By clicking the icon off right the vulnerability groups list, you can see the real request that SOAPSonar™ is going to send and all of its variations.

Figure 16—All variations of an attack

SOAPSonar™ is not just about simulating attacks; it also provides you with a patent pending technology for automatic vulnerability assessment. Simulating attacks is great, but what if you fail analyzing the results of the response correctly?

SOAPSonar™ comes with an extensible set of automatic vulnerability discovery (AVD) functions. An AVD function evaluates a Web service response, looking for vulnerability criteria such as information leakage, SOAP fault compliance, and attack susceptibility.

The standard AVD library defines a number of criteria that, if verified, configure a potential risk. All defined AVD functions are dynamically applied to each response to build a report.

Figure 17—The standard AVD library

Let’s delve deeper in the feature by examining a sample AVD function—Stack. This function examines the response of the Web service, seeking anything that looks like a stack trace. A stack trace that appears in a SOAP response indicates that an exception was not handled in the Web service. A stack trace is considered a security vulnerability because it reveals important implementation attributes (platform, language) to the attacker which can potentially be used to further attack the Web service. Each AVD function is associated with a diagnostic message and a suggested remedy. How does an AVD function work?

Any AVD function works by applying common operators to the Web service response. The operators are the same used to create the success criteria in QA mode. If an AVD function succeeds, the associated diagnostic message, severity level, and remediation are conveyed to a report document, like the one below:

Figure 18—A vulnerability report

Summary

Service-oriented architectures help companies to deploy applications while saving structural costs. At the same time, service-oriented applications generate new opportunities and, why not, new revenues by integrating internal systems with business partners.

A service-oriented architecture is based on services—whatever is the nature and implementation of the service. Web services are just one type of service, although one of the most common today. A Web service is mostly implemented over the HTTP protocol and port 80, and exposes its contract over WSDL files.

A company that deploys applications based on Web services is exposing, although behind a contracted API, the core of the internal processes and systems. What if the Web service has some vulnerabilities or doesn’t comply with recognized standards? What if it doesn’t work and perform as expected?

You need to carefully test the Web service for quality assurance and security. And you need it done before the application goes live. And possibly, you need it done without touching the code base and without recompiling.

If there were a tool with all of these capabilities, it would be great. And such a tool would be close to perfection if only it provided the ability to separate test phases, thus enabling different teams to work on the Web service at the same time. SOAPSonar™ meets all requirements, and qualifies as the premier solution to enterprise Web services vulnerability, compliance, QA, and performance testing. Try it now—go to www.crosschecknet.com.

SOAPSonar™ Measures the Distance Between Your Web Services and the Real World