SoapExtensions: A Bad Day with HTTP 400 Bad Requests

You may have found this post if you were searching for:

• HTTP 400 Bad Request web service
• Response is not well-formed XML web service
• System.Xml.XmlException: Root element is missing web service
• SoapExtension impacting all web services

Yesterday, I was debugging an inconsistent issue in production. Thankfully, we could track trending recurring errors and began to piece together all incoming and outgoing webservices were being negatively impacted for unknown reasons. This was creating a lot of pressure as backlogs of incoming calls were returning HTTP 400 Bad Request errors. Outgoing calls were silently failing without a facility to retrigger the calls later creating manual work.

We suspected SSO or SSL leading us to change settings in IIS. Being IIS 7.5, this touched the web.config which recycled the app pool. Every time a setting in IIS was changed or an iisreset was issued, it seemed to rectify the situation. But after an indeterminate amount of time, the problems would resurface.

The culprit ended up being a SoapExtension. The SoapExtension modifies the soap header for authentication when making outgoing calls to a Java webservice.

<?xml version="1.0" encoding="UTF-8"?>
<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<SOAP-ENV:Header>
<h:BasicAuth xmlns:h="http://soap-authentication.org/basic/2001/10/"
SOAP-ENV:mustUnderstand="1">
<Name>admin</Name>
<Password>broccoli</Password>
</h:BasicAuth>
</SOAP-ENV:Header>
<SOAP-ENV:Body>
<m:echoString xmlns:m="http://soapinterop.org/">
<inputString>This is a test.</inputString>
</m:echoString>
</SOAP-ENV:Body>
</SOAP-ENV:Envelope>

It does this with a dynamically loaded (that bit was my fault and made it a complete bitch to debug) SoapExtension taken from a legacy command line util which did this every 5 minutes.

This existed simply because nobody could figure out how to call the webservice directly within the web application. Once incorporated when the web service was called, perhaps hours from an iisreset, the SoapExtension is dynamically loaded. The bug was, even though it was coded to not affect anything but Vendavo, the checks performed were performed too late and therefore all web services, incoming and outgoing, were impacted.

Previously, the check was in the After Serialize message handler. The fix was to return the original stream in the ChainStream. The hard part was knowing what webservice was making the call before ChainStream was called. The check was moved to:

public overrides void Initialize(Object initializer)

The initializer object was tested setting a flag used in ChainStream to determine which stream was returned.

So lesson learned, beware SoapExtensions may impact all soap calls. While you can specify a custom attribute to limit the extension to web methods you publish, you cannot use this filtering mechanism on webservices you consume. This means you must self-filter or risk affecting all incoming and outgoing web services unintentionally.

Also, dynamically loading a setting which belongs in the web.config was a dumb idea which delayed identification of the problem. Now we use this:

<system.web>
   <webServices>
      <soapExtensionTypes>
         <add type="SBA.Data.PMMSoapExtension, SBA.Data" 
         priority="1" group="High" />
      </soapExtensionTypes>
   </webServices>
</system.web>

References

• http://www.hanselman.com/blog/ASMXSoapExtensionToStripOutWhitespaceAndNewLines.aspx
• http://msdn.microsoft.com/en-ie/magazine/cc164007(en-us).aspx