Introduction

This article will show you how to implement a multithreaded HTTP proxy server in C# with a non-standard proxy server feature of terminating and then proxying HTTPS traffic. I've added a simple caching mechanism, and have simplified the code by ignoring http/1.1 requests for keeping connections alive, etc.

Disclaimer: Understand that this code is for debugging and testing purposes only. The author does not intend for this code or the executable to be used in any way that may compromise someone's sensitive information. Do not use this server in any environment which has users that are unaware of its use. By using this code or the executable found in this article, you are taking responsibility for the data which may be collected through its use.

Background

If you are familiar with fiddler, then you already know how this proxy server works. It essentially performs a "man-in-the-middle" on the HTTP client to dump and debug HTTP traffic. The System.Net.Security.SslStream class is utilized to handle all the heavy lifting.

Using the Code

The most important part about this code is that when the client asks for a CONNECT, instead of just passing TCP traffic, we're going to handle an SSL handshake and establish an SSL session and receive a request from the client. In the mean time, we'll send the same request to the destination HTTPS server.

First, let's look at creating a server that can handle multiple concurrent TCP connections. We'll use the System.Threading.Thread object to start listening for connections in a separate thread. This thread's job will be to listen for incoming connections, and then spawn a new thread to handle processing, thus allowing the listening thread to continue listening for new connections without blocking while one client is processed.

public sealed class ProxyServer
{
   private TcpListener _listener;
   private Thread _listenerThread;

   public void Start()
   {
      _listener = new TcpListener(IPAddress.Loopback, 8888);
      _listenerThread = new Thread(new ParameterizedThreadStart(Listen));
      _listenerThread.Start(_listener);
   }
        
   public void Stop()
   {
      //stop listening for incoming connections
      _listener.Stop();
      //wait for server to finish processing current connections...
      _listenerThread.Abort();
      _listenerThread.Join();
   }

   private static void Listen(Object obj)
   {
      TcpListener listener = (TcpListener)obj;
      try
      {
         while (true)
         {
            TcpClient client = listener.AcceptTcpClient();
            while (!ThreadPool.QueueUserWorkItem
		(new WaitCallback(ProxyServer.ProcessClient), client)) ;
         }
      }
      catch (ThreadAbortException) { }
      catch (SocketException) { }
   }

   private static void ProcessClient(Object obj)
   {
      TcpClient client = (TcpClient)obj;
      try
      {
         //do your processing here
      }
      catch(Exception ex)
      {
         //handle exception
      }
      finally
      {
         client.Close();
      }
   }
}

And that's the beginning of the code to handle concurrent TCP clients in a multithreaded manner. Nothing really special there. The interesting bit is when we use SslStream to act as an HTTPS server and "trick" the client into believing it's talking to the destination server. Note that the browser should not actually be tricked because of the SSL certificate chain, but depending on their browser, it may or may not be apparent that the server's identity is in question.

Now let's take a look at the actual processing of the SSL request. Assume that we are somewhere inside the try block of the ProcessClient method shown above.

//read the first line HTTP command
Stream clientStream = client.GetStream();
StreamReader clientStreamReader = new StreamReader(clientStream);
String httpCmd = clientStreamReader.ReadLine();

//break up the line into three components
String[] splitBuffer = httpCmd.Split(spaceSplit, 3);
String method = splitBuffer[0];
String remoteUri = splitBuffer[1];
Version version = new Version(1, 0); //force everything to HTTP/1.0

//this will be the web request issued on behalf of the client
HttpWebRequest webReq;

if (method == "CONNECT")
{
   //Browser wants to create a secure tunnel
   //instead = we are going to perform a man in the middle
   //the user's browser should warn them of the certification errors however.
   //Please note: THIS IS ONLY FOR TESTING PURPOSES - 
   //you are responsible for the use of this code
   //this is the URI we'll request on behalf of the client
   remoteUri = "https://" + splitBuffer[1];
   //read and ignore headers
   while (!String.IsNullOrEmpty(clientStreamReader.ReadLine())) ;

   //tell the client that a tunnel has been established
   StreamWriter connectStreamWriter = new StreamWriter(clientStream);
   connectStreamWriter.WriteLine("HTTP/1.0 200 Connection established");
   connectStreamWriter.WriteLine
	(String.Format("Timestamp: {0}", DateTime.Now.ToString()));
   connectStreamWriter.WriteLine("Proxy-agent: matt-dot-net");
   connectStreamWriter.WriteLine();
   connectStreamWriter.Flush();

   //now-create an https "server"
   sslStream = new SslStream(clientStream, false);
   sslStream.AuthenticateAsServer(_certificate, 
	false, SslProtocols.Tls | SslProtocols.Ssl3 | SslProtocols.Ssl2, true);

   //HTTPS server created - we can now decrypt the client's traffic
   //.... 
}

Points of Interest

You can see that I have an X509Certificate2 _certificate defined elsewhere in the code. To get a certificate for this, you need to use a tool like makecert.exe to create a self-signed certificate. I found makecert.exe in the Windows SDK and it also comes with Fiddler. I have included a certificate file in the source files to allow the server to run, but because it does not include the private key, to actually handle SSL traffic, You will need to run makecert.exe

Here is the syntax I used for makecert.exe:

makecert.exe cert.cer -a sha1 -n "CN=matt-dot-net" -sr LocalMachine -ss My -sky signature -pe -len 2048

One thing to note is that on your HttpWebRequest object, you will need to set the Proxy property to null if you are using Windows internet options to specify using your proxy server. This is because your HttpWebRequest will default to the Windows internet settings and you'll have a proxy server that is trying to use itself as a proxy server!

Another interesting and frustrating hang-up was the handling of cookies. When I thought that I was processing requests/responses perfectly, I found that I could not maintain state with any websites because cookies were not properly being sent by the client. I determined (from using fiddler and firefox live HTTP headers) that cookies needed to be set individually. The server was returning several cookies in one Set-Cookie header, but I needed to parse them out and return an individual Set-Cookie header for each one. I haven't researched to find out why this is, or what is the proper handling of cookies in HTTP, but all I can assume was that when a browser is set to use a proxy server, it expects the proxy server to process cookies into individual Set-Cookie headers and if not configured to use a proxy, the browser does this itself.

History

I threw together this code on a Sunday afternoon, so I expect it to have errors and problems. Also, I make no claim that this is properly handling errors, cleaning up objects, or is in any way the best way to do anything.

Implementing a Multithreaded HTTP/HTTPS Debugging Proxy Server in C#