Using MSMQ in Mail Relay

• Download compiled version - 283 KB
• Download source code - 24.7 KB

Introduction

I have been working on a .NET project that needed to send email notifications via an SMTP server. It worked just fine, but sometimes the notifications just did not arrive. The cause was simple, but also annoying: when the SmtpClient could not reach the server (for any reason), there was no means to postpone the operation and try to re-send it later. Thus an exception was thrown, and the message abandoned.

Then I searched the web for a suitable solution, but found only fragments of what I imagined. One of these is the article by Amol Rawke Pune: Use of MSMQ for Sending Bulk Mails [1].

Well, sounds great but has a lots of limitations. But among these, the three major ones:

1. Not all fields of the MailMessage object are passed
2. The queue consumer "service" is a simple console application
3. It has no solution to handle error conditions when sending mail

It is mainly a proof of concept, but I have to admit, this article was the starting point for me. After some Googling, I found a great solution [2] for the first one - which I incorporated in my solution after some minor adaptation. I also used the ideas of Mark Gravell [3] of making the service installer not to depend on installutil.exe.

I also found Rami Vemula's work [4] which is a great demonstration of how MailMessage can be populated with rich content. Thus it was ideal for me to test the abilities of serialization and of the solution itself.

So thanks to these good people for their contribution. All the other things I had to come up with myself... but it was fun, and I hope it will be of use for many.

Background

Message queue is a very basic concept of Windows. The so-called MSMQ (Microsoft Message Queuing) technology is not new, and is a really useful and rock solid IPC (Inter-Process Communication) tool - able to transaction ally communicate even between hosts over a network.

It has its own limitations, first of all, there is a limit of 4MB for messages. Considering the overhead of serialization, Unicode, and so on, we have to deal with even smaller messages. Well, I have to live with this - but I never intended to send large files.

Where Are the Message Queues?

Should be that simple to find them:

1. Right click on “My Computer”
2. Click on “Manage”
3. Expand Services and Application
4. Now you can see “Message Queuing”
5. Expand “Private Queue”, click on “New Private queue”

Well, it is that simple on a Windows 2008 Server. But you will probably not find it on Windows 7. You will have to install Message Queuing service. It is (a standard) Windows component, so it's not a big deal.

As any other object since Windows NT, message queues have ACL-s (Access Control List). I will come back to this one later, since it can be tricky.

So my piece of code is made of four parts:

1. A test application :)
2. The library for posting messages on the queue
3. A service to consume the queue and to send emails
4. A MailMessage serializer-deserializer

I will start with this later one shortly.

What About the Error Conditions?

Well, my approach was to treat the queue not like a regular FIFO storage - MSMQ has the tools to do it. I am using a property of the message object to set up a TTL-like counter (Time To Live), and a property to store a schedule timestamp. When a message is posted, the TTL counter is set to a user-defined value, and the timestamp to the current one. The service will use the timestamp to pick only the messages that are "expired" - every message starts its life as expired. If a possibly recoverable error condition arises during email sending, a message is re-posted in the queue and re-processed after a predefined waiting time. Every time a message is re-posted, the TTL is decreased. If the TTL is consumed, the message is abandoned. While a message is waiting to be re-processed, other messages can be processed.

Using the Code

Mailmessage Serializer-Deserializer

Serialization is something very important if we want to create distributed applications. I think actually any class should be serializable - but they are not. Well, System.Net.Mail.MailMessage is not serializable by default.

Actually, SmtpClient uses some sort of serialization to save messages in .elm format (see this article), but it would be a lot of work to make MailMessage again from such an RFC compliant file - and a lot of unnecessary overhead too.

The author of [1] created a serializable class that encapsulated a small subset of MailMessage properties. But in the blog-post [2], we can see a complete binary serializable replacement (SerializeableMailMessage) of the original class.

Since MailMessage has lot of properties of different unserializable types, the author implemented the SerializeableMailAddress, SerializeableAttachment, SerializeableAlternateView, SerializeableLinkedResource, SerializeableContentDisposition, SerializeableContentType, and SerializeableCollection classes.

Finally, it is that simple to put the serializabable version of MailMessage in an MSMQ Message:

public void QueueMessage(MailMessage emailMessage)
{
  //...
  Message message = new Message(); 
  message.Body = new SerializeableMailMessage(emailMessage);
  //...
}

and to get it back:

Message message = msgQueue.Receive(); 
MailMessage mailMessage = (message.Body as SerializeableMailMessage).GetMailMessage();

My version is an adaptation of it to .NET 4.0, I have changed to generic collections, and so on. Great stuff after all, but if you find something missing, he is to blame :). No, actually, since I intend to use this in real-life applications, any comment is appreciated.

Message Sender Library

This piece of code is intended to be called by the application willing to send a mail. It contains a MailSender class. When constructed, it tries to attach to the MSMQ specified in the constructor parameter. If it does not exist, the construction is aborted.

if (!MessageQueue.Exists(queueName))
{
    throw new ArgumentException("Cannot instantiate. MSM queue does not exist.");
}
// This class will only post to the queue     
    msgQueue = new MessageQueue(queueName, QueueAccessMode.Send);
// Will use binary formatter    
    msgQueue.Formatter = new BinaryMessageFormatter();
// Messages will be by default recoverable    
    msgQueue.DefaultPropertiesToSend.Recoverable = true;

Oh, and by the way, there is logging all around the code: you will need to install NLog if you want to compile the code. Logging is a must even in production environments, and with NLog, you need only to change a configuration file to have the logs where you want them: one logger for all purposes. Really.

logger.Info("Successfully attached to MSM queue: '{0}'", queueName);

But I will omit logging statements in this article when possible.

If construction is successful, you may call the QueueMessage method.

// the integer parameter is the TTL value I was talking before
public void QueueMessage(MailMessage emailMessage, int deliveryAttempts = 3)
{
   // some checks here...

   try
   {
      Message message = new Message();
      // we copy the mail in the body of the message
      message.Body = new SerializeableMailMessage(emailMessage);
      // just to be sure...
      message.Recoverable = true;
      // wi will use binary serialization to use as much as we can from that 4MB
      message.Formatter = new BinaryMessageFormatter();
      // here we sore the TTL value
      message.AppSpecific = deliveryAttempts;
      // scheduled delivery time, 'now' for the start
      message.Extension = System.BitConverter.GetBytes(DateTime.UtcNow.ToBinary()); 
      // unique, app-specific id
      message.Label = Guid.NewGuid().ToString();
      // ...see below...
      message.UseAuthentication = useAuthentication;
      // post the message
      msgQueue.Send(message);
   }
   catch (Exception ex)
   {
      throw ex;
    }
}

The Extension property is a byte array, so I had to convert the current timestamp to bytes. I am using UTC to be sure to have the same time on both sides, if producer and consumer are not on the same host.

A queue can be set up to require authentication to access it. Only in this case are the ACL-s of the queue effective. I have not tested it yet, but it seems that authentication is working only in AD environments. Well, the code is prepared.

The Service

Let's talk first about the service installer. I wrote earlier that I have adopted a solution [3] to get rid of installutil.exe and to have the possibility to run the service application as a regular console application. This little trick is really useful during debugging.

// custom service executable installer class 
[RunInstaller(true)]
public sealed class MSMQ_MailRelyServiceProcessInstaller : ServiceProcessInstaller
{
    public MSMQ_MailRelyServiceProcessInstaller()
    {
        // run service as network service... remember the ACL-s!
        this.Account = ServiceAccount.NetworkService;
        this.Username = null;
        this.Password = null;
    }
}
// custom service installer class    
[RunInstaller(true)]
public class MSMQ_MailRelyServiceInstaller : ServiceInstaller
{
    public MSMQ_MailRelyServiceInstaller()
    {
        // set up basic parameters of the service
        this.DisplayName = "MSMQ Mail processor service";
        this.StartType = ServiceStartMode.Automatic;
        this.DelayedAutoStart = true;
        this.Description = 
          "Service is designed to send email messages posted in a messaging queue.";
        // this service depends on the Microsoft Messaging Queue service
        this.ServicesDependedOn = new string[] { "MSMQ" };
        this.ServiceName = "MSMQ Mail Rely";
    }
}
// This is the application itself
class Program
{
    // Install or uninstall the service
    static void Install(bool undo, string[] args) 
    { 
        try 
        { 
            Console.WriteLine(undo ? "uninstalling" : "installing");
            // The service is in this assembly, thus install the executable itself 
            using (AssemblyInstaller inst = new AssemblyInstaller(typeof(Program).Assembly, args)) 
            { 
                // Installer will return some messages
                IDictionary state = new Hashtable(); 
                inst.UseNewContext = true;
                // try to install or uninstall service, and rollback process if something fails
                try 
                { 
                    if (undo) 
                    { 
                        inst.Uninstall(state); 
                    } 
                    else 
                    { 
                        inst.Install(state); 
                        inst.Commit(state); 
                    } 
                } 
                catch 
                { 
                    try 
                    { 
                        inst.Rollback(state); 
                    } 
                    catch { } 
                    throw; 
                } 
            } 
        } 
        catch (Exception ex) 
        { 
            Console.Error.WriteLine(ex.Message); 
        } 
    }
    // entry point
    static int Main(string[] args)
    {
        bool install = false, uninstall = false, console = false, rethrow = false; 
        try 
        { 
            // let's parse arguments
            foreach (string arg in args) 
            { 
                switch (arg) 
                { 
                    case "-i": 
                    case "-install": 
                        install = true; break; 
                    case "-u": 
                    case "-uninstall": 
                        uninstall = true; break;
                    case "-c":
                    case "-console":
                        console = true; break; 
                    default: 
                        Console.Error.WriteLine("Argument not expected: " + arg); 
                        break; 
                } 
            } 
            // do the action
            if (uninstall) 
            { 
                Install(true, args); 
            } 
            if (install) 
            { 
                Install(false, args); 
            } 
            // this is the hack:
            if (console) 
            {
                // we construct the service class outside the service host
                MSMQ_MailRelyService service = new MSMQ_MailRelyService();
                Console.WriteLine("Starting...");
                // original event handlers are protected, so we need public methods to start up...
                service.StartUp(args);
                Console.WriteLine("Service '{0}' is running in console mode. " + 
                                  "Press any key to stop", service.ServiceName); 
                Console.ReadKey(true);
                // ...and to shut it down 
                service.ShutDown(); 
                Console.WriteLine("System stopped"); 
            } 
            // when loaded as real service we will create the host... 
            else if (!(install || uninstall)) 
            { 
                // so that windows sees error...
                rethrow = true;  
                ServiceBase[] services = { new MSMQ_MailRelyService() };
                // ...and run the service 
                ServiceBase.Run(services); 
                rethrow = false; 
            } 
            return 0; 
        } 
        catch (Exception ex) 
        { 
            if (rethrow) throw; 
            Console.Error.WriteLine(ex.Message); 
            return -1; 
        } 
    }
}

So if you want to install the service, just call MSMQ_MailRelyService.exe -i as administrator. To debug, just add -c as a command line argument in the Debug page of the application properties, and run it from the IDE.

Next, I will talk about the service itself. For the complete source code, please browse the code here or download it.

protected override void OnStart(string[] args)
{         
  try
    {
      // If queue does not exist
      if (!MessageQueue.Exists(settings.QueueName))
      {
        // create the queue itself
        msgQueue = MessageQueue.Create(settings.QueueName);
        // use authentication in AD environment
        msgQueue.Authenticate = settings.UseAuthentication;
        // label the queue
        msgQueue.Label = "MSMQ Mail Rely message queue";
      }
      else
      {
        // attach to the queue
        msgQueue = new MessageQueue(settings.QueueName);
      }
      // we will use binary serialization
      msgQueue.Formatter = new BinaryMessageFormatter();
      // retrieve all properties
      msgQueue.MessageReadPropertyFilter.SetAll();
      // only this service can retrieve from the queue
      msgQueue.DenySharedReceive = true;
      // we start the message processor thread
      MSMQMessageProcessor = new Thread(new ThreadStart(this.ProcessMSMQMessages));
      MSMQMessageProcessor.Start();
    }
    catch (Exception ex)  
    {
      throw ex;
    }
}

Remember: You may create the queue manually, but do not forget to set up the ACL. By default, everyone can post messages in the queue, but retrieving from it requires rights granted. The creator will have this by default. So if the service creates it, it will work just fine, but you need to take ownership to be able to manipulate the ACL or any other property via the MMC plug-in. If you create the queue, you have to grant necessary rights to the user running the service.

As extension methods are great things, I wrote one to get the next message to be processed. The method will traverse all messages in the queue, and take all eligible ones according to their schedule timestamp. From those eligible ones, the method will pick the oldest one, and return its ID. If no eligible message is found, the method will return null.

public static String GetScheduledMessageID(this MessageQueue q)
{
   DateTime OldestTimestamp = DateTime.MaxValue;
   String OldestMessageID = null;

   using (MessageEnumerator messageEnumerator = q.GetMessageEnumerator2())
   {                
      while (messageEnumerator.MoveNext())
      {
         DateTime ScheduledTime = DateTime.FromBinary(
            BitConverter.ToInt64(messageEnumerator.Current.Extension, 0));
         if (ScheduledTime < DateTime.UtcNow) // Take only the proper ones 
         {
            if (ScheduledTime < OldestTimestamp)
            {
               OldestTimestamp = ScheduledTime;
               OldestMessageID = messageEnumerator.Current.Id;
            }
         }
      }
   }
   return OldestMessageID;
}

The main thread will do the real work, so let's see:

private void ProcessMSMQMessages()
{
   try
   {
      // this is a tread after all
      while (true)
      {
         // wait for available messages, thread is blocked while queue is empty
         Message message = msgQueue.Peek();
         // we look for the first scheduled message with the extension method  
         String ID = msgQueue.GetScheduledMessageID(); 
         // have we found a message to be processed?
         if (ID != null)
         {
            // retrieve the elected message by it's id
            message = msgQueue.ReceiveById(ID);
            // deserialize original email
            MailMessage mailMessage = 
              (message.Body as SerializeableMailMessage).GetMailMessage();
            // we will store the error condition for later 
            Exception CachedException = null;
            // by default we will not re-post if something fails
            RetryReason retry = RetryReason.NoRetry;
            try
            {
               using (var smtpClient = new SmtpClient())
               {
                  // try to send the mail
                  // (do not forget to set up SMTP parameters on app.config)
                  smtpClient.Send(mailMessage);
               }
            }
            // look for exceptions, if any
            catch (SmtpFailedRecipientsException ex)
            {
               // store exception
               CachedException = ex;
               // traverse inner exceptions...
               for (int i = 0; i < ex.InnerExceptions.Length; i++)
               {
                  // ...to see if worth retrying
                  SmtpStatusCode status = ex.InnerExceptions[i].StatusCode;
                  if (status == SmtpStatusCode.MailboxBusy ||
                     status == SmtpStatusCode.MailboxUnavailable ||
                     status == SmtpStatusCode.InsufficientStorage)
                  {
                     // store retry reason
                     retry = RetryReason.Postmaster;
                  }
               }
            }
            catch (SmtpException ex)
            {
               CachedException = ex;
               if (ex.InnerException != null)
               {
                  // this is the case of network errors 
                  WebExceptionStatus status = (ex.InnerException as WebException).Status;
                  // we look for possibly recoverable situations...
                  if (status == System.Net.WebExceptionStatus.NameResolutionFailure ||
                        status == System.Net.WebExceptionStatus.ConnectFailure)
                  {
                     // ...and store the reason
                     retry = RetryReason.Network;
                  }
               }
            }
            catch (Exception ex)
            {
               // nothing to do in other cases
               CachedException = ex;
            }
            // if error looks recoverable...
            if (CachedException != null)
            {
               if (retry != RetryReason.NoRetry)
               {
                  // ...and we have not consumed our chances
                  if (message.AppSpecific > 0)
                  {
                     // update schedule time
                     DateTime OriginalScheduledTime = 
                       DateTime.FromBinary(BitConverter.ToInt64(message.Extension, 0));
                     // determine wait time
                     int retryDelaySeconds;
                     if (retry == RetryReason.Network)
                     {
                        // network errors might recover sooner...
                        retryDelaySeconds = settings.NetworkRetryDelay_s;
                     }
                     else
                     {
                        // ...smtp errors can last longer
                        retryDelaySeconds = settings.PostmasterRetryDelay_s;
                     }
                     // calculate new schedule timestamp
                     message.Extension = System.BitConverter.GetBytes(
                       DateTime.UtcNow.ToUniversalTime().AddSeconds(retryDelaySeconds).ToBinary());
                     // update TTL
                     message.AppSpecific--;
                     // postpone message
                     msgQueue.Send(message);
                  }
                  else
                  {
                     logger.ErrorException("Failed to deliver, no more attempts.", CachedException);
                  }
               }
               else
               {
                  logger.ErrorException("Failed to deliver, but no use to retry", CachedException);
               }
            }
         }
         // wait only if there was nothing to process
         else
         {
            Thread.Sleep(settings.SleepInterval);
         }
      }
   }
   // Catch exception raised when thread is aborted
   catch (ThreadAbortException)
   {
      logger.Info("Thread aborted.");
   }
}

It might be worth reviewing error conditions where retries are applied. Actually, it depends on the SMTP server and the addresses.

The service has several settings stored in the app.config. The defaults might be suitable for you, but you may change them as you like.

First of all, we have the queue name. As this is the service, using a local private queue is straightforward. Technically, a public queue could also be used.

<applicationSettings>
    <MSMQ_MailRelyService.Properties.Settings>
        <setting name="QueueName" serializeAs="String">
            <value>.\Private$\EmailQueue</value>
        </setting>

This is how long (in milliseconds) the thread sleeps if there was no suitable message to process:

<setting name="SleepInterval" serializeAs="String">
    <value>5000</value>
</setting>

Use or not use authentication:

<setting name="UseAuthentication" serializeAs="String">
    <value>False</value>
</setting>

The amount of time (in seconds) the postponed delivery attempt should be delayed in case of a network error:

<setting name="NetworkRetryDelay_s" serializeAs="String">
    <value>120</value>
</setting>

The delayed time in seconds in case of an SMTP error.

         <setting name="PostmasterRetryDelay_s" serializeAs="String">
            <value>3600</value>
        </setting>
    </MSMQ_MailRelyService.Properties.Settings>
</applicationSettings>

Do not forget to set up your SMTP environment in the app.config as well. This is how it can be done with GMail:

<system.net>
    <mailSettings>
      <smtp deliveryMethod="Network" from="validatedsender@gmail.com">
        <network defaultCredentials="false" enableSsl="true" 
           host="smtp.gmail.com" port="587" 
           userName="username@gmail.com" password="password"/>
      </smtp>
    </mailSettings>
</system.net>

See MSDN for future details.

The Test Application

I will not say much about this one, the code will be self-explaining. As I mentioned earlier, I have adapted [4] to have a nearly complete feature-test.

MailSender sender = new MailSender(@".\Private$\EmailQueue"); 
class Program
{
   static void Main(string[] args)
   {
      // we construct the sender object with the queue name as parameter 
      MailSender sender = new MailSender(@".\Private$\EmailQueue"); 
      // this will be the mail to be sent
      MailMessage m = new MailMessage();
      // we populate the basic fields 
      m.From = new MailAddress("sender@mailserver.com", "Sender Display Name");
      m.To.Add(new MailAddress("to@mail.com", "To Display Name"));
      m.CC.Add(new MailAddress("cc@mail.com", "CC Display Name"));
      m.Subject = "Sample message";
      m.IsBodyHtml = true;
      m.Body = @"<h1>This is sample</h1><a " + 
        @"href=""http://http://www.codeproject.com"">See this link</a>";
      // we add an attachment
      FileStream fs = new FileStream(@"C:\Windows\Microsoft.NET\Framework" + 
        @"\v4.0.30319\SetupCache\Client\SplashScreen.bmp", FileMode.Open, FileAccess.Read);
      Attachment a = new Attachment(fs, "image.bmp", MediaTypeNames.Application.Octet);
      m.Attachments.Add(a);
      // we add an alternate view...
      string str = "<html><body><h1>Picture</h1>" + 
         "<br/><img src=\"cid:image1\"></body></html>";
      AlternateView av = 
        AlternateView.CreateAlternateViewFromString(str, null, MediaTypeNames.Text.Html);
      // ...with an embedded image
      LinkedResource lr = new LinkedResource(@"C:\Windows\Microsoft.NET\Framework" + 
        @"\v4.0.30319\ASP.NETWebAdminFiles\Images\ASPdotNET_logo.jpg", MediaTypeNames.Image.Jpeg);
      lr.ContentId = "image1";
      av.LinkedResources.Add(lr);
      m.AlternateViews.Add(av);
      // and finally we try to pass the mail to the rest of the solution 
      sender.QueueMessage(m);
   }
}

That's all folks.

Points of interest

I think this solution can be used as is in a real-life application. I have tried to deal with as many situations that must be handled as possible. I think MSMQ has even more possibilities to be explored in such an application, like using report typed messages and administration queues to give asynchronous feedback to the sender, introducing some timeouts. As I will test it in an AD environment soon, I will come back and update the article with my findings.

History

• 2012.02.05: First release, but with some improvements during the writing of this article