Working with DSL Tools: An Asynchronous Graphical DSL

• Download source code - 1.09 MB

Contents

• Introduction
• Problem statement
• Making the DSL
• Arranging shapes
• Relationships
• Toolboxes, at last
• Running it!
• Concepts
• Transforming the model with T4
• Conclusion

Introduction

Broadly speaking, domain-specific languages (DSLs for short) come in three varieties. The first is a textual DSL, one defined entirely via text and with structure predicted by whatever tool processes or compiles the definitions.[1] The second is a structural DSL, where the content is defined via a tree-like or graph-like editor.[2] The third type and the one I will talk about is the graphical DSL, where the developer works with a graphical editor in order to create a visual structure that will get turned into code.

In this article, we shall create a simple graphical DSL that allows the end user to define asynchronous operations that will be orchestrated via the Pulse & Wait mechanism. In order to compile the attached example, you will need Visual Studio 2008 with the Visual Studio SDK. We shall be using Microsoft DSL Tools (part of the SDK) to create our DSL.

Problem statement

Since working with Pulse & Wait is difficult, I want to make a graphical DSL that would allow me to define sequences of operations that can be orchestrated using the Pulse & Wait mechanism. Specifically, I want to be able to drag operation blocks onto a canvas, and be able to make connections between them to indicate continuation rules for overall (asynchronous) execution.

Making the DSL

Before we begin, let me outline the most important points when working with DSL Tools:

• In DSL Tools, the graphical DSL is itself made using a graphical DSL. This may seem confusing at first but, basically, you need to realize that most of our asynchronous DSL (which I call AsyncDsl here) will be designed using visual aids – not with a programming language. Of course, there will be lots of code behind the scenes, but we won’t see much of it.
• DSL Tools make heavy use of the T4 technology.[3] Our graphical DSLs are, in reality, just visual representations of XML, and T4 turns XML into code. Thus, when you are editing visual elements with DSL tools, you’re really editing XML.
• Your DSL is still made using C#, and it does get compiled. You can extend it by using partial classes and the like, making your DSL behave in a specific way. We won’t be doing any of this in this article.
• DSL creation using DSL Tools only concerns the visual part – the part that lets the user build and XML model visually. The part that turns it into code is plain text – we’ll see it later.

To make a DSL project in Visual Studio, select New Project, then navigate to Other Project Types → Extensibility → Domain-Specific Language Designer.

After you press OK, you will be shown a wizard where you can define some features of the DSL you are creating.

• On the first page, apart from giving your language a name, you also get to pick a starting template. This template determines what starting features your DSL has – for example, choosing Task Flow adds some elements that suggest flowchart-like behavior. Whatever option you choose here, you can always redefine this later by removing the auto-generated elements and adding new ones.
• The second page lets you pick the extension to use for your DSL. This is the extension that your DSL definitions will have when living in ordinary solution files. In addition, you get to define a model icon.
• The third page lets you set some strings which define your DSL, such as the name of the product this DSL belongs to.
• The fourth page basically forces you to sign the assemblies of your DSL with a strong name key – either with one you already have, or the system will generate the key for you.

When done with the wizard, you get a skeleton DSL definition. Once again, this might seem like a culture shock for a typical programmer, so let’s take a look at what happened to Visual Studio:

The constituent parts of the DSL editing experience are as follows:

• The DSL Designer Toolbox. This toolbox contains all the elements you’ll be working with while designing your DSL. The procedure for using an item from this toolbox is the same as with, say, WinForms – you just grab an element and drag it onto the DSL window (the one with the funny boxes and whatnot).
• The DSL editor itself. The file itself has a .dsl extension but, as you can see, its editor is very visual – as I said before, DSLs are themselves made using a DSL. This particular DSL has two parts – the left part contains Classes and Relationships, while the right part contains Diagram Elements. You can think of these two parts as visuals and code-behind of a GUI, with the visual elements being on the right and the ‘logic’ being on the left.
• The Solution Explorer. When making a DSL, you get two projects – one that defines the DSL you’re making, and another that defines the editor components associated with the DSL. We’ll talk more about these later – now the only important thing is to point out the Transform All Templates button:



This button is very important. As I mentioned before, DSLs are simply XML that gets transformed into code. This means that in order to update changes in your DSL definition (itself a DSL), you need to transform all templates into C# code. This is precisely what this button does. If you ever find yourself wondering why the DSL hasn’t updated itself after you’ve made changes, you probably forgot to press this button before compiling.

• The DSL explorer. This is a new tab that presents the DSL in a treelike form:



This tree encapsulates many of the structural aspects of a DSL. It is important to note that some of the tree nodes have property pages (as per the Properties tab), which you can open by pressing F4.

• Property pages exist for DSL explorer tree elements, and also for the visual DSL elements (the various boxes). Some DSL elements are also editable on-screen – for example, you can set relationship cardinality by just typing it in the visual designer. It's your choice how you want to do this.

Time for an in-depth discussion of the tools and how to use them to make our DSL.

Arranging shapes

As I mentioned earlier, the toolbox contains all the elements you’ll be working with. These elements fall into two groups – you can call them logical and visual. The logical elements are the ones that define the structure (i.e., the concepts) in your DSL. The visual ones are the ones that correspond to rectangles and lines drawn in the actual DSL.

At the core of a logical DSL structure is the domain class. This class models anything, depending on what you’re modeling. Since we are working with asynchronous operations, one of our domain classes is called – guess what – Operation:

A domain class can have properties, which are simply values that the user can set. Our Operation domain class has properties Timeout, Name, and Description, that the end user can set after they drag an instance of an Operation class onto their model.

There’s a catch though – the end user doesn’t really drag this class onto the model – instead, they drag an OperationShape element onto the model. The OperationShape element is a type of GeometryShape (see the toolbox), and looks as follows:

Having defined both the concept Operation and the corresponding visual OperationShape, we need to link the two. This is what the Diagram Element Map element is for. Basically, it draws a thin line between one element and another, defining an association between the two. But if you were to compile the projects now and attempted to debug them, you wouldn’t see the new item in the toolbox – lots more needs to be done.

Relationships

Before we get to make toolbox items (which is the fun part), we need to talk about relationships. There are two types – an Embedding Relationship and a Reference Relationship. Care to guess what they are? Well, if you use an embedding relationship, element A will be entirely within element B. For example, if A is a swim-lane (a big chunk of visual editing space), and B is a class, this makes sense. But if I want to attach a comment to class A, then we simply use a reference relationship, thus having a comment refer to the class.

Let’s look at our specific usage again. At the ‘root’ of the model, we have our ExampleModel class (I didn’t bother changing the name because we won’t actually see it anywhere). In order to specify that my model contains processes and comments, I draw the Embedding Relationship lines between the corresponding classes and get the following:

The orange boxes are the relationships, with relationship names and cardinalities on either side. Cardinality gets enforced by the DSL designer later on when you work with the DSL. As for the relationship, the purpose of having those orange boxes is so that you can connect them to Connector shapes in the visual part of the DSL designer.

Warning: The DSL designer applies a number of rules to your DSL such that all elements you add (i.e., all domain classes) have to ‘belong’ somewhere. In other words, all elements have to have a common owner that contains them.

Just as we have seen an Embedding relationship forcing the model to contain a process, we can get the two elements which are simply linkable, but are, in fact, ‘equal partners’ in the DSL (i.e., none contains the other). Here is an illustration:

The dashed orange line implies a referencing relationship and, in our case, this means that an operation can simply refer to a comment – not contain it. Of course, such a relationship can also have its own visual element (a line drawn from one to another), and that’s precisely what we do in our DSL.

Toolboxes, at last

Okay, so you’ve got the logical and visual parts of the DSL and want to let people drop them into their model? Here’s where you would start – the Editor node in the DSL Explorer:

In order to add a toolbox item, right-click the DSL (in our case, the AsyncDsl node under Editor → Toolbox Items). You will see the following menu:

There are two options – connections and elements. Connections are the lines (with or without arrowheads) that connect elements together. Elements are actual block-shape structures.

Once you create a toolbox item, press F4 to view its properties. You will see attributes similar to the following:

The important thing about these properties is that you have to specify several of them – the mandatory ones – so that the system doesn’t complain. The obvious ones are specifying a domain class this element corresponds to, and specifying an icon for the toolbox. (Two default icons are provided already, but adding others is a matter of creating 16×16 masked bitmaps.)

Running it!

Let’s recap the steps we took to produce a DSL:

1. We created a DSL stub using the wizard.
2. Added domain classes to denote concepts useful to our model, such as process, operation, and so on.
3. Added relationships between domain classes to specify that, e.g., operations belong to a model and can have comments. We also added transition operations between the two classes as well as start and finish elements. We shall talk more about the elements in a little while.
4. Defined the visual shapes that would be drawn by our DSL.
5. Connected visual shapes to domain classes.
6. Created toolbox items and connected them to the corresponding classes.

Our DSL is half-ready: we’ve just got the visual part done. After we transform all the templates, compile the solution, and press F5, we can finally play with our DSL in the Visual Studio Experimental Hive.[4] Here is a screenshot:

Concepts

For our asynchronous DSL, we define the following entities:

• Operation - This is basically a unit of work, for example, ‘make tea’. We assume that this unit of work can be carried out by a separate thread without interference.
• Process - This is a sequence of operations presented as a graph. The only reason we have this concept in our DSL is to be able to define several sequences of operations inside a class.
• Start and Finish - These are states which must be present in any definition of a process. After all, it has to start and end somewhere, right?
• Finish-to-start transition - This transition says that an operation has to finish before another one can start. In .NET terms, it means that an operation has to pulse at the end of its method while another is doing a wait at the beginning.
• Start-to-start transition - This transition says that an operation can only start when another starts, and no earlier. In .NET terms, it means that an operation has to pulse before doing anything, while another operation waits for this before doing its work.

Let’s consider a practical example: the process of having breakfast (not too exciting I know). To have breakfast, I want to put the kettle on, and put bread in the toaster – in any order. While these asynchronous operations carry on, I want to take out the jam – but only after I’ve begun toasting bread (this is a start-to-start operation). Only when I have both the toasted bread and the jam can I make a sandwich (this is a finish-to-start op). And only when I have jam on toast and my tea is ready can I have breakfast (another finish-to-start).

Working with our DSL, the whole procedure can be defined as follows:

As you may have guessed, solid lines denote finish-to-start, and the dashed line denotes start-to-start.

Transforming the model with T4

The visual model of Breakfast only exists as a DSL, so we need to use T4 in order to turn it into meaningful code. Luckily, by the time we get to the T4 part, the model is already converted from XML format into object format for us. All we have to do is to traverse the tree and make legal C# code on the output.

The production of the output in T4 is driven by several simple functions such as WriteLine(), which writes a line of output, and PushIndent()/PopIndent() which allow us to control indentation by keeping an internal stack.

I will not present the T4 code here – you can take a look at what’s in the solution. Instead, I want to show the output of our transformation after we have defined the Breakfast process:

namespace Debugging
{
  using System.Threading;
  partial class Breakfast
  {
    private readonly object MakeSandwichLock = new object();
    private readonly object EatBreakfastLock = new object();
    private readonly object GetJamLock = new object();
    private bool MakeTeaIsDone;
    private bool ToastBreadIsDone;
    private bool GetJamIsDone;
    private bool MakeSandwichIsDone;
    private bool MakeTeaStarted;
    private bool ToastBreadStarted;
    private bool GetJamStarted;
    private bool MakeSandwichStarted;
    protected internal void MakeTea()
    {
      MakeTeaImpl();
      lock(EatBreakfastLock)
      {
        MakeTeaIsDone = true;
        Monitor.PulseAll(EatBreakfastLock);
      }
    }
    protected internal void ToastBread()
    {
      lock(GetJamLock)
      {
        ToastBreadIsDone = true;
        Monitor.PulseAll(GetJamLock);
      }
      ToastBreadImpl();
      lock(MakeSandwichLock)
      {
        ToastBreadIsDone = true;
        Monitor.PulseAll(MakeSandwichLock);
      }
    }
    protected internal void GetJam()
    {
      lock(GetJamLock)
        if(!(ToastBreadStarted))
          Monitor.Wait(GetJamLock);
      GetJamImpl();
      lock(MakeSandwichLock)
      {
        GetJamIsDone = true;
        Monitor.PulseAll(MakeSandwichLock);
      }
    }
    protected internal void MakeSandwich()
    {
      lock(MakeSandwichLock)
        if(!(ToastBreadIsDone && GetJamIsDone))
          Monitor.Wait(MakeSandwichLock);
      MakeSandwichImpl();
      lock(EatBreakfastLock)
      {
        MakeSandwichIsDone = true;
        Monitor.PulseAll(EatBreakfastLock);
      }
    }
    protected internal void EatBreakfast()
    {
      lock(EatBreakfastLock)
        if(!(MakeTeaIsDone && MakeSandwichIsDone))
          Monitor.Wait(EatBreakfastLock);
      EatBreakfastImpl();
    }
  }
}

This is quite a bit of code! However, it sticks to the structure we defined, and produces methods which implement the asynchronous aspects of our process. Now we can fill in the gaps as follows:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading;
namespace Debugging
{
  partial class Breakfast
  {
    AutoResetEvent eatHandle = new AutoResetEvent(false);
    Random rand = new Random();
    public void Prepare()
    {
      ThreadStart[] ops = new ThreadStart[] {
        MakeTea,
        GetJam,
        ToastBread,
        MakeSandwich,
        EatBreakfast };
      foreach (ThreadStart op in ops)
        op.BeginInvoke(null, null);
      eatHandle.WaitOne();
    }
    private int RandomInterval
    {
      get
      {
        return (1 + rand.Next() % 10) * 100;
      }
    }
    public void MakeTeaImpl()
    {
      Thread.Sleep(RandomInterval);
      Console.WriteLine("Make tea");
    }
    public void ToastBreadImpl()
    {
      Thread.Sleep(RandomInterval);
      Console.WriteLine("Toast bread");
    }
    public void GetJamImpl()
    {
      Thread.Sleep(RandomInterval);
      Console.WriteLine("Get jam");
    }
    public void MakeSandwichImpl()
    {
      Thread.Sleep(RandomInterval);
      Console.WriteLine("Make sandwich");
    }
    public void EatBreakfastImpl()
    {
      Thread.Sleep(RandomInterval);
      Console.WriteLine("Eat breakfast");
      eatHandle.Set();
    }
  }
}

I’ve created a very simple function for starting all those processes asynchronously, and have also added an event so we know when the whole thing is over. Running this code outputs the following:

Make tea
Toast bread
Get jam
Make sandwich
Eat breakfast
All done

Though, of course, Make tea may appear after Toast bread on a different run. [5]

Conclusion

DSL Tools are a complicated, but powerful technology. Its key characteristic is the simplicity with which DSLs can be edited once their structure is defined. However, the definition of a DSL is no simple affair. This article has only scratched the surface in terms of the possibilities of DSL Tools. However, I hope it can serve as inspiration for trying out more complex designs. (And, believe me, it really is easier the second time round :)

Notes

1. As an example, take a look at the Boo programming language. A book on building DSLs using Boo is forthcoming. Another example is the yet to be released Microsoft Oslo.
2. For an example, take a look at the JetBrains Meta Programming System.
3. T4 stands for Text Templating Transformation Toolkit. Essentially, T4 is a way to transform XML files into code (or anything else) by letting them be read by template files (files having the .tt extension). These .tt files are, essentially, C# code that traverses the XML model and turns it into whatever you wish. For a definitive guide to all things T4, you need to take a look at the website of Oleg Sych.
4. The experimental hive is another registry hive that clones the one that comes with VS. It allows you to test Visual Studio in a ‘clean’ environment.
5. You’ll notice also that the operation is sensitive to the order in which the BeginInvoke() methods are executed. This is the specific trait of the Pulse & Wait mechanism – you need to start the waiting thread before pulsing it – otherwise, the pulse is wasted.