Learn Windows Workflow Foundation 4.5 through Unit Testing: Workflow Definition Persistence, Versioning and WorkflowIdentity

Overview

And this article is focused on persisting workflow definitions and versioning for long running / persistable workflows.

Background

Other articles related to this one are given below:

• Learn Windows Workflow Foundation 4.5 through Unit Testing: Persistence with WorkflowApplication
• Learn Windows Workflow Foundation 4.5 through Unit Testing: Workflow Services

For really long running workflows, using WorflowServiceHost and Workflow Persistence is the only way to go. However, somehow Workflow 4 and 4.5 had provided "weaker" supports for persistence, also, because of the deprecation of AppFabric, Workflow Management Service is not longer a viable solution. And we application developers have to write our own custom solutions to manage the lifecycle of workflows.

To some developers with experiences in earlier versions of Workflow Foundation prior to 4.0, such "weak" support for managing lifecycle of workflows is bad news resulting in great burdens for extra maintenance works and respective mindset shifting, unless the developers are working solely on Biztalk and Sharepoint with sophisticated workflow hosting.

Remarks

I consider that the move to not having built-in solution of persisting workflow definitions in WF 4 is a good move, since I think that many workflow applications do not need to persist the workflow definitions in a persistence layer, while the respective functions have knowledge of which workflow definitions to use when resuming a workflow instance, thus have no need for serialization and deserialization of workflow definitions. For scenarios that need persisting workflow definitions, it is not hard to write custom solutions for example a dictionary. And the most tricky part I regard is scanning workflows hibernated in the persistence layer and loading them timely if not using Sharepoint or Biztalk.

References:

• Create a Long-runing Workflow Service
• Persistence Best Practices
• Workflow Service Host Extensibility
• What's New in WF 4.5
• WF 4.x Samples
• Using WorkflowIdentity and Versioning

Using the Code

The source code is available at https://github.com/zijianhuang/WorkflowDemo.

Prerequisites:

1. Visual Studio 2015 Update 1 or Visual Studio 2013 Update 4
2. xUnit (included)
3. EssentialDiagnostics (included)
4. FonlowTesting (included)
5. Workflow Persistence SQL database, with default local database WF.

Examples in this article are from a test class:  WorkflowServiceHostTests, WorkflowServiceHostPersistenceTests, WFServiceTests, WCFWithWorkflowTests, NonServiceWorkflowTests.

WFDefinitionIdentityStore

In article Learn Windows Workflow Foundation 4.5 through Unit Testing: Persistence with WorkflowApplication , I had introduce a dictionary class that provide for looking up a workflow definition through an instance ID. In a service app, there might be multiple instances of the same workflow definition, thus it is inefficient to store multiple copies of the same definition, so you may be designing a storage either in program logic or persistence that will store a definition once and support versioning. While there could be many ways to design a workflow definition store that does not store duplicate copies of the same definition, however, using WorkflowIdentity could be a good option if you want versioning for long running workflows, and the definition identity is by default persisted in SQL table DefinitionIdentityTable.

public class WFDefinitionIdentityStore
{
    private static readonly Lazy<WFDefinitionIdentityStore> lazy = new Lazy<WFDefinitionIdentityStore>(() => new WFDefinitionIdentityStore());

    public static WFDefinitionIdentityStore Instance { get { return lazy.Value; } }

    public WFDefinitionIdentityStore()
    {
        InstanceDefinitions = new System.Collections.Concurrent.ConcurrentDictionary<WorkflowIdentity, byte[]>();

        Store = new SqlWorkflowInstanceStore("Server =localhost; Initial Catalog = WF; Integrated Security = SSPI")
        {
            InstanceCompletionAction = InstanceCompletionAction.DeleteAll,
            InstanceEncodingOption = InstanceEncodingOption.GZip,

        };

        var handle = Store.CreateInstanceHandle();
        var view = Store.Execute(handle, new CreateWorkflowOwnerCommand(), TimeSpan.FromSeconds(50));
        handle.Free();
        Store.DefaultInstanceOwner = view.InstanceOwner;

    }

    public System.Collections.Concurrent.ConcurrentDictionary<WorkflowIdentity, byte[]> InstanceDefinitions { get; private set; }

    public System.Runtime.DurableInstancing.InstanceStore Store { get; private set; }

    public bool TryAdd(WorkflowIdentity definitionIdentity, Activity a)
    {
        using (MemoryStream stream = new MemoryStream())
        {
            ActivityPersistenceHelper.SaveActivity(a, stream);
            stream.Position = 0;
            return InstanceDefinitions.TryAdd(definitionIdentity, stream.ToArray());
        }

    }

    public bool TryAdd<T>(WorkflowIdentity definitionIdentity, ActivityBuilder<T> ab)
    {
        using (MemoryStream stream = new MemoryStream())
        {
            ActivityPersistenceHelper.SaveActivity(ab, stream);
            stream.Position = 0;
            return InstanceDefinitions.TryAdd(definitionIdentity, stream.ToArray());
        }

    }

    public Activity this[WorkflowIdentity definitionIdentity]
    {
        get
        {
            return ActivityPersistenceHelper.LoadActivity(InstanceDefinitions[definitionIdentity]);
        }
    }
}

Remarks:

Many tutorials you could find in the Internet probably present a dictionary of mapping between WorkflowIdneity and Activity, however, binary Activity of CLR could hardly really persisted in persistence layers rather than memory, and only XAML representing the activity is a good one to persist. It shouldn't be hard for you to modify the code above and store the definition in SQL or NoSql.

Examples for WFDefinitionIdentityStore

IDictionary<string, object> LoadAndCompleteLongRunning(Guid instanceId, WorkflowIdentity definitionIdentity)
{
    bool completed2 = false;
    bool unloaded2 = false;
    AutoResetEvent syncEvent = new AutoResetEvent(false);

    var instance = WorkflowApplication.GetInstance(instanceId, WFDefinitionIdentityStore.Instance.Store);
    var definition = WFDefinitionIdentityStore.Instance[definitionIdentity];
    IDictionary<string, object> dic = null;
    var app2 = new WorkflowApplication(definition, instance.DefinitionIdentity)
    {
        Completed = e =>
        {
            completed2 = true;
            if (e.CompletionState== ActivityInstanceState.Closed)
            {
                dic = e.Outputs;
            }
        },

        Unloaded = e =>
        {
            unloaded2 = true;
            syncEvent.Set();
        },

        InstanceStore = WFDefinitionIdentityStore.Instance.Store,
    };

    stopwatch.Restart();
    app2.Load(instance);
    Trace.TraceInformation("It took {0} seconds to load workflow", stopwatch.Elapsed.TotalSeconds);

    app2.Run();
    syncEvent.WaitOne();
    stopwatch2.Stop();
    var seconds = stopwatch2.Elapsed.TotalSeconds;

    Assert.True(completed2);
    Assert.True(unloaded2);

    return dic;

}

[Fact]
public void TestWaitForSignalOrDelayVersion1()
{
    var a = new WaitForSignalOrDelay()
    {
        Duration=TimeSpan.FromSeconds(10),
        BookmarkName="Wakeup",
    };

    var definitionIdentity = new WorkflowIdentity("WaitForSignalOrDelay", new Version(1, 0), null);

    AutoResetEvent syncEvent = new AutoResetEvent(false);

    bool completed1 = false;
    bool unloaded1 = false;
    var app = new WorkflowApplication(a, definitionIdentity)
    {
        InstanceStore = WFDefinitionStore.Instance.Store,
        PersistableIdle = (eventArgs) =>
        {
            return PersistableIdleAction.Unload;
        },

        OnUnhandledException = (e) =>
        {
            return UnhandledExceptionAction.Abort;
        },

        Completed = delegate (WorkflowApplicationCompletedEventArgs e)
        {
            completed1 = true;
            syncEvent.Set();
        },

        Unloaded = (eventArgs) =>
        {
            unloaded1 = true;
            syncEvent.Set();
        },
    };

    var id = app.Id;
    app.Run();

    syncEvent.WaitOne();
    Assert.False(completed1);
    Assert.True(unloaded1);

    WFDefinitionIdentityStore.Instance.TryAdd(definitionIdentity, a);

    Thread.Sleep(5000); // from 1 seconds to 9 seconds, the total time of the test case is the same.

    var outputs = LoadAndCompleteLongRunning(id, definitionIdentity);

    Assert.False((bool)outputs["Result"]);

}

[Fact]
public void TestWaitForSignalOrDelayVersion2()
{
    var a = new WaitForSignalOrAlarm()
    {
        AlarmTime = DateTime.Now.AddSeconds(10),
        BookmarkName = "Wakeup",
    };

    var definitionIdentity = new WorkflowIdentity("WaitForSignalOrDelay", new Version(2, 0), null);

    AutoResetEvent syncEvent = new AutoResetEvent(false);

    bool completed1 = false;
    bool unloaded1 = false;
    var app = new WorkflowApplication(a, definitionIdentity)
    {
        InstanceStore = WFDefinitionStore.Instance.Store,
        PersistableIdle = (eventArgs) =>
        {
            return PersistableIdleAction.Unload;
        },

        OnUnhandledException = (e) =>
        {
            return UnhandledExceptionAction.Abort;
        },

        Completed = delegate (WorkflowApplicationCompletedEventArgs e)
        {
            completed1 = true;
            syncEvent.Set();
        },

        Unloaded = (eventArgs) =>
        {
            unloaded1 = true;
            syncEvent.Set();
        },
    };

    var id = app.Id;
    app.Run();

    syncEvent.WaitOne();
    Assert.False(completed1);
    Assert.True(unloaded1);

    WFDefinitionIdentityStore.Instance.TryAdd(definitionIdentity, a);

    Thread.Sleep(5000); // from 1 seconds to 9 seconds, the total time of the test case is the same.

    var outputs = LoadAndCompleteLongRunning(id, definitionIdentity);

    Assert.False((bool)outputs["Result"]);
}

Remarks:

Workflow versioning is a logical concept. As you can see in the examples above, 2 versions of "WaitForSignalOrDelay" are implemented by 2 different classes, and the implementation of version 2 is not necessarily the derived class of version 1. And the version information is injected during the instantiation of a workflow definition. 

It is the responsibility of the host application (more exactly you as the application developer) to provide the correct workflow definition when resuming instances, and ensure the uniqueness of mappings.

WorkflowDefinitionIdentityFactory

WorkflowIdentity supports great separation of concerns for workflow definition persistence and versioning. Sometimes in some applications you may not want to manually defining each WorkflowIdentity object, and may want to use workflow class names as the definition identity, particularly when you have fairly clean practices of .NET component designs and release controls.

This class has almost identical interface of WorkflowDefinitionIdentityStore, however, use CLR types and assemblies as the persistence media, rather than XAML. The DefinitionIdentity of a workflow could be generated through static function GetWorkflowIdentity().

public class WFDefinitionIdentityFactory
{
    private static readonly Lazy<WFDefinitionIdentityFactory> lazy = new Lazy<WFDefinitionIdentityFactory>(() => new WFDefinitionIdentityFactory());

    public static WFDefinitionIdentityFactory Instance { get { return lazy.Value; } }

    public WFDefinitionIdentityFactory()
    {
        InstanceDefinitions = new System.Collections.Concurrent.ConcurrentDictionary<WorkflowIdentity, Activity>();

        Store = new SqlWorkflowInstanceStore("Server =localhost; Initial Catalog = WF; Integrated Security = SSPI")
        {
            InstanceCompletionAction = InstanceCompletionAction.DeleteAll,
            InstanceEncodingOption = InstanceEncodingOption.GZip,

        };

        var handle = Store.CreateInstanceHandle();
        var view = Store.Execute(handle, new CreateWorkflowOwnerCommand(), TimeSpan.FromSeconds(50));
        handle.Free();
        Store.DefaultInstanceOwner = view.InstanceOwner;

    }

    public System.Collections.Concurrent.ConcurrentDictionary<WorkflowIdentity, Activity> InstanceDefinitions { get; private set; }

    public System.Runtime.DurableInstancing.InstanceStore Store { get; private set; }

    public bool TryAdd(WorkflowIdentity definitionIdentity, Activity a)
    {
        return InstanceDefinitions.TryAdd(definitionIdentity, a);
    }

    public Activity this[WorkflowIdentity definitionIdentity]
    {
        get
        {
            Activity activity = null;
            var found = InstanceDefinitions.TryGetValue(definitionIdentity, out activity);
            if (found)
                return activity;

            var assemblyFullName = definitionIdentity.Package;
            var activityTypeName = definitionIdentity.Name;
            System.Diagnostics.Trace.Assert(assemblyFullName.Contains(definitionIdentity.Version.ToString()));
            var objectHandle=  Activator.CreateInstance(assemblyFullName, activityTypeName);//tons of exceptions needed to be handled in production
            activity = objectHandle.Unwrap() as Activity;
            if (activity==null)
            {
                throw new InvalidOperationException("You must have been crazy.");
            }

            InstanceDefinitions.TryAdd(definitionIdentity, activity);
            return activity;

        }
    }

    public static WorkflowIdentity GetWorkflowIdentity(Activity activity)
    {
        var type = activity.GetType();
        var name = type.FullName;
        var assembly = type.Assembly;
        var package = assembly.FullName;
        var version = assembly.GetName().Version;
        return new WorkflowIdentity(name, version, package);
    }
}

It is presumed that in WorkflowIdentity, Name must be the full class name, Version must be the assembly version, and Package must be the assembly's FullName. Therefore, you must follow .NET component design strictly. For workflow versioning, you may consider the following designs:

1. Use numeric suffix for next version of workflow, for example, you have a workflow class BuyWorkflow, then next version could be BuyWorkflow2.
2. Modify WFDefinitionIdentityFactory and use Activity.DisplayName rather than Type.FullName for WorkflowIdentity.Name, and WorkflowIdentity.Package will store Type.AssemblyQualifiedName.

Examples

[Fact]
public void TestPersistenceWithDelayAndResult()
{
    var a = new Fonlow.Activities.Calculation();
    a.XX = 3;
    a.YY = 7;

    bool completed1 = false;
    bool unloaded1 = false;

    AutoResetEvent syncEvent = new AutoResetEvent(false);

    var definitionIdentity = WFDefinitionIdentityFactory.GetWorkflowIdentity(a);
    var app = new WorkflowApplication(a, definitionIdentity);
    app.InstanceStore = WFDefinitionStore.Instance.Store;
    app.PersistableIdle = (eventArgs) =>
    {
        return PersistableIdleAction.Unload;//so persist and unload
    };

    app.OnUnhandledException = (e) =>
    {

        return UnhandledExceptionAction.Abort;
    };

    app.Completed = delegate (WorkflowApplicationCompletedEventArgs e)
    {
        completed1 = true;

    };

    app.Aborted = (eventArgs) =>
    {

    };

    app.Unloaded = (eventArgs) =>
    {
        unloaded1 = true;
        syncEvent.Set();
    };

    var id = app.Id;
    stopwatch.Restart();
    stopwatch2.Restart();
    app.Run();
    syncEvent.WaitOne();

    stopwatch.Stop();
    Assert.True(stopwatch.ElapsedMilliseconds < 2500, String.Format("The first one is executed for {0} milliseconds", stopwatch.ElapsedMilliseconds));
    //the ellipsed time depends on the performance of the WF runtime when handling persistence. The first case of persistence is slow.

    Assert.False(completed1);
    Assert.True(unloaded1);

    stopwatch.Restart();
    var t = WFDefinitionIdentityFactory.Instance.TryAdd(definitionIdentity, a);
    stopwatch.Stop();
    Trace.TraceInformation("It took {0} seconds to persist definition", stopwatch.Elapsed.TotalSeconds);

    //Now to use a new WorkflowApplication to load the persisted instance.
    var dic =  LoadAndCompleteLongRunning(id, definitionIdentity);
    var finalResult = (long)dic["Result"];
    Assert.Equal(21, finalResult);

}

IDictionary<string, object> LoadAndCompleteLongRunning(Guid instanceId, WorkflowIdentity definitionIdentity)
{
    bool completed2 = false;
    bool unloaded2 = false;
    AutoResetEvent syncEvent = new AutoResetEvent(false);

    var instance = WorkflowApplication.GetInstance(instanceId, WFDefinitionIdentityStore.Instance.Store);
    var definition = WFDefinitionIdentityFactory.Instance[definitionIdentity];
    IDictionary<string, object> dic = null;
    var app2 = new WorkflowApplication(definition, instance.DefinitionIdentity)
    {
        Completed = e =>
        {
            completed2 = true;
            if (e.CompletionState== ActivityInstanceState.Closed)
            {
                dic = e.Outputs;
            }
        },

        Unloaded = e =>
        {
            unloaded2 = true;
            syncEvent.Set();
        },

        InstanceStore = WFDefinitionIdentityStore.Instance.Store,
    };

    stopwatch.Restart();
    app2.Load(instance);
    Trace.TraceInformation("It took {0} seconds to load workflow", stopwatch.Elapsed.TotalSeconds);

    app2.Run();
    syncEvent.WaitOne();
    stopwatch2.Stop();
    var seconds = stopwatch2.Elapsed.TotalSeconds;

    Assert.True(completed2);
    Assert.True(unloaded2);

    return dic;

}

[Fact]
public void TestWaitForSignalOrDelay()
{
    var a = new WaitForSignalOrDelay()
    {
        Duration=TimeSpan.FromSeconds(10),
        BookmarkName="Wakeup",
    };

    var definitionIdentity = WFDefinitionIdentityFactory.GetWorkflowIdentity(a);

    AutoResetEvent syncEvent = new AutoResetEvent(false);

    bool completed1 = false;
    bool unloaded1 = false;
    var app = new WorkflowApplication(a, definitionIdentity)
    {
        InstanceStore = WFDefinitionStore.Instance.Store,
        PersistableIdle = (eventArgs) =>
        {
            return PersistableIdleAction.Unload;
        },

        OnUnhandledException = (e) =>
        {
            return UnhandledExceptionAction.Abort;
        },

        Completed = delegate (WorkflowApplicationCompletedEventArgs e)
        {
            completed1 = true;
            syncEvent.Set();
        },

        Unloaded = (eventArgs) =>
        {
            unloaded1 = true;
            syncEvent.Set();
        },
    };

    var id = app.Id;
    app.Run();

    syncEvent.WaitOne();
    Assert.False(completed1);
    Assert.True(unloaded1);

    Thread.Sleep(5000); // from 1 seconds to 9 seconds, the total time of the test case is the same.

    var outputs = LoadAndCompleteLongRunning(id, definitionIdentity); //at this point, the workflow definition is not yet added into the dictionary of WFDefinitionIdentityFactory.

    Assert.False((bool)outputs["Result"]);

}

In the 2nd test case, the initial run of the workflow does not persist the workflow definition in the dictionary of WFDefinitionIdentityFactory. However, when reloading, WorkflowIdentity is sufficient enough for creating an instance of the definition through reflection. And when the workflow instance is persisted, the WorkflowIdentity object is saved in the DefinitionIdentityTable of the WF SQL database by Workflow runtime.

Side by Side Versioning in Web-hosted Workflow Service

There exist quite a few good articles about side by side versioning:

• Side by Side Versioning in WorkflowServiceHost
• Side by side Versioning of workflow services

If you want multiple versions of a workflow service side by side and each will receive new requests and create new instances, you may:

1. Ccreate a new folder under the project folder, so the folder name will become a path segment of the new endpoint address for newer version of the workflow service,
2. Copy the xamlx file to there
3. Update the features with same contract, same binding, but different endpoint address.

However, you must not name the new folder as "app_code", since WF runtime uses this folder for special purpose, as described by MSDN below.

MSDN:

The WorkflowServiceHost side-by-side versioning introduced in .NET Framework 4.5 provides the capability to host multiple versions of a workflow service on a single endpoint. The side-by-side functionality provided allows a workflow service to be configured so that new instances of the workflow service are created using the new workflow definition, while running instances complete using the existing definition.

So no new workflow instance can be created for the old workflow definition.

If you use a self-hosted workflow service, you will need to do the house keeping works as you do in Examples for WFDeiniitionIdentityStore above, and each versions implemented in different activity classes, and use WorkflowIdentity to announce that they are the same service but at different versions.

If you use a Web-hosted Workflow Service, WF runtime will handle much of the house keeping works for you. The SupportedVersions property of the primary service will be populated during runtime when the service is instantiated. Upon receiving a client request with an existing session Id for an existing workflow instance persisted in the SQL database, WF runtime will load the old version of workflow definition stored in ~/app_code/MyVersionedService/MyVersionedServiceVx.xamlx for further execution of the same workflow instance.

Remarks:

By the nature of continuing remaining instances of old version and creating new instances of new version, it is impossible to create automatic testing to demonstrate the behaviors, because the behaviors are not repetitive within a single build of the service. However, you may observe the behaviors through following the development cycles of 2 versions of the workflow service as described in Side by Side Versioning in WorkflowServiceHost.

Points of Interest

Persistence of Workflow Definitions

Class WFDefinitionStore basically compensates what available in WF 3.5 but being missed in WF 4.0 and 4.5, that is, persisting workflow definition along with each instance of the workflow. If you have legacy WF 3.5 applications and would move to WF 4.5, this class may be a good starting point for you to migrate with least effort.

Class WFDefinitionIdentityStore enables storing one workflow definition for all instances of the workflow.

Both WFDefinitionStore and WFDefinitionIdentityStore serialize a workflow into XAML. When the application loads and runs a persisted workflow, the application does not even need to know and load the workflow type and its assembly, since the XAML is sufficient enough to provide the workflow logic, and the deserialization does not return the original workflow type but type Activity.

Class WFDefinitionIdentityFactory has almost identical interface of WFDefintionIdentityStore, however with very different mechanism of persisting workflow definitions. WFDefinitionIdentityFactory uses .NET assemblies as the ultimate persistence media rather than XAML, so there's no need for SQL database or NoSql for persistence.

WorkflowIdentity and Versioning

The move to not having built-in solution of persisting workflow definitions in WF 4 is a good move/design, since many workflow applications do not need to persist the workflow definitions in a persistence layer, while the respective functions have knowledge of which workflow definitions to use when resuming a workflow instance, thus have no need for serialization and deserialization of workflow definitions. For scenarios that need persisting workflow definitions, WFDefinitionStore and WFDefinitionIdentityStore and WFDefinitionIdentityFactory may give you some hints. And the flexibility through versioning supported by WorkflowIdentity may lead to higher performance of workflow definition reload for variety of scenarios.  

Workflow Service and WCF

If you have read WCF for the Real World, Not Hello World or similar articles, you see that you could have separated projects of WCF service library and WCF service application. And you could carry out most of the development in WCF service library and use the WCF service application as a simple facade container containing multiple WCF service library assemblies for further configuration during deployment. So a WCF service library could be freely hosted in IIS or a self-hosted application. Therefore you could have very clean separation of concerns over development and deployment, that is, during development, you as a developer don't need to think about deployment.

For WCF Workflow service, you may have to carryout most of the development inside a WCF Workflow service application project and design most of the workflow logic inside an XAMLX file. There seems no instrument or interface in WF for constructing something like WCF service library, which could be used in both IIS-hosted deployment and self-hosted deployment. So in SDLC, you may have to think about the deployment of workflows or workflow services at early stages. If you have other ideas, please leave a comment.