WPF UIAutomation test suite development

• Download AvalonEdit_05_03.CodedUITest.zip
• Download AvalonEdit_05_03.zip

• Introduction
  • Objectives
• Test Scenarios
  • Functional Testing (Black Box)
  • Test Example
  • Design
  • Record test and generate test method
  • Update the Test method and Build
• Test Case Specification
  • TestMethod
  • Initialize Methods
  • Implement a Custom TestContext
  • Test Configuration
• FAQ on CodedUI
  • Do not change the code in *.Designer.cs file manually
  • Frequent Exceptions
  • NullReferenceException
  • Cannot locate a UI control
• Conclusions
• References

Introduction

The quality of software can be determined through coding guidelines and automated tests. There are typically unit tests to ensure that the quality of small units (eg. methods) meets expectations. And in large enough projects, there are other automated tests, such as, User Interface (UI) tests, to ensure that the user experience is as expected.

This article is about test automation for User Interface (UI) tests. We use Microsoft's latest test environment CodedUI as a sample to explain how test automation can be integrated into a software development iteration. We found out during the writing of this article that CodedUI is going to be deprecated with the end of Visual Studio 2019, and while this is still sometime to go from the article's publication date in 2019, its also worthwhile to point out that many principles discussed here can also be applied to other UI test frameworks.

UI Test automation for HTML has been around for a while. And while an open source test environment like Selenium has been the state of the art, there are also other open source projects like Appium that can be used to test WPF applications.

This article lists important practices that should be implemented when developing any automated UI tests. We explain basic features and use cases of the CodeUI test framework and use this to show that methods and properties can be extended, such that previously untestable parts of the test target, can be made testable with automation.

An important pre-requisite for working with Microsoft's CodedUI is that you have Visual Studio Enterprise installed.

Objectives

UI test automation has many similarities to normal software development but it also has unique properties that set it apart from other sorts of developments. We use this section to discuss best practices that should be observed when using UI automation in a real world project. The following high-level objectives should be observed to make sure that normal development can take place despite the additional effort for UI test automation:

• Introduce automated tests in an early iteration of your WPF app,
• Keep test code easy to write and simple to maintain,
• Keep test suites simple to deploy and run.

The above points can be extended with important principles of Automation Testing. These principles ensure that you do not end up just testing anything but actually start at the most important points in the project and can work your way to less important things. These principles are:

• Start with tests on the external interface,
• Keep test targets for each test method simple and clear (mixing too much logic in a test method is not a good idea),
• Do not modify the test object or test environment,
• Minimize repetitive testing
  • It is impossible to test all combinations,
  • Choose the most useful combinations,
  • Avoid redundant test coverage,
• Balance development progress with test progress.

Some points, like the last point above address important issues like maintainability and additional effort for automated UI testing. It is easy to see that a simple project with a simple user interface hardly justifies any additional effort for test automation. But there are also projects with products that are used by a few hundred or thousand customers being produced by a large (de-centralized) team of software developers. These projects may profit from test automation despite its additional effort, because the regular testing through automation can ensure that all parts of the UI can pass a basic test before the last stage (fixing bugs) of a development cycle is entered.

Test Scenarios

There are many types of tests that can be executed during software development. These are:

• Unit Testing,
• Integration Testing,
• Functional Testing,
• Acceptance Testing,
• Performance Testing,
• and more [1].

We detail functional testing and its automation in the sections below to illustrate how a user interface (UI) can be tested by code. The development of a functional test has similarities to software development itself since we can apply best practices, patterns, and lessons learned. An "obvious" best paractice is the application of a high-level design pricinples that involve at least two principles and is applicable for each test:

1. Start with software requirements. It is critical to identify requirements for testing in a comprehensive manner, since this will directly affect the test coverage.
    
2. For each test requirement, always use:
   1. Equivalence Class Partitioning [2],
   2. Boundary Value Analysis [3], and
   3. deduce errors.
   to make test cases comprehensively.

A comprehensive test case is a case that starts with the user requirements, because it ensures that the user's likely behavior is reflected in the required quality. This can be ensured by using the listed patterns and taking the time to deduce errors that may not have been evaluated up to this point.

Functional Testing (Black Box)

Functional Testing is a type of software testing whereby the software is tested against a functional requirement in a specification. A functional test is also known as a Black Box test, because the test regards the system being tested as a box that cannot be opened to look inside it. This type of test is focused around expectations about system input, system output, and behaviors during the test, instead of looking at the inner details of a system. A story based functional requirement looks like this:

1. The User starts the System,
2. The User changes a program setting,
3. The User clicks the Close (button),
4. The System saves all changed program settings, and
5. The System stops running

A test against this requirement would be to determine that the software does indeed save all relevant data and shuts down correctly. This test could be performed manually by a test engineer, but automation has many advantages over a manual approach. This section shows how to design and perform automated-functional testing with UI Automation for WPF.

The following steps are usually involved in this process:

Microsoft UI Automation is a new accessibility framework for Microsoft Windows. It addresses the needs of assistive technology products and automated test frameworks by providing programmatic access to information about the user interface (UI) [4].

Visual Studio Coded UI [5] encapsulates the UI Automation API and provides a tool (CodedUITestBuilder.exe) to record the user operation, and generate the test code automatically. Note that Visual Studio Coded UI requires Visual Studio Enterprise and Coded UI test components for recording.

We detail automated functional testing with the above tools in the next sections below.

Test Example

The Test Target of a test is usually a technical unit: a control, a library, or a whole software product. The test target in this section is the open source control AvalonEdit (Version 5.0.3) (source code on Github) and we design a test case to verify its font setting function.

Design

Our sample test case design is the following:

1. Launch the AvalonEdit.Sample application,
2. Set the font size to a specified valid value (eg: 16),
3. Verify that the text font size is the same as it was set in the last step.

It is worthwhile to note that the above test case design is very similar to the workflow identified in the functional testing section. This similarity supports the idea that a user should be simulated by a software component that drives the test target and it enables the common practice to either implement a functional test case directly, as automated test, or implement a close enough variation of the given functional test.

Record test and generate test method

The test component that drives the test target is in our case the CodedUI Test Framework. The test recording with CodedUI requires a seperate setup and type of project [5] [6].

The Recording of test steps and the generation of a test method (with CodedUITestBuilder.exe) can be done with the CodedUI Test Builder interface: [5]

The above user interface can be used inside a CodedUI test project to record and generate test code that implements the above test case [5], click the "Record/Pause/Resume" button first, and then performe the steps from 2 to 3, and generate the test code by click the "Generate code" button, then the code should look as below (it is better to refine the SetFontSize method later to make the font size be an input parameter):

CodedUITest class

[TestMethod]
public void CodedUITestMethod1()
{
    // To generate code for this test, select "Generate Code for CodedUI Test" from the shortcut menu and select one of the menu items.
    this.UIMap.SetFontSize();
}

UIMap class

/// <summary>
/// SetFontSize - Use 'SetFontSizeParams' to pass parameters into this method.
/// </summary>
public void SetFontSize()
{
    WinEdit uIFontEdit = this.UIAvalonEditSampleWindow.UIItemToolBar.UITxtFontEdit;
    WinEdit uIFontSizeEdit = this.UIAvalonEditSampleWindow.UIItemToolBar.UITxtFontSizeEdit;
   
    // Type '16' in 'FontSize' text box
    uIFontSizeEdit.Text = this.SetFontSizeParams.UIFontSizeEditText;

    // Click 'Font' text box
    Mouse.Click(uIFontEdit);
}

The UIMap file is the core file which supports the UI elements and operations in the test project. The test project contains at least 3 types of files: *.uitest, *.cs, and *.designer.cs. The *.uitest is an xml file. Both *.uitest and *.designer.cs files are generated automatically (they should not be changed manually, because the change will be lost after the next record operation). So, to add custom code, you should double click the *.uitest file and click the Move button (marked yellow in the image below).

The RecordedMethod1 is in our sample in the UIMap.cs file after clicking the Move button. We can rename the method to SetFontSize and add a parameter. Don't worry the code won't be changed automatically by the CodedUI Test Builder tool now.  Please refer to the FAQ section for more hints like this.

Then, when we have recorded the initial test steps, its time to verify the result. To do that, just drag the Add assert button into the editor area (the editor area is automatically marked blue as shown below) to add an assert method that verifies that the text font in AvalonEdit has changed as expected.

The list of possible assertions is shown in the Add Assertions tool window. The font size property is missing here. It seems like we cannot get the font size through CodedUI Test Builder, because AvalonEdit does not expose the font property for UIAutomation. And checking the source code of AvalonEdit, leads us to find that it implements the ITextRangeProvider interface, but it returns nothing useful for us in the GetAttributeValue method:

public object GetAttributeValue(int attribute)
{
    Log("{0}.GetAttributeValue({1})", ID, attribute);
    return null;
}

So, to make the verification step testable, we can either refine the GetAttributeValue method to expose the font property, or in the situation where the source code cannot be changed, we can implement a sub class that inherits from the control to be tested (the class to be sub classed here is the TextEditor class).

The OnCreateAutomationPeer method should be overriden to return a custom AutomationPeer sub class, which implements the related pattern provider to expose related properties. For more information about programmatic access to UI elements on the desktop, please refer to UIAutomation Fundamentals [4].

In this example, the GetAttributeValue method can be refined as shown below (see download sample attached to article or source code on GitHub: ICSharpCode.AvalonEdit\Editing\TextRangeProvider.cs):

public object GetAttributeValue(int attribute)
{
  if (AutomationTextAttribute.LookupById(attribute) == TextPatternIdentifiers.FontSizeAttribute)
  {
      return this.textArea.FontSize;
  }
  else
  {
    if (AutomationTextAttribute.LookupById(attribute) == TextPatternIdentifiers.FontNameAttribute)
      return this.textArea.FontFamily.Source;
    else
      return null;
  }
}

Now we are ready to rebuild the AvalonEdit.Sample, drag the assert button again, and we will find that the Font property gets a font name value of the current text. There is still no FontSize property. Microsoft provided a way for creating custom extensions to the CodedUI test framework that can support specific user interfaces [7]. But this is hard to debug and, worse still, deprecated by Microsoft. So, this solution is not recommended here.

Direct access to the FontSize property is in fact simple with UIAutomation. Go back to the FontSize example, in the CodedUI test project and add the following verification method manually to implement the FontSize property assertion:

public void SetFontSize(double fontSize)
{
    this.UIAvalonEditSampleWindow.UIItemToolBar.UITxtFontSizeEdit.Text = fontSize.ToString();

    Mouse.Click(this.UIAvalonEditSampleWindow.UIItemToolBar.UITxtFontEdit);
}

public void AssertMethod1(double expectedFontSize)
{
    WpfControl uiTestControl = this.UIAvalonEditSampleWindow.UITextEditorDocument.UIItemDocument;

    AutomationElement automationElement = uiTestControl.NativeElement as AutomationElement;

    if (automationElement.TryGetCurrentPattern(TextPattern.Pattern, out object pattern))
    {
        object obj = ((TextPattern)pattern).DocumentRange.GetAttributeValue(TextPattern.FontSizeAttribute);
        double actualFontSize = Convert.ToDouble(obj);
        Assert.AreEqual(expectedFontSize, actualFontSize, "Font size verified.");
    }

    Assert.Fail("Assert fail message.");
}

The SetFontSize method can be used to change the font size of the test target when all precondition are met. The AssertMethod1 on the other hand verifies whether a given font size is currently set or not, and throws an Assertion failed if not. A similar pattern can be used to implement FontSize get/set methods within a test method as we will see in the next section.

Update the Test method and Build

Testing the user interaction and verifying the correct result can be implemented in 2 seperate methods. The complete test method can then be implemented with this code (see download sample Download AvalonEdit_05_03.CodedUITest.zip):

[TestMethod]
public void SetFontSizeTest()
{
    double expectedFontSize = 16;
    this.UIMap.SetFontSize(expectedFontSize);

    this.TestContext.WriteLine($"Step 1: set the font size to {expectedFontSize.ToString()}.");
    double actualFontSize = this.UIMap.GetFontSize();
    this.TestContext.WriteLine($"Step 2: get the current font size of the TextEditor: {actualFontSize}.");

    Assert.AreEqual(expectedFontSize, actualFontSize, "");

    this.TestContext.WriteLine("Step 3: confirm the current font size of the TextEditor is same as set in step 1.");
}

Now we can build and run the test method with the Visual Studio Test platform (via IDE) or via command line and the test should perform successfully. The test result can be viewed in Visual Studio:

Test Case Specification

TestMethod

A method marked with the TestMethod attribute can automatically be recognized as a test case. There may also be other instances or methods related to the test method. These may be necessary to initialize or build the precondition to use a TestContext during the execution. This section explains methods of initialization and gives an example to implement a custom TestContext.

Initialize Methods

An initialize method can be implemented at different levels of execution. Currently, there are 4 types of initialize methods listed below in order of top-down granularity:

• AssemblyInitialize: Executes only once, before all test methods, in the same assembly as the test methods,
   
• ClassInitialize: Executes only once, before all test methods, in the same class as the test methods,
   
• Constructor: Executes every time before a test method is executed (a new instance is created first, if the test method is an instance method)
   
• TestInitialize: Executes every time before a test method is executed.

Every initialize method has a cleanup method (a Dispose method can be regarded as clean up method of a constructor) and is executed in reversed sequence.

Implement a Custom TestContext

Here is a sample to show a way for implementing and using a custom TestContext to replace the default.

public class CustomTestContext : TestContext
{
    private TestContext testContext;
    public CustomTestContext(TestContext testContext) { this.testContext = testContext; }
    public override void WriteLine(string format, params object[] args)
    {
        // Perform custom behavior such as re-direct write stream.
    }
    
    public override DataRow DataRow => this.testContext.DataRow;
}

The next code sample shows how the TestContext instance can be used from within a test. The test class implements a property with type 'TestContext' and name 'TestContext', both the name and the type can't be changed because the property setter executes automatically with the MSTest Framework before the actual test is executed.

[TestClass]
public class Test
{
    [TestMethod]
    public void TestMethod1()
    {
    }

    private CustomTestContext testContext;

    public TestContext TestContext
    {
        get
        {
            return this.testContext;
        }
        
        set
        {
            this.testContext = new CustomTestContext(value);
        }
    }
}

Any test can be influenced by many global variables and we have seen here how the TestContext can be used within a test. The next section gives a short overview on test configuration files which should also influences tests on a global level.

Test Configuration

There are 2 types of configuration files in the Visual Studio Test Platform:

1. .runsettings and
2. .testsettings

The .runsettings file is used to configure unit tests [8] and the .testsettings file was used by versions of Visual Studio that are older than VS 2019. The old .testsettings file can be referenced and used with the .runsettings file as shown below:

<RunSettings>
  <MSTest> 
    <SettingsFile>my.testsettings</SettingsFile>
    <ForcedLegacyMode>true</ForcedLegacyMode>
  </MSTest>
</RunSettings>

FAQ on CodedUI

Do not change the code in *.Designer.cs file manually

There are 3 files generated by the CodedUI Test Builder. The extensions of these files are:

1. *.uitest,
2. *.cs and
3. *.Designer.cs.

The *.Designer.cs file is updated after each recording. So, you should not change it since Visual Studio is likely to overwrite your changes anyway. A custom method should always be moved to a *.cs file. You can do this through double clicking the *.uitest file and clicking the Move button shown in the following screenshot:

Frequent Exceptions

There are 2 exceptions (NullReferenceException and "Cannot locate a UI control") that are typically caused by a threading issue. This section explains the issue and suggests a solution.

NullReferenceException

The CodedUI Test is running in Single Thread Apartment (STA) mode of COM (Common Object Model). In this mode, all playback calls should be invoked from the TestMethod thread, only, and the UITestControl should not be shared across TestMethods.

The AssemblyInitialize and ClassInitialize methods, are for example, static methods that are exectued in different threads with their TestMethod. It is therefore, required to initialize a playback enviroment manually and create a different UIMap instance, before invocation of any UIMap method takes place. This can typically be done with this try pattern:

[ClassInitialize]
public static void MyClassInitialize() 
{     
    Playback.Initialize();     
    try     
    {         
         UIMap uiMap = new UIMap();
         uiMap.DoSomethin(); 
    }     
    finally     
    {         
        Playback.Cleanup();    
    } 
}

Cannot locate a UI control

The detailed error message for this issue is: "Search may have failed at 'XXControl'

This error can be caused by a bug in the CodedUI test builder that didn't record the right ancestor of the target control.

A custom control may have virtualized children. If the control being searched is a descendant of ' XXControl' Custom then including it as the parent container may solve the problem."

For example, if there is a control named A that has a child nameed B, and B has a child named C. And the CodedUI test builder faild to detected the correct tree structure and determined A as container of C. Then you need to update the structure manually to fix this issue as shown in the code snippet below:

uiMap.A.C.Container = uiMap.B;

uiMap.B.Container = uiMap.A;

Conclusions

This article has tried to introduce the reader to the world of automated UI testing. Instead of black magic we are recommending patterns from lessons learned towards integrating UI tests into an existing development cycle. We also showed how a property that seemed to be untestable with UI tests, can be made testable through a custom test class that was derived from the original test target.

We hope this article is useful to anyone interested. Please let us know if you have additional questions or mistakes you might find in the article.

References

1. Different Types of Testing
   https://www.atlassian.com/continuous-delivery/software-testing/types-of-software-testing
    
2. Equivalence class partitioning
   https://en.wikipedia.org/wiki/Equivalence_partitioning
    
3. Boundary value analysis
   https://en.wikipedia.org/wiki/Boundary-value_analysis
    
4. Microsoft UI Automation
   https://docs.microsoft.com/en-us/windows/desktop/WinAuto/entry-uiauto-win32
    
5. Use CodedUI test to test your code
   https://docs.microsoft.com/en-us/visualstudio/test/use-ui-automation-to-test-your-code?view=vs-2019
    
6. UIAutomation Fundamentals
   https://docs.microsoft.com/en-us/windows/desktop/WinAuto/entry-uiauto-win32
    
7. CodedUI Test Extension for 3rd party controls
   https://devblogs.microsoft.com/devops/coded-ui-test-extension-for-3rd-party-controls-the-basics-explained/
    
8. Configure unit tests by using a .runsettings file
   https://docs.microsoft.com/en-us/visualstudio/test/configure-unit-tests-by-using-a-dot-runsettings-file?view=vs-2017
    

History

Keep a running update of any changes or improvements you've made here.