Click here to Skip to main content
15,867,288 members
Articles / DevOps / Unit Testing

Screenshot Smoke Testing

Rate me:
Please Sign up or sign in to vote.
4.63/5 (8 votes)
28 Nov 2013CPOL5 min read 35.9K   463   22   2
Leveraging Selenium and Image Magick to perform screenshot comparisons for UI regression testing.

Introduction

We are leveraging Selenium 2 for web page functional testing. If you are not familiar with Selenium, it is basically used for web automation either on its own or within scripts. You can use a Firefox plugin to record your steps into a script and then export that script into junit (Java), rspec (ruby) or nunit (C#) unit tests. I started to write some basic selenium tests to make sure a checkin doesn't break something else within the solution. Some simple things such as making sure the page loads and a few items exist on the page. The more checks you have in your test, the more confidence you can have in your spoke test that everything is in place. This got me to thinking that it would be nice to just have the test visually compare the newly checked-in page with the page prior to the check-in and make sure everything was working.

Background

ImageMagick has a lot of capabilities but the key one we will be focusing on is its compare capabilities. If we were to take a screenshot of the page when it is in the state that we like and use that to compare against the newly built page, we could then leverage ImageMagick's compare functionality to tell us if something is wrong. There might be some light pixel shifting for whatever reason that would be acceptable but we can adjust the compare to fail when the difference is at a certain level. I saw this post that gave me the idea to do this: Stack Overflow Post. Here, you can see the use of the PAE metric - which is the Peak Average Error and seemed to work well. Magick.net over at CodePlex is a nice wrapper that makes it a bit easier to use ImageMagick within a .NET project.

Using the Code

Image 1The VisualSmoke solution is for demonstration purposes of this approach and consists of a standard ASP.NET MVC Razor 2 project called VisualSmoke.Web which is the web application our test project will hit. The VisualSmoke.Web.Tests project is the meat of the article.

Note that there are a few NuGet packages for the VisualSmoke.Web.Tests project:

  • NUnit - NUnit is a unit-testing framework for all .NET languages with a strong TDD focus.
  • Selenium packages (4) - Allows for Selenium server to get started up and leverage it during our unit tests
  • Magick.NET-Q16-x86 - A .NET API to the ImageMagick image-processing library for Desktop and Web.

Test Overview

SetupTest()

Set some properties that will be used during the test so it knows:

  • what URLs it should hit
  • what directories it should read and write to for the screenshots
  • Calls the PopulateUrls() method which we will go into detail next
  • Start the Selenium server so we can run our selenium based tests
C#
[SetUp]
public void SetupTest()
{
    ActiveShots = ConfigurationManager.AppSettings["activeShotsPath"];
    ApprovedShots = ConfigurationManager.AppSettings["approvedShotsPath"];
    ErroredShots = ConfigurationManager.AppSettings["erroredShotsPath"];
    DeltaShots = ConfigurationManager.AppSettings["deltaShotsPath"];
    BaseDomain = string.Format("http://{0}/", 
    		ConfigurationManager.AppSettings["domain"]);
    CompareDomain = string.Format("http://{0}/", 
    		ConfigurationManager.AppSettings["compareDomain"]);
    PopulateUrls();
    selenium = new DefaultSelenium("localhost", 4444, "*chrome", 
    string.Format("http://{0}/", ConfigurationManager.AppSettings["domain"]));
    selenium.Start();
    verificationErrors = new StringBuilder();
}

PopulateUrls()

  • Read the ApprovedPages.txt file for the list of pages we should be Screenshot Smoke Testing.
  • We have a simple "Page" class that contains the following properties to make it easier to deal with the data:
    • Name
    • ApprovedURL - the URL of the page that is approved
    • ActiveURL - the URL of the page that is being tested
C#
private void PopulateUrls()
{
    UrlList = new List<page>();
    const string file = "VisualSmoke.Web.Tests.Selenium.ApprovedPages.txt";
    using (var tr = GetInputFile(file))
    {
        string line;
        while ((line = tr.ReadLine()) != null)
        {
            var page = new Page();
            page.Name = line;
            if (line == "home")
            {
                page.ApprovedUrl = CompareDomain;
                page.ActiveUrl = BaseDomain;
            }
            else
            {
                page.ApprovedUrl = CompareDomain + line;
                page.ActiveUrl = BaseDomain + line;
            }
            UrlList.Add(page);
        }
    }
}

ApprovedPages.txt

The ApprovedPages.txt file allows for the control of which pages should be smoke tested without having to create a unit test for each page.

*Note* - The ApprovedPages.txt file has the properties set to Embedded Resource which makes it so the location of the file when pushed to the CI environment irrelevant.

home
products
solutions
about-us

ScreenshotCompareTest()

  • The ScreenshotCompareTest() will capture a baseline of approved pages for comparison if the AppSetting of "baseline" is true.
  • It will then iterate through the UrlList of pages to test and then Use Selenium.CaptureEntirePageScreenshot to capture and save the screenshot, leveraging our CaptureImagePath method.
  • We then compare the captured image with the approved counterpart using the CompareImage method
  • The CompareImage will return either a Match, NoMatch or Error
  • If we do not have a Match, then we make note of it in our verificationErrors StringBuilder so we can use it in the message at the end of the test for the Assert message.
C#
[Test]
public void ScreenshotCompareTest()
{
	if (ConfigurationManager.AppSettings["baseline"] == "true")
	{
		CaptureApprovedPages();
	}

	foreach (var url in UrlList)
	{
		selenium.Open(url.ActiveUrl);
		selenium.WaitForPageToLoad("30000");
		selenium.CaptureEntirePageScreenshot(CaptureImagePath(url.Name), "");
		var result = CompareImage(url.Name);
		if (result != CompareResult.Match)
		{
			if (result == CompareResult.Error)
			{
				verificationErrors.AppendLine(url.Name + " : " + 
                                                                    LastException.Message);
			}
			else
			{
				verificationErrors.AppendLine(url.Name + " 
				did not match the approved screenshot. " + string.Format
				("{0}images/screenshots/delta/{1}.png", 
                                                               UrlList[0].ActiveUrl, url.Name));
			}
		}
	}
}

CaptureApprovedPages()

When an environment is in an approved state for the QA and/or Business User, then set the appSetting baseline to true. This will iterate through the UrlList, take a screenshot and save the images to the approved image directory.

C#
private void CaptureApprovedPages()
{
	foreach (var url in UrlList)
	{
		selenium.Open(url.ApprovedUrl);
		selenium.CaptureEntirePageScreenshot(ApprovedImagePath(url.Name), "");
	}
}

CompareImage

The CompareImage method is the main method that does the ImageMagick work to compare the images. If the compare result falls outside an acceptable range, then we save the difference to the delta folder for review by a human when the test fails. Below we are using the PeakAbsoluteError. I am still playing around with the acceptable range and even if PeakAbsoluteError is the metric to use but I am fairly certain that a result of 1.0 is certainly unacceptable.

C#
private CompareResult CompareImage(string name)
{
	var activeImage = new MagickImage(CaptureImagePath(name));
	var approvedImage = new MagickImage(ApprovedImagePath(name));

	Assert.IsNotNull(activeImage);
	Assert.IsNotNull(approvedImage);
	var result = CompareResult.NoMatch;
	try
	{
		using (var delta = new MagickImage())
		{
			var compareResult = activeImage.Compare
                                   (approvedImage, Metric.PeakAbsoluteError, delta);

			if (compareResult < .9)
			{
				result = CompareResult.Match;
			}
			else
			{
				delta.Write(string.Format("{0}\\{1}.png", DeltaShots, name));
			}
		}
	}
	catch (Exception ex)
	{
		result = CompareResult.Error;
		LastException = ex;
	}

	return result;
}

Image Comparison Example

Below, you can see a test after I updated the home page to have additional text added after the approved screenshots were taken. This causes the home page comparison to not match and the additional text shows up in red on the delta comparison image.

Updated Page

Image 2

Approved Page

Image 3

Comparison

Image 4

The Unit Test Failure Report

Unit tests normally stop after a failure, but when you are doing Selenium tests, you would tend to continue running the script and save up all the issues until the end. Below, you can see the results of the test. I have also added a link to the delta image in the error message so someone reviewing the test can simply click on the link to see the issue(s) that have occurred.

Image 5

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)


Written By
Technical Lead Independent Consultant
United States United States
___________________________
J A M E S C O L E M A N
Director, Technical Services
Linked-In: http://www.linkedin.com/in/jameswcoleman
Blog: ledtalks.wordpress.com

Comments and Discussions

 
Generalvery enlightening Pin
Southmountain28-Nov-15 15:02
Southmountain28-Nov-15 15:02 
Questionerror while compiling Pin
fredatcodeproject11-Dec-14 11:59
professionalfredatcodeproject11-Dec-14 11:59 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.