Click here to Skip to main content
15,867,568 members
Articles / Programming Languages / C#

Assessment System for Tests Architecture Design- SpecFlow Based Tests

Rate me:
Please Sign up or sign in to vote.
5.00/5 (3 votes)
21 May 2017Ms-PL5 min read 4.8K   3  
We will use the previously presented assessment framework to evaluate SpecFlow based tests. Find the ratings and reasoning behind them.

Introduction

In my previous article, Assessment System for Tests' Architecture Design, I presented to you eight criteria for system tests architecture design assessment. To understand the system fully, I am going to use it to evaluate a couple of real-world examples, assign ratings to them for each criterion and tell you my reasoning behind them. The third type of tests that we will assess is the SpecFlow based tests. If you are not familiar with SpecFlow, you can check my SpecFlow Series.

SpecFlow Based Tests

Feature and Scenario

The SpecFlow uses the Gherkin DSL to describe the behaviour of the system using human-readable syntax. It uses the so-called specifications where the test scenarios are described through Gherkin sentences. On build, the DSL is compiled to MSTest tests.

C#
Feature: Create Purchase in Amazon
	In order to receive a book online
	As a client
	I want to be able to choose it through the browser and pay for it online

@testingFramework
Scenario: Create Successful Purchase 
When Billing Country Is United States with American Express Card
	When I navigate to "/Selenium-Testing-Cookbook-Gundecha-Unmesh/dp/1849515743"
	And I click the 'buy now' button
	And then I click the 'proceed to checkout' button
	#When the login page loads
	And I login with email = "g3984159@trbvm.com" and pass = "ASDFG_12345"
	#When the shipping address page loads
	And I type full name = "John Smith", 
	country = "United States", Adress = "950 Avenue of the Americas", 
	city = "New Your City", state = "New Your", 
	zip = "10001-2121" and phone = "00164644885569"
	And I choose to fill different billing, full name = "John Smith", 
	country = "United States", Adress = "950 Avenue of the Americas", 
	city = "New Your City", state = "New Your", 
	zip = "10001-2121" and phone = "00164644885569"
	And click shipping address page 'continue' button
	And click shipping payment top 'continue' button
	Then assert that order total price = "40.49"

Binding Methods

Another thing you have to do is to define binding methods for every sentence used in your scenarios. Otherwise, the tests will fail. These bindings are defined in standard C# classes marked with Bindings attribute. Each step method is marked with step type attribute containing the step's regex pattern. Inside the steps' methods, the pages' logic is called. This way, every step defines and executes a small part of the test's workflow. With this approach of tests, writing the standard MSTest classes don't exist.

C#
[Binding]
public class CreatePurchaseSteps
{
    [When(@"I navigate to ""([^""]*)""")]
    public void NavigateToItemUrl(string itemUrl)
    {
        var itemPage = UnityContainerFactory.GetContainer().Resolve<ItemPage>();
        itemPage.Navigate(itemUrl);
    }
        
    [When(@"I click the 'buy now' button")]
    public void ClickBuyNowButtonItemPage()
    {
        var itemPage = UnityContainerFactory.GetContainer().Resolve<ItemPage>();
        itemPage.ClickBuyNowButton();
    }

    [When(@"then I click the 'proceed to checkout' button")]
    public void ClickProceedToCheckoutButtonPreviewShoppingCartPage()
    {
        var previewShoppingCartPage = 
        UnityContainerFactory.GetContainer().Resolve<PreviewShoppingCartPage>();
        previewShoppingCartPage.ClickProceedToCheckoutButton();
    }
        
    [When(@"the login page loads")]
    public void SignInPageLoads()
    {
        var signInPage = UnityContainerFactory.GetContainer().Resolve<SignInPage>();
        signInPage.WaitForPageToLoad();
    }
        
    [When(@"I login with email = ""([^""]*)"" and pass = ""([^""]*)""")]
    public void LoginWithEmailAndPass(string email, string password)
    {
        var signInPage = UnityContainerFactory.GetContainer().Resolve<SignInPage>();
        signInPage.Login(email, password);
    }

    [When(@"the shipping address page loads")]
    public void ShippingPageLoads()
    {
        var shippingAddressPage = 
        UnityContainerFactory.GetContainer().Resolve<ShippingAddressPage>();
        shippingAddressPage.WaitForPageToLoad();
    }
        
    [When(@"I type full name = ""([^""]*)"", country = ""([^""]*)"", 
    Adress = ""([^""]*)"", 
    city = ""([^""]*)"", 
    state = ""([^""]*)"", 
    zip = ""([^""]*)"" and 
    phone = ""([^""]*)""")]
    public void FillShippingInfo(string fullName, string country, 
    string address, string state, string city, string zip, string phone)
    {
        var shippingAddressPage = 
        UnityContainerFactory.GetContainer().Resolve<ShippingAddressPage>();
        var clientPurchaseInfo = new ClientPurchaseInfo(
            new ClientAddressInfo()
            {
                FullName = fullName,
                Country = country,
                Address1 = address,
                State = state,
                City = city,
                Zip = zip,
                Phone = phone
            });
        shippingAddressPage.FillShippingInfo(clientPurchaseInfo);
    }
        
    [When(@"I choose to fill different billing, full name = 
    ""([^""]*)"", country = ""([^""]*)"", Adress = ""([^""]*)"", 
    city = ""([^""]*)"", state = ""([^""]*)"", zip = ""([^""]*)"" and 
    phone = ""([^""]*)""")]
    public void FillDifferentBillingInfo(string fullName, 
    string country, string address, string state, string city, string zip, string phone)
    {
        var shippingAddressPage = 
        UnityContainerFactory.GetContainer().Resolve<ShippingAddressPage>();
        var shippingPaymentPage = 
        UnityContainerFactory.GetContainer().Resolve<ShippingPaymentPage>();
        var clientPurchaseInfo = new ClientPurchaseInfo(
            new ClientAddressInfo()
            {
                FullName = fullName,
                Country = country,
                Address1 = address,
                State = state,
                City = city,
                Zip = zip,
                Phone = phone
            });
        shippingAddressPage.ClickDifferentBillingCheckBox(clientPurchaseInfo);
        shippingAddressPage.ClickContinueButton();
        shippingPaymentPage.ClickBottomContinueButton();
        shippingAddressPage.FillBillingInfo(clientPurchaseInfo);
    }
        
    [When(@"click shipping address page 'continue' button")]
    public void ClickContinueButtonShippingAddressPage()
    {
        var shippingAddressPage = 
        UnityContainerFactory.GetContainer().Resolve<ShippingAddressPage>();
        shippingAddressPage.ClickContinueButton();
    }
        
    [When(@"click shipping payment top 'continue' button")]
    public void WhenClickTopPaymentButton()
    {
        var shippingPaymentPage = PerfectSystemTestsDesign.Base.UnityContainerFactory.
        GetContainer().Resolve<ShippingPaymentPage>();
        shippingPaymentPage.ClickTopContinueButton();
    }
        
    [Then(@"assert that order total price = ""([^""]*)""")]
    public void AssertOrderTotalPrice(string itemPrice)
    {
        var placeOrderPage = PerfectSystemTestsDesign.Base.
        UnityContainerFactory.GetContainer().Resolve<PlaceOrderPage>();
        double totalPrice = double.Parse(itemPrice);
        placeOrderPage.AssertOrderTotalPrice(totalPrice);
    }
}

Hook Methods

There are so called hooks classes where a different pre/post execution logic can be defined by a run, feature, step block or step level. For example, here we start a browser before each test and register all needed pages as singletons for the test. After that, we close the browser.

C#
[Binding]
public sealed class SpecflowHooks
{
    [BeforeTestRun]
    public static void BeforeTestRun()
    {
        Driver.StartBrowser(BrowserTypes.Chrome);
        UnityContainerFactory.GetContainer().RegisterType
        <ItemPage>(new ContainerControlledLifetimeManager());
        UnityContainerFactory.GetContainer().RegisterType
        <PreviewShoppingCartPage>(new ContainerControlledLifetimeManager());
        UnityContainerFactory.GetContainer().RegisterType
        <SignInPage>(new ContainerControlledLifetimeManager());
        UnityContainerFactory.GetContainer().RegisterType
        <ShippingAddressPage>(new ContainerControlledLifetimeManager());
        UnityContainerFactory.GetContainer().RegisterType
        <ShippingPaymentPage>(new ContainerControlledLifetimeManager());
        UnityContainerFactory.GetContainer().RegisterType
        <PlaceOrderPage>(new ContainerControlledLifetimeManager());
        UnityContainerFactory.GetContainer().RegisterType
        <ItemPageBuyBehaviour>(new ContainerControlledLifetimeManager());
        UnityContainerFactory.GetContainer().RegisterType
        <ItemPageNavigationBehaviour>(new ContainerControlledLifetimeManager());
        UnityContainerFactory.GetContainer().RegisterType
        <PlaceOrderPageAssertFinalAmountsBehaviour>(new ContainerControlledLifetimeManager());
        UnityContainerFactory.GetContainer().RegisterType
        <PreviewShoppingCartPageProceedBehaviour>(new ContainerControlledLifetimeManager());
        UnityContainerFactory.GetContainer().RegisterType
        <ShippingAddressPageContinueBehaviour>(new ContainerControlledLifetimeManager());
        UnityContainerFactory.GetContainer().RegisterType
        <ShippingAddressPageFillDifferentBillingBehaviour>(new ContainerControlledLifetimeManager());
        UnityContainerFactory.GetContainer().RegisterType
        <ShippingAddressPageFillShippingBehaviour>(new ContainerControlledLifetimeManager());
        UnityContainerFactory.GetContainer().RegisterType
        <ShippingPaymentPageContinueBehaviour>(new ContainerControlledLifetimeManager());
        UnityContainerFactory.GetContainer().RegisterType
        <SignInPageLoginBehaviour>(new ContainerControlledLifetimeManager());
        UnityContainerFactory.GetContainer().RegisterType
        <ShoppingCart>(new ContainerControlledLifetimeManager());
        UnityContainerFactory.GetContainer().RegisterInstance
        <IWebDriver>(PerfectSystemTestsDesign.Core.Driver.Browser);
    }

    [AfterTestRun]
    public static void AfterTestRun()
    {
        Driver.StopBrowser();
    }

    [BeforeScenario]
    public static void BeforeScenario()
    {
    }

    [AfterScenario]
    public static void AfterScenario()
    {
    }
}

Data Driven Tests Examples Table

Specflow supports data driven tests through data tables - a new test is generated for every row in the table. Keep in mind that I formatted the table manually. The Visual Studio support for Gherkin is not on the needed level.

Scenario Outline: Successfully Convert Seconds to Minutes Table
	When I navigate to Seconds to Minutes Page
	And type seconds for <seconds>
	Then assert that <minutes> minutes are displayed as answer
Examples:
| seconds						| minutes   | 
| 1 day, 1 hour, 1 second       | 1500		| 
| 5 days, 3 minutes 			| 7203		| 
| 4 hours						| 240		| 
| 180 seconds     				| 3			| 

Pass Multiple Parameters to Step

This table will pass something like a list of dynamic object to our binding method. By the way, it is really implemented using dynamic C# objects.

Scenario: Add Amazon Products with Affiliate Codes
	When add products
	| Url                                      | AffilicateCode |
	| /dp/B00TSUGXKE/ref=ods_gw_d_h1_tab_fd_c3 | affiliate3     |
	| /dp/B00KC6I06S/ref=fs_ods_fs_tab_al      | affiliate4     |
	| /dp/B0189XYY0Q/ref=fs_ods_fs_tab_ts      | affiliate5     |
	| /dp/B018Y22C2Y/ref=fs_ods_fs_tab_fk      | affiliate6     |

Here is how we use the SpecFlow's parameters table in tests. As I pointed, it creates a dynamic list of objects and we can iterate through them.

C#
[When(@"add products")]
public void NavigateToItemUrl(Table productsTable)
{
    var itemPage = UnityContainerFactory.GetContainer().Resolve<ItemPage>();
    IEnumerable<dynamic> products = productsTable.CreateDynamicSet();
    foreach (var product in products)
    {
        itemPage.Navigate(string.Concat(product.Url, "?", product.AffilicateCode));
        itemPage.ClickBuyNowButton();
    }
}

Evaluate SpecFlow Based Tests- Assessment System

Maintainability

Everything mentioned for the behaviours is applicable for this approach too. However, the rating is decreased because additional binding files exist. Moreover, the user cannot use existing tags and constants in the feature files which leads to hard-coded data and copy-paste development.

 Facade Based Tests
Maintainability3
Readability 
Code Complexity Index 
Usability 
Flexibility 
Learning Curve 
Least Knowledge 

Readability

The main advantage of SpecFlow is the readability. All steps are described in human-readable syntax via the Gherkin DSL. Even the base initialize methods are described with a couple of sentences which makes them more meaningful to the user.

 Facade Based Tests
Maintainability3
Readability5
Code Complexity Index 
Usability 
Flexibility 
Learning Curve 
Least Knowledge 

Code Complexity Index

The code complexity index here is not entirely accurate because Visual Studio doesn't support the calculation of code metrics for Gherkin files. The index for the binding classes is marked as very good. Probably because there are not any base classes. However, they depend on multiple classes such as pages and behaviours or facades. I think you will agree with me that if we could calculate the metrics for the Gherkin files, they weren't going to be very good because of that, I decreased the overall rating with one and it is only marked as good.

 Facade Based Tests
Maintainability3
Readability5
Code Complexity Index3
Usability 
Flexibility 
Learning Curve 
Least Knowledge 

Usability

The rating for this parameter is calculated as Very Poor because there are a lot of new classes that should be created before the user can use any steps in his test (assuming that he/she is writing a new test from scratch for a new feature). The SpecFlow's integration with Visual Studio is poor and the suggested steps while writing are not very helpful (IntelliSense). It is a challenge if you need to define a couple of actions with common starting words (you should define different overridden methods in the bindings' classes + use custom regex patterns). If you use examples' table to generate tests, you should format it manually if you want to be readable.

 Facade Based Tests
Maintainability3
Readability5
Code Complexity Index3
Usability1
Flexibility 
Learning Curve 
Least Knowledge 

Flexibility

The rating is only good because in order for the SpecFlow's API to support additional steps, you need to create wrapper methods in the bindings' classes with custom regex expressions.

 Facade Based Tests
Maintainability3
Readability5
Code Complexity Index3
Usability1
Flexibility4
Learning Curve 
Least Knowledge 

Learning Curve

I guess it will be harder to write new tests compared to the approach of using only page objects, especially if there aren't existing tests using the SpecFlow's sentences steps.

 Facade Based Tests
Maintainability3
Readability5
Code Complexity Index3
Usability1
Flexibility4
Learning Curve3
Least Knowledge 

Principle of Least Knowledge

You pass only the required parameters to the concrete binding methods. So the rating is marked as excellent.

 Facade Based Tests
Maintainability3
Readability5
Code Complexity Index3
Usability1
Flexibility4
Learning Curve3
Least Knowledge5

You can watch my conference talk dedicated to the system or download the whole slide deck.

Design & Architecture

The post Assessment System for Tests - Architecture Design- SpecFlow Based Tests appeared first on Automate The Planet.

All images are purchased from DepositPhotos.com and cannot be downloaded and used for free.

License Agreement

License

This article, along with any associated source code and files, is licensed under The Microsoft Public License (Ms-PL)


Written By
CEO Automate The Planet
Bulgaria Bulgaria
CTO and Co-founder of Automate The Planet Ltd, inventor of BELLATRIX Test Automation Framework, author of "Design Patterns for High-Quality Automated Tests: High-Quality Test Attributes and Best Practices" in C# and Java. Nowadays, he leads a team of passionate engineers helping companies succeed with their test automation. Additionally, he consults companies and leads automated testing trainings, writes books, and gives conference talks. You can find him on LinkedIn every day.

Comments and Discussions

 
-- There are no messages in this forum --