Portable User Driven Tests

In this post I intend to take one basic Acceptance Criteria and create an automated user driven test which will run as a docker process. I will use a farcical Acceptance Criteria to cover the following:

For this blog post I will be using the following:

  • Dependency Management / Task Runner: Gradle
  • Language: Groovy
  • Specification Framework: Spock
  • Browser Automation: Geb

You can clone the project here: user-driven-test-example

Create a failing test

For the purpose of this post I will use the following Acceptance Criteria:

Given: I navigate to the desktop Ebay home page.

When: I am not logged in, and I enter the string 'iPhone' into the search bar, and press the 'Search' button.

Then: I am directed to a page of listings matching my keyword search with 50 listings in that page.

Given, When, Then. The three words that make up the backbone of any good BDD acceptance criteria. Acceptance Criteria should, where possible, be verified via automated tests. These tests ensure that we have not broken or changed existing functional behaviour, and should meet the following requirements:

  1. Be human readable
    • The living, evolving feature documentation
  2. Verify Acceptance Criteria
  3. Be deterministic
  4. Be reliable (i.e. not flaky)

From the user journey defined in our Acceptance Criteria we can start to build an automated BDD, user driven test which will meet our four test requirements: readable, verify AC, deterministic, reliable.

Its that last requirement that I have found most difficulty meeting, and in my experience, the major causes of unreliable user driven tests is the varying environments in which they run. I will demonstrate how we can leverage the capabilities of Docker containers to achieve this reliability.


Running user driven tests in varying environments is a good thing, it can give confidence that the software under test is consistent in different Browsers / Operating Systems / Devices, however as part of a smooth continuous integration pipeline with a tight feedback loop reliability is king.


TDD with BDD

TDD and BDD compliment each other perfectly: write a test which satisfies a behavioural requirement and iterate over the solution until it passes. So lets do just that:

import geb.spock.GebSpec


class SearchFilterAndSortSpec extends GebSpec {

    def "Verify that searching for a keyword yields a results page with 50 listings"() {

        given: "As a user I navigate to the desktop Ebay home page."
        to EbayHomePage

        when: "I am not logged in, and I enter the string _'iPhone'_ into the search bar, and press the 'Search' button"
        searchFor "iPhone"

        then: "I am directed to a page of listings"
        at SearchResultsPage

        and: "The listings match my keyword search with 50 listings in that page."
        numberOfListings() == 50
    }

Make the test pass

Lets work through the test step by step and add functionality at each point.

given

given: "As a user I navigate to the desktop Ebay home page."  
        to EbayHomePage

This first line of test code is the the cause of our failing test. to EbayHomePage. For this line of code to work we need to implement a PageObject called "EbayHomePage".

But, before we start lets define the base URL of the site we are testing:

src/test/resources/GebConfig.groovy

baseUrl = "https://ebay.co.uk"  

Now all page objects we create will prefix their static url with this baseUrl.

Back to our first Page Object, lets start simple:

src/test/groovy/com/user/drive/test/examples/pages/EbayHomePage.groovy

package com.user.drive.test.examples.pages

import geb.Page

class EbayHomePage extends Page {  
    static url = "/"
}

This Page object does nothing but define an address of https://ebay.co.uk/. This is enough for the precondition of our test to work. However it is good practice to add a boolean which can be used to confirm the page is the page expected. We can verify that the page title is correct:

package com.user.drive.test.examples.pages

import geb.Page

class EbayHomePage extends Page {

    static url = "/"

    static at = { title.contains("Electronics, Cars, Fashion, Collectibles, Coupons and More | eBay") }
}

Now the to method only returns once the at method returns true.

When

when: "I am not logged in, and I enter the string _'iPhone'_ into the search bar, and press the 'Search' button"  
        searchFor "iPhone"

searchFor This is the first method we are going to have to implement. The functionality of this method is pretty obvious:

  • Take an input string
  • Enter the string into the search bar
  • Click search

Now where to implement this method?... On the Page Object obviously!

Geb has a very nice feature, after an at method has returned true, that page object is included in the test context. Meaning we can directly call methods / content defined in the Page Object from the test.

While we are here we need to define the content that we will be interacting with:

src/test/groovy/com/user/drive/test/examples/pages/EbayHomePage.groovy

package com.user.drive.test.examples.pages

import com.user.drive.test.examples.pages.modules.SearchModule  
import geb.Page

class EbayHomePage extends Page {

    static url = "/"

    static at = { title.contains("Electronics, Cars, Fashion, Collectibles, Coupons and More | eBay") }

    static content = {
        search { module SearchModule }
    }
}

Lets take a moment to explain the use of Geb Modules. Modules are key for interacting with repeating content. This might be content that is found across multiple pages such as the Search Bar we are interacting with here, or it might be repeating content on the same page such as items in a list (we will see this later). For now we can define the search Module:

src/test/groovy/com/user/drive/test/examples/pages/modules/SearchModule.groovy

package com.user.drive.test.examples.pages.modules

import geb.Module

class SearchModule extends Module {

    static content = {
        searchTextBox     { $("input", id:"gh-ac") }
        searchButton      { $("input", id:"gh-btn") }
        catagoryContainer { $("select", id:"gh-cat") }
    }

    void searchFor(String input) {
        searchTextBox << input
        searchButton.click()
    }

    void selectCatagory(Integer catagory) {
        catagoryContainer.click()
        catagoryContainer.find("option").find { it.value() == "${catagory}" }.click()
    }

    void searchFor(String input, Integer catagory) {
        selectCatagory(catagory)
        searchFor(input)
    }
}

So there is a fair amount of complexity here. Let's break down what we have built:

We have defined a page EbayHomePage and that page has content which we can interact with. The only content we have defined is search which itself is a set of intractable content defined in a module SearchModule. Lets take a look at the actual eBay home page in a browser, and highlight some of the elements defined:


Because we defined the search content as a module we can reuse that module across multiple pages.

Now that we have the content we can interact with it. All content defined in the content closure is a Navigator object and has a number of methods provided by the Geb framework. From our test we could directly interact with the elements but a much nicer pattern is to abstract actions into methods. E.G.

We can abstract searching for an item into a single verb searchFor. By implementing a method on the Page Object which takes a String and interacts with the necessary content to search for the string provided we can create very readable tests:

// Top Level Test
when: "I am not logged in, and I enter the string _'iPhone'_ into the search bar, and press the 'Search' button"  
        searchFor "iPhone"

// Page Object
void searchFor(String input) {  
        searchTextBox << input
        searchButton.click()
}

Now that we have covered Page Objects, content, modules and test verbs we can continue to implement the remaining code required to make our test pass.

Then

I wont walk through the rest of whats required but instead check our original test and list what else needs to be created (all of which can be found in the user-driven-test-example project.

From our test we have implement both the precondition, given, and the action when and all that is left is to add the verification, then:

then: "I am directed to a page of listings"  
        at SearchResultsPage

        and: "The listings match my keyword search with 50 listings in that page."
        numberOfListings() == 50

So we need to create a Page Object called SearchResultsPage which has a method numberOfListings which returns an Integer.

Finally, lets see it run:

Dockerise the Test

I do not intend to explain the basics of Docker here, instead I encourage you to work through the getting started with Docker documentation.

Now lets build our Dockerfile which will define the environment our tests will run in.

First of all we need a base image to work from. thankfully the good people over at Selenium have done the hard work for us. They supply Ubuntu images with Chrome or Firefox installed and a JDK/JVM for us to build and run tests. More info and sources can be found here: docker-selenium

Dockerfile

FROM selenium/standalone-chrome

ENV WORKING_DIR=/usr/local/test-runner \  
    GRADLE_CMD=""

COPY resources/docker-entrypoint.sh docker-entrypoint.sh  
COPY gradle ${WORKING_DIR}/gradle  
COPY src ${WORKING_DIR}/src  
COPY build.gradle ${WORKING_DIR}  
COPY gradlew ${WORKING_DIR}

RUN sudo apt-get clean \  
    && sudo apt-key update \
    && sudo apt-get -y update \
    && sudo apt-get -y install xvfb \
    && sudo chown seluser:seluser -R ${WORKING_DIR}

ENTRYPOINT ["/docker-entrypoint.sh"]  
CMD ./gradlew ${GRADLE_CMD}  

Lets work through this:

  • FROM selenium/standalone-chrome: The base image we will build from
  • ENV WORKING_DIR=/usr/local/test-runner: Setting a working dir variable to save copy and pasting a path
  • GRADLE_CMD="": An empty string, we will use this to pass in gradle commands
  • COPY ... All of the COPY commands copy assets from our local machine to the Docker image
  • RUN ...
    • sudo apt-get -y install xvfb: Install a headless display which we attach Chrome to for running headless.
    • sudo chown seluser:seluser -R ${WORKING_DIR}:Set ownership of the working directory to the user which runs the tests
  • ENTRYPOINT ["/docker-entrypoint.sh"]: execute a script before running anything
  • CMD ./gradlew ${GRADLE_CMD}: Run a gradle command

We are missing one small piece and that is the docker-entrypoint.sh. This file allows us to get ready to execute the command defined in the CMD statement.

We don't need to do much to prepare for running our tests apart from starting a headless display for Chrome to attach to and moving to the working directory

resources/docker-entrypoint.sh

#!/bin/sh

# Start the virtual display for chrome to attach to
sudo Xvfb :10 -ac -screen 0 1024x768x8 &  
export DISPLAY=:10

cd ${WORKING_DIR}  
exec "$@"  

Now that we have defined our environment we need a way to build the image. This is done using docker build. For simplicity I have created build.sh which serves two purposes: I don't need to remember the docker build syntax and I can execute it from a gradle task.

build.sh

#!/usr/bin/env bash

set -e

IMAGE=${1:-"user-driven-test-example"}  
VERSION=${2:-"latest"}  
REPO=${3:-"local"}

docker build -t ${REPO}/${IMAGE}:${VERSION} "$(dirname $0)"

docker rmi -f $(docker images -f dangling=true -q) 2>/dev/null  

And the new gradle task:

task dockerImage(type:Exec) {  
    commandLine './build.sh'
}

Now we can simply run our new gradle task to generate the docker image. And running the image as a container is as simple as:

docker run -it --rm \  
    -e GRADLE_CMD="chromeTest" \
local/user-driven-test-example:latest  

And we are done!

This image can now be run in ANY Docker environment and we can guarantee the version of the browser and OS will always be the same.