Documentation

Table of Contents

1. Defining stories
2. The stories field
3. The @Feature annotation
4. The @Features annotation
5. Test steps
6. Creating tests steps
7. Aliases
8. Step composition
9. Named parameters
10. Regular expressions
11. Step matching
12. Group extraction
13. Projections
14. Test Composition
15. Composing tests
16. Depending on multiple tests
17. State sharing
18. Tags
19. Skipping features or scenarios
20. Ignoring features or scenarios
21. Grouping and Packs
22. Comments
23. REST reporting
24. Include reports plugin dependency
25. Configure properties file
26. IDE enabling

Defining stories

Gherkin is the language used to define user stories. Gherkin is a Business Readable, Domain Specific Language created especially for behavior descriptions. It’s perfectly usable for business analysts and product owners. At the same time it’s a perfect fit for developers. Tools like COLA Tests such as Cucumber, jBehave and Behat use Gherkin to aid in making sure that the application keeps behaving as designed.

In COLA Tests user stories can be set in a few different ways.

1. The stories field

By default, Gherkin features can be defined in your JUnit POJO in a field named "stories".

Java

...
    // Field must be final
    private final String stories =
        "Feature: Introduce addition\n"
            + "Scenario: Should add two numbers\n"
            + "Given 2\n"
            + "And 4\n"
            + "When added together\n"
            + "Then the result will be 6";
...

See Working Example.

2. The @Feature annotation

The @Feature annotation can be used to annotate Gherkin feature fields.

Java

...
    // Field must be final
    @Feature
    private final String featureOne = ...

    // Field must be final
    @Feature
    private final String featureTwo = ...
...

See Working Example.

3. The @Features annotation

The @Features annotation is used to decorate JUnit POJO classes with the name of the file or files where the Gherkin scenarios have been specified.

Java

@Features({ "feature-one", "feature-two" })
public class FeaturesTest {
...
}

Feature files can have an extension of .feature, .stories, .story or .gherkin and must be placed in the same package as the COLA Test JUnit class handling it.

See Working Example.

Test steps

In Gherkin language, test steps start with one of the following keywords: Given, When, Then, And or But.

1. Creating test steps

Test steps are mapped to Java methods by means of the @Given, @When and @Then annotations.

Here's a full scenario example containing all the possible keywords:

Scenario: Should verify something
Given a initial state
And another initial state
When something happens
Then verify result
And verify another result
But verify yet another result

And when converted to a COLA Test could look something like this:

Java

...
    @Given("a initial state")
    public void givenAInitialState() {
        ...
    }

    @Given("another initial state")
    public void givenAnotherInitialState() {
        ...
    }

    @When("something happens")
    public void whenSomethingHappens() {
        ...
    }

    @Then("verify result")
    public void thenVerifyResult() {
        ...
    }

    @Then("verify another result")
    public void thenVerifyAnotherResult() {
        ...
    }

    @Then("verify yet another result")
    public void thenVerifyYetAnotherResult() {
        ...
    }
...

Notice that all And are converted to their previous step keyword when converted to a method. If placed after a Given step, it will use the @Given annotation. If placed after a Then step, it will use the @Then annotation. All But steps have to use the @Then keyword.

2. Aliases

Test steps can have multiple alias in order to facilitate code sharing and linguistic integration for different scenarios.

@Given({
    "the item price is <price>",
    "the item price becomes <price>"
})
public void givenAnItemPrice(@Assigned("price") final Double itemPrice) {
...
}

3. Step composition

Step composition allows for implicit steps to be called before and/or after the step to be executed. The implicit steps to be invoked can be defined using the @PreSteps and @PostSteps annotations.

...

    @Given("a user is logged in")
    public void loginUser() {
        ...
    }

    @Given("the user as a item in the cart")
    public void addItemToCart() {
        ...
    }

    @PreSteps({ "Given a user is logged in", "Given the user as a item in the cart" })
    @When("a user checks out")
    public void checkoutUser() {
        ...
    }

...

Named parameters

Named parameters can be captured from the scenario steps and injected into the handling method using the @Assigned annotation. This is particularly useful because the same handling step method is shared to handle different scenarios steps.

In the following scenario the methods can be injected with the respective numbers used in the scenario.

Scenario: Should calculate leftover cucumbers
Given there are 50 cucumbers
When I eat 20 cucumbers
Then I should have 30 cucumbers

And the COLA Test step handlers methods could look something like:

Java

...
    @Given("there are <start> cucumbers")
    public void given(@Assigned("start") final String start) {
        ...
    }

    @When("I eat <eaten> cucumbers")
    public void when(@Assigned("eaten") final Integer eaten) {
        ...
    }

    @Then("I should have <left> cucumbers")
    public void then(@Assigned("left") final Long left) {
        ...
    }
...

See Working Example.

Regular expressions

1. Step matching

COLA Tests step annotations can include regular expressions in order to match against one or more scenario steps.

For example, the following steps:

...
Given a beer
And a pop
...

Could both be handled by the same method by means of a matching regular expression:

Java

...
    @Given("a (beer|pop)")
    public void givenADrink() {
        ...
    }
...

See Working Example.

2. Group extraction

Regular expressions groups can also be extracted from the scenario steps and injected in the handling method by means of the @Group annotation.

In the following scenario step:

...
Given 50 beers per 5 developers
...

The values could be extracted and injected in the handling step method:

Java

...
    @Given("(\\d+) beers per (\\d+) developers")
    public void given(@Group(1) final Integer beers, @Group(2) final Integer developers) {
        ...
    }
...

See Working Example.

Projections

Projections are COLA Tests way of handling Gherkin Scenario Outlines examples. Example data can be injected in COLA Test annotated step methods by means of the @Projection annotation.

For example, the following scenario outline:

Scenario Outline: Should parse examples
Given there are <start> cucumbers
When I eat <eat> cucumbers
Then I should have <left> cucumbers
        
Examples:
 | start | eat | left |
 | 12    | 5   | 7    |
 | 20    | 5   | 15   |

Could be handled by the following COLA Test step handling methods:

Java

...
    @Given("there are <start> cucumbers")
    public void given(@Projection("start") final String start) {
        ...
    }

    @When("I eat <eat> cucumbers")
    public void when(@Projection("eat") final Integer eat) {
        ...
    }

    @Then("I should have <left> cucumbers")
    public void then(@Projection("left") final Long left) {
        ...
    }
...

See Working Example.

Test Composition

What better way to guarantee application state than a test? After all, a test is nothing more than a simple state validation. In this spirit, COLA Tests Test Composition is designed to facilitate that test classes become building blocks to other test classes.

COLA Tests can be composed of any number of COLA Tests and/or plain simple JUnit tests. In addition, state can be shared between COLA Tests.

1. Composing tests

A test can depend on another test through the @DependsOn annotation.

• COLA Test classes annotated with @DependsOn will execute the configured class(es) once per Scenario.
• COLA Test steps annotated with @DependsOn will be executed right before the annotated step.
• The @DependsOn annotation will execute the configured COLA or JUnit test in the configured order, left first.
• The @DependsOn annotation will only execute the selected methods, or all methods if none set.

Java

@Features("contacts.feature")
@DependsOn(DBInitializeConfigTest.class)
public class ContactsRestTest extends BaseColaTest {

    @Given("a new contact")
    @DependsOn(value="DBCreateContactTest.class", methods="createNewContact")
    public void givenANewContact() {
        ...
    }
...

2. Depending on multiple tests

Multiple test dependencies are defined using the @Dependencies annotation. Each dependency will be executed in the provided order.

    @Dependencies({
      @DependsOn(ExamplesColaTest.class),
      @DependsOn(RegexColaTest.class)
    })

3. State sharing

State sharing helps sharing state between different tests. For example, say you have a tests that performs a login into a web site using Selenium driver. This test can be reused on the logout tests since it ensures that a user is logged in before the logout tests is executed. The problem here is that in order for this arrangement to be possible, it is required to share the Selenium driver instance between tests.

COLA Tests shares state through dependency injection. To share an attribute, mark it with the @ColaInjectable annotation.

@Features("LogoutTest")
public class LogoutTest extends BaseColaTest {

    @ColaInjectable
    public WebDriver outDriver;

    ...
    
    @Given("a user is logged in")
    @DependsOn(LoginTest.class)
    public void givenAUserIsLoggedIn() {
        welcomePage = new WelcomePage(getDriver());
        welcomePage.isVisible();
    }
    
    ...
}

See Working Example.

The above example marks the WebDriver attribute for sharing. The depending tests specified through @DependsOn can be injected with this WebDriver if the class is marked as a @ColaInjected class and the target attribute is marked as @Inject.

@Features("LoginTest")
@ColaInjected
public class LoginTest extends BaseColaTest {

    @Inject
    public WebDriver inDriver;

    ...
}

See Working Example.

Sharing will be done by type, meaning that when injecting the target attribute will be set with the first instance found that matches the receiving type. To share different attributes of the same type, @ColaInjectable can be named.

    ...
    @ColaInjectable("beer_count")
    public Integer beers;

    @ColaInjectable("spirits_count")
    public Integer spirits;
    ...    

And to inject them, simply use the @Named annotation.

    ...
    @Inject @Named("beer_count")
    private String numberOfBeers;

    @Inject @Named("spirits_count")
    private String numberOfSpirits;
    ...

Simple Test Composition Example

State Sharing Example

Tags

Tags can be added to either Gherkin Features or Scenarios. Currently only the tags @skip and @ignore are processed and handled by COLA Tests. Users can also add tags for easy file grepping or for future execution filter/selection.

1. Skipping features or scenarios

Skipped features or scenarios will not be injected in the JUnit POJO and therefore no output or notification will be available in the logs.

The following Gherkin feature will be skipped by COLA Tests:

@skip
Feature: Skip Feature

Scenario: Should skip feature
Given A
When B
Then not executed

And in the following Gherkin feature, only the first scenario will be executed:

Feature: Skip Feature without scenarios

Scenario: Should execute
Given A
When B
Then it was executed

@skip
Scenario: Should be skipped
Given A
When B
Then not executed

See Working Example.

2. Ignoring features or scenarios

Ignored features or scenarios will have test methods injected only so that a ignore notification can be logged for user information.

The following Gherkin feature will be ignored by COLA Tests:

@ignore
Feature: Ignore Feature
Scenario: Should ignore feature
Given A
When B
Then a ignored notification will be logged

And in the following Gherkin feature, the first scenario will be executed and the second will log a test ignored notification:

Feature: Skip scenario

Scenario: Should execute
Given A
When B
Then it was executed

@ignore
Scenario: Should be ignored
Given A
When B
Then a notification will be logged

See Working Example.

3. Grouping and Packs

User defined tags can be used in multiple features and/or scenarios to create groups or, as Gojko Adzic puts it, packs. These packs can then be singled out for execution or exclusion. This is done through the following JVM properties:

• Singled-out for execution: -Dcola.group=pack1,pack2,...,packN
• Excluded from execution: -D~cola.group=pack1,pack2,...,packN

@my-feature
Feature: Execute feature

@my-pack2
Scenario: Should execute my-pack2
...

@my-other-pack
Scenario: Should execute my-other-pack
...

For example:

• To single out my-pack2 for execution use -Dcola.group=pack2. To exclude it use -D~cola.group=my-pack2

Comments

Comments are any lines starting with # followed by some text.

# This is a feature comment
Feature: Allow comments

# This is a scenario comment
Scenario: Should have comment
Given ...

REST reporting

In some cases you might want to report the result of your test straight to an issue/feature tracker such as GitHub, GitLab, JetBrains You Track, Atlassian JIRA or any other issue/feature tracker with an available REST API. Another option might be to simply log the result to a REST based logger such as GrayLog2 or Splunk.

COLA Tests provide the cola-tests-reports plugin dependency for this purpose.

1. Include reports plugin dependency

    <dependency>
        <groupId>com.github.bmsantos</groupId>
        <artifactId>cola-tests-reports</artifactId>
        <version>0.3.0</version>
        <scope>test</scope>
    </dependency>

2. Configure properties file

The following is an example setup for COLA Tests reports properties file:

Classpath file: /cola-test-report.properties

# The target URL Template
cola.tests.report.url=http://localhost:7919/rest/to/${type}

# HTTP Method: get/post/put
cola.tests.report.method=post

# State Codes
cola.tests.report.code.fail=Fail
cola.tests.report.code.pass=Pass

# Request Body Template
cola.tests.report.body.template={ "type":"${type}", "state":"${state}", "error":"${error}" }
cola.tests.report.body.content-type=application/json

COLA Tests Reports work by replacing template parameters in both report url and/or the body template. By default it provides two parameters:

1. state - The state of the executed test. If the test has failed, the state will be the value set in cola.tests.report.code.fail. If the state has passed, the state will be the value set in cola.tests.report.code.pass.

2. error - Error message and stack trace of the failure.

Custom parameters, such as type in the above example, can be passed as arguments to the template as shown in the next section.

3. Introduce it in your Gherkin stories

COLA Test Reports can then be added to Feature and/or Scenarios. Reports declared at the Feature level will be applied to all Scenarios. Reports declared at the Scenario level will only be applied to the Scenario itself.

The following example shows a Feature and its Scenario annotated with COLA Test REST Report plugin:

#report> type:feature
Feature: Ignore Feature

#report> type:scenario
Scenario: Should ignore feature
Given A
When B
Then not executed

The notation follows the following convention:

#[Report Plugin Name]> key1:value1 ... keyN:valueN

And from the above Feature example we have that the default plugin is named report and passes the custom key type to the report plugin so that it can be used in the URL and/or body template.

Users are encouraged to create their own report plugins.

See Working Example.

IDE enabling

IDE JUnit 4 tests runners look into the JUnit POJO in order to find out if there's an annotated @Test method. If there's one, the IDE will allow the class to be executed. Because COLA Tests are injected, the IDE never detects that the POJO is to be executed as a JUnit test.

To get around this limitation, COLA Tests provide the @IdeEnabler annotation that can be used to annotate any @Test method used to trigger the IDE JUnit runner. The annotated method used to trigger the IDE is removed from the class and never executed. It is recommended for this method to live in a base class common to all tests.

See Working Example.