JUnit 5 Dynamic Tests — Generate Tests at Runtime

In this article, I’d like to introduce about JUnit 5 Dynamic Tests feature which allows us to declare and run test cases generated at runtime.

1. Static Tests vs. Dynamic Tests

1.1. Static Tests

To get to know about the Dynamic Tests vs. Static Tests, let take a look at an example below. We have a very simple TranslatorEngine class which is responsible for translating text from English to French. Note that the class is implemented basically, without optimization, to make it easy to understand.

public class TranslatorEngine {

  public String tranlate(String text) {
    if (StringUtils.isBlank(text)) {
      throw new IllegalArgumentException("Translated text must not be empty.");
    }
    if ("Hello".equalsIgnoreCase(text)) {
      return "Bonjour";

    } else if ("Yes".equalsIgnoreCase(text)) {
      return "Oui";

    } else if ("No".equalsIgnoreCase(text)) {
      return "Non";

    } else if ("Goodbye".equalsIgnoreCase(text)) {
      return "Au revoir";

    } else if ("Good night".equalsIgnoreCase(text)) {
      return "Bonne nuit";

    } else if ("Thank you".equalsIgnoreCase(text)) {
      return "Merci";
    } else {
      return "Not found";
    }
  }

}

Now, let’s write some tests for this class. Basically, we can come up with several test cases as follows. Note that these test cases are still not enough for the translate method yet.

import static org.junit.jupiter.api.Assertions.assertEquals;

import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.platform.runner.JUnitPlatform;
import org.junit.runner.RunWith;

@RunWith(JUnitPlatform.class)
public class TranslationEngineTest {

  private TranslatorEngine translatorEngine;

  @BeforeEach
  public void setUp() {
    translatorEngine = new TranslatorEngine();
  }

  @Test
  public void testTranlsateHello() {
    assertEquals("Bonjour", translatorEngine.tranlate("Hello"));
  }

  @Test
  public void testTranlsateYes() {
    assertEquals("Oui", translatorEngine.tranlate("Yes"));
  }

  @Test
  public void testTranlsateNo() {
    assertEquals("Non", translatorEngine.tranlate("No"));
  }
}

These above tests are called Static Tests. They are are static in the sense that they are fully specified at compile time, and their behavior cannot be changed at run-time. Note that we have just written 3 test cases while the current translate method can support translate 6 words or phrases. We may need to add three more static tests for three remaining words or phrases and other cases like Null or empty word or phrases, not supported words or phrases, etc.

When we run the test, basically, it will look for all the tests which were defined by annotating the @Test annotation and run them.

If the translate method supports for more words, phrases, or sentences, say 1000, we may need to add 1000 more test cases tediously for this method.

1.2. Dynamic Tests

Contrary to the Static Tests, which allow us to define a static number of fixed test cases at compile time, Dynamic Tests allow us to define the test case dynamically at runtime.

Let’s see an example about Dynamic Tests:

public void translateDynamicTests() {

    List<String> inPhrases = new ArrayList<>(Arrays.asList("Hello", "Yes", "No"));
    List<String> outPhrases = new ArrayList<>(Arrays.asList("Bonjour", "Oui", "Non"));
    Collection<DynamicTest> dynamicTests = new ArrayList<>();

    for (int i = 0; i < inPhrases.size(); i++) {

      String phr = inPhrases.get(i);
      String outPhr = outPhrases.get(i);

      // create an test execution
      Executable exec = () -> assertEquals(outPhr, translatorEngine.tranlate(phr));

      // create a test display name
      String testName = " Test tranlate " + phr;
      // create dynamic test
      DynamicTest dTest = DynamicTest.dynamicTest(testName, exec);

      // add the dynamic test to collection
      dynamicTests.add(dTest);
    }
  }

We have tried to create a collection of test cases by iterating through the list of data. Those tests are dynamic in the sense that they are generated at runtime, and the number of test cases are dependant on the data: the number of the input words or phrases.

That was an simple example of dynamic tests. Let’s see in detail how we can fully create dynamic tests by using JUnit 5.

2. JUnit 5 Dynamic Tests

In JUnit 5, dynamic test cases are represented by DynamicTest class. Here are some essential points:

• Dynamic tests can be generated by a factory method annotated with @TestFactory, which is a new annotation of JUnit 5.

• @TestFactory method must return a Stream, Collection, Iterable, or Iterator of DynamicTest instances.

• @TestFactory methods must not be private or static, and may optionally declare parameters to be resolved by ParameterResolvers.

2.1. Dynamic Tests Examples

All the example source code can be found on Github. You need to get JUnit 5 be ready for your environment. You can do this by yourselves or refer to this guideline to get JUnit 5 setup with Eclipse, Maven, and Gradle.

We will modify the above test method to make it comply with the syntax of JUnit 5.

 @TestFactory
  public Collection<DynamicTest> translateDynamicTests() {

    List<String> inPhrases =
        new ArrayList<>(Arrays.asList("Hello", "Yes", "No", "Goodbye", "Good night", "Thank you"));
    List<String> outPhrases =
        new ArrayList<>(Arrays.asList("Bonjour", "Oui", "Non", "Au revoir", "Bonne nuit", "Merci"));

    Collection<DynamicTest> dynamicTests = new ArrayList<>();

    for (int i = 0; i < inPhrases.size(); i++) {

      String phr = inPhrases.get(i);
      String outPhr = outPhrases.get(i);

      // create an test execution
      Executable exec = () -> assertEquals(outPhr, translatorEngine.tranlate(phr));

      // create a test display name
      String testName = "Test tranlate " + phr;
      // create dynamic test
      DynamicTest dTest = DynamicTest.dynamicTest(testName, exec);

      // add the dynamic test to collection
      dynamicTests.add(dTest);
    }
    return dynamicTests;
  }

We have annotated the method with the @TestFactory annotation and changed the return type to Collection<DynamicTest>.

This example is very straightforward and easy to understand. Let’s tune it to be conformed with Java 8 style and return a Stream of DynamicTest instead of collection.

@TestFactory
  public Stream<DynamicTest> translateDynamicTestsFromStream() {

    List<String> inPhrases =
        new ArrayList<>(Arrays.asList("Hello", "Yes", "No", "Goodbye", "Good night", "Thank you"));
    List<String> outPhrases =
        new ArrayList<>(Arrays.asList("Bonjour", "Oui", "Non", "Au revoir", "Bonne nuit", "Merci"));

    return inPhrases.stream().map(phrs -> DynamicTest.dynamicTest("Test translate " + phrs, () -> {
      int idx = inPhrases.indexOf(phrs);
      assertEquals(outPhrases.get(idx), translatorEngine.tranlate(phrs));
    }));
  }

As mentioned above, the factory method must return a Stream, Collection, Iterable, or Iterator. We will try return an Iterable of DynamicTest instances.

TestFactory
  public Iterable<DynamicTest> translateDynamicTestsFromIterate() {

    List<String> inPhrases =
        new ArrayList<>(Arrays.asList("Hello", "Yes", "No", "Goodbye", "Good night", "Thank you"));
    List<String> outPhrases =
        new ArrayList<>(Arrays.asList("Bonjour", "Oui", "Non", "Au revoir", "Bonne nuit", "Merci"));

    return inPhrases.stream().map(phrs -> DynamicTest.dynamicTest("Test translate " + phrs, () -> {
      int idx = inPhrases.indexOf(phrs);
      assertEquals(outPhrases.get(idx), translatorEngine.tranlate(phrs));
    })).collect(Collectors.toList());
  }

And finally, we will try to return an Iterator of DynamicTest instances.

  @TestFactory
  public Iterator<DynamicTest> translateDynamicTestsFromIterator() {

    List<String> inPhrases =
        new ArrayList<>(Arrays.asList("Hello", "Yes", "No", "Goodbye", "Good night", "Thank you"));
    List<String> outPhrases =
        new ArrayList<>(Arrays.asList("Bonjour", "Oui", "Non", "Au revoir", "Bonne nuit", "Merci"));

    return inPhrases.stream().map(phrs -> DynamicTest.dynamicTest("Test translate " + phrs, () -> {
      int idx = inPhrases.indexOf(phrs);
      assertEquals(outPhrases.get(idx), translatorEngine.tranlate(phrs));
    })).iterator();
  }

2.2. Running Tests

To run the test on Eclipse, simply Right Click –> Run As –> JUnit Tests.

As for running with Maven and Gradle, you can refer to this post: JUnit 5 Basic Introduction.

Here is the output in Eclipse:

3. Summary

We have gotten to know about JUnit 5's Dynamic Test feature, which allow us to create test cases at runtime. In my opinion, dynamic testing is necessary to help reduce the effort in writing tests. However, Dynamic Tests is still an experimental feature in the current version of JUnit 5, 5.0.0-M2. This feature may be removed without prior notice. Besides, the execution lifecycle of a dynamic test is quite different with a standard @Test case. One essential point should be noted that there are not any lifecycle callbacks for dynamic tests. This means that @BeforeEach and @AfterEach methods and their corresponding extension callbacks are not executed for dynamic tests.

In future posts, I’d like to continue deep dive into other features of JUnit 5. Recently, I have some posts related to JUnit 5 tutorial. If you’re interesting in them, you can find them here.