Testing Ansible

Why test your Ansible contributions?

If you’re a developer, one of the most valuable things you can do is to look at GitHub issues and help fix bugs, since bug-fixing is almost always prioritized over feature development. Even for non-developers, helping to test pull requests for bug fixes and features is still immensely valuable.

Ansible users who understand how to write playbooks and roles should be able to test their work. GitHub pull requests will automatically run a variety of tests (for example, Azure Pipelines) that show bugs in action. However, contributors must also test their work outside of the automated GitHub checks and show evidence of these tests in the pull request to ensure that their work will be more likely to be reviewed and merged.

Read on to learn how Ansible is tested, how to test your contributions locally, and how to extend testing capabilities.

If you want to learn about testing collections, read Testing collections

Types of tests

At a high level, we have the following classifications of tests:

sanity:

  • Sanity Tests

  • Sanity tests are made up of scripts and tools used to perform static code analysis.

  • The primary purpose of these tests is to enforce Ansible coding standards and requirements.

integration:
units:

  • Unit Tests

  • Tests directly against individual parts of the code base.

Testing within GitHub & Azure Pipelines

Organization

When Pull Requests (PRs) are created they are tested using Azure Pipelines, a Continuous Integration (CI) tool. Results are shown at the end of every PR.

When Azure Pipelines detects an error and it can be linked back to a file that has been modified in the PR then the relevant lines will be added as a GitHub comment. For example:

The test `ansible-test sanity --test pep8` failed with the following errors:

lib/ansible/modules/network/foo/bar.py:509:17: E265 block comment should start with '# '

The test `ansible-test sanity --test validate-modules` failed with the following error:
lib/ansible/modules/network/foo/bar.py:0:0: E307 version_added should be 2.4. Currently 2.3

From the above example we can see that --test pep8 and --test validate-modules have identified an issue. The commands given allow you to run the same tests locally to ensure you’ve fixed all issues without having to push your changes to GitHub and wait for Azure Pipelines, for example:

If you haven’t already got Ansible available, use the local checkout by running:

source hacking/env-setup

Then run the tests detailed in the GitHub comment:

ansible-test sanity --test pep8
ansible-test sanity --test validate-modules

If there isn’t a GitHub comment stating what’s failed you can inspect the results by clicking on the “Details” button under the “checks have failed” message at the end of the PR.

Rerunning a failing CI job

Occasionally you may find your PR fails due to a reason unrelated to your change. This could happen for several reasons, including:

  • a temporary issue accessing an external resource, such as a yum or Git repo

  • a timeout creating a virtual machine to run the tests on

If either issue appears to be the case, you can rerun the Azure Pipelines test by:

  • adding a comment with /rebuild (full rebuild) or /rebuild_failed (rebuild only failed CI nodes) to the pull request

  • closing and re-opening the pull request (full rebuild)

  • making another change to the branch and pushing to GitHub

If the issue persists, please contact the community. Visit the Ansible communication guide for details.

How to test a PR

Ideally, the code should add tests that prove that the code works. That’s not always possible and tests are not always comprehensive, especially when a user doesn’t have access to a wide variety of platforms, or is using an API or web service. In these cases, live testing against real equipment can be more valuable than automation that runs against simulated interfaces. In any case, things should always be tested manually the first time as well.

Thankfully, helping to test Ansible is pretty straightforward, assuming you are familiar with how Ansible works.