ansible/test/integration
Brian Coca e29dc49a49 moved to exceptions for basic skip/fails
better handling of checkmode and async
fix test to follow new flow control
2017-04-25 14:48:56 -04:00
..
group_vars Fixes loginBanner plucking over eapi transport when loginBanner isn't set. (#23363) 2017-04-06 10:58:04 -06:00
host_vars
roles ec2_group: description property is immutable (#19790) 2017-03-26 09:33:29 -04:00
targets moved to exceptions for basic skip/fails 2017-04-25 14:48:56 -04:00
vars Split integration tests out from Makefile. (#17976) 2016-10-12 14:57:53 -07:00
amazon.yml Adding support for Amazon ECR (#19306) 2017-01-17 14:45:43 -05:00
asa.yaml Migrate Network Tests into ansible/ansible (#18233) 2016-10-28 19:50:29 +01:00
azure.yml
cleanup_azure.py
cleanup_ec2.py PEP 8 E111 & E114 cleanup. (#20838) 2017-01-30 15:01:47 -08:00
cleanup_gce.py PEP 8 E111 & E114 cleanup. (#20838) 2017-01-30 15:01:47 -08:00
cleanup_rax.py
cloudflare.yml Fix cosmetic problems in YAML source 2016-11-11 14:50:57 -08:00
cloudscale.yml new module cloudscale_server (#20175) 2017-01-17 23:15:20 +01:00
cloudstack.yml
cnos.yaml New module: cnos_vlan and various utility files (#21107) 2017-02-16 08:11:39 +00:00
consul.yml
consul_inventory.yml
consul_running.py
credentials.template Changes to enable make gce to run. Added sys import so libcloud error is displayed; renamed credentials keys in template file so they work properly with gce_credentials.py. (#16607) 2016-09-16 10:02:18 -04:00
dellos6.yaml Migrate Network Tests into ansible/ansible (#18233) 2016-10-28 19:50:29 +01:00
dellos9.yaml Migrate Network Tests into ansible/ansible (#18233) 2016-10-28 19:50:29 +01:00
dellos10.yaml Migrate Network Tests into ansible/ansible (#18233) 2016-10-28 19:50:29 +01:00
destructive.yml Fix locale_gen to compare native strings rather than mixing byte and text strings 2017-01-04 10:11:33 -08:00
eos.yaml Test eos_banner (#22201) 2017-03-10 15:20:09 +00:00
exoscale.yml exo_dns: new module utils and integration tests for exoscale DNS (#17230) 2016-08-30 21:48:28 +02:00
galaxy_playbook.yml
galaxy_playbook_git.yml
galaxy_roles.yml Fix cosmetic problems in YAML source 2016-11-11 14:50:57 -08:00
galaxy_rolesfile
gce.yml [GCE] gce_net integration test (#21502) 2017-02-16 16:39:25 -05:00
gce_credentials.py Integration Playbook for Google Cloud DNS module. (#17698) 2016-09-22 14:40:57 -04:00
integration_config.yml Fix cosmetic problems in YAML source 2016-11-11 14:50:57 -08:00
inventory Configurable fact path (#18147) 2017-01-12 10:49:04 -05:00
inventory.network veos-dut-01 is down, so use veos01 (#19299) 2016-12-13 14:40:52 +00:00
inventory.remote.template Add support for OS X CI on Shippable. (#17160) 2016-08-23 10:49:46 -07:00
inventory.winrm.template Add binary module tests to Windows CI. (#16914) 2016-08-02 10:42:07 -07:00
ios.yaml Add ios_banner module (#22687) 2017-03-16 16:35:44 +01:00
iosxr.yaml Test system (#22420) 2017-03-08 12:21:45 -07:00
jenkins.yml jenkins_job: add integration tests (#17499) 2016-09-13 16:03:58 +02:00
junos.yaml Migrate Network Tests into ansible/ansible (#18233) 2016-10-28 19:50:29 +01:00
Makefile Fix 'task name is not templated in retry callback' (add task_name property to TaskResult) (#21214) 2017-02-24 12:33:24 -05:00
net_command.yml Add files via upload (#21062) 2017-02-11 12:12:32 +00:00
network-all.yaml Ops (#18652) 2016-11-28 16:22:17 +00:00
non_destructive.yml known_hosts: support --diff (#20349) 2017-02-08 09:56:03 -05:00
nxos.yaml Add nxos_interface integration testing (#23030) 2017-03-29 11:01:38 +02:00
ops.yaml Ops (#18652) 2016-11-28 16:22:17 +00:00
rackspace.yml
README.md Use ansible-test in integration test README. 2017-01-16 19:01:59 -08:00
setup_gce.py PEP 8 E111 & E114 cleanup. (#20838) 2017-01-30 15:01:47 -08:00
target-prefixes.network ops not openswitch (#18256) 2016-10-30 19:27:01 +00:00
test_win_group1.yml Migrate Windows CI roles to test targets. (#18005) 2016-10-13 18:03:19 +01:00
test_win_group2.yml Revert "win_service_stat: Added module" (#22184) 2017-03-02 01:34:11 -08:00
test_win_group3.yml Added win_region module (#19147) 2017-02-20 11:50:27 +00:00
vyos.yaml Migrate Network Tests into ansible/ansible (#18233) 2016-10-28 19:50:29 +01:00

Integration tests

The ansible integration system.

Tests for playbooks, by playbooks.

Some tests may require credentials. Credentials may be specified with credentials.yml.

Some tests may require root.

Quick Start

It is highly recommended that you install and activate the argcomplete python package. It provides tab completion in bash for the ansible-test test runner.

To get started quickly using Docker containers for testing, see Tests in Docker containers.

Configuration

Making your own version of integration_config.yml can allow for setting some tunable parameters to help run the tests better in your environment. Some tests (e.g. cloud) will only run when access credentials are provided. For more information about supported credentials, refer to credentials.template.

Prerequisites

The tests will assume things like hg, svn, and git are installed and in path.

(Complete list pending)

Non-destructive Tests

These tests will modify files in subdirectories, but will not do things that install or remove packages or things outside of those test subdirectories. They will also not reconfigure or bounce system services.

Run as follows for all POSIX platform tests executed by our CI system:

test/runner/ansible-test integration -v posix/ci/

You can select specific tests as well, such as for individual modules:

test/runner/ansible-test integration -v ping

Destructive Tests

These tests are allowed to install and remove some trivial packages. You will likely want to devote these to a virtual environment. They won't reformat your filesystem, however :)

test/runner/ansible-test integration -v destructive/

Cloud Tests

Cloud tests exercise capabilities of cloud modules (e.g. ec2_key). These are not 'tests run in the cloud' so much as tests that leverage the cloud modules and are organized by cloud provider.

In order to run cloud tests, you must provide access credentials in a file named credentials.yml. A sample credentials file named credentials.template is available for syntax help.

Provide cloud credentials:

cp credentials.template credentials.yml
${EDITOR:-vi} credentials.yml

Run the tests: make cloud

WARNING running cloud integration tests will create and destroy cloud resources. Running these tests may result in additional fees associated with your cloud account. Care is taken to ensure that created resources are removed. However, it is advisable to inspect your AWS console to ensure no unexpected resources are running.

Windows Tests

These tests exercise the winrm connection plugin and Windows modules. You'll need to define an inventory with a remote Windows 2008 or 2012 Server to use for testing, and enable PowerShell Remoting to continue.

Running these tests may result in changes to your Windows host, so don't run them against a production/critical Windows environment.

Enable PowerShell Remoting (run on the Windows host via Remote Desktop): Enable-PSRemoting -Force

Define Windows inventory:

cp inventory.winrm.template inventory.winrm
${EDITOR:-vi} inventory.winrm

Run the Windows tests executed by our CI system:

test/runner/ansible-test windows-integration -v windows/ci/

Tests in Docker containers

If you have a Linux system with Docker installed, running integration tests using the same Docker containers used by the Ansible continuous integration (CI) system is recommended.

Using Docker Engine to run Docker on a non-Linux host is not recommended. Some tests may fail, depending on the image used for testing. Using the --docker-privileged option may resolve the issue.

Running Integration Tests

To run all CI integration test targets for POSIX platforms in a Ubuntu 16.04 container:

test/runner/ansible-test integration -v posix/ci/ --docker

You can also run specific tests or select a different Linux distribution. For example, to run tests for the ping module on a Ubuntu 14.04 container:

test/runner/ansible-test integration -v ping --docker ubuntu1404

Container Images

Python 2

Most container images are for testing with Python 2:

  • centos6
  • centos7
  • fedora24
  • fedora25
  • opensuse42.1
  • opensuse42.2
  • ubuntu1204
  • ubuntu1404
  • ubuntu1604

Python 3

To test with Python 3 use the following images:

  • ubuntu1604py3

Network Tests

Note: From Ansible 2.3, for any new Network Module to be accepted it must be accompanied by a corresponding test.

For further help with this please contact gundalow in #ansible-devel on FreeNode IRC.

$ ANSIBLE_ROLES_PATH=targets ansible-playbook network-all.yaml

NOTE To run the network tests you will need a number of test machines and sutabily configured inventory file, a sample is included in test/integration/inventory.network

NOTE As with the rest of the integration tests, they can be found grouped by module in test/integration/targets/MODULENAME/

To filter a set of test cases set limit_to to the name of the group, generally this is the name of the module:

$ ANSIBLE_ROLES_PATH=targets ansible-playbook -i inventory.network network-all.yaml -e "limit_to=eos_command"

To filter a singular test case set the tags options to eapi or cli, set limit_to to the test group, and test_cases to the name of the test:

$ ANSIBLE_ROLES_PATH=targets ansible-playbook -i inventory.network network-all.yaml --tags="cli" -e "limit_to=eos_command test_case=notequal"

Contributing Test Cases

Test cases are added to roles based on the module being testing. Test cases should include both cli and eapi test cases. Cli test cases should be added to test/integration/targets/modulename/tests/cli and eapi tests should be added to test/integration/targets/modulename/tests/eapi.

In addition to positive testing, negative tests are required to ensure user friendly warnings & errors are generated, rather than backtraces, for example:

- name: test invalid subset (foobar)
  eos_facts:
    provider: "{{ cli }}"
    gather_subset:
      - "foobar"
  register: result
  ignore_errors: true

- assert:
    that:
      # Failures shouldn't return changes
      - "result.changed == false"
      # It's a failure
      - "result.failed == true"
      # Sensible Failure message
      - "'Subset must be one of' in result.msg"

Conventions

  • Each test case should generally follow the pattern:

    setup —> test —> assert —> test again (idempotent) —> assert —> -teardown (if needed) -> done

    This keeps test playbooks from becoming monolithic and difficult to troubleshoot.

  • Include a name for each task that is not an assertion. (It's OK to add names to assertions too. But to make it easy to identify the broken task within a failed test, at least provide a helpful name for each task.)

  • Files containing test cases must end in .yaml

Adding a new Network Platform

A top level playbook is required such as ansible/test/integration/eos.yaml which needs to be references by ansible/test/integration/network-all.yaml