ansible/docs/docsite/rst/dev_guide/testing_integration.rst
Sandra McCann ceb474bb9e [WIP] Backport/2.7/batch port (#45859)
Batch of docs backports:

* docs: Clarify include_task v import_tasks with conditionals (#43856)
(cherry picked from commit 6be42a2a0e)

* Add single quotes around package name (#45152)
(cherry picked from commit 0d81386144)

* prefer ansible_facts namespace and dict notation (#44980)
(cherry picked from commit 44510448b0)

* fix cherrypick conflict - scenario_guides

* Update implicit_localhost.rst (#45455)
(cherry picked from commit f68cd1acc6)

* updated fbsd install instructions (#45309)
(cherry picked from commit e9c2695ce7)

* Change "Defaulting Undefined Variables" (#41379)
(cherry picked from commit e35c4be1c1)

* adds license details to dev guide pages (#45574)
(cherry picked from commit 6e68d77f6d)

* FAQ: fix a typo, add link to 'vars' lookup (#42412)
(cherry picked from commit 95649dc793)

* Fix link and toctree (#45595)
(cherry picked from commit 6999bf318f)

* Improve the local toctree (and title) (#45590)
(cherry picked from commit afea00fa9f)

* Add undocumented configuration parameter and explain in porting guide (#36059)
(cherry picked from commit a892a6ef03)

* Simplify PPA installation for Ubuntu (#45690)
(cherry picked from commit 78e9f452a5)

* adding git+ssh uri scheme (#36025)
(cherry picked from commit 84a4257774)

* Add workaround for non-standard kerberos environments (#41465)
(cherry picked from commit 4e532e0ad9)

* Restore license agreement (#45809)
(cherry picked from commit f430f60541)

* partial cherry-pick - lenovo doc update PR 45483
2018-09-19 22:02:01 -05:00

251 lines
9.3 KiB
ReStructuredText

:orphan:
.. _testing_integration:
*****************
Integration tests
*****************
.. contents:: Topics
The Ansible integration Test system.
Tests for playbooks, by playbooks.
Some tests may require credentials. Credentials may be specified with `credentials.yml`.
Some tests may require root.
Quick Start
===========
It is highly recommended that you install and activate the ``argcomplete`` python package.
It provides tab completion in ``bash`` for the ``ansible-test`` test runner.
Configuration
=============
Making your own version of ``integration_config.yml`` can allow for setting some
tunable parameters to help run the tests better in your environment. Some
tests (e.g. cloud) will only run when access credentials are provided. For
more information about supported credentials, refer to ``credentials.template``.
Prerequisites
=============
The tests will assume things like hg, svn, and git are installed and in path. Some tests
(such as those for Amazon Web Services) need separate definitions, which will be covered
later in this document.
(Complete list pending)
Non-destructive Tests
=====================
These tests will modify files in subdirectories, but will not do things that install or remove packages or things
outside of those test subdirectories. They will also not reconfigure or bounce system services.
.. note:: Running integration tests within Docker
To protect your system from any potential changes caused by integration tests, and to ensure the a sensible set of dependencies are available we recommend that you always run integration tests with the ``--docker`` option. See the `list of supported docker images <https://github.com/ansible/ansible/blob/devel/test/runner/completion/docker.txt>`_ for options.
.. note:: Avoiding pulling new Docker images
Use the ``--docker-no-pull`` option to avoid pulling the latest container image. This is required when using custom local images that are not available for download.
Run as follows for all POSIX platform tests executed by our CI system::
test/runner/ansible-test integration --docker fedora25 -v posix/ci/
You can select specific tests as well, such as for individual modules::
test/runner/ansible-test integration -v ping
By installing ``argcomplete`` you can obtain a full list by doing::
test/runner/ansible-test integration <tab><tab>
Destructive Tests
=================
These tests are allowed to install and remove some trivial packages. You will likely want to devote these
to a virtual environment, such as Docker. They won't reformat your filesystem::
test/runner/ansible-test integration --docker fedora25 -v destructive/
Windows Tests
=============
These tests exercise the ``winrm`` connection plugin and Windows modules. You'll
need to define an inventory with a remote Windows 2008 or 2012 Server to use
for testing, and enable PowerShell Remoting to continue.
Running these tests may result in changes to your Windows host, so don't run
them against a production/critical Windows environment.
Enable PowerShell Remoting (run on the Windows host via Remote Desktop)::
Enable-PSRemoting -Force
Define Windows inventory::
cp inventory.winrm.template inventory.winrm
${EDITOR:-vi} inventory.winrm
Run the Windows tests executed by our CI system::
test/runner/ansible-test windows-integration -v windows/ci/
Tests in Docker containers
==========================
If you have a Linux system with Docker installed, running integration tests using the same Docker containers used by
the Ansible continuous integration (CI) system is recommended.
.. note:: Docker on non-Linux
Using Docker Engine to run Docker on a non-Linux host (such as macOS) is not recommended.
Some tests may fail, depending on the image used for testing.
Using the ``--docker-privileged`` option may resolve the issue.
Running Integration Tests
-------------------------
To run all CI integration test targets for POSIX platforms in a Ubuntu 16.04 container::
test/runner/ansible-test integration -v posix/ci/ --docker
You can also run specific tests or select a different Linux distribution.
For example, to run tests for the ``ping`` module on a Ubuntu 14.04 container::
test/runner/ansible-test integration -v ping --docker ubuntu1404
Container Images
----------------
Python 2
````````
Most container images are for testing with Python 2:
- centos6
- centos7
- fedora24
- fedora25
- opensuse42.1
- opensuse42.2
- ubuntu1204
- ubuntu1404
- ubuntu1604
Python 3
````````
To test with Python 3 use the following images:
- ubuntu1604py3
Legacy Cloud Tests
==================
Some of the cloud tests run as normal integration tests, and others run as legacy tests; see the
:doc:`testing_integration_legacy` page for more information.
Other configuration for Cloud Tests
===================================
In order to run some tests, you must provide access credentials in a file named
``cloud-config-aws.yml`` or ``cloud-config-cs.ini`` in the test/integration
directory. Corresponding .template files are available for for syntax help. The newer AWS
tests now use the file test/integration/cloud-config-aws.yml
IAM policies for AWS
====================
Ansible needs fairly wide ranging powers to run the tests in an AWS account. This rights can be provided to a dedicated user. These need to be configured before running the test.
testing-policies
----------------
``hacking/aws_config/testing_policies`` contains a set of policies that are required for all existing AWS module tests.
The ``hacking/aws_config/setup_iam.yml`` playbook can be used to add all of those policies to an IAM group (using
``-e iam_group=GROUP_NAME``. Once the group is created, you'll need to create a user and make the user a member of the
group. The policies are designed to minimize the rights of that user. Please note that while this policy does limit
the user to one region, this does not fully restrict the user (primarily due to the limitations of the Amazon ARN
notation). The user will still have wide privileges for viewing account definitions, and will also able to manage
some resources that are not related to testing (for example, AWS lambdas with different names). Tests should not
be run in a primary production account in any case.
Other Definitions required
--------------------------
Apart from installing the policy and giving it to the user identity running the tests, a
lambda role `ansible_integration_tests` has to be created which has lambda basic execution
privileges.
Network Tests
=============
Starting with Ansible 2.4, all network modules MUST include unit tests that cover all functionality. You must add unit tests for each new network module and for each added feature. Please submit the unit tests and the code in a single PR. Integration tests are also strongly encouraged.
Writing network integration tests
---------------------------------
For guidance on writing network test see the `adding tests for Network modules guide <https://github.com/ansible/community/blob/master/group-network/network_test.rst>`_.
Running network integration tests locally
-----------------------------------------
Ansible uses Shippable to run an integration test suite on every PR, including new tests introduced by that PR. To find and fix problems in network modules, run the network integration test locally before you submit a PR.
To run the network integration tests, use a command in the form::
ansible-test network-integration --inventory /path/to/inventory tests_to_run
First, define a network inventory file::
cd test/integration
cp inventory.network.template inventory.networking
${EDITOR:-vi} inventory.networking
# Add in machines for the platform(s) you wish to test
To run all Network tests for a particular platform::
ansible-test network-integration --inventory /path/to/ansible/test/integration/inventory.networking vyos_.*
This example will run against all vyos modules. Note that ``vyos_.*`` is a regex match, not a bash wildcard - include the `.` if you modify this example.
To run integration tests for a specific module::
ansible-test network-integration --inventory /path/to/ansible/test/integration/inventory.networking vyos_vlan
To run a single test case on a specific module::
# Only run vyos_vlan/tests/cli/basic.yaml
ansible-test network-integration --inventory /path/to/ansible/test/integration/inventory.networking vyos_vlan --testcase basic
To run integration tests for a specific transport::
# Only run nxapi test
ansible-test network-integration --inventory /path/to/ansible/test/integration/inventory.networking --tags="nxapi" nxos_.*
# Skip any cli tests
ansible-test network-integration --inventory /path/to/ansible/test/integration/inventory.networking --skip-tags="cli" nxos_.*
See `test/integration/targets/nxos_bgp/tasks/main.yaml <https://github.com/ansible/ansible/blob/devel/test/integration/targets/nxos_bgp/tasks/main.yaml>`_ for how this is implemented in the tests.
For more options::
ansible-test network-integration --help
If you need additional help or feedback, reach out in ``#ansible-network`` on Freenode.
Where to find out more
======================
If you'd like to know more about the plans for improving testing Ansible, join the `Testing Working Group <https://github.com/ansible/community/blob/master/meetings/README.md>`_.