diff --git a/test/README.md b/test/README.md deleted file mode 100644 index bf7d3de497d..00000000000 --- a/test/README.md +++ /dev/null @@ -1,31 +0,0 @@ -Ansible Test System -=================== - -Folders -======= - -units ------ - -Unit tests that test small pieces of code not suited for the integration test layer, usually very API based, and should leverage -mock interfaces rather than producing side effects. - -Playbook engine code is better suited for integration tests. - -Requirements: `pip install -r test/runner/requirements/units.txt` - -integration ------------ - -Integration test layer, constructed using playbooks. - -Some tests may require cloud credentials, others will not, and destructive tests are separated from non-destructive so a subset -can be run on development machines. - -learn more ----------- - -hop into a subdirectory and see the associated README.md for more info. - - - diff --git a/test/README.rst b/test/README.rst new file mode 100644 index 00000000000..20d42dd12f6 --- /dev/null +++ b/test/README.rst @@ -0,0 +1,140 @@ +Testing Ansible +=============== + +How to run and create tests for the Ansible core engine and modules with ``ansible-test``. + +Requirements +============ + +There are no special requirements for running ``ansible-test`` on Python 2.7 or later. +The ``argparse`` package is required for Python 2.6. +The requirements for each ``ansible-test`` command are covered later. + +Setup +===== + +#. Fork the `ansible/ansible `_ repository on Git Hub. +#. Clone your fork: ``git clone git@github.com:USERNAME/ansible.git`` +#. Install the optional ``argcomplete`` package for tab completion (highly recommended): + + #. ``pip install argcomplete`` + #. ``activate-global-python-argcomplete`` + #. Restart your shell to complete global activation. + +#. Configure your environment to run from your clone (once per shell): ``. hacking/env-setup`` + +Test Environments +================= + +Most ``ansible-test`` commands support running in one or more isolated test environments to simplify testing. + +Local +----- + +The ``--local`` option runs tests locally without the use of an isolated test environment. +This is the default behavior. + + Recommended for ``compile`` tests. + +See the `command requirements directory `_ for the requirements for each ``ansible-test`` command. +Requirements files are named after their respective commands. +See also the `constraints `_ applicable to all commands. + + Use the ``--requirements`` option to automatically install ``pip`` requirements relevant to the command being used. + +Docker +------ + +The ``--docker`` option runs tests in a docker container. + + Recommended for ``integration`` tests. + +This option accepts an optional docker container image. +See the `list of supported docker images `_ for options. + + Use the ``--docker-no-pull`` option to avoid pulling the latest container image. + This is required when using custom local images that are not available for download. + +Tox +--- + +The ``--tox`` option run tests in a ``tox`` managed Python virtual environment. + + Recommended for ``windows-integration`` and ``units`` tests. + +The following Python versions are supported: + +* 2.6 +* 2.7 +* 3.5 +* 3.6 + +By default, test commands will run against all supported Python versions when using ``tox``. + + Use the ``--python`` option to specify a single Python version to use for test commands. + +Remote +------ + +The ``--remote`` option runs tests in a cloud hosted environment. +An API key is required to use this feature. + + Recommended for integration tests. + +See the `list of supported platforms and versions `_ for additional details. + +General Usage +============= + +Tests are run with the ``ansible-test`` command. +Consult ``ansible-test --help`` for usage information not covered here. + + Use the ``--explain`` option to see what commands will be executed without actually running them. + +Running Tests +============= + +There are four main categories of tests, each in their own directory. + +* `compile `_ - Python syntax checking for supported versions. Examples: + + * ``ansible-test compile`` - Check syntax for all supported versions. + * ``ansible-test compile --python 3.5`` - Check only Python 3.5 syntax. + +* `sanity `_ - Static code analysis and general purpose script-based tests. Examples: + + * ``ansible-test sanity --tox --python 2.7`` - Run all sanity tests on Python 2.7 using ``tox``. + * ``ansible-test sanity --test pep8`` - Run the ``pep8`` test without ``tox``. + +* `integration `_ - Playbook based tests for modules and core engine functionality. Examples: + + * ``ansible-test integration ping --docker`` - Run the ``ping`` module test using ``docker``. + * ``ansible-test windows-integration windows/ci/`` - Run all Windows tests covered by CI. + +* `units `_ - API oriented tests using mock interfaces for modules and core engine functionality. Examples: + + * ``ansible-test units --tox`` - Run all unit tests on all supported Python versions using ``tox``. + * ``ansible-test units --tox --python 2.7 test/units/vars/`` - Run specific tests on Python 2.7 using ``tox``. + +Consult each of the test directories for additional details on usage and requirements. + +Interactive Shell +================= + +Use the ``ansible-test shell`` command to get an interactive shell in the same environment used to run tests. Examples: + +* ``ansible-test shell --docker`` - Open a shell in the default docker container. +* ``ansible-test shell --tox --python 3.6`` - Open a shell in the Python 3.6 ``tox`` environment. + +Code Coverage +============= + +Add the ``--coverage`` option to any test command to collect code coverage data. + +Reports can be generated in several different formats: + +* ``ansible-test coverage report`` - Console report. +* ``ansible-test coverage html`` - HTML report. +* ``ansible-test coverage xml`` - XML report. + +To clear data between test runs, use the ``ansible-test coverage erase`` command. diff --git a/test/compile/README.rst b/test/compile/README.rst new file mode 100644 index 00000000000..0801f69def5 --- /dev/null +++ b/test/compile/README.rst @@ -0,0 +1,13 @@ +Compile Tests +============= + +Compile tests check source files for valid syntax on all supported python versions: + +- 2.6 +- 2.7 +- 3.5 +- 3.6 + +Tests are run with ``ansible-test compile``. +All versions are tested unless the ``--python`` option is used. +All ``*.py`` files are tested unless specific files are specified. diff --git a/test/sanity/README.rst b/test/sanity/README.rst new file mode 100644 index 00000000000..2bf94b532c5 --- /dev/null +++ b/test/sanity/README.rst @@ -0,0 +1,70 @@ +Sanity Tests +============ + +Sanity tests are made up of scripts and tools used to perform static code analysis. +The primary purpose of these tests is to enforce Ansible coding standards and requirements. + +Tests are run with ``ansible-test sanity``. +All available tests are run unless the ``--test`` option is used. + +Available Tests +=============== + +Tests can be listed with ``ansible-test sanity --list-tests``. + +This list is a combination of two different categories of tests. + +Code Smell Tests +---------------- + +Miscellaneous `scripts `_ used for enforcing coding standards and requirements, identifying trip hazards, etc. + + These tests are listed and accessed by script name. There is no actual test named ``code-smell``. + +All executable scripts added to the ``code-smell`` directory are automatically detected and executed by ``ansible-test``. + +Scripts in the directory which fail can be skipped by adding them to `skip.txt `_. +This is useful for scripts which identify issues that have not yet been resolved in the code base. + +Files tested are specific to the individual test scripts and are not affected by command line arguments. + +Built-in Tests +-------------- + +These tests are integrated directly into ``ansible-test``. +All files relevant to each test are tested unless specific files are specified. + +ansible-doc +~~~~~~~~~~~ + +Verifies that ``ansible-doc`` can parse module documentation on all supported python versions. + +pep8 +~~~~ + +Python static analysis for PEP 8 style guideline compliance. + +pylint +~~~~~~ + +Python static analysis for common programming errors. + +rstcheck +~~~~~~~~ + +Check reStructuredText files for syntax and formatting issues. + +shellcheck +~~~~~~~~~~ + +Static code analysis for shell scripts using the excellent `shellcheck `_ tool. + +validate-modules +~~~~~~~~~~~~~~~~ + +Analyze modules for common issues in code and documentation. + +yamllint +~~~~~~~~ + +Check YAML files for syntax and formatting issues.