Commit graph

120 commits

Author SHA1 Message Date
Matt Clay
831e1bf2e0 Fix ansible-test PYTHONPATH handling.
Running from an installed version of ansible-test now results in tests using a dedicated directory for PYTHONPATH instead of using the site-packages directory where ansible is installed.

This provides consistency with tests running from source, which already used a dedicated directory.

Resolves https://github.com/ansible/ansible/issues/62716
2019-09-26 14:02:19 -04:00
Gonéri Le Bouder
5ecbe9cbbb vcenter provider: Only rely on VMWARE_TEST_PLATFORM
Until now, the vcenter provider was switching between `static` and
`govcsim` depending on the presence of the following configuration file:
`test/integration/cloud-config-vcenter.ini`.

This was not consistent with Worldstream, which we enable with the
`VMWARE_TEST_PLATFORM` environment variable.

We now only rely on `VMWARE_TEST_PLATFORM` to know which platform should be
used. `govcsim` is still the default, this to preserve the original
behaviour.

This commit also rename the following variables to be consistent with the rest
of the code base. It also ensures they are alway defined, even with `govcsim`:

- `VCENTER_HOSTNAME`
- `VCENTER_USERNAME`
- `VCENTER_PASSWORD`
2019-09-20 10:33:44 -04:00
Matt Clay
9f7b124a6f Mark ansible-test cloud credentials as sensitive. 2019-09-17 13:08:14 -07:00
Matt Clay
0631e057e9 Redact sensitive values by default in ansible-test 2019-09-17 08:09:17 -07:00
Felix Fontein
d8d3790d6a Bump acme-test-container version. (#62381) 2019-09-17 13:06:33 +02:00
Matt Clay
b73e7721df Mark AWS credentials in ansible-test as sensitive.
This avoids displaying the credentials in CI when retrying tests at maximum verbosity.
2019-09-16 20:21:23 -07:00
Matt Clay
32d965e2c6 Fix display of sanity test doc links.
The documentation links are now displayed when running from an install.

Previously the links were only displayed when running from source.
This was due to ansible-test checking for the presence of documentation files locally, which are only present when running from source.
The check is no longer necessary since there is a sanity test in place to enforce the presence of documentation for all sanity tests.
2019-09-13 22:24:14 -07:00
Matt Clay
cd4882e229 Fix ansible-test vcenter test filtering.
The default behavior of the ansible-test vcenter plugin is to use the govcsim container to run tests.

However, unless the govcsim mode was specified using the VMWARE_TEST_PLATFORM environment variable, the filter code would skip the tests unless the tests ran on Shippable or the user had an ansible-core-ci key.

Now the filter correctly recognizes that govcsim is the default.
2019-09-12 02:13:57 -07:00
Matt Clay
cdc4926340
Fix ansible-test collections requirements installation. (#62181)
* Fix location of unit test requirements.

* Preserve ansible-test unit test requirements.

* Remove redundant unit test requirements.

* Fix location of network test requirements.

* Preserve ansible-test network test requirements.

* Remove redundant network test requirements.

* Add missing ordereddict requirements.

* Load collection requirements correctly.

* Add changelog fragment.
2019-09-12 02:00:33 -07:00
Matt Clay
aaa6d2ecb0
Fix ansible-test pytest plugin loading. (#62119)
* Avoid assertion rewriting in pytest plugins.

Adding PYTEST_DONT_REWRITE to the ansible-test pytest plugin docstrings disables assertion rewriting in pytest for those plugins.

This avoids warnings during test execution if the plugins are loaded multiple times (such as being imported within tests).

* Run ansible-test pytest plugins early.

The ansible-test pytest plugins need to load and run earlier than conftest modules.

To facilitate this, the pytest_configure function is run during loading, which works since they are loaded (but not always run) before conftest modules are loaded.

A check has also been added to the pytest_configure functions to prevent them from running multiple times in the same process.

* Load pytest plugins using an env var.

The -p command line option loads plugins before conftest, but only during collection.
The PYTEST_PLUGINS environment variable loads plugins before confest, both during collection and test execution.
2019-09-10 23:27:05 -07:00
Jordan Borean
6fb1d56fdc
Fix ansible-test coverage --all (#62115) 2019-09-11 15:12:38 +10:00
Matt Clay
a7bc11ce67
Work around virtualenv/venv issue in ansible-test. (#62111)
Creating a virtual environment using `venv` when running in a virtual environment created by `virtualenv` results in a copy of the original virtual environment instead of creation of a new one.

To work around this, `ansible-test` now identifies when it is running in a `virtualenv` created virtual environment and uses the real Python interpreter to create the `venv` virtual environment.
2019-09-10 19:27:12 -07:00
Matt Clay
983f0ad5a0 Properly exclude tests/output/ from code coverage. 2019-09-10 15:50:22 -07:00
Matt Clay
f110abb806
Fix ansible-test to ignore tests/output/. (#62084)
The `test/results/` directory for Ansible test output was already ignored when not using git.

When Ansible Collections were switched to `tests/output/` the ignore entry was previously overlooked.
2019-09-10 15:49:40 -07:00
Matt Clay
064e8e1ef4
Fix ansible-doc traceback and sanity test. (#62040)
* Fix ansible-doc traceback for removed modules.

This avoids tracebacks with errors like the following when a module has been removed:

module module_name missing documentation (or could not parse documentation): 'NoneType' object does not support item assignment

* Fix ansible-doc sanity test warning handling.

Warnings about removed modules/plugins on stderr are now properly ignored.

Previously an ansible-doc error could result in unrelated errors going undetected because tests were stopped early and the underlying error was ignored.
2019-09-10 00:29:34 -07:00
Matt Clay
c77ab11051
Fix issues with ansible-test --venv option. (#62033)
* Fix ansible-test venv activation.

When using the ansible-test --venv option, an execv wrapper for each python interpreter is now used instead of a symbolic link.

* Fix ansible-test execv wrapper generation.

Use the currently running Python interpreter for the shebang in the execv wrapper instead of the selected interpreter.

This allows the wrapper to work when the selected interpreter is a script instead of a binary.

* Fix ansible-test sanity requirements install.

When running sanity tests on multiple Python versions, install requirements for all versions used instead of only the default version.

* Fix ansible-test --venv when installed.

When running ansible-test from an install, the --venv delegation option needs to make sure the ansible-test code is available in the created virtual environment.

Exposing system site packages does not work because the virtual environment may be for a different Python version than the one on which ansible-test is installed.
2019-09-09 18:32:29 -07:00
Gonéri Le Bouder
8e22a60f97 ansible-test: vcenter+Worldstream: hide password
Hide the temporary password when `ansible-test` is called with the
`--redact` parameter.
2019-09-03 10:34:34 -04:00
Sam Doran
e218c9814c ansible-test - Continue if the git command returns an error (#61605)
* ansible-test - Contiune if the git command returns an error

* Just return stdout

* Use to_text() when displaying exception

* Add a message property to SubprocessError
2019-08-30 12:03:39 -07:00
Matt Clay
e4e5005640
Fix ansible-test coverage path handling. (#61528)
* Fix ansible-test coverage path handling.
* Split CI unit tests into two groups.
2019-08-30 00:18:49 -07:00
Matt Clay
4063d58339 Use relative submodule status in ansible-test.
The `git submodule status` command is relative to the current git repository by default.
When running from a repository subdirectory paths can be returned above the current directory.
Specifying the current directory with `git submodule status` avoids listing submodules above that directory.

This will fix issues when testing a collection that is rooted below the repository root when that repository uses submodules.
2019-08-29 20:29:38 -07:00
Matt Clay
2215a62057 Fix ansible-test handling of submodules.
Resolves https://github.com/ansible/ansible/issues/61550
2019-08-29 12:21:28 -07:00
Matt Clay
f90587069c
Update collections integration targets path. (#61523)
* Update collections integration targets path.

* Fix integration path handling.

* Add collections test target fallback.

Also add warnings and errors for common path mistakes.

* Improve role target detection.
2019-08-29 03:39:03 -07:00
Matt Clay
032bd3a030 Move the ansible-test output_dir.
This directory is currently a fixed location to make troubleshooting easier.
It is cleared before each test target runs, but is preserved when a test target finishes.
This allows the contents to be inspected when a test fails.

The previous location was `~/ansible_testing/`.
The new location is within the content root:

- `test/results/.tmp/output_dir` for Ansible
- `tests/output/.tmp/output_dir` for Ansible Collections

Moving the directory reduces the number of places on the filesystem where tests create output.
It also enables the results to be returned from delegated systems.
2019-08-29 02:32:31 -07:00
Matt Clay
d92e2a6b0e
Simplify ansible-test target processing. (#61506)
* Clean up layout paths for integration tests.

* Remove "special" integration test target type.

* Remove unnecessary role detection logic.

* Remove support for non-sh runme scripts.

* Simplify reading of aliases.
2019-08-28 22:18:20 -07:00
Matt Clay
81ca44b7e5 Change location of collections tests.
The directories used for collections tests are changing as follows:

`test/` -> `tests/`
`test/results/` -> `tests/output/`

This is a breaking change for collections tests executed by ansible-test.
All collections will need to be updated to use the new directory.

The `tests/output/` directory should be added to the `.gitignore` or equivalent in each collection.

This change is being made before the first pre-release of Ansible 2.9 since ansible-test has not yet been shipped.
Using the `tests/` directory matches the other collections directories `plugins/` and `roles/`.

This resolves https://github.com/ansible/ansible/issues/60218
2019-08-28 19:15:36 -07:00
Matt Clay
460b583dff Improve ansible-test --venv error handling.
This allows it to fall back to virtualenv when venv is broken.
2019-08-28 16:50:42 -07:00
Sam Doran
8dd79fbbd2 Show doc URL for sanity test failures 2019-08-28 15:32:31 -07:00
Matt Clay
4afe757f93
More ansible-test path handling updates. (#61484)
* Move test results vars into layout.

* Add test_path to layout.

* Add sanity_path to layout.

* Clean up layout.

* Update change classification.

* Improve classification layout.

* Relocate common classification code.

* Use is_subdir for relocated code.

* Relocate ansible test/units/compat/ hack.

* Fix variable shadowing.

* Remove unused code.

* Fix ordering of path tests.

* Clean up ansible classification.

* Fix hard-coded plugin paths in classification.

* Relocate more common classification.

* Fix logic.

* Fix pylint issue.

* Add missing licenses classification.
2019-08-28 15:13:00 -07:00
Sam Doran
73248bf27d validate-modules - Use error codes instead of numbers (#60711) 2019-08-28 14:12:56 -07:00
Matt Clay
1db78d4ba4 Attempt quick fix for CI failures. 2019-08-28 09:42:23 -07:00
Matt Clay
830f995ed4
Add a --venv option to ansible-test. (#61422)
* Add --venv delegation to ansible-test.

* Update import sanity test venv creation.

* Fix import test when using --venv on Python 2.x.

* Improve virtualenv setup overhead.

* Hide pip noise for import sanity test.

* Raise verbosity on venv info messages.

* Get rid of base branch noise for collections.

* Add missing --requirements check.
2019-08-28 09:10:17 -07:00
Matt Clay
f5d829392a
Overhaul ansible-test test path handling. (#61416)
* Remove .keep files from test/results/ dirs.

* Remove classification of test/results/ dir.

* Add results_relative to data context.

* Use variables in delegation paths.

* Standardize file writing and results paths.

* Fix issues reported by PyCharm.

* Clean up invocation of coverage command.

It now runs through the injector.

* Hack to allow intercept_command in cover.py.

* Simplify git ignore for test results.

* Use test result tmp dir instead of cache dir.

* Remove old .pytest_cache reference.

* Fix unit test docker delegation.

* Show HTML report link.

* Clean up more results references.

* Move import sanity test output to .tmp dir.

* Exclude test results dir from coverage.

* Fix import sanity test lib paths.

* Fix hard-coded import test paths.

* Fix most hard-coded integration test paths.

* Fix PyCharm warnings.

* Fix import placement.

* Fix integration test dir path.

* Fix Shippable scripts.

* Fix Shippable matrix check.

* Overhaul key pair management.
2019-08-27 23:40:06 -07:00
Jordan Borean
66f52b74b1 Change collection PS util import pattern (#61307)
* Change collection PS util import pattern

* Add changes for py2 compat

* fix up regex and doc errors

* fix up import analysis

* Sanity fix for 2.6 CI workers

* Get collection util path for coverage collection
2019-08-27 16:02:27 -07:00
Jordan Borean
faaa669764
PowerShell - Added coverage collector (#59009)
* Added coverage collection for PowerShell - ci_complete ci_coverage

* uncomment out coverage uploader call

* Generate XML for PowerShell coverage

* Use whitelist to exclude coverage run on non content plugins

* Remove uneeded ignore entry

* Try to reduce diff in cover.py

* Fix up coverage report package - ci_complete ci_coverage
2019-08-28 07:03:23 +10:00
Gonéri Le Bouder
e7a9d71ac0
ansible-test: async deployment w/ vcenter provider (#59984)
ansible-test: async deployment w/ vcenter provider

So far, `ansible-test`'s vcenter provider was trigger by a blocking POST
call. Since the depoyment take a lot of time (> 8m), we decided to use
an async call instead and poll the API server until the deployment
is ready.
2019-08-27 14:56:43 -04:00
Matt Clay
8a7b89c66f Support delegation testing collections with deps.
All collections in the same collection root as the collection being tested will be sent to the remote system/container.
2019-08-27 07:29:42 -07:00
Matt Clay
e258ba703e
Improve ansible-test env display. (#61339)
* Improve ansible-test env display.
* Fix PyCharm warnings.
* Fix ansible-test ansible-doc sanity test with -e.
2019-08-26 14:02:55 -07:00
Matt Clay
02c38ed4fd
Relocate ansible-test self tests outside package. (#61255)
* Relocate ansible-test self tests outside package.

We don't want to include the tests for verifying ansible-test within the ansible-test package.

* Add `test/ansible_test/` to classification.py.

* Fix test invocation.

* Relocate tests in MANIFEST.in.

* Improve package-data sanity test error checking.

* Only use includes for ansible-test in MANIFEST.in.

* Improve readability of MANIFEST.in.
2019-08-24 11:38:30 -07:00
Matt Clay
2a9b996bd2 Use .cfg extension for pylint configs. 2019-08-24 07:25:19 -07:00
Matt Clay
6ec0b4ec86
Sanity test updates for collections support. (#61248)
* Run no-unwanted-files sanity test only on Ansible.

Since collections should be able to use binary modules there is not really any limit on what could exist in a collection `plugins` directory.

* Add support for symlinks in sanity target lists:

- Sanity tests that need to analyze symlinks can do so using the supplied target list.
- Tests that analyze directories will now only look at symlinks if requested.
- Directory symlinks will now be seen as directories instead of files.

* Enable symlinks on filename based sanity tests.

Sanity tests that evalulate filenames instead of content should include symlinks.

* Update symlinks sanity test.

Use the sanity test target list now that it can include symlinks.
2019-08-23 18:08:21 -07:00
Felix Fontein
4551965af1 ACME modules: bump test container version (#61192)
* Fix root cert path.

* Bump testing container version.

* Improve check (needed due to letsencrypt/pebble#257).
2019-08-23 15:30:28 -07:00
Matt Clay
2eff25aef8 Update ansible-test network inventory handling.
DCI uses an inventory directory instead of a file, along with --no-temp-workdir.

This changes the inventory check to keep that scenario working until all DCI jobs can be migrated to Zuul.
2019-08-22 17:35:41 -07:00
Matt Martz
adb886e4ce
Speed up units using time.sleep by mocking or shortening sleep time (#61064)
* Speed up units using time.sleep by mocking or shortening sleep time

* Only use durations when not a collection
2019-08-21 16:52:58 -05:00
Matt Clay
3551d83289 Fix ansible-test payload bin creation. 2019-08-21 12:45:55 -07:00
Matt Clay
29ac0273d4
Update ansible-test collection inventory handling. (#61031)
* Update ansible-test collection inventory handling.

- The `windows-integration` command now supports the `--inventory` option.
- The incomplete support for host_vars and group_vars directories has been removed.
- The incomplete support for an inventory directory has been removed.
- The inventory specified by `--inventory` can now reside outside the install and content roots.
- Using `ansible_ssh_private_key_file` with `--docker` or `--remote` results in a warning about the combination being unsupported and likely to fail.

* Fix config handling.

* Fix payload handling of ssh keys.

* Disable pylint no-self-use rule for ansible-test.

* De-duplicate payload paths.
2019-08-21 12:12:37 -07:00
Matt Clay
32b0a72547
Support windows-integration on collections. (#60984)
* Support windows-integration on collections.

* Update configure-remoting-ps1 sanity test.

* Add copy of ConfigureRemotingForAnsible.ps1.

* Add ignore entry for copied file.
2019-08-20 22:23:28 -07:00
Matt Clay
92380d7050
Update ansible-test --windows inventory creation. (#60911)
Host specific settings are created as part of the generated inventory instead of using a host_vars directory.

Also update connection tests to work properly with the generated inventory changes.
2019-08-20 07:17:32 -07:00
Matt Clay
ebe55e83c3 Fix ansible-test bin creation.
Use symlink instead of link.
2019-08-16 14:00:43 -07:00
Matt Martz
def3d1f815
validate-modules: support collections (#60247)
* Start of work to support collections

* remove version_added from base schema

* If a collection, pass that to validate-modules

* clean ups

* Allow version_added in a collection, just make it optional

* Don't traceback on missing doc_fragment

* Don't validate metadata in a collection
2019-08-16 14:28:34 -05:00
Matt Clay
d9c44897a6 Fix ansible-test cloud vars overwrite. 2019-08-15 10:20:14 -07:00
Matt Clay
7968dc4c00 Load integration config test vars from content.
The `test/integration/integration_config.yml` vars file will now be loaded from the content under test and is now optional.

The `output_dir` and `win_output_dir` vars are now provided by ansible-test.
2019-08-14 20:40:59 -07:00
Matt Clay
ba1b4565d3 Clean up ansible-test results path handling.
This brings us closer to being able to use `tests/` instead of `test/` in collections.
2019-08-13 21:51:57 -07:00
Matt Clay
1b8aa798df
Update ansible-test integration config handling. (#60448)
* Move Ansible test config into env vars.

This allows ansible-test to use an empty Ansible config file, leaving open the option for users to customize the one used to run tests (although such usage is discouraged).

* Use config from content under test when present.
2019-08-13 13:18:55 -07:00
Matt Clay
25d396fdfc Fix testing of Windows binary modules. 2019-08-12 22:19:35 -07:00
Gonéri Le Bouder
a3407682ed ansible-test: fix the vcenter+vcsim support
The vcenter provider allow one to use the `VMWARE_TEST_PLATFORM` to
manually select a test platform. The test platforms are:

- `govcsim`
- `static`
- and `worldstream`.

Before this commit, the `govcsim` value was ignored.

Example:

```shell
$ VMWARE_TEST_PLATFORM=govcsim ansible-test integration --python 3.7 needs/target/prepare_vmware_tests/
```
2019-08-12 11:55:38 +02:00
Matt Clay
3fac6a0adb Fix ansible-test TypeVar type annotations. 2019-08-09 17:29:41 -07:00
Matt Clay
cb68110296
Clean up ansible-test inventory templates. (#60321)
* Move ansible-test inventory templates.
* Remove obsolete inventoy.remote.template file.
2019-08-09 11:06:21 -07:00
Matt Clay
2a9964ede8
Code cleanup in ansible-test. (#60318)
* Move ansible-test units code into separate dir.
* Fix --explain on "code smell" sanity tests.
* Add --strict-markers to pytest invoctaion.
* Fix classification of units code changes.
2019-08-09 09:38:19 -07:00
Matt Clay
df59f06d6a
Additional ansible-test path fixes. (#60300)
* Add missing ansible-test injector symlink.
* More ansible-test path fixes.
* Expand bin-symlinks docs.
2019-08-08 23:51:03 -07:00
Matt Clay
39b3fc0926 Update ansible-test layout and payload handling. 2019-08-08 21:34:38 -07:00
Abhijeet Kasurde
647b78a09c
VMware: Check if datastore is valid (#58872)
Check validation of datastore selected using autoselect_datastore API.

Fixes: #58541

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2019-08-09 07:56:52 +05:30
Matt Clay
57dc7ec265
Prepare ansible-test for inclusion in setup.py (#60294)
* Cache ansible version lookup.
* Fix method of determining ANSIBLE_ROOT.
* Clean up based on PyCharm inspections.
* Generate minimal PKG-INFO without setup.py.
* Use ANSIBLE_LIB_ROOT where possible.
* Use import instead of subprocess to get version.
* Fix install layout type.
* Correct required paths message for installs.
* Update list of files copied during delegation.
* Fix ansible-test entry point.
* Fix pylint issue.
* Fix version lookup on Python 2.x.
* Fix pylint issue.
* Remove unwanted print statement.
2019-08-08 16:14:19 -07:00
Matt Clay
5f67b5a757 Move ansible-test coverage config to test/lib/ 2019-08-07 18:32:46 -07:00
Matt Clay
2e7d36a3f9
Fix placement and handling of ansible-test cloud config (#60253)
* Move cloud templates.
* Update paths for cloud config templates.
* Fix cloud config paths.
* Fix cloud-config classification.
2019-08-07 18:32:30 -07:00
Matt Clay
278044358c Split up ansible-test sanity test requirements. 2019-08-07 09:34:26 -07:00
Matt Clay
a93a0c8fd6
Relocate validate-modules for easier testing. (#60214)
* Relocate module validator code and tests.
* Fix validate-modules entry point and imports.
* Fix paths and test entry points.
* Fix up unit tests.
* Fix shebang and execute bit.
2019-08-07 09:34:11 -07:00
Matt Clay
18867847f4
Split up "code smell" sanity tests. (#60174)
* Relocate ansible-only sanity tests.

* Get "code smell" sanity tests from multiple dirs.

- `test/lib/ansible_test/_data/sanity/code-smell/` - General purpose tests used for both Ansible and Ansible Collections.
- `test/sanity/code-smell/` - Tests specific to Ansible, will not be used for Ansible Collections.
2019-08-06 17:17:57 -07:00
Matt Clay
709a8c5026 Fix ansible-test classification of sanity changes. 2019-08-06 17:17:31 -07:00
Matt Clay
9da5908afb Fix ansible-test layout path generation. 2019-08-06 16:07:53 -07:00
Matt Clay
d651bda123
Relocate ansible-test code. (#60147)
* Initial move of `test/runner/` content.

`test/runner/lib/` -> `test/lib/ansible_test/_internal/`
`test/runner/`     -> `test/lib/ansible_test/_internal/data/`

* Initial move of `test/sanity/` content.

`test/sanity/` -> `test/lib/ansible_test/_internal/data/sanity/` (except `test/sanity/ignore.txt`)

* Initial move of `test/units/pytest/` content.

`test/units/pytest/` -> `test/lib/ansible_test/_internal/data/pytest/`

* Follow-up move of `test/runner/unit/` content.

`test/lib/ansible_test/_internal/data/unit/` -> `test/lib/ansible_test/tests/unit/`

* Initial move of `ansible.cfg` content.

`test/units/ansible.cfg` -> `test/lib/ansible_test/_internal/data/units/ansible.cfg`
`test/env/ansible.cfg` -> `test/lib/ansible_test/_internal/data/env/ansible.cfg`

* Follow-up move of `data` directory.

`test/lib/ansible_test/_internal/data/` -> `test/lib/ansible_test/_data/`

* Update import statements.

* Add missing __init__.py for unit tests.

* Fix path references and miscellaneous issues.
2019-08-06 14:43:29 -07:00