The last task in a play should now properly report code coverage.
This change should also eliminate empty coverage files, as well as incomplete coverage files resulting from early worker termination.
* Fix ansible-test coverage analysis option usage.
The `--input-dir` option for `coverage analyze targets generate` was being ignored.
No changelog entry since this feature has not yet been released.
* Move coverage config to fix type annotations.
Declaring the types before referencing them makes sure they're recognized by tools such as PyCharm.
Now empty `*.py` files are ignored during module_utils import analysis for change detection.
This eliminates "No imports found" warnings for files which should have no imports.
* Internally redirect win modules to collection name
* Added comment for how this should be fixed in a subsystem plugin
* add collection plugins for base tests
Requirements were incorrectly added to ansible-test in https://github.com/ansible/ansible/pull/61813
These requirements should have been placed into `test/units/requirements.txt` instead.
Now that the relevant content has been migrated out of the repository, the requirements are no longer necessary there either.
No changelog entry for this change since the original changes were not included in any release and also lacked a changelog entry.
* ansible-galaxy - optimise some paths and use fake galaxy int tests
* Added init, built, and publish tests
* Test against both mocked Galaxy and AH server
* Finish off writing the install tests
* Fix up broken tests
* Rename test target and add migrated tests
* Use cloud provider for Galaxy implementation
* Added blank static config
* Use correct alias group
* Set release version and fix copy typo
* Remove reset step as it is no longer needed
* Use sane env var names for test container name
Make AWS test plugin to refer to the aws-permissions-for-integration-tests section of the AWS devel docs now that we have them, rather than suggesting to flag down specific users.
* Support private test plugins in ansible-test.
This feature is exclusively used for testing Ansible itself and is not available to collections.
Content in the newly supported directories will be added in follow-up PRs.
* Add new directory to path classification.
* Support new paths in shebang test.
* Add new directory to manifest.
* Add warning when running devel. Addresses #67362
* Add config entry to disable devel warning, so it doesn't impact CI
* Update warning about using devel
* required_if checks should have three or four parts.
* Validate mutually_exclusive, required_together, required_one_of, required_if and required_by.
* Simplify code.
* Improve messages.
* Add changelog.
* Sanity check.
* Update docs.
* Update ignore.txt.
* Don't continue with tests when terms are not strings.
* Remove ignore.txt entry.
* Make sure validate-modules doesn't choke on things already flagged by schema test.
* Check required_if requirements list for strings.
* 'message' parameter is replaced by 'commit_message' in grafana_dashboard
* 'message' parameter is replaced by 'notification_message' in datadog_monitor
This change is required since 'message' as parameter name is used internally by
Ansible core engine.
Fixes: #39295#45362#47132#59617
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* Make validate-modules stop ignore FILE_COMMON_ARGUMENTS.
* Add types to FILE_COMMON_ARGUMENTS and update document fragment to match it.
* Update ignore.txt.
* Add changelog.
* Refactor coverage file enumeration.
* Relocate sanitize_filename function.
* Support sets when writing JSON files.
* Generalize setting of info_stderr mode.
* Split out coverage path checking.
* Split out collection regex logic.
* Improve sanitize_filename type hints and docs.
* Clean up coverage erase command.
* Fix docs and type hints for initialize_coverage.
* Update type hints on CoverageConfig.
* Split out logic for finding modules.
* Split out arc enumeration.
* Split out powershell coverage enumeration.
* Raise verbosity level of empty coverage warnings.
* Add code coverage target analysis to ansible-test.
This change moves all code for the `ansible-test coverage` command into the `coverage` directory.
Each subcommand is split into a separate file.
Only minor spelling changes were made aside from code relocation.
PR #66898
This change introduces a new sanity check with code
`parameter-state-invalid-choice` in the `ansible-test sanity`
validator. It enforces modules not to support `list` or `info`
as their `state`.
Co-Authored-By: Sviatoslav Sydorenko <webknjaz@redhat.com>
Co-Authored-By: Felix Fontein <felix@fontein.de>
Bind the govcsim service on port 1443, this way we don't need root privilege
to start the container. This allow us to use ansible-test with podman
seamlessly.
The commit also avoid the mapping of port 80. We don't need it.
Check that all yaml we ship is parsable by the pyyaml c backend. Since
Ansible uses Pyyaml for docs and playbooks, if the yaml files aren't
parsable, they will error out if they were used.
Warn and skip yamllint if libyaml backend is not present
Ignore new errors in examples until someone can fix them
* Add test for print() call in module_utils and modules.
* Add changelog fragment.
* Add ignore.txt entries.
* Use blacklist plugin instead of adding a new.
* Update ignore.txt
* Revert most of PR #61605 commit e218c9814c
This removes the git error handling that converted all git errors into warnings.
* Fix ansible-test handling of git submodules.
Use the --venv option instead.
This option was only available when running from source to test the ansible/ansible repository.
This will have no effect on testing collections or running from an installed version of Ansible.
Also update docs to reference the --venv option instead of the --tox option.
* Set alter_sys=True instead of False to address backwards incompat
* ci_complete
* Add integration test
* ci_complete
* sanity
* ci_complete
* Changelog fragment
* Update import test and validate-modules to match
* Fix validate-modules support for collections.
- Relative imports now work correctly.
- The collection loader is now used.
- Modules are invoked as `__main__`.
* Remove obsolete validate-modules code ignores.
* Handle sys.exit in validate-modules.
* Add check for AnsibleModule initialization.
* Remove `missing-module-utils-import` check.
This check does not support relative imports or collections.
Instead of trying to overhaul the test, we can rely on the `ansible-module-not-initialized` test instead.
* Fix badly named error codes with `c#` in the name.
The `#` conflicts with comments in the sanity test ignore files.
* Add changelog entries.
* Add contains: validation for return values.
* Only require returned: on top level.
* Fix various return value problems.
* Update ignore.txt.
* Two more.
* Improve return value documentation by allowing entry for return values.
* Add docs formatting, adjust styling.
* Fix sample return value. (Taken from https://tools.ietf.org/html/rfc7517#appendix-A.1.)
* Work around abuse of .
* fix default collection resolution in adhoc
* if an adhoc command is run with a playbook-dir under a configured collection, default collection resolution is used to resolve unqualified module/action names
* Set ANSIBLE_PLAYBOOK_DIR in integration tests.
* Fix config conflict in ansible integration test.
* add adhoc default collection test
* text-ify warning string
Newer versions of ssh-keygen create PEM keys that are not recognized by Paramiko.
Now ansible-test compensates for this by updating they keys it generates so Paramiko will recognize them.
Previously the temporary directory used to run integration tests resided under the user's home directory. This prevented ansible-playbook from detecting the default collection when running tests.
Now the temporary directory is created within the collection to facilitate default collection detection.
The following modules depend on `vSphere Automation SDK`:
- `vmware_rest_client`
- `vmware_guest_info`
- `vmware_tag_manager`
- `vmware_vm_inventory`
The associated test cannot be run with `govcsim`. But the situation is
changing since we will soon run them on a regular lab, and so, we
need to install the dependency.
Bumping the default-test-container version to 1.9.3 to get a fresh version
of pip and requests.
Depends-On: https://github.com/ansible/ansible/pull/62412
This fixes test errors related to failures copying temporary test results files from a remote system back to the local system.
It also speeds up processing of test results and reduces network utilization by avoiding the temporary files.
Running from an installed version of ansible-test now results in tests using a dedicated directory for PYTHONPATH instead of using the site-packages directory where ansible is installed.
This provides consistency with tests running from source, which already used a dedicated directory.
Resolves https://github.com/ansible/ansible/issues/62716
Until now, the vcenter provider was switching between `static` and
`govcsim` depending on the presence of the following configuration file:
`test/integration/cloud-config-vcenter.ini`.
This was not consistent with Worldstream, which we enable with the
`VMWARE_TEST_PLATFORM` environment variable.
We now only rely on `VMWARE_TEST_PLATFORM` to know which platform should be
used. `govcsim` is still the default, this to preserve the original
behaviour.
This commit also rename the following variables to be consistent with the rest
of the code base. It also ensures they are alway defined, even with `govcsim`:
- `VCENTER_HOSTNAME`
- `VCENTER_USERNAME`
- `VCENTER_PASSWORD`
VMware VSphere SDK needs an up to date version of `pip` for the
installation step. With the current image, we face the following error:
```
(...)
02:27 Collecting git+https://github.com/vmware/vsphere-automation-sdk-python.git (from -r /root/ansible/test/lib/ansible_test/_data/requirements/integration.cloud.vcenter.txt (line 2))
02:27 Cloning https://github.com/vmware/vsphere-automation-sdk-python.git to /tmp/pip-req-build-pm27t16b
02:33 Requirement already satisfied: pyvmomi in /usr/local/lib/python3.6/dist-packages (from -r /root/ansible/test/lib/ansible_test/_data/requirements/integration.cloud.vcenter.txt (line 1)) (6.7.1.2018.12)
02:33 Requirement already satisfied: lxml>=4.3.0 in /usr/local/lib/python3.6/dist-packages (from vSphere-Automation-SDK==1.4.0->-r /root/ansible/test/lib/ansible_test/_data/requirements/integration.cloud.vcenter.txt (line 2)) (4.4.0)
02:33 Processing ./\\localhost/tmp/pip-req-build-pm27t16b/lib/vapi-runtime/vapi_runtime-2.12.0-py2.py3-none-any.whl
02:33 Could not install packages due to an EnvironmentError: [Errno 2] No such file or directory: '/root/ansible/\\\\localhost/tmp/pip-req-build-pm27t16b/lib/vapi-runtime/vapi_runtime-2.12.0-py2.py3-none-any.whl'
```
Bump default-test-container to 1.9.3 to get an up to date release of
`pip` (was 19.0.2, is now 19.2.3).
The documentation links are now displayed when running from an install.
Previously the links were only displayed when running from source.
This was due to ansible-test checking for the presence of documentation files locally, which are only present when running from source.
The check is no longer necessary since there is a sanity test in place to enforce the presence of documentation for all sanity tests.
The default behavior of the ansible-test vcenter plugin is to use the govcsim container to run tests.
However, unless the govcsim mode was specified using the VMWARE_TEST_PLATFORM environment variable, the filter code would skip the tests unless the tests ran on Shippable or the user had an ansible-core-ci key.
Now the filter correctly recognizes that govcsim is the default.
* Fix location of unit test requirements.
* Preserve ansible-test unit test requirements.
* Remove redundant unit test requirements.
* Fix location of network test requirements.
* Preserve ansible-test network test requirements.
* Remove redundant network test requirements.
* Add missing ordereddict requirements.
* Load collection requirements correctly.
* Add changelog fragment.
* Avoid assertion rewriting in pytest plugins.
Adding PYTEST_DONT_REWRITE to the ansible-test pytest plugin docstrings disables assertion rewriting in pytest for those plugins.
This avoids warnings during test execution if the plugins are loaded multiple times (such as being imported within tests).
* Run ansible-test pytest plugins early.
The ansible-test pytest plugins need to load and run earlier than conftest modules.
To facilitate this, the pytest_configure function is run during loading, which works since they are loaded (but not always run) before conftest modules are loaded.
A check has also been added to the pytest_configure functions to prevent them from running multiple times in the same process.
* Load pytest plugins using an env var.
The -p command line option loads plugins before conftest, but only during collection.
The PYTEST_PLUGINS environment variable loads plugins before confest, both during collection and test execution.
Creating a virtual environment using `venv` when running in a virtual environment created by `virtualenv` results in a copy of the original virtual environment instead of creation of a new one.
To work around this, `ansible-test` now identifies when it is running in a `virtualenv` created virtual environment and uses the real Python interpreter to create the `venv` virtual environment.
The `test/results/` directory for Ansible test output was already ignored when not using git.
When Ansible Collections were switched to `tests/output/` the ignore entry was previously overlooked.
* bump hcloud version to 1.4.1
`hcloud`<=1.4.0 has requirement `requests==2.20.0`. This prevents the
installation of the Vcenter Automation SDK which depends on `requests>=2.22.0`.
`hcloud` 1.4.1 does not have the problem: 8bff356efb
Bumping the dependency will resolve the issue.
* Fix ansible-doc traceback for removed modules.
This avoids tracebacks with errors like the following when a module has been removed:
module module_name missing documentation (or could not parse documentation): 'NoneType' object does not support item assignment
* Fix ansible-doc sanity test warning handling.
Warnings about removed modules/plugins on stderr are now properly ignored.
Previously an ansible-doc error could result in unrelated errors going undetected because tests were stopped early and the underlying error was ignored.
* Fix ansible-test venv activation.
When using the ansible-test --venv option, an execv wrapper for each python interpreter is now used instead of a symbolic link.
* Fix ansible-test execv wrapper generation.
Use the currently running Python interpreter for the shebang in the execv wrapper instead of the selected interpreter.
This allows the wrapper to work when the selected interpreter is a script instead of a binary.
* Fix ansible-test sanity requirements install.
When running sanity tests on multiple Python versions, install requirements for all versions used instead of only the default version.
* Fix ansible-test --venv when installed.
When running ansible-test from an install, the --venv delegation option needs to make sure the ansible-test code is available in the created virtual environment.
Exposing system site packages does not work because the virtual environment may be for a different Python version than the one on which ansible-test is installed.
* ansible-test - Contiune if the git command returns an error
* Just return stdout
* Use to_text() when displaying exception
* Add a message property to SubprocessError
The `git submodule status` command is relative to the current git repository by default.
When running from a repository subdirectory paths can be returned above the current directory.
Specifying the current directory with `git submodule status` avoids listing submodules above that directory.
This will fix issues when testing a collection that is rooted below the repository root when that repository uses submodules.
* Update collections integration targets path.
* Fix integration path handling.
* Add collections test target fallback.
Also add warnings and errors for common path mistakes.
* Improve role target detection.
This directory is currently a fixed location to make troubleshooting easier.
It is cleared before each test target runs, but is preserved when a test target finishes.
This allows the contents to be inspected when a test fails.
The previous location was `~/ansible_testing/`.
The new location is within the content root:
- `test/results/.tmp/output_dir` for Ansible
- `tests/output/.tmp/output_dir` for Ansible Collections
Moving the directory reduces the number of places on the filesystem where tests create output.
It also enables the results to be returned from delegated systems.
* Clean up layout paths for integration tests.
* Remove "special" integration test target type.
* Remove unnecessary role detection logic.
* Remove support for non-sh runme scripts.
* Simplify reading of aliases.
The directories used for collections tests are changing as follows:
`test/` -> `tests/`
`test/results/` -> `tests/output/`
This is a breaking change for collections tests executed by ansible-test.
All collections will need to be updated to use the new directory.
The `tests/output/` directory should be added to the `.gitignore` or equivalent in each collection.
This change is being made before the first pre-release of Ansible 2.9 since ansible-test has not yet been shipped.
Using the `tests/` directory matches the other collections directories `plugins/` and `roles/`.
This resolves https://github.com/ansible/ansible/issues/60218
