Commit graph

56 commits

Author SHA1 Message Date
Matt Clay
fb0d5609cb
Code cleanup in ansible-test. (#74529)
* Fix get_available_python_versions calls.
* Make run_playbook vars optional.
* Use ansible_pipelining in inventory.
* Fix type hint.
* Fix order of conditional evaluation.
* Remove unused ibmi platform.
* Add changelog fragment.
2021-04-30 18:50:00 -07:00
Matt Clay
c4e76a7f80 Add collection config support to ansible-test. 2021-04-27 12:09:34 -07:00
Matt Clay
6cfe36fa1f
Set ansible-test min controller Python to 3.8. (#74395)
- Version neutral sanity tests now require Python 3.8 or later.
- Unit tests for controller-only code now require Python 3.8 or later.
2021-04-23 00:26:19 -07:00
Matt Clay
7786dbbdc1 Split out module/module_utils unit test execution.
- Unit tests for `modules` and `module_utils` are now limited to importing only `ansible.module_utils` from the `ansible` module.
- Unit tests other than `modules` and `module_utils` are now run only on Python versions supported by the controller (Python 3.8+).
- Unit tests are now run in separate contexts (`controller`, `modules`, `module_utils`), each using separate invocations of `pytest`.
2021-04-19 15:44:58 -07:00
Matt Clay
b752d07163 Overhaul ansible-test container management.
This brings ansible-test closer to being able to support split controller/remote testing.
2021-04-12 12:40:36 -07:00
Felix Fontein
29aef842d7
ansible-test: make sure tests are also run for the new plugin types (#73599)
* Make sure tests are also run for the new plugin types.
* New plugin types are not documentable.
2021-02-13 14:05:16 -08:00
Felix Fontein
1f3a90270b
Import sanity test for plugins (#72497) 2021-02-12 14:09:50 -08:00
Matt Clay
f533d46572
Fix ansible-test constraints issues. (#73578)
* Silence Python 3.5 EOL notice in pip.
* Skip import test on compat/selinux.py utils.
* Improve Python version skip warning.
* Use Python 3.6 as minimum Python for sanity tests.
* Improve min Python handling for code-smell tests.
* Overhaul test-constraints sanity test.
* Merge sanity test constraints with requirements.
* Remove legacy content specific constraints.
* Add changelog fragment.
2021-02-11 16:45:53 -08:00
Matt Clay
8a175f59c9
Upgrade pylint used by ansible-test. (#70155)
* Upgrade pylint and deps in ansible-test.
* Enable pylint on Python 3.9.
* Update pylint config.
* Add ignore for vendored six.
* Add ignores for support plugins.
* Fix issue reported by pylint.
2021-02-10 15:52:41 -08:00
Matt Clay
1b157ef42f
Limit rstcheck sanity test to ansible-core. (#73391) 2021-01-27 21:06:00 -08:00
Matt Clay
73fadc5e97
Clean up code in ansible-test. (#73379)
* Relocate code to fix type dependencies.

* Fix missing and unused imports.

* Fix type hints.

* Suppress PyCharm false positives.

* Avoid shadowing `file` built-in.

* Use json.JSONEncoder directly instead of super().

This matches the recommended usage and avoids a PyCharm warning.

* Remove redundant regex escape.

* Remove redundant find_python call.

* Use tarfile.open directly.

* Add changelog fragment.
2021-01-26 14:02:08 -08:00
Matt Clay
37d09f2488
Update ansible-test pylint Python support. (#72972)
* Update ansible-test pylint Python support.

* Python 3.8 is now officially supported.
* Python 3.9 is now skipped with a warning.
2020-12-15 10:27:32 -08:00
Matt Clay
221c50b57c Use AZP config for integration-aliases test.
No changelog entry since this test is limited to the ansible/ansible repo.
2020-12-03 15:35:58 -08:00
Matt Clay
8d92df4537 Test netconf plugins in ansible-doc sanity test. 2020-06-24 15:53:32 -07:00
Matt Martz
4a4a11d282
Add mccabe complexity testing (#64623)
* Add mccabe complexity testing

* Make mccabe complexity an optional error

* Add mccabe to new sanity pylint requirements

* Add a changelog fragment.

Co-authored-by: Matt Clay <matt@mystile.com>
2020-06-11 13:02:06 -07:00
Felix Fontein
a114da80ee
Use antsibull-changelog instead of packaged changelog generator (#69313)
Replace the ansible-base changelog linting and generation tool with antsibull-changelog and make it available for linting collections. Previously changelog linting was limited to ansible-base.
2020-06-11 11:21:21 -07:00
Matt Clay
6c2be04b6f Intercept code-smell tests using subprocess.
Some code-smell sanity tests for ansible-base use subprocess to invoke ansible commands.
Intercept these commands to make sure the correct script and python version are used.
2020-06-10 16:07:47 -07:00
Matt Clay
a5679d4dd1
Additional ansible-test sanity adjustments. (#69974)
* Fix ansible-test check_pyyaml usage.
* Avoid unnecessary pip commands.
* Avoid pointless pip check.
* Only install ansible-test requirements as needed.
2020-06-10 09:39:25 -07:00
Matt Clay
7bff3d312f
Split out and install sanity test requirements. (#69971)
* Split out sanity test requirements.

* Run each --venv test separately.

This provides verification that the requirements for each test are properly specified.

* Use a separate requirements file per sanity test.

* Skip setuptools/cryptography setup for sanity.

* Eliminate pyyaml missing warning.

* Eliminate more pip noise.

* Fix conflicting generate_pip_install commands.

* Add changelog fragment.
2020-06-09 13:38:36 -07:00
Felix Fontein
0b82d4499e
ansible-test ansible-doc sanity test: also run with --json parameter (#69288) 2020-05-29 08:52:29 -07:00
Felix Fontein
40f21dfd3c
Version source tagging (automatic and manual) for version_added and deprecation versions (#69680)
* Track collection for version_added.
Validate *all* version numbers in validate-modules.
For tagged version numbers (i.e. version_added), consider source collection to chose validation.

* Make tagging/untagging functions more flexible.

* Tag all versions in doc fragments.

* Tag all deprecation versions issued by code.

* Make Display.deprecated() understand tagged versions.

* Extend validation to enforce tagged version numbers.

* Tag versions in tests.

* Lint and fix test.

* Mention collection name in collection loader's deprecation/removal messages.

* Fix error IDs.

* Handle tagged dates in Display.deprecated().

* Also require that removed_at_date and deprecated_aliases.date are tagged.

* Also automatically tag/untag removed_at_date; fix sanity module removal version check.

* Improve error message when invalid version number is used (like '2.14' in collections).
2020-05-28 22:46:16 -07:00
Jordan Borean
f5f3ba7ab5
ansible-test - fix up relative util import for powershell validate-modules (#69753)
* ansible-test - fix up relative util import for powershell validate-modules

* Use different tactic for generic group

* Use python 2 and 3
2020-05-29 14:31:59 +10:00
Felix Fontein
ea04e0048d
Allow to deprecate options and aliases by date (#68177)
* Allow to deprecate options and aliases by date instead of only by version.

* Update display.deprecate().

* Adjust behavior to conform to tested behavior, extend tests, and improve C# style.

* Parse date and fail on invalid date.

This is mainly to make sure that people start using invalid dates, and we eventually have a mess to clean up.

* C# code: improve validation and update/extend tests.

* Make sure that deprecate() is not called with both date and version.

* Forgot to remove no longer necessary formatting.

* Adjust order of warnings in C# code.

* Adjust unrelated test.

* Fix grammar (and make that test pass).

* Don't parse date, and adjust message to be same as in #67684.

* Sanity tests: disable date in past test.

* Validate-modules: validate ISO 8601 date format.

* Validate-modules: switch schema declaration for deprecated_aliases to improve error messages for invalid dates.

* Use DateTime instead of string for date deprecation.

* Validate that date in deprecated_aliases is actually a DateTime.

* Fix tests.

* Fix rebasing error.

* Adjust error codes for pylint, and add removed_at_date and deprecated_aliases.date checks to validate-modules.

* Make deprecation date in the past error codes optional.

* Make sure not both version and date are specified for AnsibleModule.deprecate() calls.

* Stop using Python 3.7+ API.

* Make sure errors are actually reported. Re-add 'ansible-' prefix.

* Avoid crashing when 'name' isn't there.

* Linting.

* Update lib/ansible/module_utils/csharp/Ansible.Basic.cs

Co-authored-by: Jordan Borean <jborean93@gmail.com>

* Adjust test to latest change.

* Prefer date over version if both end up in Display.deprecated().

Co-authored-by: Jordan Borean <jborean93@gmail.com>
2020-05-26 20:23:56 -04:00
Matt Davis
f7dfa817ae
collection routing (#67684)
* `meta/` directory in collections
* runtime metadata for redirection/deprecation/removal of plugin loads
* a compatibility layer to keep existing content working on ansible-base + collections
* a Python import redirection layer to keep collections-hosted (and otherwise moved) content importable by things that don't know better
* supported Ansible version validation on collection loads
2020-05-26 09:42:06 -07:00
Matt Clay
c495c92a6e Code cleanup in ansible-test. 2020-05-15 18:17:11 -07:00
Matt Clay
d8e0aadc0d Update ansible-test support for CI providers.
Refactored CI provider code to simplify multiple provider support and addition of new providers.
2020-05-14 17:08:30 -07:00
Felix Fontein
0e15375ffe
Add deprecated removed_in_version and deprecated_aliases version tests (#66920) 2020-05-13 13:58:09 -07:00
Matt Clay
c06f442409 Fix ansible-test import sanity test venv setup. 2020-03-01 13:59:24 -08:00
Matt Clay
7c8b046b5f
Fourth batch of incidental integration tests. (#67873)
* Copy in incidental posix tests.

* Update incidental test aliases.

* Update target names.

* Add support plugins.

* Fix paths.

* Update ignores.

* Update integration-aliases sanity test.

* Add incidental tests to CI.
2020-02-28 20:37:50 -08:00
Matt Clay
f735fd672a
Third batch of incidental integration tests. (#67830)
* Copy in incidental windows tests.

* Update incidental test aliases.

* Add support plugins.

* Update target references.

* Update sanity ignores.

* Update integration-aliases test.

* Add to CI.
2020-02-27 16:05:47 -08:00
Matt Clay
e3591223a0
Second batch of incidental integration tests. (#67765)
* Update incidental test aliases.

* Rewrite target references for renamed targets.

* Add incidental tests to CI.

* Update sanity tests for incidental cloud tests.

* Initial copy of incidental tests.

* Copy contrib files into test.

* Update paths in test.

* Add support plugins.

* Update plugin to work around missing deps.

* Update sanity ignores.

* Fix matrix entries.

* Remove debug echo.
2020-02-25 23:18:50 -08:00
Matt Clay
f4a80bb600
Code cleanup and refactoring in ansible-test. (#67063)
* Code cleanup in ansible-test.
* Split out encoding functions.
* Consoldate loading of JSON files.
* Split out disk IO functions.
* Simplify file access.
* Add functions for opening files.
* Replace open calls with appropriate functions.
* Expose more types from typing module.
* Support writing compact JSON.
* Add verbosity argument to display.warning.
* Add changelog entry.
* Update files overlooked during rebase.
* Use `io.open` instead of `open`.
* Fix file opening for imp.load_module.
* Remove use of `r+` mode to access files.
* Add missing import.
* Fix httptester on Python 2.x.
* Clarify changelog fragment.
* Consolidate imports. Remove extra newlines.
* Fix indirect imports.
2020-02-04 11:21:53 -08:00
Toshio Kuratomi
2b6d94c4c8 Change the check_pyyaml() function to return the features of pyyaml that we check 2019-12-20 10:29:03 -08:00
Toshio Kuratomi
ece306b201 Add a sanity test for yaml to make sure we can parse all yaml
Check that all yaml we ship is parsable by the pyyaml c backend.  Since
Ansible uses Pyyaml for docs and playbooks, if the yaml files aren't
parsable, they will error out if they were used.

Warn and skip yamllint if libyaml backend is not present

Ignore new errors in examples until someone can fix them
2019-12-20 10:29:03 -08:00
Matt Clay
fb69d68821
Fix plugin paths for ansible-test pylint test. (#65526)
* Fix plugin paths for ansible-test pylint test.
2019-12-04 18:07:16 -08:00
Matt Clay
e9f8a34dce
Fixes for validate-modules import handling. (#63932)
* Fix validate-modules support for collections.

- Relative imports now work correctly.
- The collection loader is now used.
- Modules are invoked as `__main__`.

* Remove obsolete validate-modules code ignores.

* Handle sys.exit in validate-modules.

* Add check for AnsibleModule initialization.

* Remove `missing-module-utils-import` check.

This check does not support relative imports or collections.

Instead of trying to overhaul the test, we can rely on the `ansible-module-not-initialized` test instead.

* Fix badly named error codes with `c#` in the name.

The `#` conflicts with comments in the sanity test ignore files.

* Add changelog entries.
2019-10-30 09:48:21 -07:00
Matt Clay
92ccdeac31 Fix various import sanity test issues.
- Relative imports are now properly recognized.
- Correct script invocation of Ansible modules is used.
- Warnings are now consistently reported as errors.
- Errors are now consistently reported with the file tested.

Resolves https://github.com/ansible/ansible/issues/62723
Resolves https://github.com/ansible/ansible/issues/61884
2019-10-22 23:00:35 -07:00
Matt Clay
064e8e1ef4
Fix ansible-doc traceback and sanity test. (#62040)
* Fix ansible-doc traceback for removed modules.

This avoids tracebacks with errors like the following when a module has been removed:

module module_name missing documentation (or could not parse documentation): 'NoneType' object does not support item assignment

* Fix ansible-doc sanity test warning handling.

Warnings about removed modules/plugins on stderr are now properly ignored.

Previously an ansible-doc error could result in unrelated errors going undetected because tests were stopped early and the underlying error was ignored.
2019-09-10 00:29:34 -07:00
Matt Clay
c77ab11051
Fix issues with ansible-test --venv option. (#62033)
* Fix ansible-test venv activation.

When using the ansible-test --venv option, an execv wrapper for each python interpreter is now used instead of a symbolic link.

* Fix ansible-test execv wrapper generation.

Use the currently running Python interpreter for the shebang in the execv wrapper instead of the selected interpreter.

This allows the wrapper to work when the selected interpreter is a script instead of a binary.

* Fix ansible-test sanity requirements install.

When running sanity tests on multiple Python versions, install requirements for all versions used instead of only the default version.

* Fix ansible-test --venv when installed.

When running ansible-test from an install, the --venv delegation option needs to make sure the ansible-test code is available in the created virtual environment.

Exposing system site packages does not work because the virtual environment may be for a different Python version than the one on which ansible-test is installed.
2019-09-09 18:32:29 -07:00
Matt Clay
f90587069c
Update collections integration targets path. (#61523)
* Update collections integration targets path.

* Fix integration path handling.

* Add collections test target fallback.

Also add warnings and errors for common path mistakes.

* Improve role target detection.
2019-08-29 03:39:03 -07:00
Matt Clay
4afe757f93
More ansible-test path handling updates. (#61484)
* Move test results vars into layout.

* Add test_path to layout.

* Add sanity_path to layout.

* Clean up layout.

* Update change classification.

* Improve classification layout.

* Relocate common classification code.

* Use is_subdir for relocated code.

* Relocate ansible test/units/compat/ hack.

* Fix variable shadowing.

* Remove unused code.

* Fix ordering of path tests.

* Clean up ansible classification.

* Fix hard-coded plugin paths in classification.

* Relocate more common classification.

* Fix logic.

* Fix pylint issue.

* Add missing licenses classification.
2019-08-28 15:13:00 -07:00
Sam Doran
73248bf27d validate-modules - Use error codes instead of numbers (#60711) 2019-08-28 14:12:56 -07:00
Matt Clay
830f995ed4
Add a --venv option to ansible-test. (#61422)
* Add --venv delegation to ansible-test.

* Update import sanity test venv creation.

* Fix import test when using --venv on Python 2.x.

* Improve virtualenv setup overhead.

* Hide pip noise for import sanity test.

* Raise verbosity on venv info messages.

* Get rid of base branch noise for collections.

* Add missing --requirements check.
2019-08-28 09:10:17 -07:00
Matt Clay
f5d829392a
Overhaul ansible-test test path handling. (#61416)
* Remove .keep files from test/results/ dirs.

* Remove classification of test/results/ dir.

* Add results_relative to data context.

* Use variables in delegation paths.

* Standardize file writing and results paths.

* Fix issues reported by PyCharm.

* Clean up invocation of coverage command.

It now runs through the injector.

* Hack to allow intercept_command in cover.py.

* Simplify git ignore for test results.

* Use test result tmp dir instead of cache dir.

* Remove old .pytest_cache reference.

* Fix unit test docker delegation.

* Show HTML report link.

* Clean up more results references.

* Move import sanity test output to .tmp dir.

* Exclude test results dir from coverage.

* Fix import sanity test lib paths.

* Fix hard-coded import test paths.

* Fix most hard-coded integration test paths.

* Fix PyCharm warnings.

* Fix import placement.

* Fix integration test dir path.

* Fix Shippable scripts.

* Fix Shippable matrix check.

* Overhaul key pair management.
2019-08-27 23:40:06 -07:00
Matt Clay
e258ba703e
Improve ansible-test env display. (#61339)
* Improve ansible-test env display.
* Fix PyCharm warnings.
* Fix ansible-test ansible-doc sanity test with -e.
2019-08-26 14:02:55 -07:00
Matt Clay
2a9b996bd2 Use .cfg extension for pylint configs. 2019-08-24 07:25:19 -07:00
Matt Clay
6ec0b4ec86
Sanity test updates for collections support. (#61248)
* Run no-unwanted-files sanity test only on Ansible.

Since collections should be able to use binary modules there is not really any limit on what could exist in a collection `plugins` directory.

* Add support for symlinks in sanity target lists:

- Sanity tests that need to analyze symlinks can do so using the supplied target list.
- Tests that analyze directories will now only look at symlinks if requested.
- Directory symlinks will now be seen as directories instead of files.

* Enable symlinks on filename based sanity tests.

Sanity tests that evalulate filenames instead of content should include symlinks.

* Update symlinks sanity test.

Use the sanity test target list now that it can include symlinks.
2019-08-23 18:08:21 -07:00
Matt Martz
def3d1f815
validate-modules: support collections (#60247)
* Start of work to support collections

* remove version_added from base schema

* If a collection, pass that to validate-modules

* clean ups

* Allow version_added in a collection, just make it optional

* Don't traceback on missing doc_fragment

* Don't validate metadata in a collection
2019-08-16 14:28:34 -05:00
Matt Clay
ba1b4565d3 Clean up ansible-test results path handling.
This brings us closer to being able to use `tests/` instead of `test/` in collections.
2019-08-13 21:51:57 -07:00
Matt Clay
2a9964ede8
Code cleanup in ansible-test. (#60318)
* Move ansible-test units code into separate dir.
* Fix --explain on "code smell" sanity tests.
* Add --strict-markers to pytest invoctaion.
* Fix classification of units code changes.
2019-08-09 09:38:19 -07:00