Commit graph

26 commits

Author SHA1 Message Date
Matt Clay
e3591223a0
Second batch of incidental integration tests. (#67765)
* Update incidental test aliases.

* Rewrite target references for renamed targets.

* Add incidental tests to CI.

* Update sanity tests for incidental cloud tests.

* Initial copy of incidental tests.

* Copy contrib files into test.

* Update paths in test.

* Add support plugins.

* Update plugin to work around missing deps.

* Update sanity ignores.

* Fix matrix entries.

* Remove debug echo.
2020-02-25 23:18:50 -08:00
Matt Clay
f4a80bb600
Code cleanup and refactoring in ansible-test. (#67063)
* Code cleanup in ansible-test.
* Split out encoding functions.
* Consoldate loading of JSON files.
* Split out disk IO functions.
* Simplify file access.
* Add functions for opening files.
* Replace open calls with appropriate functions.
* Expose more types from typing module.
* Support writing compact JSON.
* Add verbosity argument to display.warning.
* Add changelog entry.
* Update files overlooked during rebase.
* Use `io.open` instead of `open`.
* Fix file opening for imp.load_module.
* Remove use of `r+` mode to access files.
* Add missing import.
* Fix httptester on Python 2.x.
* Clarify changelog fragment.
* Consolidate imports. Remove extra newlines.
* Fix indirect imports.
2020-02-04 11:21:53 -08:00
Toshio Kuratomi
2b6d94c4c8 Change the check_pyyaml() function to return the features of pyyaml that we check 2019-12-20 10:29:03 -08:00
Toshio Kuratomi
ece306b201 Add a sanity test for yaml to make sure we can parse all yaml
Check that all yaml we ship is parsable by the pyyaml c backend.  Since
Ansible uses Pyyaml for docs and playbooks, if the yaml files aren't
parsable, they will error out if they were used.

Warn and skip yamllint if libyaml backend is not present

Ignore new errors in examples until someone can fix them
2019-12-20 10:29:03 -08:00
Matt Clay
fb69d68821
Fix plugin paths for ansible-test pylint test. (#65526)
* Fix plugin paths for ansible-test pylint test.
2019-12-04 18:07:16 -08:00
Matt Clay
e9f8a34dce
Fixes for validate-modules import handling. (#63932)
* Fix validate-modules support for collections.

- Relative imports now work correctly.
- The collection loader is now used.
- Modules are invoked as `__main__`.

* Remove obsolete validate-modules code ignores.

* Handle sys.exit in validate-modules.

* Add check for AnsibleModule initialization.

* Remove `missing-module-utils-import` check.

This check does not support relative imports or collections.

Instead of trying to overhaul the test, we can rely on the `ansible-module-not-initialized` test instead.

* Fix badly named error codes with `c#` in the name.

The `#` conflicts with comments in the sanity test ignore files.

* Add changelog entries.
2019-10-30 09:48:21 -07:00
Matt Clay
92ccdeac31 Fix various import sanity test issues.
- Relative imports are now properly recognized.
- Correct script invocation of Ansible modules is used.
- Warnings are now consistently reported as errors.
- Errors are now consistently reported with the file tested.

Resolves https://github.com/ansible/ansible/issues/62723
Resolves https://github.com/ansible/ansible/issues/61884
2019-10-22 23:00:35 -07:00
Matt Clay
064e8e1ef4
Fix ansible-doc traceback and sanity test. (#62040)
* Fix ansible-doc traceback for removed modules.

This avoids tracebacks with errors like the following when a module has been removed:

module module_name missing documentation (or could not parse documentation): 'NoneType' object does not support item assignment

* Fix ansible-doc sanity test warning handling.

Warnings about removed modules/plugins on stderr are now properly ignored.

Previously an ansible-doc error could result in unrelated errors going undetected because tests were stopped early and the underlying error was ignored.
2019-09-10 00:29:34 -07:00
Matt Clay
c77ab11051
Fix issues with ansible-test --venv option. (#62033)
* Fix ansible-test venv activation.

When using the ansible-test --venv option, an execv wrapper for each python interpreter is now used instead of a symbolic link.

* Fix ansible-test execv wrapper generation.

Use the currently running Python interpreter for the shebang in the execv wrapper instead of the selected interpreter.

This allows the wrapper to work when the selected interpreter is a script instead of a binary.

* Fix ansible-test sanity requirements install.

When running sanity tests on multiple Python versions, install requirements for all versions used instead of only the default version.

* Fix ansible-test --venv when installed.

When running ansible-test from an install, the --venv delegation option needs to make sure the ansible-test code is available in the created virtual environment.

Exposing system site packages does not work because the virtual environment may be for a different Python version than the one on which ansible-test is installed.
2019-09-09 18:32:29 -07:00
Matt Clay
f90587069c
Update collections integration targets path. (#61523)
* Update collections integration targets path.

* Fix integration path handling.

* Add collections test target fallback.

Also add warnings and errors for common path mistakes.

* Improve role target detection.
2019-08-29 03:39:03 -07:00
Matt Clay
4afe757f93
More ansible-test path handling updates. (#61484)
* Move test results vars into layout.

* Add test_path to layout.

* Add sanity_path to layout.

* Clean up layout.

* Update change classification.

* Improve classification layout.

* Relocate common classification code.

* Use is_subdir for relocated code.

* Relocate ansible test/units/compat/ hack.

* Fix variable shadowing.

* Remove unused code.

* Fix ordering of path tests.

* Clean up ansible classification.

* Fix hard-coded plugin paths in classification.

* Relocate more common classification.

* Fix logic.

* Fix pylint issue.

* Add missing licenses classification.
2019-08-28 15:13:00 -07:00
Sam Doran
73248bf27d validate-modules - Use error codes instead of numbers (#60711) 2019-08-28 14:12:56 -07:00
Matt Clay
830f995ed4
Add a --venv option to ansible-test. (#61422)
* Add --venv delegation to ansible-test.

* Update import sanity test venv creation.

* Fix import test when using --venv on Python 2.x.

* Improve virtualenv setup overhead.

* Hide pip noise for import sanity test.

* Raise verbosity on venv info messages.

* Get rid of base branch noise for collections.

* Add missing --requirements check.
2019-08-28 09:10:17 -07:00
Matt Clay
f5d829392a
Overhaul ansible-test test path handling. (#61416)
* Remove .keep files from test/results/ dirs.

* Remove classification of test/results/ dir.

* Add results_relative to data context.

* Use variables in delegation paths.

* Standardize file writing and results paths.

* Fix issues reported by PyCharm.

* Clean up invocation of coverage command.

It now runs through the injector.

* Hack to allow intercept_command in cover.py.

* Simplify git ignore for test results.

* Use test result tmp dir instead of cache dir.

* Remove old .pytest_cache reference.

* Fix unit test docker delegation.

* Show HTML report link.

* Clean up more results references.

* Move import sanity test output to .tmp dir.

* Exclude test results dir from coverage.

* Fix import sanity test lib paths.

* Fix hard-coded import test paths.

* Fix most hard-coded integration test paths.

* Fix PyCharm warnings.

* Fix import placement.

* Fix integration test dir path.

* Fix Shippable scripts.

* Fix Shippable matrix check.

* Overhaul key pair management.
2019-08-27 23:40:06 -07:00
Matt Clay
e258ba703e
Improve ansible-test env display. (#61339)
* Improve ansible-test env display.
* Fix PyCharm warnings.
* Fix ansible-test ansible-doc sanity test with -e.
2019-08-26 14:02:55 -07:00
Matt Clay
2a9b996bd2 Use .cfg extension for pylint configs. 2019-08-24 07:25:19 -07:00
Matt Clay
6ec0b4ec86
Sanity test updates for collections support. (#61248)
* Run no-unwanted-files sanity test only on Ansible.

Since collections should be able to use binary modules there is not really any limit on what could exist in a collection `plugins` directory.

* Add support for symlinks in sanity target lists:

- Sanity tests that need to analyze symlinks can do so using the supplied target list.
- Tests that analyze directories will now only look at symlinks if requested.
- Directory symlinks will now be seen as directories instead of files.

* Enable symlinks on filename based sanity tests.

Sanity tests that evalulate filenames instead of content should include symlinks.

* Update symlinks sanity test.

Use the sanity test target list now that it can include symlinks.
2019-08-23 18:08:21 -07:00
Matt Martz
def3d1f815
validate-modules: support collections (#60247)
* Start of work to support collections

* remove version_added from base schema

* If a collection, pass that to validate-modules

* clean ups

* Allow version_added in a collection, just make it optional

* Don't traceback on missing doc_fragment

* Don't validate metadata in a collection
2019-08-16 14:28:34 -05:00
Matt Clay
ba1b4565d3 Clean up ansible-test results path handling.
This brings us closer to being able to use `tests/` instead of `test/` in collections.
2019-08-13 21:51:57 -07:00
Matt Clay
2a9964ede8
Code cleanup in ansible-test. (#60318)
* Move ansible-test units code into separate dir.
* Fix --explain on "code smell" sanity tests.
* Add --strict-markers to pytest invoctaion.
* Fix classification of units code changes.
2019-08-09 09:38:19 -07:00
Matt Clay
df59f06d6a
Additional ansible-test path fixes. (#60300)
* Add missing ansible-test injector symlink.
* More ansible-test path fixes.
* Expand bin-symlinks docs.
2019-08-08 23:51:03 -07:00
Matt Clay
39b3fc0926 Update ansible-test layout and payload handling. 2019-08-08 21:34:38 -07:00
Matt Clay
57dc7ec265
Prepare ansible-test for inclusion in setup.py (#60294)
* Cache ansible version lookup.
* Fix method of determining ANSIBLE_ROOT.
* Clean up based on PyCharm inspections.
* Generate minimal PKG-INFO without setup.py.
* Use ANSIBLE_LIB_ROOT where possible.
* Use import instead of subprocess to get version.
* Fix install layout type.
* Correct required paths message for installs.
* Update list of files copied during delegation.
* Fix ansible-test entry point.
* Fix pylint issue.
* Fix version lookup on Python 2.x.
* Fix pylint issue.
* Remove unwanted print statement.
2019-08-08 16:14:19 -07:00
Matt Clay
a93a0c8fd6
Relocate validate-modules for easier testing. (#60214)
* Relocate module validator code and tests.
* Fix validate-modules entry point and imports.
* Fix paths and test entry points.
* Fix up unit tests.
* Fix shebang and execute bit.
2019-08-07 09:34:11 -07:00
Matt Clay
18867847f4
Split up "code smell" sanity tests. (#60174)
* Relocate ansible-only sanity tests.

* Get "code smell" sanity tests from multiple dirs.

- `test/lib/ansible_test/_data/sanity/code-smell/` - General purpose tests used for both Ansible and Ansible Collections.
- `test/sanity/code-smell/` - Tests specific to Ansible, will not be used for Ansible Collections.
2019-08-06 17:17:57 -07:00
Matt Clay
d651bda123
Relocate ansible-test code. (#60147)
* Initial move of `test/runner/` content.

`test/runner/lib/` -> `test/lib/ansible_test/_internal/`
`test/runner/`     -> `test/lib/ansible_test/_internal/data/`

* Initial move of `test/sanity/` content.

`test/sanity/` -> `test/lib/ansible_test/_internal/data/sanity/` (except `test/sanity/ignore.txt`)

* Initial move of `test/units/pytest/` content.

`test/units/pytest/` -> `test/lib/ansible_test/_internal/data/pytest/`

* Follow-up move of `test/runner/unit/` content.

`test/lib/ansible_test/_internal/data/unit/` -> `test/lib/ansible_test/tests/unit/`

* Initial move of `ansible.cfg` content.

`test/units/ansible.cfg` -> `test/lib/ansible_test/_internal/data/units/ansible.cfg`
`test/env/ansible.cfg` -> `test/lib/ansible_test/_internal/data/env/ansible.cfg`

* Follow-up move of `data` directory.

`test/lib/ansible_test/_internal/data/` -> `test/lib/ansible_test/_data/`

* Update import statements.

* Add missing __init__.py for unit tests.

* Fix path references and miscellaneous issues.
2019-08-06 14:43:29 -07:00