Commit graph

78 commits

Author SHA1 Message Date
Matt Clay
2198ecb1d2
Improve checking for unusable sanity test ignores. (#59833)
* Check sanity ignore paths against test paths.

This prevents ignores from being added for paths which will never be tested by the test being ignored or skipped.

* Fix sanity ignore handling for no/all targets.

This allows checking of ignores for tests which do not use a target list.

It also allows checking of the full ignore list on every test run for tests that always use all targets.
2019-07-30 18:58:37 -07:00
Matt Clay
f61b044bf0 Support directories in sanity tests. 2019-07-30 18:22:09 -07:00
Matt Clay
e07c4f41d7 Standardize ansible-test sanity test filters. 2019-07-30 15:31:33 -07:00
Matt Clay
1351962eee Move "code smell" sanity test config loading. 2019-07-30 12:07:33 -07:00
Matt Clay
5d7cc993dd
Clean up sanity test ignore handling. (#59749)
* Allow sanity tests to easily ignore themselves.

Useful for simple regex based sanity tests that error on their own script due to the regex or error message generated.

* Simplify no-smart-quotes sanity test.

Pruning of unversioned directories is handled by ansible-test.

* Remove directory pruning from boilerplate tests.

Files not passing these tests should be ignored instead.

* Add missing sanity ignore entries.
2019-07-29 14:37:44 -07:00
Matt Clay
66654475e1 Update ansible-test Python version handling.
Minimum version requirements for sanity tests have been standardized:

- All single version sanity tests now require Python 3.5 or later.
- All multiple version sanity tests continue to use all supported Python versions.
- All version neutral sanity tests continue to work on any supported Python version.

Previously some tests required 3.5 or later with most of the remaining tests requiring 2.7 or later.

When using the `--python` option to specify a Python version:

- Tests which do not support the specified Python version will be skipped with a warning.
- If the specified Python version is not available, any test attempting to use it will generate an error.

When not using the `--python` option to specify a Python version:

- Multiple version tests will attempt to run on all supported versions.
- Single version tests will use the current version if supported and available, or if no supported version is available.
- Single version tests will use the lowest available and supported version if the current version is not supported.
- Any versions which are not available or supported will be skipped with a warning.

Unit tests automatically skip unavailable Python versions unless `--python` was used to specify a version.
2019-07-29 11:54:03 -07:00
Matt Clay
2f6de3aea4 Add missing decorator to ansible-test sanity.
This fixes sanity test errors so they show the proper error code instead of "bound method".
2019-07-26 09:04:37 -07:00
Matt Clay
db7e9aa051
Use consolidated ignore.txt in ansible-test. (#59618) 2019-07-25 23:29:23 -07:00
Matt Clay
7c728118e6
Fix import sanity test for collections. (#59484) 2019-07-23 18:10:24 -07:00
Matt Clay
d65d8d6e13 Fix default pylint config for collections. 2019-07-23 17:14:10 -07:00
Matt Clay
44a262714c
Add common sanity ignore & skip to ansible-test. (#59416)
* Add common sanity ignore & skip to ansible-test.
* Switch ansible-doc sanity test to single version.
* Remove duplicate ignore entry.
2019-07-23 13:31:54 -07:00
Matt Clay
79eca9c8fb
Initial ansible-test support for collections. (#59197)
* Initial ansible-test support for collections.
* Include cloud config in delegation payload.
* Add missing types import and fix `t` shadowing.
* Fix plugin traceback when config_path not set.
* Fix encoding issues.
* Remove unused imports.
* More encoding fixes.
* Handle delegation outside exception handler.
* Inject ssh keys only if not already in place.
* More defensive approach to getting remote pwd.
* Add missing string format var.
* Correct PowerShell require regex.
* Rename `is_install` and `INSTALL_ROOT`.
2019-07-22 19:24:48 -07:00
Matt Clay
4f1d0f1b39 Skip using paths for ansible-doc sanity test. 2019-07-19 16:17:56 -07:00
Matt Clay
cb0dfc9793 Prepare pylint test for collections support. 2019-07-16 11:24:16 -07:00
Matt Clay
4255ee8e5a Prepare ansible-test for collections support.
Another round of changes to prepare ansible-test for supporting collections to help keep later PRs a manageable size.
2019-07-15 17:46:29 -07:00
Matt Clay
d910c971b4
Minor ansible-test code cleanup. (#59052)
* Minor ansible-test code cleanup.
* Fix type annotations.
2019-07-12 21:58:19 -07:00
Matt Clay
6645054329
Code cleanup for test infrastructure. (#59046)
* Remove useless object inheritance in test code.
* Remove unnecessary pass statements from test code.
* Comment on why certain pylint rules are ignored.
* Add more pylint test contexts.
* Fix import order.
* Remove unused variables.
* Remove unnecessary pass statement.
* Fix bad continuations.
* Remove unnecessary elif.
* Allow legitimate broad except statements.
* Allow legitimate protected access.
* Clean up names to make pylint pass.
2019-07-12 13:17:20 -07:00
Matt Clay
3f2b766d10 Add future and metaclass boilerplate to test code.
Continue to ignore:

- test/integration/
- test/legacy/
- test/units/
2019-07-12 09:18:24 -07:00
Matt Clay
bc3a599678 Fix invalid type hints in ansible-test. 2019-07-11 14:25:57 -07:00
Matt Clay
86ad4c99ba Reorganize util code in ansible-test.
Code in util.py that depends on CommonConfig is now in util_common.py.
2019-07-11 08:54:13 -07:00
Matt Clay
520af5cb47 Sanity test fixes for future collections support. 2019-07-10 07:03:48 -07:00
Matt Clay
1e1463401d
Prepare ansible-test for supporting collections. (#58886)
This is a small but incomplete set of the initial changes for supporting testing of collections with ansible-test.
2019-07-09 17:31:04 -07:00
Sam Doran
8cfd0ba039 Allow for test groups with double digits (#58532) 2019-06-28 16:24:14 -07:00
Matt Clay
b06f5634c5 Simplify ansible-test target expansion.
Targets are always expanded to full lists now instead of optimizing
for shorter lists by collapsing directories.

This change only affects unit tests and the ansible-doc sanity test,
as they were the only remaining tests using the old behavior.
2019-06-27 08:27:48 -07:00
Matt Clay
535e646a66 Add test for invalid ignore entries. 2019-06-24 13:13:32 -07:00
Matt Clay
817ec5ab48 Add PyYAML check for libyaml support. 2019-06-05 15:20:47 -07:00
Matt Clay
d758d5bab1 Disable code-smell tests on Python 2.6. 2019-06-05 15:20:47 -07:00
Jordan Borean
04cae4134f
Bump deps for ntlm-auth and PSScriptAnalyzer (#55269) 2019-04-16 06:43:41 +10:00
Matt Clay
ade6977d66 Fix typo in doc_fragments path. 2019-03-21 15:36:05 -07:00
Jordan Borean
0bf14d61d0
Add PSCustomUseLiteralPath and update PSSA for pslint (#54080) 2019-03-20 16:26:12 +10:00
Matt Clay
63ce588382 Update pylint and split runs more to avoid hangs. 2019-03-19 13:17:37 -07:00
Matt Martz
07ff2a6494 cliconf and httpapi are documentable plugins now 2019-03-13 10:28:44 -07:00
Matt Clay
a8e328f474
Overhaul ansible-test code coverage and injector. (#53510) 2019-03-13 07:14:12 -07:00
Matt Martz
a639dc6d88
Add a new check to ensure that module arguments are valid python identifiers (#53278)
* Add a new check to ensure that module arguments are valid python identifiers

* Move the check up higher, out of the if docs block

* Skip validate-modules on py2

* Remove errant alias
2019-03-05 10:26:06 -06:00
Matt Clay
06d83bae05 Support ignore_changes in code-smell tests. 2019-02-05 15:21:01 -08:00
Brian Coca
96b3ef5553
Doc fragments to plugins (#50172)
* promote doc_fragments into actual plugins

  change tests hardcoded path to doc fragments
  avoid sanity in fragments
  avoid improper testing of doc_fragments
  also change runner paths
 fix botmeta
 updated comment for fragments
 updated docs
2019-01-23 20:03:47 -05:00
Matt Clay
38eba60849 Fix ansible-test --exclude with delegation.
Previously the option worked with integration commands but not units or sanity.
2018-10-15 14:05:37 -07:00
Matt Clay
76bf861308
Upgrade pylint to version 2.1.1. (#47036)
* Fix issues reported by the latest pylint.
* Split pylint runs into more contexts.
* Upgrade pylint.
2018-10-14 22:59:52 -07:00
Matt Clay
e2b6047514
Add symlinks sanity test. (#46467)
* Add symlinks sanity test.
* Replace legacy test symlinks with actual content.
* Remove dir symlink from template_jinja2_latest.
* Update import test to use generated library dir.
* Fix copy test symlink setup.
2018-10-04 07:02:42 -07:00
Matt Clay
33a8be9109 Add file exists check in integration-aliases test. 2018-10-03 15:40:30 -07:00
Matt Clay
e53390b3b1
Fix ansible-test multi-group smoke test handling. (#46363)
* Fix ansible-test smoke tests across groups.
* Fix ansible-test list arg defaults.
* Fix ansible-test require and exclude delegation.
* Fix detection of Windows specific changes.
* Add minimal Windows testing for Python 3.7.
2018-10-02 12:26:14 -07:00
Matt Clay
ac492476e5
Bug fixes and cleanup for ansible-test. (#45991)
* Remove unused imports.
* Clean up ConfigParser usage in ansible-test.
* Fix bare except statements in ansible-test.
* Miscellaneous cleanup from PyCharm inspections.
* Enable pylint no-self-use for ansible-test.
* Remove obsolete pylint ignores for Python 3.7.
* Fix shellcheck issuers under newer shellcheck.
* Use newer path for ansible-test.
* Fix issues in code-smell tests.
2018-09-21 11:38:22 -07:00
Matt Clay
2056c981ae Improve ansible-test match error handling. 2018-09-21 00:05:33 -07:00
Matt Clay
5a3000af19 Support comments in ansible-test flat files. 2018-09-20 00:48:00 -07:00
Matt Clay
998badbda5 Fix ansible-test virtualenv use in import test. 2018-09-04 10:07:48 -07:00
Matt Clay
4e489d1be8
Update Shippable integration test groups. (#43118)
* Update Shippable integration test groups.
* Update integration test group aliases.
* Rebalance AWS and Azure tests with extra group.
* Rebalance Windows tests with another group.
2018-07-23 20:46:22 -07:00
Matt Clay
dff662fa0f Add plugins to ansible-doc test and fix issues. 2018-05-23 13:10:09 -07:00
Matt Clay
15b6837daf Add yamllint for plugin docs and fix issues. 2018-05-23 09:19:30 -07:00
Matt Clay
a7d7df1450 Make docs-build sanity test disabled by default. 2018-05-09 17:55:00 -07:00
Matt Clay
8a223009ca
Improve handling of integration test aliases. (#38698)
* Include change classification data in metadata.
* Add support for disabled tests.
* Add support for unstable tests.
* Add support for unsupported tests.
* Overhaul integration aliases sanity test.
* Update Shippable scripts to handle unstable tests.
* Mark unstable Azure tests.
* Mark unstable Windows tests.
* Mark disabled tests.
2018-04-12 16:15:28 -07:00