Commit graph

215 commits

Author SHA1 Message Date
Matt Clay a51266ba85 Skip removed windows CI groups after migration. 2020-03-04 12:08:05 -08:00
Matt Clay 3508e1197e
Update test matrix to prepare for migration. (#67983)
* Split fallaxy tests into their own group.

This keeps expected pass/fail tests separate during migration prep.

* Move network tests down in matrix.

This will help keep pass/fail tests grouped together during migration prep.

* Group all incidental tests together.

This will keep pass/fail tests separate leading up to migration.
2020-03-03 16:23:48 -08:00
Matt Clay f4b923a29b Automatically enable test plugins after migration. 2020-03-03 16:23:27 -08:00
Matt Clay 64782dd815
Split unit tests into 3 groups. (#67917) 2020-03-01 22:57:48 -08:00
Matt Clay 42eba3ce25 Increase unit test timeout. 2020-03-01 19:22:56 -08:00
Matt Clay bcddf2bda9 Increase unit test coverage timeout. 2020-03-01 13:59:09 -08:00
Matt Clay 7c8b046b5f
Fourth batch of incidental integration tests. (#67873)
* Copy in incidental posix tests.

* Update incidental test aliases.

* Update target names.

* Add support plugins.

* Fix paths.

* Update ignores.

* Update integration-aliases sanity test.

* Add incidental tests to CI.
2020-02-28 20:37:50 -08:00
Matt Clay f735fd672a
Third batch of incidental integration tests. (#67830)
* Copy in incidental windows tests.

* Update incidental test aliases.

* Add support plugins.

* Update target references.

* Update sanity ignores.

* Update integration-aliases test.

* Add to CI.
2020-02-27 16:05:47 -08:00
Matt Clay e3591223a0
Second batch of incidental integration tests. (#67765)
* Update incidental test aliases.

* Rewrite target references for renamed targets.

* Add incidental tests to CI.

* Update sanity tests for incidental cloud tests.

* Initial copy of incidental tests.

* Copy contrib files into test.

* Update paths in test.

* Add support plugins.

* Update plugin to work around missing deps.

* Update sanity ignores.

* Fix matrix entries.

* Remove debug echo.
2020-02-25 23:18:50 -08:00
Matt Clay 4e571248a9
First batch of incidental integration tests. (#67717)
* Initial copy of incidental network tests.

* Update incidental test aliases.

* Add incidental tests to CI.

* Rewrite module references in tests.

This should not be necessary once module redirection is supported.

* Rewrite target references for renamed targets.

* Add support collections for incidental tests.

* Add ignores for test support code.

* Remove echo used for debugging.
2020-02-24 17:31:14 -08:00
Matt Clay 2dadc259d3
Split hcloud tests into two CI groups. (#67683)
* Split hcloud tests into two CI groups.

This will hopefully resolve CI failures when running with code coverage.

* Fix smoketest logic.
2020-02-23 21:17:03 -08:00
Matt Clay 4a885072af Split vyos tests to avoid timeouts with coverage. 2020-02-23 20:43:05 -08:00
Matt Clay a64b562071
Fix up network testing on Shippable. (#67678)
This prepares for code coverage collection for initial setup of incidental test coverage.
2020-02-23 18:37:26 -08:00
Matt Clay 19f6fc68d4
Relocate Shippable tools. (#67556)
* Move Shippable tools to hacking directory.

These limits the `test/utils/shippable/` directory to scripts required for CI.

* Fix `test/utils/shippable/` file classification.

* Update package-data sanity test.
2020-02-19 09:55:34 -08:00
Matt Clay 80561ccef0 Improve sanity test handling in CI after migration. 2020-02-07 21:29:39 -08:00
Matt Clay a0242b2173 Improve unit test handling in CI after migration. (#67217)
* Improve unit test handling in CI after migration.

* Fix undefined var in bash 4.3 and earlier.

See https://git.savannah.gnu.org/cgit/bash.git/tree/CHANGES?id=3ba697465bc74fab513a26dea700cc82e9f4724e#n878
2020-02-07 21:28:46 -08:00
Matt Clay 5e68bb3d93
Add code coverage target analysis to ansible-test. (#67141)
* Refactor coverage file enumeration.
* Relocate sanitize_filename function.
* Support sets when writing JSON files.
* Generalize setting of info_stderr mode.
* Split out coverage path checking.
* Split out collection regex logic.
* Improve sanitize_filename type hints and docs.
* Clean up coverage erase command.
* Fix docs and type hints for initialize_coverage.
* Update type hints on CoverageConfig.
* Split out logic for finding modules.
* Split out arc enumeration.
* Split out powershell coverage enumeration.
* Raise verbosity level of empty coverage warnings.
* Add code coverage target analysis to ansible-test.
2020-02-05 22:16:15 -08:00
Yanis Guenane 4fd2dce7f3
Testing: Add support for AIX platform (#65802) 2020-01-29 15:56:51 -08:00
Matt Clay 0971a0e7bc Consolidate remote Shippable scripts. 2020-01-28 17:48:54 -08:00
Matt Clay 5c1fe78685
Move CI coverage stub to sanity/5. (#66827) 2020-01-27 16:19:49 -08:00
Matt Clay 452ade007e Clean up Shippable scripts in tools directory. 2020-01-07 14:38:15 -08:00
Matt Clay 78e476eb20 Increase unit test timeout in CI. 2019-11-01 17:16:07 -07:00
Matt Clay d2562cde1c Temporarily increase CI timeout. 2019-11-01 10:10:02 -07:00
Matt Clay ada02f1966 Limit Shippable matrix check to ansible repo. 2019-10-01 09:53:19 -07:00
Gonéri Le Bouder cc758ea574 Revert "shippable: temporary bump the timeout for Azure jobs"
The Azure timeout should not accure anymore since
173d47d1f4.

This reverts commit aaa8835311.
2019-09-09 18:17:25 -07:00
Sam Doran c9e7a2122d
Increase unit test timeout to account for long image download times (#61717) 2019-09-03 12:14:55 -04:00
Gonéri Le Bouder aaa8835311 shippable: temporary bump the timeout for Azure jobs 2019-08-30 18:47:51 -04:00
Matt Clay e4e5005640
Fix ansible-test coverage path handling. (#61528)
* Fix ansible-test coverage path handling.
* Split CI unit tests into two groups.
2019-08-30 00:18:49 -07:00
Sam Doran e5b24a68cc Add additional group for sanity tests
Put validate-modules in its own group
2019-08-29 00:21:07 -07:00
Matt Clay 20d9c34862 Fix CI test result placement. 2019-08-28 09:58:00 -07:00
Matt Clay f5d829392a
Overhaul ansible-test test path handling. (#61416)
* Remove .keep files from test/results/ dirs.

* Remove classification of test/results/ dir.

* Add results_relative to data context.

* Use variables in delegation paths.

* Standardize file writing and results paths.

* Fix issues reported by PyCharm.

* Clean up invocation of coverage command.

It now runs through the injector.

* Hack to allow intercept_command in cover.py.

* Simplify git ignore for test results.

* Use test result tmp dir instead of cache dir.

* Remove old .pytest_cache reference.

* Fix unit test docker delegation.

* Show HTML report link.

* Clean up more results references.

* Move import sanity test output to .tmp dir.

* Exclude test results dir from coverage.

* Fix import sanity test lib paths.

* Fix hard-coded import test paths.

* Fix most hard-coded integration test paths.

* Fix PyCharm warnings.

* Fix import placement.

* Fix integration test dir path.

* Fix Shippable scripts.

* Fix Shippable matrix check.

* Overhaul key pair management.
2019-08-27 23:40:06 -07:00
Jordan Borean faaa669764
PowerShell - Added coverage collector (#59009)
* Added coverage collection for PowerShell - ci_complete ci_coverage

* uncomment out coverage uploader call

* Generate XML for PowerShell coverage

* Use whitelist to exclude coverage run on non content plugins

* Remove uneeded ignore entry

* Try to reduce diff in cover.py

* Fix up coverage report package - ci_complete ci_coverage
2019-08-28 07:03:23 +10:00
Matt Clay 4d424d0830 Only run package-data sanity once in CI. 2019-08-20 14:09:40 -07:00
Jordan Borean 3a3727d200
Remove Python 2.6 from Windows tests (#60235) 2019-08-08 07:32:09 +10:00
Matt Clay b446ca100f
Prepare ansible-test code for relocation. (#60110)
* Fix imports in cli.py.

* Fix imports in executor.py.

* Remove old test/runner/ansible-test entry point.

Use the official bin/ansible-test entry point instead, which has been around since Ansible 2.5.

* Use bin/ansible-test on Shippable.
2019-08-05 14:56:38 -07:00
Matt Clay d3da8e4a5b
Replace old shippable.py with new check_matrix.py. (#60022)
This new script does not depend on ansible-test and provides much more robust job matrix testing.

It is also run on every job in the matrix now, to detect issues with jobs being re-run after matrix changes are made.
2019-08-05 11:27:37 -07:00
Matt Clay f944bd2358 Update ansible-test to properly skip unit tests.
Unit tests will no longer run on "remote only" Python versions (2.6) for tests which are not "remote" (modules and module_utils).
2019-07-31 00:27:07 -07:00
Matt Martz 95f4282c42
Verify package data in setup.py installs all files (#59537)
* Add sanity test to ensure all non-py files are installed

* Fix mode and regex

* Fix role skel inventory package_data

* Add docs

* Update package_data for inventory files

* Address pylint concerns

* Another tweak to package_data

* Address review feedback

* Change index to 1

* add to ansible-only.txt
2019-07-24 13:58:13 -05:00
Matt Clay 13bbfacb04 Show ansible-test --explain errors in CI. 2019-07-17 15:34:00 -07:00
Matt Clay 3f2b766d10 Add future and metaclass boilerplate to test code.
Continue to ignore:

- test/integration/
- test/legacy/
- test/units/
2019-07-12 09:18:24 -07:00
Matt Clay 79685a4a2d
Use Python 3.7 to run ansible-test coverage in CI. (#57555)
Generating XML for coverage reports on Python 3.7 uses far less
memory than on Python 2.7. This change should avoid the MemoryError
we've been seeing in the nightly coverage runs.
2019-06-07 15:48:17 -07:00
Matt Clay b1e6ad9754
Limit Python 2.6 units to modules & module_utils. (#57377) 2019-06-04 15:15:36 -07:00
Matt Clay 85ae8f5258 Add Hetzner Cloud to CI integration tests. 2019-04-09 15:41:50 -07:00
Matt Clay c846501829 Increase unit test timeout for --coverage. 2019-04-05 09:43:42 -07:00
Matt Clay 2a8252ceb9 Slightly increase non-coverage unit test timeout. 2019-03-21 09:30:43 -07:00
Matt Clay 8fa294a33a
Fix ansible-test coverage stub memory usage. (#53652)
Split coverage stub files into multiple parts based on the number
of lines in files included in each stub. This will keep memory usage
down to avoid hanging on Shippable.
2019-03-11 15:14:09 -07:00
Matt Clay 887ab35656
Add support for ansible-test --coverage-check. (#53573) 2019-03-11 15:13:09 -07:00
Matt Clay 9546761407 Fix coverage stub generation in CI. 2019-03-08 16:29:14 -08:00
Matt Clay a8d829d9c3
Add timeout support to ansible-test. (#53302)
* Add timeout support to ansible-test.
* Fix ansible-test tar filename filter bug.
* Update timeouts used on Shippable.
* Kill subprocesses when parent process terminates.
* Require explicit use of env --show option.
2019-03-05 11:58:13 -08:00
Matt Clay b2f354ce29 Ignore errors during container cleanup in CI. 2019-02-21 21:20:48 -08:00