Commit graph

227 commits

Author SHA1 Message Date
Matt Clay
13bbfacb04 Show ansible-test --explain errors in CI. 2019-07-17 15:34:00 -07:00
Matt Clay
3f2b766d10 Add future and metaclass boilerplate to test code.
Continue to ignore:

- test/integration/
- test/legacy/
- test/units/
2019-07-12 09:18:24 -07:00
Matt Clay
79685a4a2d
Use Python 3.7 to run ansible-test coverage in CI. (#57555)
Generating XML for coverage reports on Python 3.7 uses far less
memory than on Python 2.7. This change should avoid the MemoryError
we've been seeing in the nightly coverage runs.
2019-06-07 15:48:17 -07:00
Matt Clay
b1e6ad9754
Limit Python 2.6 units to modules & module_utils. (#57377) 2019-06-04 15:15:36 -07:00
Matt Clay
85ae8f5258 Add Hetzner Cloud to CI integration tests. 2019-04-09 15:41:50 -07:00
Matt Clay
c846501829 Increase unit test timeout for --coverage. 2019-04-05 09:43:42 -07:00
Matt Clay
2a8252ceb9 Slightly increase non-coverage unit test timeout. 2019-03-21 09:30:43 -07:00
Matt Clay
8fa294a33a
Fix ansible-test coverage stub memory usage. (#53652)
Split coverage stub files into multiple parts based on the number
of lines in files included in each stub. This will keep memory usage
down to avoid hanging on Shippable.
2019-03-11 15:14:09 -07:00
Matt Clay
887ab35656
Add support for ansible-test --coverage-check. (#53573) 2019-03-11 15:13:09 -07:00
Matt Clay
9546761407 Fix coverage stub generation in CI. 2019-03-08 16:29:14 -08:00
Matt Clay
a8d829d9c3
Add timeout support to ansible-test. (#53302)
* Add timeout support to ansible-test.
* Fix ansible-test tar filename filter bug.
* Update timeouts used on Shippable.
* Kill subprocesses when parent process terminates.
* Require explicit use of env --show option.
2019-03-05 11:58:13 -08:00
Matt Clay
b2f354ce29 Ignore errors during container cleanup in CI. 2019-02-21 21:20:48 -08:00
Matt Clay
a1fec259c1
Reduce console noise at start of CI session. (#52387)
* Reduce console noise at start of CI session.
* Use allowed shebang.
2019-02-18 13:41:45 -08:00
Matt Clay
6a82827ddd Add Python 3.8 to ansible-test and CI. 2019-02-14 23:24:24 -08:00
Matt Clay
d2a9b16f7e Make Shippable timing script more resilient.
This will permit the script to pass through content which could
not be previously decoded or encoded. This could occur when running
some tests on macOS using file paths with non-ASCII characters.
2019-01-17 00:03:38 -08:00
John R Barker
721a28202e
Revert Debug: Size of coverage file #50380 (#50945) 2019-01-15 20:36:12 +00:00
Matt Clay
31c1bdd6a8 Fix run.py --branch argument.
The API docs state that both `branch` and `branchName` are valid,
but only `branchName` appears to work.
2019-01-07 13:56:35 -08:00
John R Barker
e6ced9c73f
Debug: Size of coverage file (#50380) 2018-12-29 13:01:35 +00:00
John R Barker
7ec82cdfc9
Revert TEMP HACK: Upload all to codecov (#50279)
Codecov.io have made some improvements to results processing.
Upload all the coverage data to see is codecov can process this.

Reverts https://github.com/ansible/ansible/pull/47515
2018-12-23 09:42:29 +00:00
Matt Clay
01833b6fb1
Add env command to ansible-test and run in CI. (#50176)
* Add `env` command to ansible-test and run in CI.
* Avoid unnecessary docker pull.
2018-12-20 22:08:57 -08:00
Matt Clay
9979a32e5c Fix passing of env vars to Shippable. 2018-11-14 10:50:31 -08:00
John R Barker
fb2e838a42
TEMP HACK: Avoid overloading codecov.io (#47515)
Without this patch we are overloading codecov.io by uploading 90+ tests.

As a workaround limit uploading to only "Group 1"

Will be removed/updated based on codecov.io's support team
2018-10-23 17:32:34 +01:00
Matt Clay
6a51b5ed26 Use 4 nodes for sanity tests in CI. 2018-10-16 09:55:26 -07:00
Matt Clay
2761fe8272 Revert "Use 4 nodes for sanity tests in CI."
This reverts commit b887333029.
2018-10-15 12:11:17 -07:00
Matt Clay
b887333029 Use 4 nodes for sanity tests in CI. 2018-10-15 12:03:43 -07:00
Jordan Borean
25c627d256
shippable: fix py3 issues with downloader (#46522) 2018-10-05 18:21:59 +10:00
Matt Clay
e53390b3b1
Fix ansible-test multi-group smoke test handling. (#46363)
* Fix ansible-test smoke tests across groups.
* Fix ansible-test list arg defaults.
* Fix ansible-test require and exclude delegation.
* Fix detection of Windows specific changes.
* Add minimal Windows testing for Python 3.7.
2018-10-02 12:26:14 -07:00
Matt Clay
ac492476e5
Bug fixes and cleanup for ansible-test. (#45991)
* Remove unused imports.
* Clean up ConfigParser usage in ansible-test.
* Fix bare except statements in ansible-test.
* Miscellaneous cleanup from PyCharm inspections.
* Enable pylint no-self-use for ansible-test.
* Remove obsolete pylint ignores for Python 3.7.
* Fix shellcheck issuers under newer shellcheck.
* Use newer path for ansible-test.
* Fix issues in code-smell tests.
2018-09-21 11:38:22 -07:00
Matt Clay
771768966f Flush output from timing script. 2018-08-31 07:58:12 -07:00
Matt Clay
a96a51b0c6 List quay.io images during Shippable run. 2018-08-29 18:04:22 -07:00
Matt Clay
5461bfbf31 Split sanity tests into 3 groups. 2018-08-27 13:43:11 -07:00
Matt Clay
31a5b874a1 Use runtime instead of datetime for test output. 2018-08-23 12:05:10 -07:00
Matt Clay
4e489d1be8
Update Shippable integration test groups. (#43118)
* Update Shippable integration test groups.
* Update integration test group aliases.
* Rebalance AWS and Azure tests with extra group.
* Rebalance Windows tests with another group.
2018-07-23 20:46:22 -07:00
Matt Clay
528de5b82f Remove unwanted containers at CI job start. 2018-07-09 21:48:11 -07:00
Matt Clay
a7d7df1450 Make docs-build sanity test disabled by default. 2018-05-09 17:55:00 -07:00
Matt Clay
8a223009ca
Improve handling of integration test aliases. (#38698)
* Include change classification data in metadata.
* Add support for disabled tests.
* Add support for unstable tests.
* Add support for unsupported tests.
* Overhaul integration aliases sanity test.
* Update Shippable scripts to handle unstable tests.
* Mark unstable Azure tests.
* Mark unstable Windows tests.
* Mark disabled tests.
2018-04-12 16:15:28 -07:00
Matt Clay
11ad559010
Terminate Tower instances after CI ends. (#37265)
* Remove obsolete Tower support from manage_ci.
* Add missing remote settings to cloud tests.
2018-03-09 16:17:29 -08:00
Matt Clay
1d419c737d Run each Windows version independently in CI. 2018-02-09 16:38:27 -08:00
Matt Clay
7abdab6c9e Convert ansible-test compile into a sanity test. 2018-01-25 09:45:36 -08:00
Matt Clay
eae23d2dbd Skip coverage on routine CI runs. 2018-01-23 10:38:38 -08:00
Matt Clay
20824b8f3d Simplify Windows versions in ansible-test. 2018-01-17 16:16:15 -08:00
Matt Clay
d8673ed72a Preserve test/results/data/ in Shippable jobs. 2018-01-12 08:28:57 -08:00
Matt Davis
fbce62795d enable Windows Server 2016 in CI (#34127)
* enable Windows Server 2016 in CI
* avoid use of Write-Host in config script
2018-01-09 17:22:24 -08:00
Matt Clay
82b5a6a0c9
Fix ansible-test network-integration command. (#34661)
* Fix ansible-test network platform init filter.
* Fix ansible-test network inventory generation.
* Remove ios/csr1000v from CI.
* Run network tests on Python 2.7 and 3.6.
2018-01-09 14:52:36 -08:00
Matt Clay
1b5c4b72bd Add Azure provider support to ansible-test and CI.
ci_complete
2017-12-06 00:34:54 -08:00
Matt Clay
887f227994 Use vyos/1.1.8 in CI. 2017-11-22 10:02:25 -08:00
Matt Clay
6472723ba8
Add missing ansible-test --remote-terminate support. (#32918)
* Expand ansible-test --remote-terminate support:

- windows-integration
- network-integration

These commands previously accepted the option, but did not support it.

* Terminate windows and network instances when done.
2017-11-14 17:08:48 -08:00
Matt Clay
cf1337ca9a Update ansible-test sanity command. (#31958)
* Use correct pip version in ansible-test.
* Add git fallback for validate-modules.
* Run sanity tests in a docker container.
* Use correct python version for sanity tests.
* Pin docker completion images and add default.
* Split pylint execution into multiple contexts.
* Only test .py files in use-argspec-type-path test.
* Accept identical python interpeter name or binary.
* Switch cloud tests to default container.
* Remove unused extras from pip install.
* Filter out empty pip commands.
* Don't force running of pip list.
* Support delegation for windows and network tests.
* Fix ansible-test python version usage.
* Fix ansible-test python version skipping.
* Use absolute path for log in ansible-test.
* Run vyos_command test on python 3.
* Fix windows/network instance persistence.
* Add `test/cache` dir to classification.
* Enable more python versions for network tests.
* Fix cs_router test.
2017-10-26 00:21:46 -07:00
Matt Clay
03e18aa52e Remove cryptography install from CI other test. 2017-10-12 00:36:12 -07:00
Matt Clay
5a91ed0a3b Switch CI for OS X back to a single job. (#31265)
* Update Shippable scripts for group/non-group use.
* Switch CI for OS X back to a single job.
2017-10-03 14:37:05 -07:00
Matt Clay
72237b63e7 Run OS X and FreeBSD CI tests using groups. (#31254)
* Run OS X and FreeBSD CI tests using groups.
* Improve readability of Shippable jobs.
2017-10-03 12:43:46 -07:00
Matt Clay
765718bed2 Run all Windows tests if change detection disabled (#30320)
* Run all Windows tests if change detection disabled
* Run all Windows versions if change detection disabled
2017-09-13 22:21:19 -07:00
Matt Clay
8f4c8844c4 Divide Windows integration tests into 2 groups. (#29029)
* Divide Windows integration tests into 2 groups.
* Support `none` for `--changed-all-target`.
* Run 2 separate Windows groups on Shippable.
* Only run smoketest and minimal for the group1 job.
2017-09-05 15:32:26 -07:00
Matt Clay
55fb18b85c Split cloud tests into two CI groups. 2017-09-01 17:33:54 -07:00
Matt Clay
e513ec10da Add safety check for Shippable "Rebuild with SSH" (#28857)
* Add safety check for Shippable "Rebuild with SSH".
* Run shippable.py for the `other` test.
2017-08-31 17:44:33 -07:00
Matt Clay
1a98962e19 Switch default cloud tests to smoketest. 2017-08-18 22:47:18 -07:00
Matt Clay
e0e5b4cf64 Optimize running of CI cloud integration tests. 2017-07-14 22:28:14 -07:00
Matt Clay
7931e11437 Use subset of Windows tests for "all" tests. (#26830) 2017-07-14 16:51:32 -07:00
Matt Clay
0b784c65b1 Add sanity import test to ansible-test. (#26730)
* Add sanity import test to ansible-test.
* Run sanity import test on all python versions.
2017-07-12 20:04:22 -07:00
Matt Clay
4a013d6d69 Add RHEL 7.3 to ansible-test and CI. (#26550)
* Add support for RHEL to ansible-test.
* Enable RHEL 7.3 on Shippable.
2017-07-07 16:52:40 -07:00
Toshio Kuratomi
e238ae999b Cyptography pr 20566 rebase (#25560)
Make pyca/cryptography the preferred backend for cryptographic needs (mainly vault) falling back to pycrypto

pyca/cryptography is already implicitly a dependency in many cases
through paramiko (2.0+) as well as the new openssl_publickey module,
which requires pyOpenSSL 16.0+. Additionally, pyca/cryptography is
an optional dep for better performance with vault already.

This commit leverages cryptography's padding, constant time comparisons,
and CBC/CTR modes to reduce the amount of code ansible needs to
maintain.

* Handle wrong password given for VaultAES format

* Do not display deprecation warning for cryptography on python-2.6

* Namespace all of the pycrypto imports and always import them

  Makes unittests better and the code less likely to get stupid mistakes
  (like using HMAC from cryptogrpahy when the one from pycrypto is needed)

* Add back in atfork since we need pycrypto to reinitialize its RNG just in case we're being used with old paramiko

* contrib/inventory/gce: Remove spurious require on pycrypto

(cherry picked from commit 9e16b9db275263b3ea8d1b124966fdebfc9ab271)

* Add cryptography to ec2_win_password module requirements
  * Fix python3 bug which would pass text strings to a function which
    requires byte strings.

* Attempt to add pycrypto version to setup deps

* Change hacking README for dual pycrypto/cryptography

* update dependencies for various CI scripts

* additional CI dockerfile/script updates

* add paramiko to the windows and sanity requirement set

  This is needed because ansible lists it as a requirement. Previously
  the missing dep wasn't enforced, but cryptography imports pkg_resources
  so you can't ignore a requirement any more

* Add integration test cases for old vault and for wrong passwords

* helper script for manual testing of pycrypto/cryptography

* Skip the pycrypto tests so that users without it installed can still run the unittests

* Run unittests for vault with both cryptography and pycrypto backend
2017-06-27 06:00:15 -07:00
Matt Clay
5babe2daea Increase python version coverage for tests. (#24762)
* Improve ansible-test inventory handling.
* Fix python 3 re-raise of exception from thread.
* Fix python 3 encoding for windows-integration.
* Run network tests on multiple python versions.
* Run windows tests on multiple python versions.
* Support Shippable delegation using --tox.
* Skip vyos_command on python 3 tests until fixed.
* Add python 3 filtering to local and tox.
* Fix tests to support back to back runs.
* Temporarily test networking with python 2.7 only.

Running the tests back to back causes intermittent test failures
which need to be addressed before we can test multiple versions
in a single test run.
2017-05-19 01:37:53 +08:00
Matt Clay
036ba7eeec Clean up Shippable tools and update job handling.
- Tools are now in a tools subdirectory.
- Removed obsolete ansible-core-ci tool.
- Added run.py for starting new CI runs.
- Improved handling of run IDs and URLs.
- General code cleanup and docs updates.
- Nightly CI runs use complete coverage.
2017-05-17 14:56:40 +08:00
Matt Clay
6e160650e9 Fix run filter on Shippable downloader. 2017-05-17 10:08:08 +08:00
Matt Clay
4daa16f9ce Add --all and --stub coverage options.
ci_coverage
ci_complete
2017-05-16 12:02:03 +08:00
Matt Clay
6a2a7a2392 Enable codecov.io and add coverage grouping. 2017-05-15 19:14:05 +08:00
Matt Clay
fdcad0f44d Improve on-demand coverage for Shippable. 2017-05-14 18:27:58 +08:00
Matt Clay
5cb5228cde Implement on-demand coverage on Shippable. (#24535) 2017-05-14 01:08:42 +08:00
Matt Clay
af5db3e4a9 Revert "Add support for on-demand coverage on Shippable."
This reverts commit 7b3f0b8870.
2017-05-12 15:47:28 +08:00
Matt Clay
7b3f0b8870 Add support for on-demand coverage on Shippable.
Triggered by setting the COVERAGE environment var to a non-empty value.
Alternatively the string `ci_coverage` can be added to a commit message.
2017-05-12 14:55:48 +08:00
Matt Clay
fb9efd3c7e Add --remote-terminate option to ansible-test.
Set `--remote-terminate success` for osx tests.
2017-05-11 22:42:52 +08:00
Matt Clay
17e07a27b2 Enable cloud tests for use with ansible-test. 2017-05-05 21:46:29 +08:00
Matt Clay
0587d59266 Use Shippable image: drydock/u16pytall:master (#23997)
* Use Shippable image: drydock/u16pytall:master
* Do not install python 3.6 on Shippable.
2017-04-26 15:04:28 +08:00
Matt Clay
e7bb508ad6 Fix var precedence check to support python 3. (#23552)
* Fix var precedence check to support python 3.
* Run CI sanity tests using python 3.5.
* Disable pylint non-iterator-returned test to pass on python 3.5.
2017-04-13 00:07:40 -07:00
Matt Clay
eec01e2137 Set IO encoding for ansible-test on Shippable. 2017-04-08 23:06:44 -07:00
Matt Clay
dfe8eca3d2 Update Shippable downloader to support new API.
- Download run/jobs by URL (old or new).
- Download run metadata.
- Download job metadata.
- Format downloaded JSON.
- Handle and save coverage/testresults download errors.
2017-03-24 14:24:38 -07:00
Matt Clay
12f6beaebd Remove python2.4 package install from other test. 2017-03-15 16:40:05 -07:00
Matt Clay
869449e288 Add test verification to ansible-test. (#22636)
* Add unified git diff parser.
* Add metadata and diff handling.
* Add test confidence/verification to bot output.
2017-03-15 12:17:42 -07:00
Matt Clay
89559f78de Add --failure-ok option to ansible-test. (#22623) 2017-03-14 16:32:21 -07:00
Matt Clay
a64665ae91 Fix error handling in CI other script. 2017-03-08 01:31:53 -08:00
Matt Clay
3ab344e8bd Use lint/junit output for compile test. (#22395)
* Refactor sanity classes for use in all tests.
* Use lint/junit output for compile test.
* Add missing options for compile test.
* Fix early bailout on requirements install.
2017-03-08 00:47:21 -08:00
Matt Clay
9acf551674 Run all other tests on Shippable before failing. 2017-03-07 20:26:10 -08:00
Matt Clay
8f463fcdd2 Create bot friendly sanity output. (#22381) 2017-03-07 14:59:50 -08:00
Matt Clay
72c96b3ac3 Fix encoding issue in shippable downloader. 2017-03-03 16:38:36 -08:00
Matt Clay
d66ce40ecb Overhaul ansible-test sanity implementation. (#22177)
- Tests are run to completion instead of stopping on first failure.
- Test results are now parsed instead of passing through to the console.
- Test results can be saved in junit xml format.
- Test results will show up on the Shippable "Tests" result tab.
- Added an experimental --lint option for easier integration with other tools.
- Code smell tests are now usable with the --list-tests, --test and --skip-test options.
- Code split out from executor.py into sanity.py.
- Rename download-logs to download.py and add support for test and coverage results.
- Miscellaneous improvements.
2017-03-02 12:36:46 -08:00
Matt Clay
ae2d2860f5 Reduce CI output processing code coverage. 2017-02-27 22:39:35 -08:00
Matt Clay
2804d2e848 Split up unit tests on Shippable. (#21931)
* Add more verbose output to coverage combine.
* Run unit tests on CI separately by version.
2017-02-24 16:58:56 -08:00
Matt Clay
b2bd75a408 Restore 2008-SP2 and 2008-R2_SP1 CI testing. 2017-02-21 20:55:25 -08:00
Matt Davis
8527013fbe Complete rewrite of Windows exec wrapper (#21510)
* supports pipelining for faster execution
* supports become (runas), creates interactive subsession under WinRM batch logon
* supports usage of arbitrary module_utils files
* modular exec wrapper payload supports easier extension
* integrates async wrapper behavior for pipelined/become'd async
* module_utils are loaded as true Powershell modules, no more runtime modifications to module code
2017-02-17 00:09:56 -08:00
Matt Clay
f16ee2841b Add retries to Shippable CI scripts. (#21108)
* Add retry.py for auto-retry in CI scripts.
* Add retries to other and units CI scripts.
2017-02-07 16:33:45 -08:00
Matt Clay
f42fa8879b Temporarily remove Windows Server 2016 from CI. 2017-01-27 11:56:50 -08:00
Matt Clay
7ed67746cf Enable Windows 2016 on Shippable.
Also launch Windows instances on us-east-2 and all others on us-east-1.
2017-01-18 18:37:42 -08:00
Matt Clay
3c51ce56fd Enable more network support in ansible-test. (#20372)
* Only start platform instances with tests selected.
* Enable ios on Shippable.
* Show inventory in explain mode.
* Fix indentation of generated network inventory.
* Update classification of network module_utils.
2017-01-17 16:24:05 -08:00
Matt Clay
203e375449 Fix python 3.6 install. 2017-01-13 15:30:27 -08:00
Matt Clay
f534573dcf Enable first network tests on Shippable. (#20208) 2017-01-12 18:23:53 -08:00
Matt Clay
cd3fdca540 Switch tests to pytest and ansible-test.
- Replace nose usage with pytest.
- Remove legacy Shippable integration.sh.
- Update Makefile to use pytest and ansible-test.
- Convert most yield unit tests to pytest parametrize.
2017-01-11 12:34:59 -08:00
Matt Clay
f340b8d14f Add test support for python 3.6. 2017-01-10 14:34:42 -08:00
Matt Clay
422857166b Remove legacy test alias migration script. (#19629)
* Require code-smell tests to be files.
* Add sanity check for integration test aliases.
* Remove migration script for test aliases.
2016-12-21 16:46:28 -08:00
Matt Clay
f789a20f29 Update log download url. 2016-12-13 16:26:10 -08:00
Toshio Kuratomi
c6b42028c4 Fixes for uri under python3 and local (non-httptester) testing 2016-12-13 15:46:07 -08:00