Commit graph

198 commits

Author SHA1 Message Date
Matt Clay
7931e11437 Use subset of Windows tests for "all" tests. (#26830) 2017-07-14 16:51:32 -07:00
Matt Clay
76158db10c Improve sanity import test messaging. (#26782) 2017-07-14 08:04:43 -07:00
Matt Clay
789218c215 Initial ansible-test sanity docs. (#26775)
* Rename no-iterkeys test for consistency.

* Require docs for all ansible-test sanity tests.

* Initial ansible-test sanity docs.

* Fix capitalization of Python.

* Fix sanity code smell test false positives.

* Fix another code-smell false positive.
2017-07-14 14:24:45 +01:00
Matt Clay
e1b9af9082 Add retries to remote test package installs. 2017-07-13 22:29:11 -07:00
Pilou
4b3d6dfa8a Use pycodestyle instead of pep8 (#25947) 2017-07-13 11:46:31 -07:00
Matt Clay
fad469efca Ignore platform_agnostic.yaml in ansible-test. 2017-07-13 11:23:59 -07:00
Matt Clay
0b784c65b1 Add sanity import test to ansible-test. (#26730)
* Add sanity import test to ansible-test.
* Run sanity import test on all python versions.
2017-07-12 20:04:22 -07:00
Matt Clay
20d49af475 Add ANSIBLE_TEST_CI env var to integration tests. 2017-07-12 17:46:12 -07:00
Ganesh Nalawade
e14e37ee1e Fix junos unit test failures (#26676)
*  Use lxml api's in unit test to parse xml
*  Remove unwanted import in unit test
*  Add ncclient dependency in unit test requirement
2017-07-12 08:36:16 -07:00
Matt Clay
ab5dd24430 Handle pip/python errors in ansible-test better. 2017-07-11 16:37:16 -07:00
David Newswanger
64f0eb50ff remove vyos/1.0.5 from test/runner/completion/network.txt 2017-07-10 21:27:08 -07:00
Matt Clay
82558baaf6 Add role dependencies to change classification. 2017-07-10 18:20:09 -07:00
Matt Clay
a66bd8f7c3 Don't update RHEL instances before running tests. 2017-07-07 23:27:25 -07:00
Matt Clay
2146bb57ce Simplify FreeBSD setup for tests. 2017-07-07 23:27:01 -07:00
Matt Clay
4a013d6d69 Add RHEL 7.3 to ansible-test and CI. (#26550)
* Add support for RHEL to ansible-test.
* Enable RHEL 7.3 on Shippable.
2017-07-07 16:52:40 -07:00
Andrea Tartaglia
0d48732e7e Integration tests requirements (#26545)
* Removed pexpect and jmespath requirement, install it in task directly
* Removed passlib dependency
2017-07-07 14:11:53 -07:00
David Newswanger
294c0f91c0 added option to ansible-test to run test playbooks using the debug strategy (#26499) 2017-07-07 12:37:08 -07:00
Matt Clay
a94156227d Improve error handling for cs tests. 2017-07-06 17:55:51 -07:00
Matt Clay
45e377566c Refactor ansible-test config classes. (#26505)
* Move Config classes from executor.py to config.py.
* Move Environment and Test config to config.py.
* Move Coverage/CoverageReport Config to config.py.
* Clean up type hints.
2017-07-06 16:14:44 -07:00
Matt Clay
bd8ea89b1b Improve handling of network test changes. 2017-07-06 14:25:58 -07:00
Matt Clay
c26a556929 Add check for known_hosts changing during tests. (#26470)
* Add check for known_hosts changing during tests.
* Don't use known hosts file for delegate_to test.
2017-07-06 00:47:28 -07:00
Matt Clay
fba76444e0 Add ansible-test coverage report --show-missing. 2017-07-05 17:11:41 -07:00
Matt Clay
69ab6fa803 Remove legacy pycrypto test requirement. 2017-07-05 16:47:53 -07:00
Adrien Vergé
4ee1d0dd35 Revert "Limit yamllint version on python 2.6." (55aec8e)
Thanks to the report by Matt Clay at [1], yamllint now supports Python
2.6. Tests were enabled on Travis for 2.6 to make sure there will be no
regressions in the future.

[1]: https://github.com/adrienverge/yamllint/issues/55
2017-07-05 08:23:21 -07:00
David Newswanger
0a6dcf4169 changed ansible-test so that it replaces . with - instead of . with _ so that the hostname is valid 2017-07-03 20:11:50 -07:00
Matt Clay
cd3a795e65 Permissions fix for remote coverage downloads. 2017-07-02 14:09:16 -07:00
Matt Clay
554423cf04 Expose docker socket to test containers. 2017-06-30 16:46:41 -07:00
Matt Clay
7ab4a889ad Update tox.ini to use ansible-test. 2017-06-30 16:41:30 -07:00
René Moser
77d0542985 test, integration: add diff support (#26296) 2017-06-30 20:43:34 +02:00
Pierre-Louis Bonicoli
d08ff86169 ansible-test: pass --[skip-]tags to ansible-playbook
Thanks to Olaf Klischat for the idea.
2017-06-30 10:35:39 -07:00
Matt Clay
55aec8ebc9 Limit yamllint version on python 2.6. 2017-06-28 08:35:19 -07:00
Toshio Kuratomi
e238ae999b Cyptography pr 20566 rebase (#25560)
Make pyca/cryptography the preferred backend for cryptographic needs (mainly vault) falling back to pycrypto

pyca/cryptography is already implicitly a dependency in many cases
through paramiko (2.0+) as well as the new openssl_publickey module,
which requires pyOpenSSL 16.0+. Additionally, pyca/cryptography is
an optional dep for better performance with vault already.

This commit leverages cryptography's padding, constant time comparisons,
and CBC/CTR modes to reduce the amount of code ansible needs to
maintain.

* Handle wrong password given for VaultAES format

* Do not display deprecation warning for cryptography on python-2.6

* Namespace all of the pycrypto imports and always import them

  Makes unittests better and the code less likely to get stupid mistakes
  (like using HMAC from cryptogrpahy when the one from pycrypto is needed)

* Add back in atfork since we need pycrypto to reinitialize its RNG just in case we're being used with old paramiko

* contrib/inventory/gce: Remove spurious require on pycrypto

(cherry picked from commit 9e16b9db275263b3ea8d1b124966fdebfc9ab271)

* Add cryptography to ec2_win_password module requirements
  * Fix python3 bug which would pass text strings to a function which
    requires byte strings.

* Attempt to add pycrypto version to setup deps

* Change hacking README for dual pycrypto/cryptography

* update dependencies for various CI scripts

* additional CI dockerfile/script updates

* add paramiko to the windows and sanity requirement set

  This is needed because ansible lists it as a requirement. Previously
  the missing dep wasn't enforced, but cryptography imports pkg_resources
  so you can't ignore a requirement any more

* Add integration test cases for old vault and for wrong passwords

* helper script for manual testing of pycrypto/cryptography

* Skip the pycrypto tests so that users without it installed can still run the unittests

* Run unittests for vault with both cryptography and pycrypto backend
2017-06-27 06:00:15 -07:00
Matt Clay
e56ea435f0 Skip dependency check for deleted files. (#26112) 2017-06-26 11:48:46 -07:00
David Newswanger
b2d88f8b73 updated ansible-test integration to skip any files in test/integration/targets/ that aren't directories.
Fixes #26007
2017-06-22 13:03:42 -07:00
jctanner
996dcb989c Add the cloud provider script for the vcenter simulator container (#25964) 2017-06-22 10:49:58 -07:00
Matt Clay
9d8aa43c67 Temporary work-around for setuptools 36.0.0 bug. (#25243)
* Temporary work-around for setuptools 36.0.0 bug.
* Use older setuptools for pip integration test.
* Limit isort version to avoid new test failures.
2017-06-01 16:23:32 +08:00
kkjang
daada2000c Fix expect for python 3 (#24912)
* Fix expect for python 3

- Change generator next to python 3 compatible
- Added tests for expect

* Add pexpect to integration.txt

- add pexpect library to requirements for integration tests

* Use ansible_python_interpreter in integration tests for expect

* Use double-quotes for expect integration tests

* Cast user input to string for expect integration tests

* Cast user input to string earlier in expect integration tests

* Use ansible.module_utils.six.moves input for expect integration tests

* Fix yamllint errors in the expect test

* Use cat to trigger timeout for expect integration tests

* Use realpath filter in expect integration tests
2017-05-23 15:51:46 -07:00
Ricardo Carrillo Cruz
b12beca3ed Refactos nxos_ip_interface module (#24885) 2017-05-22 19:48:59 +02:00
Tim Rupp
c1397626fc Adds the bigip_user module to Ansible (#24753)
The patch adds the bigip_user module to Ansible to support managing
users on an F5 BIG-IP.

Unit tests are provided. Integration tests can be found here

  * https://github.com/F5Networks/f5-ansible/blob/devel/test/integration/bigip_user.yaml
  * https://github.com/F5Networks/f5-ansible/tree/devel/test/integration/targets/bigip_user/tasks
2017-05-18 20:21:18 +01:00
Matt Clay
53180ebb57 Fix Shippable BASE_BRANCH lookup on merges. 2017-05-19 01:51:24 +08:00
Matt Clay
5babe2daea Increase python version coverage for tests. (#24762)
* Improve ansible-test inventory handling.
* Fix python 3 re-raise of exception from thread.
* Fix python 3 encoding for windows-integration.
* Run network tests on multiple python versions.
* Run windows tests on multiple python versions.
* Support Shippable delegation using --tox.
* Skip vyos_command on python 3 tests until fixed.
* Add python 3 filtering to local and tox.
* Fix tests to support back to back runs.
* Temporarily test networking with python 2.7 only.

Running the tests back to back causes intermittent test failures
which need to be addressed before we can test multiple versions
in a single test run.
2017-05-19 01:37:53 +08:00
Matt Clay
9178e176b5 Limit sphinx version on python 2.6. (#24678)
* Limit sphinx version on python 2.6.
* Fix issues identified by rstcheck.
2017-05-17 01:00:15 +08:00
Matt Clay
4daa16f9ce Add --all and --stub coverage options.
ci_coverage
ci_complete
2017-05-16 12:02:03 +08:00
Matt Clay
6a2a7a2392 Enable codecov.io and add coverage grouping. 2017-05-15 19:14:05 +08:00
Matt Clay
fdcad0f44d Improve on-demand coverage for Shippable. 2017-05-14 18:27:58 +08:00
John R Barker
e9e2d67035 Run vyos_command Network tests in Shippable (#24514)
* Run VyOS Network tests in Shippable

* net_command no longer exists

* network/ci

* Disable other network tests

* pep8 fix

* Fix PEP 8 issue.
2017-05-12 12:49:12 +01:00
Matt Clay
dfd19a812f Miscellaneous bug fixes for ansible-test.
- Overhauled coverage injector to fix issues with non-local tests.
- Updated integration tests to work with the new coverage injector.
- Fix concurrency issue by using random temp files for delegation.
- Fix handling of coverage files from root user.
- Fix handling of coverage files without arcs.
- Make sure temp copy of injector is world readable and executable.
2017-05-12 14:55:48 +08:00
Matt Clay
fb9efd3c7e Add --remote-terminate option to ansible-test.
Set `--remote-terminate success` for osx tests.
2017-05-11 22:42:52 +08:00
Matt Clay
b8cb3f519b Detect and fix environment tampering in tests. 2017-05-11 19:45:15 +08:00
Matt Clay
e6934e42be Update ansible-test change handling and CS plugin. (#24395)
* Use CloudStack simulator from Ansible repo.
* Recognize cloud specific ansible-test changes.
* Hide ansible-test cs notice in explain mode.
2017-05-09 16:15:27 +08:00
Matt Clay
0573b8bcd2 Fix ansible-test cloud integration bugs. 2017-05-05 21:46:29 +08:00
Matt Clay
ac72fd9d2c Add missing return statements. 2017-05-05 18:37:11 +08:00
Matt Clay
201242f302 Fix call to wrong method in ansible-test. 2017-05-05 18:18:40 +08:00
Matt Clay
1582e9bf7d Fix handling of Shippable env vars. 2017-05-05 17:44:51 +08:00
Matt Clay
a07d42e16d Add support for cloud tests to ansible-test. (#24315)
* Split out ansible-test docker functions.
* Add cloud support to ansible-test.
2017-05-05 16:23:00 +08:00
Matt Clay
6522d703a1 Fix ansible-test pylint and pep8 issues. (#24245)
* Fix pylint useless-super-delegation issues.
* Fix miscellaneous pylint issues.
* Fix PEP 8 issues.
* Fix warnings reported by PyCharm.
2017-05-03 23:19:44 +08:00
Matt Clay
d662f6f0db Fix bugs in ansible-test units command. (#24044)
* Handle old versions of coverage.
* Handle old versions of setuptools.
* Detect python version for docker/remote units.
* Add sanity override for test constraints.
2017-04-27 11:21:11 +08:00
Matt Clay
cb1f57d4e5 Add rstcheck to ansible-test and correct issues. (#23550)
* Add rstcheck to ansible-test.
* Fix rst code-block languages and syntax errors.
* Fix rst inline literals.
* Update python 2 code block to pass tests on py 3.
2017-04-13 10:28:52 -07:00
Matt Clay
4ee96b20a3 Temporarily cap pylint version < 1.7.0 in tests. 2017-04-13 09:10:29 -07:00
Toshio Kuratomi
4e8e366739 Ryanpineo fix ipaddr netmask with 32 cidr (#23506)
* Fix bug in ipaddr netmask filter when cidr is 32

* Add requirement on netaddr to the units.txt file
2017-04-11 14:10:43 -07:00
Matt Clay
92ad9a19b6 Fix ansible-test empty diff handling on Shippable. 2017-04-08 22:08:08 -07:00
Matt Clay
0cf1e8e061 Fix ansible-test handling of import/syntax errors. 2017-04-07 12:20:31 -07:00
Matt Clay
1daa69d685 Initial pylint support for ansible-test sanity. 2017-04-04 16:16:21 -07:00
Ryan Brown
2196fa0e95 [cloud][tests] fixtures for placebo to test AWS modules using boto3 (#21253)
* [cloud][tests] Create fixtures for using placebo to test boto3-using modules

* Use pytest's importorskip instead of manually skipping on missing deps

* Fix imports in cloudformation module

* Delete unused code

* Add maybe_sleep fixtures to speed up recorded test runs

* Build basic placebo-CFN tests

* Commit placebo recordings of basic stack operations

* Add placebo to test-requires

* Allow unit tests to run regardless of environment by setting a default region

* Use explicit relative import for Python 3 compat

* Use __name__ attribute that works on Python 2 and 3
2017-03-31 15:31:12 -04:00
Matt Clay
4fdeade389 Overhaul ansible-test import analysis. (#22888)
- Add support for module_utils packages for better organization.
- Add support for "virtual" module_utils packages such as `six`.
2017-03-22 18:07:53 -07:00
Matt Clay
042e387f27 Fix metadata handling for shell command. (#22801) 2017-03-20 15:31:57 -07:00
Matt Clay
991bf0a1c3 Fix ansible-test base branch handling on merge. (#22759) 2017-03-17 23:33:32 -07:00
Matt Clay
47887a2edd Fix ansible-test base branch usage on Shippable. 2017-03-17 17:35:09 -07:00
Matt Clay
f7c83ccdc0 Fix ansible-test PR diff handling.
Permits use of PRs on branches other than devel.
2017-03-16 10:34:31 -07:00
Matt Clay
3a22467b38 Remove python 2.4 compile test. 2017-03-15 15:46:12 -07:00
Matt Clay
869449e288 Add test verification to ansible-test. (#22636)
* Add unified git diff parser.
* Add metadata and diff handling.
* Add test confidence/verification to bot output.
2017-03-15 12:17:42 -07:00
Matt Clay
5e9a2b8528 Verify SHA in Shippable change detection. 2017-03-15 11:31:55 -07:00
Matt Clay
89559f78de Add --failure-ok option to ansible-test. (#22623) 2017-03-14 16:32:21 -07:00
Matt Clay
771b14acb2 Sort test messages for deterministic output. (#22581) 2017-03-13 20:45:10 -07:00
Matt Clay
cfd313bced Disable ansible color output on sanity tests. 2017-03-08 12:18:16 -08:00
Matt Clay
3ab344e8bd Use lint/junit output for compile test. (#22395)
* Refactor sanity classes for use in all tests.
* Use lint/junit output for compile test.
* Add missing options for compile test.
* Fix early bailout on requirements install.
2017-03-08 00:47:21 -08:00
Matt Clay
fbf55886ba Use lint friendly output for python syntax checks. (#22390) 2017-03-07 20:27:50 -08:00
Matt Clay
8f463fcdd2 Create bot friendly sanity output. (#22381) 2017-03-07 14:59:50 -08:00
Matt Clay
3e4be156d7 Pre-compile regexes to speed up target processing.
Without this, changing a large number of files results in target
processing taking a very long time due to repeatedly compiling
the same patterns in a loop over many targets.
2017-03-06 12:35:12 -08:00
Matt Clay
d66ce40ecb Overhaul ansible-test sanity implementation. (#22177)
- Tests are run to completion instead of stopping on first failure.
- Test results are now parsed instead of passing through to the console.
- Test results can be saved in junit xml format.
- Test results will show up on the Shippable "Tests" result tab.
- Added an experimental --lint option for easier integration with other tools.
- Code smell tests are now usable with the --list-tests, --test and --skip-test options.
- Code split out from executor.py into sanity.py.
- Rename download-logs to download.py and add support for test and coverage results.
- Miscellaneous improvements.
2017-03-02 12:36:46 -08:00
Matt Clay
28b882e737 Support ansible commands in sanity tests. (#22172) 2017-03-01 15:48:14 -08:00
Matt Clay
4f3dade266 Add retries for ansible-test docker run. 2017-02-27 22:55:46 -08:00
Matt Clay
45cec93e99 Disable newer shellcheck tests which don't pass. 2017-02-27 22:31:47 -08:00
Matt Clay
a7bb04800d Add boto to units requirements for s3 test. 2017-02-27 20:55:32 -08:00
Matt Clay
fcac261eef Run unit tests in isolation w/ coverage support. 2017-02-27 17:16:45 -08:00
Matt Clay
c87ebb6215 Deduplicate arcs when combining coverage data. (#22022) 2017-02-27 15:46:15 -08:00
Matt Clay
2797cff773 Run destructive win tests when creating instances. 2017-02-27 10:36:53 -08:00
Matt Clay
2804d2e848 Split up unit tests on Shippable. (#21931)
* Add more verbose output to coverage combine.
* Run unit tests on CI separately by version.
2017-02-24 16:58:56 -08:00
Matt Clay
bc8186543a Pass base branch to module validator on delegate. 2017-02-20 14:28:11 -08:00
Matt Martz
9b1bd2eb7e Re-enable module comparisons in validate-modules (#21582)
* Re-enable module comparisons, specifically for new module detection and for finding new options/arguments
* Only do new module checks in shippable, local will display warning
2017-02-17 15:13:18 -08:00
Matt Clay
1ce2ea0b6c Fix ansible-test handling of deleted test targets. 2017-02-17 11:16:24 -08:00
Matt Clay
2192c1eb02 Fix ansible-test selection of inventory file. 2017-02-16 15:03:27 -08:00
John R Barker
5e28d3a15c network_cli shouldn't be specified anymore (#21458) 2017-02-15 12:58:12 +00:00
Matt Clay
d54bc09fae Consider module_utils deps when running tests. (#21382)
* Skip pep8 analysis when --explain is used.
* Fix return type annotations.
* Match line length requirement of PEP 8 config.
* Consider module_utils deps when running tests.
2017-02-13 18:49:36 -08:00
Peter Sprygada
7f1c43e597 updates ios modules to support persistent socket (#21258)
* updates all ios modules to support persistent socket
* adds ios action plugin to connect to device
* adds exec_command() to ios shared module
* fixes ios_config and ios_template local action
* update all unit test cases
* adds base test module for ios module testing
2017-02-13 15:22:14 +00:00
Matt Clay
4189592701 Revert "Enable process isolation for unit tests."
This reverts commit 91526cd9f2.

Removing this feature primarily because it interferes with
collecting proper code coverage results. I may restore the
feature later if that can be resolved.
2017-02-09 11:10:06 -08:00
Matt Clay
91526cd9f2 Enable process isolation for unit tests. 2017-02-08 14:36:42 -08:00
Matt Clay
f16ee2841b Add retries to Shippable CI scripts. (#21108)
* Add retry.py for auto-retry in CI scripts.
* Add retries to other and units CI scripts.
2017-02-07 16:33:45 -08:00
Matt Clay
bce5fc7d96 Use us-east-2 for Windows CI on Shippable. 2017-02-06 15:01:20 -08:00
Matt Clay
309a37de86 Add --debug option to ansible-test. 2017-02-01 16:05:03 -08:00