Commit graph

5826 commits

Author SHA1 Message Date
Gonéri Le Bouder
4869874337
ansible-test: vcenter initialize group/vmware (#70011)
Ensure the vcenter provider initialize the `module_defaults` of all
the vmware modules, not just `vmware_guest`.
The VMware CI relies on this for the authentication of the different
VMware modules.

The commit adjust `incidental_vmware_prepare_tests/tasks/init_vcsim.yml`.
The test-suite uses a copy of `vmware_guest` that is not in the
`group/vmware` group. As a result, we need to manually pass the
authentification parameter.
2020-06-11 12:21:44 -04:00
Brian Coca
ebb22655e4
implemented 'prefix' for file based cache (#69872)
* implemented 'prefix' for file based cache

Co-authored-by: s-hertel <shertel@redhat.com>
2020-06-11 10:40:25 -04:00
Jordan Borean
d30fc6c0b3
galaxy - preserve symlinks on build/install (#69959)
* galaxy - preserve symlinks on build/install

* Handle directory symlinks

* py2 compat change

* Updated changelog fragment
2020-06-11 06:46:42 +10:00
Sam Doran
3815ed67d5
ansible-galaxy - Fix role info when role is not installed (#69924)
* ansible-galaxy - Fix role info when role is not installed

Only report the role not found if in offline mode, otherwise query the galaxy API
to get role information.

Fixes #69867

* Improve error message when role is not found in Ansible Galaxy
2020-06-10 15:36:24 -04:00
Kshitij Chawla
32c392e622
Raise Warning when import_playbook receives additional parameters (#64156)
* Extra whitespace around imported playbook filename are stripped.
* Corrected call for display.deprecated
* Changed warning type & added test
* Added auto verification of raised warning
* More accurate warning message
2020-06-10 13:22:40 -04:00
Sloane Hertel
51f6d129cb
support hard coded module_defaults.yml groups for collections (#69919)
* Only allow groups which were hardcoded in module_defaults.yml

only load action groups from the collection if module_defaults contains a potential group for the action

* Fix tests using modules that override those whitelisted in lib/ansible/config/module_defaults.yml

Third party modules should not be using group/ - use the action name instead

* add externalized module_defaults tests

add the missing group and collections

ci_complete

Co-authored-by: Matt Davis <mrd@redhat.com>

* changelog

ci_complete

* Fix import in tests

ci_complete

* Update with requested changes

ci_complete

* don't traceback since we don't validate the contents of module_defaults

ci_complete

Co-authored-by: Matt Davis <mrd@redhat.com>
2020-06-09 15:38:57 -07:00
Felix Fontein
a862ff2d43
Deprecation revisited (#69926)
* Allow to specify collection_name separately for deprecation.

* Use new functionality in Ansible.

* Use new functionality in tests.

* Update tagging/untagging functions.

* Update pylint deprecated sanity test.

* Update validate-modules. Missing are basic checks for version_added (validate semantic version format for collections).

* Improve version validation. Re-add version_added validation.

* Make sure collection names are added to return docs before schema validation.

* Extra checks to avoid crashes on bad data.

* Make C# module utils code work, and update/extend tests.

* Add changelog fragment.

* Stop extracting collection name from potentially tagged versions/dates.

* Simplify C# code.

* Update Windows modules docs.

* Forgot semicolons.
2020-06-09 15:21:19 -07:00
Matt Clay
7bff3d312f
Split out and install sanity test requirements. (#69971)
* Split out sanity test requirements.

* Run each --venv test separately.

This provides verification that the requirements for each test are properly specified.

* Use a separate requirements file per sanity test.

* Skip setuptools/cryptography setup for sanity.

* Eliminate pyyaml missing warning.

* Eliminate more pip noise.

* Fix conflicting generate_pip_install commands.

* Add changelog fragment.
2020-06-09 13:38:36 -07:00
Brian Coca
ce199ef0e1
Add intg test to repro #36045 (add_host traceback) (#69912)
* Add intg test to repro #36045 (add_host traceback)

* fix raw_params usage in add_host

Co-authored-by: Adrian Likins <alikins@redhat.com>
2020-06-09 15:25:56 -04:00
Matt Clay
03f8bf46c3
Update ansible-test integration tests. (#69970)
* Extend ansible-test collection test.

It now runs `rstcheck` and `empty-init` tests.

* Add galaxy.yml to test collections.
2020-06-09 11:16:48 -07:00
Matt Martz
9667f221a5
Make AnsibleVaultEncryptedUnicode work more like a string (#67823)
* Make AnsibleVaultEncryptedUnicode work more like a string. Fixes #24425

* Remove debugging

* Wrap some things

* Reduce diff

* data should always result in text

* add tests

* Don't just copy and paste, kids

* Add eq and ne back

* Go full UserString copy/paste

* Various version related fixes

* Remove trailing newline

* py2v3

* Add a test that can evaluate whether a variable is vault encrypted

* map was introduces in jinja2 2.7

* moar jinja

* type fix

Co-Authored-By: Sam Doran <sdoran@redhat.com>

* Remove duplicate __hash__

* Fix typo

* Add changelog fragment

* ci_complete

Co-authored-by: Sam Doran <sdoran@redhat.com>
2020-06-08 16:30:14 -05:00
Matt Clay
86f1cba3a6 Minor cleanup for ansible-test integration test. 2020-06-08 11:06:23 -07:00
Matt Martz
c1c6f61a18
Auto unroll generators produced by jinja filters (#68014)
* Auto unroll generators produced by jinja filters

* Unroll for native in finalize

* Fix indentation

Co-authored-by: Sam Doran <sdoran@redhat.com>

* Add changelog fragment

* ci_complete

* Always unroll regardless of jinja2

* ci_complete

Co-authored-by: Sam Doran <sdoran@redhat.com>
2020-06-08 12:58:03 -05:00
Felix Fontein
840d3a9dd7
ansible-doc: properly handle suboptions (#69795)
* Specifically handle suboptions, contains, etc in ansible-doc.
2020-06-08 11:51:33 -04:00
Matt Davis
984216f52e
various deprecation, display, warning, error fixes for collections redirection (#69822)
* various deprecation, display, warning, error fixes

* Update lib/ansible/utils/display.py

Co-authored-by: Felix Fontein <felix@fontein.de>

* Update lib/ansible/utils/display.py

Co-authored-by: Felix Fontein <felix@fontein.de>

* Update lib/ansible/utils/display.py

Co-authored-by: Felix Fontein <felix@fontein.de>

* cleanup, test fixes

* add collection name to deprecated() calls

* clean up redirect entries from uncommitted tests

* fix dep warning/error header text to match previous

Co-authored-by: Felix Fontein <felix@fontein.de>
2020-06-05 00:28:40 -07:00
Rick Elrod
efe103cdff
Make add_host be idempotent/show changed status (#69897)
Change:
- The `add_host` action now shows an accurate change status.

Test Plan:
- Added a plethora of integration tests.

Tickets:
Fixes #69881

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-06-04 21:01:56 -05:00
Brian Coca
062e780a68
starting metadata sunset (#69454)
* starting metadata sunset

 - purged metadata from any requirements
 - fix indent in generic handler for yaml content (whey metadata display was off)
 - make more resilient against bad formed docs
 - removed all metadata from docs template
 - remove metadata from schemas
 - removed mdata tests and from unrelated tests

Co-authored-by: Felix Fontein <felix@fontein.de>
Co-authored-by: Rick Elrod <rick@elrod.me>
2020-06-04 21:01:46 -04:00
Alex Schultz
247e43b252
Fix IncludedFile equality check (#69524)
In the case of a free style strategy, it is possible to end up with
multiple hosts trying to include from the same role, however the tasks
being included may be different with the use of tasks_from.  Previously
if you had two hosts that were included the same role when the
process_include_results function tries to determine if a included needs
to be run on a specific host, it would end up merging two different
tasks into which ever one was processed first.

This change updates the equality check to also check if the task uuid
associated with the IncludedFile is the same. The previous check only
checked if the task's parent uuid was the same. This breaks down when
both includes have the same parent.

    - hosts: all
      strategy: free
      gather_facts: false
      tasks:
        - include_role:
            name: random_sleep
        - block:
          - name: set a fact (1)
            include_role:
              name: set_a_fact
              tasks_from: fact1.yml
          - name: set a fact (2)
            include_role:
              name: set_a_fact
              tasks_from: fact2.yml
        - name: include didn't run
          fail:
            msg: >
              set_a_fact didn't run
              fact1: {{ fact1 | default('not defined')}}
              fact2: {{ fact2 | default('not defined') }}"
          when: (fact1 is not defined or fact2 is not defined)

Closes #69521
2020-06-04 10:41:46 -05:00
Brian Coca
22da903e9c
return changed for group_by (#32057)
* return changed for group_by

* added tests and fixed 'early registeration'
2020-06-03 12:46:40 -04:00
Andrew Klychkov
00ead98595
systemd: should fail in check_mode when service not found on host (#68136)
* systemd: should fail in check_mode when service not found on host
2020-06-02 11:52:12 -04:00
Jack
80f09efd03
do not return the body even if it failed (#69706)
* do not return the body even if it failed

* add some tests for this and rebase

* import test task

* ignore_errors when fails

Co-authored-by: Jack Zhang <jack.zhang@aspiraconnect.com>
2020-06-01 09:55:38 -04:00
Abhijeet Kasurde
723a904f4e
Handle disabled service units (#69349)
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2020-06-01 09:40:52 -04:00
Hideki Saito
79ab798427 Enable logging using setup_loggers() API in dnf-4.2.17-6 or later
- Fixed issue #28061
- https://bugzilla.redhat.com/show_bug.cgi?id=1788212

Signed-off-by: Hideki Saito <saito@fgrep.org>
2020-05-30 20:37:27 -05:00
Rick Elrod
61a1fe1d7d
Fix network_cli test to use a trap for cleanup (#69762)
Signed-off-by: Rick Elrod <rick@elrod.me>
2020-05-29 15:45:33 -05:00
Martin Krizek
3606dcfe65
Ensure --version works with non-ascii project path (#66624)
Fixes #66617
2020-05-29 14:42:44 -04:00
Sloane Hertel
e40889e711
Add support to install collections from git repositories (#69154)
* Enable installing collections from git repositories

* Add tests for installing individual and multiple collections from git repositories

* Test to make sure recursive dependencies with different syntax are deduplicated

* Add documentation

* add a changelog

* Skip Python 2.6

* Only fail if no collections are located in a git repository

Add support for a 'type' key for collections in requirement.yml files.
Update the changelog and document the supported keys and allowed values for the type.

Add a note that the collection(s) in the repo must contain a galaxy.yml

* Add a warning about embedding credentials in SCM URLs

* Update with review suggestions

* suppress sanity compile failure for Python 2.6
2020-05-29 13:33:32 -04:00
Rick Elrod
8e19ab178a
connection plugins: try config, then play_context (#69751)
Change:
Rather than only using config, have base connection plugins fall back to
play_context.

Test Plan:
- Tested ansible-connection logic against an IOS device
- Tested -k against a VM
- CI

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-05-29 10:00:21 -05:00
Jordan Borean
f81f5da20e
Ansible.Basic added generic fragment merger for module options (#69719) 2020-05-29 16:11:38 +10:00
Felix Fontein
40f21dfd3c
Version source tagging (automatic and manual) for version_added and deprecation versions (#69680)
* Track collection for version_added.
Validate *all* version numbers in validate-modules.
For tagged version numbers (i.e. version_added), consider source collection to chose validation.

* Make tagging/untagging functions more flexible.

* Tag all versions in doc fragments.

* Tag all deprecation versions issued by code.

* Make Display.deprecated() understand tagged versions.

* Extend validation to enforce tagged version numbers.

* Tag versions in tests.

* Lint and fix test.

* Mention collection name in collection loader's deprecation/removal messages.

* Fix error IDs.

* Handle tagged dates in Display.deprecated().

* Also require that removed_at_date and deprecated_aliases.date are tagged.

* Also automatically tag/untag removed_at_date; fix sanity module removal version check.

* Improve error message when invalid version number is used (like '2.14' in collections).
2020-05-28 22:46:16 -07:00
Jordan Borean
f5f3ba7ab5
ansible-test - fix up relative util import for powershell validate-modules (#69753)
* ansible-test - fix up relative util import for powershell validate-modules

* Use different tactic for generic group

* Use python 2 and 3
2020-05-29 14:31:59 +10:00
Sam Doran
d7618712fd
blockinfile - fix line end regression (#69734)
PR #66461 introduced a regression that resulted in an in correct block in the file
if the block to be inserted did not end with a line separator. Fix this bug and add
tests to cover this scenario.
2020-05-27 16:54:38 -04:00
Riyad Preukschas
e5cc12a64f
blockinfile - Preserve line endings (#66461)
Fixes #64966
* Fix "TypeError: splitlines() takes no keyword arguments" on Python2.7
* Add changelog fragment
* Don't use `grep -P` for BSD/macOS compatibility
* Fix sanity checks complaining about test fixtures with mixed line endings
* Update changelogs/fragments/66461-blockinfile_preserve_line_endings.yaml
2020-05-27 11:05:07 -04:00
Rick Elrod
304c3e57e8
[ssh] Add new sshpass_prompt option (#68874)
Change:
Allows the user to configure sshpass (1.06+) to look for a different
substring than the default "assword" that it comes with.

Test Plan:
Set a custom ssh password prompt on a VM with PAM and tried connecting to
it. Without `ansible_sshpass_prompt` set in inventory: experienced hang.
With `ansible_sshpass_prompt` in inventory: connected successfully.

Tried setting `ansible_sshpass_prompt` with an older `sshpass` in PATH
and got a loud error, as expected.

Tickets:
Fixes #34722, fixes #54743, refs #11565.

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-05-26 19:58:31 -05:00
Felix Fontein
ea04e0048d
Allow to deprecate options and aliases by date (#68177)
* Allow to deprecate options and aliases by date instead of only by version.

* Update display.deprecate().

* Adjust behavior to conform to tested behavior, extend tests, and improve C# style.

* Parse date and fail on invalid date.

This is mainly to make sure that people start using invalid dates, and we eventually have a mess to clean up.

* C# code: improve validation and update/extend tests.

* Make sure that deprecate() is not called with both date and version.

* Forgot to remove no longer necessary formatting.

* Adjust order of warnings in C# code.

* Adjust unrelated test.

* Fix grammar (and make that test pass).

* Don't parse date, and adjust message to be same as in #67684.

* Sanity tests: disable date in past test.

* Validate-modules: validate ISO 8601 date format.

* Validate-modules: switch schema declaration for deprecated_aliases to improve error messages for invalid dates.

* Use DateTime instead of string for date deprecation.

* Validate that date in deprecated_aliases is actually a DateTime.

* Fix tests.

* Fix rebasing error.

* Adjust error codes for pylint, and add removed_at_date and deprecated_aliases.date checks to validate-modules.

* Make deprecation date in the past error codes optional.

* Make sure not both version and date are specified for AnsibleModule.deprecate() calls.

* Stop using Python 3.7+ API.

* Make sure errors are actually reported. Re-add 'ansible-' prefix.

* Avoid crashing when 'name' isn't there.

* Linting.

* Update lib/ansible/module_utils/csharp/Ansible.Basic.cs

Co-authored-by: Jordan Borean <jborean93@gmail.com>

* Adjust test to latest change.

* Prefer date over version if both end up in Display.deprecated().

Co-authored-by: Jordan Borean <jborean93@gmail.com>
2020-05-26 20:23:56 -04:00
Rick Elrod
4aff87770e
[yum] Make package removal confirmation strict (#69592)
* [yum] Make package removal confirmation strict

Change:
After removing packages, the yum module does a final check to ensure the
packages are really installed. The check would include packages that
were RPM `Provides:` values of another package.

This means that, for example, if a third-party kernel RPM spec had
`Provides: kernel` in it, removing the stock kernel would be successful
but the check to see if it was really removed would fail and cause
Ansible to report a failure.

Test Plan:
Tested on local CentOS 7 VM with kernel from elrepo which is known to
`Provides: kernel`.

Tickets:
Fixes #69237
Refs #35672
Refs #40723

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-05-26 14:47:39 -04:00
Matt Davis
f7dfa817ae
collection routing (#67684)
* `meta/` directory in collections
* runtime metadata for redirection/deprecation/removal of plugin loads
* a compatibility layer to keep existing content working on ansible-base + collections
* a Python import redirection layer to keep collections-hosted (and otherwise moved) content importable by things that don't know better
* supported Ansible version validation on collection loads
2020-05-26 09:42:06 -07:00
Rick Elrod
fe9696be52
Fix ansible -K become_pass regression (#69629)
* Fix `ansible -K` become_pass regression

Change:
- This fixes a breaking change introduced in
  2165f9ac40

Test Plan:
- Local VM for now, with plans to add an integration test for -K going
  forward.

Tickets:
Refs #69244
2020-05-22 09:34:26 -04:00
Brian Coca
de3f7c7739
fix delegated interpreter discovery (#69604)
* fix delegated interpeter
* allow returning fact if it is 'the right host'
* added note for future fix/efficiency
 as it stands we rerun discovery for the delegated host
unless its saving facts to itself
 * fixed test lacking delegate_to mock
2020-05-22 09:31:34 -04:00
James Cammarata
a4072ad0e9
Split regular and handler results into their own queues (#69498)
When mixed with the free strategy (or any custom strategy that does not behave in
a lock-step manner), the linear methodology of _wait_on_handler_results may cause
race conditions with regular task result processing if the strategy uses
_process_pending_results directly. This patch addresses that by splitting the queues
used for results and adding a flag to _process_pending_results to determine which
queue to check.

Fixes #69457
2020-05-21 15:55:08 -05:00
Sam Doran
cd8920af99
file - return 'state': 'absent' when a file does not exist (#66503)
This was changed in ansible/ansible#51350.
2020-05-21 16:35:45 -04:00
Matt Martz
e0f25a2b1f
Strip no log values from module response keys. Fixes #68400 (#69653) 2020-05-21 16:17:57 -04:00
Brian Coca
9281148b62
correctly merge multiple facts results (#68987)
* correctly merge multiple facts results

  fixes #68532
2020-05-20 18:53:37 -04:00
Sam Doran
fe941a4045 Pin Docker version at 19.03.1 2020-05-20 15:10:41 -07:00
Matt Martz
b748edea45
Add multipart/form-data functionality (#69376)
* Add multipart/form-data functionality

* Fix some linting issues

* Fix error message

* Allow filename to be provided with content

* Add integration test

* Update examples

* General improvements to multipart handling

* Use prepare_multipart for galaxy collection publish

* Properly account for py2 vs py3, ensuring no max header length

* Address test assumptions

* Add unit tests

* Add changelog

* Ensure to use CRLF instead of NL

* Ignore line-endings in fixture

* Consolidate code, add comment

* Bump fallaxy container version

* ci_complete
2020-05-20 15:44:01 -05:00
Martin Krizek
ecd986006e
Properly handle unicode in safe_eval (#68576)
* Properly handle unicode in safe_eval

Fixes #66943

* Update lib/ansible/template/safe_eval.py

Co-authored-by: Sam Doran <sdoran@redhat.com>
2020-05-20 12:08:50 -04:00
Sam Doran
889da811d7
Pin docker-ce-cli version in tests (#69620)
Installing docker-ce has a dependency of docker-ce-cli. If the version of docker-ci-cli is not specified, it installs the latest version.
2020-05-20 11:38:01 -04:00
Brian Coca
df4e83deda
added 'task timeout' feature (#69284)
* added 'task timeout' feature


Co-authored-by: Abhijeet Kasurde <akasurde@redhat.com>
2020-05-19 17:20:08 -04:00
Jordan Borean
ecea15c508
Unify ansible-galaxy install -r (#67843)
* Unify ansible-galaxy install -r

* Minor nit fixes for docs

* Re-align warnings

* Fix up integration test

* Fix up test where no roles/collections were in file
2020-05-19 05:09:42 +10:00
Matt Clay
01e7915b0a Remove obsolete incidental_azure_rm_webapp test. 2020-05-15 22:23:17 -07:00
Matt Clay
ca6da5ad63 Remove obsolete incidental test. 2020-05-14 17:02:11 -07:00
Brian Coca
2165f9ac40
fix delegation vars usage (debug still shows inventory_hostname) (#69244)
* fix delegation vars usage and reporting

 - just pass delegated host vars + task vars to plugins
   and avoid poluting with original host vars
 - updated tests
2020-05-14 16:36:13 -04:00
Sloane Hertel
55e29a1464
ansible-galaxy - fall back to galaxy.yml when listing collections (#68925)
Fallback to galaxy.yml for listing collections that don't have a FILES.json or MANIFEST.json

Co-authored-by: Sam Doran <sdoran@redhat.com>
2020-05-14 12:28:08 -04:00
Brian Coca
87d9b49de2
Fix listing of colleciton plugins with symlinks (#69305)
* Fix listing of colleciton plugins with symlinks
2020-05-14 11:45:02 -04:00
Brian Coca
cedfe34619
added unvault lookup plugin (#69087)
* added unvault lookup plugin
2020-05-14 11:19:19 -04:00
Matt Martz
4c4406b2df
Flatten the directory hierarchy of modules (#68966)
* Flatten the directory hierarchy of modules

* Update ignore.txt, flatten units

* Update imports

* Completely flatten the modules directory

* Update sanity ignore

* Fix some sanity test ignores

* Fix relative import

* Fix docs builds without category

* ci_complete

* Clean up docs. ci_complete

* Adjust needs/file alias

* ci_complete

* fix hardcoded ping module paths

Co-authored-by: Matt Davis <mrd@redhat.com>
2020-05-13 19:14:53 -07:00
Brian Coca
0aa76503dc
avoid fatal tb on bad fqcn callback name (#69440) 2020-05-13 12:02:31 -04:00
Mark Goddard
aa36b02ede
Fix fileglob plugin with non-existent subdirectory (#69451)
Since Ansible 2.9.8, if the fileglob plugin is passed a path containing
a subdirectory of a non-existent directory, it will fail. For example:

lookup('fileglob', '/'): ok
lookup('fileglob', '/foo'): (non-existent): ok
lookup('fileglob', '/foo/bar'): (non-existent): FAIL

The exact error depends on Python 2 or 3, but here is the error on
Python 2:

    AttributeError: 'NoneType' object has no attribute 'endswith'

And on Python 3:

    TypeError: expected str, bytes or os.PathLike object, not NoneType

This change fixes the issue by skipping paths that are falsey before
passing them to os.path.join().

Fixes: #69450
2020-05-12 09:12:21 -04:00
Martin Krizek
9645304da3
Validate args for includes in handlers too (#57537) 2020-05-12 09:35:58 +02:00
Rick Elrod
de59b17c7f
Add Fedora 32 to CI (#69222)
Change:
Adds Fedora 32 to shippable and alters tests slightly for new Fedora.

Test Plan:
CI

Tickets:
Fixes #69230

Co-authored-by: Matt Clay <matt@mystile.com>
2020-05-08 18:59:43 -05:00
Toshio Kuratomi
049800c063 Fix metadata defaults
* Metadata defaults were not being set if only a few fields were missing.

* ansible-doc with no documentation and no status in metadata should
  return empty, just like if there was no documentation and no metadata
  at all.
2020-05-05 09:06:52 -07:00
Abhijeet Kasurde
8d43d79191
galaxy: Handle empty roles and collections (#69199)
Galaxy collection install command raised indexError,
when requirements.yml contains empty roles and collections.

This fix handles empty roles and/or empty collections.

Fixes: #68186

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2020-05-05 10:34:04 -05:00
Brian Coca
bbdf77a59f
preserve json parsing error (#58461)
* preserve json parsing error
* added test
2020-05-01 15:03:07 -04:00
Brian Coca
23ab8f37df
fix a-doc listing plugins and add tests (#68600)
* add docs listing tests
* added collection module docs test
* always safe_load
* force 'type consistency' for uniquing paths
* bytified
* use our json encoder

Co-Authored-By: Matt Clay <matt@mystile.com>
2020-04-30 17:20:56 -04:00
David Shrewsbury
1142faa213
Do not pass file mode during recursive copy on symlink files. (#69011)
* Do not pass file mode during recursive copy on symlink files.

The 'file' module cannot deal with mode=preserve. Do not pass that
mode to the module when 'preserve' is used.

* Fix changelog fragment filename
2020-04-30 14:22:16 -05:00
Rick Elrod
eb65a01598 Enable dnf integration tests on CentOS 8
Change:
We were only testing dnf on RHEL previously.
Test on CentOS 8 as well.

Test Plan:
Ran locally in docker.

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-04-29 20:51:28 -05:00
Rick Elrod
f6cfcba346 [dnf] Make behavior/errors compatible for new DNF
Change:
Extend the logic for custom error handling in the dnf module, so that on
newer DNF (such as DNF that ships with modern Fedora 31 container
images, and ships with RHEL 8.2) we report errors consistently with
older DNF.

Test Plan:
Ran dnf integration tests against an old Fedora 31 container image and a
brand new Fedora 32 container image; tess passed on both.

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-04-29 20:51:28 -05:00
Matt Martz
bc41dd4514
Handle non-ascii paths during role installation. Fixes #69133 (#69213) 2020-04-28 15:33:44 -05:00
Brian Coca
babac66f9c
Be strict about what is a boolean for keywords (#67625)
* be strict about what is a boolean for keywords

 - found and fixed typo in test , 'yes' != 'yes.'
2020-04-28 13:55:26 -04:00
Abhijeet Kasurde
7714f691eb wait_for simplehttpserver to come up online
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2020-04-28 09:43:07 -07:00
David Shrewsbury
ff47d3f766
Make sure collection is a list if a str is given (#69081)
* Make sure collection is a list if a str is given

* Call field validation early on collections

Because we are doing work on modifying the collections value before
it is actually validated, we can validate it ourselves early to make
sure the user supplies either a string or list. Dicts are not valid.

The new validation allows us to simplify the _ensure_default_collection()
function. And since the field is now static, we no longer need to specify
a default for it, which also allows us to simplify the function. Since
the default is now removed, we can also remove the sanity/ignore.txt entry
for collectionsearch.py.

New unit tests are added (and the existing one modified) that allow us to
make sure that we throw a parser error if a user specifies something other
than a string or list for the collections value everywhere it can be specified.

* Revert removing the collection default

The default is actually used, so restore it.

* Fix unit tests in test_helpers.py affected by early collection validation
2020-04-28 10:47:11 -05:00
Alexandre Chouinard
91bb5af688
Update ansible-galaxy to handle role requirements (#68288)
* Update galaxy role object to handle requirements

Co-Authored-By: Sandra McCann <samccann@redhat.com>
2020-04-28 10:42:57 -04:00
Joe Cropper
e8d3358427 Ignore GPG key checks on test repo
Disable GPG key checks when installing packages from the test repo.
2020-04-23 14:48:55 -07:00
Joe Cropper
e663391e77 Fix yum test case on ppc64le
This patch covers a few changes to get the yum test case working on ppc64le
CentOS.  Specifically we needed to enable the EPEL repository on CentOS
as well as ensure some of the architecture-specific tasks use the right
set of binaries during their test.
2020-04-23 14:48:30 -07:00
Jesse Pretorius (odyssey4me)
da390b297e [pip] Enable virtualenv_command to have arguments
Currently if virtualenv_command has arguments, then the
search for the binary in the path does not work so the
user has to specify the full path to it.

To allow arguments to be used without having to specify
the path to the binary, we split the module argument into
the command and anything after the first space.

This makes using this module argument more flexible and
user friendly.

Fixes: #52275
2020-04-23 18:13:30 +02:00
Rick Elrod
85bb804cda
Revert "Cap cryptography version for macOS openssl test (#69083)" (#69126)
This reverts commit 1e08bb7a6f.
2020-04-23 08:49:09 -05:00
Adam Miller
8b30360ca9
fixed handling of releasever for all known scenarios (#69057)
Signed-off-by: Adam Miller <admiller@redhat.com>
2020-04-23 09:02:00 +02:00
Rick Elrod
cace616aab
Filter BLACKLIST_EXTS in PluginLoader (#69029)
Change:
Rather than hardcoding .pyo and .pyc, filter on all BLACKLIST_EXTS in
the non-legacy logic of PluginLoader (_find_fq_plugin). The two harcoded
extensions are part of BLACKLIST_EXTS already and this simply adds the
rest of the blacklisted extensions to the check.

In addition, check .endswith() instead of an exact match of the suffix,
like everywhere else that uses BLACKLIST_EXTS. This allows for
blacklisting, for example, emacs's backup files which can appear after
any extension, leading to things like `foo.py~`.

Test Plan:
Ran `ansible-playbook` against a collection where a `foo.py~` module was
getting executed instead of `foo.py` which also appeared in the same
directory. `foo.py~` is no longer executed.

Tickets:
Fixes #22268
Refs #27235

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-04-22 17:20:12 -05:00
Matt Martz
da98fc267a
Allow a collection role to call a standalone role by default (#69102)
* Allow a collection role to call a standalone role by default. Fixes #69101

* tweaked changelog text

* Guard against NoneType

Co-authored-by: Matt Davis <nitzmahone@users.noreply.github.com>
2020-04-22 13:58:08 -05:00
rwagnergit
977b58740b
update ActionBase._low_level_execute_command to honor executable (#68315)
* update ActionBase._low_level_execute_command to honor executable

* adding changelog fragment

* renaming changelog fragment to .yml

* noop change to bump shippable

* adding raw_executable integration test

* copying aliases from raw

* removing blank lines

* skipping aix and freebsd

* noop to bump shippable

* moving tests to raw/

* removing become_method: sudo ; it doesn't work on AIX

* removing trailing blank line

* forcing become_method: su to try to get AIX to work

Co-authored-by: Rob Wagner <rob.wagner@sas.com>
2020-04-22 11:56:35 -05:00
Rick Elrod
1e08bb7a6f
Cap cryptography version for macOS openssl test (#69083)
Change:
New `cryptography` statically links an openssl that is too new for macOS
10.11, so limit to an older cryptography for now.

Test Plan:
Ran the test with `--remote osx/10.11` and it passed.

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-04-21 15:13:46 -05:00
Rick Elrod
d17a44b0c5
Add new arches/OSes for binary_modules tests (#69052) 2020-04-21 00:52:23 -07:00
David Shrewsbury
af44bd4ddd
Fix --start-at-task when skipping tasks with no name (#68951)
Using --start-at-task on a playbook with tasks with no name would fail
if those unnamed tasks were encountered before the targetted start task.
2020-04-21 09:39:17 +02:00
Matt Clay
a8a61829b8 Enable passing tests on power. 2020-04-20 10:07:54 -07:00
Rick Elrod
648b3d43d3
[copy] Add some test coverage for check_mode (#68895)
Change:
Adds some missing coverage for the copy module when `check_mode: True`.

Test Plan:
Ran test with --coverage and looked at the resulting report.

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-04-17 16:09:32 -05:00
Rick Elrod
44a471f6ba
Enable service integration tests for FreeBSD (#68980)
* Enable service integration tests for FreeBSD

Change:
Adds necessary rc file for freebsd, and gets tests passing for it.

Test Plan:
Ran test with `--remote freebsd/12.1` and `--remote freebsd/11.1`. Both
passed.

Signed-off-by: Rick Elrod <rick@elrod.me>

* Update test/integration/targets/service/tasks/rc_setup.yml

Co-Authored-By: Matt Clay <matt@mystile.com>

* fix up comment

Signed-off-by: Rick Elrod <rick@elrod.me>

Co-authored-by: Matt Clay <matt@mystile.com>
2020-04-17 10:41:08 -07:00
Brian Coca
d3cab602a5
Fix fileglob when using 'file*' vs 'stuff/file*' (#68945)
* Fix fileglob when using 'file*' vs 'stuff/file*'

 when not having dir in glob, files/ subdir was being ignored.

* tests for fileglob
2020-04-17 09:51:05 -04:00
Martin Krizek
acdc9eb76d
native types: literal_eval all the things (#68938)
With https://github.com/pallets/jinja/pull/1190 merged our short-circuit
is no longer valid (has it ever been?) as now data like ' True ' may go
through our ansible_native_concat function as opposed to going through
intermediate call to Jinja2's native_concat before. Now we need to always
send data through literal_eval to ensure native types are returned.
2020-04-17 14:59:52 +02:00
Abhijeet Kasurde
3591451bc7
include_role: Strictly check string datatype for *_from (#68958)
Strictly check string datatype for 'tasks_from', 'vars_from',
'defaults_from', and 'handlers_from' in include_role

Fixes: #68515

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2020-04-17 10:57:41 +05:30
Brian Coca
ac509d489b
Revert "stricter permissions on atomic_move when creating new file (#68970)" (#68983)
This reverts commit 566f2467f6.
2020-04-16 12:52:15 -04:00
Brian Coca
566f2467f6
stricter permissions on atomic_move when creating new file (#68970)
fixes #67794
  updated some tests that expected previous defaults
  CVE-2020-1736
2020-04-16 09:06:18 -04:00
Jordan Borean
702949e64c
windows - fix up env var quoting for unicode single quotes (#68968)
* windows - fix up env var quoting for unicode single quotes

* Add sanity ignore check for smart quotes
2020-04-16 20:24:23 +10:00
Matt Clay
77a916e763
Add more integration tests. (#68972) 2020-04-16 00:53:08 -07:00
Yanis Guenane
46d82179d8
Testing: Add support for CentOS Linux On Power platform (#68130)
* Testing: Add CentOS Linux On Power platform

* Add arch designation to remotes.

This avoids overloading the provider with the arch.

Also add a changelog entry.

Co-authored-by: Matt Clay <matt@mystile.com>
2020-04-15 16:22:17 -07:00
Sam Doran
cdb24e0078
ansible-galaxy - fix listing specific role and role description (#67409)
* ansible-galaxy - fix listing specific role

If the role was not in the first search path, it was reported as not found

* Properly display role description

Default to description to top level description, falling back to the description from within galaxy_info

* Display proper message when a role does not exist

* Add integration tests

* Use context manager

* BSD and macOS ruining all the fun
2020-04-15 15:26:45 -04:00
Martin Krizek
ff1ba39c8a
Prevent templating unused variables for {%include%} (#68749)
Fixes #68699
2020-04-14 10:27:02 +02:00
Sloane Hertel
d91658ec0c
subversion module - provide password securely when possible or warn (#67829)
* subversion module - provide password securely with svn command line option --password-from-stdin when possible, and provide a warning otherwise.
* Update lib/ansible/modules/source_control/subversion.py.
* Add a test.

Co-authored-by: Sam Doran <sdoran@redhat.com>
2020-04-13 10:21:10 -04:00
Matt Martz
d86d20a378
Fix label lookup in the default callback for includes (#68822)
* Replace included_file._args w/ included_file._vars

* Fix item value in output of include_tasks loop

Signed-off-by: Yadnyawalk Tale <ytale@redhat.com>

* Update tests for loop callback fix. Add changelog

Co-authored-by: Yadnyawalk Tale <ytale@redhat.com>

Fixes #65904
Fixes #66018
2020-04-09 13:27:58 -05:00
Rick Elrod
38b7ceb75a
Update docker.txt to use the OpenSUSE 15.1 container image (devel edition) (#68785)
* Update docker.txt to use the OpenSUSE 15.1 container image

Signed-off-by: Rick Elrod <rick@elrod.me>

* handle installing mysql on suse

Signed-off-by: Rick Elrod <rick@elrod.me>

* add changelog fragment

Signed-off-by: Rick Elrod <rick@elrod.me>

* Update changelogs/fragments/ansible-test-opensuse-15.1.yml

Co-Authored-By: Matt Clay <matt@mystile.com>

Co-authored-by: Matt Clay <matt@mystile.com>
2020-04-09 00:26:59 -07:00
Matt Clay
04edd77c42
Update tests to use RHEL 7.8. (#68787)
* Update tests to use RHEL 7.8.

Keeping support for RHEL 7.6 since collections are still using it.

* Fix tests for RHEL 7.7+ due to extras repo name change.
2020-04-08 21:09:52 -07:00
Brian Coca
ba87c225cd
fixed fetch traversal from slurp (#68720)
* fixed fetch traversal from slurp

  * ignore slurp result for dest
  * fixed naming when source is relative
  * fixed bug in local connection plugin
  * added tests with fake slurp
  * moved existing role tests into runme.sh
  * normalized on action excepts
  * moved dest transform down to when needed
  * added is_subpath check
  * fixed bug in local connection

fixes #67793

CVE-2019-3828
2020-04-08 14:28:51 -04:00
Matt Martz
087be1da50
Allow tasks to notify a fqcn handler name (#68213)
* Allow tasks to notify a fqcn handler name

* Add tests. Fixes #68181

* Add changelog fragment

* Add test to ensure handlers are deduped properly with fqcn, role, and just handler names

* Add some docs about new special vars
2020-04-08 11:36:38 -05:00
Matt Clay
148e83f832
Fix ansible-test submodule handling. (#68759)
* Refactor ansible-test integration test.
* Add env --list-files option.
* Add tests for collection files tracked by git.
* Fix ansible-test submodule usage on older git.
* Fix submodule directory detection as files.
* Improve handling of nested source control.
2020-04-08 01:15:49 -07:00
Rick Elrod
4916be24fd
[git] make force=True apply to git fetches (#68691)
Fixes #67972

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-04-06 15:25:24 -05:00
Rick Elrod
f50df67517
coverage for lookup_list (#68683)
* coverage for lookup_list

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-04-06 13:39:18 -05:00
Rick Elrod
82c60db49b
pip - Fix check_mode for prerelease packages (#68690)
* pip - Fix check_mode for prerelease packages

Fixes #68592.

Signed-off-by: Rick Elrod <rick@elrod.me>
Co-authored-by: Matt Martz <matt@sivel.net>
2020-04-06 11:18:48 -05:00
Jordan Borean
ae1cd27b57
WebRequest - Fix use_proxy: no on module options (#68603)
* WebRequest - Fix use_proxy: no on module options

* Fix up changelog fragment
2020-04-02 07:17:50 +10:00
Jordan Borean
6db66bcadd
Add relative module_util support for powershell (#68321)
* Add relative module_util support for powershell

* Added ansible-test classification support
2020-04-01 09:01:18 +10:00
Jordan Borean
a20a527014
ansible-galaxy - Fix tar path traversal issue during install - CVE-2020-10691 (#68596) 2020-04-01 06:39:02 +10:00
Rick Elrod
541384e7b7
coverage for lookup_config (#68566)
Signed-off-by: Rick Elrod <rick@elrod.me>
2020-03-31 09:57:50 -05:00
Matt Clay
7323d5dd0d Fix references to old egg-info directory. 2020-03-30 13:56:43 -07:00
Matt Clay
0fb5593abf
Add initial ansible-test tests for collections. (#68533) 2020-03-29 09:04:27 -07:00
Rick Elrod
835ad75a0a
add test coverage for core filters (#68518)
Also remove now-useless exception handling.
2020-03-27 17:18:27 -07:00
Matt Clay
6a7e438012 Fix code coverage in tests.
Code coverage < 5 must be used.

ci_coverage
2020-03-26 22:41:12 -07:00
Rick Elrod
2af76f16be
Unify yaml style in unarchive tests to be dict-style (refs #68272) (#68477)
Signed-off-by: Rick Elrod <rick@elrod.me>
2020-03-26 18:02:17 -05:00
Sloane Hertel
5945415398
fix using inventory and cache plugins in a collection (#56469)
* Allow custom inventory plugins and cache plugins

If _load_name is not set correctly the cache plugin can't load the documentation (which is also the arg spec)

Fix the existing inventory plugin in the collections tests

Add integration tests for using a cache plugin in a collection

* Set the attribute on the instance instead of the class

Deprecate importing custom CacheModules directly - they should use the cache_loader
2020-03-26 12:05:46 -04:00
Rick Elrod
0651e1c25f
Add some more test coverage for unarchive (#68272)
* Add some more test coverage for unarchive

This moves over (and slightly extends) coverage which was found in
incidental_flatpak_remote.

Signed-off-by: Rick Elrod <rick@elrod.me>

* add a group for testing too, user creation does not mean group creation on all platforms

Signed-off-by: Rick Elrod <rick@elrod.me>

* update the test group assert

Signed-off-by: Rick Elrod <rick@elrod.me>

* Fix style

Signed-off-by: Rick Elrod <rick@elrod.me>

* block/always

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-03-25 16:23:55 -05:00
Rick Elrod
11e678b1b0
coverage for filter/urls.py, including fallback to custom implementations (#68412)
Signed-off-by: Rick Elrod <rick@elrod.me>
2020-03-25 16:23:14 -05:00
Rick Elrod
01638e0ea2
[mathstuff] Get coverage into the 90% range (#68395)
Signed-off-by: Rick Elrod <rick@elrod.me>
2020-03-25 15:33:37 -05:00
Brian Coca
6452a82452
fix vault temp file handling (#68433)
* fix vault tmpe file handling

 * use local temp dir instead of system temp
 * ensure each worker clears dataloader temp files
 * added test for dangling temp files
 * added notes to data loader

CVE-2020-10685
2020-03-25 15:24:04 -04:00
Brian Coca
ecd66a6a6e
Action module xross coll (#68080)
* added test to verify xrs collection action/module

* added to runme

* sanity fixes

* corrected name

* avoid windoez
2020-03-25 11:57:53 -04:00
Jordan Borean
a2deeb8fa2
ansible-galaxy - add download option (#67632)
* ansible-galaxy - add download option

* Fix sanity issues and added integration tests

* Fix doc suggestions

* Added --pre option
2020-03-25 08:32:43 +10:00
Brian Coca
a9d2ceafe4
prevent ansible_facts injection (#68431)
- also only replace when needed
 - switched from replace to index
 - added test to verify bogus_facts are not accepted

CVE-2020-10684
2020-03-24 15:46:56 -04:00
Rick Elrod
97e5179745
Test installing a .deb that has deps, from a URL (#68332)
Improve coverage of the apt module and remove some incidental
coverage from incidental_lookup_rabbitmq.

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-03-24 13:59:42 -05:00
Matt Martz
8d8c73729f
Merge pull request #67093 from sivel/acd-content-dir
Add content dir where Ansible will look for the content provided by ACD
2020-03-23 23:30:40 +01:00
Rick Elrod
dbd8d0a492
Pull timezone's incidental file coverage into file tests (#68247)
* Remove some unreachable code in the file module

Remove some cases in file.py which are covered by conditionals a few
lines earlier. Remove the duplicate code which will never be hit.

Signed-off-by: Rick Elrod <rick@elrod.me>

* Restore incidental file coverage from timezone module

Signed-off-by: Rick Elrod <rick@elrod.me>

* Combine two conditionals, add a changelog entry

Signed-off-by: Rick Elrod <rick@elrod.me>

* Make new test syntax consistent, add two stat tests

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-03-23 15:28:53 -07:00
Matt Martz
d3ec31f8d5
Support pre-releases via new SemanticVersion (#68258)
* Support pre-releases via new SemanticVersion. Fixes #64905

* Don't treat buildmeta as prerelease

* Don't inherit from str and int

* Add helper method to try and construct a SemanticVersion from a LooseVersion

* Don't count major 0 as pre-release, it's different

* Guard against invalid or no version in LooseVersion

* return a bool

* Add integration tests for pre-release

* Fix up lingering issues with comparisons

* typo fix

* Always allow pre-releases in verify

* Move pre-release filtering into CollectionRequirement, add messaging when a collection only contains pre-releases

* Update changelog

* If explicit requirement allow pre releases

* Enable pre-releases for tar installs, and collections already installed when they are pre-releases

* Drop --pre-release alias, make arg name more clear

* Simplify code into a single line

* Remove build metadata precedence, add some comments, and is_stable helper

* Improve from_loose_version

* Increase test coverage

* linting fix

* Update changelog
2020-03-23 16:04:07 -05:00
Matt Martz
ee6413af47
Address fixme and handle filter/test errors for collections better (#68047)
* Address fixme and handle fitler/test errors for collections better. Fixes #66721

* Re-arrange code
2020-03-23 14:47:58 -05:00
Matt Clay
5a1b59adf9 Rename tests (#68356)
* Rename `tests` test to match plugin type.

* Rename `test_infra` test to avoid confusion.

This test target is not a test for test plugins.

* Rename `vars_prompt` test to avoid confusion.

* Update sanity ignores.
2020-03-23 11:14:21 -05:00
Matt Clay
a63dd92a45 Clean up network test config. 2020-03-23 11:14:21 -05:00
Matt Martz
26da443fd2 Address compat issue for collection loading on py26 (#68219)
* Address compat issue for collection loading on py26

* Move import_module shim to utils for compat across the codebase

* Enable collection tests on py2.6

* Update changelog fragment

* Simplify code using sys.moduls

* Move compat to module_utils/compat/importlib

* Add back errantly deleted newline

* Remove hack comment

Co-Authored-By: Matt Clay <matt@mystile.com>

Co-authored-by: Matt Clay <matt@mystile.com>
2020-03-23 11:14:21 -05:00
Matt Clay
41f6c73be0 Update ansible-test network test provisioning. (#68220)
* Update network provisioning defaults.

* Update network test requirements.
2020-03-23 11:14:21 -05:00
Matt Clay
9126ac53e3 Add dynamic and static include tests. (#68200)
* Add dynamic and static include tests.

* Update task names for junit output.
2020-03-23 11:14:21 -05:00
Matt Clay
09614d90ac Consolidate Windows tests to one group. 2020-03-23 11:14:21 -05:00
Matt Clay
5e28323f32 Fix incidental test stability. (#68125) 2020-03-23 11:14:21 -05:00
Ansible Core Team
e0e57b4beb migration final cleanup 2020-03-23 11:14:21 -05:00
Ansible Core Team
cc2feea51b Migrated to vyos.vyos 2020-03-23 11:14:21 -05:00
Ansible Core Team
40714dfca9 Migrated to openvswitch.openvswitch 2020-03-23 11:14:21 -05:00
Ansible Core Team
ab914b9ab6 Migrated to azure.azcollection 2020-03-23 11:14:21 -05:00
Ansible Core Team
7449ec1546 Migrated to google.cloud 2020-03-23 11:14:21 -05:00
Ansible Core Team
dd5a8dec21 Migrated to hetzner.hcloud 2020-03-23 11:14:21 -05:00
Ansible Core Team
36a545553e Migrated to dellemc_networking.os6 2020-03-23 11:14:21 -05:00
Ansible Core Team
bc40fe0b6f Migrated to dellemc_networking.os9 2020-03-23 11:14:21 -05:00
Ansible Core Team
0b633b3703 Migrated to dellemc_networking.os10 2020-03-23 11:14:21 -05:00
Ansible Core Team
02541a15b2 Migrated to awx.awx 2020-03-23 11:14:21 -05:00
Ansible Core Team
35063de590 Migrated to junipernetworks.junos 2020-03-23 11:14:21 -05:00
Ansible Core Team
675129ae19 Migrated to cisco.ucs 2020-03-23 11:14:21 -05:00
Ansible Core Team
3149db0869 Migrated to cisco.nxos 2020-03-23 11:14:21 -05:00
Ansible Core Team
ae8fb5e371 Migrated to cisco.mso 2020-03-23 11:14:21 -05:00
Ansible Core Team
8a3f3e41f8 Migrated to cisco.meraki 2020-03-23 11:14:21 -05:00
Ansible Core Team
5aa37733c3 Migrated to cisco.iosxr 2020-03-23 11:14:21 -05:00