Commit graph

10871 commits

Author SHA1 Message Date
Brian Coca
f8aa307659
add extra vars to inventory plugins (#70970)
* add extra vars to inventory plugins
* disable by default for backwards compat
* also allow extra vars for options
2020-08-19 12:56:28 -04:00
Rick Elrod
ea58d7c233
Make it so callback plugins can act on implicit/explicit meta tasks (#71009)
Change:
- Now sends meta tasks to the task start callback
- Lets callback plugins opt-in to receiving implicit tasks

Test Plan:
- New integration tests

Tickets:
- Indirectly fixes #71007 by allowing custom callbacks with this data

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-08-18 16:56:48 -05:00
Abhijeet Kasurde
4f993922c8
Add documentation about info/facts module development (#71250)
Fixes: #40151

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2020-08-18 15:56:43 -04:00
David Shrewsbury
f2f6c34632
Fix play stats when rescue block is a child block (#70922)
* check run state of current block only

* Add changelog and test

* Add test for issue 29047

* Fix for both tests

* blerg

* Change test messages

* fix tests

* Add multi-level block in rescue test case

* Add recursive rescue check and multi-level test

* Should probably run the new test

* ci_complete

* Merge new tests

* ci_complete
2020-08-18 11:20:55 +02:00
Matt Martz
6289570234
epoch can be a float with strftime filter. Fixes #71257 (#71314) 2020-08-17 13:46:49 -05:00
Edwin Hermans
eb8b3a8479
get_url - Allow checksum file to be local file:// url (#71205)
This would be a partial solution for #69364 in that the SHASUMS file can be downloaded and gpg verified but then used from the downloaded location to verify the get_url's file.
* Make checksum url parsing more explicit

Use urlsplit to test if the checksum string has a (currently tested and) supported url scheme.

* Fix whitespace
* Changelog fragment
* Added tests
* Fix typo in test setup
2020-08-17 12:21:15 -04:00
Lorenzo Castelli
a1a50bb3cd
systemd - supports new systemctl output message for chroot (#71197)
The message generated by systemctl has been updated in 9321e23c40, which requires a corresponding change in the systemd module.

In addition, this fixes the module when the SYSTEMD_OFFLINE environment variable is set.
2020-08-17 12:09:07 -04:00
Matt Martz
5821128995
Allow callbacks from forks (#70501)
* POC for supporting callback events that come from the worker

* linting fixes. ci_complete

* fix up units. ci_complete

* Try moving the sentinel put higher. ci_complete

* safeguards. ci_complete

* Move queue killing to terminate

* LINTING. ci_complete

* Subclass Queue, to add helper send_callback method

* Just use _final_q instead of adding another queue and thread

* Revert a few changes

* Add helper for inserting a TaskResult into the _final_q

* Add changelog fragment

* Address rebase issue

* ci_complete

* Add test to assert async poll callback from fork

* Don't use full path

* ci_complete

* Use _results_lock as a context manager

* Add new generic lock decorator, and use it with send_callback
2020-08-17 10:51:01 -05:00
Toshio Kuratomi
d816a5966e Force an upgrade to a newer version of antsibull 2020-08-14 13:01:29 -07:00
Florent PIGOUT
336c176ebc
Fix cron file regression (#71207) 2020-08-12 14:04:29 -05:00
Sam Doran
888be697cb
Revert "Change default file permissions so they are not world readable (#70221)" (#71231)
* Revert "Change default file permissions so they are not world readable (#70221)"

This reverts commit 5260527c4a.

* Revert "Fix warning for new default permissions when mode is not specified (#70976)"

This reverts commit dc79528cc6.
2020-08-12 12:29:04 -05:00
Matt Martz
c59472715a
Add support for non-yaml EXAMPLES (#71184)
* Add support for non-yaml EXAMPLES

* Make pattern raw string

* fmt tag should be the first non-whitespace line

* Add changelog fragment
2020-08-12 09:17:45 -05:00
Martin Krizek
b66d66027e
Skip literal_eval for string filters results in native jinja. (#70988)
Fixes #70831
2020-08-11 10:19:49 +02:00
Matt Martz
ae1291004e
Make sure that apache2-webdav related packages get upgraded (#71187) 2020-08-10 13:10:20 -04:00
Rick Elrod
80ad1a17df
Remove incidental_lvg target (#71174)
Change:
- This no longer has any incidental coverage.

Test Plan:
- Coverage report

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-08-10 00:34:34 -05:00
Rick Elrod
4f54fb8d43
Remove incidental azure tests (#71164)
Change:
- All incidental coverage from these tests has moved to intentional
  coverage.

Test Plan:
- CI

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-08-08 00:56:53 -05:00
Jordan Borean
3f22f79e73
Ensure -k is set to delegated hosts without a pass (#71136)
* Ensure -k is set to delegated hosts without a pass

* Fix up some broken tests

* Update task_executor.py

one possible fix, the other is updating winrm to normalize on 'password' like the other connection plugins

* Add alias for winrm and fix incorrect assumption

* Make sure aliases are used for keyword options

* Conditionally run test if sshpass is present, fix sanity

Co-authored-by: Brian Coca <bcoca@users.noreply.github.com>
2020-08-08 09:06:32 +10:00
Matt Martz
7c60dadb9a
Updates to Integration tests to pass against Alpine (#70946)
* Start of alpine testing

* More updates

* Add forgotten file

* remove debug

* Add alpine3

* equal

* group 4

* group 4

* group 5

* Try to decrease test length

* libuser only available in testing

* Remove debug

* Make loops target work on hosts without gnu date

* Enable alpine testing

* ci_complete

* Don't specify uid for creating test user

* ci_complete

* Re-sort docker completion

* use newer container image

* ci_complete

* fix indentation

Co-authored-by: Matt Clay <matt@mystile.com>

Co-authored-by: Matt Clay <matt@mystile.com>
2020-08-07 14:28:10 -05:00
Matt Martz
410b6a7109
Update integration tests to support rpmfluff-0.6 (#71155) 2020-08-07 14:09:09 -05:00
Felix Fontein
050841324c
ansible-test: bump acme test container version to 2.0.0 (#71097) 2020-08-07 09:34:52 -07:00
Sam Doran
6ac4439a6a
Drop "rhui-" prefix from RHEL repositories in AMI (#71130)
The repository names seem to have changed and no longer have the "rhui-" prefix
2020-08-06 16:07:35 -05:00
Felix Fontein
fbfc0f99eb
Make changelog tool be more strict about suffixes (#70909)
* Bump antsibull-changelog version.

* Flag all dotfiles, except .keep and .gitkeep.

* Enable ignoring other fragment extensions.
2020-08-06 10:49:39 -05:00
Toshio Kuratomi
fb144c4414
Update ansible doc formats (#71070)
* Fix tty_ify bugs and refactor

* Move tty_ify() and supporting attributes to the DocCLI class as that's
  the only thing using it.
* Add unittest for the code.
* Fix a bug where the substitution macros can be detected when they are
  a part of another word.
* Add support for L(), R(), and HORIZONTALLINE which were added to the
  website docs many years ago.

* Update test/units/cli/test_doc.py

Co-authored-by: Matt Clay <matt@mystile.com>

Co-authored-by: Matt Clay <matt@mystile.com>
2020-08-05 10:53:25 -07:00
Brian Coca
662d34b9a7
add jinja2 global to reserved names (#71088)
* add jinja2 global to reserved names

  also allow expansion by additional context provided from caller

  fixes #41955

Co-authored-by: Abhijeet Kasurde <akasurde@redhat.com>
2020-08-05 10:43:31 -04:00
Abhijeet Kasurde
504ef607f3
Misc typo fixes (#71089) 2020-08-05 10:16:44 +05:30
Brian Coca
f9af27c631
dont clobber facts in loop (#71032)
* dont clobber facts in loop

	fixes #71031
2020-08-04 15:41:42 -04:00
Rick Elrod
0d7c144ce4
Allow macOS ACLs to work for unpriv -> unpriv (#70785)
Change:
- Use `chmod +a` in the fallback chain to allow MacOS to use ACLs to
  allow an unprivileged user to become an unprivileged user.

Test Plan:
- CI, new tests

Tickets:
- Fixes #70648

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-08-04 14:32:48 -04:00
Matt Martz
4e27569347
Add more include and yaml parsing tests (#70506)
These additional tests should provide coverage for features currently tested by the postgres incidental tests.
2020-08-04 11:24:52 -07:00
Rick Elrod
cdcf0aa42a
Intentional coverage, incidental_azure_rm_resource (#71052)
Change:
- Adds some intentional coverage around PluginLoader for cases that
  incidental_azure_rm_resource covered.
- Specifically, modules starting with an underscore, and starting with
  an underscore but a symlink.

Test Plan:
- CI

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-08-04 12:53:31 -05:00
Rick Elrod
8af3079640
Add intentional coverage for incidental_lvg (#71043)
Change:
- Add hardware_facts test target which manually sets up some LVM devices
  and tests facts against them.

Test Plan:
- New integration tests

Tickets:
- Refs #71041 and #71042 both of which I discovered during this

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-08-04 12:52:38 -05:00
Rick Elrod
810a9a5593
find: Allow reading whole file for contains regex (#71083)
Change:
- Add a parameter `read_whole_file` which allows for reading the whole
  file when doing a `contains` regex search.
- This allows for (for example) matching a pattern at the very end of
  a file.

Test Plan:
- New integration tests

Tickets:
- Fixes #63378

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-08-04 12:49:45 -05:00
Martin Krizek
5ca3aec3c4
native types: properly handle Undefined in nested data (#68432) 2020-08-04 19:06:07 +02:00
Rick Elrod
3bec27dc34
Allow cache_timeout=0 to reach inv. cache plugins (#70977)
Change:
- Previously, `cache_timeout=0` was seen as falsey and not passed along
  to cache plugins as an option. Now only "nulley" values are not sent.

Test Plan:
- New integration tests

Tickets:
- Fixes #70702

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-08-04 11:54:28 -05:00
Jordan Borean
f6b3b4b430
ansible-galaxy - fix download for subdirs in SCM (#71005) 2020-08-04 11:10:00 -04:00
Matt Davis
86b6c4bbb6
fix collection package root location under pytest >=6.0.0 (#70963) 2020-08-03 17:31:32 -07:00
Matt Clay
b53e7f8720 Fix ansible-test self-test change classification.
Changes to sanity and unit tests now trigger the ansible-test self-test integration tests.

No changelog entry since this only affects tests for ansible itself and not collections.
2020-08-03 15:24:54 -07:00
Rick Elrod
7f62a9d7b5
Make cache adjudicator's flush call plugin flush (#70987)
Change:
- Previously CachePluginAdjudicator#flush only removed entries from the
  cache backend that it knew about by using them earlier. Now it calls
  the underlying plugin's flush() method.

Test Plan:
- New unit tests

Tickets:
- Fixes #68770

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-08-03 17:16:15 -05:00
Toshio Kuratomi
e10902d744
Add the ansible_2_10 intersphinx reference. (#71028)
There are links from the scenario guides to collections but collections
docs aren't built in testing or locally when we're on the devel branch.
Due to that we need to make sure those references resolve to the
production docsite.  We can use intersphinx to make sure that happens.

* The test for binary files wasn't reading enough of the file.

Checking for null bytes in the first 1024 bytes failed to diagnose the
ansible_2_10.inv file as binary
2020-08-03 12:31:36 -05:00
Sloane Hertel
991714b9d1
copy - redact 'content' from invocation in check mode (#71033)
* sanitize copy module invocation secrets in check mode
2020-08-03 11:41:45 -04:00
Rick Elrod
74c14c6743
Strip spaces in module names in explicit actions (#71040)
* Strip spaces in module names in explicit actions

Change:
- When an action is called like "action: copy foo=bar", strip spaces
  around the action name.
- This allows "action: copy        foo=bar" to work as expected.

Test Plan:
- New integration tests

Tickets:
- Fixes #62136

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-08-03 11:30:45 -04:00
Ganesh Nalawade
6160e82bf2
Update network integration auth timeout (#71057)
*  Incase of network integration test for connection local
   test the paramiko_ssh auth_timeout is the value of timeout
   under defaults section which is 10 seconds.
*  For slower connection 10sec timout value result in authentication
   timeout error hence increase the timeout value to 90 seconds
2020-08-03 19:28:30 +05:30
Rick Elrod
96aa81e8ff
Remove incidental_azure_rm_mariadbserver tests (#71037)
Change:
- Remove incidental test which has no remaining incidental coverage.

Test Plan:
- CI

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-08-01 15:34:35 -05:00
Sam Doran
f99f96ceb6
Fix unstable unarchive test (#71004)
* Add mode to copy tasks
* Fix unreliable test by ignoring errors
2020-07-30 19:25:45 -04:00
Martin Krizek
bf7276a4e8
Emit proper error for x in y when y is undefined (#70990)
Fixes #70984
2020-07-30 15:57:01 -04:00
Philip Douglass
ac5f3f8bef
unarchive - Check 'fut_gid' against 'run_gid' in addition to supplemental groups (#65666)
Add integration tests for unarchiving as unprivileged user
Break tasks into separate files for easier reading and maintenance

Create a user by specifying a default group of 'staff' for macOS.

The user module does not actually remove the user directory on macOS,
so explicitly remove it.

Put the removal tasks in an always block to ensure they always run

Co-authored-by: Philip Douglass <philip.douglass@amadeus.com>
Co-authored-by: Sam Doran <sdoran@redhat.com>
2020-07-30 15:28:05 -04:00
Sloane Hertel
54e2ae79e7
fix downloading collections in git repos and tar.gz artifacts (#70524)
* Fix downloading tar files

* Fix downloading SCM collections

* changelog
2020-07-30 13:55:29 -04:00
Sam Doran
e6bf202738
linux facts - return proper broadcast address (#64528)
* linux facts - return proper broadcast address

Check that the value being returned is actually a broadcast address

* Add tests

* Cleanup tests
2020-07-30 19:40:14 +02:00
Sam Doran
c4f442ed5a
facts - fix incorrect time for some date_time_facts (#70665)
The iso8601_micro and iso8601 facts incorrectly called now.utcnow(), resulting
in a new timestamp at the time it was called, not a conversion of the previously
stored timestamp.

Correct this by capturing the UTC timestamp once then calculating the local
time using the UTC offset of the current system.

* Use time.time() for getting the current time
* Convert from that stored epoch timestamp to local and UTC times
* Used existing timestamp for epoch time
* Add unit tests that validate the formate of the return value rather than an exact value since mocking time and timezone is non-trivial
2020-07-30 11:39:58 -04:00
Matt Clay
2e0097ada3 Fix ansible-test relative import analysis. 2020-07-30 08:05:28 -07:00
Sam Doran
dc79528cc6
Fix warning for new default permissions when mode is not specified (#70976)
Follow up to #70221
Related to #67794
CVE-2020-1736

When set_mode_if_different() is called with mode of 'None', ensure we issue
a warning about the change in default permissions.

Add integration tests to ensure the warning works properly.

* Fix tests
- actually use custom module 🤦‍♂️
- verify file permission on created files
- use remote_tmp_dir so we're ready for split controller
- improve test module so we can skip the call to set_fs_attributes_if_different()
- fix tests for CentOS 6
2020-07-30 00:05:29 -04:00
Rick Elrod
14dc4de424
Update docs for --tags default, and add some tests (#70939)
Change:
- Clarify that not passing `--tags` will cause `ansible_run_tags` to
  default to `["all"]`.
- Add some extra coverage around `ansible_run_tags`

Test Plan:
- New integration and unit tests

Tickets:
- Fixes #69619

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-07-29 17:16:57 -05:00
Jordan Borean
b1cb2553af
ansible-galaxy - fix fallback for AH searches (#70957) 2020-07-30 07:28:43 +10:00
Rick Elrod
20a7476064
git: verify, only use --raw when we need it (#70900)
Change:
- Allow older git to verify tags again
- Enable verification tests everywhere, even if most of them only work
  on newer git. Some of them work on older git and they test the --raw
  parameter.

Test Plan:
- Re-enabled subset of git tests

Tickets:
- Fixes #64469

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-07-29 10:45:51 -05:00
Brian Coca
f9c3c6cba6
Allow changed/failed mgmt on strategy actions (#70919)
* Allow changed/failed mgmt on strategy actions
2020-07-29 10:44:46 -04:00
Matt Clay
3a8ac62596 Cap pytest version to avoid relative import issue. 2020-07-28 14:51:39 -07:00
Sloane Hertel
5c8520fbd9
Fix verbosity for var_templating test (#70943)
* Fix verbosity for var_templating test

Display the subtests

* fix specificity and set ANSIBLE_BECOME_ALLOW_SAME_USER

Co-authored-by: Matt Martz <matt@sivel.net>

Co-authored-by: Matt Martz <matt@sivel.net>
2020-07-28 12:02:43 -05:00
Rick Elrod
7d32129efb
[dnf] show installations/removals in check_mode (#70892)
Change:
- Previously, we only showed that something would have changed, not what
  would have changed. This allows us to show what will chang as well.

Test Plan:
- Local RHEL8 VM
- New integration tests

Tickets:
- Fixes #66132

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-07-28 11:23:55 -04:00
Felix Fontein
7e2cc7db12
validate-modules: fix version_added validation for top-level, fix error codes (#70869)
* Also validate top-level version_added.

* Fix error code.

* Produce same version_added validation error in schema than in code (and stop returning it twice).

* Return correct error codes for invalid version_added for options and return values.

* Add changelog.

* Fix forgotten closing braket.

* Accept 'historical' for some top-level version_added.
2020-07-28 10:10:35 -05:00
Rick Elrod
be4be926c4
subversion Give subversion module a validate_certs option (#70890)
* Give subversion module a validate_certs option

Change:
- Add `validate_certs` option to subversion module. Defaults to off for
  backwards compatibility.

Tickets:
- Fixes #22599

Signed-off-by: Rick Elrod <rick@elrod.me>

* Update changelogs/fragments/22599_svn_validate_certs.yml

Co-authored-by: Abhijeet Kasurde <akasurde@redhat.com>

* test verify_certs codepaths

Signed-off-by: Rick Elrod <rick@elrod.me>

Co-authored-by: Abhijeet Kasurde <akasurde@redhat.com>
2020-07-27 14:18:54 -04:00
Amin Vakil
9d2982549d
dnf: Add nobest option (#70318)
* dnf: Add nobest option

* dnf: Fix indent, add nobest specifically to dnf not yum

* Add changelog for dnf: add nobest option

* dnf: Add nobest to yumdnf module argument_spec

* dnf: remove nobest from module paramaters in yumdnf.py

* dnf: Add test for nobest option

* dnf: Cleanup packages in nobest test at last

* dnf: Cleanup manually added repos in nobest test at last

* dnf: Remove dnf-plugins-core as well in nobest test

* dnf: Change nobest release version to 2.11

* Change changelog number according to change in PR number

* Change changelog number according to change in PR number
2020-07-27 12:02:07 +02:00
Brian Coca
1223ce656a
reset logging to INFO (#70878)
- due to CVE-2019-14846
 - also added comments and test to avoid 'oportunistic' reversion
2020-07-24 15:53:17 -04:00
Matt Clay
8b96caf712
Update ansible-base test container to 1.5.0. (#70856) 2020-07-23 20:49:57 -05:00
Jordan Borean
5e1a968983
Do not add connection vars to the output results (#70853)
* Do not add connection vars to the output results

* Also revert the delgated scenario JIC

* Added regression test
2020-07-24 09:53:08 +10:00
Matt Davis
4c0af6c808
fix internal cases of actions calling unqualified module names (#70818)
* fix internal cases of actions calling unqualified module names

* add porting_guide entry
* misc other fixes around action/module resolution broken by redirection

ci_complete

* Update docs/docsite/rst/porting_guides/porting_guide_2.10.rst

Co-authored-by: Rick Elrod <rick@elrod.me>

* Update docs/docsite/rst/porting_guides/porting_guide_2.10.rst

Co-authored-by: Rick Elrod <rick@elrod.me>

* address review feedback

* pep8

* unit test fixes

* win fixes

* gather_facts fix module args ignores

* docs sanity

* pep8

* fix timeout test

* fix win name rewrites

Co-authored-by: Rick Elrod <rick@elrod.me>
2020-07-23 09:02:57 -07:00
Abhijeet Kasurde
e439194c8c
basic: Add name of aliases in error message (#69427)
With this patch, user will be notified with available aliases
of arg parameter.

Fixes: #58752

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2020-07-23 16:02:18 +05:30
Sam Doran
5260527c4a
Change default file permissions so they are not world readable (#70221)
* Change default file permissions so they are not world readable

CVE-2020-1736

Set the default permissions for files we create with atomic_move() to 0o0660. Track
which files we create that did not exist and warn if the module supports 'mode'
and it was not specified and the module did not call set_mode_if_different(). This allows the user to take action and specify a mode rather than using the defaults.

A code audit is needed to find all instances of modules that call atomic_move()
but do not call set_mode_if_different(). The findings need to be documented in
a changelog since we are not warning. Warning in those instances would be frustrating
to the user since they have no way to change the module code.

- use a set for storing list of created files
- just check the argument spac and params rather than using another property
- improve the warning message to include the default permissions
2020-07-22 17:05:38 -04:00
David Shrewsbury
bf98f031f3
Sanitize URI module keys with no_log values (#70762)
* Add sanitize_keys() to module_utils.

* More robust tests

* Revert 69653 change

* Allow list or dict

* fix pep8

* Sanitize lists within dict values

* words

* First pass at uri module

* Fix insane sanity tests

* fix integration tests

* Add changelog

* Remove unit test introduced in 69653

* Add ignore_keys param

* Sanitize all-the-things

* Ignore '_ansible*' keys

* cleanup

* Use module.no_log_values

* Avoid deep recursion issues by using deferred removal structure.

* Nit cleanups

* Add doc blurb

* spelling

* ci_complete
2020-07-22 14:49:37 -05:00
Brian Coca
84adaba6f5
Allow hostvars delegation (#70331)
* ensure hostvars are available on delegation
* also inventory_hostname must point to current host and not delegated one
* fix get_connection since it was still mixing original host vars and delegated ones
* also return connection vars for delegation and non delegation alike
* add test to ensure we have expected usage when directly assigning for non delegated host
2020-07-22 11:13:57 -04:00
Matt Martz
96b74d3e0b
Ensure single vaulted values aren't counted as sequences. Fixes #70784 (#70786) 2020-07-21 16:48:35 -05:00
Baptiste Mille-Mathias
0ead4306a8
user - Create home and parent directories only when requested (#70600)
The home user and the parents directories should only be created when
create_home == True
2020-07-21 15:50:44 -04:00
psi / Ryo Hirafuji
5ce47646ad
cron - Allow non-ascii (UTF-8) chars in cron file paths and jobs (#70426)
* Encode/Decode files in UTF-8
* Use helper function in ansible
* Add an integration test
* Use emoji in test data.
* add changelog
* Also support non-ascii chars in filepath and add tests about this.
* Also use non-ascii chars in replaced text and ensure not to break cron syntax.
* rename self.existing to self.n_existing
* rename crontab.existing to crontab.n_existing
2020-07-21 15:00:21 -04:00
Abhijeet Kasurde
4f96f9826c
distribution: Add support for DragonFly (#70748)
partially fixes #43739

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2020-07-21 13:47:11 -04:00
Rick Elrod
69472a5f8d
Refactor _fixup_perms2 to remove way-nested logic (#70701)
Change:
- Refactoring to make it harder to get wrong and easier to read.
- Generalize become_unprivileged tests and fix some that never worked
  but also never failed.

Test Plan:
- CI, new units/integration tests

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-07-20 18:46:47 -05:00
Brian Kohles
77d0effcc5
Fix missing quoting for remote_tmp in second mkdir of shell module. Issue #69577 (#69578)
* Fix missing quoting for remote_tmp in second mkdir of shell module. Issue #69577

* adding changelog

* fixing typo in changelog entry

* adding test case

Adding test case written by bmillemayhias.

* using $HOME instead of ~

* fixing commit measage

* Update 69578-shell-remote_tmp-quoting.yaml

Co-authored-by: Brian Kohles <me@briankohles.com>
2020-07-20 11:25:29 -05:00
Abhijeet Kasurde
c1402ddee0
validate_modules: added missing single quote (#70744)
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2020-07-20 09:26:17 -05:00
Matt Martz
f90aa5599f
Don't treat empty excludes as a match. Fixes #70640 (#70710) 2020-07-17 17:34:24 -04:00
Eduard Rozenberg
566c5e6ce1
Handle Slackware OS version strings containing a plus (“+”) (#68142)
A couple of years ago Slackware -current began using a plus (“+”) at the end of the distribution version string to indicate a future version work-in-progress.

Rearrange distribution_files unit tests to easily support more tests
  - add conftest with common fixtures 
  - use parametrize for testing multiple scenarios

* Add changelog
* Add unit tests for Slackware distribution parsing
* Use correct fixtures for Slackware
Data comes from /etc/slackware-version

Co-authored-by: Sam Doran <sdoran@redhat.com>
Co-authored-by: <Eduard Rozenberg <eduardr@pobox.com>>
2020-07-17 16:39:26 -04:00
Toshio Kuratomi
9dda393d70
Collections docs generation (#59761)
* Build documentation for Ansible-2.10 (formerly known as ACD).

Builds plugin docs from collections whose source is on galaxy

The new command downloads collections from galaxy, then finds the
plugins inside of them to get the documentation for those plugins.

* Update the python syntax checks
  * docs builds can now require python 3.6+.

* Move plugin formatter code out to an external tool, antsibull-docs.
  Collection owners want to be able to extract docs for their own
  websites as well.
* The jinja2 filters, tests, and other support code have moved to antsibull
* Remove document_plugins as that has now been integrated into antsibull-docs

* Cleanup and bugfix to other build script code:
  * The Commands class needed to have its metaclass set for abstractmethod
    to work correctly
  * Fix lint issues in some command plugins

* Add the docs/docsite/rst/collections to .gitignore as
  everything in that directory will be generated so we don't want any of
  it saved in the git repository
* gitignore the build dir and remove edit docs link on module pages

* Add docs/rst/collections as a directory to remove on make clean
* Split the collections docs from the main docs

* remove version and edit on github
* remove version banner for just collections
* clarify examples need collection keyword defined

* Remove references to plugin documentation locations that no longer exist.
  * Perhaps the pages in plugins/*.rst should be deprecated
    altogether and their content moved?
  * If not, perhaps we want to rephrase and link into the collection
    documentation?
  * Or perhaps we want to link to the plugins which are present in
    collections/ansible/builtin?

* Remove PYTHONPATH from the build-ansible calls
  One of the design goals of the build-ansible.py script was for it to
  automatically set its library path to include the checkout of ansible
  and the library of code to implement itself.  Because it automatically
  includes the checkout of ansible, we don't need to set PYTHONPATH in
  the Makefile any longer.

* Create a command to only build ansible-base plugin docs
  * When building docs for devel, only build the ansible-base docs for
    now.  This is because antsibull needs support for building a "devel
    tree" of docs.  This can be changed once that is implemented
  * When building docs for the sanity tests, only build the ansible-base
    plugin docs for now.  Those are the docs which are in this repo so
    that seems appropriate for now.
2020-07-17 13:07:35 -07:00
Matt Davis
c616e54a6e
refactor Python module_utils locator (#70610)
* refactor Python module_utils locator

* no longer recursive
* embed special-case module code internally
* share common code between collections/not cases
* fixes #70134
* properly support subpackage redirection
* adds support for FQCN redirect targets used by migration (expands to FQ Python name)
* add tests

* add changelog
2020-07-16 17:57:47 -07:00
Sloane Hertel
8c213c9334
template connection variables accessed directly before using (#70657)
* template variables accessed directly when using them instead of FieldAttributes
2020-07-16 11:21:39 -04:00
Abhijeet Kasurde
055871cbb8
api: time.clock compatible code (#70650)
time.clock is removed in Python 3.8. Add time.clock
compatible code.

Fixes: #70649

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2020-07-16 12:24:22 +05:30
Sam Doran
9d27d7c8b1
ansible_runner test - Add constraints (#70667)
A recent updated to psutil, which is a dependency of ansible-runner, fails
to install on older versions of pip.

Commit with the breaking change:

  135628639b
2020-07-15 18:22:44 -04:00
Rick Elrod
8455bc6838
Some more coverage for csvfile (#70661)
* Some more coverage for csvfile

Change:
- Add a few tests

Test Plan:
- CI, new tests

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-07-15 14:44:52 -05:00
Rick Elrod
e1a33a6a84
inventory_hostnames lookup plugin -> 100% coverage (#70628)
Change:
- Add tests to get inventory_hostnames up to 100% coverage.

Test Plan:
- CI

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-07-15 11:31:48 -05:00
Rick Elrod
01e30993fd
Get subelements lookup plugin coverage to 100% (#70634)
Change:
- Add a bunch of tests to bump coverage to 100%

Test Plan:
- CI, new tests

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-07-15 11:00:57 -05:00
Mykola Grygoriev
fe86a93482
Add a new date_time fact to provide DST timezone
PR #70449
2020-07-14 18:22:51 +02:00
Jordan Borean
e22e103cdf
winrm - Added kinit_args to control the args for kinit calls (#70624) 2020-07-14 12:05:11 -04:00
Matt Martz
a77dbf0866
Allow single vault encrypted values to be used directly as module parameters. Fixes #68275 (#70607) 2020-07-14 11:56:26 -04:00
jabdr
0690b68bd3
Support datetime.date object in module result (#70595)
* Support datetime.date object in module result

Fixes #70583

* change blank lines for pep8 sanity test
2020-07-14 11:42:40 -04:00
Mark Chappell
8d160b1881
Remove remaining examples of 1.1.1.1 (#70552)
* Remove remaining examples of 1.1.1.1

* Update ec2_group.py
2020-07-14 11:19:59 -04:00
Matt Clay
1e02a201a6 Fix ansible-test virtualenv management. 2020-07-14 01:26:06 -07:00
Matt Clay
09f02980a4 Update ansible-test change detection for plugins. 2020-07-13 18:28:59 -07:00
Rick Elrod
d372ce2c5d
Get m_u.facts.utils coverage up to 100% (#70614)
* Get m_u.facts.utils coverage up to 100%

Change:
- Add tests to 'gathering_facts' integration target to get
  module_utils.facts.utils coverage up to 100%.
- This also clears incidental coverage from incidental_selinux.

Test Plan:
- CI

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-07-13 16:52:56 -07:00
Rick Elrod
d16018fe72
Add intentional coverage for an async_wrapper case (#70593)
Change:
- Test async_wrapper when the module it runs has stderr output

Test Plan:
- CI
- Looked at coverage report and saw green for a few lines that weren't
  previously green.

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-07-13 11:58:14 -05:00
Rick Elrod
d5480572c8
varnames: add tests, fix exception grammar (#70573)
Change:
- Add integration tests for various cases
- Fix wrong use of "its" in an exception thrown in varnames when it
  throws an AnsibleError, given a term of the wrong type.

Test Plan:
- new tests, CI

Tickets:
- Fixes #70546

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-07-10 21:04:02 -05:00
Matt Clay
df45dcdae0
Add integration tests for test plugins. (#70576) 2020-07-10 18:48:51 -07:00
Brian Coca
cf89ca8a03
Make filter type errors 'loop friendly' (#70417)
- ensure we preserve the typeerror part of the exception so loop defereed error handling
 can postpone those caused by undefined variables until the when check is done.
 - fix tests to comply with the 'new normal'

 - human_to_bytes and others can issue TypeError not only on 'non string'
 but also bad string that is not convertable.

Co-authored-by: Sloane Hertel <shertel@redhat.com>

Co-authored-by: Sloane Hertel <shertel@redhat.com>
2020-07-10 18:49:57 -04:00
Felix Fontein
24dcaf8974
plugin loader: return collection name; ansible-doc: handle ansible.builtin correctly (#70026)
* Determine collection in plugin loader.

* Fix test.

* Use PluginPathContext objects in PluginLoader._plugin_path_cache instead of tuples.
2020-07-11 00:24:08 +02:00
Rick Elrod
1b4fd23ba6
csvfile: use parse_kv() for args, add tests (#70550)
Change:
- Use parse_kv() for parsing in the csvfile lookup plugin. This allows
  us to handle multi-word search keys and filenames. Previously, the
  plugin split on space and so none of these things worked as expected.
- Add integration tests for csvfile, testing a plethora of weird cases.

Test Plan:
- New integration tests, CI

Tickets:
- Fixes #70545

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-07-10 16:21:03 -05:00
Felix Fontein
f4c89eab23
ansible-doc: include collection name in text output (#70401)
* ansible-doc: include collection name in text output

* Be more careful to not accidentally pass ansible.builtin for user-supplied modules.
2020-07-10 22:52:47 +02:00
Amin Vakil
9c40b1b2ff
Replace filename with file in apt_key (#70492)
* Replace filename with file from apt_key check

  one is internal variable, the other is actual parameter used and required for parameter check.
2020-07-10 12:42:49 -04:00
Rick Elrod
9fbd65958d
hostname: hostnamectl check -> SystemdStrategy (#70532)
Change:
- Move hostnamectl check out of GenericStrategy because it was incorrect
  for everything except the SystemdStrategy which is where it belongs.
- Add some initial tests for the hostname module, though we are limited
  by the fact that we can't do much testing with it in containers.

Test Plan:
- new hostname integration tests

Signed-off-by: Rick Elrod <rick@elrod.me>

Co-authored-by: Sam Doran <sdoran@redhat.com>
2020-07-09 20:04:23 -05:00
Matt Clay
bbd8f15a58 Correct name of fileglob lookup integration test. 2020-07-09 12:28:06 -07:00
Mykola Grygoriev
71c378e139
Fix decrypt argument in assemble module (#70465)
* Do not pass decrypt parameter to assemble module

* Add integration tests where decrypt=True

* Add changelog #70465
2020-07-09 15:24:12 -04:00
Alan Rominger
b0d9deeae3
Fix ansible-test error in community.aws (#70507)
* Fix ansible-test error in community.aws

* Add changelog entry for fix

* Change check from None to string_types

* Update changelogs/fragments/70507-validate-null-author.yaml

clarify wording "or a list of strings"

Co-authored-by: Felix Fontein <felix@fontein.de>

* Update test/lib/ansible_test/_data/sanity/validate-modules/validate_modules/schema.py

clarify wording - single string or not specified valid

Co-authored-by: Felix Fontein <felix@fontein.de>

* Do not fail but return None when given outside list

Co-authored-by: Felix Fontein <felix@fontein.de>
2020-07-08 15:17:12 -05:00
David Shrewsbury
8aca464b8b
Make sure ansible_become treated as a boolean (#70484)
* Make sure ansible_become treated as a boolean
2020-07-08 14:53:38 -04:00
Rick Elrod
f3ef4ed076
Remove some constants.py deprecated items (#70466)
Change:
- Remove mk_boolean
- Remove BECOME_METHODS
- Remove get_config

Test Plan:
- CI, removed mk_boolean unit tests

Tickets:
- Fixes #69678

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-07-07 18:41:19 -05:00
Rick Elrod
41414ed475
Deprecate command warnings feature (#70504)
Change:
- The command warnings feature which suggests that users use modules
  instead of certain commands is now deprecated. Its `warn` paramater
  and `COMMAND_WARNINGS` configuration options are also deprecated.
  Their use will become an error in version 2.13.

Test Plan:
- CI

Signed-off-by: Rick Elrod <rick@elrod.me>
Co-authored-by: Sam Doran <sdoran@redhat.com>
2020-07-07 18:40:38 -05:00
Rick Elrod
688cd8657b
Fix string/bytestring comparsion in m_u.basic (#70439)
Change:
- module_utils.basic.is_special_selinux_path() used a string ==
  bytestring comparison which returned False and made Ansible think that
  certain filesystems aren't, in fact, special-cased, when they should
  be. Ensure both sides of the == are bytestrings.

Test Plan:
- Added `copy` integration tests for this case.

Tickets:
- Fixes #70244

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-07-07 11:48:05 -04:00
Abhijeet Kasurde
f5037314e3
sanity: remove ansible-bad-function (#70431)
* Updated docs and ignore.txt
* Replaced with correct function calls

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2020-07-07 11:46:28 -04:00
Sloane Hertel
5b03267a1f
Add ansible-doc tests for documentation containing YAML anchors (#70436)
Co-authored-by: Tadej Borovšak <tadej.borovsak@xlab.si>
2020-07-07 08:33:55 -04:00
Felix Fontein
9164b96774
ansible-doc man formatter: fail with better error message when description isn't there (#70046)
* ansible-doc man formatter: do not crash when description isn't there.
* Change to report a better error message when description is not there.
* Add test.
2020-07-06 13:29:03 -04:00
Rick Elrod
1be78dbfc5
Updated bundled libraries (#70418)
Change:
- Update bundled six to 1.13 (last with py2.6 support)
- Make it pass lint
- Fix check to allow skipping over compat __init__.py files we authored
- Fix check to allow files that can't be updated for some reason

Test Plan:
- ansible-test sanity --docker
- CI

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-07-02 12:39:00 -05:00
Brian Coca
7a15a3a109
fix flatten handling of nulls/nones (#70141)
* fix flatten loop control issue (break -> continue)

fix issue #69012

(cherry picked from commit 2127be5ec5)

* fixed null break bug and added option to include

 fixes #69012
 fixes #69013

Co-authored-by: pseudocoder <borisovano@users.noreply.github.com>
Co-authored-by: David Shrewsbury <Shrews@users.noreply.github.com>
2020-07-01 18:52:03 -04:00
Matt Martz
ce570691e7
Test against galaxy_ng (#70303)
* Test against galaxy_ng

* Switch container image

* Remove redundant |default

* Re-enable

* Update image

* Update wording

* Don't use pulp as the container name
2020-07-01 16:20:03 -04:00
Rick Elrod
91aea92c62
Add ability to fallback to chgrp remote_tmp and its files. (#68627)
* Add ability to fallback to chgrp remote_tmp and its files.

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-07-01 15:16:56 -04:00
Rick Elrod
f7078c1f8f
Throw a prettier error in m_u.basic syslog (#70312)
Change:
- In certain situations, such as when the input string contains null
  bytes (\0), syslog.syslog will throw a TypeError. Handle that and
  fail_json instead.

Test Plan:
- New test
- ansible-test --docker centos[68] (for py2 and py3 respectively)

Tickets:
- Refs #70269

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-07-01 09:28:48 -05:00
Jordan Borean
40ce448657
ansible-test - do not validate blacklisted ps modules (#70376)
* ansible-test - do not validate blacklisted ps modules

* Update changelogs/fragments/validate-modules-ps-doc-blacklist.yaml

Co-authored-by: Matt Clay <matt@mystile.com>

Co-authored-by: Matt Clay <matt@mystile.com>
2020-07-01 07:59:53 +10:00
Abhijeet Kasurde
a8ae8f4d72
setup: Handle CIFS share with backward slash (#70006)
CIFS can be mounted using backward slash as well in /etc/fstab like

\\Windows\share /data/ cifs credentials=/root/.creds 0 0

Handle this condition while gather mount information in Linux.

Fixes: #48813

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2020-06-30 16:09:35 -04:00
Dmitriy Rabotyagov
8324db7cb5
Add new option for YUM4/DNF repositories (#70042)
Documentation reference:
  https://dnf.readthedocs.io/en/stable/conf_ref.html

Co-authored-by: Estelle Poulin <dev@inspiredby.es>
2020-06-30 15:48:36 -04:00
Abhijeet Kasurde
74bedab8a9
known_hosts: update documentation (#70342)
* Update documentation as per sanity tests
* Added example about custom SSH port in example section

Fixes: #29236

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2020-06-30 12:03:31 -04:00
psi / Ryo Hirafuji
4997063b4a
apt - add fail_on_autoremove option to avoid unintended package removals (#70056)
* Ensure not to remove existing packages while installing apt packages.
* Make all lines shorter than 160 characters
* Allow removing packages only when upgrading.
* Add integration tests
2020-06-30 09:53:14 -04:00
Martin Krizek
b05e00e99a
Fix delegate_facts with interpreter not being set (#70293)
Fixes #70168

ci_complete

Co-authored-by: Brian Coca <bcoca@users.noreply.github.com>
Co-authored-by: Matt Clay <matt@mystile.com>
2020-06-30 15:49:09 +02:00
Sloane Hertel
30e70f4b63
Handle post_validate templating errors and fix tests (#70240)
* Handle unexpected templating errors

* Fixes #70050

Fix up tests that weren't running and add tests for graceful templating error handling
2020-06-30 09:31:58 -04:00
Rick Elrod
cc2cee6980
Fix -vvvvv unicode error in executor.module_common (#70368)
Change:
- Fix a UnicodeDecodeError in executor.module_common that could get
  triggered with -vvvvv.

Test Plan:
- `ansible-test integration --docker centos7 module_utils -vvvvv`
  This would show the error previously, and no loner does after this
  patch.

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-06-29 18:02:03 -05:00
Laurent Coustet
e396715d7b
git - add single_branch option (#28465)
In some usecases, we want to be able to clone a single branch
of a repository, without using --depth (which implies --single-branch).

* Use branch name when available
  -  update description of parameter
  - consolidate branch or tag checking for easy reuse

* Add changelog
* Use static task imports rather than dynamic includes
* Add integration tests for single_branch
* Account for older versions of git
* Minor tweak to warnings

Co-authored-by: Laurent Coustet <laurent.coustet@clarisys.fr>
Co-authored-by: Sam Doran <sdoran@redhat.com>
2020-06-29 15:40:54 -04:00
Hideki Saito
2d59e548f6
Improve ansible-galaxy STDOUT messages for collections (#70040)
- Fix issue #70010
- Add installation successful message
- This feature targets "collection" sub-command and does not affect "role" sub-command

Signed-off-by: Hideki Saito <saito@fgrep.org>
2020-06-29 15:18:30 -04:00
s-hamann
e39a9bf583
Add support for Parrot Linux, a Debian derivate (#69158)
Co-authored-by: black <invalid>
2020-06-29 20:22:44 +05:30
Matt Clay
d6fb42d1c5 Add integration tests for basic.py _set_cwd.
These tests verify that AnsibleModule can be instantiated when cwd does not exist or is unreadable.
2020-06-26 16:40:52 -07:00
Rick Elrod
7fdd8fcfcb
pip tests, use py2 compat sampleproject fork (#70313)
Change:
- sampleproject has gone py3 only. Use a py2 compatible fork.

Test Plan:
- CI

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-06-25 22:38:17 -05:00
Martin Krizek
0cd07eb3fd
hash filter - fail when unsupported type is passed as an argument (#70292)
Fixes #70258
2020-06-25 14:55:21 -05:00
jctanner
b019029bf3
Add intentional unit tests for basic._set_cwd and common.dict_merge (#70283)
* Add unit tests for basic._set_cwd

* incidental coverage for dict_merge

* add test for async stderr inclusion
2020-06-25 14:45:13 -05:00
Abhijeet Kasurde
29169ae847
stat: Handle colon in filename (#70259)
Handle colon appearing in filename while parsing the mimetype and charset
using file command.

Fixes: #70256

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2020-06-25 14:21:50 -05:00
Abhijeet Kasurde
3fe48ecba2
Add collection path in CLI version info (#68633)
This will provide user default path of collection

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2020-06-25 01:58:56 -04:00
Matt Clay
74494a1908 Code cleanup for ansible-test coverage support. 2020-06-24 17:39:35 -07:00
Matt Clay
8d92df4537 Test netconf plugins in ansible-doc sanity test. 2020-06-24 15:53:32 -07:00
Sam Doran
a3c1ea886f
Revert commits related to delegated facts changes (#70272)
* Revert "Fix storing delegate_to facts (#70231)"

This reverts commit 88bb76f248.

* Revert "Fix storing local task_vars facts for the retry (#70171)"

This reverts commit eaf6086eea.
2020-06-24 15:36:51 -04:00
Felix Fontein
ebd20ddca6
Include changelogs/changelog.yaml in distribution. (#70260) 2020-06-24 11:38:20 -05:00
Matt Clay
262f93e22c Code cleanup in the import sanity test. 2020-06-23 23:09:00 -07:00
David Shrewsbury
dd07bdf22d
Fix galaxy role info bug to support multiple roles (#70148)
* Add changelog fragment
* Update changelog
2020-06-23 16:57:25 -04:00
Matt Clay
70c59423fc Pin ansible-test requirements for RHEL.
The `packaging` and `pyparsing` packages are now installed by `ansible-test` during provisioning of RHEL instances to match the downstream vendored versions.
2020-06-23 11:52:25 -07:00
Rick Elrod
0073ab5fa6
Nuke strategy.SharedPluginLoaderObj, depr. 2.11 (#70235)
* Nuke strategy.SharedPluginLoaderObj, depr. 2.11

Change:
- Nuke SharedPluginLoaderObj class
- Update tests (which seemingly didn't use it anyway)
- Changelog

Test Plan:
CI, grep

Signed-off-by: Rick Elrod <rick@elrod.me>

* Nuke from ignore.txt

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-06-23 13:09:26 -04:00
Rick Elrod
7584e145a9
Nuke _get_item() from callback, deprecated 2.11 (#70233)
Change:
- Remove _get_item() alias as it has been deprecated
- Update tests
- Remove relevant sanity curtailment
- Add changelog

Test Plan:
CI, grep

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-06-23 12:03:27 -04:00
Martin Krizek
88bb76f248
Fix storing delegate_to facts (#70231)
* Fix storing delegate_to facts
2020-06-23 12:02:16 -04:00
Rick Elrod
c9edb35652
Nuke systemd module user option, 2.11 deprecation (#70211)
Change:
Remove all references to the 'user' param in systemd module.

Test Plan:
CI and grep.

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-06-23 11:55:01 -04:00
Rick Elrod
9cfc76a221
Nuke with_ squashing, deprecated for 2.11 (#70209)
Change:
Removes with_* loop squashing and tests for 2.11

Test Plan:
CI, and grepped for with_items in package manager integration targets.
There might be some test cases in collections which need to stop testing
this behavior.

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-06-23 11:53:25 -04:00
Matt Clay
fa48678a08
Rename pylint plugin and add tests. (#70225)
* Renamed custom pylint plugin for unwanted names.
* Add integration tests for sanity test failures.
2020-06-22 20:25:35 -07:00
Matt Clay
4816bb4f43
More boilerplate fixes. (#70224)
* Fix boilerplate in hacking dir.
* Fix boilerplate in docs dir.
* Fix boilerplate in integration tests.
* Fix boilerplate in examples.
2020-06-22 19:05:30 -07:00
Matt Clay
234994fc07 Fix boilerplate in setup.py and lib/ansible/ dir. 2020-06-22 16:29:16 -07:00
Matt Martz
1fedb95e4b
Use libc wcwidth to calculate print width in display (#66214)
* Use libc wcwidth to calculate print width in display. Fixes #63105

* Remove errantly added blank lines

* Fixes

* Move setlocale, adjust tests to work around py2 oddity with characters following null

* Don't change cli stub

* emojis

* Remove to_text call

* Special accounting for deletions

* Add initialization function, expand tests, ensure fallback to len

* get_text_width requires text, ensure banner deals with it

* Handle setlocale errors

* Move variable decrement

* Remove unused import
2020-06-22 16:57:01 -05:00
Matt Clay
26e8c07f32 Remove incidental_lookup_rabbitmq test.
It is no longer contributing unique code coverage.
2020-06-22 14:33:03 -07:00
Matt Clay
98a0995fd0 Clean up unit test boilerplate. 2020-06-22 14:20:33 -07:00
Martin Krizek
eaf6086eea
Fix storing local task_vars facts for the retry (#70171)
* Fix storing local task_vars facts for the retry

Fixes #70168
2020-06-22 10:07:47 -04:00
Sam Doran
b2d6db7916
Rebalance CI groups to avoid macOS timeouts (#70126) 2020-06-19 16:40:51 -04:00
Sam Doran
86606d3ca7
Disable ansible-galaxy-collection test (#70177)
It it currently failing
2020-06-19 14:20:06 -04:00
Sam Doran
bc05415109
Only pass kwargs to our string checker not callable checkers (#70151)
Since only check_type_str() accepts extra param, only pass to our checker and
do not pass kwargs to custom checkers.

* Add unit tests
2020-06-19 09:52:05 -04:00
Abhijeet Kasurde
87406890cf
find: Specify elements type in paths (#70139)
Fixes: #35990

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2020-06-19 06:56:02 +05:30
Matt Clay
89afb60999 Fix transport defs in test connection plugins. 2020-06-18 11:50:51 -07:00
Matt Clay
c67efe0bd1 Avoid unnecessary comprehensions. 2020-06-18 11:50:31 -07:00
Matt Clay
1a0d8a51cd Fix yamllint sanity test line numbers.
Parse errors from libyaml now compensate for the offset of the documentation within a module.
2020-06-18 10:03:21 -07:00
Martin Krizek
0ed5b77377
Detect failure in always block after rescue (#70094)
* Detect failure in always block after rescue

Fixes #70000

ci_complete

* Add more tests
2020-06-18 11:03:36 -05:00
Matt Clay
feb163e0de
Improve consistency of ansible-test pylint results. (#70132)
* Disable too-many-ancestors pylint check.

This check was only enabled for ansible-test code.

In some cases this check counts 1 more ancestor in python 3.7+ than under python 3.6, making the check inconsistent and unsuitable for use in ansible-test.

* Disable additional pylint checks:

- import-error
- no-name-in-module

These checks were only enabled for ansible-test code.

These checks provide inconsistent results since they are dependent on available imports, which vary between environments.

It may be practical to enable these checks in the future if changes are made to ensure a consistent test environment for pylint.
2020-06-17 17:03:41 -07:00
Matt Clay
e3f3b4002e
Clean up ansible-test pylint configs. (#70129)
* Sort disabled pylint codes in config.
* Fix comments in pylint config.
* Add comments to pylint config.
* Prepare to sort pylint config.
* Fix pylint config indentation.
* Sort pylint good/bad names in config.
* Add changelog fragment.
2020-06-17 15:06:04 -07:00
Matt Martz
fa1fb2d13b
Test ansible-galaxy against pulp/pulp_ansible (#69605)
* Test galaxy cli against pulp

* linting fix

* Renames and small fixes

* Better handling for resetting pulp

* Clean up some things, add a comment

* I can't spell

* Bump fallaxy, use alternate pulp image

* Only reset pulp when we're are executing against pulp

* Update for updated pulp container

* Update some comments with correct URLs and typos

* Linting fix

* Pin pulp-fedora31 to a digest

* Address review comments for documentation
2020-06-17 16:06:07 -05:00
Rick Elrod
cc071cdea6
Bump devel to 2.11.0 (#70121)
Signed-off-by: Rick Elrod <rick@elrod.me>
2020-06-17 14:22:38 -05:00
Matt Clay
8152d8bc1a Disabled inconsistent pylint checks. 2020-06-17 10:10:17 -07:00
Monty Taylor
0133757d44 Add an author exception for OpenStack Ansible SIG
The current author line wants to match a github author id. But
some people, including the OpenStack project, do not use github,
and additionally do not claim individual ownership but instead
group ownership.

Since there are already a couple of hard-coded examples in the
regex, just add one more. Alternately we could come up with some
mechanism to indicate that the author is purposely not listing
a github id, but that seems a bit heavywight.
2020-06-17 08:21:14 -07:00
Matt Davis
29c6aae2fc
try to load unqualified plugins from whitelist (#70086)
* try to load unqualified plugins from whitelist

* necessary for backcompat loading of unqualified collectionized callback plugins redirected from <= 2.9 core
* also added de-duping from actual loaded name

* add tests

* add warning test

* group test script entries by topic

* shorten warning text grep because wrapping is dumb

* fix adhoc callback loading behavior

* collections pass over whitelist wasn't respecting `_run_additional_callbacks`
* adds regression tests for same

* avoid `grep -L` in tests since it breaks the world
2020-06-16 17:17:38 -07:00
Matt Clay
598786e16d Update ansible-test default test containers.
The main change is the upgrade to Python 3.9.0b3.
2020-06-16 16:50:04 -07:00
Matt Davis
3dd324b837
disable slow/unstable hetzner incidental tests (#70103)
* disable slow/unstable hetzner incidental tests

* zap disabled tests

* verified no incidental coverage loss
2020-06-16 16:30:56 -07:00
Matt Clay
d295b81920 Improve stability of postgresql tests. 2020-06-16 16:29:25 -07:00
Matt Clay
0faec62354 Remove temporary migration hack from CI scripts. 2020-06-16 11:25:39 -07:00
Matt Clay
f69b1846f9 Add pyparsing constraint to ansible-test.
The upcoming pyparsing 3 release will require Python 3.5 or later, see:

https://github.com/pypa/packaging/issues/313

Unfortunately pip 8.x and earlier versions do not support python version requirements, which is why this constraint is needed.
2020-06-16 10:38:16 -07:00
Matt Davis
de63cba7e8
transparent downstream vendoring (#69850)
* builtin downstream vendoring support

* allows downstream packagers to install packages to `ansible/_vendor` that will automatically be added to head of sys.path during `ansible` package load
* tests

* sort conflicting package names in warning text

* sanity fixes

* skip unnecessary comparison
2020-06-15 16:22:25 -07:00
Matt Clay
7641d32f8e
Fix ansible-test import sanity test issues. (#70084) 2020-06-15 15:09:15 -07:00
Sam Doran
d45cb01b84
ansible-galaxy - fix collection installation with trailing slashes (#70016)
If we fail to find a member when extracting a directory, try adding a trailing
slash to the member name. In certain cases, the member in the tarfile will
contain a trailing slash but the file name in FILES.json will never contain
the trailing slash.

If unable to find the member, handle the KeyError and print a nicer error.

Also check if a directory exists before creating it since it may have been
extracted from the archive.

Fixes #70009

* Add unit tests
* Use loop for trying to get members
2020-06-15 17:36:07 -04:00
Felix Fontein
80410f292b Fix missing allow_date=False. 2020-06-15 11:18:51 -07:00
Matt Davis
16796a6170
enable collections module_utils subpkg tests (#69957)
* fixed by collection loader rewrite, just needed to be reenabled
2020-06-15 09:34:25 -07:00
Felix Fontein
0617514176
Boolean error: fix deprecation date sanity check. (#70047) 2020-06-13 09:05:40 -04:00
Matt Clay
5b3719805e Update ansible-test test containers. 2020-06-12 09:19:18 -07:00
Felix Fontein
2fa65eb5e1
validate-modules: allow YAML dates in module documentation and meta/runtime.yml. (#70025) 2020-06-12 15:36:38 +01:00
John R Barker
37785255f2
validate ansible-base's and collections runtime.yml (#69742)
* Validate ansible-base & collection's runtime.yml

Add new test `runtime-metadata`

* Schema validation of file
* Error if a a legacy meta/routing.yml exist in a collection
* removal_date OR removal_version

* Add tombstone validation.

* Allow both ISO 8601 date strings and datetime.date objects (from YAML dates).

* Address review comments.

* Add metadata to test collection.

* Add requirements file.

Co-authored-by: Felix Fontein <felix@fontein.de>
Co-authored-by: Matt Clay <matt@mystile.com>
2020-06-12 15:34:34 +01:00
Matt Clay
9f49db1f99 Avoid use of deprecated junit-xml method. 2020-06-12 00:47:07 -07:00
Matt Clay
26f318d277 Ignore return code from antsibull_changelog lint. 2020-06-11 19:27:40 -07:00
Brian Coca
4c9d9dbb56
fix discovery on loop with delegation (#70013)
* fix discovery on loop with delegation

fixes #69963
2020-06-11 18:04:17 -04:00
Matt Clay
8ffaed00f8 Add Azure Pipelines support to ansible-test. 2020-06-11 14:57:42 -07:00
Matt Martz
4a4a11d282
Add mccabe complexity testing (#64623)
* Add mccabe complexity testing

* Make mccabe complexity an optional error

* Add mccabe to new sanity pylint requirements

* Add a changelog fragment.

Co-authored-by: Matt Clay <matt@mystile.com>
2020-06-11 13:02:06 -07:00
David Shrewsbury
0ae4dac65a
Fix copy module file perms with remote_src (#69993)
When using 'remote_src: yes' and 'mode: preserve', the code handling
the file modes has to be handled on the remote node because it's
the one that has access to the source files. This means that the
copy module itself must handle this, rather than the copy action
plugin (which is where all that logic exists). The copy module
handles this when we copy a single file over. But when it is a
directory as the src parameter value, the mode of the files
beneath it are not considered. Subdirectories are copied with
shutil.copytree() which will preserve permissions automatically.
Individual files are copied with shutil.copyfile() which does NOT
preserve permissions. We need to add some calls to shutil.copymode()
to correct that.

Note: This *always* retains individial file permissions. Specifying
a 'mode' other than 'preserve' when giving a source directory for
the 'src' param does not make sense so will be ignored in that case
only.

Fixes #69783

* Add changelog and test
2020-06-11 15:18:59 -04:00
Matt Clay
2251b239a3 Fix changelog sanity test config detection. 2020-06-11 11:57:58 -07:00
James Cassell
47d14a33bd
config: singular ANSIBLE_COLLECTIONS_PATH (#70007)
* config: singular ANSIBLE_COLLECTIONS_PATH

Every other *_PATH setting in ansible is singular, and the traditional
$PATH variable is also singular despite containing a list of
directories.  Let's be consistent both internally and with POSIX
tradition.

* update all ANSIBLE_COLLECTIONS_PATHS env references to be singular

* deprecate plural ANSIBLE_COLLECTIONS_PATHS setting
2020-06-11 11:40:13 -07:00
Felix Fontein
a114da80ee
Use antsibull-changelog instead of packaged changelog generator (#69313)
Replace the ansible-base changelog linting and generation tool with antsibull-changelog and make it available for linting collections. Previously changelog linting was limited to ansible-base.
2020-06-11 11:21:21 -07:00
Felix Fontein
8d93ba9120
Plugin/module docs: parse return values, add collection names in them (version_added_collection), and format them nicely in ansible-doc (#69796)
* Tag return value docs if they are a dict (and not str/None).

* Try to parse return docs as YAML.

* Properly dump return values in ansible-doc.

* Adjust plugin formatter.

* Add changelog fragment.

* Don't add 'default' for return values.

* Fix plugin_formatter.

* Only try to parse return docs if they are still a string.

* Add tests.

* Warn if RETURN cannot be parsed.

* Adjust tests. Also test for warning.

* if -> elif (otherwise EXAMPLE will be parsed too).

* Always parse return documentation, and fail if it is invalid YAML.

* Polishing.

* Mostly re-enable ansible-doc tests.

Listing from the local collection seems to be somewhat broken. I assume this
is why the test was disabled.

* Lint and make tests work with Python 2.

* Keep FQCNs in plugins (not modules), i.e. restore previous state.
2020-06-11 11:03:43 -07:00
Nilashish Chakraborty
9d6b0f2b03
Support removed_at_date in ansible-doc (#70002)
* Support removed_at_date in ansible-doc

Signed-off-by: NilashishC <nilashishchakraborty8@gmail.com>

Changes:
  * ansible-doc does not support `removed_at_date` and assumes that
    deprecated dict will either have `removed_in` or `version`. This
    results in ansible-doc (and hence "sanity --test=ansible-doc")
    failing for modules having only `removed_at_date`.

  * This patch adds support for `removed_at_date` and also gives it
    precedence over `removed_in` or `version`.

* Add tests and changelog

Signed-off-by: NilashishC <nilashishchakraborty8@gmail.com>
2020-06-11 13:18:18 -04:00
Gonéri Le Bouder
4869874337
ansible-test: vcenter initialize group/vmware (#70011)
Ensure the vcenter provider initialize the `module_defaults` of all
the vmware modules, not just `vmware_guest`.
The VMware CI relies on this for the authentication of the different
VMware modules.

The commit adjust `incidental_vmware_prepare_tests/tasks/init_vcsim.yml`.
The test-suite uses a copy of `vmware_guest` that is not in the
`group/vmware` group. As a result, we need to manually pass the
authentification parameter.
2020-06-11 12:21:44 -04:00
Brian Coca
ebb22655e4
implemented 'prefix' for file based cache (#69872)
* implemented 'prefix' for file based cache

Co-authored-by: s-hertel <shertel@redhat.com>
2020-06-11 10:40:25 -04:00
Matt Clay
6c2be04b6f Intercept code-smell tests using subprocess.
Some code-smell sanity tests for ansible-base use subprocess to invoke ansible commands.
Intercept these commands to make sure the correct script and python version are used.
2020-06-10 16:07:47 -07:00
Jordan Borean
d30fc6c0b3
galaxy - preserve symlinks on build/install (#69959)
* galaxy - preserve symlinks on build/install

* Handle directory symlinks

* py2 compat change

* Updated changelog fragment
2020-06-11 06:46:42 +10:00
Jordan Borean
a58fcde3a0
Use common ps sanity requirements file (#69992)
* Use common pssanity requirements file

* Fix up sanity ignore
2020-06-11 06:38:17 +10:00
Sam Doran
3815ed67d5
ansible-galaxy - Fix role info when role is not installed (#69924)
* ansible-galaxy - Fix role info when role is not installed

Only report the role not found if in offline mode, otherwise query the galaxy API
to get role information.

Fixes #69867

* Improve error message when role is not found in Ansible Galaxy
2020-06-10 15:36:24 -04:00
Kshitij Chawla
32c392e622
Raise Warning when import_playbook receives additional parameters (#64156)
* Extra whitespace around imported playbook filename are stripped.
* Corrected call for display.deprecated
* Changed warning type & added test
* Added auto verification of raised warning
* More accurate warning message
2020-06-10 13:22:40 -04:00
Matt Clay
a5679d4dd1
Additional ansible-test sanity adjustments. (#69974)
* Fix ansible-test check_pyyaml usage.
* Avoid unnecessary pip commands.
* Avoid pointless pip check.
* Only install ansible-test requirements as needed.
2020-06-10 09:39:25 -07:00
Matt Clay
dc9c1d202b Fix ansible-test pip filter on Fedora 32.
The original version works on CentOS 8 but not Fedora 32.
2020-06-10 09:37:01 -07:00
Felix Fontein
f1ab7cf0c6
Validate-modules: adjust test to fixes in #69977 (#69978) 2020-06-10 18:02:01 +10:00
Jordan Borean
15f07e7a6c
Fix up schema for collection deprecation (#69977)
* Fix up schema for collection deprecation

* Fix up modules using wrong name

* Another fix
2020-06-10 16:33:08 +10:00
Matt Clay
a81dd4f06a
Update ansible-test default containers. (#69819)
* Move ansible-test completion code.

* Fix a few type hints.

* Change docker completion based on context.

Collections now use version 2.0 of the default-test-container.
This is an updated version of the earlier 1.x default-test-container with ansible-base and cloud specific requirements removed.

Testing of ansible-base now uses version 1.0 of a new ansible-base-test-container.
This container is similar to the earlier 1.x default-test-container, but with unnecessary duplication of requirements across Python versions removed.

Collections which have tests that depend on requirements no longer present in the default test container should specify them in their test requirements files:

* tests/integration/requirements.txt
* tests/unit/requirements.txt

* Bump test container versions

Co-authored-by: Jordan Borean <jborean93@gmail.com>
2020-06-09 15:40:56 -07:00
Sloane Hertel
51f6d129cb
support hard coded module_defaults.yml groups for collections (#69919)
* Only allow groups which were hardcoded in module_defaults.yml

only load action groups from the collection if module_defaults contains a potential group for the action

* Fix tests using modules that override those whitelisted in lib/ansible/config/module_defaults.yml

Third party modules should not be using group/ - use the action name instead

* add externalized module_defaults tests

add the missing group and collections

ci_complete

Co-authored-by: Matt Davis <mrd@redhat.com>

* changelog

ci_complete

* Fix import in tests

ci_complete

* Update with requested changes

ci_complete

* don't traceback since we don't validate the contents of module_defaults

ci_complete

Co-authored-by: Matt Davis <mrd@redhat.com>
2020-06-09 15:38:57 -07:00
Felix Fontein
a862ff2d43
Deprecation revisited (#69926)
* Allow to specify collection_name separately for deprecation.

* Use new functionality in Ansible.

* Use new functionality in tests.

* Update tagging/untagging functions.

* Update pylint deprecated sanity test.

* Update validate-modules. Missing are basic checks for version_added (validate semantic version format for collections).

* Improve version validation. Re-add version_added validation.

* Make sure collection names are added to return docs before schema validation.

* Extra checks to avoid crashes on bad data.

* Make C# module utils code work, and update/extend tests.

* Add changelog fragment.

* Stop extracting collection name from potentially tagged versions/dates.

* Simplify C# code.

* Update Windows modules docs.

* Forgot semicolons.
2020-06-09 15:21:19 -07:00
Matt Clay
7bff3d312f
Split out and install sanity test requirements. (#69971)
* Split out sanity test requirements.

* Run each --venv test separately.

This provides verification that the requirements for each test are properly specified.

* Use a separate requirements file per sanity test.

* Skip setuptools/cryptography setup for sanity.

* Eliminate pyyaml missing warning.

* Eliminate more pip noise.

* Fix conflicting generate_pip_install commands.

* Add changelog fragment.
2020-06-09 13:38:36 -07:00
Brian Coca
ce199ef0e1
Add intg test to repro #36045 (add_host traceback) (#69912)
* Add intg test to repro #36045 (add_host traceback)

* fix raw_params usage in add_host

Co-authored-by: Adrian Likins <alikins@redhat.com>
2020-06-09 15:25:56 -04:00
Matt Clay
03f8bf46c3
Update ansible-test integration tests. (#69970)
* Extend ansible-test collection test.

It now runs `rstcheck` and `empty-init` tests.

* Add galaxy.yml to test collections.
2020-06-09 11:16:48 -07:00
Sam Doran
c41a160951
Update test images to version 1.16.0 (#66771) 2020-06-09 10:08:31 -04:00
Matt Martz
9667f221a5
Make AnsibleVaultEncryptedUnicode work more like a string (#67823)
* Make AnsibleVaultEncryptedUnicode work more like a string. Fixes #24425

* Remove debugging

* Wrap some things

* Reduce diff

* data should always result in text

* add tests

* Don't just copy and paste, kids

* Add eq and ne back

* Go full UserString copy/paste

* Various version related fixes

* Remove trailing newline

* py2v3

* Add a test that can evaluate whether a variable is vault encrypted

* map was introduces in jinja2 2.7

* moar jinja

* type fix

Co-Authored-By: Sam Doran <sdoran@redhat.com>

* Remove duplicate __hash__

* Fix typo

* Add changelog fragment

* ci_complete

Co-authored-by: Sam Doran <sdoran@redhat.com>
2020-06-08 16:30:14 -05:00
Matt Clay
86f1cba3a6 Minor cleanup for ansible-test integration test. 2020-06-08 11:06:23 -07:00
Matt Martz
c1c6f61a18
Auto unroll generators produced by jinja filters (#68014)
* Auto unroll generators produced by jinja filters

* Unroll for native in finalize

* Fix indentation

Co-authored-by: Sam Doran <sdoran@redhat.com>

* Add changelog fragment

* ci_complete

* Always unroll regardless of jinja2

* ci_complete

Co-authored-by: Sam Doran <sdoran@redhat.com>
2020-06-08 12:58:03 -05:00
Felix Fontein
840d3a9dd7
ansible-doc: properly handle suboptions (#69795)
* Specifically handle suboptions, contains, etc in ansible-doc.
2020-06-08 11:51:33 -04:00
Felix Fontein
3e566768c1 Fix broken validate-modules. 2020-06-06 11:00:10 -07:00
Matt Clay
06efc41fb0
Reduce pip output noise in ansible-test. (#69925) 2020-06-05 18:25:48 -07:00
Matt Clay
f733517e6e Support dates in metadata in import sanity test.
No changelog entry since this feature and the associated bug have not been included in a release.
2020-06-05 14:46:08 -07:00
Jordan Borean
112251c7f4
Bump fallaxy version (#69917) 2020-06-06 06:00:57 +10:00
Matt Davis
984216f52e
various deprecation, display, warning, error fixes for collections redirection (#69822)
* various deprecation, display, warning, error fixes

* Update lib/ansible/utils/display.py

Co-authored-by: Felix Fontein <felix@fontein.de>

* Update lib/ansible/utils/display.py

Co-authored-by: Felix Fontein <felix@fontein.de>

* Update lib/ansible/utils/display.py

Co-authored-by: Felix Fontein <felix@fontein.de>

* cleanup, test fixes

* add collection name to deprecated() calls

* clean up redirect entries from uncommitted tests

* fix dep warning/error header text to match previous

Co-authored-by: Felix Fontein <felix@fontein.de>
2020-06-05 00:28:40 -07:00
Matt Davis
d79b23910a
fix sys.modules cleanup and blacklist behavior (#69898)
* fix sys.modules cleanup and blacklist behavior

* fix map-as-generator py2/py3 issue
* clear path_importer_cache between runs

* sanity fix

* don't be stupid with moving target generators
2020-06-04 19:02:23 -07:00
Rick Elrod
efe103cdff
Make add_host be idempotent/show changed status (#69897)
Change:
- The `add_host` action now shows an accurate change status.

Test Plan:
- Added a plethora of integration tests.

Tickets:
Fixes #69881

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-06-04 21:01:56 -05:00
Brian Coca
062e780a68
starting metadata sunset (#69454)
* starting metadata sunset

 - purged metadata from any requirements
 - fix indent in generic handler for yaml content (whey metadata display was off)
 - make more resilient against bad formed docs
 - removed all metadata from docs template
 - remove metadata from schemas
 - removed mdata tests and from unrelated tests

Co-authored-by: Felix Fontein <felix@fontein.de>
Co-authored-by: Rick Elrod <rick@elrod.me>
2020-06-04 21:01:46 -04:00
Felix Fontein
2dbd5dc2ae
ansible-test: do not accept empty string as valid version number (#69816)
* Work around strange behavior of StrictVersion and SemanticVersion constructors that they accept an falsy value.

* Do not accept empty strings as versions.
2020-06-04 13:25:05 -07:00
Rick Elrod
3b00c161dc
Re-add documentation for yum "install_repoquery" (#69871)
Change:
- This was removed in 2014 in 122a7021bc.
- The option still exists and is enabled by default and can lead to user
  confusion when people aren't expecting packages (or updated
  dependencies for it) to get installed and they do.
- Add the option documentation back with a few notes to make it clear what
  is happening.

Test Plan:
N/A, no code change, just documentation

Tickets:
- Refs #69497

Signed-off-by: Rick Elrod <rick@elrod.me>

* Fix sanity errors

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-06-04 15:45:22 -04:00
Alex Schultz
247e43b252
Fix IncludedFile equality check (#69524)
In the case of a free style strategy, it is possible to end up with
multiple hosts trying to include from the same role, however the tasks
being included may be different with the use of tasks_from.  Previously
if you had two hosts that were included the same role when the
process_include_results function tries to determine if a included needs
to be run on a specific host, it would end up merging two different
tasks into which ever one was processed first.

This change updates the equality check to also check if the task uuid
associated with the IncludedFile is the same. The previous check only
checked if the task's parent uuid was the same. This breaks down when
both includes have the same parent.

    - hosts: all
      strategy: free
      gather_facts: false
      tasks:
        - include_role:
            name: random_sleep
        - block:
          - name: set a fact (1)
            include_role:
              name: set_a_fact
              tasks_from: fact1.yml
          - name: set a fact (2)
            include_role:
              name: set_a_fact
              tasks_from: fact2.yml
        - name: include didn't run
          fail:
            msg: >
              set_a_fact didn't run
              fact1: {{ fact1 | default('not defined')}}
              fact2: {{ fact2 | default('not defined') }}"
          when: (fact1 is not defined or fact2 is not defined)

Closes #69521
2020-06-04 10:41:46 -05:00
Brian Coca
22da903e9c
return changed for group_by (#32057)
* return changed for group_by

* added tests and fixed 'early registeration'
2020-06-03 12:46:40 -04:00
Andrew Klychkov
00ead98595
systemd: should fail in check_mode when service not found on host (#68136)
* systemd: should fail in check_mode when service not found on host
2020-06-02 11:52:12 -04:00
Dongsu Park
598e3392a9
Discover Flatcar Linux properly for hostname (#69627)
To avoid issues with Flatcar Container Linux being unable to be found,
detect Flatcar distro name especially for hostname, just like CoreOS
Container Linux was supported.

See also https://github.com/ansible/ansible/issues/69516
2020-06-02 18:41:53 +05:30
Jack
80f09efd03
do not return the body even if it failed (#69706)
* do not return the body even if it failed

* add some tests for this and rebase

* import test task

* ignore_errors when fails

Co-authored-by: Jack Zhang <jack.zhang@aspiraconnect.com>
2020-06-01 09:55:38 -04:00
Abhijeet Kasurde
723a904f4e
Handle disabled service units (#69349)
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2020-06-01 09:40:52 -04:00
Hideki Saito
79ab798427 Enable logging using setup_loggers() API in dnf-4.2.17-6 or later
- Fixed issue #28061
- https://bugzilla.redhat.com/show_bug.cgi?id=1788212

Signed-off-by: Hideki Saito <saito@fgrep.org>
2020-05-30 20:37:27 -05:00
Rick Elrod
61a1fe1d7d
Fix network_cli test to use a trap for cleanup (#69762)
Signed-off-by: Rick Elrod <rick@elrod.me>
2020-05-29 15:45:33 -05:00
Martin Krizek
3606dcfe65
Ensure --version works with non-ascii project path (#66624)
Fixes #66617
2020-05-29 14:42:44 -04:00
Sloane Hertel
e40889e711
Add support to install collections from git repositories (#69154)
* Enable installing collections from git repositories

* Add tests for installing individual and multiple collections from git repositories

* Test to make sure recursive dependencies with different syntax are deduplicated

* Add documentation

* add a changelog

* Skip Python 2.6

* Only fail if no collections are located in a git repository

Add support for a 'type' key for collections in requirement.yml files.
Update the changelog and document the supported keys and allowed values for the type.

Add a note that the collection(s) in the repo must contain a galaxy.yml

* Add a warning about embedding credentials in SCM URLs

* Update with review suggestions

* suppress sanity compile failure for Python 2.6
2020-05-29 13:33:32 -04:00
Karl Fogel
225ae65b0f
Make Python path warning say what it means to say (#69669) 2020-05-29 13:15:11 -04:00
Felix Fontein
0b82d4499e
ansible-test ansible-doc sanity test: also run with --json parameter (#69288) 2020-05-29 08:52:29 -07:00
Felix Fontein
4794b98f2a
Fix version_added test (#69756)
* Fix version_added test for changes in #69680.

* Fix error message.
2020-05-29 08:25:14 -07:00
Rick Elrod
8e19ab178a
connection plugins: try config, then play_context (#69751)
Change:
Rather than only using config, have base connection plugins fall back to
play_context.

Test Plan:
- Tested ansible-connection logic against an IOS device
- Tested -k against a VM
- CI

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-05-29 10:00:21 -05:00
Abhijeet Kasurde
7b3d9cbcb8
distribution: Added support for openEuler OS (#69324)
Detect os_family for openEuler OS as 'RedHat', instead of 'openEuler'.

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2020-05-29 12:43:16 +05:30
Jordan Borean
f81f5da20e
Ansible.Basic added generic fragment merger for module options (#69719) 2020-05-29 16:11:38 +10:00
Felix Fontein
40f21dfd3c
Version source tagging (automatic and manual) for version_added and deprecation versions (#69680)
* Track collection for version_added.
Validate *all* version numbers in validate-modules.
For tagged version numbers (i.e. version_added), consider source collection to chose validation.

* Make tagging/untagging functions more flexible.

* Tag all versions in doc fragments.

* Tag all deprecation versions issued by code.

* Make Display.deprecated() understand tagged versions.

* Extend validation to enforce tagged version numbers.

* Tag versions in tests.

* Lint and fix test.

* Mention collection name in collection loader's deprecation/removal messages.

* Fix error IDs.

* Handle tagged dates in Display.deprecated().

* Also require that removed_at_date and deprecated_aliases.date are tagged.

* Also automatically tag/untag removed_at_date; fix sanity module removal version check.

* Improve error message when invalid version number is used (like '2.14' in collections).
2020-05-28 22:46:16 -07:00
Jordan Borean
f5f3ba7ab5
ansible-test - fix up relative util import for powershell validate-modules (#69753)
* ansible-test - fix up relative util import for powershell validate-modules

* Use different tactic for generic group

* Use python 2 and 3
2020-05-29 14:31:59 +10:00
Felix Fontein
31bf3a5622
Deprecate module in collection: allow removal date in documentation, make validate-modules ensure version and date match (#69727)
* Allow to deprecate module by date in documentation.

* Make sure deprecation date/version match between module docs and meta/runtime.yml.

* Unrelated fix: don't compare deprecated module version to Ansible's version in collection.

* Allow documentation's removal version to be something else than fixed list of Ansible versions for collections.

* Linting.

* Allow to deprecate plugin options by date.

* Add changelog fragment for deprecation by date (also covers #68177).
2020-05-28 14:20:52 -07:00
Toshio Kuratomi
412af6ae91
Fix docs return formatting. (#69632)
Gets rid of the unknown field names in the return data.
Allows the plugin return docs to format under the new docs pipeline.
* Expect that package_facts will pass return-syntax-error now.
2020-05-28 15:23:30 -05:00
Sam Doran
d7618712fd
blockinfile - fix line end regression (#69734)
PR #66461 introduced a regression that resulted in an in correct block in the file
if the block to be inserted did not end with a line separator. Fix this bug and add
tests to cover this scenario.
2020-05-27 16:54:38 -04:00
Matt Clay
65c57a7f5b Update default-test-container to version 1.14. 2020-05-27 11:31:38 -07:00
Riyad Preukschas
e5cc12a64f
blockinfile - Preserve line endings (#66461)
Fixes #64966
* Fix "TypeError: splitlines() takes no keyword arguments" on Python2.7
* Add changelog fragment
* Don't use `grep -P` for BSD/macOS compatibility
* Fix sanity checks complaining about test fixtures with mixed line endings
* Update changelogs/fragments/66461-blockinfile_preserve_line_endings.yaml
2020-05-27 11:05:07 -04:00
Abhijeet Kasurde
564907d8ac
distribution: Refactor test_distribution_version testcases (#69300) 2020-05-27 14:01:31 +05:30
Rick Elrod
304c3e57e8
[ssh] Add new sshpass_prompt option (#68874)
Change:
Allows the user to configure sshpass (1.06+) to look for a different
substring than the default "assword" that it comes with.

Test Plan:
Set a custom ssh password prompt on a VM with PAM and tried connecting to
it. Without `ansible_sshpass_prompt` set in inventory: experienced hang.
With `ansible_sshpass_prompt` in inventory: connected successfully.

Tried setting `ansible_sshpass_prompt` with an older `sshpass` in PATH
and got a loud error, as expected.

Tickets:
Fixes #34722, fixes #54743, refs #11565.

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-05-26 19:58:31 -05:00
Felix Fontein
ea04e0048d
Allow to deprecate options and aliases by date (#68177)
* Allow to deprecate options and aliases by date instead of only by version.

* Update display.deprecate().

* Adjust behavior to conform to tested behavior, extend tests, and improve C# style.

* Parse date and fail on invalid date.

This is mainly to make sure that people start using invalid dates, and we eventually have a mess to clean up.

* C# code: improve validation and update/extend tests.

* Make sure that deprecate() is not called with both date and version.

* Forgot to remove no longer necessary formatting.

* Adjust order of warnings in C# code.

* Adjust unrelated test.

* Fix grammar (and make that test pass).

* Don't parse date, and adjust message to be same as in #67684.

* Sanity tests: disable date in past test.

* Validate-modules: validate ISO 8601 date format.

* Validate-modules: switch schema declaration for deprecated_aliases to improve error messages for invalid dates.

* Use DateTime instead of string for date deprecation.

* Validate that date in deprecated_aliases is actually a DateTime.

* Fix tests.

* Fix rebasing error.

* Adjust error codes for pylint, and add removed_at_date and deprecated_aliases.date checks to validate-modules.

* Make deprecation date in the past error codes optional.

* Make sure not both version and date are specified for AnsibleModule.deprecate() calls.

* Stop using Python 3.7+ API.

* Make sure errors are actually reported. Re-add 'ansible-' prefix.

* Avoid crashing when 'name' isn't there.

* Linting.

* Update lib/ansible/module_utils/csharp/Ansible.Basic.cs

Co-authored-by: Jordan Borean <jborean93@gmail.com>

* Adjust test to latest change.

* Prefer date over version if both end up in Display.deprecated().

Co-authored-by: Jordan Borean <jborean93@gmail.com>
2020-05-26 20:23:56 -04:00
Matt Davis
fa81cc6a0c
fix delegated loading when path is not a directory (#69713)
* find_module can't pop ImportError- we need to just translate to `None` since this is a normal condition with files on sys.path (eg `/usr/lib/python36.zip`)
* added test
2020-05-26 15:55:17 -07:00
Matt Clay
5fc01c0a1d
Use a custom build container on Shippable. (#69660)
* Use a custom build container on Shippable.

This supports key generation before git_sync, to avoid issues with pre-migration PRs.

If the node pool is switched to another version, a matching build container should be built and used.

* Don't kill our own custom container.
2020-05-26 14:05:46 -07:00
Rick Elrod
4aff87770e
[yum] Make package removal confirmation strict (#69592)
* [yum] Make package removal confirmation strict

Change:
After removing packages, the yum module does a final check to ensure the
packages are really installed. The check would include packages that
were RPM `Provides:` values of another package.

This means that, for example, if a third-party kernel RPM spec had
`Provides: kernel` in it, removing the stock kernel would be successful
but the check to see if it was really removed would fail and cause
Ansible to report a failure.

Test Plan:
Tested on local CentOS 7 VM with kernel from elrepo which is known to
`Provides: kernel`.

Tickets:
Fixes #69237
Refs #35672
Refs #40723

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-05-26 14:47:39 -04:00
Matt Davis
f7dfa817ae
collection routing (#67684)
* `meta/` directory in collections
* runtime metadata for redirection/deprecation/removal of plugin loads
* a compatibility layer to keep existing content working on ansible-base + collections
* a Python import redirection layer to keep collections-hosted (and otherwise moved) content importable by things that don't know better
* supported Ansible version validation on collection loads
2020-05-26 09:42:06 -07:00
Nilashish Chakraborty
dae3ba71a8
Add FQCN mappings for network facts modules (#69601)
* Add FQCN mappings for network facts modules

Signed-off-by: NilashishC <nilashishchakraborty8@gmail.com>

* Add unit tests

Signed-off-by: NilashishC <nilashishchakraborty8@gmail.com>

* update entry for ironware

Signed-off-by: NilashishC <nilashishchakraborty8@gmail.com>
2020-05-22 23:20:21 +05:30
Rick Elrod
fe9696be52
Fix ansible -K become_pass regression (#69629)
* Fix `ansible -K` become_pass regression

Change:
- This fixes a breaking change introduced in
  2165f9ac40

Test Plan:
- Local VM for now, with plans to add an integration test for -K going
  forward.

Tickets:
Refs #69244
2020-05-22 09:34:26 -04:00
Brian Coca
de3f7c7739
fix delegated interpreter discovery (#69604)
* fix delegated interpeter
* allow returning fact if it is 'the right host'
* added note for future fix/efficiency
 as it stands we rerun discovery for the delegated host
unless its saving facts to itself
 * fixed test lacking delegate_to mock
2020-05-22 09:31:34 -04:00
James Cammarata
a4072ad0e9
Split regular and handler results into their own queues (#69498)
When mixed with the free strategy (or any custom strategy that does not behave in
a lock-step manner), the linear methodology of _wait_on_handler_results may cause
race conditions with regular task result processing if the strategy uses
_process_pending_results directly. This patch addresses that by splitting the queues
used for results and adding a flag to _process_pending_results to determine which
queue to check.

Fixes #69457
2020-05-21 15:55:08 -05:00
Sam Doran
cd8920af99
file - return 'state': 'absent' when a file does not exist (#66503)
This was changed in ansible/ansible#51350.
2020-05-21 16:35:45 -04:00
Matt Martz
e0f25a2b1f
Strip no log values from module response keys. Fixes #68400 (#69653) 2020-05-21 16:17:57 -04:00
Martin Krizek
1ee70fc272
ansible.utils.vars.isidentifier improvements (#58278)
ci_complete
2020-05-21 19:54:40 +02:00
Brian Coca
9281148b62
correctly merge multiple facts results (#68987)
* correctly merge multiple facts results

  fixes #68532
2020-05-20 18:53:37 -04:00
Sam Doran
fe941a4045 Pin Docker version at 19.03.1 2020-05-20 15:10:41 -07:00
Matt Clay
8fe812b212
Remove conflicting plugin from test/support/ dir. (#69628) 2020-05-20 14:52:26 -07:00
Matt Martz
b748edea45
Add multipart/form-data functionality (#69376)
* Add multipart/form-data functionality

* Fix some linting issues

* Fix error message

* Allow filename to be provided with content

* Add integration test

* Update examples

* General improvements to multipart handling

* Use prepare_multipart for galaxy collection publish

* Properly account for py2 vs py3, ensuring no max header length

* Address test assumptions

* Add unit tests

* Add changelog

* Ensure to use CRLF instead of NL

* Ignore line-endings in fixture

* Consolidate code, add comment

* Bump fallaxy container version

* ci_complete
2020-05-20 15:44:01 -05:00
Martin Krizek
ecd986006e
Properly handle unicode in safe_eval (#68576)
* Properly handle unicode in safe_eval

Fixes #66943

* Update lib/ansible/template/safe_eval.py

Co-authored-by: Sam Doran <sdoran@redhat.com>
2020-05-20 12:08:50 -04:00
Sam Doran
889da811d7
Pin docker-ce-cli version in tests (#69620)
Installing docker-ce has a dependency of docker-ce-cli. If the version of docker-ci-cli is not specified, it installs the latest version.
2020-05-20 11:38:01 -04:00
Brian Coca
06ecdaa7b1
comment heavy playbook (#68981)
* create examples/play.yml


Co-authored-by: flowerysong <junk+github@flowerysong.com>
2020-05-19 17:27:08 -04:00
Brian Coca
df4e83deda
added 'task timeout' feature (#69284)
* added 'task timeout' feature


Co-authored-by: Abhijeet Kasurde <akasurde@redhat.com>
2020-05-19 17:20:08 -04:00
Sam Doran
f01de15d25
ansible-galaxy - remove warning during collection install (#69541)
* ansible-galaxy - remove warning during collection install

If existing collections do not contain a MANIFEST.json, which is common
for collections under development that were not installed from Ansible
Galaxy, fall back to inspecting galaxy.yml rather than displaying a
warning.

A warning will still be displayed in neither a MANIFEST.json nor
galaxy.yml are present.

* Update unit tests
2020-05-19 10:31:49 -05:00
Felix Fontein
43acd61901
ansible-test local change detection: use --base-branch if specified (#69508) 2020-05-18 13:18:31 -07:00
Jordan Borean
ecea15c508
Unify ansible-galaxy install -r (#67843)
* Unify ansible-galaxy install -r

* Minor nit fixes for docs

* Re-align warnings

* Fix up integration test

* Fix up test where no roles/collections were in file
2020-05-19 05:09:42 +10:00
Matt Clay
01e7915b0a Remove obsolete incidental_azure_rm_webapp test. 2020-05-15 22:23:17 -07:00
Matt Clay
c495c92a6e Code cleanup in ansible-test. 2020-05-15 18:17:11 -07:00
Matt Clay
159d79995c Freeze pycodestyle in ansible-test. 2020-05-15 16:38:50 -07:00
Matt Clay
e7c2eb519b
Add Shippable request signing to ansible-test. (#69526) 2020-05-15 15:38:02 -07:00
Lukas Pirl
34db57a47f
introduce fact "ansible_processor_nproc": number of usable vcpus (#66569)
This fact reflects the number of usable vcpus (which might be different
from ansible_processor_vcpus, e.g., in containers with limits). See
also #51504.

* Add fixture data and update unit tests

Co-authored-by: Sam Doran <sdoran@redhat.com>
2020-05-15 09:38:56 -04:00
Matt Clay
83381cd912 Update the default-test-container in ansible-test.
The updated container includes fewer requirements now that the collection migration has completed.

Collections which encounter test issues with this new container should update their test requirements files to include the necessary requirements.
2020-05-14 21:24:58 -07:00
Matt Clay
d0484fbf74 Cleanup for Python 3.9 and pycodestyle compat. 2020-05-14 21:06:16 -07:00
Matt Clay
d8e0aadc0d Update ansible-test support for CI providers.
Refactored CI provider code to simplify multiple provider support and addition of new providers.
2020-05-14 17:08:30 -07:00
Matt Clay
ca6da5ad63 Remove obsolete incidental test. 2020-05-14 17:02:11 -07:00
Brian Coca
2165f9ac40
fix delegation vars usage (debug still shows inventory_hostname) (#69244)
* fix delegation vars usage and reporting

 - just pass delegated host vars + task vars to plugins
   and avoid poluting with original host vars
 - updated tests
2020-05-14 16:36:13 -04:00
Sloane Hertel
55e29a1464
ansible-galaxy - fall back to galaxy.yml when listing collections (#68925)
Fallback to galaxy.yml for listing collections that don't have a FILES.json or MANIFEST.json

Co-authored-by: Sam Doran <sdoran@redhat.com>
2020-05-14 12:28:08 -04:00
Sloane Hertel
343ffaa18b
better error for "ansible-galaxy collection verify" if there's no MANIFEST.json (#67498)
* Add a better error for "ansible-galaxy verify" if the MANIFEST.json has been deleted from the installed collection or if the collection hasn't been installed via normal means

* Fix unit tests for the remote collection

If there's something wrong with the local collection's version it will fail before the remote collection is sought

* Add a test for the new error msg

* Prevent the duplicate warning

Mock the new isfile call where needed

* Update lib/ansible/galaxy/collection.py

Co-Authored-By: Martin Krizek <martin.krizek@gmail.com>

Co-authored-by: Martin Krizek <martin.krizek@gmail.com>
2020-05-14 12:25:19 -04:00
Toshio Kuratomi
25c5388fde Update compile skip; all release and docs build scripts require 3.6+ 2020-05-14 09:14:37 -07:00
Bob Weinand
f200487414
Fix filedescriptor out of range in select() when running commands (#65058)
* Fix filedescriptor out of range in select() when running commands

* Simplify the run_command() code

Now that we're using selectors in run_command(), we can simplify some of
the code.

* Use fileobj.read() instead of os.read()
* No longer use get_buffer_size() as we can just slurp all of the data
  instead.

Also use a simpler conditional check of whether the selector map is
empty

Co-authored-by: Toshio Kuratomi <a.badger@gmail.com>
2020-05-14 11:46:34 -04:00
Brian Coca
87d9b49de2
Fix listing of colleciton plugins with symlinks (#69305)
* Fix listing of colleciton plugins with symlinks
2020-05-14 11:45:02 -04:00
Brian Coca
cedfe34619
added unvault lookup plugin (#69087)
* added unvault lookup plugin
2020-05-14 11:19:19 -04:00
Matt Martz
4c4406b2df
Flatten the directory hierarchy of modules (#68966)
* Flatten the directory hierarchy of modules

* Update ignore.txt, flatten units

* Update imports

* Completely flatten the modules directory

* Update sanity ignore

* Fix some sanity test ignores

* Fix relative import

* Fix docs builds without category

* ci_complete

* Clean up docs. ci_complete

* Adjust needs/file alias

* ci_complete

* fix hardcoded ping module paths

Co-authored-by: Matt Davis <mrd@redhat.com>
2020-05-13 19:14:53 -07:00
Felix Fontein
0e15375ffe
Add deprecated removed_in_version and deprecated_aliases version tests (#66920) 2020-05-13 13:58:09 -07:00
Brian Coca
0aa76503dc
avoid fatal tb on bad fqcn callback name (#69440) 2020-05-13 12:02:31 -04:00
Matt Davis
776f4840fc
fix ansible-test units to work(ish) under podman (#69462)
* ignore the missing `Networks` key, issue a warning that network disconnection won't function
2020-05-12 10:46:22 -07:00
Mark Goddard
aa36b02ede
Fix fileglob plugin with non-existent subdirectory (#69451)
Since Ansible 2.9.8, if the fileglob plugin is passed a path containing
a subdirectory of a non-existent directory, it will fail. For example:

lookup('fileglob', '/'): ok
lookup('fileglob', '/foo'): (non-existent): ok
lookup('fileglob', '/foo/bar'): (non-existent): FAIL

The exact error depends on Python 2 or 3, but here is the error on
Python 2:

    AttributeError: 'NoneType' object has no attribute 'endswith'

And on Python 3:

    TypeError: expected str, bytes or os.PathLike object, not NoneType

This change fixes the issue by skipping paths that are falsey before
passing them to os.path.join().

Fixes: #69450
2020-05-12 09:12:21 -04:00
Martin Krizek
9645304da3
Validate args for includes in handlers too (#57537) 2020-05-12 09:35:58 +02:00
Felix Fontein
5b9418c06c
Fix SemanticVersion comparison (#69395)
* Fix SemanticVersion comparison.

* Complete tests for _Alpha and _Numeric comparators.

* Linting, and add comment.
2020-05-11 11:58:15 -05:00
Rick Elrod
de59b17c7f
Add Fedora 32 to CI (#69222)
Change:
Adds Fedora 32 to shippable and alters tests slightly for new Fedora.

Test Plan:
CI

Tickets:
Fixes #69230

Co-authored-by: Matt Clay <matt@mystile.com>
2020-05-08 18:59:43 -05:00
Rick Elrod
98fbd1b72f lint
Signed-off-by: Rick Elrod <rick@elrod.me>
2020-05-07 21:44:13 -05:00
John R Barker
084587913a
validate-modules: deprecated modules in collections (#68646)
* validate-modules: deprecated modules in collections

In Collections a module is marked as deprecated via meta/routing.yml

Use this file, rather than the leading `_` as part of the deprecated
test.

* Correct variable

* review comments

* indentation

* Read routing.yml only once

* pep8

* Apply suggestions from code review

Co-authored-by: Matt Clay <matt@mystile.com>

* review: remove duplicated conditional

Co-authored-by: Matt Clay <matt@mystile.com>
2020-05-07 08:11:46 +01:00
Toshio Kuratomi
39b3942048 Remove left hand side slicing
Left hand side slicing is confusing and slower but maybe more memory
efficient in some circumstances.  There is one case where it adds to
code safety: when it's used to substitute a different list in place of a
slice of the original list and the original list could have been bound
to a different variable in some other code.  (The most likely case of
this is when it's a global variable and some other code might import
that variable name).

Because of the confusion factor we think it should only be used for the
safety case or where it's been benchmarked and shown to have some sort
of documentatble improvement.  At the moment, only one piece of code
falls into those categories so this PR removes all the other instances
of left hand side slicing.
2020-05-06 09:56:40 -07:00
Abhijeet Kasurde
794d269a4d
Distribution: Add PopOS as Debian OS_FAMILY (#69294)
PopOS is a Debian based OS distribution, added support to detect
ansible_os_family as 'debian' instead of 'Pop!_OS'

Fixes: #69286

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2020-05-05 11:43:28 -05:00
Felix Fontein
947fa3bad3
ansible-test validate-modules: don't allow arbitrary lists and dicts for 'default', 'sample' and 'example' (#69287)
* Don't allow arbitrary lists and dicts for 'default', 'sample' and 'example'.

* Add changelog.

* Make compile with Python 2.
2020-05-05 11:38:10 -05:00
Toshio Kuratomi
049800c063 Fix metadata defaults
* Metadata defaults were not being set if only a few fields were missing.

* ansible-doc with no documentation and no status in metadata should
  return empty, just like if there was no documentation and no metadata
  at all.
2020-05-05 09:06:52 -07:00
Abhijeet Kasurde
8d43d79191
galaxy: Handle empty roles and collections (#69199)
Galaxy collection install command raised indexError,
when requirements.yml contains empty roles and collections.

This fix handles empty roles and/or empty collections.

Fixes: #68186

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2020-05-05 10:34:04 -05:00
Clément Notin
9d48884e36
Fix support for Kali Linux detection (#69194)
* Fix support for Kali Linux detection

* Add test for Kali Linux detection

* Improve path matching with "in list"

Co-Authored-By: Abhijeet Kasurde <akasurde@redhat.com>

Co-authored-by: Abhijeet Kasurde <akasurde@redhat.com>
2020-05-05 10:31:32 -05:00
Brian Coca
bbdf77a59f
preserve json parsing error (#58461)
* preserve json parsing error
* added test
2020-05-01 15:03:07 -04:00
Brian Coca
23ab8f37df
fix a-doc listing plugins and add tests (#68600)
* add docs listing tests
* added collection module docs test
* always safe_load
* force 'type consistency' for uniquing paths
* bytified
* use our json encoder

Co-Authored-By: Matt Clay <matt@mystile.com>
2020-04-30 17:20:56 -04:00
David Shrewsbury
1142faa213
Do not pass file mode during recursive copy on symlink files. (#69011)
* Do not pass file mode during recursive copy on symlink files.

The 'file' module cannot deal with mode=preserve. Do not pass that
mode to the module when 'preserve' is used.

* Fix changelog fragment filename
2020-04-30 14:22:16 -05:00
Matt Clay
1cf26896c5 Default ansible-test provisioning to us-east-1.
Previously windows instances on Shippable would be automatically directed to us-east-2.
2020-04-30 11:04:37 -07:00
Rick Elrod
1c5c89baa6 Add RHEL 8.2 to CI
Change:
RHEL 8.2 GA was released this week, test it in CI instead of 8.1.

Test Plan:
CI

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-04-29 20:51:28 -05:00
Rick Elrod
eb65a01598 Enable dnf integration tests on CentOS 8
Change:
We were only testing dnf on RHEL previously.
Test on CentOS 8 as well.

Test Plan:
Ran locally in docker.

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-04-29 20:51:28 -05:00
Rick Elrod
f6cfcba346 [dnf] Make behavior/errors compatible for new DNF
Change:
Extend the logic for custom error handling in the dnf module, so that on
newer DNF (such as DNF that ships with modern Fedora 31 container
images, and ships with RHEL 8.2) we report errors consistently with
older DNF.

Test Plan:
Ran dnf integration tests against an old Fedora 31 container image and a
brand new Fedora 32 container image; tess passed on both.

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-04-29 20:51:28 -05:00
Matt Martz
bc41dd4514
Handle non-ascii paths during role installation. Fixes #69133 (#69213) 2020-04-28 15:33:44 -05:00
Brian Coca
babac66f9c
Be strict about what is a boolean for keywords (#67625)
* be strict about what is a boolean for keywords

 - found and fixed typo in test , 'yes' != 'yes.'
2020-04-28 13:55:26 -04:00
Abhijeet Kasurde
7714f691eb wait_for simplehttpserver to come up online
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2020-04-28 09:43:07 -07:00
David Shrewsbury
ff47d3f766
Make sure collection is a list if a str is given (#69081)
* Make sure collection is a list if a str is given

* Call field validation early on collections

Because we are doing work on modifying the collections value before
it is actually validated, we can validate it ourselves early to make
sure the user supplies either a string or list. Dicts are not valid.

The new validation allows us to simplify the _ensure_default_collection()
function. And since the field is now static, we no longer need to specify
a default for it, which also allows us to simplify the function. Since
the default is now removed, we can also remove the sanity/ignore.txt entry
for collectionsearch.py.

New unit tests are added (and the existing one modified) that allow us to
make sure that we throw a parser error if a user specifies something other
than a string or list for the collections value everywhere it can be specified.

* Revert removing the collection default

The default is actually used, so restore it.

* Fix unit tests in test_helpers.py affected by early collection validation
2020-04-28 10:47:11 -05:00
Alexandre Chouinard
91bb5af688
Update ansible-galaxy to handle role requirements (#68288)
* Update galaxy role object to handle requirements

Co-Authored-By: Sandra McCann <samccann@redhat.com>
2020-04-28 10:42:57 -04:00
Matt Clay
a050d892d8 Remove obsolete vcenter setup from ansible-test. 2020-04-26 19:56:40 -07:00
Matt Clay
d7da1d9bd0 Fix ansible-test coverage constraints. 2020-04-24 10:52:21 -07:00
Joe Cropper
e8d3358427 Ignore GPG key checks on test repo
Disable GPG key checks when installing packages from the test repo.
2020-04-23 14:48:55 -07:00
Joe Cropper
e663391e77 Fix yum test case on ppc64le
This patch covers a few changes to get the yum test case working on ppc64le
CentOS.  Specifically we needed to enable the EPEL repository on CentOS
as well as ensure some of the architecture-specific tasks use the right
set of binaries during their test.
2020-04-23 14:48:30 -07:00
Jesse Pretorius (odyssey4me)
da390b297e [pip] Enable virtualenv_command to have arguments
Currently if virtualenv_command has arguments, then the
search for the binary in the path does not work so the
user has to specify the full path to it.

To allow arguments to be used without having to specify
the path to the binary, we split the module argument into
the command and anything after the first space.

This makes using this module argument more flexible and
user friendly.

Fixes: #52275
2020-04-23 18:13:30 +02:00
Matt Martz
f27c417fc4
Use Templar for galaxy skeletons (#69106)
* Use Templar for galaxy skeletons. Fixes #69104

* Update checksum, our templar doesn't remove trailing newline, jinja2 seems to remove it
2020-04-23 10:36:14 -05:00
Rick Elrod
85bb804cda
Revert "Cap cryptography version for macOS openssl test (#69083)" (#69126)
This reverts commit 1e08bb7a6f.
2020-04-23 08:49:09 -05:00
Adam Miller
8b30360ca9
fixed handling of releasever for all known scenarios (#69057)
Signed-off-by: Adam Miller <admiller@redhat.com>
2020-04-23 09:02:00 +02:00
Rick Elrod
cace616aab
Filter BLACKLIST_EXTS in PluginLoader (#69029)
Change:
Rather than hardcoding .pyo and .pyc, filter on all BLACKLIST_EXTS in
the non-legacy logic of PluginLoader (_find_fq_plugin). The two harcoded
extensions are part of BLACKLIST_EXTS already and this simply adds the
rest of the blacklisted extensions to the check.

In addition, check .endswith() instead of an exact match of the suffix,
like everywhere else that uses BLACKLIST_EXTS. This allows for
blacklisting, for example, emacs's backup files which can appear after
any extension, leading to things like `foo.py~`.

Test Plan:
Ran `ansible-playbook` against a collection where a `foo.py~` module was
getting executed instead of `foo.py` which also appeared in the same
directory. `foo.py~` is no longer executed.

Tickets:
Fixes #22268
Refs #27235

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-04-22 17:20:12 -05:00
Matt Martz
da98fc267a
Allow a collection role to call a standalone role by default (#69102)
* Allow a collection role to call a standalone role by default. Fixes #69101

* tweaked changelog text

* Guard against NoneType

Co-authored-by: Matt Davis <nitzmahone@users.noreply.github.com>
2020-04-22 13:58:08 -05:00
rwagnergit
977b58740b
update ActionBase._low_level_execute_command to honor executable (#68315)
* update ActionBase._low_level_execute_command to honor executable

* adding changelog fragment

* renaming changelog fragment to .yml

* noop change to bump shippable

* adding raw_executable integration test

* copying aliases from raw

* removing blank lines

* skipping aix and freebsd

* noop to bump shippable

* moving tests to raw/

* removing become_method: sudo ; it doesn't work on AIX

* removing trailing blank line

* forcing become_method: su to try to get AIX to work

Co-authored-by: Rob Wagner <rob.wagner@sas.com>
2020-04-22 11:56:35 -05:00
Rick Elrod
1e08bb7a6f
Cap cryptography version for macOS openssl test (#69083)
Change:
New `cryptography` statically links an openssl that is too new for macOS
10.11, so limit to an older cryptography for now.

Test Plan:
Ran the test with `--remote osx/10.11` and it passed.

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-04-21 15:13:46 -05:00
Matt Clay
5a3ca0ae5f Fix ansible-test delegation from pip install. 2020-04-21 09:48:16 -07:00
Rick Elrod
d17a44b0c5
Add new arches/OSes for binary_modules tests (#69052) 2020-04-21 00:52:23 -07:00
David Shrewsbury
af44bd4ddd
Fix --start-at-task when skipping tasks with no name (#68951)
Using --start-at-task on a playbook with tasks with no name would fail
if those unnamed tasks were encountered before the targetted start task.
2020-04-21 09:39:17 +02:00
Matt Clay
122743ae30
Update ansible-test handling of delegation paths. (#69056) 2020-04-20 22:04:25 -07:00
Matt Clay
a8a61829b8 Enable passing tests on power. 2020-04-20 10:07:54 -07:00
Rick Elrod
648b3d43d3
[copy] Add some test coverage for check_mode (#68895)
Change:
Adds some missing coverage for the copy module when `check_mode: True`.

Test Plan:
Ran test with --coverage and looked at the resulting report.

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-04-17 16:09:32 -05:00
Rick Elrod
44a471f6ba
Enable service integration tests for FreeBSD (#68980)
* Enable service integration tests for FreeBSD

Change:
Adds necessary rc file for freebsd, and gets tests passing for it.

Test Plan:
Ran test with `--remote freebsd/12.1` and `--remote freebsd/11.1`. Both
passed.

Signed-off-by: Rick Elrod <rick@elrod.me>

* Update test/integration/targets/service/tasks/rc_setup.yml

Co-Authored-By: Matt Clay <matt@mystile.com>

* fix up comment

Signed-off-by: Rick Elrod <rick@elrod.me>

Co-authored-by: Matt Clay <matt@mystile.com>
2020-04-17 10:41:08 -07:00
Brian Coca
d3cab602a5
Fix fileglob when using 'file*' vs 'stuff/file*' (#68945)
* Fix fileglob when using 'file*' vs 'stuff/file*'

 when not having dir in glob, files/ subdir was being ignored.

* tests for fileglob
2020-04-17 09:51:05 -04:00
Martin Krizek
acdc9eb76d
native types: literal_eval all the things (#68938)
With https://github.com/pallets/jinja/pull/1190 merged our short-circuit
is no longer valid (has it ever been?) as now data like ' True ' may go
through our ansible_native_concat function as opposed to going through
intermediate call to Jinja2's native_concat before. Now we need to always
send data through literal_eval to ensure native types are returned.
2020-04-17 14:59:52 +02:00
Abhijeet Kasurde
3591451bc7
include_role: Strictly check string datatype for *_from (#68958)
Strictly check string datatype for 'tasks_from', 'vars_from',
'defaults_from', and 'handlers_from' in include_role

Fixes: #68515

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2020-04-17 10:57:41 +05:30
Jordan Borean
79fff7da69
Expose to_<str> as a public function (#68965)
* Expose to_<str> as a public function

* Fix sanity checks

* Move docstring to start of util
2020-04-17 07:54:00 +10:00
Brian Coca
ac509d489b
Revert "stricter permissions on atomic_move when creating new file (#68970)" (#68983)
This reverts commit 566f2467f6.
2020-04-16 12:52:15 -04:00
Brian Coca
566f2467f6
stricter permissions on atomic_move when creating new file (#68970)
fixes #67794
  updated some tests that expected previous defaults
  CVE-2020-1736
2020-04-16 09:06:18 -04:00
Jordan Borean
702949e64c
windows - fix up env var quoting for unicode single quotes (#68968)
* windows - fix up env var quoting for unicode single quotes

* Add sanity ignore check for smart quotes
2020-04-16 20:24:23 +10:00
Matt Clay
77a916e763
Add more integration tests. (#68972) 2020-04-16 00:53:08 -07:00
Yanis Guenane
46d82179d8
Testing: Add support for CentOS Linux On Power platform (#68130)
* Testing: Add CentOS Linux On Power platform

* Add arch designation to remotes.

This avoids overloading the provider with the arch.

Also add a changelog entry.

Co-authored-by: Matt Clay <matt@mystile.com>
2020-04-15 16:22:17 -07:00
Sam Doran
cdb24e0078
ansible-galaxy - fix listing specific role and role description (#67409)
* ansible-galaxy - fix listing specific role

If the role was not in the first search path, it was reported as not found

* Properly display role description

Default to description to top level description, falling back to the description from within galaxy_info

* Display proper message when a role does not exist

* Add integration tests

* Use context manager

* BSD and macOS ruining all the fun
2020-04-15 15:26:45 -04:00
Matt Clay
7d51706278 Revert "Switch network incidental tests to VyOS 1.1.7-R2."
This reverts commit 7d5177d6a0.
2020-04-14 12:18:23 -07:00
jctanner
f11b1454bb
prepare for future ansibot collection changes (#68944) 2020-04-14 13:11:05 -04:00
Martin Krizek
ff1ba39c8a
Prevent templating unused variables for {%include%} (#68749)
Fixes #68699
2020-04-14 10:27:02 +02:00
Matt Clay
7d5177d6a0 Switch network incidental tests to VyOS 1.1.7-R2.
The previous version being used, VyOS 1.1.8, is currently unavailable in AWS.
2020-04-14 01:12:48 -07:00