Commit graph

2370 commits

Author SHA1 Message Date
Brian Coca
9b992c0b78
reset logging to INFO (#70878) (#70881)
- due to CVE-2019-14846
 - also added comments and test to avoid 'oportunistic' reversion

(cherry picked from commit 1223ce656a)
2020-07-24 15:47:57 -05:00
Rick Elrod
c528e648a6 New release v2.10.0rc2 2020-07-23 21:38:44 -05:00
Jordan Borean
e9c9c02e0a
Do not add connection vars to the output results (#70853) (#70855)
* Do not add connection vars to the output results

* Also revert the delgated scenario JIC

* Added regression test

(cherry picked from commit 5e1a968983)
2020-07-23 19:48:45 -05:00
Rick Elrod
3ee5b46ba7 New release v2.10.0rc1 2020-07-23 14:44:50 -05:00
Matt Davis
cc8d180801
fix internal cases of actions calling unqualified module names (#70818) (#70840)
* fix internal cases of actions calling unqualified module names

* add porting_guide entry
* misc other fixes around action/module resolution broken by redirection

ci_complete

* Update docs/docsite/rst/porting_guides/porting_guide_2.10.rst

Co-authored-by: Rick Elrod <rick@elrod.me>

* Update docs/docsite/rst/porting_guides/porting_guide_2.10.rst

Co-authored-by: Rick Elrod <rick@elrod.me>

* address review feedback

* pep8

* unit test fixes

* win fixes

* gather_facts fix module args ignores

* docs sanity

* pep8

* fix timeout test

* fix win name rewrites

Co-authored-by: Rick Elrod <rick@elrod.me>
(cherry picked from commit 4c0af6c808)
2020-07-23 10:29:09 -07:00
Sloane Hertel
180eea8089
Update default from True to False for CONDITIONAL_BARE_VARS (#70709) (#70838)
ci_complete

(cherry picked from commit 2811d9486f)
2020-07-23 10:09:58 -07:00
Sam Doran
7e4cffc5d2
[stable-2.10] Change default file permissions so they are not world readable (#70221) (#70824)
* Change default file permissions so they are not world readable

CVE-2020-1736

Set the default permissions for files we create with atomic_move() to 0o0660. Track
which files we create that did not exist and warn if the module supports 'mode'
and it was not specified and the module did not call set_mode_if_different(). This allows the user to take action and specify a mode rather than using the defaults.

A code audit is needed to find all instances of modules that call atomic_move()
but do not call set_mode_if_different(). The findings need to be documented in
a changelog since we are not warning. Warning in those instances would be frustrating
to the user since they have no way to change the module code.

- use a set for storing list of created files
- just check the argument spac and params rather than using another property
- improve the warning message to include the default permissions.
(cherry picked from commit 5260527c4a)

Co-authored-by: Sam Doran <sdoran@redhat.com>
2020-07-23 09:07:18 -07:00
Brian Coca
8c2754e6d3
Allow hostvars delegation (#70331) (#70810)
* ensure hostvars are available on delegation
* also inventory_hostname must point to current host and not delegated one
* fix get_connection since it was still mixing original host vars and delegated ones
* also return connection vars for delegation and non delegation alike
* add test to ensure we have expected usage when directly assigning for non delegated host

(cherry picked from commit 84adaba6f5)
2020-07-22 18:29:07 -07:00
David Shrewsbury
7cdba7c923
Sanitize URI module keys with no_log values (#70762) (#70820)
* Add sanitize_keys() to module_utils.

* More robust tests

* Revert 69653 change

* Allow list or dict

* fix pep8

* Sanitize lists within dict values

* words

* First pass at uri module

* Fix insane sanity tests

* fix integration tests

* Add changelog

* Remove unit test introduced in 69653

* Add ignore_keys param

* Sanitize all-the-things

* Ignore '_ansible*' keys

* cleanup

* Use module.no_log_values

* Avoid deep recursion issues by using deferred removal structure.

* Nit cleanups

* Add doc blurb

* spelling

* ci_complete

(cherry picked from commit bf98f031f3)
2020-07-22 18:28:24 -07:00
Nathaniel Case
4cbfb08da5
[stable-2.10] Document existing ansi_re sequences and add ESC[m (#70683) (#70807)
* Document existing ansi_re sequences and add `ESC[m`

* Add changelog
(cherry picked from commit 06a4fc2)

Co-authored-by: Nathaniel Case <ncase@redhat.com>
2020-07-22 18:27:21 -07:00
Abhijeet Kasurde
18dd73c147
[2.10] basic: use PollSelector implementation (#70800)
Some platform such as ESXi does not implement EpollSelector,
which is selected by DefaultSelector. Use PollSelector.
This works perfectly with a platform like VMware ESXi.

Fixes: #70238

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
(cherry picked from commit 8cccede0d4)
2020-07-22 18:26:47 -07:00
psi / Ryo Hirafuji
61f8f8ce7f
cron - Allow non-ascii (UTF-8) chars in cron file paths and jobs (#70426) (#70794)
* Encode/Decode files in UTF-8
* Use helper function in ansible
* Add an integration test
* Use emoji in test data.
* add changelog
* Also support non-ascii chars in filepath and add tests about this.
* Also use non-ascii chars in replaced text and ensure not to break cron syntax.
* rename self.existing to self.n_existing
* rename crontab.existing to crontab.n_existing
2020-07-22 18:26:10 -07:00
Matt Martz
7eb5f53294
[stable-2.10] Ensure single vaulted values aren't counted as sequences. Fixes #70784 (#70786) (#70791)
(cherry picked from commit 96b74d3)

Co-authored-by: Matt Martz <matt@sivel.net>
2020-07-22 18:25:38 -07:00
Baptiste Mille-Mathias
1eb2afac63
Create home and parent directories only when requested (#70790)
The home user and the parents directories should only be created when
create_home == True

(cherry picked from commit f3dd8d3052)
2020-07-22 18:23:47 -07:00
Martin Krizek
4170786cd9
2.10: Detect failure in always block after rescue (#70094) (#70204)
* Detect failure in always block after rescue (#70094)

* Detect failure in always block after rescue

Fixes #70000

ci_complete

* Add more tests

(cherry picked from commit 0ed5b77377)

* add changelog

Co-authored-by: Matt Davis <mrd@redhat.com>
2020-07-22 14:00:27 -07:00
Matt Martz
448f17e9a5
[stable-2.10] Guard against allowing ansible to ansible-base upgrades (#70529) (#70760)
* Fix building Ansible dist w/ setuptools>=48,<49.1 (#70525)

* Fix building Ansible dist w/ setuptools>=48,<49.1

This change addresses the deprecation of the use of stdlib
`distutils`. It's a short-term hotfix for the problem and we'll
need to consider dropping the use of `distutils` from our `setup.py`.

Refs:
* https://github.com/ansible/ansible/issues/70456
* https://github.com/pypa/setuptools/issues/2230
* https://github.com/pypa/setuptools/commit/bd110264

Co-Authored-By: Jason R. Coombs <jaraco@jaraco.com>

* Add a change note for PR #70525

Co-authored-by: Jason R. Coombs <jaraco@jaraco.com>
(cherry picked from commit 918388b85f)

* Guard against allowing ansible to ansible-base upgrades (#70529)

* Guard against allowing ansible to ansible-base upgrades

* newline

* use alias

* Add an explicit line detailing this is a 1 time thing

* period

* Read __version__ and __author__ rather than import, update working, and add ability to skip conflict checks

* Remove commented code

* Re introduce removed changes from rebase

* Just use open

* Nuke unused import

(cherry picked from commit 54b002e1ac)

Co-authored-by: Sviatoslav Sydorenko <webknjaz@redhat.com>
2020-07-21 11:23:55 -07:00
Baptiste Mille-Mathias
ffd3757fc3
Fix missing quoting for remote_tmp in second mkdir of shell module. Issue #69577 (#69578) (#70757)
* Fix missing quoting for remote_tmp in second mkdir of shell module. Issue #69577

* adding changelog

* fixing typo in changelog entry

* adding test case

Adding test case written by bmillemayhias.

* using $HOME instead of ~

* fixing commit measage

* Update 69578-shell-remote_tmp-quoting.yaml

Co-authored-by: Brian Kohles <me@briankohles.com>
(cherry picked from commit 77d0effcc5)

Co-authored-by: Brian Kohles <briankohles@users.noreply.github.com>
2020-07-21 11:23:25 -07:00
Abhijeet Kasurde
0d230b4f56
[2.10] debconf: add a note about no_log usage (#70752)
debconf module exposes sensitive information to logs, console.
Add a note to user about using no_log=True to hide such
information from console.

Fixes: #32386

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
(cherry picked from commit 84b4387702)
2020-07-21 11:21:45 -07:00
Sam Doran
9b8a649f2e
[stable-2.10] Handle Slackware OS version strings containing a plus (“+”) (#68142) (#70717)
A couple of years ago Slackware -current began using a plus (“+”) at the end of the distribution version string to indicate a future version work-in-progress.

Rearrange distribution_files unit tests to easily support more tests
  - add conftest with common fixtures
  - use parametrize for testing multiple scenarios

* Add changelog
* Add unit tests for Slackware distribution parsing
* Use correct fixtures for Slackware
Data comes from /etc/slackware-version

Co-authored-by: Sam Doran <sdoran@redhat.com>
Co-authored-by: <Eduard Rozenberg <eduardr@pobox.com>>
(cherry picked from commit 566c5e6ce1)

Co-authored-by: Eduard Rozenberg <2648417+edrozenberg@users.noreply.github.com>

Co-authored-by: Eduard Rozenberg <2648417+edrozenberg@users.noreply.github.com>
2020-07-21 11:21:07 -07:00
Jordan Borean
a7f4c6a37f
win async - use full path for powershell (#70703) (#70714)
(cherry picked from commit 154efd97f2)
2020-07-21 11:20:35 -07:00
Sloane Hertel
d329985d4c
[2.10] template connection variables accessed directly before using (#70657) (#70688)
* template connection variables accessed directly before using (#70657)

* template variables accessed directly when using them instead of FieldAttributes

(cherry picked from commit 8c213c9334)

* changelog
2020-07-21 11:19:52 -07:00
Abhijeet Kasurde
ed07821a59
[2.10] api: time.clock compatible code (#70677)
time.clock is removed in Python 3.8. Add time.clock
compatible code.

Fixes: #70649

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
(cherry picked from commit 055871cbb8)
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2020-07-21 11:19:27 -07:00
Matt Martz
255dfca7f6
[stable-2.10] Allow single vault encrypted values to be used directly as module parameters. Fixes #68275 (#70607) (#70641)
(cherry picked from commit a77dbf0)

Co-authored-by: Matt Martz <matt@sivel.net>
2020-07-17 12:54:32 -07:00
Abhijeet Kasurde
963bdd9983
[2.10] pipe: update docs for Popen with shell=True usage (#70602)
pipe lookup plugin uses Popen with shell=True intentionally.
This is considered a security issue if user input is not validated.
Updated docs to reflect this information for the user. Also, added
Bandit B602 documentation link for further reading.

Fixes: #70159

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
(cherry picked from commit e5649ca3e8)
2020-07-17 12:51:35 -07:00
Brian Coca
6cd015d7e2
Make filter type errors 'loop friendly' (#70417) (#70574)
- ensure we preserve the typeerror part of the exception so loop defereed error handling
 can postpone those caused by undefined variables until the when check is done.
 - fix tests to comply with the 'new normal'

 - human_to_bytes and others can issue TypeError not only on 'non string'
 but also bad string that is not convertable.

Co-authored-by: Sloane Hertel <shertel@redhat.com>

Co-authored-by: Sloane Hertel <shertel@redhat.com>
(cherry picked from commit cf89ca8a03)
2020-07-17 12:51:18 -07:00
Felix Fontein
15355ed059
[2.10] ansible-doc: include collection name in text output / plugin loader: return collection name; ansible-doc: handle ansible.builtin correctly (#70572)
* ansible-doc: include collection name in text output (#70401)

* ansible-doc: include collection name in text output

* Be more careful to not accidentally pass ansible.builtin for user-supplied modules.

(cherry picked from commit f4c89eab23)

* plugin loader: return collection name; ansible-doc: handle ansible.builtin correctly (#70026)

* Determine collection in plugin loader.

* Fix test.

* Use PluginPathContext objects in PluginLoader._plugin_path_cache instead of tuples.

(cherry picked from commit 24dcaf8974)
2020-07-17 12:50:23 -07:00
Brian Coca
827c47d9bc
try to capture better winrm/put_file error (#70508) (#70570)
* try to capture better winrm/put_file error

fixes #70361

* Update lib/ansible/plugins/connection/winrm.py

Co-authored-by: Abhijeet Kasurde <akasurde@redhat.com>
Co-authored-by: Matt Davis <nitzmahone@users.noreply.github.com>
(cherry picked from commit 8789d7968d)
2020-07-17 12:49:11 -07:00
Mykola Grygoriev
5ea6de4e7d
Fix decrypt argument in assemble module (#70465) (#70560)
* Do not pass decrypt parameter to assemble module

* Add integration tests where decrypt=True

* Add changelog #70465

(cherry picked from commit 71c378e139)
2020-07-17 12:48:38 -07:00
David Shrewsbury
94a81f7b44
Make sure ansible_become treated as a boolean (#70484) (#70526)
* Make sure ansible_become treated as a boolean

(cherry picked from commit 8aca464b8b)
2020-07-17 12:46:33 -07:00
Felix Fontein
a5c0b11913
ansible-doc man formatter: fail with better error message when description isn't there (#70046) (#70485)
* ansible-doc man formatter: do not crash when description isn't there.
* Change to report a better error message when description is not there.
* Add test.

(cherry picked from commit 9164b96774)
2020-07-17 12:44:03 -07:00
Sam Doran
422e976649
[stable-2.10] apt - make errors more transparent (#70099) (#70478)
Include error from apt Python library in module error output

Co-authored-by: Andreas Schleifer <aschleifer@bigpoint.net>
(cherry picked from commit 7d7f15fc9b)

Co-authored-by: Andreas Schleifer <a.schleifer@bigpoint.net>

Co-authored-by: Andreas Schleifer <a.schleifer@bigpoint.net>
2020-07-17 12:43:19 -07:00
Sam Doran
df3246b7d1
[stable-2.10] ssh connection - use get_option() rather than _play_context (#70438) (#70443)
The ssh_args were sometimes not correctly applied to the connection
when using _play_context. Use get_option() instead to ensure the
correct ssh_args are always applied.
(cherry picked from commit b4184aa50e)

Co-authored-by: Sam Doran <sdoran@redhat.com>
2020-07-17 12:42:51 -07:00
Matt Davis
865ba1953b
misc collection metadata fixes (#70403) (#70428)
* misc collection metadata fixes

* parse collection meta with libyaml if available
* require only Mapping for validation
* add explanatory text for _meta_yml_to_dict

* ignore custom pylint rule

* this code shouldn't import a bunch of stuff from ansible, since it's run under the import sanity test

(cherry picked from commit b9e38e8b55)
2020-07-17 12:42:20 -07:00
Sloane Hertel
212d2024f4
[2.10] Handle post_validate templating errors and fix tests (#70240) (#70389)
* Handle post_validate templating errors and fix tests (#70240)

* Handle unexpected templating errors

* Fixes #70050

Fix up tests that weren't running and add tests for graceful templating error handling

(cherry picked from commit 30e70f4b63)

* changelog

ci_complete
2020-07-17 12:41:43 -07:00
Sam Doran
d0e37aeec1
[stable-2.10] Improve error message when ssh client is not found on the host (#70122) (#70386)
* Add changlelog
(cherry picked from commit 70bb2b33ca)

Co-authored-by: Mykola Grygoriev <nickgryg@gmail.com>

Co-authored-by: Mykola Grygoriev <nickgryg@gmail.com>
2020-07-17 12:40:38 -07:00
Martin Krizek
7dfda4026e
Fix delegate_facts with interpreter not being set (#70293) (#70384)
Fixes #70168

ci_complete

Co-authored-by: Brian Coca <bcoca@users.noreply.github.com>
Co-authored-by: Matt Clay <matt@mystile.com>
(cherry picked from commit b05e00e99a)
2020-07-17 12:40:00 -07:00
Abhijeet Kasurde
e1c0688e43
[2.10] Improve ansible-galaxy STDOUT messages for collections (#70379)
- Fix issue #70010
- Add installation successful message
- This feature targets "collection" sub-command and does not affect "role" sub-command

Signed-off-by: Hideki Saito <saito@fgrep.org>
(cherry picked from commit 2d59e548f6)

Co-authored-by: Hideki Saito <saito@fgrep.org>
2020-07-17 12:39:16 -07:00
Felix Fontein
f9a737b9b2
Top-level deprecation of plugin did not get collection_name added when deprecating by version (#70344) (#70370)
* Top-level deprecation of plugin did not get collection_name added when deprecating by version.

* Add changelog fragment.

(cherry picked from commit 689cfd1983)
2020-07-17 10:58:54 -07:00
Matt Davis
1e03b54d23
refactor Python module_utils locator (#70610) (#70711)
* refactor Python module_utils locator

* no longer recursive
* embed special-case module code internally
* share common code between collections/not cases
* fixes #70134
* properly support subpackage redirection
* adds support for FQCN redirect targets used by migration (expands to FQ Python name)
* add tests

* add changelog

(cherry picked from commit c616e54a6e)
2020-07-17 10:57:44 -07:00
Sam Doran
367c9b7eda
[stable-2.10] pause - fix curses.setupterm() error (#47851) (#70363)
* [Bugfix] curses.setupterm() error

When run playbook in celery task, curses.setupterm()  will be failed

```
  File "<frozen importlib._bootstrap>", line 675, in _load
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed
  File "/Users/guang/.virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/action/pause.py", line 45, in <module>
    curses.setupterm()
TypeError: argument must be an int, or have a fileno() method.

```

* Add changelog

Co-authored-by: Sam Doran <sdoran@redhat.com>
(cherry picked from commit c3fc2d27d9)

Co-authored-by: 老广 <ibuler@qq.com>

Co-authored-by: 老广 <ibuler@qq.com>
2020-07-17 10:57:20 -07:00
Abhijeet Kasurde
3afa0424b3
[2.10] display: use stdout for column width (#70290)
stdout may differ from stdin so it should be used to determine the column
width, especially since it is the target file descriptor.

(cherry picked from commit 45e0f74702)

Co-authored-by: Pavel Březina <pbrezina@redhat.com>
2020-07-17 10:56:40 -07:00
Felix Fontein
273159da4c
Include changelogs/changelog.yaml in distribution. (#70260) (#70266)
(cherry picked from commit ebd20ddca6)
2020-07-17 10:55:25 -07:00
Sam Doran
95ec1618ef
[stable-2.10] Only pass kwargs to our string checker not callable checkers (#70151) (#70170)
Since only check_type_str() accepts extra param, only pass to our checker and
do not pass kwargs to custom checkers.

* Add unit tests
(cherry picked from commit bc05415109)

Co-authored-by: Sam Doran <sdoran@redhat.com>
2020-07-17 10:53:56 -07:00
Sam Doran
b6f6067dfd
[stable-2.10] ansible_runner test - Add constraints (#70667) (#70669)
A recent updated to psutil, which is a dependency of ansible-runner, fails
to install on older versions of pip.

Commit with the breaking change:

  135628639b
(cherry picked from commit 9d27d7c8b1)

Co-authored-by: Sam Doran <sdoran@redhat.com>
2020-07-15 17:32:09 -05:00
Matt Clay
c2b69fd30e [stable-2.10] Fix ansible-test virtualenv management.
(cherry picked from commit 1e02a201a6)

Co-authored-by: Matt Clay <matt@mystile.com>
2020-07-14 02:08:17 -07:00
Matt Clay
ddc78622a4 [stable-2.10] Update ansible-test change detection for plugins.
(cherry picked from commit 09f02980a4)

Co-authored-by: Matt Clay <matt@mystile.com>
2020-07-13 23:55:15 -07:00
Matt Clay
957f0c27fc [stable-2.10] Fix boilerplate in setup.py and lib/ansible/ dir..
(cherry picked from commit 234994fc07)

Co-authored-by: Matt Clay <matt@mystile.com>
2020-07-13 18:57:17 -07:00
Matt Clay
a707466c0b [stable-2.10] Fix yamllint sanity test line numbers.
Parse errors from libyaml now compensate for the offset of the documentation within a module.
(cherry picked from commit 1a0d8a51cd)

Co-authored-by: Matt Clay <matt@mystile.com>
2020-07-13 18:28:30 -07:00
Matt Clay
73d9054b32 [stable-2.10] Test netconf plugins in ansible-doc sanity test.
(cherry picked from commit 8d92df4537)

Co-authored-by: Matt Clay <matt@mystile.com>
2020-07-13 18:14:42 -07:00
Sandra McCann
374d4ecd98
Doc backportapalooza 2 (#70440)
* doc: avoid mix of single and double quotes (#70115)

Avoid mix of single and double quotes in the `ternary`, this way
we can copy/past the example without any surprise.

(cherry picked from commit b491f776b9)

* document FQCN for M()  and :seealso: in DOCUMENTATION blocks (#70245)

* document FQCN for M() in DOCUMENTATION blocks

* add note about c

(cherry picked from commit 83f6e4850b)

* Fix bullet points in intro_getting_started.rst. (#70365)

The layout was jumbled due to issues with whitespace.

(cherry picked from commit dc6f4b6502)

* Add steps for how to create changelog.rst for a collection (#70262)

* Update docs/docsite/rst/dev_guide/developing_collections.rst
* add steps to create changelogs, add sentence about not using the tool
* add note for rerunning the command

Co-authored-by: Felix Fontein <felix@fontein.de>
(cherry picked from commit 5a28b2b86c)

* ansible-doc: avoid problems with YAML anchors when formatting man page (#70045)

* Avoid problems with YAML anchors when formatting man page.

* Add changelog.

(cherry picked from commit 5e4f708241)

* Minor grammatical fix (#70405)

'you' -> 'your'

(cherry picked from commit a1ac595d42)

* incorporate minimalism feedback on filters page (#70366)

Co-authored-by: Alicia Cozine <acozine@users.noreply.github.com>
(cherry picked from commit c89f3cda9e)

* more correct info about role main.yml (#70326)

fixes #40496

(cherry picked from commit 5d3d097de3)

* Fix a small typo in cache plugin description @ `config/base.yml`

PR #70420

(cherry picked from commit 626df08d9d)

* with_sequence: example using vars (#69369)

Added an example for using vars in with_sequence.

Fixes: #68836

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
(cherry picked from commit 5709173c32)

* Update pull.py (#70393)

(cherry picked from commit 46ad3c1162)

* Update playbooks.rst (#70317)

(cherry picked from commit 7c90a2d2a6)

* Add documentation for ipaddr filters  (#70343)

(cherry picked from commit 9eb904ea61)

* update platform table with links to collections (#70373)

(cherry picked from commit aa59c23aed)

* Add description of collections and become_exe keywords (#68055)

* Add description of collections keyword
* Update based on feedback.
- Add link to become plugins.
- Add note about how the collections keyword works with roles.

(cherry picked from commit 5833af9e2a)

Co-authored-by: Gonéri Le Bouder <goneri@lebouder.net>
Co-authored-by: Mark Sanders <ziplokk.mark.sanders@gmail.com>
Co-authored-by: Felix Fontein <felix@fontein.de>
Co-authored-by: Sir Mobus Gochfulshigan Dorphin Esquire XXIII <celestialtuba@gmail.com>
Co-authored-by: Alicia Cozine <879121+acozine@users.noreply.github.com>
Co-authored-by: Brian Coca <bcoca@users.noreply.github.com>
Co-authored-by: Michael Scherer <mscherer@users.noreply.github.com>
Co-authored-by: Abhijeet Kasurde <akasurde@redhat.com>
Co-authored-by: Ethan <smithe2413@gmail.com>
Co-authored-by: jafiala <56597272+jafiala@users.noreply.github.com>
Co-authored-by: Baptiste Mille-Mathias <baptiste.millemathias@gmail.com>
Co-authored-by: Sam Doran <sdoran@redhat.com>
2020-07-08 16:05:41 -05:00
Rick Elrod
43ef13d206
command module, nix warnings for moved modules (#70444)
Change:
- Remove warnings from command module which point to modules that no
  longer ship with ansible-base but have moved to collections.

Test Plan:
- CI

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-07-02 14:26:18 -05:00
Adam Miller
ad61774acf
fix runtime.yml splunk.enterprise_security -> splunk.es (#70305) (#70310)
Signed-off-by: Adam Miller <admiller@redhat.com>
2020-07-02 10:32:01 -05:00
Jordan Borean
70da336de9 ansible-test - do not validate blacklisted ps modules (#70376)
* ansible-test - do not validate blacklisted ps modules

* Update changelogs/fragments/validate-modules-ps-doc-blacklist.yaml

Co-authored-by: Matt Clay <matt@mystile.com>

Co-authored-by: Matt Clay <matt@mystile.com>
(cherry picked from commit 40ce448657)
2020-06-30 15:15:13 -07:00
Matt Clay
73ee85145b [stable-2.10] Pin ansible-test requirements for RHEL.
The `packaging` and `pyparsing` packages are now installed by `ansible-test` during provisioning of RHEL instances to match the downstream vendored versions.
(cherry picked from commit 70c59423fc)

Co-authored-by: Matt Clay <matt@mystile.com>
2020-06-26 11:32:16 -07:00
Jordan Borean
a45d4fe80d
Fix up the redirection of the win setup module (#70158) (#70181)
(cherry picked from commit 8b24a4c5ed)
2020-06-19 17:12:32 -07:00
Rick Elrod
cbfe2e642f New release v2.10.0b1 2020-06-17 14:38:15 -05:00
Matt Clay
8152d8bc1a Disabled inconsistent pylint checks. 2020-06-17 10:10:17 -07:00
Matt Clay
598786e16d Update ansible-test default test containers.
The main change is the upgrade to Python 3.9.0b3.
2020-06-16 16:50:04 -07:00
David Shrewsbury
fc448ea54f
Clarify blockinfile docs for insertafter/insertbefore (#69396)
* Clarify blockinfile docs for insertafter/insertbefore

It's not clear from the docs that these options take effect
only when no marker lines are found in the document.

* Add changelog fragment
2020-06-16 14:48:08 -04:00
Matt Clay
f69b1846f9 Add pyparsing constraint to ansible-test.
The upcoming pyparsing 3 release will require Python 3.5 or later, see:

https://github.com/pypa/packaging/issues/313

Unfortunately pip 8.x and earlier versions do not support python version requirements, which is why this constraint is needed.
2020-06-16 10:38:16 -07:00
Matt Davis
de63cba7e8
transparent downstream vendoring (#69850)
* builtin downstream vendoring support

* allows downstream packagers to install packages to `ansible/_vendor` that will automatically be added to head of sys.path during `ansible` package load
* tests

* sort conflicting package names in warning text

* sanity fixes

* skip unnecessary comparison
2020-06-15 16:22:25 -07:00
Matt Clay
7641d32f8e
Fix ansible-test import sanity test issues. (#70084) 2020-06-15 15:09:15 -07:00
Martin Krizek
843751a00d
yum/dnf: check type of elements in a name (#70072) 2020-06-15 11:01:42 -04:00
Matt Clay
5b3719805e Update ansible-test test containers. 2020-06-12 09:19:18 -07:00
John R Barker
37785255f2
validate ansible-base's and collections runtime.yml (#69742)
* Validate ansible-base & collection's runtime.yml

Add new test `runtime-metadata`

* Schema validation of file
* Error if a a legacy meta/routing.yml exist in a collection
* removal_date OR removal_version

* Add tombstone validation.

* Allow both ISO 8601 date strings and datetime.date objects (from YAML dates).

* Address review comments.

* Add metadata to test collection.

* Add requirements file.

Co-authored-by: Felix Fontein <felix@fontein.de>
Co-authored-by: Matt Clay <matt@mystile.com>
2020-06-12 15:34:34 +01:00
Matt Clay
9f49db1f99 Avoid use of deprecated junit-xml method. 2020-06-12 00:47:07 -07:00
Brian Coca
4c9d9dbb56
fix discovery on loop with delegation (#70013)
* fix discovery on loop with delegation

fixes #69963
2020-06-11 18:04:17 -04:00
Matt Clay
8ffaed00f8 Add Azure Pipelines support to ansible-test. 2020-06-11 14:57:42 -07:00
Brian Coca
8c3eb7ae4a
fix configurable pipelining (#69920)
* fix configurable pipelining

Co-authored-by: Sloane Hertel <shertel@redhat.com>
2020-06-11 16:17:58 -04:00
Matt Martz
4a4a11d282
Add mccabe complexity testing (#64623)
* Add mccabe complexity testing

* Make mccabe complexity an optional error

* Add mccabe to new sanity pylint requirements

* Add a changelog fragment.

Co-authored-by: Matt Clay <matt@mystile.com>
2020-06-11 13:02:06 -07:00
David Shrewsbury
0ae4dac65a
Fix copy module file perms with remote_src (#69993)
When using 'remote_src: yes' and 'mode: preserve', the code handling
the file modes has to be handled on the remote node because it's
the one that has access to the source files. This means that the
copy module itself must handle this, rather than the copy action
plugin (which is where all that logic exists). The copy module
handles this when we copy a single file over. But when it is a
directory as the src parameter value, the mode of the files
beneath it are not considered. Subdirectories are copied with
shutil.copytree() which will preserve permissions automatically.
Individual files are copied with shutil.copyfile() which does NOT
preserve permissions. We need to add some calls to shutil.copymode()
to correct that.

Note: This *always* retains individial file permissions. Specifying
a 'mode' other than 'preserve' when giving a source directory for
the 'src' param does not make sense so will be ignored in that case
only.

Fixes #69783

* Add changelog and test
2020-06-11 15:18:59 -04:00
James Cassell
47d14a33bd
config: singular ANSIBLE_COLLECTIONS_PATH (#70007)
* config: singular ANSIBLE_COLLECTIONS_PATH

Every other *_PATH setting in ansible is singular, and the traditional
$PATH variable is also singular despite containing a list of
directories.  Let's be consistent both internally and with POSIX
tradition.

* update all ANSIBLE_COLLECTIONS_PATHS env references to be singular

* deprecate plural ANSIBLE_COLLECTIONS_PATHS setting
2020-06-11 11:40:13 -07:00
Felix Fontein
a114da80ee
Use antsibull-changelog instead of packaged changelog generator (#69313)
Replace the ansible-base changelog linting and generation tool with antsibull-changelog and make it available for linting collections. Previously changelog linting was limited to ansible-base.
2020-06-11 11:21:21 -07:00
Felix Fontein
8d93ba9120
Plugin/module docs: parse return values, add collection names in them (version_added_collection), and format them nicely in ansible-doc (#69796)
* Tag return value docs if they are a dict (and not str/None).

* Try to parse return docs as YAML.

* Properly dump return values in ansible-doc.

* Adjust plugin formatter.

* Add changelog fragment.

* Don't add 'default' for return values.

* Fix plugin_formatter.

* Only try to parse return docs if they are still a string.

* Add tests.

* Warn if RETURN cannot be parsed.

* Adjust tests. Also test for warning.

* if -> elif (otherwise EXAMPLE will be parsed too).

* Always parse return documentation, and fail if it is invalid YAML.

* Polishing.

* Mostly re-enable ansible-doc tests.

Listing from the local collection seems to be somewhat broken. I assume this
is why the test was disabled.

* Lint and make tests work with Python 2.

* Keep FQCNs in plugins (not modules), i.e. restore previous state.
2020-06-11 11:03:43 -07:00
Alicia Cozine
f509a22f9d
add changelog categories, update CVE fragments to use security_fix category (#69968)
* use security_fix category in changelogs for CVEs

* these fragments do not say CVE but are security fixes

Co-authored-by: Alicia Cozine <acozine@users.noreply.github.com>
2020-06-11 10:24:01 -07:00
Nilashish Chakraborty
9d6b0f2b03
Support removed_at_date in ansible-doc (#70002)
* Support removed_at_date in ansible-doc

Signed-off-by: NilashishC <nilashishchakraborty8@gmail.com>

Changes:
  * ansible-doc does not support `removed_at_date` and assumes that
    deprecated dict will either have `removed_in` or `version`. This
    results in ansible-doc (and hence "sanity --test=ansible-doc")
    failing for modules having only `removed_at_date`.

  * This patch adds support for `removed_at_date` and also gives it
    precedence over `removed_in` or `version`.

* Add tests and changelog

Signed-off-by: NilashishC <nilashishchakraborty8@gmail.com>
2020-06-11 13:18:18 -04:00
Brian Coca
ebb22655e4
implemented 'prefix' for file based cache (#69872)
* implemented 'prefix' for file based cache

Co-authored-by: s-hertel <shertel@redhat.com>
2020-06-11 10:40:25 -04:00
Jordan Borean
d30fc6c0b3
galaxy - preserve symlinks on build/install (#69959)
* galaxy - preserve symlinks on build/install

* Handle directory symlinks

* py2 compat change

* Updated changelog fragment
2020-06-11 06:46:42 +10:00
Brian Coca
37e7191304
added more ways to config new options in url (#69950)
* added more ways to config new options in url
2020-06-10 15:48:20 -04:00
Martin Krizek
ac20466375
Do not keep empty blocks after tag filtering (#69987)
This prevents PlayIterator having to go through empty blocks
that were created in filter_tagged_tasks. This should
be a performance improvement for playbooks that mostly skip
tasks with tags.

ci_complete
2020-06-10 12:12:11 -05:00
Brian Coca
026bf4a9a8
do we really need update vars from pc? (#69952)
* don't we really need update vars from pc
2020-06-10 08:39:47 -04:00
Brian Coca
805dff4129
fixed missing default (#69972)
* fixed missing default

 also deprecated so new options wont have to go through this
2020-06-10 08:35:36 -04:00
Matt Clay
a81dd4f06a
Update ansible-test default containers. (#69819)
* Move ansible-test completion code.

* Fix a few type hints.

* Change docker completion based on context.

Collections now use version 2.0 of the default-test-container.
This is an updated version of the earlier 1.x default-test-container with ansible-base and cloud specific requirements removed.

Testing of ansible-base now uses version 1.0 of a new ansible-base-test-container.
This container is similar to the earlier 1.x default-test-container, but with unnecessary duplication of requirements across Python versions removed.

Collections which have tests that depend on requirements no longer present in the default test container should specify them in their test requirements files:

* tests/integration/requirements.txt
* tests/unit/requirements.txt

* Bump test container versions

Co-authored-by: Jordan Borean <jborean93@gmail.com>
2020-06-09 15:40:56 -07:00
Sloane Hertel
51f6d129cb
support hard coded module_defaults.yml groups for collections (#69919)
* Only allow groups which were hardcoded in module_defaults.yml

only load action groups from the collection if module_defaults contains a potential group for the action

* Fix tests using modules that override those whitelisted in lib/ansible/config/module_defaults.yml

Third party modules should not be using group/ - use the action name instead

* add externalized module_defaults tests

add the missing group and collections

ci_complete

Co-authored-by: Matt Davis <mrd@redhat.com>

* changelog

ci_complete

* Fix import in tests

ci_complete

* Update with requested changes

ci_complete

* don't traceback since we don't validate the contents of module_defaults

ci_complete

Co-authored-by: Matt Davis <mrd@redhat.com>
2020-06-09 15:38:57 -07:00
Felix Fontein
a862ff2d43
Deprecation revisited (#69926)
* Allow to specify collection_name separately for deprecation.

* Use new functionality in Ansible.

* Use new functionality in tests.

* Update tagging/untagging functions.

* Update pylint deprecated sanity test.

* Update validate-modules. Missing are basic checks for version_added (validate semantic version format for collections).

* Improve version validation. Re-add version_added validation.

* Make sure collection names are added to return docs before schema validation.

* Extra checks to avoid crashes on bad data.

* Make C# module utils code work, and update/extend tests.

* Add changelog fragment.

* Stop extracting collection name from potentially tagged versions/dates.

* Simplify C# code.

* Update Windows modules docs.

* Forgot semicolons.
2020-06-09 15:21:19 -07:00
Matt Clay
7bff3d312f
Split out and install sanity test requirements. (#69971)
* Split out sanity test requirements.

* Run each --venv test separately.

This provides verification that the requirements for each test are properly specified.

* Use a separate requirements file per sanity test.

* Skip setuptools/cryptography setup for sanity.

* Eliminate pyyaml missing warning.

* Eliminate more pip noise.

* Fix conflicting generate_pip_install commands.

* Add changelog fragment.
2020-06-09 13:38:36 -07:00
Brian Coca
ce199ef0e1
Add intg test to repro #36045 (add_host traceback) (#69912)
* Add intg test to repro #36045 (add_host traceback)

* fix raw_params usage in add_host

Co-authored-by: Adrian Likins <alikins@redhat.com>
2020-06-09 15:25:56 -04:00
Sam Doran
6065638e00
Display parameter name in string conversion warning (#57145)
Add prefix to error message for nested options. This is helpful if a subelement key has the same name as a top level key.
2020-06-09 13:05:53 -04:00
Brian Coca
a64418c2b3
avoid constant checking controlpersist (#69910)
* avoid constant checking controlpersist
2020-06-09 11:59:38 -04:00
Sam Doran
c41a160951
Update test images to version 1.16.0 (#66771) 2020-06-09 10:08:31 -04:00
Matt Martz
bfff95393c
Add custom globals to the environment, and not per template (#69278)
* Add custom globals to the environment, and not per template

* Add changelog fragment
2020-06-08 16:30:40 -05:00
Matt Martz
9667f221a5
Make AnsibleVaultEncryptedUnicode work more like a string (#67823)
* Make AnsibleVaultEncryptedUnicode work more like a string. Fixes #24425

* Remove debugging

* Wrap some things

* Reduce diff

* data should always result in text

* add tests

* Don't just copy and paste, kids

* Add eq and ne back

* Go full UserString copy/paste

* Various version related fixes

* Remove trailing newline

* py2v3

* Add a test that can evaluate whether a variable is vault encrypted

* map was introduces in jinja2 2.7

* moar jinja

* type fix

Co-Authored-By: Sam Doran <sdoran@redhat.com>

* Remove duplicate __hash__

* Fix typo

* Add changelog fragment

* ci_complete

Co-authored-by: Sam Doran <sdoran@redhat.com>
2020-06-08 16:30:14 -05:00
Sam Doran
8dd0356719
Update bundled version of distro from 1.4.0 to 1.5.0 (#69709)
Includes changes we made in our version that were merged upstream

ci_complete
2020-06-08 16:34:37 -04:00
Matt Martz
c1c6f61a18
Auto unroll generators produced by jinja filters (#68014)
* Auto unroll generators produced by jinja filters

* Unroll for native in finalize

* Fix indentation

Co-authored-by: Sam Doran <sdoran@redhat.com>

* Add changelog fragment

* ci_complete

* Always unroll regardless of jinja2

* ci_complete

Co-authored-by: Sam Doran <sdoran@redhat.com>
2020-06-08 12:58:03 -05:00
Felix Fontein
840d3a9dd7
ansible-doc: properly handle suboptions (#69795)
* Specifically handle suboptions, contains, etc in ansible-doc.
2020-06-08 11:51:33 -04:00
Matt Clay
06efc41fb0
Reduce pip output noise in ansible-test. (#69925) 2020-06-05 18:25:48 -07:00
Rick Elrod
efe103cdff
Make add_host be idempotent/show changed status (#69897)
Change:
- The `add_host` action now shows an accurate change status.

Test Plan:
- Added a plethora of integration tests.

Tickets:
Fixes #69881

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-06-04 21:01:56 -05:00
Sam Doran
f5718a354c
lineinfile - Add alternative backrefs example to docs (#69844)
* lineinfile - Add alternative backrefs example to docs

* Add changelog
2020-06-04 16:13:42 -05:00
the-antz
66bd385f6f
Fix documentation for the assemble modules remote_src property. (#69845) 2020-06-04 15:30:45 -04:00
Ganesh Nalawade
646536643f
Use fqcr from command module invocation using shell module (#69790)
* Use fqcr from command module invocation using shell module

Fixes https://github.com/ansible/ansible/issues/69788

Use fully qualified collection reference while invoking
command module from shell module

* Fox review comment
2020-06-04 10:00:11 -07:00
Alex Schultz
247e43b252
Fix IncludedFile equality check (#69524)
In the case of a free style strategy, it is possible to end up with
multiple hosts trying to include from the same role, however the tasks
being included may be different with the use of tasks_from.  Previously
if you had two hosts that were included the same role when the
process_include_results function tries to determine if a included needs
to be run on a specific host, it would end up merging two different
tasks into which ever one was processed first.

This change updates the equality check to also check if the task uuid
associated with the IncludedFile is the same. The previous check only
checked if the task's parent uuid was the same. This breaks down when
both includes have the same parent.

    - hosts: all
      strategy: free
      gather_facts: false
      tasks:
        - include_role:
            name: random_sleep
        - block:
          - name: set a fact (1)
            include_role:
              name: set_a_fact
              tasks_from: fact1.yml
          - name: set a fact (2)
            include_role:
              name: set_a_fact
              tasks_from: fact2.yml
        - name: include didn't run
          fail:
            msg: >
              set_a_fact didn't run
              fact1: {{ fact1 | default('not defined')}}
              fact2: {{ fact2 | default('not defined') }}"
          when: (fact1 is not defined or fact2 is not defined)

Closes #69521
2020-06-04 10:41:46 -05:00
Brian Coca
22da903e9c
return changed for group_by (#32057)
* return changed for group_by

* added tests and fixed 'early registeration'
2020-06-03 12:46:40 -04:00
David Shrewsbury
2d394382cf
Reap collections changelog fragments - 6 (#69491) 2020-06-02 11:39:46 -05:00
Andrew Klychkov
00ead98595
systemd: should fail in check_mode when service not found on host (#68136)
* systemd: should fail in check_mode when service not found on host
2020-06-02 11:52:12 -04:00
Abhijeet Kasurde
d757995bb6
service_facts: Update documentation about service status (#69757)
Update documentation about undocumented service status provided
by systemd

Fixes: #69752

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2020-06-02 11:07:02 -04:00
Dongsu Park
598e3392a9
Discover Flatcar Linux properly for hostname (#69627)
To avoid issues with Flatcar Container Linux being unable to be found,
detect Flatcar distro name especially for hostname, just like CoreOS
Container Linux was supported.

See also https://github.com/ansible/ansible/issues/69516
2020-06-02 18:41:53 +05:30
Jack
80f09efd03
do not return the body even if it failed (#69706)
* do not return the body even if it failed

* add some tests for this and rebase

* import test task

* ignore_errors when fails

Co-authored-by: Jack Zhang <jack.zhang@aspiraconnect.com>
2020-06-01 09:55:38 -04:00
Rick Elrod
2abaf320d7
[ansiballz] ensure that '' is not in sys.path (#69342)
Change:
On OpenBSD when using pipelining, we do not set cwd which results in a
permissions fatal. Ensure that `''` - cwd - is not in `sys.path`.

Test Plan:
Tested against local OpenBSD VM

Tickets:
Fixes #69320

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-06-01 03:43:20 -05:00
Hideki Saito
79ab798427 Enable logging using setup_loggers() API in dnf-4.2.17-6 or later
- Fixed issue #28061
- https://bugzilla.redhat.com/show_bug.cgi?id=1788212

Signed-off-by: Hideki Saito <saito@fgrep.org>
2020-05-30 20:37:27 -05:00
Martin Krizek
3606dcfe65
Ensure --version works with non-ascii project path (#66624)
Fixes #66617
2020-05-29 14:42:44 -04:00
Sloane Hertel
e40889e711
Add support to install collections from git repositories (#69154)
* Enable installing collections from git repositories

* Add tests for installing individual and multiple collections from git repositories

* Test to make sure recursive dependencies with different syntax are deduplicated

* Add documentation

* add a changelog

* Skip Python 2.6

* Only fail if no collections are located in a git repository

Add support for a 'type' key for collections in requirement.yml files.
Update the changelog and document the supported keys and allowed values for the type.

Add a note that the collection(s) in the repo must contain a galaxy.yml

* Add a warning about embedding credentials in SCM URLs

* Update with review suggestions

* suppress sanity compile failure for Python 2.6
2020-05-29 13:33:32 -04:00
Felix Fontein
0b82d4499e
ansible-test ansible-doc sanity test: also run with --json parameter (#69288) 2020-05-29 08:52:29 -07:00
Abhijeet Kasurde
7b3d9cbcb8
distribution: Added support for openEuler OS (#69324)
Detect os_family for openEuler OS as 'RedHat', instead of 'openEuler'.

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2020-05-29 12:43:16 +05:30
Jordan Borean
f81f5da20e
Ansible.Basic added generic fragment merger for module options (#69719) 2020-05-29 16:11:38 +10:00
Felix Fontein
40f21dfd3c
Version source tagging (automatic and manual) for version_added and deprecation versions (#69680)
* Track collection for version_added.
Validate *all* version numbers in validate-modules.
For tagged version numbers (i.e. version_added), consider source collection to chose validation.

* Make tagging/untagging functions more flexible.

* Tag all versions in doc fragments.

* Tag all deprecation versions issued by code.

* Make Display.deprecated() understand tagged versions.

* Extend validation to enforce tagged version numbers.

* Tag versions in tests.

* Lint and fix test.

* Mention collection name in collection loader's deprecation/removal messages.

* Fix error IDs.

* Handle tagged dates in Display.deprecated().

* Also require that removed_at_date and deprecated_aliases.date are tagged.

* Also automatically tag/untag removed_at_date; fix sanity module removal version check.

* Improve error message when invalid version number is used (like '2.14' in collections).
2020-05-28 22:46:16 -07:00
Felix Fontein
31bf3a5622
Deprecate module in collection: allow removal date in documentation, make validate-modules ensure version and date match (#69727)
* Allow to deprecate module by date in documentation.

* Make sure deprecation date/version match between module docs and meta/runtime.yml.

* Unrelated fix: don't compare deprecated module version to Ansible's version in collection.

* Allow documentation's removal version to be something else than fixed list of Ansible versions for collections.

* Linting.

* Allow to deprecate plugin options by date.

* Add changelog fragment for deprecation by date (also covers #68177).
2020-05-28 14:20:52 -07:00
Martin Krizek
061c6c7c6f
Prevent losing error info by including both loop and cond error msgs (#68485)
Fixes #66529
2020-05-28 17:13:39 -04:00
Ganesh Biradar
187de7a8aa
Updated CLI help for 'ansible-galaxy' for 'collection' subcommand (#69458)
* Added YAML header line and link to existing github issue
2020-05-28 10:38:48 -04:00
Sam Doran
d7618712fd
blockinfile - fix line end regression (#69734)
PR #66461 introduced a regression that resulted in an in correct block in the file
if the block to be inserted did not end with a line separator. Fix this bug and add
tests to cover this scenario.
2020-05-27 16:54:38 -04:00
Matt Clay
65c57a7f5b Update default-test-container to version 1.14. 2020-05-27 11:31:38 -07:00
Riyad Preukschas
e5cc12a64f
blockinfile - Preserve line endings (#66461)
Fixes #64966
* Fix "TypeError: splitlines() takes no keyword arguments" on Python2.7
* Add changelog fragment
* Don't use `grep -P` for BSD/macOS compatibility
* Fix sanity checks complaining about test fixtures with mixed line endings
* Update changelogs/fragments/66461-blockinfile_preserve_line_endings.yaml
2020-05-27 11:05:07 -04:00
Abhijeet Kasurde
564907d8ac
distribution: Refactor test_distribution_version testcases (#69300) 2020-05-27 14:01:31 +05:30
Rick Elrod
304c3e57e8
[ssh] Add new sshpass_prompt option (#68874)
Change:
Allows the user to configure sshpass (1.06+) to look for a different
substring than the default "assword" that it comes with.

Test Plan:
Set a custom ssh password prompt on a VM with PAM and tried connecting to
it. Without `ansible_sshpass_prompt` set in inventory: experienced hang.
With `ansible_sshpass_prompt` in inventory: connected successfully.

Tried setting `ansible_sshpass_prompt` with an older `sshpass` in PATH
and got a loud error, as expected.

Tickets:
Fixes #34722, fixes #54743, refs #11565.

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-05-26 19:58:31 -05:00
Matt Davis
f7dfa817ae
collection routing (#67684)
* `meta/` directory in collections
* runtime metadata for redirection/deprecation/removal of plugin loads
* a compatibility layer to keep existing content working on ansible-base + collections
* a Python import redirection layer to keep collections-hosted (and otherwise moved) content importable by things that don't know better
* supported Ansible version validation on collection loads
2020-05-26 09:42:06 -07:00
Brian Coca
de3f7c7739
fix delegated interpreter discovery (#69604)
* fix delegated interpeter
* allow returning fact if it is 'the right host'
* added note for future fix/efficiency
 as it stands we rerun discovery for the delegated host
unless its saving facts to itself
 * fixed test lacking delegate_to mock
2020-05-22 09:31:34 -04:00
James Cammarata
a4072ad0e9
Split regular and handler results into their own queues (#69498)
When mixed with the free strategy (or any custom strategy that does not behave in
a lock-step manner), the linear methodology of _wait_on_handler_results may cause
race conditions with regular task result processing if the strategy uses
_process_pending_results directly. This patch addresses that by splitting the queues
used for results and adding a flag to _process_pending_results to determine which
queue to check.

Fixes #69457
2020-05-21 15:55:08 -05:00
Sam Doran
cd8920af99
file - return 'state': 'absent' when a file does not exist (#66503)
This was changed in ansible/ansible#51350.
2020-05-21 16:35:45 -04:00
Matt Martz
e0f25a2b1f
Strip no log values from module response keys. Fixes #68400 (#69653) 2020-05-21 16:17:57 -04:00
Brian Coca
9281148b62
correctly merge multiple facts results (#68987)
* correctly merge multiple facts results

  fixes #68532
2020-05-20 18:53:37 -04:00
Matt Martz
b748edea45
Add multipart/form-data functionality (#69376)
* Add multipart/form-data functionality

* Fix some linting issues

* Fix error message

* Allow filename to be provided with content

* Add integration test

* Update examples

* General improvements to multipart handling

* Use prepare_multipart for galaxy collection publish

* Properly account for py2 vs py3, ensuring no max header length

* Address test assumptions

* Add unit tests

* Add changelog

* Ensure to use CRLF instead of NL

* Ignore line-endings in fixture

* Consolidate code, add comment

* Bump fallaxy container version

* ci_complete
2020-05-20 15:44:01 -05:00
Martin Krizek
ecd986006e
Properly handle unicode in safe_eval (#68576)
* Properly handle unicode in safe_eval

Fixes #66943

* Update lib/ansible/template/safe_eval.py

Co-authored-by: Sam Doran <sdoran@redhat.com>
2020-05-20 12:08:50 -04:00
Sam Doran
f01de15d25
ansible-galaxy - remove warning during collection install (#69541)
* ansible-galaxy - remove warning during collection install

If existing collections do not contain a MANIFEST.json, which is common
for collections under development that were not installed from Ansible
Galaxy, fall back to inspecting galaxy.yml rather than displaying a
warning.

A warning will still be displayed in neither a MANIFEST.json nor
galaxy.yml are present.

* Update unit tests
2020-05-19 10:31:49 -05:00
Felix Fontein
43acd61901
ansible-test local change detection: use --base-branch if specified (#69508) 2020-05-18 13:18:31 -07:00
Jordan Borean
ecea15c508
Unify ansible-galaxy install -r (#67843)
* Unify ansible-galaxy install -r

* Minor nit fixes for docs

* Re-align warnings

* Fix up integration test

* Fix up test where no roles/collections were in file
2020-05-19 05:09:42 +10:00
Matt Clay
86743292ec Remove incorrect function args in unarchive. 2020-05-15 18:31:43 -07:00
Matt Clay
c495c92a6e Code cleanup in ansible-test. 2020-05-15 18:17:11 -07:00
Matt Clay
ed4fd9be67 Code cleanup in ansible-inventory. 2020-05-15 18:02:18 -07:00
Matt Clay
159d79995c Freeze pycodestyle in ansible-test. 2020-05-15 16:38:50 -07:00
Matt Clay
e7c2eb519b
Add Shippable request signing to ansible-test. (#69526) 2020-05-15 15:38:02 -07:00
Sam Doran
b8469d5c7a
lineinfile - Use ANSIBLE_REMOTE_TMP for temporary file (#69543) 2020-05-15 15:52:17 -04:00
Lukas Pirl
34db57a47f
introduce fact "ansible_processor_nproc": number of usable vcpus (#66569)
This fact reflects the number of usable vcpus (which might be different
from ansible_processor_vcpus, e.g., in containers with limits). See
also #51504.

* Add fixture data and update unit tests

Co-authored-by: Sam Doran <sdoran@redhat.com>
2020-05-15 09:38:56 -04:00
Matt Clay
83381cd912 Update the default-test-container in ansible-test.
The updated container includes fewer requirements now that the collection migration has completed.

Collections which encounter test issues with this new container should update their test requirements files to include the necessary requirements.
2020-05-14 21:24:58 -07:00
Matt Clay
d8e0aadc0d Update ansible-test support for CI providers.
Refactored CI provider code to simplify multiple provider support and addition of new providers.
2020-05-14 17:08:30 -07:00
Richard Lau
9d8cf1ed21
Add path for yum on IBM i to PKG_MGRS (#69484)
* Add path for yum on IBM i to PKG_MGRS

On IBM i, yum is installed under the `/QOpenSys/pkgs` prefix, see:
https://bitbucket.org/ibmi/opensource/src/master/docs/yum/#markdown-header-must-know-usage-notes-read-this-after-you-install

* Add changelog fragment
2020-05-14 19:09:09 -04:00
Chris Holland
dfa2863dee
Remove FIXME where no change is needed (#69466)
* Added exception

* Added changelog fragment

* Remove FIXME comment

* Edit changelog fragment
2020-05-14 15:44:33 -04:00
Bob Weinand
f200487414
Fix filedescriptor out of range in select() when running commands (#65058)
* Fix filedescriptor out of range in select() when running commands

* Simplify the run_command() code

Now that we're using selectors in run_command(), we can simplify some of
the code.

* Use fileobj.read() instead of os.read()
* No longer use get_buffer_size() as we can just slurp all of the data
  instead.

Also use a simpler conditional check of whether the selector map is
empty

Co-authored-by: Toshio Kuratomi <a.badger@gmail.com>
2020-05-14 11:46:34 -04:00
Brian Coca
87d9b49de2
Fix listing of colleciton plugins with symlinks (#69305)
* Fix listing of colleciton plugins with symlinks
2020-05-14 11:45:02 -04:00
Ted Pearson
c13b040e67
systemd: set scope default to "system" (#67006)
- Make "system" default scope for systemd
- Remove extra None checks since there is now a default
2020-05-14 11:36:47 -04:00
Matt Martz
4c4406b2df
Flatten the directory hierarchy of modules (#68966)
* Flatten the directory hierarchy of modules

* Update ignore.txt, flatten units

* Update imports

* Completely flatten the modules directory

* Update sanity ignore

* Fix some sanity test ignores

* Fix relative import

* Fix docs builds without category

* ci_complete

* Clean up docs. ci_complete

* Adjust needs/file alias

* ci_complete

* fix hardcoded ping module paths

Co-authored-by: Matt Davis <mrd@redhat.com>
2020-05-13 19:14:53 -07:00
Felix Fontein
0e15375ffe
Add deprecated removed_in_version and deprecated_aliases version tests (#66920) 2020-05-13 13:58:09 -07:00