Commit graph

10462 commits

Author SHA1 Message Date
Sam Doran
be5aab177c
[stable-2.10] Drop "rhui-" prefix from RHEL repositories in AMI (#71130) (#71131)
The repository names seem to have changed and no longer have the "rhui-" prefix
(cherry picked from commit 6ac4439a6a)

Co-authored-by: Sam Doran <sdoran@redhat.com>
2020-08-06 16:58:58 -05:00
Sloane Hertel
9f4748cb2d
fix downloading collections in git repos and tar.gz artifacts (#70524) (#71000)
* Fix downloading tar files

* Fix downloading SCM collections

* changelog

(cherry picked from commit 54e2ae79e7)
2020-07-30 13:37:59 -07:00
Sam Doran
719c40bfdf
[stable-2.10] facts - fix incorrect time for some date_time_facts (#70665) (#70996)
* [stable-2.10] facts - fix incorrect time for some date_time_facts (#70665)

The iso8601_micro and iso8601 facts incorrectly called now.utcnow(), resulting
in a new timestamp at the time it was called, not a conversion of the previously
stored timestamp.

Correct this by capturing the UTC timestamp once then calculating the local
time using the UTC offset of the current system.

* Use time.time() for getting the current time
* Convert from that stored epoch timestamp to local and UTC times
* Used existing timestamp for epoch time
* Add unit tests that validate the formate of the return value rather than an exact value since mocking time and timezone is non-trivial
(cherry picked from commit c4f442ed5a)

Co-authored-by: Sam Doran <sdoran@redhat.com>

* Remove tests for tz_dst since that only exists in newer versions
2020-07-30 13:37:19 -07:00
Matt Clay
b764d381f0
[stable-2.10] Fix ansible-test relative import analysis. (#70993)
(cherry picked from commit 2e0097ada3)

Co-authored-by: Matt Clay <matt@mystile.com>
2020-07-30 13:36:49 -07:00
Sam Doran
5cb96087e6
Fix warning for new default permissions when mode is not specified (#70976) (#70985)
Follow up to #70221
Related to #67794
CVE-2020-1736

When set_mode_if_different() is called with mode of 'None', ensure we issue
a warning about the change in default permissions.

Add integration tests to ensure the warning works properly.

* Fix tests
- actually use custom module 🤦‍♂️
- verify file permission on created files
- use remote_tmp_dir so we're ready for split controller
- improve test module so we can skip the call to set_fs_attributes_if_different()
- fix tests for CentOS 6

(cherry picked from commit dc79528cc6)
2020-07-30 10:10:23 -07:00
Jordan Borean
a8217f1bd4
ansible-galaxy - fix fallback for AH searches (#70957) - 2.10 (#70980)
* ansible-galaxy - fix fallback for AH searches (#70957)

(cherry picked from commit b1cb2553af)

* Fix tests after backport
2020-07-30 10:08:25 -07:00
Brian Coca
a75b3601d9
Allow changed/failed mgmt on strategy actions (#70919) (#70968)
* Allow changed/failed mgmt on strategy actions

(cherry picked from commit f9c3c6cba6)
2020-07-29 14:32:33 -07:00
Felix Fontein
c39e536d84
validate-modules: fix version_added validation for top-level, fix error codes (#70869) (#70947)
* Also validate top-level version_added.

* Fix error code.

* Produce same version_added validation error in schema than in code (and stop returning it twice).

* Return correct error codes for invalid version_added for options and return values.

* Add changelog.

* Fix forgotten closing braket.

* Accept 'historical' for some top-level version_added.

(cherry picked from commit 7e2cc7db12)
2020-07-29 13:54:45 -07:00
Felix Fontein
155041d8ee
Fix ansible-test error in community.aws (#70507) (#70873)
* Fix ansible-test error in community.aws

* Add changelog entry for fix

* Change check from None to string_types

* Update changelogs/fragments/70507-validate-null-author.yaml

clarify wording "or a list of strings"

Co-authored-by: Felix Fontein <felix@fontein.de>

* Update test/lib/ansible_test/_data/sanity/validate-modules/validate_modules/schema.py

clarify wording - single string or not specified valid

Co-authored-by: Felix Fontein <felix@fontein.de>

* Do not fail but return None when given outside list

Co-authored-by: Felix Fontein <felix@fontein.de>
(cherry picked from commit b0d9deeae3)

Co-authored-by: Alan Rominger <arominge@redhat.com>
2020-07-29 13:51:47 -07:00
Sviatoslav Sydorenko
be28e98cf2
[backport][2.10][PR #70446] Refactor Python API examples and docs (#70850)
* Add boilerplate snippet into `examples/`

It is a partial backport of #70224

(partially cherry picked from commit 4816bb4f43)

* Refactor Python API examples and docs

PR #70446: it's a follow-up for #70445.

It includes a merge of `examples/scripts/uptime.py` and a similar
code snippet from `docs/docsite/rst/dev_guide/developing_api.rst`.

This patch also changes the docs RST file to include contents of
the example file instead of holding a copy of a similar code.

(cherry picked from commit 20bb915092)
2020-07-29 13:51:21 -07:00
Matt Clay
4280efccc4 [stable-2.10] Cap pytest version to avoid relative import issue.
(cherry picked from commit 3a8ac62596)

Co-authored-by: Matt Clay <matt@mystile.com>
2020-07-28 17:14:30 -07:00
Brian Coca
9b992c0b78
reset logging to INFO (#70878) (#70881)
- due to CVE-2019-14846
 - also added comments and test to avoid 'oportunistic' reversion

(cherry picked from commit 1223ce656a)
2020-07-24 15:47:57 -05:00
Rick Elrod
c53a68bf16
Update ansible-base test container to 1.5.0. (#70856) (#70857)
Co-authored-by: Matt Clay <mclay@redhat.com>
2020-07-23 21:36:41 -05:00
Jordan Borean
e9c9c02e0a
Do not add connection vars to the output results (#70853) (#70855)
* Do not add connection vars to the output results

* Also revert the delgated scenario JIC

* Added regression test

(cherry picked from commit 5e1a968983)
2020-07-23 19:48:45 -05:00
Matt Davis
cc8d180801
fix internal cases of actions calling unqualified module names (#70818) (#70840)
* fix internal cases of actions calling unqualified module names

* add porting_guide entry
* misc other fixes around action/module resolution broken by redirection

ci_complete

* Update docs/docsite/rst/porting_guides/porting_guide_2.10.rst

Co-authored-by: Rick Elrod <rick@elrod.me>

* Update docs/docsite/rst/porting_guides/porting_guide_2.10.rst

Co-authored-by: Rick Elrod <rick@elrod.me>

* address review feedback

* pep8

* unit test fixes

* win fixes

* gather_facts fix module args ignores

* docs sanity

* pep8

* fix timeout test

* fix win name rewrites

Co-authored-by: Rick Elrod <rick@elrod.me>
(cherry picked from commit 4c0af6c808)
2020-07-23 10:29:09 -07:00
Sam Doran
7e4cffc5d2
[stable-2.10] Change default file permissions so they are not world readable (#70221) (#70824)
* Change default file permissions so they are not world readable

CVE-2020-1736

Set the default permissions for files we create with atomic_move() to 0o0660. Track
which files we create that did not exist and warn if the module supports 'mode'
and it was not specified and the module did not call set_mode_if_different(). This allows the user to take action and specify a mode rather than using the defaults.

A code audit is needed to find all instances of modules that call atomic_move()
but do not call set_mode_if_different(). The findings need to be documented in
a changelog since we are not warning. Warning in those instances would be frustrating
to the user since they have no way to change the module code.

- use a set for storing list of created files
- just check the argument spac and params rather than using another property
- improve the warning message to include the default permissions.
(cherry picked from commit 5260527c4a)

Co-authored-by: Sam Doran <sdoran@redhat.com>
2020-07-23 09:07:18 -07:00
Brian Coca
8c2754e6d3
Allow hostvars delegation (#70331) (#70810)
* ensure hostvars are available on delegation
* also inventory_hostname must point to current host and not delegated one
* fix get_connection since it was still mixing original host vars and delegated ones
* also return connection vars for delegation and non delegation alike
* add test to ensure we have expected usage when directly assigning for non delegated host

(cherry picked from commit 84adaba6f5)
2020-07-22 18:29:07 -07:00
David Shrewsbury
7cdba7c923
Sanitize URI module keys with no_log values (#70762) (#70820)
* Add sanitize_keys() to module_utils.

* More robust tests

* Revert 69653 change

* Allow list or dict

* fix pep8

* Sanitize lists within dict values

* words

* First pass at uri module

* Fix insane sanity tests

* fix integration tests

* Add changelog

* Remove unit test introduced in 69653

* Add ignore_keys param

* Sanitize all-the-things

* Ignore '_ansible*' keys

* cleanup

* Use module.no_log_values

* Avoid deep recursion issues by using deferred removal structure.

* Nit cleanups

* Add doc blurb

* spelling

* ci_complete

(cherry picked from commit bf98f031f3)
2020-07-22 18:28:24 -07:00
psi / Ryo Hirafuji
61f8f8ce7f
cron - Allow non-ascii (UTF-8) chars in cron file paths and jobs (#70426) (#70794)
* Encode/Decode files in UTF-8
* Use helper function in ansible
* Add an integration test
* Use emoji in test data.
* add changelog
* Also support non-ascii chars in filepath and add tests about this.
* Also use non-ascii chars in replaced text and ensure not to break cron syntax.
* rename self.existing to self.n_existing
* rename crontab.existing to crontab.n_existing
2020-07-22 18:26:10 -07:00
Matt Martz
7eb5f53294
[stable-2.10] Ensure single vaulted values aren't counted as sequences. Fixes #70784 (#70786) (#70791)
(cherry picked from commit 96b74d3)

Co-authored-by: Matt Martz <matt@sivel.net>
2020-07-22 18:25:38 -07:00
Baptiste Mille-Mathias
1eb2afac63
Create home and parent directories only when requested (#70790)
The home user and the parents directories should only be created when
create_home == True

(cherry picked from commit f3dd8d3052)
2020-07-22 18:23:47 -07:00
Martin Krizek
4170786cd9
2.10: Detect failure in always block after rescue (#70094) (#70204)
* Detect failure in always block after rescue (#70094)

* Detect failure in always block after rescue

Fixes #70000

ci_complete

* Add more tests

(cherry picked from commit 0ed5b77377)

* add changelog

Co-authored-by: Matt Davis <mrd@redhat.com>
2020-07-22 14:00:27 -07:00
Baptiste Mille-Mathias
ffd3757fc3
Fix missing quoting for remote_tmp in second mkdir of shell module. Issue #69577 (#69578) (#70757)
* Fix missing quoting for remote_tmp in second mkdir of shell module. Issue #69577

* adding changelog

* fixing typo in changelog entry

* adding test case

Adding test case written by bmillemayhias.

* using $HOME instead of ~

* fixing commit measage

* Update 69578-shell-remote_tmp-quoting.yaml

Co-authored-by: Brian Kohles <me@briankohles.com>
(cherry picked from commit 77d0effcc5)

Co-authored-by: Brian Kohles <briankohles@users.noreply.github.com>
2020-07-21 11:23:25 -07:00
Sam Doran
9b8a649f2e
[stable-2.10] Handle Slackware OS version strings containing a plus (“+”) (#68142) (#70717)
A couple of years ago Slackware -current began using a plus (“+”) at the end of the distribution version string to indicate a future version work-in-progress.

Rearrange distribution_files unit tests to easily support more tests
  - add conftest with common fixtures
  - use parametrize for testing multiple scenarios

* Add changelog
* Add unit tests for Slackware distribution parsing
* Use correct fixtures for Slackware
Data comes from /etc/slackware-version

Co-authored-by: Sam Doran <sdoran@redhat.com>
Co-authored-by: <Eduard Rozenberg <eduardr@pobox.com>>
(cherry picked from commit 566c5e6ce1)

Co-authored-by: Eduard Rozenberg <2648417+edrozenberg@users.noreply.github.com>

Co-authored-by: Eduard Rozenberg <2648417+edrozenberg@users.noreply.github.com>
2020-07-21 11:21:07 -07:00
Sloane Hertel
d329985d4c
[2.10] template connection variables accessed directly before using (#70657) (#70688)
* template connection variables accessed directly before using (#70657)

* template variables accessed directly when using them instead of FieldAttributes

(cherry picked from commit 8c213c9334)

* changelog
2020-07-21 11:19:52 -07:00
Abhijeet Kasurde
ed07821a59
[2.10] api: time.clock compatible code (#70677)
time.clock is removed in Python 3.8. Add time.clock
compatible code.

Fixes: #70649

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
(cherry picked from commit 055871cbb8)
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2020-07-21 11:19:27 -07:00
Toshio Kuratomi
46b1a999c6
Collections docs generation backport (#70515)
* Build documentation for Ansible-2.10 (formerly known as ACD).

Builds plugin docs from collections whose source is on galaxy

The new command downloads collections from galaxy, then finds the
plugins inside of them to get the documentation for those plugins.

* Update the python syntax checks
  * docs builds can now require python 3.6+.

* Move plugin formatter code out to an external tool, antsibull-docs.
  Collection owners want to be able to extract docs for their own
  websites as well.
* The jinja2 filters, tests, and other support code have moved to antsibull
* Remove document_plugins as that has now been integrated into antsibull-docs

* Cleanup and bugfix to other build script code:
  * The Commands class needed to have its metaclass set for abstractmethod
    to work correctly
  * Fix lint issues in some command plugins

* Add the docs/docsite/rst/collections to .gitignore as
  everything in that directory will be generated so we don't want any of
  it saved in the git repository
* gitignore the build dir and remove edit docs link on module pages

* Add docs/rst/collections as a directory to remove on make clean
* Split the collections docs from the main docs

* remove version and edit on github
* remove version banner for just collections
* clarify examples need collection keyword defined

* Remove references to plugin documentation locations that no longer exist.
  * Perhaps the pages in plugins/*.rst should be deprecated
    altogether and their content moved?
  * If not, perhaps we want to rephrase and link into the collection
    documentation?
  * Or perhaps we want to link to the plugins which are present in
    collections/ansible/builtin?

* Remove PYTHONPATH from the build-ansible calls
  One of the design goals of the build-ansible.py script was for it to
  automatically set its library path to include the checkout of ansible
  and the library of code to implement itself.  Because it automatically
  includes the checkout of ansible, we don't need to set PYTHONPATH in
  the Makefile any longer.

* Create a command to only build ansible-base plugin docs
  * When building docs for devel, only build the ansible-base docs for
    now.  This is because antsibull needs support for building a "devel
    tree" of docs.  This can be changed once that is implemented
  * When building docs for the sanity tests, only build the ansible-base
    plugin docs for now.  Those are the docs which are in this repo so
    that seems appropriate for now.

* Docs: User guide overhaul, part 5 (#70307)

(cherry picked from commit db354c0300)

* Need to return any error code from running antsibull-docs (#70763)

This way we fail early if there's a problem

(cherry picked from commit 1e3989c9f7)

Co-authored-by: Alicia Cozine <879121+acozine@users.noreply.github.com>
2020-07-20 14:28:35 -07:00
Matt Martz
255dfca7f6
[stable-2.10] Allow single vault encrypted values to be used directly as module parameters. Fixes #68275 (#70607) (#70641)
(cherry picked from commit a77dbf0)

Co-authored-by: Matt Martz <matt@sivel.net>
2020-07-17 12:54:32 -07:00
Rick Elrod
86b24498b7
Add intentional coverage for an async_wrapper case (#70593) (#70630)
* Add intentional coverage for an async_wrapper case (#70593)

Change:
- Test async_wrapper when the module it runs has stderr output

Test Plan:
- CI
- Looked at coverage report and saw green for a few lines that weren't
  previously green.

Signed-off-by: Rick Elrod <rick@elrod.me>

* sigh

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-07-17 12:52:45 -07:00
Rick Elrod
33e5f1d661
Get m_u.facts.utils coverage up to 100% (#70614) (#70629)
* Get m_u.facts.utils coverage up to 100%

Change:
- Add tests to 'gathering_facts' integration target to get
  module_utils.facts.utils coverage up to 100%.
- This also clears incidental coverage from incidental_selinux.

Test Plan:
- CI

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-07-17 12:52:00 -07:00
Brian Coca
6cd015d7e2
Make filter type errors 'loop friendly' (#70417) (#70574)
- ensure we preserve the typeerror part of the exception so loop defereed error handling
 can postpone those caused by undefined variables until the when check is done.
 - fix tests to comply with the 'new normal'

 - human_to_bytes and others can issue TypeError not only on 'non string'
 but also bad string that is not convertable.

Co-authored-by: Sloane Hertel <shertel@redhat.com>

Co-authored-by: Sloane Hertel <shertel@redhat.com>
(cherry picked from commit cf89ca8a03)
2020-07-17 12:51:18 -07:00
Felix Fontein
15355ed059
[2.10] ansible-doc: include collection name in text output / plugin loader: return collection name; ansible-doc: handle ansible.builtin correctly (#70572)
* ansible-doc: include collection name in text output (#70401)

* ansible-doc: include collection name in text output

* Be more careful to not accidentally pass ansible.builtin for user-supplied modules.

(cherry picked from commit f4c89eab23)

* plugin loader: return collection name; ansible-doc: handle ansible.builtin correctly (#70026)

* Determine collection in plugin loader.

* Fix test.

* Use PluginPathContext objects in PluginLoader._plugin_path_cache instead of tuples.

(cherry picked from commit 24dcaf8974)
2020-07-17 12:50:23 -07:00
Mykola Grygoriev
5ea6de4e7d
Fix decrypt argument in assemble module (#70465) (#70560)
* Do not pass decrypt parameter to assemble module

* Add integration tests where decrypt=True

* Add changelog #70465

(cherry picked from commit 71c378e139)
2020-07-17 12:48:38 -07:00
David Shrewsbury
94a81f7b44
Make sure ansible_become treated as a boolean (#70484) (#70526)
* Make sure ansible_become treated as a boolean

(cherry picked from commit 8aca464b8b)
2020-07-17 12:46:33 -07:00
Felix Fontein
a5c0b11913
ansible-doc man formatter: fail with better error message when description isn't there (#70046) (#70485)
* ansible-doc man formatter: do not crash when description isn't there.
* Change to report a better error message when description is not there.
* Add test.

(cherry picked from commit 9164b96774)
2020-07-17 12:44:03 -07:00
Sloane Hertel
212d2024f4
[2.10] Handle post_validate templating errors and fix tests (#70240) (#70389)
* Handle post_validate templating errors and fix tests (#70240)

* Handle unexpected templating errors

* Fixes #70050

Fix up tests that weren't running and add tests for graceful templating error handling

(cherry picked from commit 30e70f4b63)

* changelog

ci_complete
2020-07-17 12:41:43 -07:00
Martin Krizek
7dfda4026e
Fix delegate_facts with interpreter not being set (#70293) (#70384)
Fixes #70168

ci_complete

Co-authored-by: Brian Coca <bcoca@users.noreply.github.com>
Co-authored-by: Matt Clay <matt@mystile.com>
(cherry picked from commit b05e00e99a)
2020-07-17 12:40:00 -07:00
Abhijeet Kasurde
e1c0688e43
[2.10] Improve ansible-galaxy STDOUT messages for collections (#70379)
- Fix issue #70010
- Add installation successful message
- This feature targets "collection" sub-command and does not affect "role" sub-command

Signed-off-by: Hideki Saito <saito@fgrep.org>
(cherry picked from commit 2d59e548f6)

Co-authored-by: Hideki Saito <saito@fgrep.org>
2020-07-17 12:39:16 -07:00
Matt Davis
1e03b54d23
refactor Python module_utils locator (#70610) (#70711)
* refactor Python module_utils locator

* no longer recursive
* embed special-case module code internally
* share common code between collections/not cases
* fixes #70134
* properly support subpackage redirection
* adds support for FQCN redirect targets used by migration (expands to FQ Python name)
* add tests

* add changelog

(cherry picked from commit c616e54a6e)
2020-07-17 10:57:44 -07:00
Felix Fontein
273159da4c
Include changelogs/changelog.yaml in distribution. (#70260) (#70266)
(cherry picked from commit ebd20ddca6)
2020-07-17 10:55:25 -07:00
Sam Doran
95ec1618ef
[stable-2.10] Only pass kwargs to our string checker not callable checkers (#70151) (#70170)
Since only check_type_str() accepts extra param, only pass to our checker and
do not pass kwargs to custom checkers.

* Add unit tests
(cherry picked from commit bc05415109)

Co-authored-by: Sam Doran <sdoran@redhat.com>
2020-07-17 10:53:56 -07:00
Sam Doran
b6f6067dfd
[stable-2.10] ansible_runner test - Add constraints (#70667) (#70669)
A recent updated to psutil, which is a dependency of ansible-runner, fails
to install on older versions of pip.

Commit with the breaking change:

  135628639b
(cherry picked from commit 9d27d7c8b1)

Co-authored-by: Sam Doran <sdoran@redhat.com>
2020-07-15 17:32:09 -05:00
Matt Clay
c2b69fd30e [stable-2.10] Fix ansible-test virtualenv management.
(cherry picked from commit 1e02a201a6)

Co-authored-by: Matt Clay <matt@mystile.com>
2020-07-14 02:08:17 -07:00
Matt Clay
ddc78622a4 [stable-2.10] Update ansible-test change detection for plugins.
(cherry picked from commit 09f02980a4)

Co-authored-by: Matt Clay <matt@mystile.com>
2020-07-13 23:55:15 -07:00
Matt Clay
957f0c27fc [stable-2.10] Fix boilerplate in setup.py and lib/ansible/ dir..
(cherry picked from commit 234994fc07)

Co-authored-by: Matt Clay <matt@mystile.com>
2020-07-13 18:57:17 -07:00
Matt Clay
a707466c0b [stable-2.10] Fix yamllint sanity test line numbers.
Parse errors from libyaml now compensate for the offset of the documentation within a module.
(cherry picked from commit 1a0d8a51cd)

Co-authored-by: Matt Clay <matt@mystile.com>
2020-07-13 18:28:30 -07:00
Matt Clay
8cd66ce95a [stable-2.10] Clean up unit test boilerplate.
(cherry picked from commit 98a0995fd0)

Co-authored-by: Matt Clay <matt@mystile.com>
2020-07-13 18:28:02 -07:00
Matt Clay
3204d260dd [stable-2.10] Add integration tests for basic.py _set_cwd.
These tests verify that AnsibleModule can be instantiated when cwd does not exist or is unreadable.
(cherry picked from commit d6fb42d1c5)

Co-authored-by: Matt Clay <matt@mystile.com>
2020-07-13 18:16:00 -07:00
Matt Clay
f28fabe6ef [stable-2.10] Add integration tests for test plugins. (#70576)
(cherry picked from commit df45dcdae0)

Co-authored-by: Matt Clay <mclay@redhat.com>
2020-07-13 18:15:21 -07:00
Matt Clay
ba645400bf [stable-2.10] Correct name of fileglob lookup integration test.
(cherry picked from commit bbd8f15a58)

Co-authored-by: Matt Clay <matt@mystile.com>
2020-07-13 18:15:00 -07:00