* check run state of current block only
* Add changelog and test
* Add test for issue 29047
* Fix for both tests
* blerg
* Change test messages
* fix tests
* Add multi-level block in rescue test case
* Add recursive rescue check and multi-level test
* Should probably run the new test
* ci_complete
* Merge new tests
* ci_complete
* POC for supporting callback events that come from the worker
* linting fixes. ci_complete
* fix up units. ci_complete
* Try moving the sentinel put higher. ci_complete
* safeguards. ci_complete
* Move queue killing to terminate
* LINTING. ci_complete
* Subclass Queue, to add helper send_callback method
* Just use _final_q instead of adding another queue and thread
* Revert a few changes
* Add helper for inserting a TaskResult into the _final_q
* Add changelog fragment
* Address rebase issue
* ci_complete
* Add test to assert async poll callback from fork
* Don't use full path
* ci_complete
* Use _results_lock as a context manager
* Add new generic lock decorator, and use it with send_callback
Password lookup can be used to generate random string
with desired length. This is useful in various scenarios.
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* Ensure -k is set to delegated hosts without a pass
* Fix up some broken tests
* Update task_executor.py
one possible fix, the other is updating winrm to normalize on 'password' like the other connection plugins
* Add alias for winrm and fix incorrect assumption
* Make sure aliases are used for keyword options
* Conditionally run test if sshpass is present, fix sanity
Co-authored-by: Brian Coca <bcoca@users.noreply.github.com>
Change:
- Use `chmod +a` in the fallback chain to allow MacOS to use ACLs to
allow an unprivileged user to become an unprivileged user.
Test Plan:
- CI, new tests
Tickets:
- Fixes#70648
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- Previously, `cache_timeout=0` was seen as falsey and not passed along
to cache plugins as an option. Now only "nulley" values are not sent.
Test Plan:
- New integration tests
Tickets:
- Fixes#70702
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- Previously CachePluginAdjudicator#flush only removed entries from the
cache backend that it knew about by using them earlier. Now it calls
the underlying plugin's flush() method.
Test Plan:
- New unit tests
Tickets:
- Fixes#68770
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- Refactoring to make it harder to get wrong and easier to read.
- Generalize become_unprivileged tests and fix some that never worked
but also never failed.
Test Plan:
- CI, new units/integration tests
Signed-off-by: Rick Elrod <rick@elrod.me>
* Fix missing quoting for remote_tmp in second mkdir of shell module. Issue #69577
* adding changelog
* fixing typo in changelog entry
* adding test case
Adding test case written by bmillemayhias.
* using $HOME instead of ~
* fixing commit measage
* Update 69578-shell-remote_tmp-quoting.yaml
Co-authored-by: Brian Kohles <me@briankohles.com>
pipe lookup plugin uses Popen with shell=True intentionally.
This is considered a security issue if user input is not validated.
Updated docs to reflect this information for the user. Also, added
Bandit B602 documentation link for further reading.
Fixes: #70159
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
Change:
- Add integration tests for various cases
- Fix wrong use of "its" in an exception thrown in varnames when it
throws an AnsibleError, given a term of the wrong type.
Test Plan:
- new tests, CI
Tickets:
- Fixes#70546
Signed-off-by: Rick Elrod <rick@elrod.me>
- ensure we preserve the typeerror part of the exception so loop defereed error handling
can postpone those caused by undefined variables until the when check is done.
- fix tests to comply with the 'new normal'
- human_to_bytes and others can issue TypeError not only on 'non string'
but also bad string that is not convertable.
Co-authored-by: Sloane Hertel <shertel@redhat.com>
Co-authored-by: Sloane Hertel <shertel@redhat.com>
Change:
- Use parse_kv() for parsing in the csvfile lookup plugin. This allows
us to handle multi-word search keys and filenames. Previously, the
plugin split on space and so none of these things worked as expected.
- Add integration tests for csvfile, testing a plethora of weird cases.
Test Plan:
- New integration tests, CI
Tickets:
- Fixes#70545
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- Followup to #70504. We need to not pass the 'warn' parameter from the
action plugin either, unless it's True. Otherwise, even though it
defaults to false, we always show the deprecation.
Test Plan:
- Local
Signed-off-by: Rick Elrod <rick@elrod.me>
When user uses home directory in --playbook-dir option
of ansible-inventory command, it warns user about this.
This PR suppress the warning message for user's home directory usage
in ansible-inventory command.
Fixes: #65262
Signed-off-by: Satyajit Bulage <sbulage@redhat.com>
The ssh_args were sometimes not correctly applied to the connection
when using _play_context. Use get_option() instead to ensure the
correct ssh_args are always applied.
* [Bugfix] curses.setupterm() error
When run playbook in celery task, curses.setupterm() will be failed
```
File "<frozen importlib._bootstrap>", line 675, in _load
File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed
File "/Users/guang/.virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/action/pause.py", line 45, in <module>
curses.setupterm()
TypeError: argument must be an int, or have a fileno() method.
```
* Add changelog
Co-authored-by: Sam Doran <sdoran@redhat.com>
Change:
- Remove _get_item() alias as it has been deprecated
- Update tests
- Remove relevant sanity curtailment
- Add changelog
Test Plan:
CI, grep
Signed-off-by: Rick Elrod <rick@elrod.me>
* Only allow groups which were hardcoded in module_defaults.yml
only load action groups from the collection if module_defaults contains a potential group for the action
* Fix tests using modules that override those whitelisted in lib/ansible/config/module_defaults.yml
Third party modules should not be using group/ - use the action name instead
* add externalized module_defaults tests
add the missing group and collections
ci_complete
Co-authored-by: Matt Davis <mrd@redhat.com>
* changelog
ci_complete
* Fix import in tests
ci_complete
* Update with requested changes
ci_complete
* don't traceback since we don't validate the contents of module_defaults
ci_complete
Co-authored-by: Matt Davis <mrd@redhat.com>
* Allow to specify collection_name separately for deprecation.
* Use new functionality in Ansible.
* Use new functionality in tests.
* Update tagging/untagging functions.
* Update pylint deprecated sanity test.
* Update validate-modules. Missing are basic checks for version_added (validate semantic version format for collections).
* Improve version validation. Re-add version_added validation.
* Make sure collection names are added to return docs before schema validation.
* Extra checks to avoid crashes on bad data.
* Make C# module utils code work, and update/extend tests.
* Add changelog fragment.
* Stop extracting collection name from potentially tagged versions/dates.
* Simplify C# code.
* Update Windows modules docs.
* Forgot semicolons.
* Make AnsibleVaultEncryptedUnicode work more like a string. Fixes#24425
* Remove debugging
* Wrap some things
* Reduce diff
* data should always result in text
* add tests
* Don't just copy and paste, kids
* Add eq and ne back
* Go full UserString copy/paste
* Various version related fixes
* Remove trailing newline
* py2v3
* Add a test that can evaluate whether a variable is vault encrypted
* map was introduces in jinja2 2.7
* moar jinja
* type fix
Co-Authored-By: Sam Doran <sdoran@redhat.com>
* Remove duplicate __hash__
* Fix typo
* Add changelog fragment
* ci_complete
Co-authored-by: Sam Doran <sdoran@redhat.com>
* various deprecation, display, warning, error fixes
* Update lib/ansible/utils/display.py
Co-authored-by: Felix Fontein <felix@fontein.de>
* Update lib/ansible/utils/display.py
Co-authored-by: Felix Fontein <felix@fontein.de>
* Update lib/ansible/utils/display.py
Co-authored-by: Felix Fontein <felix@fontein.de>
* cleanup, test fixes
* add collection name to deprecated() calls
* clean up redirect entries from uncommitted tests
* fix dep warning/error header text to match previous
Co-authored-by: Felix Fontein <felix@fontein.de>
Change:
- The `add_host` action now shows an accurate change status.
Test Plan:
- Added a plethora of integration tests.
Tickets:
Fixes#69881
Signed-off-by: Rick Elrod <rick@elrod.me>
* starting metadata sunset
- purged metadata from any requirements
- fix indent in generic handler for yaml content (whey metadata display was off)
- make more resilient against bad formed docs
- removed all metadata from docs template
- remove metadata from schemas
- removed mdata tests and from unrelated tests
Co-authored-by: Felix Fontein <felix@fontein.de>
Co-authored-by: Rick Elrod <rick@elrod.me>
* Use fqcr from command module invocation using shell module
Fixes https://github.com/ansible/ansible/issues/69788
Use fully qualified collection reference while invoking
command module from shell module
* Fox review comment
ModuleNotFoundError is a subclass of ImportError but only exists in
Python 3.6 or newer. Instead of doing hacks to be able to catch that on
older Pythons, just always only catch ImportError, which will also catch
ModuleNotFoundError on Python 3.6 or later
Change:
Rather than only using config, have base connection plugins fall back to
play_context.
Test Plan:
- Tested ansible-connection logic against an IOS device
- Tested -k against a VM
- CI
Signed-off-by: Rick Elrod <rick@elrod.me>
* Track collection for version_added.
Validate *all* version numbers in validate-modules.
For tagged version numbers (i.e. version_added), consider source collection to chose validation.
* Make tagging/untagging functions more flexible.
* Tag all versions in doc fragments.
* Tag all deprecation versions issued by code.
* Make Display.deprecated() understand tagged versions.
* Extend validation to enforce tagged version numbers.
* Tag versions in tests.
* Lint and fix test.
* Mention collection name in collection loader's deprecation/removal messages.
* Fix error IDs.
* Handle tagged dates in Display.deprecated().
* Also require that removed_at_date and deprecated_aliases.date are tagged.
* Also automatically tag/untag removed_at_date; fix sanity module removal version check.
* Improve error message when invalid version number is used (like '2.14' in collections).
Change:
Rather than connection plugins directly accessing play_context and pulling
the password from there, have them pull it from the config system, and
have TaskExecutor store it there for now.
Internally, it still routes through play_context for now, but this is
the first step away from that.
Test Plan:
- Local test with `ansible -c ssh`
- grep -R play_context.pass lib/ansible/plugins/connection/
- CI
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
Allows the user to configure sshpass (1.06+) to look for a different
substring than the default "assword" that it comes with.
Test Plan:
Set a custom ssh password prompt on a VM with PAM and tried connecting to
it. Without `ansible_sshpass_prompt` set in inventory: experienced hang.
With `ansible_sshpass_prompt` in inventory: connected successfully.
Tried setting `ansible_sshpass_prompt` with an older `sshpass` in PATH
and got a loud error, as expected.
Tickets:
Fixes#34722, fixes#54743, refs #11565.
Signed-off-by: Rick Elrod <rick@elrod.me>
* `meta/` directory in collections
* runtime metadata for redirection/deprecation/removal of plugin loads
* a compatibility layer to keep existing content working on ansible-base + collections
* a Python import redirection layer to keep collections-hosted (and otherwise moved) content importable by things that don't know better
* supported Ansible version validation on collection loads
* fix delegated interpeter
* allow returning fact if it is 'the right host'
* added note for future fix/efficiency
as it stands we rerun discovery for the delegated host
unless its saving facts to itself
* fixed test lacking delegate_to mock
When mixed with the free strategy (or any custom strategy that does not behave in
a lock-step manner), the linear methodology of _wait_on_handler_results may cause
race conditions with regular task result processing if the strategy uses
_process_pending_results directly. This patch addresses that by splitting the queues
used for results and adding a flag to _process_pending_results to determine which
queue to check.
Fixes#69457
* Add multipart/form-data functionality
* Fix some linting issues
* Fix error message
* Allow filename to be provided with content
* Add integration test
* Update examples
* General improvements to multipart handling
* Use prepare_multipart for galaxy collection publish
* Properly account for py2 vs py3, ensuring no max header length
* Address test assumptions
* Add unit tests
* Add changelog
* Ensure to use CRLF instead of NL
* Ignore line-endings in fixture
* Consolidate code, add comment
* Bump fallaxy container version
* ci_complete
This doesn't seem to be used by config although it came in with the
config PR. It's not displayed in docs. Removing it as this is the only
place where it's present.
* Fix filedescriptor out of range in select() when running commands
* Simplify the run_command() code
Now that we're using selectors in run_command(), we can simplify some of
the code.
* Use fileobj.read() instead of os.read()
* No longer use get_buffer_size() as we can just slurp all of the data
instead.
Also use a simpler conditional check of whether the selector map is
empty
Co-authored-by: Toshio Kuratomi <a.badger@gmail.com>
Since Ansible 2.9.8, if the fileglob plugin is passed a path containing
a subdirectory of a non-existent directory, it will fail. For example:
lookup('fileglob', '/'): ok
lookup('fileglob', '/foo'): (non-existent): ok
lookup('fileglob', '/foo/bar'): (non-existent): FAIL
The exact error depends on Python 2 or 3, but here is the error on
Python 2:
AttributeError: 'NoneType' object has no attribute 'endswith'
And on Python 3:
TypeError: expected str, bytes or os.PathLike object, not NoneType
This change fixes the issue by skipping paths that are falsey before
passing them to os.path.join().
Fixes: #69450
* Do not pass file mode during recursive copy on symlink files.
The 'file' module cannot deal with mode=preserve. Do not pass that
mode to the module when 'preserve' is used.
* Fix changelog fragment filename
Change:
Rather than hardcoding .pyo and .pyc, filter on all BLACKLIST_EXTS in
the non-legacy logic of PluginLoader (_find_fq_plugin). The two harcoded
extensions are part of BLACKLIST_EXTS already and this simply adds the
rest of the blacklisted extensions to the check.
In addition, check .endswith() instead of an exact match of the suffix,
like everywhere else that uses BLACKLIST_EXTS. This allows for
blacklisting, for example, emacs's backup files which can appear after
any extension, leading to things like `foo.py~`.
Test Plan:
Ran `ansible-playbook` against a collection where a `foo.py~` module was
getting executed instead of `foo.py` which also appeared in the same
directory. `foo.py~` is no longer executed.
Tickets:
Fixes#22268
Refs #27235
Signed-off-by: Rick Elrod <rick@elrod.me>
* update ActionBase._low_level_execute_command to honor executable
* adding changelog fragment
* renaming changelog fragment to .yml
* noop change to bump shippable
* adding raw_executable integration test
* copying aliases from raw
* removing blank lines
* skipping aix and freebsd
* noop to bump shippable
* moving tests to raw/
* removing become_method: sudo ; it doesn't work on AIX
* removing trailing blank line
* forcing become_method: su to try to get AIX to work
Co-authored-by: Rob Wagner <rob.wagner@sas.com>
The chapter that described more information about become to an unprivileged user has moved. This commit points to an existing chapter
Relates to #69021
* also consolidated temp dir name generation, added pid for more 'uniqness'
* generalize error message
* added notes about remote expansion
CVE-2020-1733
fixes#67791
* fixed fetch traversal from slurp
* ignore slurp result for dest
* fixed naming when source is relative
* fixed bug in local connection plugin
* added tests with fake slurp
* moved existing role tests into runme.sh
* normalized on action excepts
* moved dest transform down to when needed
* added is_subpath check
* fixed bug in local connection
fixes#67793
CVE-2019-3828
* Allow tasks to notify a fqcn handler name
* Add tests. Fixes#68181
* Add changelog fragment
* Add test to ensure handlers are deduped properly with fqcn, role, and just handler names
* Add some docs about new special vars
* Always set the discovered interpreter on the delegated host. Fixes#63180
* Make code a little more generic
* Move code into a function
* Implement some changes based on reviews
* Add changelog fragment
* Allow custom inventory plugins and cache plugins
If _load_name is not set correctly the cache plugin can't load the documentation (which is also the arg spec)
Fix the existing inventory plugin in the collections tests
Add integration tests for using a cache plugin in a collection
* Set the attribute on the instance instead of the class
Deprecate importing custom CacheModules directly - they should use the cache_loader
* Internally redirect win modules to collection name
* Added comment for how this should be fixed in a subsystem plugin
* add collection plugins for base tests
* Address compat issue for collection loading on py26
* Move import_module shim to utils for compat across the codebase
* Enable collection tests on py2.6
* Update changelog fragment
* Simplify code using sys.moduls
* Move compat to module_utils/compat/importlib
* Add back errantly deleted newline
* Remove hack comment
Co-Authored-By: Matt Clay <matt@mystile.com>
Co-authored-by: Matt Clay <matt@mystile.com>
* Run modules on windows container
This provides an ability to run Powershell modules on windows container via docker connection. Otherwise, Ansible tries to run python modules on windows containers and fails.
* Removing whitespace in the blank lines
* Adding a changelog fragment
* Allow botocore configuration to be configurable for boto3 modules
* Allow modification of the boto user agent
* play nicely with modules that might be modifying config
* changelog
* Added additional error handling inside the hcloud.py inventory plugin. Otherwise, errors may happen, if the HCloud API is not able to correctly identify the server image.
* Corrected the previous commit to not check for None on variables that are always set. Additionally, used image.description if image.name was not set, instead of the previously added default string.
The `/api/v2/reports` endpoint has been deprecated a long time ago and should
no longer be used. This changes to callback to use the correct API
endpoint and paramerter.
* Hotfix for FortiAnalyzer HTTPAPI Plugin in response to connection issues introduced in Ansible 2.9.
* Changed _connect behavior per Mr. Case.
Co-authored-by: Luke Weighall <lweighall@users.noreply.github.com>
Co-authored-by: Nathaniel Case <this.is@nathanielca.se>
* Hotfix for connection manager persistent connection initialization.
* PEP8 Fixes
* First Round of fixes requested by Qualthos.
* Removed _connection check per Mr. Case.
* Somehow the wrong version got sent... should be all good.
Co-authored-by: Luke Weighall <lweighall@users.noreply.github.com>
Co-authored-by: Nathaniel Case <this.is@nathanielca.se>
* Make validate-modules stop ignore FILE_COMMON_ARGUMENTS.
* Add types to FILE_COMMON_ARGUMENTS and update document fragment to match it.
* Update ignore.txt.
* Add changelog.
* Clean up FILE_COMMON_ARGUMENTS.
* postgresql_pg_hba doesn't declare the backup option.
* uri doesn't declare the remote_src option.
* Add documentation.
* maven_artifact seems to use directory_mode, which it doesn't declare.
* Update changelogs/fragments/66389-file-common-arguments.yml
Update docs/docsite/rst/porting_guides/porting_guide_2.10.rst
ci_complete
Co-Authored-By: Jill R <4121322+jillr@users.noreply.github.com>
self._discovered_interpreter_key is None unless a previous iteration
has attempted discovery. In that case, force re-discovery, as the
previous attempt certainly failed.
* Fix UNC path joining in the powershell shell plugin, add test
* Remove testy bits and a redundant line
* Fix style nits
* Update to use os.ntpath
* Add changelog for #66604
