This is in reference to the issue: https://github.com/ansible/ansible/issues/41130 (Cannot decrypt with copy module when remote_src: yes)
Autodecryption of the vaulted files does not work when we want to copy files with remote_src=yes.
Co-authored-by: ziaurrehman72 <ziaurrheman72@yahoo.com>
* Move _syslog_facitily to __init__
No good reason it should not be set for each object
* Move internal property setting to private method
* Create check_arguments() function
* Remove unused import
* Rename function to better match its behavior
Change the behavior to return a set, either empty or populated, with unsupported keys.
Accept legal_inputs as optional which will not required calling handle_aliases before calling
get_unsupported_parameters().
* Add changelog
* Rework function behavior and documentation
I realized I missed the original intent of this method when moving it to a function. It
is meant to compared the parameter keys to legal inputs always, not compare
parameter keys to argument spec keys, even though the argument spec keys should
be a subset of legal inputs.
* Add tests
* Fix typo.
* Set internal properties when handling suboptions
* Support listing roles in text and JSON
* Change tests for unfrack'd playbook_dir var
These tests were using '/tmp' for testing the setting of the playbook_dir
var. Now that we unfrack that var, MacOS will change this to '/private/tmp'
causing the tests to fail. We can choose a path that does not exist (since
unfrack does not validate existence) so that we can guarantee unfracking
will not change the value.
Change:
- Initial set of changes for renaming to ansible-core
- Includes changelog fragment changes from base -> core
- Does NOT include docs changes
- Modifies detection stuff in setup.py to support ansible<2.9 and ansible-base
Test Plan:
- ci_complete
* Fix type of mode field in copy module
* Remove module type checking from ignore.txt sanity checks
* Remove mode type
Co-authored-by: Matt Martz <matt@sivel.net>
* Fix super annoying Python 2.6 multiprocessing.Queue stack trace in CI
A bug exists in Python 2.6 that sometimes raises an exception during interpreter shutdown. We
encounter this frequently in our CI since we run tests on CentOS 6 as the control node, which
has Python 2.6.6 with this bug.
This PR adds a very minor sleep only on Python 2.6 which gets around this issue. I did lot of testing
using a standalon script I found that easily duplicated the issue to find the minimum sleep value
needed to avoid this issue.
CPython issue: https://bugs.python.org/issue4106
Fix in CPython: https://hg.python.org/cpython/rev/d316315a8781
* Use correct attribute
* Consolidate logic for determining whether or not session is interactive
into a single function, is_interactive()
* Increase test coverage
I wasn't able to find a good way of simulating running a backgrounded test with CI since the
whole test is essentially run not in a TTY, which is similar enough to cause the new is_interactive()
function to always return false.
* add return documentation for unarchive, add fragment for changelog, apply suggestions from code review
Co-authored-by: Amin Vakil <info@aminvakil.com>
* Clarify collection paths in docs, fix rst syntax, rephrase docs about using collection from folder/local git
Co-authored-by: Felix Fontein <felix@fontein.de>
* Added caching mechanism for Galaxy API requests
* Add cache options and split up code
* Added unit tests
* Fix sanity test
* Use modified date and fix up caching for explicit servers
* Make sure credentials are not in cached server name
* Added test for getting updated cache version
* Changes from review
* Reorder comment postition
* Add comment unit test
* Fix unit test
* Fix unit test
* Add changelog
* Add paramaters which would be problematic without this fix
* Fix typo
* Fix unit test
* Fix unit test
* Emit warning when running on the controller with a Python older than 3.8
* Add spaces
Co-authored-by: Matt Clay <matt@mystile.com>
* and more spaces
Co-authored-by: Matt Clay <matt@mystile.com>
* s/Py/Python/
Co-authored-by: Matt Clay <matt@mystile.com>
* Add note to Control node requirements about Py3.8 requirement
* Add collection_name to deprecated call
* more spaces
Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
* Clarify that we are only packaging for py3.8+
Co-authored-by: Matt Clay <matt@mystile.com>
Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
* Fix 'role_name : tast_name' notation if task contains role name
* Add tests for notifying handler names which contain the role name
Co-authored-by: Thomas Wouters <thomaswouters@gmail.com>
* Support docker and k8s action groups for moved modules in community.docker and community.kubevirt.
* Also support k8s action group for community.okd.
* Also add kubernetes.core.
* Adjust PR #.
* Fix changelog fragment.
* Remove community.okd.
* Revert "Remove community.okd."
This reverts commit 812b5aa6e2.
Change:
- Docs: Add note that security/bugfix apply to dependencies too, like
the dnf command.
- dnf: security/bugfix only makes sense for updates, so limit the
package query sack to available updates.
- tests: Limit tests to our known-good test packages, so that RHEL
packages marked security/bugfix without similarly marked dependencies
don't fail our tests.
Test Plan:
- Tested with `dnf upgrade-minimal --bugfix` and reproduced the same
error currently seen in CI, showing that we are consistent with what
dnf does.
Tickets:
- Likely fixes#72316
Signed-off-by: Rick Elrod <rick@elrod.me>
* Indent Jinja2 directives.
* Show version_added and deprecations for ini settings and environment variables.
* Make default conditional, add choices. Copied from #55474.
* Add Ansible variables. Copied from #55474.
* Turn ini keys and environment variables into lists. Copied from #55474.
* Improve formatting. Copied from #55474.
* hint in doc of git module to abort on missing http password
improves #69489
* fix syntax in doc of git module
Co-authored-by: Matt Martz <matt@sivel.net>
Co-authored-by: Matt Martz <matt@sivel.net>
* ansible-doc: export has_action when --json is used.
* Remove docuri and now_data, which were not used resp. ignored in format_plugin_doc and the functions it calls anyway.
* Add function _combine_plugin_doc.
* added timeout options to adhoc and console
* added test
* fix typosesz
* fix conflict
* task_timeout
* fix timeout option, added extra vars to console
* actually use right cli switch .. DUH!
* added timeout to include 'valid' but ignored keys
* fix default
* fixes per review
* fixes for collection playbooks
- add fqcn invocation, also handles extensions
- bring import_playbook into new normal
- avoid adding collection playbooks to adjacency
- deal with extensions or lack of em
- added tests
- fix bugs with multiple playbooks supplied
- unicode/bytes/native fixes
- fix modulenotfound for < py3.7
- tests
* Add DarwinStrategy class and integration tests
macOS has three seprate hostname params that need to be set. One of those params, LocalHostName, has more stringent requirements than the other two, which accept special characters and spaces. Create a method to scrub the hostname to ensure it works well with the system requirements.
* Update documentation
* Account for virtualization type returned on Azure Pipelines
* Do not be dependent on order of self.name_types
Use the scrubbed name when the name type is LocalHostName
A bug existed in systemd 245 that did not properly handle unknown kernel
capabilities gracefully. This resulted in incomplete output when querying
for the service status. It is possible to get service status by other means.
This PR works around this issue by getting service status using other commands
in the event of a failure due to this bug.
* GitHub is removing the underlying API used to implement the `login` command. Since the general consensus seems to be that relatively nobody currently uses this command (in favor of explicit token passing), support was simply removed for interactive login. If a future need arises, this command should be reimplemented via OAuth Device Auth Grants.
* login or role login commands now produce a fatal error with a descriptive message
* updated 2.10 and 2.11 porting guide entries
* remove dead code/config, update messages and porting guides
* Fix the reversed order of return values
AnsibleModule.run_command returns a tuple of return code, stdout and stderr.
The module main function of the user module expects user.create_user to
return a tuple of return code, stdout and stderr.
Fix the locations where stdout and stderr got reversed.
* Fix another missed occurance
* Add changelog
* Fix further reversed order of return values occurrences
Followup to 72c3ba62c8:
Fix further reversed order of return values occurrences.
Change:
- containerd is now recognized as container tech
Test Plan:
- Tested with Docker inside Github Actions in a private repo:
"ansible_virtualization_role": "guest",
"ansible_virtualization_tech_guest": [
"containerd",
"container",
"VirtualPC"
],
"ansible_virtualization_tech_host": [],
"ansible_virtualization_type": "containerd",
Tickets:
- Fixes#66304 because this is what docker containers show up as
in Github Actions.
Signed-off-by: Rick Elrod <rick@elrod.me>
* Ensure we call action_loader.get with collection_list. Fixes#72170
* Add tests and changelog
* Remove grep, do assertion in playbook. ci_complete
* Skip old jinja2 versions
* ci_complete
* dedupe
make collections whitelist follow normal flow
* fixes missing set_options call and adhoc and stdout processing rules
* avoid dupes
* fixed to handle redirects
* also updated tests with new and more accurate skip message
* fix callback tests for envs with cowsay installed
* lots MOAR comments on why the code is as it is, some todos to refactor in future
Co-authored-by: Matt Davis <nitzmahone@users.noreply.github.com>
Change:
- When a plugin defines `type: str` on a parameter, treat more kinds of
input as a string instead of whatever it is parsed as.
Test Plan:
- New unit tests
- CI
Signed-off-by: Rick Elrod <rick@elrod.me>
* Add support for GSSAPI/Kerberos to urls.py
* Test out changes with the latest test container
* Get remote hosts working
* Fix up httptester_krb5_password reader
* Fix tests for opensuse and macOS
* Hopefully last lot of testing changes
* Dont do CBT on macOS
* Fixes from review
* Fix various sanity errors in plugins.
* Revert callback_type -> type transform.
* Undo paramiko_ssh 'connection' change, since this discrepancy is hardcoded in the plugin loader.
* Standardize on name and type (for callbacks).
* Fix existing author entries.
* Add 'Unknown (!UNKNOWN)' as author when author is missing.
* These are actually parsed as integers in the code.
* Revert "Add 'Unknown (!UNKNOWN)' as author when author is missing."
This reverts commit 29d1438aca.
Change:
- No longer fail due to old Fedora/RHEL and our failure to try to cast
gids to integers before trying to pass them to getgrgid() before
trying to use them.
- Add tests for user/mode for various unarchive formats.
Test Plan:
- New integration tests, ran against centos6 container
Tickets:
- Fixes#71903
Change:
- Make KubeVirt be identified as such, and with a guest role.
- This is a breaking change, since playbooks could be relying on the
incorrect behavior previously.
Test Plan:
- Issue author
Tickets:
- Fixes#72001
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- Use `sysctl -n` for openbsd uptime information
- Allow `get_sysctl()` to account for multi-line sysctl settings
- Add unit tests for `get_sysctl()`
Test Plan:
- New unit tests
Tickets:
- Fixes#71968
- Refs #72025
- Refs #72067
Signed-off-by: Rick Elrod <rick@elrod.me>
Co-authored-by: Brian Coca <brian.coca+git@gmail.com>
Co-authored-by: Brian Coca <bcoca@ansible.com>
The luseradd / lusermod commands do not support the -e option. Set
the expiry time in this case via lchage after the user was
created / modified.
Fixes: #71942
In Python3 math.floor returns an integer whereas Python2 returns a float.
Hence always convert the result of math.floor to an int to ensure that
lexpires is an integer.
Move local expires tests in a separate file and import the tasks to the
main.yml to keep main.yml smaller.
Module package is actually a proxy to a native package manager
modules (internally it delegates to the module that matches the
results of ansible_pkg_mgr).
Fixes#69371
b6b238a fixed the SLES4SAP detection, which was at this time ok.
Sadly Suse changed with SLES 15 the /etc/os-release file, so the above
change will no longer work.
This commit updates the SLES4SAP detection regarding
https://www.suse.com/support/kb/doc/?id=000019341.
The symlink realpath is matched with endswith, because in SLES 12+ the
link target is SLES_SAP.prod, but in SLES 11 the link target is
SUSE_SLES_SAP.prod.
This change preserves how all the external imports refer to this code
while allowing us to start cutting the spaghetti into more easily
maintainable pieces.
This is a start of the upcoming refactoring effort destined to
eliminate tight coupling, implicit data manipulation, god objects,
abstraction leaks and other code smells.
Essentially, `ansible.galaxy.collection` is going to be a package that
holds parts of the collection management code spread across loosely
coupled modules.
PR: #71931
* Let get_file_attributes() work without `lsattr -v`
Change:
- module_utils's get_file_attributes() expects `lsattr -v` to work, but
in some cases, it may not.
- The function now takes an optional include_version bool parameter,
which removes this expectation.
- Places where we call get_file_attributes() without using the 'version'
it returns, we now call it with include_version=False.
Test Plan:
- New unit tests
Signed-off-by: Rick Elrod <rick@elrod.me>
* Reorder test/sanity/ignore.txt to better see changes
* Remove extra empty line causing pass fail
* Apply suggestions made by Akasurde
* Minor fix in package_facts and remove two unnecessary ignores
* Fix subversion based on a suggestion made by felixfontein
* Apply suggestions made by felixfontein and Andersson007
* Fix subversion.py as suggested by felixfontein
* Minor reformatting in yum_repository description
* Reformat changelog
* Add key to apt_key, add deprecated changelog
* Add PR url to changelog
* Ignore paramater-type-not-in-doc in favour of adding key back to apt_key
* Fix apt_key
* Remove undocumented-paramater from apt_key ignore
* Ignore doc-choices-do-not-match-spec in package_facts
* Fix package_facts
* Fix filter option in setup module
Change:
- In some cases (always with free strategy, sometimes with linear), the
default callback would not show the task banner for include_tasks.
- This only affects the include_tasks task itself, not the tasks in the
included file.
Test Plan:
- Updated default callback tests
Tickets:
- Fixes#71277
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- Previously when `security: true` and `bugfix: true` were both given,
only security updates would get applied. Filters now accumulate so
that both get applied in this case.
Test Plan:
- New integration tests for both check_mode and not. These tests make
use of a contrived yum repository which is stored in S3.
Tickets:
- Fixes#70854
Signed-off-by: Rick Elrod <rick@elrod.me>
Co-authored-by: Matt Martz <matt@sivel.net>
Co-authored-by: Matt Martz <matt@sivel.net>
* fix coverage output from synthetic packages
* synthetic packages (eg, implicit collection packages without `__init__.py`) were always created at runtime with empty string source, which was compiled to a code object and exec'd during the package load. When run with code coverage, it created a bogus coverage entry (since the `__synthetic__`-suffixed `__file__` entry didn't exist on disk).
* modified collection loader `get_code` to preserve the distinction between `None` (eg synthetic package) and empty string (eg empty `__init__.py`) values from `get_source`, and to return `None` when the source is `None`. This allows the package loader to skip `exec`ing things that truly have no source file on disk, thus not creating bogus coverage entries, while preserving behavior and coverage reporting for empty package inits that actually exist.
* add unit test
The setup module can now filter out multiple pattern by providing a list
to the filter parameter instead of just a string. Single string sill
works. Previous behaviour remains.
(cherry picked from commit b5c36dac483fdd74d6c570d77cc8f3e396720366)
* Use InventoryManager for inventory_hostnames lookup. Fixes#17268
* Add test for using patterns
* s/it's/its/
* Allow bypassing parse_sources in InventoryManager
* Remove unneeded method call
Change:
- By default the dnf API does not gpg-verify packages. This is a feature
that is executed in its CLI code. It never made it into Ansible's
usage of the API, so packages were previously not verified.
- This fixes CVE-2020-14365.
Test Plan:
- New integration tests
Signed-off-by: Rick Elrod <rick@elrod.me>
Remove "mode: preserve" option from documentation in doc fragments "FILE_COMMON_ARGUMENTS", as it was incorrectly included in the documentation for the `lineinfile`, `unarchive` and other file-related modules.
The `copy` and `templates` modules documentation remains untouched and still contain "mode: preserve", as intended.
* Validate salt when using crypt. Respect salt_size in password lookup. Repair salt for bcrypt. Fixes#71107. Fixes#53750. Fixes#36129.
* Handle algorithms we don't know about, and make sure to return the salt
* Account for old passlib
* Add tests for salt constraints
* Add changelog fragment
* Add test for #36129
Some parameters for systemd are ignored, add a note about such parameters in documentation.
Fixes: #23144
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* added configurable dir for tree callback
- allows usage in playbooks
- also made errors more specific if bad dir is supplied
- ensure we expand paths
* docfix
This script is mostly unmaintained and relies on the deprecated and
unmaintained `boto` library. Featuring it prominently in the docs
leads to many new users using it instead of the supported `aws_ec2`
inventory plugin.
Change:
- Now sends meta tasks to the task start callback
- Lets callback plugins opt-in to receiving implicit tasks
Test Plan:
- New integration tests
Tickets:
- Indirectly fixes#71007 by allowing custom callbacks with this data
Signed-off-by: Rick Elrod <rick@elrod.me>
skip_missing parameter in subelements lookup plugin is accepted from
inside the dictionary.
Fixes: #38182
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* check run state of current block only
* Add changelog and test
* Add test for issue 29047
* Fix for both tests
* blerg
* Change test messages
* fix tests
* Add multi-level block in rescue test case
* Add recursive rescue check and multi-level test
* Should probably run the new test
* ci_complete
* Merge new tests
* ci_complete
This would be a partial solution for #69364 in that the SHASUMS file can be downloaded and gpg verified but then used from the downloaded location to verify the get_url's file.
* Make checksum url parsing more explicit
Use urlsplit to test if the checksum string has a (currently tested and) supported url scheme.
* Fix whitespace
* Changelog fragment
* Added tests
* Fix typo in test setup
The message generated by systemctl has been updated in 9321e23c40, which requires a corresponding change in the systemd module.
In addition, this fixes the module when the SYSTEMD_OFFLINE environment variable is set.
* POC for supporting callback events that come from the worker
* linting fixes. ci_complete
* fix up units. ci_complete
* Try moving the sentinel put higher. ci_complete
* safeguards. ci_complete
* Move queue killing to terminate
* LINTING. ci_complete
* Subclass Queue, to add helper send_callback method
* Just use _final_q instead of adding another queue and thread
* Revert a few changes
* Add helper for inserting a TaskResult into the _final_q
* Add changelog fragment
* Address rebase issue
* ci_complete
* Add test to assert async poll callback from fork
* Don't use full path
* ci_complete
* Use _results_lock as a context manager
* Add new generic lock decorator, and use it with send_callback
* Revert "Change default file permissions so they are not world readable (#70221)"
This reverts commit 5260527c4a.
* Revert "Fix warning for new default permissions when mode is not specified (#70976)"
This reverts commit dc79528cc6.
Password lookup can be used to generate random string
with desired length. This is useful in various scenarios.
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* Ensure -k is set to delegated hosts without a pass
* Fix up some broken tests
* Update task_executor.py
one possible fix, the other is updating winrm to normalize on 'password' like the other connection plugins
* Add alias for winrm and fix incorrect assumption
* Make sure aliases are used for keyword options
* Conditionally run test if sshpass is present, fix sanity
Co-authored-by: Brian Coca <bcoca@users.noreply.github.com>
Change:
- Allow systems to declare multiple virt techs. For example if a system
is both a docker container, but virtualized on KVM, show both. If a
system is set up to run virtualbox and KVM VMs, show both.
- This is done by introducing new facts keys:
- virtualization_tech_guest
- virtualization_tech_host
- Backwards compatibility is preserved by keeping track of the previous
return-points and refusing to update those keys after we would have
returned, but now returning them at the end, so that the new keys can
accumulate their data.
Test Plan:
- Local
- CI
Tickets:
- Refs #66304
- Refs #17151
- Refs #17058
- Probably others
Signed-off-by: Rick Elrod <rick@elrod.me>
* Fix tty_ify bugs and refactor
* Move tty_ify() and supporting attributes to the DocCLI class as that's
the only thing using it.
* Add unittest for the code.
* Fix a bug where the substitution macros can be detected when they are
a part of another word.
* Add support for L(), R(), and HORIZONTALLINE which were added to the
website docs many years ago.
* Update test/units/cli/test_doc.py
Co-authored-by: Matt Clay <matt@mystile.com>
Co-authored-by: Matt Clay <matt@mystile.com>
* add jinja2 global to reserved names
also allow expansion by additional context provided from caller
fixes#41955
Co-authored-by: Abhijeet Kasurde <akasurde@redhat.com>
Change:
- Use `chmod +a` in the fallback chain to allow MacOS to use ACLs to
allow an unprivileged user to become an unprivileged user.
Test Plan:
- CI, new tests
Tickets:
- Fixes#70648
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- Add a parameter `read_whole_file` which allows for reading the whole
file when doing a `contains` regex search.
- This allows for (for example) matching a pattern at the very end of
a file.
Test Plan:
- New integration tests
Tickets:
- Fixes#63378
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- Previously, `cache_timeout=0` was seen as falsey and not passed along
to cache plugins as an option. Now only "nulley" values are not sent.
Test Plan:
- New integration tests
Tickets:
- Fixes#70702
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- Previously CachePluginAdjudicator#flush only removed entries from the
cache backend that it knew about by using them earlier. Now it calls
the underlying plugin's flush() method.
Test Plan:
- New unit tests
Tickets:
- Fixes#68770
Signed-off-by: Rick Elrod <rick@elrod.me>
* Strip spaces in module names in explicit actions
Change:
- When an action is called like "action: copy foo=bar", strip spaces
around the action name.
- This allows "action: copy foo=bar" to work as expected.
Test Plan:
- New integration tests
Tickets:
- Fixes#62136
Signed-off-by: Rick Elrod <rick@elrod.me>
Add integration tests for unarchiving as unprivileged user
Break tasks into separate files for easier reading and maintenance
Create a user by specifying a default group of 'staff' for macOS.
The user module does not actually remove the user directory on macOS,
so explicitly remove it.
Put the removal tasks in an always block to ensure they always run
Co-authored-by: Philip Douglass <philip.douglass@amadeus.com>
Co-authored-by: Sam Doran <sdoran@redhat.com>
The iso8601_micro and iso8601 facts incorrectly called now.utcnow(), resulting
in a new timestamp at the time it was called, not a conversion of the previously
stored timestamp.
Correct this by capturing the UTC timestamp once then calculating the local
time using the UTC offset of the current system.
* Use time.time() for getting the current time
* Convert from that stored epoch timestamp to local and UTC times
* Used existing timestamp for epoch time
* Add unit tests that validate the formate of the return value rather than an exact value since mocking time and timezone is non-trivial
Follow up to #70221
Related to #67794
CVE-2020-1736
When set_mode_if_different() is called with mode of 'None', ensure we issue
a warning about the change in default permissions.
Add integration tests to ensure the warning works properly.
* Fix tests
- actually use custom module 🤦♂️
- verify file permission on created files
- use remote_tmp_dir so we're ready for split controller
- improve test module so we can skip the call to set_fs_attributes_if_different()
- fix tests for CentOS 6
Change:
- Clarify that not passing `--tags` will cause `ansible_run_tags` to
default to `["all"]`.
- Add some extra coverage around `ansible_run_tags`
Test Plan:
- New integration and unit tests
Tickets:
- Fixes#69619
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- Allow older git to verify tags again
- Enable verification tests everywhere, even if most of them only work
on newer git. Some of them work on older git and they test the --raw
parameter.
Test Plan:
- Re-enabled subset of git tests
Tickets:
- Fixes#64469
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- Previously, we only showed that something would have changed, not what
would have changed. This allows us to show what will chang as well.
Test Plan:
- Local RHEL8 VM
- New integration tests
Tickets:
- Fixes#66132
Signed-off-by: Rick Elrod <rick@elrod.me>
* dnf: Add nobest option
* dnf: Fix indent, add nobest specifically to dnf not yum
* Add changelog for dnf: add nobest option
* dnf: Add nobest to yumdnf module argument_spec
* dnf: remove nobest from module paramaters in yumdnf.py
* dnf: Add test for nobest option
* dnf: Cleanup packages in nobest test at last
* dnf: Cleanup manually added repos in nobest test at last
* dnf: Remove dnf-plugins-core as well in nobest test
* dnf: Change nobest release version to 2.11
* Change changelog number according to change in PR number
* Change changelog number according to change in PR number
Change:
- On Linux, there are situations where a host might be a KVM host but
not have the kernel module enabled (it might be compiled in instead).
In these cases, /dev/kvm will still exist, and rather than reporting
NA, we should report that the host is a KVM host.
Test Plan:
- Local
Signed-off-by: Rick Elrod <rick@elrod.me>
* Change default file permissions so they are not world readable
CVE-2020-1736
Set the default permissions for files we create with atomic_move() to 0o0660. Track
which files we create that did not exist and warn if the module supports 'mode'
and it was not specified and the module did not call set_mode_if_different(). This allows the user to take action and specify a mode rather than using the defaults.
A code audit is needed to find all instances of modules that call atomic_move()
but do not call set_mode_if_different(). The findings need to be documented in
a changelog since we are not warning. Warning in those instances would be frustrating
to the user since they have no way to change the module code.
- use a set for storing list of created files
- just check the argument spac and params rather than using another property
- improve the warning message to include the default permissions
* ensure hostvars are available on delegation
* also inventory_hostname must point to current host and not delegated one
* fix get_connection since it was still mixing original host vars and delegated ones
* also return connection vars for delegation and non delegation alike
* add test to ensure we have expected usage when directly assigning for non delegated host
Some platform such as ESXi does not implement EpollSelector,
which is selected by DefaultSelector. Use SelectSelector which is
based upon 'select' implementation. This works perfectly with
a platform like VMware ESXi.
Fixes: #70238
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* Encode/Decode files in UTF-8
* Use helper function in ansible
* Add an integration test
* Use emoji in test data.
* add changelog
* Also support non-ascii chars in filepath and add tests about this.
* Also use non-ascii chars in replaced text and ensure not to break cron syntax.
* rename self.existing to self.n_existing
* rename crontab.existing to crontab.n_existing
Change:
- Refactoring to make it harder to get wrong and easier to read.
- Generalize become_unprivileged tests and fix some that never worked
but also never failed.
Test Plan:
- CI, new units/integration tests
Signed-off-by: Rick Elrod <rick@elrod.me>
* Fix missing quoting for remote_tmp in second mkdir of shell module. Issue #69577
* adding changelog
* fixing typo in changelog entry
* adding test case
Adding test case written by bmillemayhias.
* using $HOME instead of ~
* fixing commit measage
* Update 69578-shell-remote_tmp-quoting.yaml
Co-authored-by: Brian Kohles <me@briankohles.com>
debconf module exposes sensitive information to logs, console.
Add a note to user about using no_log=True to hide such
information from console.
Fixes: #32386
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* Followup to #70610: fix redirects to wrong collections
* gcp module_utils is in community.general.
* dellos10_* modules are in dellemc_networking.os10, not dellemc.os10.
* intersight_facts was renamed to intersight_info in cisco.intersight in version 1.0.7.
* Fix more dellos10 redirects.
A couple of years ago Slackware -current began using a plus (“+”) at the end of the distribution version string to indicate a future version work-in-progress.
Rearrange distribution_files unit tests to easily support more tests
- add conftest with common fixtures
- use parametrize for testing multiple scenarios
* Add changelog
* Add unit tests for Slackware distribution parsing
* Use correct fixtures for Slackware
Data comes from /etc/slackware-version
Co-authored-by: Sam Doran <sdoran@redhat.com>
Co-authored-by: <Eduard Rozenberg <eduardr@pobox.com>>
pipe lookup plugin uses Popen with shell=True intentionally.
This is considered a security issue if user input is not validated.
Updated docs to reflect this information for the user. Also, added
Bandit B602 documentation link for further reading.
Fixes: #70159
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
Change:
- Add integration tests for various cases
- Fix wrong use of "its" in an exception thrown in varnames when it
throws an AnsibleError, given a term of the wrong type.
Test Plan:
- new tests, CI
Tickets:
- Fixes#70546
Signed-off-by: Rick Elrod <rick@elrod.me>
- ensure we preserve the typeerror part of the exception so loop defereed error handling
can postpone those caused by undefined variables until the when check is done.
- fix tests to comply with the 'new normal'
- human_to_bytes and others can issue TypeError not only on 'non string'
but also bad string that is not convertable.
Co-authored-by: Sloane Hertel <shertel@redhat.com>
Co-authored-by: Sloane Hertel <shertel@redhat.com>
Change:
- Use parse_kv() for parsing in the csvfile lookup plugin. This allows
us to handle multi-word search keys and filenames. Previously, the
plugin split on space and so none of these things worked as expected.
- Add integration tests for csvfile, testing a plethora of weird cases.
Test Plan:
- New integration tests, CI
Tickets:
- Fixes#70545
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- Move hostnamectl check out of GenericStrategy because it was incorrect
for everything except the SystemdStrategy which is where it belongs.
- Add some initial tests for the hostname module, though we are limited
by the fact that we can't do much testing with it in containers.
Test Plan:
- new hostname integration tests
Signed-off-by: Rick Elrod <rick@elrod.me>
Co-authored-by: Sam Doran <sdoran@redhat.com>
Change:
- Followup to #70504. We need to not pass the 'warn' parameter from the
action plugin either, unless it's True. Otherwise, even though it
defaults to false, we always show the deprecation.
Test Plan:
- Local
Signed-off-by: Rick Elrod <rick@elrod.me>
* speed up modify_module
* Remove debugging
* ci_complete
* Simplify generic_visit, alias to visit, eliminate some attr lookups
* ci_complete
* Add changelog fragment
Change:
- The command warnings feature which suggests that users use modules
instead of certain commands is now deprecated. Its `warn` paramater
and `COMMAND_WARNINGS` configuration options are also deprecated.
Their use will become an error in version 2.13.
Test Plan:
- CI
Signed-off-by: Rick Elrod <rick@elrod.me>
Co-authored-by: Sam Doran <sdoran@redhat.com>
Change:
Our handling of NetBSD virtualization facts led to facts that were just
plain incorrect. One example is reporting Xen even when the system is
running on something completely different (like KVM).
As stated by the reporter of #69352, NetBSD has a better sysctl setting
to use for this information, machdep.hypervisor.
This PR does the following:
- Try to use machdep.hypervisor sysctl value if the other sysctl values
we check don't end up with enough information to be useful
- Only look for /dev/xencons and assume Xen if nothing else works
(Really this should probably return 'unknown' since the file exists on
non-Xen systems and is not very useful).
- Add a few more patterns (Xen matches and also Hyper-V) to
VirtualSysctlDetectionMixin#detect_virt_product.
This change is slightly breaking:
- If the first two attempts at using sysctl worked before,
(machdep.dmi.system-product and machdep.dmi.system-vendor), they will
continue to work.
- For cases when those values didn't work, previously the existence of
/dev/xencons was checked, and if found, we reported 'xen' (even on
non-Xen systems when the file existed). After this PR, we try the
machdep.hypervisor sysctl key before still falling back to
/dev/xencons. This means that in some cases, we might go from
(wrongly) saying "xen" to giving a more accurate value such as "kvm"
or "Hyper-V".
Test Plan:
- Tested with local NetBSD VM and got 'kvm' instead of 'xen' back.
Tickets:
- Fixes#69352
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- module_utils.basic.is_special_selinux_path() used a string ==
bytestring comparison which returned False and made Ansible think that
certain filesystems aren't, in fact, special-cased, when they should
be. Ensure both sides of the == are bytestrings.
Test Plan:
- Added `copy` integration tests for this case.
Tickets:
- Fixes#70244
Signed-off-by: Rick Elrod <rick@elrod.me>
* ansible-doc man formatter: do not crash when description isn't there.
* Change to report a better error message when description is not there.
* Add test.
When user uses home directory in --playbook-dir option
of ansible-inventory command, it warns user about this.
This PR suppress the warning message for user's home directory usage
in ansible-inventory command.
Fixes: #65262
Signed-off-by: Satyajit Bulage <sbulage@redhat.com>
The ssh_args were sometimes not correctly applied to the connection
when using _play_context. Use get_option() instead to ensure the
correct ssh_args are always applied.
Change:
- Update bundled six to 1.13 (last with py2.6 support)
- Make it pass lint
- Fix check to allow skipping over compat __init__.py files we authored
- Fix check to allow files that can't be updated for some reason
Test Plan:
- ansible-test sanity --docker
- CI
Signed-off-by: Rick Elrod <rick@elrod.me>
* Use the first galaxy server supporting v1 for roles. Fixes#65440
* Add changelog fragment
* This is best effort, fall back to original behavior if something bad happens
* misc collection metadata fixes
* parse collection meta with libyaml if available
* require only Mapping for validation
* add explanatory text for _meta_yml_to_dict
* ignore custom pylint rule
* this code shouldn't import a bunch of stuff from ansible, since it's run under the import sanity test
Change:
- In certain situations, such as when the input string contains null
bytes (\0), syslog.syslog will throw a TypeError. Handle that and
fail_json instead.
Test Plan:
- New test
- ansible-test --docker centos[68] (for py2 and py3 respectively)
Tickets:
- Refs #70269
Signed-off-by: Rick Elrod <rick@elrod.me>
CIFS can be mounted using backward slash as well in /etc/fstab like
\\Windows\share /data/ cifs credentials=/root/.creds 0 0
Handle this condition while gather mount information in Linux.
Fixes: #48813
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* Update documentation as per sanity tests
* Added example about custom SSH port in example section
Fixes: #29236
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* Ensure not to remove existing packages while installing apt packages.
* Make all lines shorter than 160 characters
* Allow removing packages only when upgrading.
* Add integration tests
Change:
- Fix a UnicodeDecodeError in executor.module_common that could get
triggered with -vvvvv.
Test Plan:
- `ansible-test integration --docker centos7 module_utils -vvvvv`
This would show the error previously, and no loner does after this
patch.
Signed-off-by: Rick Elrod <rick@elrod.me>
In some usecases, we want to be able to clone a single branch
of a repository, without using --depth (which implies --single-branch).
* Use branch name when available
- update description of parameter
- consolidate branch or tag checking for easy reuse
* Add changelog
* Use static task imports rather than dynamic includes
* Add integration tests for single_branch
* Account for older versions of git
* Minor tweak to warnings
Co-authored-by: Laurent Coustet <laurent.coustet@clarisys.fr>
Co-authored-by: Sam Doran <sdoran@redhat.com>
* [Bugfix] curses.setupterm() error
When run playbook in celery task, curses.setupterm() will be failed
```
File "<frozen importlib._bootstrap>", line 675, in _load
File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed
File "/Users/guang/.virtualenvs/ansible/lib/python3.6/site-packages/ansible/plugins/action/pause.py", line 45, in <module>
curses.setupterm()
TypeError: argument must be an int, or have a fileno() method.
```
* Add changelog
Co-authored-by: Sam Doran <sdoran@redhat.com>
Handle colon appearing in filename while parsing the mimetype and charset
using file command.
Fixes: #70256
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
Change:
- On Python 3.9, `crypt.crypt` will throw instead of returning `None`
when the algorithm isn't supported. So we catch that and handle it
the same way we handled the algorithm not being supported on 3.8: by
throwing AnsibleError.
Test Plan:
- CI for <=3.8.
- Local for 3.9b3:
ansible -m debug -a "msg=\"{{ 'changeme' | password_hash('bcrypt') }}\"" localhost
Before:
localhost | FAILED! => {
"msg": "Unexpected failure during module execution.",
"stdout": ""
}
After:
localhost | FAILED! => {
"msg": "crypt.crypt does not support 'bcrypt' algorithm"
}
Tickets:
- Fixes#69930
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
- Remove _get_item() alias as it has been deprecated
- Update tests
- Remove relevant sanity curtailment
- Add changelog
Test Plan:
CI, grep
Signed-off-by: Rick Elrod <rick@elrod.me>
Change:
Removes with_* loop squashing and tests for 2.11
Test Plan:
CI, and grepped for with_items in package manager integration targets.
There might be some test cases in collections which need to stop testing
this behavior.
Signed-off-by: Rick Elrod <rick@elrod.me>
* Test galaxy cli against pulp
* linting fix
* Renames and small fixes
* Better handling for resetting pulp
* Clean up some things, add a comment
* I can't spell
* Bump fallaxy, use alternate pulp image
* Only reset pulp when we're are executing against pulp
* Update for updated pulp container
* Update some comments with correct URLs and typos
* Linting fix
* Pin pulp-fedora31 to a digest
* Address review comments for documentation
* try to load unqualified plugins from whitelist
* necessary for backcompat loading of unqualified collectionized callback plugins redirected from <= 2.9 core
* also added de-duping from actual loaded name
* add tests
* add warning test
* group test script entries by topic
* shorten warning text grep because wrapping is dumb
* fix adhoc callback loading behavior
* collections pass over whitelist wasn't respecting `_run_additional_callbacks`
* adds regression tests for same
* avoid `grep -L` in tests since it breaks the world
* Clarify blockinfile docs for insertafter/insertbefore
It's not clear from the docs that these options take effect
only when no marker lines are found in the document.
* Add changelog fragment
* document return values of file module
* Clarify whether path or dest input is used as dest output in file module
Co-authored-by: Alicia Cozine <879121+acozine@users.noreply.github.com>
* builtin downstream vendoring support
* allows downstream packagers to install packages to `ansible/_vendor` that will automatically be added to head of sys.path during `ansible` package load
* tests
* sort conflicting package names in warning text
* sanity fixes
* skip unnecessary comparison
If we fail to find a member when extracting a directory, try adding a trailing
slash to the member name. In certain cases, the member in the tarfile will
contain a trailing slash but the file name in FILES.json will never contain
the trailing slash.
If unable to find the member, handle the KeyError and print a nicer error.
Also check if a directory exists before creating it since it may have been
extracted from the archive.
Fixes#70009
* Add unit tests
* Use loop for trying to get members
When using 'remote_src: yes' and 'mode: preserve', the code handling
the file modes has to be handled on the remote node because it's
the one that has access to the source files. This means that the
copy module itself must handle this, rather than the copy action
plugin (which is where all that logic exists). The copy module
handles this when we copy a single file over. But when it is a
directory as the src parameter value, the mode of the files
beneath it are not considered. Subdirectories are copied with
shutil.copytree() which will preserve permissions automatically.
Individual files are copied with shutil.copyfile() which does NOT
preserve permissions. We need to add some calls to shutil.copymode()
to correct that.
Note: This *always* retains individial file permissions. Specifying
a 'mode' other than 'preserve' when giving a source directory for
the 'src' param does not make sense so will be ignored in that case
only.
Fixes#69783
* Add changelog and test
* config: singular ANSIBLE_COLLECTIONS_PATH
Every other *_PATH setting in ansible is singular, and the traditional
$PATH variable is also singular despite containing a list of
directories. Let's be consistent both internally and with POSIX
tradition.
* update all ANSIBLE_COLLECTIONS_PATHS env references to be singular
* deprecate plural ANSIBLE_COLLECTIONS_PATHS setting
* Tag return value docs if they are a dict (and not str/None).
* Try to parse return docs as YAML.
* Properly dump return values in ansible-doc.
* Adjust plugin formatter.
* Add changelog fragment.
* Don't add 'default' for return values.
* Fix plugin_formatter.
* Only try to parse return docs if they are still a string.
* Add tests.
* Warn if RETURN cannot be parsed.
* Adjust tests. Also test for warning.
* if -> elif (otherwise EXAMPLE will be parsed too).
* Always parse return documentation, and fail if it is invalid YAML.
* Polishing.
* Mostly re-enable ansible-doc tests.
Listing from the local collection seems to be somewhat broken. I assume this
is why the test was disabled.
* Lint and make tests work with Python 2.
* Keep FQCNs in plugins (not modules), i.e. restore previous state.
* Support removed_at_date in ansible-doc
Signed-off-by: NilashishC <nilashishchakraborty8@gmail.com>
Changes:
* ansible-doc does not support `removed_at_date` and assumes that
deprecated dict will either have `removed_in` or `version`. This
results in ansible-doc (and hence "sanity --test=ansible-doc")
failing for modules having only `removed_at_date`.
* This patch adds support for `removed_at_date` and also gives it
precedence over `removed_in` or `version`.
* Add tests and changelog
Signed-off-by: NilashishC <nilashishchakraborty8@gmail.com>
* ansible-galaxy - Fix role info when role is not installed
Only report the role not found if in offline mode, otherwise query the galaxy API
to get role information.
Fixes#69867
* Improve error message when role is not found in Ansible Galaxy
* Extra whitespace around imported playbook filename are stripped.
* Corrected call for display.deprecated
* Changed warning type & added test
* Added auto verification of raised warning
* More accurate warning message
This prevents PlayIterator having to go through empty blocks
that were created in filter_tagged_tasks. This should
be a performance improvement for playbooks that mostly skip
tasks with tags.
ci_complete
* Only allow groups which were hardcoded in module_defaults.yml
only load action groups from the collection if module_defaults contains a potential group for the action
* Fix tests using modules that override those whitelisted in lib/ansible/config/module_defaults.yml
Third party modules should not be using group/ - use the action name instead
* add externalized module_defaults tests
add the missing group and collections
ci_complete
Co-authored-by: Matt Davis <mrd@redhat.com>
* changelog
ci_complete
* Fix import in tests
ci_complete
* Update with requested changes
ci_complete
* don't traceback since we don't validate the contents of module_defaults
ci_complete
Co-authored-by: Matt Davis <mrd@redhat.com>
* Allow to specify collection_name separately for deprecation.
* Use new functionality in Ansible.
* Use new functionality in tests.
* Update tagging/untagging functions.
* Update pylint deprecated sanity test.
* Update validate-modules. Missing are basic checks for version_added (validate semantic version format for collections).
* Improve version validation. Re-add version_added validation.
* Make sure collection names are added to return docs before schema validation.
* Extra checks to avoid crashes on bad data.
* Make C# module utils code work, and update/extend tests.
* Add changelog fragment.
* Stop extracting collection name from potentially tagged versions/dates.
* Simplify C# code.
* Update Windows modules docs.
* Forgot semicolons.
* Make AnsibleVaultEncryptedUnicode work more like a string. Fixes#24425
* Remove debugging
* Wrap some things
* Reduce diff
* data should always result in text
* add tests
* Don't just copy and paste, kids
* Add eq and ne back
* Go full UserString copy/paste
* Various version related fixes
* Remove trailing newline
* py2v3
* Add a test that can evaluate whether a variable is vault encrypted
* map was introduces in jinja2 2.7
* moar jinja
* type fix
Co-Authored-By: Sam Doran <sdoran@redhat.com>
* Remove duplicate __hash__
* Fix typo
* Add changelog fragment
* ci_complete
Co-authored-by: Sam Doran <sdoran@redhat.com>
* Auto unroll generators produced by jinja filters
* Unroll for native in finalize
* Fix indentation
Co-authored-by: Sam Doran <sdoran@redhat.com>
* Add changelog fragment
* ci_complete
* Always unroll regardless of jinja2
* ci_complete
Co-authored-by: Sam Doran <sdoran@redhat.com>
* various deprecation, display, warning, error fixes
* Update lib/ansible/utils/display.py
Co-authored-by: Felix Fontein <felix@fontein.de>
* Update lib/ansible/utils/display.py
Co-authored-by: Felix Fontein <felix@fontein.de>
* Update lib/ansible/utils/display.py
Co-authored-by: Felix Fontein <felix@fontein.de>
* cleanup, test fixes
* add collection name to deprecated() calls
* clean up redirect entries from uncommitted tests
* fix dep warning/error header text to match previous
Co-authored-by: Felix Fontein <felix@fontein.de>
Change:
- The `add_host` action now shows an accurate change status.
Test Plan:
- Added a plethora of integration tests.
Tickets:
Fixes#69881
Signed-off-by: Rick Elrod <rick@elrod.me>
* starting metadata sunset
- purged metadata from any requirements
- fix indent in generic handler for yaml content (whey metadata display was off)
- make more resilient against bad formed docs
- removed all metadata from docs template
- remove metadata from schemas
- removed mdata tests and from unrelated tests
Co-authored-by: Felix Fontein <felix@fontein.de>
Co-authored-by: Rick Elrod <rick@elrod.me>
Change:
- This was removed in 2014 in 122a7021bc.
- The option still exists and is enabled by default and can lead to user
confusion when people aren't expecting packages (or updated
dependencies for it) to get installed and they do.
- Add the option documentation back with a few notes to make it clear what
is happening.
Test Plan:
N/A, no code change, just documentation
Tickets:
- Refs #69497
Signed-off-by: Rick Elrod <rick@elrod.me>
* Fix sanity errors
Signed-off-by: Rick Elrod <rick@elrod.me>
* Use fqcr from command module invocation using shell module
Fixes https://github.com/ansible/ansible/issues/69788
Use fully qualified collection reference while invoking
command module from shell module
* Fox review comment
In the case of a free style strategy, it is possible to end up with
multiple hosts trying to include from the same role, however the tasks
being included may be different with the use of tasks_from. Previously
if you had two hosts that were included the same role when the
process_include_results function tries to determine if a included needs
to be run on a specific host, it would end up merging two different
tasks into which ever one was processed first.
This change updates the equality check to also check if the task uuid
associated with the IncludedFile is the same. The previous check only
checked if the task's parent uuid was the same. This breaks down when
both includes have the same parent.
- hosts: all
strategy: free
gather_facts: false
tasks:
- include_role:
name: random_sleep
- block:
- name: set a fact (1)
include_role:
name: set_a_fact
tasks_from: fact1.yml
- name: set a fact (2)
include_role:
name: set_a_fact
tasks_from: fact2.yml
- name: include didn't run
fail:
msg: >
set_a_fact didn't run
fact1: {{ fact1 | default('not defined')}}
fact2: {{ fact2 | default('not defined') }}"
when: (fact1 is not defined or fact2 is not defined)
Closes#69521
To avoid issues with Flatcar Container Linux being unable to be found,
detect Flatcar distro name especially for hostname, just like CoreOS
Container Linux was supported.
See also https://github.com/ansible/ansible/issues/69516
ModuleNotFoundError is a subclass of ImportError but only exists in
Python 3.6 or newer. Instead of doing hacks to be able to catch that on
older Pythons, just always only catch ImportError, which will also catch
ModuleNotFoundError on Python 3.6 or later
* do not return the body even if it failed
* add some tests for this and rebase
* import test task
* ignore_errors when fails
Co-authored-by: Jack Zhang <jack.zhang@aspiraconnect.com>
Change:
On OpenBSD when using pipelining, we do not set cwd which results in a
permissions fatal. Ensure that `''` - cwd - is not in `sys.path`.
Test Plan:
Tested against local OpenBSD VM
Tickets:
Fixes#69320
Signed-off-by: Rick Elrod <rick@elrod.me>
