Fixes#64966
* Fix "TypeError: splitlines() takes no keyword arguments" on Python2.7
* Add changelog fragment
* Don't use `grep -P` for BSD/macOS compatibility
* Fix sanity checks complaining about test fixtures with mixed line endings
* Update changelogs/fragments/66461-blockinfile_preserve_line_endings.yaml
Change:
Allows the user to configure sshpass (1.06+) to look for a different
substring than the default "assword" that it comes with.
Test Plan:
Set a custom ssh password prompt on a VM with PAM and tried connecting to
it. Without `ansible_sshpass_prompt` set in inventory: experienced hang.
With `ansible_sshpass_prompt` in inventory: connected successfully.
Tried setting `ansible_sshpass_prompt` with an older `sshpass` in PATH
and got a loud error, as expected.
Tickets:
Fixes#34722, fixes#54743, refs #11565.
Signed-off-by: Rick Elrod <rick@elrod.me>
* `meta/` directory in collections
* runtime metadata for redirection/deprecation/removal of plugin loads
* a compatibility layer to keep existing content working on ansible-base + collections
* a Python import redirection layer to keep collections-hosted (and otherwise moved) content importable by things that don't know better
* supported Ansible version validation on collection loads
* fix delegated interpeter
* allow returning fact if it is 'the right host'
* added note for future fix/efficiency
as it stands we rerun discovery for the delegated host
unless its saving facts to itself
* fixed test lacking delegate_to mock
When mixed with the free strategy (or any custom strategy that does not behave in
a lock-step manner), the linear methodology of _wait_on_handler_results may cause
race conditions with regular task result processing if the strategy uses
_process_pending_results directly. This patch addresses that by splitting the queues
used for results and adding a flag to _process_pending_results to determine which
queue to check.
Fixes#69457
* Add multipart/form-data functionality
* Fix some linting issues
* Fix error message
* Allow filename to be provided with content
* Add integration test
* Update examples
* General improvements to multipart handling
* Use prepare_multipart for galaxy collection publish
* Properly account for py2 vs py3, ensuring no max header length
* Address test assumptions
* Add unit tests
* Add changelog
* Ensure to use CRLF instead of NL
* Ignore line-endings in fixture
* Consolidate code, add comment
* Bump fallaxy container version
* ci_complete
* ansible-galaxy - remove warning during collection install
If existing collections do not contain a MANIFEST.json, which is common
for collections under development that were not installed from Ansible
Galaxy, fall back to inspecting galaxy.yml rather than displaying a
warning.
A warning will still be displayed in neither a MANIFEST.json nor
galaxy.yml are present.
* Update unit tests
* Unify ansible-galaxy install -r
* Minor nit fixes for docs
* Re-align warnings
* Fix up integration test
* Fix up test where no roles/collections were in file
This fact reflects the number of usable vcpus (which might be different
from ansible_processor_vcpus, e.g., in containers with limits). See
also #51504.
* Add fixture data and update unit tests
Co-authored-by: Sam Doran <sdoran@redhat.com>
The updated container includes fewer requirements now that the collection migration has completed.
Collections which encounter test issues with this new container should update their test requirements files to include the necessary requirements.
* Fix filedescriptor out of range in select() when running commands
* Simplify the run_command() code
Now that we're using selectors in run_command(), we can simplify some of
the code.
* Use fileobj.read() instead of os.read()
* No longer use get_buffer_size() as we can just slurp all of the data
instead.
Also use a simpler conditional check of whether the selector map is
empty
Co-authored-by: Toshio Kuratomi <a.badger@gmail.com>
* only show_vars when showing vars
avoid processing function params that can be very expensive
and might not be used at all in called function.
fixes#69357
* Update changelogs/fragments/69357_optimize_inventory_graph_wo_vars.yml
Co-authored-by: Sloane Hertel <shertel@redhat.com>
Co-authored-by: Sloane Hertel <shertel@redhat.com>
Since Ansible 2.9.8, if the fileglob plugin is passed a path containing
a subdirectory of a non-existent directory, it will fail. For example:
lookup('fileglob', '/'): ok
lookup('fileglob', '/foo'): (non-existent): ok
lookup('fileglob', '/foo/bar'): (non-existent): FAIL
The exact error depends on Python 2 or 3, but here is the error on
Python 2:
AttributeError: 'NoneType' object has no attribute 'endswith'
And on Python 3:
TypeError: expected str, bytes or os.PathLike object, not NoneType
This change fixes the issue by skipping paths that are falsey before
passing them to os.path.join().
Fixes: #69450
* Fix galaxy publish sha256 value format.
The multipart/form content used for the body
of the POST to /api/automation-hub/v3/collections
was missing a newline before the line with the value
of the sha256.
automation-hub/galaxy/django skips the field entirely in
that case and automation-hub code will use None for default
to indicate that no sha256 is provided (an available option).
Fixesansible/galaxy-dev#246
* Add changelog fragment
Co-authored-by: Matt Martz <matt@sivel.net>
Change:
Adds Fedora 32 to shippable and alters tests slightly for new Fedora.
Test Plan:
CI
Tickets:
Fixes#69230
Co-authored-by: Matt Clay <matt@mystile.com>
* validate-modules: deprecated modules in collections
In Collections a module is marked as deprecated via meta/routing.yml
Use this file, rather than the leading `_` as part of the deprecated
test.
* Correct variable
* review comments
* indentation
* Read routing.yml only once
* pep8
* Apply suggestions from code review
Co-authored-by: Matt Clay <matt@mystile.com>
* review: remove duplicated conditional
Co-authored-by: Matt Clay <matt@mystile.com>
* use orderdict for yaml dictionaries
* clog
* SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS
* allow user to toggle odict off
* removed config, since requored to load config
* remove unused import
* Update changelogs/fragments/yaml_orderd_mappings.yml
Co-authored-by: Matt Clay <matt@mystile.com>
* Update lib/ansible/parsing/yaml/objects.py
Co-authored-by: Matt Clay <matt@mystile.com>
* Update lib/ansible/parsing/yaml/objects.py
Co-authored-by: Matt Clay <matt@mystile.com>
Co-authored-by: Matt Clay <matt@mystile.com>
PopOS is a Debian based OS distribution, added support to detect
ansible_os_family as 'debian' instead of 'Pop!_OS'
Fixes: #69286
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* Metadata defaults were not being set if only a few fields were missing.
* ansible-doc with no documentation and no status in metadata should
return empty, just like if there was no documentation and no metadata
at all.
* Address FIXME's in sysctl.py
* Added changelog fragment
* Updated check
* Update conditions
* if not instead of is None
* Restore and delete FIXME comments
* Do not pass file mode during recursive copy on symlink files.
The 'file' module cannot deal with mode=preserve. Do not pass that
mode to the module when 'preserve' is used.
* Fix changelog fragment filename
Add the ability to pass allowerasing to alter the dnf transaction
behavior.
Fixes#24161
This is effectively a port of the original pull request from
poettler-ric who has since abandoned the PR
https://github.com/ansible/ansible/pull/34111
Signed-off-by: Adam Miller <admiller@redhat.com>
Change:
Extend the logic for custom error handling in the dnf module, so that on
newer DNF (such as DNF that ships with modern Fedora 31 container
images, and ships with RHEL 8.2) we report errors consistently with
older DNF.
Test Plan:
Ran dnf integration tests against an old Fedora 31 container image and a
brand new Fedora 32 container image; tess passed on both.
Signed-off-by: Rick Elrod <rick@elrod.me>
With inventory script migrated to their respective collection,
redirect links in documentation to their respective collection
location.
Fixes: #69139
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* Make sure collection is a list if a str is given
* Call field validation early on collections
Because we are doing work on modifying the collections value before
it is actually validated, we can validate it ourselves early to make
sure the user supplies either a string or list. Dicts are not valid.
The new validation allows us to simplify the _ensure_default_collection()
function. And since the field is now static, we no longer need to specify
a default for it, which also allows us to simplify the function. Since
the default is now removed, we can also remove the sanity/ignore.txt entry
for collectionsearch.py.
New unit tests are added (and the existing one modified) that allow us to
make sure that we throw a parser error if a user specifies something other
than a string or list for the collections value everywhere it can be specified.
* Revert removing the collection default
The default is actually used, so restore it.
* Fix unit tests in test_helpers.py affected by early collection validation
The call to daemonize() in sysvinit.py was missing the module parameter included in the function definition in service.py.
This pull request simply adds that parameter, as the module is used for error handling in daemonize().
* service_facts: fix for systemd 245
Since systemd 245, `systemctl list-unit-files` comes with a new column
"VENDOR PRESET" [1] and breaks the service_facts module:
$ ansible localhost -m service_facts
localhost | FAILED! => {
"changed": false,
"msg": "Malformed output discovered from systemd list-unit-files: auditd.service disabled disabled "
}
This patch drops the third column to make it work with old and new
systemd. With the new slice operation, IndexError instead of ValueError
is raised if the output contains less than 2 columns.
Test plan: running `ansible-test integration -v service_facts` on
up-to-date Arch Linux
[1] https://github.com/systemd/systemd/pull/14445
* add changelog
Signed-off-by: Rick Elrod <rick@elrod.me>
Co-authored-by: Rick Elrod <rick@elrod.me>
Openstack dynamic inventory script is moved from community.general to
openstack.cloud, adjust the bot meta accordingly.
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
Currently if virtualenv_command has arguments, then the
search for the binary in the path does not work so the
user has to specify the full path to it.
To allow arguments to be used without having to specify
the path to the binary, we split the module argument into
the command and anything after the first space.
This makes using this module argument more flexible and
user friendly.
Fixes: #52275
Change:
Rather than hardcoding .pyo and .pyc, filter on all BLACKLIST_EXTS in
the non-legacy logic of PluginLoader (_find_fq_plugin). The two harcoded
extensions are part of BLACKLIST_EXTS already and this simply adds the
rest of the blacklisted extensions to the check.
In addition, check .endswith() instead of an exact match of the suffix,
like everywhere else that uses BLACKLIST_EXTS. This allows for
blacklisting, for example, emacs's backup files which can appear after
any extension, leading to things like `foo.py~`.
Test Plan:
Ran `ansible-playbook` against a collection where a `foo.py~` module was
getting executed instead of `foo.py` which also appeared in the same
directory. `foo.py~` is no longer executed.
Tickets:
Fixes#22268
Refs #27235
Signed-off-by: Rick Elrod <rick@elrod.me>
* Allow a collection role to call a standalone role by default. Fixes#69101
* tweaked changelog text
* Guard against NoneType
Co-authored-by: Matt Davis <nitzmahone@users.noreply.github.com>
* update ActionBase._low_level_execute_command to honor executable
* adding changelog fragment
* renaming changelog fragment to .yml
* noop change to bump shippable
* adding raw_executable integration test
* copying aliases from raw
* removing blank lines
* skipping aix and freebsd
* noop to bump shippable
* moving tests to raw/
* removing become_method: sudo ; it doesn't work on AIX
* removing trailing blank line
* forcing become_method: su to try to get AIX to work
Co-authored-by: Rob Wagner <rob.wagner@sas.com>
Change:
This corrects an incorrect CVE identifier in the changelog entry for
CVE-2020-1735.
Test Plan:
N/A
Tickets:
Refs #67793, #68720
Signed-off-by: Rick Elrod <rick@elrod.me>
This patch fixes the issue where nested Block copies were created from
incorrect Block object. This resulted in nested Blocks data like ``name``
or ``_uuid`` to contain values from the Block the filter_tagged_tasks
method was called on.
* Testing: Add CentOS Linux On Power platform
* Add arch designation to remotes.
This avoids overloading the provider with the arch.
Also add a changelog entry.
Co-authored-by: Matt Clay <matt@mystile.com>
* ansible-galaxy - fix listing specific role
If the role was not in the first search path, it was reported as not found
* Properly display role description
Default to description to top level description, falling back to the description from within galaxy_info
* Display proper message when a role does not exist
* Add integration tests
* Use context manager
* BSD and macOS ruining all the fun
* also consolidated temp dir name generation, added pid for more 'uniqness'
* generalize error message
* added notes about remote expansion
CVE-2020-1733
fixes#67791
* Force collections to be static
Templating of collection names does not work at all. Force them to
be static so that a warning is generated for the user.
* Add collectionsearch unit test and fix for reviews
New unit test validates the new _load_collections() code and moves
the new check to the end of the method.
* Change unit test to pytest
* Adjust unit test to use capsys instead of monkeypatch
* Fix pep8 error
* Add changelog fragment
Closes#68704
* subversion module - provide password securely with svn command line option --password-from-stdin when possible, and provide a warning otherwise.
* Update lib/ansible/modules/source_control/subversion.py.
* Add a test.
Co-authored-by: Sam Doran <sdoran@redhat.com>
* Don't trigger full CI run for changes to changelogs/ and docs/ in collections.
* Add changelog fragment.
* Update changelogs/fragments/68550-ansible-test-docs-changelogs.yml
Co-Authored-By: Matt Clay <matt@mystile.com>
Co-authored-by: Matt Clay <matt@mystile.com>
* Update docker.txt to use the OpenSUSE 15.1 container image
Signed-off-by: Rick Elrod <rick@elrod.me>
* handle installing mysql on suse
Signed-off-by: Rick Elrod <rick@elrod.me>
* add changelog fragment
Signed-off-by: Rick Elrod <rick@elrod.me>
* Update changelogs/fragments/ansible-test-opensuse-15.1.yml
Co-Authored-By: Matt Clay <matt@mystile.com>
Co-authored-by: Matt Clay <matt@mystile.com>
* Update tests to use RHEL 7.8.
Keeping support for RHEL 7.6 since collections are still using it.
* Fix tests for RHEL 7.7+ due to extras repo name change.
* fixed fetch traversal from slurp
* ignore slurp result for dest
* fixed naming when source is relative
* fixed bug in local connection plugin
* added tests with fake slurp
* moved existing role tests into runme.sh
* normalized on action excepts
* moved dest transform down to when needed
* added is_subpath check
* fixed bug in local connection
fixes#67793
CVE-2019-3828
* Allow tasks to notify a fqcn handler name
* Add tests. Fixes#68181
* Add changelog fragment
* Add test to ensure handlers are deduped properly with fqcn, role, and just handler names
* Add some docs about new special vars
fial_json() requires a message be given to it to inform the end user of
why the module failed. Prior to this commit, the message had to be a
keyword argument:
module.fail_json(msg='Failed due to error')
Since this is a required parameter, this commit allows the message to be
given as a positional argument instead:
module.fail_json('Failed due to an error')
* Always set the discovered interpreter on the delegated host. Fixes#63180
* Make code a little more generic
* Move code into a function
* Implement some changes based on reviews
* Add changelog fragment
* when possible, use filedescriptors from mkstemp to avoid race
* when using path strings, ensure we are always creating the file
CVE-2020-1740
Fixes#67798
Co-authored-by: samdoran
The last task in a play should now properly report code coverage.
This change should also eliminate empty coverage files, as well as incomplete coverage files resulting from early worker termination.
* Fix test_check_mutually_exclusive exception-checking
Asserting inside of the `with` context of `pytest.raises`
doesn't actually have any effect. So we move the assert
out, using the exception that gets placed into the scope
after we leave the context, and ensure that it actually gets
checked.
This is also what the pytest documentation says to do:
https://docs.pytest.org/en/latest/assert.html#assertions-about-expected-exceptions
Signed-off-by: Rick Elrod <rick@elrod.me>
* Add some tests for check_required_together
Signed-off-by: Rick Elrod <rick@elrod.me>
* use to_native instead of str, for consistency
Signed-off-by: Rick Elrod <rick@elrod.me>
* Add newlines for pep8
Signed-off-by: Rick Elrod <rick@elrod.me>
* Add tests for check_required_arguments
Signed-off-by: Rick Elrod <rick@elrod.me>
* Sort missing keys in error message, since hashes are unsorted and this can be random
Signed-off-by: Rick Elrod <rick@elrod.me>
* Add changelog entry
Signed-off-by: Rick Elrod <rick@elrod.me>
* fix vault tmpe file handling
* use local temp dir instead of system temp
* ensure each worker clears dataloader temp files
* added test for dangling temp files
* added notes to data loader
CVE-2020-10685
* Remove some unreachable code in the file module
Remove some cases in file.py which are covered by conditionals a few
lines earlier. Remove the duplicate code which will never be hit.
Signed-off-by: Rick Elrod <rick@elrod.me>
* Restore incidental file coverage from timezone module
Signed-off-by: Rick Elrod <rick@elrod.me>
* Combine two conditionals, add a changelog entry
Signed-off-by: Rick Elrod <rick@elrod.me>
* Make new test syntax consistent, add two stat tests
Signed-off-by: Rick Elrod <rick@elrod.me>
* Support pre-releases via new SemanticVersion. Fixes#64905
* Don't treat buildmeta as prerelease
* Don't inherit from str and int
* Add helper method to try and construct a SemanticVersion from a LooseVersion
* Don't count major 0 as pre-release, it's different
* Guard against invalid or no version in LooseVersion
* return a bool
* Add integration tests for pre-release
* Fix up lingering issues with comparisons
* typo fix
* Always allow pre-releases in verify
* Move pre-release filtering into CollectionRequirement, add messaging when a collection only contains pre-releases
* Update changelog
* If explicit requirement allow pre releases
* Enable pre-releases for tar installs, and collections already installed when they are pre-releases
* Drop --pre-release alias, make arg name more clear
* Simplify code into a single line
* Remove build metadata precedence, add some comments, and is_stable helper
* Improve from_loose_version
* Increase test coverage
* linting fix
* Update changelog
* remove azure extras and extras_require support
* Since Azure will be collectionized, the requirements will float more frequently than Ansible releases; the Azure collection needs to host the requirements now.
* Removed the dynamic extras support as well, since Azure was the only thing using it. If we need it again, it's easy to pull back from history.
* Mark azure-requirements as orhpaned.
This keeps the docs around so that existing links from old test runs remain valid.
Co-authored-by: Matt Clay <matt@mystile.com>
Now empty `*.py` files are ignored during module_utils import analysis for change detection.
This eliminates "No imports found" warnings for files which should have no imports.
* Address compat issue for collection loading on py26
* Move import_module shim to utils for compat across the codebase
* Enable collection tests on py2.6
* Update changelog fragment
* Simplify code using sys.moduls
* Move compat to module_utils/compat/importlib
* Add back errantly deleted newline
* Remove hack comment
Co-Authored-By: Matt Clay <matt@mystile.com>
Co-authored-by: Matt Clay <matt@mystile.com>
* win_timezone - Allow for _dstoff timezones
* Update win_timezone-Allow-dstoff.yml
* Added doc entry for new format
Co-authored-by: Jordan Borean <jborean93@gmail.com>
* ansible-galaxy - optimise some paths and use fake galaxy int tests
* Added init, built, and publish tests
* Test against both mocked Galaxy and AH server
* Finish off writing the install tests
* Fix up broken tests
* Rename test target and add migrated tests
* Use cloud provider for Galaxy implementation
* Added blank static config
* Use correct alias group
* Set release version and fix copy typo
* Remove reset step as it is no longer needed
* Use sane env var names for test container name
* Run modules on windows container
This provides an ability to run Powershell modules on windows container via docker connection. Otherwise, Ansible tries to run python modules on windows containers and fails.
* Removing whitespace in the blank lines
* Adding a changelog fragment
* fix:git module ignores remote_tmp
* WIP: added the changelog fragment and edit comment
* Revert "WIP: added the changelog fragment and edit comment"
This reverts commit 2f739df619.
* WIP: added changelog fragments
* WIP: added changelog fragment
* WIP: fixed spelling in changelog fragment
* reworked sqs_queue
* Switch default purge_tags behaviour to false.
This matches the behaviour of ec2_tag and ecs_tag.
* Minor lint / review fixups
* Add missing AWS IAM policy for SQS tests
* Move integration tests to using module_defaults: group/aws:...
* add changelog
* Break out the 'compatability' map from our spec definition (gets flagged by the schema validation)
* Tweaks based on review
* add basic examples
* Lint fixups
* Switch out NonExistentQueue logic so it's easier to follow
* Reorder name argument options for consistency
Co-authored-by: Dennis Podkovyrin <dennis.podkovyrin@gmail.com>
* Allow passing through of (almost) all params available on boto methods in aws_api_gateway
* Linting and docs fixes
* Refactored method signature of create_deployment() to use keyword args instead of named args
* Updated version_added flags to 2.10
* Cleanup and improve aws_api__gateway integration test play. Also included new params into test.
* Fixed RETURN docs and some ttests
* Completed RETURN docs and made integration tests match
* Fixed variable names in test and YAML syntax in docs
* Comment out critical sections of integration test
* Fixed update test after figuring out what the error message means. Also updated error message to be more descriptive.
* Fixed test assertion
* Update docs and make tests reflect that endpoint type wont be changed on updates
* Syntax fix
* Add changelog fragment
* Improve aws_api_gateway docs, fix typos.
* Quote doc lines with colon
* Cleanup tests
* Auto-Retry on ResourceNotFound and RequestInProgress exceptions
* Use AnsibleModule options for required_if logic
* changelog
* Remove (now) duplicate RequestInProgressException catching
* Allow a single retry when attempting to fetch the information about a cert directly after deleting it.
There is a small chance that it goes away while we pull the details.
* add key rotation option
* add changelog fragment
* provide version added as string
* change changelog to minor_changes
* Update changelogs/fragments/67651-aws-kms-key-rotation.yml
Co-Authored-By: Mark Chappell <mchappel@redhat.com>
* Update lib/ansible/modules/cloud/amazon/aws_kms.py
Co-Authored-By: Mark Chappell <mchappel@redhat.com>
* prevent key upgrade if key rotation was enabled manually. In that case, the key rotation would be disabled, if not mentioned in the playbook
* Update lib/ansible/modules/cloud/amazon/aws_kms.py
Co-Authored-By: Mark Chappell <mchappel@redhat.com>
* Update lib/ansible/modules/cloud/amazon/aws_kms.py
Co-Authored-By: Mark Chappell <mchappel@redhat.com>
* Update lib/ansible/modules/cloud/amazon/aws_kms.py
Co-Authored-By: Mark Chappell <mchappel@redhat.com>
Co-authored-by: Mark Chappell <mchappel@redhat.com>
Handle NoneType error occured due to accessing host system service info
in vmware_host_service_info module.
Fixes: #67615
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* fixed#47050
* added changelog fragment
* added quick and basic test
* Revert "added quick and basic test"
This reverts commit 75f4141656.
* added better tests
* now also creating files to copy on the remote
* removed tests for recursive copying which is not supported by remote_src
* sns_topic: (integration tests) Move the tests over to using module defaults
* sns_topic: (integration tests) Add test for behaviour of changed when using delivery_policy
* sns_topic: ensure "changed" behaves properly when managing delivery policies
- a delivery_policy isn't an IAM policy, so compare_policies didn't cope with it
- AWS automatically adds an additional option when you set an HTTP delivery
policy
* Parse the delivery policies so we can test the changes properly
* Remove dead code key_matches_filter/key_matches_filters
* Fail more cleanly when we don't recognise the 'shape' of KMS policy
* Refactor aws_kms to bring down the complexity
* Minor docs tweaks
* Changelog fragment
* Fixups from review
