PR #72591
This change:
* Adds an artifacts manager that abstracts away extracting the
metadata from artifacts, downloading and caching them in a
temporary location.
* Adds `resolvelib` to direct ansible-core dependencies[0].
* Implements a `resolvelib`-based dependency resolver for
`collection` subcommands that replaces the legacy
in-house code.
This is a dependency resolution library that pip 20.3+ uses
by default. It's now integrated for use for the collection
dependency resolution in ansible-galaxy CLI.
* Refactors of the `ansible-galaxy collection` CLI.
In particular, it:
- reimplements most of the `download`, `install`, `list` and
`verify` subcommands from scratch;
- reuses helper bits previously moved out into external modules;
- replaces the old in-house resolver with a more clear
implementation based on the resolvelib library[0][1][2].
* Adds a multi Galaxy API proxy layer that abstracts accessing the
version and dependencies via API or local artifacts manager.
* Makes `GalaxyAPI` instances sortable.
* Adds string representation methods to `GalaxyAPI`.
* Adds dev representation to `GalaxyAPI`.
* Removes unnecessary integration and unit tests.
* Aligns the tests with the new expectations.
* Adds more tests, integration ones in particular.
[0]: https://pypi.org/p/resolvelib
[1]: https://github.com/sarugaku/resolvelib
[2]: https://pradyunsg.me/blog/2020/03/27/pip-resolver-testing
Co-Authored-By: Jordan Borean <jborean93@gmail.com>
Co-Authored-By: Matt Clay <matt@mystile.com>
Co-Authored-By: Sam Doran <sdoran@redhat.com>
Co-Authored-By: Sloane Hertel <shertel@redhat.com>
Co-Authored-By: Sviatoslav Sydorenko <webknjaz@redhat.com>
Signed-Off-By: Sviatoslav Sydorenko <webknjaz@redhat.com>
While getting policy name in get_chain_policy API,
module does not require any additional parameters except chain
Enabling flag in get_chain_policy API call fixes this.
Fixes: #68612
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
Use correct ssh executable and options in all cases on connection plugin
* Also nicer naming/comments
Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
Ability to add local variables into AnsibleJ2Vars was added in
18a9eff11f to fix#6653. Local variables
are added using ``AnsibleJ2Vars.add_locals()`` method when creating a
new context - typically when including/importing a template with
context. For that use case local template variables created using
``set`` should override variables from higher contexts - either from the
play or any parent template, or both; Jinja behaves the same way.
Also removes AnsibleJ2Vars.extras instance variable which is not used.
Also adds missing test for #6653.
Fixes#72262Fixes#72615
ci_complete
The `command` module does not return stdout & sterr when calling
a non existing executable or an unknown exception arises. This fix
lets the module return empty byte strings in those cases.
* Consolidate filters/tests handling into JinjaPluginIntercept
ci_complete
* Postpone loading all ansible plugins
* Do we need to create an overlay?
ci_complete
* Typo
ci_complete
* Add FIXME
* conditional.py: use public Environment.parse() method
* Remove remaining occurrences of shared_loader_obj being passed to Templar
* __UNROLLED__ not needed with this change anymore
* Incorrect rebase at some point?
When calling compile(), the filename argument should be either a real file
name or a string. According to Python docs, suggested one is '<string>'.
Keep the current behaviour (encapsulate the actual expression),
but enclose it into angle brackets.
Signed-off-by: Oldřich Jedlička <oldium.pro@gmail.com>
* Fix YAML error message when error is at the end of the file
If a YAML file fails to load due to a syntax error in a file, or there is an error in the last line of a
file, PyYAML reports the last line number of the file as the index where the error occurred.
When reading the file lines, we use that index to the get the relevant line. If the index value is out
of range, the relevant line is lost for error reporting.
Subtract one from the index value to avoid the IndexError in this specific scenario. It is possible
to still get an IndexError, which will be handled as it is currently.
* Update existing tests and add new tests
* Allow restricting config values to enumerated list
* dont document internal entries
* also ignore private defs for ansible-config
* remove invalid value from tests
* added porting entry
When the pause module is run in the background and seconds parameter is provided,
do not warn.
* Add tests
* Fix existing tests
The test wasn't failing when it should have.
Change:
- On CentOS Stream, make distribution_release be "Stream"
- On CentOS Core, it continues to be "Core"
- Implement custom distribution file parser for CentOS, so we can look
for "CentOS Linux" and "CentOS Stream"
- Two new fixtures introduced (CentOS Linux 8.1 and CentOS Stream 8)
- Removed two dicts from `Distribution` class that were seemingly not
used anywhere.
Test Plan:
- ci_complete
- New test fixtures
Tickets:
- Fixes#73027
Signed-off-by: Rick Elrod <rick@elrod.me>
Do the right thing on Linux when password lock and a password hash are provided by writing
out the password hash prepended by the appropriate lock string rather than using -U and -L.
This is the correct way to set and lock the account in one command.
On BSD, run separate commands as appropriate since locking and setting the password cannot
be done in a single action.
FreeBSD requires running several commands to get the account in the desired state. As a result,
the rc, output, and error from all commands need to be combined and evaluated so an accurate
and complete summary can be given at the end of module execution.
* Improve integration tests to cover this scenario.
* Break up user integration tests into smaller files
* Properly lock account when creating a new account and password is supplied
* Simplify rc collection in FreeBSD class
Since the _handle_lock() method was added, the rc would be set to None, which could make
task change reporting incorrect. My first attempt to solve this used a set and was a bit too
complicated. Simplify it my comparing the rc from _handle_lock() and the current value of rc.
* Improve the Linux password hash and locking behavior
If password lock and hash are provided, set the hash and lock the account by using a password
hash since -L cannot be used with -p.
* Ensure -U and -L are not combined with -p since they are mutually exclusive to usermod.
* Clarify password_lock behavior.
The old example failed with:
fatal: [localhost]: FAILED! => {"msg": "to count backwards make stride negative"}
I fixed this and placed a start parameter to consolidate with the rest of the examples provided.
Co-authored-by: Jorge Rua <jorua@redhat.com>
Change:
- Internally, use dnf.subject.Subject#get_best_query for state: absent
- Add a bunch of tests for removing packages, given a bunch of different
pkg specs (nv, nvr, nvra, wildcard, etc.)
Test Plan:
- New tests
- Local experiments with DNF API via PDB.
Tickets:
- Fixes#72809
Signed-off-by: Rick Elrod <rick@elrod.me>
* Replace the use of the function string.replace() with the method str.replace()
because that's what works for both Python 2 and 3.
* Cleanup the unused string import.
Added a changelog fragment.
* The documentation for os.write() seems a bit iffy, but in Python 3 we
definitely cannot pass it a string and we need to encode it into
bytes.
The Python documentation at
https://docs.python.org/3/howto/pyporting.html#text-versus-binary-data
says that this code will work in Python 2 as well.
Co-authored-by: Lloyd Parkes <lloyd@must-have-coffee.gen.nz>
Change:
- The FreeBSD release can contain -RC or -PRERELEASE in addition to
-RELEASE, -STABLE, or -CURRENT.
Test Plan:
- Added new fixed from an RC version of TrueNAS which uses a -PRERELEASE
version of FreeBSD.
Tickets:
- Fixes#72331
Signed-off-by: Rick Elrod <rick@elrod.me>
* arg_spec - move type checking lookup method to a function
* Change get_wanted_type name and behavior
Change the name to get_validator to bette describe what it is doing.
Change the interface to always return a value. This lines up with the behavior of get_*
functions always returning something or None and check_* functions raising an
Exception if something went wrong during the check.
* Add param to check_type_str()
Not meant to be a long term fix, but gets tests passing. More work is needed to figure
out how to solve this cleanly.
* Remove private attribute mapping types to validator
Since the function that needs it has moved to parameters.py, there is no need to have it as
a attribute of AnsibleModule.
Update tests that were referencing the private attribute.
* Use private method for 'str' type
To avoid having to put the string conversion warning behavior in the check_type_str() method,
use the private _check_type_str() method for 'str' type.
Import CHECK_ARGUMENT_TYPES_DISPATCHER for backwards compalitibility and store it as
a private attribute.
Revert changes to support plugins that are referencing serf._CHECK_ARGUMENT_TYPES_DISPATCHER.
* Add changelog
* Change function name to better reflect its... function
* Change dict name to better reflect its contents
CHECK_ARGUMENT_TYPES_DISPATCHER --> DEFAULT_TYPE_VALIDATORS
* Fix changelog
* systemd - do not overwrite unit name when searching
PR #72702 introduced a bug that changed the unit name when splitting it up for the purpose
of searching for the unit. This only happens on unit file templates on systems that have a 5.8
or newer kernel and a version of systemd that does not contain a bugfix that causes systmed
to fail to parse dbus.
* Use facts rather than a manual probe to determine if systmed is present
* Remov unnecessary block
* Use vars files instead of set_fact
* Add tests for using a templated unit file
* Update changelog fragment
* Use template to get correct path to sleep binary
* import_playbook - change additional params to deprecation
I incorrectly recommended this be set as a warning when it should have been a deprecation.
* Fix deprecation sanity test to not required a collection name when not inside a collection
* Change role argspec file used by ansible-doc
This changes the file used for role argument specs from meta/argument_specs.yml
to meta/main.yml. The argument specs are expected to be in that file under the
top-level entry of `argument_spec`.
* Switch to argument_specs
* Update yum/dnf module docs to include version comp
Fixes#61234
Signed-off-by: Adam Miller <admiller@redhat.com>
* fix up docs syntax to sanity tests are happy
Signed-off-by: Adam Miller <admiller@redhat.com>
* ansible-galaxy collection list and verify now utilize collections in site-packages.
This is a short term fix for #70147. The long term fix needs to handle
install (but that discussion is also bound up in how upgrade is going to
work and where things can get installed so it's deferred for 2.11.)
* Add test for ansible-galaxy collection list with site-packages
* Fix sanity issue
Co-authored-by: David Moreau Simard <moi@dmsimard.com>
Co-authored-by: Jordan Borean <jborean93@gmail.com>
On python3 sys.stdin is an encoded file object that does not support
reading raw binary data. Use the supplied buffer object to do so.
Signed-off-by: Sven Wegener <sven.wegener@inovex.de>
Co-authored-by: Sven Wegener <sven.wegener@inovex.de>
This should allow users to extract specific files from an archive as
desired.
Fixes#16130, #27081.
* Rebase and make a few minor changes
* Add changelog
* Improve tests
- move to separate tasks file
- change assertions to check for exactly one file
- use remote_tmp_dir for output dir
* Make exclude and include mutually exclusive
* Don't remove files needed by other tasks
* Fix sanity tests
* Improve feature documentation
* Skip tests that use map() on CentOS 6
* Use fnmatch on include for zip archives
This matches the behavior of exclude
Co-authored-by: Sam Doran <sdoran@redhat.com>
* Remove space before comma in '... bla , use ...'
* 'why' is inserted in the middle of a sentence, between two commas.
* Make deprecations from base.yml show source ansible-core.
* Add changelog fragment.
* Improve some more 'why's.
* Add PR URL to fragment.
* file: add symlink is in a sticky directory tests
* file: handle symlink in a sticky directory
Co-Authored-By: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
* Add changelog and fix unit test
The builtins import was removed since it was unused, but it is now needed.
This is in reference to the issue: https://github.com/ansible/ansible/issues/41130 (Cannot decrypt with copy module when remote_src: yes)
Autodecryption of the vaulted files does not work when we want to copy files with remote_src=yes.
Co-authored-by: ziaurrehman72 <ziaurrheman72@yahoo.com>
* Move _syslog_facitily to __init__
No good reason it should not be set for each object
* Move internal property setting to private method
* Create check_arguments() function
* Remove unused import
* Rename function to better match its behavior
Change the behavior to return a set, either empty or populated, with unsupported keys.
Accept legal_inputs as optional which will not required calling handle_aliases before calling
get_unsupported_parameters().
* Add changelog
* Rework function behavior and documentation
I realized I missed the original intent of this method when moving it to a function. It
is meant to compared the parameter keys to legal inputs always, not compare
parameter keys to argument spec keys, even though the argument spec keys should
be a subset of legal inputs.
* Add tests
* Fix typo.
* Set internal properties when handling suboptions
* Support listing roles in text and JSON
* Change tests for unfrack'd playbook_dir var
These tests were using '/tmp' for testing the setting of the playbook_dir
var. Now that we unfrack that var, MacOS will change this to '/private/tmp'
causing the tests to fail. We can choose a path that does not exist (since
unfrack does not validate existence) so that we can guarantee unfracking
will not change the value.
Change:
- Initial set of changes for renaming to ansible-core
- Includes changelog fragment changes from base -> core
- Does NOT include docs changes
- Modifies detection stuff in setup.py to support ansible<2.9 and ansible-base
Test Plan:
- ci_complete
* Fix type of mode field in copy module
* Remove module type checking from ignore.txt sanity checks
* Remove mode type
Co-authored-by: Matt Martz <matt@sivel.net>
* Fix super annoying Python 2.6 multiprocessing.Queue stack trace in CI
A bug exists in Python 2.6 that sometimes raises an exception during interpreter shutdown. We
encounter this frequently in our CI since we run tests on CentOS 6 as the control node, which
has Python 2.6.6 with this bug.
This PR adds a very minor sleep only on Python 2.6 which gets around this issue. I did lot of testing
using a standalon script I found that easily duplicated the issue to find the minimum sleep value
needed to avoid this issue.
CPython issue: https://bugs.python.org/issue4106
Fix in CPython: https://hg.python.org/cpython/rev/d316315a8781
* Use correct attribute
* Consolidate logic for determining whether or not session is interactive
into a single function, is_interactive()
* Increase test coverage
I wasn't able to find a good way of simulating running a backgrounded test with CI since the
whole test is essentially run not in a TTY, which is similar enough to cause the new is_interactive()
function to always return false.
* add return documentation for unarchive, add fragment for changelog, apply suggestions from code review
Co-authored-by: Amin Vakil <info@aminvakil.com>
* Clarify collection paths in docs, fix rst syntax, rephrase docs about using collection from folder/local git
Co-authored-by: Felix Fontein <felix@fontein.de>
* Added caching mechanism for Galaxy API requests
* Add cache options and split up code
* Added unit tests
* Fix sanity test
* Use modified date and fix up caching for explicit servers
* Make sure credentials are not in cached server name
* Added test for getting updated cache version
* Changes from review
* Reorder comment postition
* Add comment unit test
* Fix unit test
* Fix unit test
* Add changelog
* Add paramaters which would be problematic without this fix
* Fix typo
* Fix unit test
* Fix unit test
* Emit warning when running on the controller with a Python older than 3.8
* Add spaces
Co-authored-by: Matt Clay <matt@mystile.com>
* and more spaces
Co-authored-by: Matt Clay <matt@mystile.com>
* s/Py/Python/
Co-authored-by: Matt Clay <matt@mystile.com>
* Add note to Control node requirements about Py3.8 requirement
* Add collection_name to deprecated call
* more spaces
Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
* Clarify that we are only packaging for py3.8+
Co-authored-by: Matt Clay <matt@mystile.com>
Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
* Fix 'role_name : tast_name' notation if task contains role name
* Add tests for notifying handler names which contain the role name
Co-authored-by: Thomas Wouters <thomaswouters@gmail.com>
* Support docker and k8s action groups for moved modules in community.docker and community.kubevirt.
* Also support k8s action group for community.okd.
* Also add kubernetes.core.
* Adjust PR #.
* Fix changelog fragment.
* Remove community.okd.
* Revert "Remove community.okd."
This reverts commit 812b5aa6e2.
Change:
- Docs: Add note that security/bugfix apply to dependencies too, like
the dnf command.
- dnf: security/bugfix only makes sense for updates, so limit the
package query sack to available updates.
- tests: Limit tests to our known-good test packages, so that RHEL
packages marked security/bugfix without similarly marked dependencies
don't fail our tests.
Test Plan:
- Tested with `dnf upgrade-minimal --bugfix` and reproduced the same
error currently seen in CI, showing that we are consistent with what
dnf does.
Tickets:
- Likely fixes#72316
Signed-off-by: Rick Elrod <rick@elrod.me>
* Indent Jinja2 directives.
* Show version_added and deprecations for ini settings and environment variables.
* Make default conditional, add choices. Copied from #55474.
* Add Ansible variables. Copied from #55474.
* Turn ini keys and environment variables into lists. Copied from #55474.
* Improve formatting. Copied from #55474.
* hint in doc of git module to abort on missing http password
improves #69489
* fix syntax in doc of git module
Co-authored-by: Matt Martz <matt@sivel.net>
Co-authored-by: Matt Martz <matt@sivel.net>
* ansible-doc: export has_action when --json is used.
* Remove docuri and now_data, which were not used resp. ignored in format_plugin_doc and the functions it calls anyway.
* Add function _combine_plugin_doc.
* added timeout options to adhoc and console
* added test
* fix typosesz
* fix conflict
* task_timeout
* fix timeout option, added extra vars to console
* actually use right cli switch .. DUH!
* added timeout to include 'valid' but ignored keys
* fix default
* fixes per review
* fixes for collection playbooks
- add fqcn invocation, also handles extensions
- bring import_playbook into new normal
- avoid adding collection playbooks to adjacency
- deal with extensions or lack of em
- added tests
- fix bugs with multiple playbooks supplied
- unicode/bytes/native fixes
- fix modulenotfound for < py3.7
- tests
* Add DarwinStrategy class and integration tests
macOS has three seprate hostname params that need to be set. One of those params, LocalHostName, has more stringent requirements than the other two, which accept special characters and spaces. Create a method to scrub the hostname to ensure it works well with the system requirements.
* Update documentation
* Account for virtualization type returned on Azure Pipelines
* Do not be dependent on order of self.name_types
Use the scrubbed name when the name type is LocalHostName
A bug existed in systemd 245 that did not properly handle unknown kernel
capabilities gracefully. This resulted in incomplete output when querying
for the service status. It is possible to get service status by other means.
This PR works around this issue by getting service status using other commands
in the event of a failure due to this bug.
* GitHub is removing the underlying API used to implement the `login` command. Since the general consensus seems to be that relatively nobody currently uses this command (in favor of explicit token passing), support was simply removed for interactive login. If a future need arises, this command should be reimplemented via OAuth Device Auth Grants.
* login or role login commands now produce a fatal error with a descriptive message
* updated 2.10 and 2.11 porting guide entries
* remove dead code/config, update messages and porting guides
* Fix the reversed order of return values
AnsibleModule.run_command returns a tuple of return code, stdout and stderr.
The module main function of the user module expects user.create_user to
return a tuple of return code, stdout and stderr.
Fix the locations where stdout and stderr got reversed.
* Fix another missed occurance
* Add changelog
* Fix further reversed order of return values occurrences
Followup to 72c3ba62c8:
Fix further reversed order of return values occurrences.
Change:
- containerd is now recognized as container tech
Test Plan:
- Tested with Docker inside Github Actions in a private repo:
"ansible_virtualization_role": "guest",
"ansible_virtualization_tech_guest": [
"containerd",
"container",
"VirtualPC"
],
"ansible_virtualization_tech_host": [],
"ansible_virtualization_type": "containerd",
Tickets:
- Fixes#66304 because this is what docker containers show up as
in Github Actions.
Signed-off-by: Rick Elrod <rick@elrod.me>
* Ensure we call action_loader.get with collection_list. Fixes#72170
* Add tests and changelog
* Remove grep, do assertion in playbook. ci_complete
* Skip old jinja2 versions
* ci_complete
* dedupe
make collections whitelist follow normal flow
* fixes missing set_options call and adhoc and stdout processing rules
* avoid dupes
* fixed to handle redirects
* also updated tests with new and more accurate skip message
* fix callback tests for envs with cowsay installed
* lots MOAR comments on why the code is as it is, some todos to refactor in future
Co-authored-by: Matt Davis <nitzmahone@users.noreply.github.com>
Change:
- When a plugin defines `type: str` on a parameter, treat more kinds of
input as a string instead of whatever it is parsed as.
Test Plan:
- New unit tests
- CI
Signed-off-by: Rick Elrod <rick@elrod.me>
* Add support for GSSAPI/Kerberos to urls.py
* Test out changes with the latest test container
* Get remote hosts working
* Fix up httptester_krb5_password reader
* Fix tests for opensuse and macOS
* Hopefully last lot of testing changes
* Dont do CBT on macOS
* Fixes from review
* Fix various sanity errors in plugins.
* Revert callback_type -> type transform.
* Undo paramiko_ssh 'connection' change, since this discrepancy is hardcoded in the plugin loader.
* Standardize on name and type (for callbacks).
* Fix existing author entries.
* Add 'Unknown (!UNKNOWN)' as author when author is missing.
* These are actually parsed as integers in the code.
* Revert "Add 'Unknown (!UNKNOWN)' as author when author is missing."
This reverts commit 29d1438aca.
Change:
- No longer fail due to old Fedora/RHEL and our failure to try to cast
gids to integers before trying to pass them to getgrgid() before
trying to use them.
- Add tests for user/mode for various unarchive formats.
Test Plan:
- New integration tests, ran against centos6 container
Tickets:
- Fixes#71903
Change:
- Make KubeVirt be identified as such, and with a guest role.
- This is a breaking change, since playbooks could be relying on the
incorrect behavior previously.
Test Plan:
- Issue author
Tickets:
- Fixes#72001
Signed-off-by: Rick Elrod <rick@elrod.me>
