Commit graph

10826 commits

Author SHA1 Message Date
Alexander Sowitzki
218f5c3648
user - Prevent user fact lookup failure if LOGNAME is set (#17029) (#73439)
The `UserFactCollector` queries the user login name via
`getpass.getuser()` and looks up the corresponding entry
in the password database.
The login name may differ from the actual user name,
eg. if the `LOGNAME` env variable is set. The lookup
fails in this case. Added a fallback in this case that
tries to get the entry via the user ID.
2021-02-02 10:36:02 -05:00
Rick Elrod
bce01655cc
[ansible-test] Bump CentOS 6 image (yum repos) (#73446)
Change:
- Bump centos6 image version to one which includes multiple fallbacks
  for vault.centos.org content.

Signed-off-by: Rick Elrod <rick@elrod.me>
2021-02-01 16:24:21 -06:00
Sloane Hertel
840bdc1e10
Fix warning for nonexistent inventory cache (#72840)
* Fix inventory cache warning by checking if the key exists before loading it
* changelog
2021-02-01 14:29:36 -05:00
Matt Clay
81cd8e46f2 Update ansible-test distro containers to 1.29.0. 2021-02-01 09:24:28 -08:00
Sloane Hertel
728dafb6eb
Make ansible-galaxy collection verify error out on no MANIFEST.json
PR #73403

Co-Authored-By: Sviatoslav Sydorenko <webknjaz@redhat.com>
2021-02-01 09:55:21 +01:00
Sviatoslav Sydorenko
6f4b4c345b
Match user-requested transitive pre-releases in collection dependency resolver (#73416)
* Add direct+transitive pre-release regression test

* Match user-requested transitive pre-releases

This change makes sure that in scenario when a user requests
collectionA-pre and collectionB that depends on collectionA,
collectionA-pre would actually match that collectionA requirement
while `--pre` is unset.

Co-authored-by: Jordan Borean <jborean93@gmail.com>
2021-02-01 14:26:49 +10:00
Matt Clay
c7cb944315 Always use python exec wrapper in ansible-test. 2021-01-28 15:59:55 -08:00
Matt Clay
902c402b6f Update ansible-test default containers. 2021-01-28 09:58:54 -08:00
Matt Clay
ded2d0b90e Use sh for ansible-test raw remote shell. 2021-01-28 09:00:55 -08:00
Matt Clay
b1344a1847 Avoid sphinx usage in changelog sanity test. 2021-01-27 23:52:13 -08:00
Matt Clay
418b9b781d No longer warn about missing pip/python pair.
This warning was intended to help improve test environment configuration.
However, it has ended up producing mostly non-actionable warning noise instead.

In most situations a missing pip or python should result in test failures.
Reporting a missing pip also implies that it should be used by tests, which is not the case.
Tests should be invoking pip as a python module with the appropriate python interpreter instead.
2021-01-27 23:51:48 -08:00
Matt Clay
672941fb0d
Add test support for Python 3.8 on RHEL and FreeBSD. (#73393)
* Support Python 3.8 on RHEL 8.2+ in ansible-test.

* Support Python 3.8 on FreeBSD in ansible-test.

* Use libyaml with PyYAML on FreeBSD.
2021-01-27 21:35:29 -08:00
Matt Clay
1b157ef42f
Limit rstcheck sanity test to ansible-core. (#73391) 2021-01-27 21:06:00 -08:00
Brian Coca
5aa4295d74
change detection and check mode fixes for apt_key (#73334)
Change detection and check mode fixes for apt_key

* allow apt-key to use proxies
* add note about deprecation of apt-key itself
* expanded error msgs
* show all keys
* fix short_format parsing
* added more return info and documented it
2021-01-27 18:40:58 -05:00
Sviatoslav Sydorenko
595413d113
Replace the inhouse collection dependency resolver with resolvelib
PR #72591

This change:

  * Adds an artifacts manager that abstracts away extracting the
    metadata from artifacts, downloading and caching them in a
    temporary location.

  * Adds `resolvelib` to direct ansible-core dependencies[0].

  * Implements a `resolvelib`-based dependency resolver for
    `collection` subcommands that replaces the legacy
    in-house code.

    This is a dependency resolution library that pip 20.3+ uses
    by default. It's now integrated for use for the collection
    dependency resolution in ansible-galaxy CLI.

  * Refactors of the `ansible-galaxy collection` CLI.
    In particular, it:

      - reimplements most of the `download`, `install`, `list` and
        `verify` subcommands from scratch;

      - reuses helper bits previously moved out into external modules;

      - replaces the old in-house resolver with a more clear
        implementation based on the resolvelib library[0][1][2].

  * Adds a multi Galaxy API proxy layer that abstracts accessing the
    version and dependencies via API or local artifacts manager.

  * Makes `GalaxyAPI` instances sortable.

  * Adds string representation methods to `GalaxyAPI`.

  * Adds dev representation to `GalaxyAPI`.

  * Removes unnecessary integration and unit tests.

  * Aligns the tests with the new expectations.

  * Adds more tests, integration ones in particular.

[0]: https://pypi.org/p/resolvelib
[1]: https://github.com/sarugaku/resolvelib
[2]: https://pradyunsg.me/blog/2020/03/27/pip-resolver-testing

Co-Authored-By: Jordan Borean <jborean93@gmail.com>
Co-Authored-By: Matt Clay <matt@mystile.com>
Co-Authored-By: Sam Doran <sdoran@redhat.com>
Co-Authored-By: Sloane Hertel <shertel@redhat.com>
Co-Authored-By: Sviatoslav Sydorenko <webknjaz@redhat.com>

Signed-Off-By: Sviatoslav Sydorenko <webknjaz@redhat.com>
2021-01-27 22:23:22 +01:00
Brian Coca
c7d4acc12f
allow env to override unspecified unsafe_writes (#73282)
* allow env var for fallback value for unspecified unsafe_writes
2021-01-27 14:16:10 -05:00
Brian Coca
2b0cd2c13f
use correct executable and options in all cases (#73323)
Use correct ssh executable and options in all cases on connection plugin

  * Also nicer naming/comments

Co-authored-by: Sviatoslav Sydorenko <wk.cvs.github@sydorenko.org.ua>
2021-01-26 20:21:24 -05:00
Matt Clay
13d08d232c Rename ansible-test functions to match constants. 2021-01-26 14:24:16 -08:00
Matt Clay
73fadc5e97
Clean up code in ansible-test. (#73379)
* Relocate code to fix type dependencies.

* Fix missing and unused imports.

* Fix type hints.

* Suppress PyCharm false positives.

* Avoid shadowing `file` built-in.

* Use json.JSONEncoder directly instead of super().

This matches the recommended usage and avoids a PyCharm warning.

* Remove redundant regex escape.

* Remove redundant find_python call.

* Use tarfile.open directly.

* Add changelog fragment.
2021-01-26 14:02:08 -08:00
Martin Krizek
a2af8432f3
Local vars should have highest precedence in AnsibleJ2Vars (#72830)
Ability to add local variables into AnsibleJ2Vars was added in
18a9eff11f to fix #6653. Local variables
are added using ``AnsibleJ2Vars.add_locals()`` method when creating a
new context - typically when including/importing a template with
context. For that use case local template variables created using
``set`` should override variables from higher contexts - either from the
play or any parent template, or both; Jinja behaves the same way.

Also removes AnsibleJ2Vars.extras instance variable which is not used.

Also adds missing test for #6653.

Fixes #72262
Fixes #72615

ci_complete
2021-01-26 11:37:56 +01:00
Matt Clay
fc590aeb21 Use a versioned pip bootstrapper in ansible-test. 2021-01-25 12:31:11 -08:00
Matt Clay
484e4af4d0 Handle get-pip.py breaking change on Python 2.7. 2021-01-23 11:45:53 -08:00
Alexander Sowitzki
e6da544310 Let command always return stdout & stderr (#73004)
The `command` module does not return stdout & sterr when calling
a non existing executable or an unknown exception arises. This fix
lets the module return empty byte strings in those cases.
2021-01-22 08:40:53 +01:00
Rick Elrod
1934ca9a55
Add Ubuntu 20.04 to CI and ansible-test (#69161)
Change:
- Add Ubuntu 20.04 to CI now that venv is default instead of virtualenv in ansible-test.

Test Plan:
- CI

Tickets:
- Fixes #69203

Signed-off-by: Rick Elrod <rick@elrod.me>
2021-01-21 22:58:31 -06:00
Rick Elrod
4357a78130
[ansible-test] First attempt at freebsd/11.4 (#72655)
Change:
- Try CI against freebsd 11.4

Test Plan:
- ci_complete

Tickets:
- Fixes #48782

Signed-off-by: Rick Elrod <rick@elrod.me>
2021-01-21 17:12:18 -06:00
Alexander Sowitzki
fe792fdcd2
Vanquish rare container bootstrap failure (#73288)
The init script for the test container writes additional lines to
the .bashrc of the user. This was done via a `cat` multiline
instruction, which is implemented internally by writing a
temporary file to TMPDIR (/tmp in this case) first. Docker fails
to provide /tmp just after creation, which results in a race
condition that rarely makes the init fail. Changed the `cat`
statement to multiple `echo`s.

Co-authored-by: Matt Clay <matt@mystile.com>
2021-01-21 12:22:58 -05:00
Joshua Bayfield
823c72bcb5
Shadow input for encrypt_string by default unless asked (fixes #71618) (#73263)
* Shadow input for encrypt_string by default unless asked (fixes #71618)
2021-01-20 15:50:24 -05:00
Jordan Borean
bc60d8ccda
Galaxy - make versions list consistent across versions (#72932)
* Galaxy - make versions list consistent across versions

* Fix up unit tests
2021-01-20 21:19:29 +01:00
Sam Doran
e8d4b62b41
Fix YAML error message when error is at the end of the file (#73241)
* Fix YAML error message when error is at the end of the file

If a YAML file fails to load due to a syntax error in a file, or there is an error in the last line of a
file, PyYAML reports the last line number of the file as the index where the error occurred.

When reading the file lines, we use that index to the get the relevant line.  If the index value is out
of range, the relevant line is lost for error reporting.

Subtract one from the index value to avoid the IndexError in this specific scenario. It is possible
to still get an IndexError, which will be handled as it is currently.

* Update existing tests and add new tests
2021-01-19 10:07:36 -05:00
Brian Coca
30d93995dd
Use a pty for local connections (#73023)
* Use a pty for local connections

Fixes #38696

Co-authored-by: James Cammarata <jimi@sngx.net>
2021-01-18 16:02:04 -05:00
Brian Coca
1e27d4052a
fix inventory source parse error handling (#73160)
fixes #51025

added test cases
2021-01-18 09:33:51 -05:00
Brian Coca
1202dd000f
Allow restricting config values to enumerated list (#73162)
* Allow restricting config values to enumerated list
* dont document internal entries
* also ignore private defs for ansible-config
* remove invalid value from tests
* added porting entry
2021-01-14 15:11:30 -05:00
Sandra McCann
88763c7347
update rstcheck and sphinx versions in contraints (#73176)
* update rstceck and sphinx versions in contraints
* fix test error
2021-01-14 13:50:44 -06:00
Sloane Hertel
0c4de6839b
Add integration tests to ansible-galaxy-collection for 'ansible-galaxy collection verify' (#73229) 2021-01-14 11:49:53 -05:00
Sam Doran
df451636e7
facts - properly report virtualization facts for Linux guests on bhyve (#73204) 2021-01-14 10:53:03 -05:00
Sam Doran
0e6c334115
pause - adjust warning when run in background (#73182)
When the pause module is run in the background and seconds parameter is provided,
do not warn.

* Add tests
* Fix existing tests
  The test wasn't failing when it should have.
2021-01-14 09:35:39 -05:00
Rick Elrod
7f0eb7ad79
[facts] Differentiate CentOS vs CentOS Stream (#73034)
Change:
- On CentOS Stream, make distribution_release be "Stream"
- On CentOS Core, it continues to be "Core"
- Implement custom distribution file parser for CentOS, so we can look
  for "CentOS Linux" and "CentOS Stream"
- Two new fixtures introduced (CentOS Linux 8.1 and CentOS Stream 8)
- Removed two dicts from `Distribution` class that were seemingly not
  used anywhere.

Test Plan:
- ci_complete
- New test fixtures

Tickets:
- Fixes #73027

Signed-off-by: Rick Elrod <rick@elrod.me>
2021-01-13 17:54:04 -05:00
Matt Clay
4164cb26f0 Remove obsolete tower plugin from ansible-test. 2021-01-13 12:37:21 -08:00
Alexander Sowitzki
3e8c8cd536 Make test_adhoc succeed from within a git checkout
The test currently only expects the literal `ansible` followed
by a semver in the first output line of `ansible --version`.
When running from within a git checkout, additional information
like the currently checkout branch are attached, which lets
the test fail. This commit allows arbitrary information to
follow the semver.
2021-01-13 21:22:21 +01:00
Alexander Sowitzki
6459fbb7bc Make test_prepare_multipart succeed on non Debian distros
`test_prepare_multipart` fails in non Debian environments since
Debian installations map the file ending `.key` to the MIME type
`application/pgp-keys`, which is not IANA conformant. This commit
explicitly sets the corresponding file type to
`application/octet-stream` and adjusts the expected serialized
result.
2021-01-13 21:21:33 +01:00
Matt Clay
21f1811ddf
Cleanup provisioning code in ansible-test. (#73207)
* Remove unused code in ansible-test.

* Remove obsolete endpoint logic from ansible-test.

* Remove obsolete region selection in ansible-test.

* Remove obsolete port logic in ansible-test.

* Clean up ansible-test remote providers.
2021-01-12 16:18:28 -08:00
Sloane Hertel
9de2da8a7e
Add tests for merging and replacing vars from inventory sources (#73181) 2021-01-11 17:37:34 -05:00
Martin Zimmermann
be59caa59b
iptables: Added set module/match_set support (#72984) 2021-01-11 16:32:53 -05:00
Sam Doran
a7e834071c
Add macOS 11 to CI (#72622)
* Fix connection_paramiko_ssh test for macOS 11
* Update Azure Pipelines config
* Add changelog
2021-01-11 12:20:47 -05:00
Sam Doran
264e08f21a
user - properly handle password and password lock when used together (#73016)
Do the right thing on Linux when password lock and a password hash are provided by writing
out the password hash prepended by the appropriate lock string rather than using -U and -L.
This is the correct way to set and lock the account in one command.

On BSD, run separate commands as appropriate since locking and setting the password cannot 
be done in a single action.

FreeBSD requires running several commands to get the account in the desired state. As a result,
the rc, output, and error from all commands need to be combined and evaluated so an accurate
and complete summary can be given at the end of module execution.

* Improve integration tests to cover this scenario.
* Break up user integration tests into smaller files
* Properly lock account when creating a new account and password is supplied

* Simplify rc collection in FreeBSD class
  Since the _handle_lock() method was added, the rc would be set to None, which could make
  task change reporting incorrect. My first attempt to solve this used a set and was a bit too
  complicated. Simplify it my comparing the rc from _handle_lock() and the current value of rc.

* Improve the Linux password hash and locking behavior
  If password lock and hash are provided, set the hash and lock the account by using a password
  hash since -L cannot be used with -p.

* Ensure -U and -L are not combined with -p since they are mutually exclusive to usermod.

* Clarify password_lock behavior.
2021-01-11 11:42:35 -05:00
Felix Fontein
ae74d871a7
Document install_python_apt option of apt_repository (#72959)
* Update lib/ansible/modules/apt_repository.py
* Remove unnecessary ignored sanity tests
Co-authored-by: Felix Fontein <felix@fontein.de>
Co-authored-by: Alicia Cozine <879121+acozine@users.noreply.github.com>
Co-authored-by: Amin Vakil <info@aminvakil.com>
2021-01-08 11:30:23 -06:00
Matt Clay
456e9b7a33 Update pip integration test to use venv on py3. 2021-01-07 20:12:39 -08:00
Matt Clay
05ba49cbba ansible-test - No virtualenv install on Python 3
Tests should use the Python built-in ``venv`` module on Python 3 instead of the standalone ``virtualenv`` module.

On Python 2 the ``virtualenv`` module continues to be the only option.
The version installed is either the OS packaged version or the last release to support Python 2, which is version 16.7.10.
2021-01-07 20:12:39 -08:00
Rick Elrod
44ee04bd1f
[dnf] Make "remove" filtering closer to dnf CLI (#73033)
Change:
- Internally, use dnf.subject.Subject#get_best_query for state: absent
- Add a bunch of tests for removing packages, given a bunch of different
  pkg specs (nv, nvr, nvra, wildcard, etc.)

Test Plan:
- New tests
- Local experiments with DNF API via PDB.

Tickets:
- Fixes #72809

Signed-off-by: Rick Elrod <rick@elrod.me>
2021-01-07 11:32:06 -06:00
Matt Clay
77942acefc Refactor ansible-test cryptography install code. 2021-01-07 09:04:33 -08:00