Commit graph

1347 commits

Author SHA1 Message Date
Matt Clay
1a2da990a4
Temporary fix for cryptography issues. (#73530) 2021-02-08 14:35:00 -08:00
Matt Clay
1b157ef42f
Limit rstcheck sanity test to ansible-core. (#73391) 2021-01-27 21:06:00 -08:00
Sviatoslav Sydorenko
595413d113
Replace the inhouse collection dependency resolver with resolvelib
PR #72591

This change:

  * Adds an artifacts manager that abstracts away extracting the
    metadata from artifacts, downloading and caching them in a
    temporary location.

  * Adds `resolvelib` to direct ansible-core dependencies[0].

  * Implements a `resolvelib`-based dependency resolver for
    `collection` subcommands that replaces the legacy
    in-house code.

    This is a dependency resolution library that pip 20.3+ uses
    by default. It's now integrated for use for the collection
    dependency resolution in ansible-galaxy CLI.

  * Refactors of the `ansible-galaxy collection` CLI.
    In particular, it:

      - reimplements most of the `download`, `install`, `list` and
        `verify` subcommands from scratch;

      - reuses helper bits previously moved out into external modules;

      - replaces the old in-house resolver with a more clear
        implementation based on the resolvelib library[0][1][2].

  * Adds a multi Galaxy API proxy layer that abstracts accessing the
    version and dependencies via API or local artifacts manager.

  * Makes `GalaxyAPI` instances sortable.

  * Adds string representation methods to `GalaxyAPI`.

  * Adds dev representation to `GalaxyAPI`.

  * Removes unnecessary integration and unit tests.

  * Aligns the tests with the new expectations.

  * Adds more tests, integration ones in particular.

[0]: https://pypi.org/p/resolvelib
[1]: https://github.com/sarugaku/resolvelib
[2]: https://pradyunsg.me/blog/2020/03/27/pip-resolver-testing

Co-Authored-By: Jordan Borean <jborean93@gmail.com>
Co-Authored-By: Matt Clay <matt@mystile.com>
Co-Authored-By: Sam Doran <sdoran@redhat.com>
Co-Authored-By: Sloane Hertel <shertel@redhat.com>
Co-Authored-By: Sviatoslav Sydorenko <webknjaz@redhat.com>

Signed-Off-By: Sviatoslav Sydorenko <webknjaz@redhat.com>
2021-01-27 22:23:22 +01:00
Felix Fontein
ae74d871a7
Document install_python_apt option of apt_repository (#72959)
* Update lib/ansible/modules/apt_repository.py
* Remove unnecessary ignored sanity tests
Co-authored-by: Felix Fontein <felix@fontein.de>
Co-authored-by: Alicia Cozine <879121+acozine@users.noreply.github.com>
Co-authored-by: Amin Vakil <info@aminvakil.com>
2021-01-08 11:30:23 -06:00
Matt Clay
37d09f2488
Update ansible-test pylint Python support. (#72972)
* Update ansible-test pylint Python support.

* Python 3.8 is now officially supported.
* Python 3.9 is now skipped with a warning.
2020-12-15 10:27:32 -08:00
Felix Fontein
5226ac5778
The implicit default for type=bool options is not 'false', but 'none' (#72699)
* The implicit default for type=bool options is not 'false', but 'none'.

* Fix modules, resp. add ignore.txt entries.
2020-12-04 12:13:14 -05:00
Rick Elrod
6894ae7d1d
Rename to ansible-core (#72594)
Change:
- Initial set of changes for renaming to ansible-core
- Includes changelog fragment changes from base -> core
- Does NOT include docs changes
- Modifies detection stuff in setup.py to support ansible<2.9 and ansible-base

Test Plan:
- ci_complete
2020-11-13 15:42:10 -06:00
Mohammad Sadegh Dehghan
85570e2dc0
Fix documented type of mode field in copy module (#72593)
* Fix type of mode field in copy module

* Remove module type checking from ignore.txt sanity checks

* Remove mode type

Co-authored-by: Matt Martz <matt@sivel.net>
2020-11-12 16:28:01 -05:00
Brian Coca
8eab113cb1
show keyword documentation in ansible-doc (#72476)
* show keyword documentation in ansible-doc

Co-authored-by: Felix Fontein <felix@fontein.de>
2020-11-09 09:55:17 -05:00
Matt Martz
c8590c7482
Various intentional tests (#72485)
* Add tests for argspec choices type=list

* Add explicit interpreter discovery tests to validate modules returning ansible_facts still set interp

* Add explicit tests for missing_required_lib

* Add explicit tests for recursive_diff

* ci_complete ci_coverage

* Update data to cover more code/tests

* ci_complete ci_coverage

* Add argspec tests for aliases, and no_log

* Forgotten file

* ci_complete ci_coverage

* Add argspec tests for type int

* ci_complete ci_coverage

* Remove incidental_k8s

* ci_complete ci_coverage

* fix missing newline

* Remove incidental_sts_assume_role

* ci_complete ci_coverage
2020-11-06 08:41:41 -06:00
Matt Martz
880087748c
Remove incidental tower integration tests (#72461)
* Add explicit test for dict repr in argspec

* Add explicit test for undefined repr

* ci_complete ci_coverage

* Skip old jinja2

* ci_complete ci_coverage

* Remove incidental_tower_receive

* ci_complete ci_coverage

* Remove incidental_tower_credential_type

* ci_complete ci_coverage

* Remove ignore entries

* ci_complete ci_coverage
2020-11-04 09:46:08 -06:00
Jordan Borean
1ae3683d0c
Fix example to use correct shebang (#72129)
* Fix example to use correct shebang

* Fix other example modules as well

* Ignore shebang test
2020-10-07 10:07:27 -05:00
Matt Martz
9081b22868
Remove incidental_vyos_static_route (#72024)
* Add explicit tests for required_together suboptions

* ci_complete ci_coverage

* Remove incidental_vyos_static_route

* ci_complete ci_coverage

* Add explicit coverage of suboptions required_if

* ci_complete ci_coverage

* Remove incidental_vyos_logging

* ci_complete ci_coverage
2020-09-30 15:38:12 -05:00
Matt Martz
6f4aed5377
Remove incidental_nios_txt_record (#72009)
* Add explicit coverage of argspec type=dict

* Non string mapping failure

* ci_complete ci_coverage

* Remove incidental_nios_txt_record and associated files

* Don't forget the ignore.txt changes

* ci_complete ci_coverage
2020-09-30 08:56:27 -05:00
flowerysong
91a153434a
Fix sorting of ignore.txt (#71930)
Mostly out of order from directory flattening
2020-09-29 11:10:23 -04:00
Sviatoslav Sydorenko
ece0be30ab
Move ansible.galaxy.collection module into package
This change preserves how all the external imports refer to this code
while allowing us to start cutting the spaghetti into more easily
maintainable pieces.

This is a start of the upcoming refactoring effort destined to
eliminate tight coupling, implicit data manipulation, god objects,
abstraction leaks and other code smells.

Essentially, `ansible.galaxy.collection` is going to be a package that
holds parts of the collection management code spread across loosely
coupled modules.

PR: #71931
2020-09-25 16:15:10 +02:00
Matt Clay
92b66e3e31
CI provider fixes for ansible-test. (#71929)
* Make Azure Pipelines resource_prefix lowercase.

* Make classification of CI files consistent.

* Update package-data sanity test for AZP.
2020-09-24 17:34:07 -07:00
Amin Vakil
3e9943bc5e
Reduce ignored module sanity tests (#70319)
* Reorder test/sanity/ignore.txt to better see changes
* Remove extra empty line causing pass fail
* Apply suggestions made by Akasurde
* Minor fix in package_facts and remove two unnecessary ignores
* Fix subversion based on a suggestion made by felixfontein
* Apply suggestions made by felixfontein and Andersson007
* Fix subversion.py as suggested by felixfontein
* Minor reformatting in yum_repository description
* Reformat changelog
* Add key to apt_key, add deprecated changelog
* Add PR url to changelog
* Ignore paramater-type-not-in-doc in favour of adding key back to apt_key
* Fix apt_key
* Remove undocumented-paramater from apt_key ignore
* Ignore doc-choices-do-not-match-spec in package_facts
* Fix package_facts
* Fix filter option in setup module
2020-09-22 16:07:29 -04:00
Matt Martz
d63d880383
Remove incidental_synchronize tests (#71776)
* Remove incidental_synchronize tests

* Update sanity ignore
2020-09-17 09:31:47 -05:00
Jérôme Loyet
daca3ade99
Allow list of filters for the setup module (#68551)
The setup module can now filter out multiple pattern by providing a list
to the filter parameter instead of just a string. Single string sill
works. Previous behaviour remains.

(cherry picked from commit b5c36dac483fdd74d6c570d77cc8f3e396720366)
2020-09-09 20:40:48 +02:00
Jordan Borean
72a7cb4a2c
powershell - fix quoting values (#71411)
* powershell - fix quoting values

* Add ignore for smart quote skip
2020-08-26 07:06:51 +10:00
Abhijeet Kasurde
4f993922c8
Add documentation about info/facts module development (#71250)
Fixes: #40151

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2020-08-18 15:56:43 -04:00
Rick Elrod
4f54fb8d43
Remove incidental azure tests (#71164)
Change:
- All incidental coverage from these tests has moved to intentional
  coverage.

Test Plan:
- CI

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-08-08 00:56:53 -05:00
Toshio Kuratomi
9dda393d70
Collections docs generation (#59761)
* Build documentation for Ansible-2.10 (formerly known as ACD).

Builds plugin docs from collections whose source is on galaxy

The new command downloads collections from galaxy, then finds the
plugins inside of them to get the documentation for those plugins.

* Update the python syntax checks
  * docs builds can now require python 3.6+.

* Move plugin formatter code out to an external tool, antsibull-docs.
  Collection owners want to be able to extract docs for their own
  websites as well.
* The jinja2 filters, tests, and other support code have moved to antsibull
* Remove document_plugins as that has now been integrated into antsibull-docs

* Cleanup and bugfix to other build script code:
  * The Commands class needed to have its metaclass set for abstractmethod
    to work correctly
  * Fix lint issues in some command plugins

* Add the docs/docsite/rst/collections to .gitignore as
  everything in that directory will be generated so we don't want any of
  it saved in the git repository
* gitignore the build dir and remove edit docs link on module pages

* Add docs/rst/collections as a directory to remove on make clean
* Split the collections docs from the main docs

* remove version and edit on github
* remove version banner for just collections
* clarify examples need collection keyword defined

* Remove references to plugin documentation locations that no longer exist.
  * Perhaps the pages in plugins/*.rst should be deprecated
    altogether and their content moved?
  * If not, perhaps we want to rephrase and link into the collection
    documentation?
  * Or perhaps we want to link to the plugins which are present in
    collections/ansible/builtin?

* Remove PYTHONPATH from the build-ansible calls
  One of the design goals of the build-ansible.py script was for it to
  automatically set its library path to include the checkout of ansible
  and the library of code to implement itself.  Because it automatically
  includes the checkout of ansible, we don't need to set PYTHONPATH in
  the Makefile any longer.

* Create a command to only build ansible-base plugin docs
  * When building docs for devel, only build the ansible-base docs for
    now.  This is because antsibull needs support for building a "devel
    tree" of docs.  This can be changed once that is implemented
  * When building docs for the sanity tests, only build the ansible-base
    plugin docs for now.  Those are the docs which are in this repo so
    that seems appropriate for now.
2020-07-17 13:07:35 -07:00
Rick Elrod
1b4fd23ba6
csvfile: use parse_kv() for args, add tests (#70550)
Change:
- Use parse_kv() for parsing in the csvfile lookup plugin. This allows
  us to handle multi-word search keys and filenames. Previously, the
  plugin split on space and so none of these things worked as expected.
- Add integration tests for csvfile, testing a plethora of weird cases.

Test Plan:
- New integration tests, CI

Tickets:
- Fixes #70545

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-07-10 16:21:03 -05:00
Amin Vakil
9c40b1b2ff
Replace filename with file in apt_key (#70492)
* Replace filename with file from apt_key check

  one is internal variable, the other is actual parameter used and required for parameter check.
2020-07-10 12:42:49 -04:00
Abhijeet Kasurde
f5037314e3
sanity: remove ansible-bad-function (#70431)
* Updated docs and ignore.txt
* Replaced with correct function calls

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2020-07-07 11:46:28 -04:00
Rick Elrod
1be78dbfc5
Updated bundled libraries (#70418)
Change:
- Update bundled six to 1.13 (last with py2.6 support)
- Make it pass lint
- Fix check to allow skipping over compat __init__.py files we authored
- Fix check to allow files that can't be updated for some reason

Test Plan:
- ansible-test sanity --docker
- CI

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-07-02 12:39:00 -05:00
Abhijeet Kasurde
74bedab8a9
known_hosts: update documentation (#70342)
* Update documentation as per sanity tests
* Added example about custom SSH port in example section

Fixes: #29236

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2020-06-30 12:03:31 -04:00
Felix Fontein
ebd20ddca6
Include changelogs/changelog.yaml in distribution. (#70260) 2020-06-24 11:38:20 -05:00
Rick Elrod
0073ab5fa6
Nuke strategy.SharedPluginLoaderObj, depr. 2.11 (#70235)
* Nuke strategy.SharedPluginLoaderObj, depr. 2.11

Change:
- Nuke SharedPluginLoaderObj class
- Update tests (which seemingly didn't use it anyway)
- Changelog

Test Plan:
CI, grep

Signed-off-by: Rick Elrod <rick@elrod.me>

* Nuke from ignore.txt

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-06-23 13:09:26 -04:00
Rick Elrod
7584e145a9
Nuke _get_item() from callback, deprecated 2.11 (#70233)
Change:
- Remove _get_item() alias as it has been deprecated
- Update tests
- Remove relevant sanity curtailment
- Add changelog

Test Plan:
CI, grep

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-06-23 12:03:27 -04:00
Rick Elrod
c9edb35652
Nuke systemd module user option, 2.11 deprecation (#70211)
Change:
Remove all references to the 'user' param in systemd module.

Test Plan:
CI and grep.

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-06-23 11:55:01 -04:00
Rick Elrod
9cfc76a221
Nuke with_ squashing, deprecated for 2.11 (#70209)
Change:
Removes with_* loop squashing and tests for 2.11

Test Plan:
CI, and grepped for with_items in package manager integration targets.
There might be some test cases in collections which need to stop testing
this behavior.

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-06-23 11:53:25 -04:00
Matt Clay
fa48678a08
Rename pylint plugin and add tests. (#70225)
* Renamed custom pylint plugin for unwanted names.
* Add integration tests for sanity test failures.
2020-06-22 20:25:35 -07:00
Matt Clay
4816bb4f43
More boilerplate fixes. (#70224)
* Fix boilerplate in hacking dir.
* Fix boilerplate in docs dir.
* Fix boilerplate in integration tests.
* Fix boilerplate in examples.
2020-06-22 19:05:30 -07:00
Matt Clay
234994fc07 Fix boilerplate in setup.py and lib/ansible/ dir. 2020-06-22 16:29:16 -07:00
Matt Clay
98a0995fd0 Clean up unit test boilerplate. 2020-06-22 14:20:33 -07:00
Abhijeet Kasurde
87406890cf
find: Specify elements type in paths (#70139)
Fixes: #35990

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2020-06-19 06:56:02 +05:30
Rick Elrod
cc071cdea6
Bump devel to 2.11.0 (#70121)
Signed-off-by: Rick Elrod <rick@elrod.me>
2020-06-17 14:22:38 -05:00
Felix Fontein
a114da80ee
Use antsibull-changelog instead of packaged changelog generator (#69313)
Replace the ansible-base changelog linting and generation tool with antsibull-changelog and make it available for linting collections. Previously changelog linting was limited to ansible-base.
2020-06-11 11:21:21 -07:00
Matt Clay
6c2be04b6f Intercept code-smell tests using subprocess.
Some code-smell sanity tests for ansible-base use subprocess to invoke ansible commands.
Intercept these commands to make sure the correct script and python version are used.
2020-06-10 16:07:47 -07:00
Jordan Borean
a58fcde3a0
Use common ps sanity requirements file (#69992)
* Use common pssanity requirements file

* Fix up sanity ignore
2020-06-11 06:38:17 +10:00
Sloane Hertel
51f6d129cb
support hard coded module_defaults.yml groups for collections (#69919)
* Only allow groups which were hardcoded in module_defaults.yml

only load action groups from the collection if module_defaults contains a potential group for the action

* Fix tests using modules that override those whitelisted in lib/ansible/config/module_defaults.yml

Third party modules should not be using group/ - use the action name instead

* add externalized module_defaults tests

add the missing group and collections

ci_complete

Co-authored-by: Matt Davis <mrd@redhat.com>

* changelog

ci_complete

* Fix import in tests

ci_complete

* Update with requested changes

ci_complete

* don't traceback since we don't validate the contents of module_defaults

ci_complete

Co-authored-by: Matt Davis <mrd@redhat.com>
2020-06-09 15:38:57 -07:00
Matt Clay
7bff3d312f
Split out and install sanity test requirements. (#69971)
* Split out sanity test requirements.

* Run each --venv test separately.

This provides verification that the requirements for each test are properly specified.

* Use a separate requirements file per sanity test.

* Skip setuptools/cryptography setup for sanity.

* Eliminate pyyaml missing warning.

* Eliminate more pip noise.

* Fix conflicting generate_pip_install commands.

* Add changelog fragment.
2020-06-09 13:38:36 -07:00
Rick Elrod
3b00c161dc
Re-add documentation for yum "install_repoquery" (#69871)
Change:
- This was removed in 2014 in 122a7021bc.
- The option still exists and is enabled by default and can lead to user
  confusion when people aren't expecting packages (or updated
  dependencies for it) to get installed and they do.
- Add the option documentation back with a few notes to make it clear what
  is happening.

Test Plan:
N/A, no code change, just documentation

Tickets:
- Refs #69497

Signed-off-by: Rick Elrod <rick@elrod.me>

* Fix sanity errors

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-06-04 15:45:22 -04:00
Sloane Hertel
e40889e711
Add support to install collections from git repositories (#69154)
* Enable installing collections from git repositories

* Add tests for installing individual and multiple collections from git repositories

* Test to make sure recursive dependencies with different syntax are deduplicated

* Add documentation

* add a changelog

* Skip Python 2.6

* Only fail if no collections are located in a git repository

Add support for a 'type' key for collections in requirement.yml files.
Update the changelog and document the supported keys and allowed values for the type.

Add a note that the collection(s) in the repo must contain a galaxy.yml

* Add a warning about embedding credentials in SCM URLs

* Update with review suggestions

* suppress sanity compile failure for Python 2.6
2020-05-29 13:33:32 -04:00
Jordan Borean
f5f3ba7ab5
ansible-test - fix up relative util import for powershell validate-modules (#69753)
* ansible-test - fix up relative util import for powershell validate-modules

* Use different tactic for generic group

* Use python 2 and 3
2020-05-29 14:31:59 +10:00
Toshio Kuratomi
412af6ae91
Fix docs return formatting. (#69632)
Gets rid of the unknown field names in the return data.
Allows the plugin return docs to format under the new docs pipeline.
* Expect that package_facts will pass return-syntax-error now.
2020-05-28 15:23:30 -05:00
Matt Davis
f7dfa817ae
collection routing (#67684)
* `meta/` directory in collections
* runtime metadata for redirection/deprecation/removal of plugin loads
* a compatibility layer to keep existing content working on ansible-base + collections
* a Python import redirection layer to keep collections-hosted (and otherwise moved) content importable by things that don't know better
* supported Ansible version validation on collection loads
2020-05-26 09:42:06 -07:00