PR #72591
This change:
* Adds an artifacts manager that abstracts away extracting the
metadata from artifacts, downloading and caching them in a
temporary location.
* Adds `resolvelib` to direct ansible-core dependencies[0].
* Implements a `resolvelib`-based dependency resolver for
`collection` subcommands that replaces the legacy
in-house code.
This is a dependency resolution library that pip 20.3+ uses
by default. It's now integrated for use for the collection
dependency resolution in ansible-galaxy CLI.
* Refactors of the `ansible-galaxy collection` CLI.
In particular, it:
- reimplements most of the `download`, `install`, `list` and
`verify` subcommands from scratch;
- reuses helper bits previously moved out into external modules;
- replaces the old in-house resolver with a more clear
implementation based on the resolvelib library[0][1][2].
* Adds a multi Galaxy API proxy layer that abstracts accessing the
version and dependencies via API or local artifacts manager.
* Makes `GalaxyAPI` instances sortable.
* Adds string representation methods to `GalaxyAPI`.
* Adds dev representation to `GalaxyAPI`.
* Removes unnecessary integration and unit tests.
* Aligns the tests with the new expectations.
* Adds more tests, integration ones in particular.
[0]: https://pypi.org/p/resolvelib
[1]: https://github.com/sarugaku/resolvelib
[2]: https://pradyunsg.me/blog/2020/03/27/pip-resolver-testing
Co-Authored-By: Jordan Borean <jborean93@gmail.com>
Co-Authored-By: Matt Clay <matt@mystile.com>
Co-Authored-By: Sam Doran <sdoran@redhat.com>
Co-Authored-By: Sloane Hertel <shertel@redhat.com>
Co-Authored-By: Sviatoslav Sydorenko <webknjaz@redhat.com>
Signed-Off-By: Sviatoslav Sydorenko <webknjaz@redhat.com>
Change:
- Initial set of changes for renaming to ansible-core
- Includes changelog fragment changes from base -> core
- Does NOT include docs changes
- Modifies detection stuff in setup.py to support ansible<2.9 and ansible-base
Test Plan:
- ci_complete
* Build documentation for Ansible-2.10 (formerly known as ACD).
Builds plugin docs from collections whose source is on galaxy
The new command downloads collections from galaxy, then finds the
plugins inside of them to get the documentation for those plugins.
* Update the python syntax checks
* docs builds can now require python 3.6+.
* Move plugin formatter code out to an external tool, antsibull-docs.
Collection owners want to be able to extract docs for their own
websites as well.
* The jinja2 filters, tests, and other support code have moved to antsibull
* Remove document_plugins as that has now been integrated into antsibull-docs
* Cleanup and bugfix to other build script code:
* The Commands class needed to have its metaclass set for abstractmethod
to work correctly
* Fix lint issues in some command plugins
* Add the docs/docsite/rst/collections to .gitignore as
everything in that directory will be generated so we don't want any of
it saved in the git repository
* gitignore the build dir and remove edit docs link on module pages
* Add docs/rst/collections as a directory to remove on make clean
* Split the collections docs from the main docs
* remove version and edit on github
* remove version banner for just collections
* clarify examples need collection keyword defined
* Remove references to plugin documentation locations that no longer exist.
* Perhaps the pages in plugins/*.rst should be deprecated
altogether and their content moved?
* If not, perhaps we want to rephrase and link into the collection
documentation?
* Or perhaps we want to link to the plugins which are present in
collections/ansible/builtin?
* Remove PYTHONPATH from the build-ansible calls
One of the design goals of the build-ansible.py script was for it to
automatically set its library path to include the checkout of ansible
and the library of code to implement itself. Because it automatically
includes the checkout of ansible, we don't need to set PYTHONPATH in
the Makefile any longer.
* Create a command to only build ansible-base plugin docs
* When building docs for devel, only build the ansible-base docs for
now. This is because antsibull needs support for building a "devel
tree" of docs. This can be changed once that is implemented
* When building docs for the sanity tests, only build the ansible-base
plugin docs for now. Those are the docs which are in this repo so
that seems appropriate for now.
Change:
- Update bundled six to 1.13 (last with py2.6 support)
- Make it pass lint
- Fix check to allow skipping over compat __init__.py files we authored
- Fix check to allow files that can't be updated for some reason
Test Plan:
- ansible-test sanity --docker
- CI
Signed-off-by: Rick Elrod <rick@elrod.me>
Replace the ansible-base changelog linting and generation tool with antsibull-changelog and make it available for linting collections. Previously changelog linting was limited to ansible-base.
Some code-smell sanity tests for ansible-base use subprocess to invoke ansible commands.
Intercept these commands to make sure the correct script and python version are used.
* Split out sanity test requirements.
* Run each --venv test separately.
This provides verification that the requirements for each test are properly specified.
* Use a separate requirements file per sanity test.
* Skip setuptools/cryptography setup for sanity.
* Eliminate pyyaml missing warning.
* Eliminate more pip noise.
* Fix conflicting generate_pip_install commands.
* Add changelog fragment.
* `meta/` directory in collections
* runtime metadata for redirection/deprecation/removal of plugin loads
* a compatibility layer to keep existing content working on ansible-base + collections
* a Python import redirection layer to keep collections-hosted (and otherwise moved) content importable by things that don't know better
* supported Ansible version validation on collection loads
Left hand side slicing is confusing and slower but maybe more memory
efficient in some circumstances. There is one case where it adds to
code safety: when it's used to substitute a different list in place of a
slice of the original list and the original list could have been bound
to a different variable in some other code. (The most likely case of
this is when it's a global variable and some other code might import
that variable name).
Because of the confusion factor we think it should only be used for the
safety case or where it's been benchmarked and shown to have some sort
of documentatble improvement. At the moment, only one piece of code
falls into those categories so this PR removes all the other instances
of left hand side slicing.
* remove azure extras and extras_require support
* Since Azure will be collectionized, the requirements will float more frequently than Ansible releases; the Azure collection needs to host the requirements now.
* Removed the dynamic extras support as well, since Azure was the only thing using it. If we need it again, it's easy to pull back from history.
* Mark azure-requirements as orhpaned.
This keeps the docs around so that existing links from old test runs remain valid.
Co-authored-by: Matt Clay <matt@mystile.com>
* BOTMETA support migrated_to
Allow BOTMETA to define if this part of the codebase has moved
into a Collection on Galaxy.
See also https://github.com/ansible/ansible/pull/63935
* Enforce migrated_to URL format
* pep8
Add a list of previously used release names to make it easy to tell what
release names are no longer usable.
Add a test that new release names have been added to the used list.
Fixes#61616
* Remove pointless sanity tests on `bin` dir.
* Update action-plugin-docs test for collections.
* Update no-main-display test for collections.
* Update empty-init test for collections.
* Update no-assert test for collections.
* Move required-and-default-attributes test.
This test only applies to Ansible itself.
* Update use-argspec-type-path test for collections.
* Relocate ansible-test self tests outside package.
We don't want to include the tests for verifying ansible-test within the ansible-test package.
* Add `test/ansible_test/` to classification.py.
* Fix test invocation.
* Relocate tests in MANIFEST.in.
* Improve package-data sanity test error checking.
* Only use includes for ansible-test in MANIFEST.in.
* Improve readability of MANIFEST.in.
* Run no-unwanted-files sanity test only on Ansible.
Since collections should be able to use binary modules there is not really any limit on what could exist in a collection `plugins` directory.
* Add support for symlinks in sanity target lists:
- Sanity tests that need to analyze symlinks can do so using the supplied target list.
- Tests that analyze directories will now only look at symlinks if requested.
- Directory symlinks will now be seen as directories instead of files.
* Enable symlinks on filename based sanity tests.
Sanity tests that evalulate filenames instead of content should include symlinks.
* Update symlinks sanity test.
Use the sanity test target list now that it can include symlinks.
* Install ansible-test
Modify the install script to install ansible-test and its supporting
code. Alternative to #60701 that doesn't change package_dir ansible for
fear that it might regress https://github.com/ansible/ansible/issues/10437
Also:
* No longer use package_data. Everything in the package dirs is going
to be installed. Anything that shouldn't be installed needs to be
moved elsewhere.
* modify the algorithm to store symlinks which are in the same tree
instead of same directory
* Add ansible_test files to package-data sanity test
* MANIFEST.in cleanups
* Add lib/ansible/config/*.yml
* Make most things in code directories (lib/ansible and test/lib/ansible_test/)
use explicit file extensions instead of wildcards for maintainability
* Exclude common file extensions that we don't want included in the code
directories
* Change package-data test to be more complete
* Now compares the repository, sdist, and install
* Compares both that everything in the sdist is in the repo and
everything in the install is in the sdist in addition to comparing that
everything in the repo that we want is in the install
* Leave out test artifacts
Only include the directory structure for test/results and test/cache not
any files that may have been generated by test runs
Remove test/utils files from the sdist as these are only needed for our CI
cleanup of docs in MANIFEST.in; getting rid of build files.
* Add the ability to output sdist and snapshot to specific directory
* Add a warning about modifying the heuristic to setup.py
* Address generated files
* Use make snapshot instead of sdist to generate changelog and man pages
and make sure they're included
* Ignore both the test/utils and generated test files (results, cache)
* Deal with Python3 __pycache__ byte code caches
* Don't check documentation, that isn't built for the sdist
* Restructure for clarity
* Add cli web docs to make clean
This was causing problems when attempting to test that the sdist didn't
have extra files
* Fix bug constructing python names from __pycache__ names
* Create a clean repo to work from
* Exclude test/legacy and be more explicit on extensions
* Exclude the legacy directory from sdist
This will help prevent accidental merging of content to recently obsoleted directories when adding new files.
It may also help contributors who have modified obsolete files understand where their changes should now be made.
* Relocate ansible-only sanity tests.
* Get "code smell" sanity tests from multiple dirs.
- `test/lib/ansible_test/_data/sanity/code-smell/` - General purpose tests used for both Ansible and Ansible Collections.
- `test/sanity/code-smell/` - Tests specific to Ansible, will not be used for Ansible Collections.
* Allow sanity tests to easily ignore themselves.
Useful for simple regex based sanity tests that error on their own script due to the regex or error message generated.
* Simplify no-smart-quotes sanity test.
Pruning of unversioned directories is handled by ansible-test.
* Remove directory pruning from boilerplate tests.
Files not passing these tests should be ignored instead.
* Add missing sanity ignore entries.
* Add sanity test to ensure all non-py files are installed
* Fix mode and regex
* Fix role skel inventory package_data
* Add docs
* Update package_data for inventory files
* Address pylint concerns
* Another tweak to package_data
* Address review feedback
* Change index to 1
* add to ansible-only.txt
* Initial ansible-test support for collections.
* Include cloud config in delegation payload.
* Add missing types import and fix `t` shadowing.
* Fix plugin traceback when config_path not set.
* Fix encoding issues.
* Remove unused imports.
* More encoding fixes.
* Handle delegation outside exception handler.
* Inject ssh keys only if not already in place.
* More defensive approach to getting remote pwd.
* Add missing string format var.
* Correct PowerShell require regex.
* Rename `is_install` and `INSTALL_ROOT`.
* Generate galaxy.yml based on single source of truth
* Fix up tests and align file names
* Minor Makefile tweak
* Remove link in galaxy.yml file and make it a template file
* Moved collections docs to dev_guide
* change Makefile clean path
* Added readme to example meta file
* review fixes
* Use newer style for doc generation script
* Fix mistake in dev_guide index
* removed uneeded file, fixed links and added preview banner
* Moved banner for sanity test
