Commit graph

17 commits

Author SHA1 Message Date
Sviatoslav Sydorenko
595413d113
Replace the inhouse collection dependency resolver with resolvelib
PR #72591

This change:

  * Adds an artifacts manager that abstracts away extracting the
    metadata from artifacts, downloading and caching them in a
    temporary location.

  * Adds `resolvelib` to direct ansible-core dependencies[0].

  * Implements a `resolvelib`-based dependency resolver for
    `collection` subcommands that replaces the legacy
    in-house code.

    This is a dependency resolution library that pip 20.3+ uses
    by default. It's now integrated for use for the collection
    dependency resolution in ansible-galaxy CLI.

  * Refactors of the `ansible-galaxy collection` CLI.
    In particular, it:

      - reimplements most of the `download`, `install`, `list` and
        `verify` subcommands from scratch;

      - reuses helper bits previously moved out into external modules;

      - replaces the old in-house resolver with a more clear
        implementation based on the resolvelib library[0][1][2].

  * Adds a multi Galaxy API proxy layer that abstracts accessing the
    version and dependencies via API or local artifacts manager.

  * Makes `GalaxyAPI` instances sortable.

  * Adds string representation methods to `GalaxyAPI`.

  * Adds dev representation to `GalaxyAPI`.

  * Removes unnecessary integration and unit tests.

  * Aligns the tests with the new expectations.

  * Adds more tests, integration ones in particular.

[0]: https://pypi.org/p/resolvelib
[1]: https://github.com/sarugaku/resolvelib
[2]: https://pradyunsg.me/blog/2020/03/27/pip-resolver-testing

Co-Authored-By: Jordan Borean <jborean93@gmail.com>
Co-Authored-By: Matt Clay <matt@mystile.com>
Co-Authored-By: Sam Doran <sdoran@redhat.com>
Co-Authored-By: Sloane Hertel <shertel@redhat.com>
Co-Authored-By: Sviatoslav Sydorenko <webknjaz@redhat.com>

Signed-Off-By: Sviatoslav Sydorenko <webknjaz@redhat.com>
2021-01-27 22:23:22 +01:00
Jordan Borean
d30fc6c0b3
galaxy - preserve symlinks on build/install (#69959)
* galaxy - preserve symlinks on build/install

* Handle directory symlinks

* py2 compat change

* Updated changelog fragment
2020-06-11 06:46:42 +10:00
Sloane Hertel
e40889e711
Add support to install collections from git repositories (#69154)
* Enable installing collections from git repositories

* Add tests for installing individual and multiple collections from git repositories

* Test to make sure recursive dependencies with different syntax are deduplicated

* Add documentation

* add a changelog

* Skip Python 2.6

* Only fail if no collections are located in a git repository

Add support for a 'type' key for collections in requirement.yml files.
Update the changelog and document the supported keys and allowed values for the type.

Add a note that the collection(s) in the repo must contain a galaxy.yml

* Add a warning about embedding credentials in SCM URLs

* Update with review suggestions

* suppress sanity compile failure for Python 2.6
2020-05-29 13:33:32 -04:00
Jordan Borean
ecea15c508
Unify ansible-galaxy install -r (#67843)
* Unify ansible-galaxy install -r

* Minor nit fixes for docs

* Re-align warnings

* Fix up integration test

* Fix up test where no roles/collections were in file
2020-05-19 05:09:42 +10:00
Sloane Hertel
343ffaa18b
better error for "ansible-galaxy collection verify" if there's no MANIFEST.json (#67498)
* Add a better error for "ansible-galaxy verify" if the MANIFEST.json has been deleted from the installed collection or if the collection hasn't been installed via normal means

* Fix unit tests for the remote collection

If there's something wrong with the local collection's version it will fail before the remote collection is sought

* Add a test for the new error msg

* Prevent the duplicate warning

Mock the new isfile call where needed

* Update lib/ansible/galaxy/collection.py

Co-Authored-By: Martin Krizek <martin.krizek@gmail.com>

Co-authored-by: Martin Krizek <martin.krizek@gmail.com>
2020-05-14 12:25:19 -04:00
Jordan Borean
a20a527014
ansible-galaxy - Fix tar path traversal issue during install - CVE-2020-10691 (#68596) 2020-04-01 06:39:02 +10:00
Sloane Hertel
97f011cf94
Add verify subcommand to 'ansible-galaxy collection' (#65618)
* [WIP] Add verify subcommand command to 'ansible-galaxy collection'

* Fix pep8 and use consistent display order

* WIP - docs

* Remove some redundancy in verify display messages by using an error queue for each collection

* Share common code and improve output format

* clean up documentation

* typo

* Apply suggestions from code review

Co-Authored-By: Sandra McCann <samccann@redhat.com>

* Move ModifiedContent namedtuple to the global scope

Add a public metadata property

Rename function to _get_json_from_tar_file

* Add some unit tests

* fix using common functions after rebase

* changelog

* Improve efficiency finding specific installed collections

Improve efficiency by only downloading the tar.gz from the galaxy server for comparison after checking that the collection has been installed

Handle multiple collection paths

Fix up tests

* pep8

* reword that for accuracy

* use more common code and verify collection name

* Improve error message and add documentation

* Update unit tests and add coverage for both local and remote collections that are unable to be located

* Only validate collections using the format collection_namespace.collection_name

Fix tests to reflect that

Fix documentation

* Fix a bug when a local collection does not contain a file

* pep8

* Fix formatting

Co-authored-by: Sandra McCann <samccann@redhat.com>
2020-02-14 08:57:04 -06:00
Sam Doran
9a51dff0b1
Rename private function (#66577)
Make find_existing_collections() a public function since it is being used across mulitple files now to support listing and verifying collections
2020-01-17 15:27:15 -05:00
Jordan Borean
f8f7662850
Add the ability to ignore files and collection build (#64688) 2019-11-14 05:02:58 +10:00
Toshio Kuratomi
4cad7e479c
Galaxy publish fix (#63580)
* Handle galaxy v2/v3 API diffs for artifact publish response

For publishing a collection artifact
(POST /v3/collections/artifacts/), the response
format is different between v2 and v3.

For v2 galaxy, the 'task' url returned is
a full url with scheme:

        {"task": "https://galaxy-dev.ansible.com/api/v2/collection-imports/35573/"}

For v3 galaxy, the task url is relative:

        {"task": "/api/automation-hub/v3/imports/collections/838d1308-a8f4-402c-95cb-7823f3806cd8/"}

So check which API we are using and update the task url approriately.

* Use full url for all wait_for_import messages

Update unit tests to parameterize the expected
responses and urls.

* update explanatory comment

* Rename n_url to full_url.

* Fix issue with overwrite of the complete path

* Fixes overwrite of the complete path in case there's extra path stored
  in self.api_sever
* Normalizes the input to the wait_import_task function so it receives
  the same value on both v2 and v3

Builds on #63523

* Update unittests for new call signature

* Add changelog for ansible-galaxy publish API fixes.
2019-10-16 15:23:12 -07:00
Jordan Borean
a7fd6e99d9
Refactor galaxy collection API for v3 support (#61510)
* Refactor galaxy collection API for v3 support

* Added unit tests for GalaxyAPI and starting to fix other failures

* finalise tests

* more unit test fixes
2019-08-30 11:55:19 +10:00
Adrian Likins
af01cb114c
Support galaxy v3/autohub API in ansible-galaxy (#60982)
* Add galaxy collections API v3 support

Issue: ansible/galaxy-dev#60

- Determine if server supports v3

Use 'available_versions' from `GET /api`
to determine if 'v3' api is available on
the server.

- Support v3 pagination style

ie, 'limit/offset style', with the paginated
responses based on https://jsonapi.org/format/#fetching-pagination

v2 galaxy uses pagination that is more or less
'django rest framework style' or 'page/page_size style',
based on the default drf pagination described
at https://www.django-rest-framework.org/api-guide/pagination/#pagenumberpagination

- Support galaxy v3 style error response

The error objects returned by the galaxy v3 api are based
on the JSONAPI response/errors format
(https://jsonapi.org/format/#errors).

This handles that style response. At least for publish_collection
for now. Needs extracting/generalizing.

Handle HTTPError in CollectionRequirement.from_name()
with _handle_http_error(). It will raise AnsibleError
based on the json in an error response.

- Update unit tests

update test/unit/galaxy/test_collection*
to paramaterize calls to test against
mocked v2 and v3 servers apis.

Update artifacts_versions_json() to tale an
api version paramater.

Add error_json() for generating v3/v3 style error
responses.

So now, the urls generated and the pagination schema
of the response will use the v3 version if
the passed in GalaxyAPI 'galaxy_api' instance
has 'v3' in it's available_api_versions

* Move checking of server avail versions to collections.py

collections.py needs to know the server api versions
supported before it makes collection related calls,
so the 'lazy' server version check in api.GalaxyAPI
is never called and isn't set, so 'v3' servers weren't
found.

Update unit tests to mock the return value of the
request instead of GalaxyAPI itself.
2019-08-28 16:59:34 -04:00
Jordan Borean
e04b2a9697
ansible-galaxy - Add timeout and progress indicator for publish and install (#60660)
* ansible-galaxy - Add timeout and progress indicator for publish

* add progress indicator to install phase as well
2019-08-23 06:27:28 +10:00
Jordan Borean
e747487720
ansible-galaxy - define multiple galaxy instances in ansible.cfg (#60553)
* ansible-galaxy: support multiple servers on install

* Added docs for the server configuration file

* Fix up doc string for requirements file format

* Fix bugs after testing

* Fix kwarg doc and added version

* Fix typo and doc improvement

* Fix base64 encoding and allow --server to override list
2019-08-21 07:49:05 +10:00
Jordan Borean
aa0de421d2
Ignore collection build release files in the root collection directory (#59121) 2019-07-23 07:52:30 +10:00
Jordan Borean
65049620ee
Generate galaxy.yml based on single source of truth (#59170)
* Generate galaxy.yml based on single source of truth

* Fix up tests and align file names

* Minor Makefile tweak

* Remove link in galaxy.yml file and make it a template file

* Moved collections docs to dev_guide

* change Makefile clean path

* Added readme to example meta file

* review fixes

* Use newer style for doc generation script

* Fix mistake in dev_guide index

* removed uneeded file, fixed links and added preview banner

* Moved banner for sanity test
2019-07-23 06:50:46 +10:00
Jordan Borean
b6791e6ae3
ansible-galaxy: add collection sub command (#57106)
* ansible-galaxy: add collection init sub command

* Fix changelog and other sanity issues

* Slim down skeleton structure, fix encoding issue on template

* Fix doc generation code to include sub commands

* Added build step

* Tidy up the build action

* Fixed up doc changes and slight testing tweaks

* Re-organise tests to use pytest

* Added publish step and fixed up issues after working with Galaxy

* Unit test improvments

* Fix unit test on 3.5

* Add remaining build tests

* Test fixes, make the integration tests clearer to debug on failures

* Removed unicode name tests until I've got further clarification

* Added publish unit tests

* Change expected length value

* Added collection install steps, tests forthcoming

* Added unit tests for collection install entrypoint

* Added some more tests for collection install

* follow proper encoding rules and added more tests

* Add remaining tests

* tidied up tests and code based on review

* exclude pre-release versions from galaxy API
2019-07-10 05:47:25 +10:00