Commit graph

85 commits

Author SHA1 Message Date
Brian Coca
ddaa539ab1
Better handling of rstisms in ansible-doc (#74596)
* Better handling of rstisms

  replace tags more intelligently to make things more readable
  unit tests + minor adjustments
2021-05-11 11:31:20 -04:00
Matt Davis
d0e991e892
Clarify CLI version number as core version (#72287)
* clarify CLI version number as core version

* reduce confusion with `ansible` PyPI package >= 2.10 drifting from core version

* fix units
2021-03-01 11:14:56 -06:00
Sviatoslav Sydorenko
f327e65d11
Use iskeyword and str.isidentifier for "is FQCN" (#73279)
* Use valid FQCN in test_verbosity_arguments

* Use iskeyword and str.isidentifier for "is FQCN"
2021-02-11 18:23:57 -08:00
Shane McDonald
67f5bb39c7
galaxy - add format options for collection list (#73474)
* Include all collections in single json object / yaml document
* Add tests
* For galaxy list yaml/json output, use dictionary of dictionaries instead of list
* Add tests for listing single collection in yaml / output format
* --output -> --format
* Add explicit test for listing collection in human format
* Fix bug where empty json object was emitted + add test
2021-02-05 13:24:59 -05:00
Sviatoslav Sydorenko
595413d113
Replace the inhouse collection dependency resolver with resolvelib
PR #72591

This change:

  * Adds an artifacts manager that abstracts away extracting the
    metadata from artifacts, downloading and caching them in a
    temporary location.

  * Adds `resolvelib` to direct ansible-core dependencies[0].

  * Implements a `resolvelib`-based dependency resolver for
    `collection` subcommands that replaces the legacy
    in-house code.

    This is a dependency resolution library that pip 20.3+ uses
    by default. It's now integrated for use for the collection
    dependency resolution in ansible-galaxy CLI.

  * Refactors of the `ansible-galaxy collection` CLI.
    In particular, it:

      - reimplements most of the `download`, `install`, `list` and
        `verify` subcommands from scratch;

      - reuses helper bits previously moved out into external modules;

      - replaces the old in-house resolver with a more clear
        implementation based on the resolvelib library[0][1][2].

  * Adds a multi Galaxy API proxy layer that abstracts accessing the
    version and dependencies via API or local artifacts manager.

  * Makes `GalaxyAPI` instances sortable.

  * Adds string representation methods to `GalaxyAPI`.

  * Adds dev representation to `GalaxyAPI`.

  * Removes unnecessary integration and unit tests.

  * Aligns the tests with the new expectations.

  * Adds more tests, integration ones in particular.

[0]: https://pypi.org/p/resolvelib
[1]: https://github.com/sarugaku/resolvelib
[2]: https://pradyunsg.me/blog/2020/03/27/pip-resolver-testing

Co-Authored-By: Jordan Borean <jborean93@gmail.com>
Co-Authored-By: Matt Clay <matt@mystile.com>
Co-Authored-By: Sam Doran <sdoran@redhat.com>
Co-Authored-By: Sloane Hertel <shertel@redhat.com>
Co-Authored-By: Sviatoslav Sydorenko <webknjaz@redhat.com>

Signed-Off-By: Sviatoslav Sydorenko <webknjaz@redhat.com>
2021-01-27 22:23:22 +01:00
Joshua Bayfield
823c72bcb5
Shadow input for encrypt_string by default unless asked (fixes #71618) (#73263)
* Shadow input for encrypt_string by default unless asked (fixes #71618)
2021-01-20 15:50:24 -05:00
Alexander Sowitzki
3e8c8cd536 Make test_adhoc succeed from within a git checkout
The test currently only expects the literal `ansible` followed
by a semver in the first output line of `ansible --version`.
When running from within a git checkout, additional information
like the currently checkout branch are attached, which lets
the test fail. This commit allows arbitrary information to
follow the semver.
2021-01-13 21:22:21 +01:00
David Shrewsbury
be2c376ab8
Extract embedded function to RoleMixin method and add tests (#72754)
* Add changelog
* Simplify return
2020-12-07 13:07:23 -05:00
Martin Krizek
a2593b5e27 CLI - Specify jinja version in --version output 2020-11-05 11:37:55 -08:00
Brian Coca
cb94c0cc55
added timeout options to adhoc and console (#71230)
* added timeout options to adhoc and console

* added test

* fix typosesz

* fix conflict

* task_timeout

* fix timeout option, added extra vars to console

* actually use right cli switch .. DUH!

* added timeout to include 'valid' but ignored keys

* fix default

* fixes per review
2020-10-30 10:09:30 -04:00
Matt Davis
83909bfa22
Remove ansible-galaxy login (#72288)
* GitHub is removing the underlying API used to implement the `login` command. Since the general consensus seems to be that relatively nobody currently uses this command (in favor of explicit token passing), support was simply removed for interactive login. If a future need arises, this command should be reimplemented via OAuth Device Auth Grants.
* login or role login commands now produce a fatal error with a descriptive message
* updated 2.10 and 2.11 porting guide entries

* remove dead code/config, update messages and porting guides
2020-10-23 09:11:45 -07:00
Matt Martz
fb03ac7019
Add libyaml info to version output, and restore git info (#71948)
* Add libyaml info to version, restore git functionality

* Add clog fragment

* Update tests to reflect new version output
2020-10-02 14:04:49 -05:00
Toshio Kuratomi
fb144c4414
Update ansible doc formats (#71070)
* Fix tty_ify bugs and refactor

* Move tty_ify() and supporting attributes to the DocCLI class as that's
  the only thing using it.
* Add unittest for the code.
* Fix a bug where the substitution macros can be detected when they are
  a part of another word.
* Add support for L(), R(), and HORIZONTALLINE which were added to the
  website docs many years ago.

* Update test/units/cli/test_doc.py

Co-authored-by: Matt Clay <matt@mystile.com>

Co-authored-by: Matt Clay <matt@mystile.com>
2020-08-05 10:53:25 -07:00
Abhijeet Kasurde
3fe48ecba2
Add collection path in CLI version info (#68633)
This will provide user default path of collection

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2020-06-25 01:58:56 -04:00
Sam Doran
d45cb01b84
ansible-galaxy - fix collection installation with trailing slashes (#70016)
If we fail to find a member when extracting a directory, try adding a trailing
slash to the member name. In certain cases, the member in the tarfile will
contain a trailing slash but the file name in FILES.json will never contain
the trailing slash.

If unable to find the member, handle the KeyError and print a nicer error.

Also check if a directory exists before creating it since it may have been
extracted from the archive.

Fixes #70009

* Add unit tests
* Use loop for trying to get members
2020-06-15 17:36:07 -04:00
Sloane Hertel
e40889e711
Add support to install collections from git repositories (#69154)
* Enable installing collections from git repositories

* Add tests for installing individual and multiple collections from git repositories

* Test to make sure recursive dependencies with different syntax are deduplicated

* Add documentation

* add a changelog

* Skip Python 2.6

* Only fail if no collections are located in a git repository

Add support for a 'type' key for collections in requirement.yml files.
Update the changelog and document the supported keys and allowed values for the type.

Add a note that the collection(s) in the repo must contain a galaxy.yml

* Add a warning about embedding credentials in SCM URLs

* Update with review suggestions

* suppress sanity compile failure for Python 2.6
2020-05-29 13:33:32 -04:00
Jordan Borean
ecea15c508
Unify ansible-galaxy install -r (#67843)
* Unify ansible-galaxy install -r

* Minor nit fixes for docs

* Re-align warnings

* Fix up integration test

* Fix up test where no roles/collections were in file
2020-05-19 05:09:42 +10:00
Matt Martz
f27c417fc4
Use Templar for galaxy skeletons (#69106)
* Use Templar for galaxy skeletons. Fixes #69104

* Update checksum, our templar doesn't remove trailing newline, jinja2 seems to remove it
2020-04-23 10:36:14 -05:00
Jordan Borean
127d54b363
galaxy - preserve mode properly on artifact (#68418)
* galaxy - preserve mode properly on artifact

* Fix py2 encoding issue

* Update lib/ansible/galaxy/collection.py

Co-Authored-By: Matt Clay <matt@mystile.com>

* Use sane defaults instead of sourcing from tarfile

Co-authored-by: Matt Clay <matt@mystile.com>
2020-03-25 08:08:23 +10:00
Rick Elrod
eab914426b
Specify display width and no color in test_execute_list_collection_no_valid_paths (#67680)
This ensures the test passes regardless of the test environment.
Previously, it would pass in some environments but fail locally
in other environments, due to the width being calculated as
different or support for color not being consistently detected.

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-02-24 13:43:32 -05:00
Sam Doran
f506fd4730
Add ansible-galaxy collection list command (#65022)
* Start adding ansible-galaxy collection list options

* Working list all collections and list a specific collection

* Nuke debugging cruft

* Use to_text to get a string of the FQCN for sorting

* Improve collection output formatting

- add header
- display collection name and version in separate columns
- width of columns is dynamic based on collection name and version length

* Make role list output match collection output list

- add header
- add columns for role name and version
- make column width dynamic based on name and version length

* remove debug statemnt and extra header

* Revert "Make role list output match collection output list"

This reverts commit a0b3db47bb3b198aafd34c1f1be5b6561af2f928.

* Add validate_collection_path function

Utility function for ensuring a collection target ends with 'ansible_collection'

* Use validate_collection_path

* Do not warn if a specific collection in found in any search path

* Fix extraneous warning and remove duplicate code

Do not warn when listing a specific collection and it does not exist
in other collection paths.

Restructure the code that loops through collection paths to remove
duplicate conditional code.

* Indicate role path was found

* Use new function name

* WIP Save Point

* Use separate functions for role and collection list

* Wrap error message

There may be a better way to do this besides hard coding a line break, but this
does make the message a lot more readable.

* Add validate_collection_path function (#66441)

* Add validate_collection_path function

Utility function for ensuring a collection target ends with 'ansible_collection'

* Fix bad syntax

* Correct docstring

* Bikeshed the names

* Properly list a single role

* Simplify _display_warnings()

Only display warnings. Move exception raise back to each caller.

* Move private methods to private functions

They don't need self, so it makes sense to have them as functions
Get rid of _display_warnings() function since it doesn't do anything worthy of
an independent function.

* Add integration tests for ansible-galaxy collection list

* Fix docs sanity test

* Fix bug where ansible_collections dir does not exist

The path may exist, but if there is no ansible_collections dir inside that path,
an exception was raised in find_existing_collections().

Add integration test for this scenario

* Put execute_list() method back

* Add some informational messages for debugging

* Add unit tests

Units tests for the various private methods in support of collection list

* Start adding unit tests for test_execute_list

* Display collection path when listing specific collection

* Add unit tests for listing all collections and specific collection

- Create fixture for creating test objects
- Add function for controlling os.path.isdir results

* Set defaults for minimum collection widths

Ensure that collections with small FQCNs display correctly.
Add unit tests

* Split up unit tests and fix fixtures

Add more fixtures for mocking objects during the specific collection tests

* Change help message for -p in list subcommand

Give accurate description of what it actually does rather than trying to use language shared between sub commands.

* Disable colorized output in unit test

* Add docs for collection list

* Fix integration test on macOS

The temp file path is really long on macOS, so the warning message gets wrapped
across multiple lines. That make seth grep fail. Switch to matching on a smaller
part of the warning.

* Recreate common path options for collections

Improve help about what the '-p' option does and how it works.

* Remove unnecessary elif after continue statements

* Account for duplicate paths in collections_searh_paths

If someone specifies the same path via '-p' that is the COLLECTIONS_PATHS,
do not list the collections twice.

* Docs updates
2020-02-14 14:00:00 -06:00
Jordan Borean
694ef5660d
Fix using a URL for galaxy collection install (#65272)
* Fix using a URL for galaxy collection install

* Update lib/ansible/galaxy/collection.py

Co-Authored-By: Sloane Hertel <shertel@redhat.com>
2019-12-03 04:55:31 +10:00
Mads Jensen
0f491c0289 Replace TestCase.assertEquals with TestCase.assertEqual. 2019-11-11 10:24:21 -08:00
Sorin Sbarnea
a6e610a9ed Prevent generating roles that do not pass linting (#63709)
Fixes bug where newly created role fails linting due to extra space
at end of line: [201] Trailing whitespace
2019-10-22 11:26:07 -04:00
Jordan Borean
85eba9d860 Fix test_galaxy failing test 2019-09-19 13:20:43 -07:00
Jordan Borean
e747487720
ansible-galaxy - define multiple galaxy instances in ansible.cfg (#60553)
* ansible-galaxy: support multiple servers on install

* Added docs for the server configuration file

* Fix up doc string for requirements file format

* Fix bugs after testing

* Fix kwarg doc and added version

* Fix typo and doc improvement

* Fix base64 encoding and allow --server to override list
2019-08-21 07:49:05 +10:00
Jordan Borean
4ebac7d5b7
ansible-galaxy - more fixes for verbosity without sub type (#60604)
* ansible-galaxy - more fixes for verbosity without sub type

* remove testing line
2019-08-15 11:53:40 +10:00
Andrey Klychkov
7955405a23 optparse_helpers: improve unit tests for version() (#59626)
optparse_helpers: improve unit tests for version()
2019-08-14 10:32:26 -05:00
Jordan Borean
14a7722e39
ansible-galaxy tidy up arg parse with better validation (#59957)
* ansible-galaxy tidy up arg parse with better validation

* Add support back in for -v before sub aprser

* Added deprecation warning for manually parsed verbosity
2019-08-14 06:36:29 +10:00
Matt Clay
6c09b5c659
Unit test cleanup. (#60315)
* Clean up option parsing unit tests.
* Remove unused unit test marker and import.
2019-08-09 08:31:13 -07:00
Abhijeet Kasurde
056aac1e30 Minor fixes in galaxy command for collection (#59846)
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2019-08-02 07:48:34 +10:00
Andrey Klychkov
4e8df9a4b8 unit tests: remove unused imports (#59636) 2019-07-26 10:53:31 -04:00
Jordan Borean
65049620ee
Generate galaxy.yml based on single source of truth (#59170)
* Generate galaxy.yml based on single source of truth

* Fix up tests and align file names

* Minor Makefile tweak

* Remove link in galaxy.yml file and make it a template file

* Moved collections docs to dev_guide

* change Makefile clean path

* Added readme to example meta file

* review fixes

* Use newer style for doc generation script

* Fix mistake in dev_guide index

* removed uneeded file, fixed links and added preview banner

* Moved banner for sanity test
2019-07-23 06:50:46 +10:00
Matt Martz
284dafe476
Perfy McPerferton (#58400)
* InventoryManager start of perf improvements

* 0 not 1

* More startswith to [0] improvements

* Remove unused var

* The hash doesn't need to be a string, start as a list, make it into a tuple

* set actually appears faster than frozenset, and these don't need to be frozen

* Cache hosts lists, to avoid extra get_hosts calls, pass to get_vars too

* negligible perf improvement, it could help with memory later

* Try the fast way, fallback to the safe way

* Revert to previous logic, linting fix

* Extend pre-caching to free

* Address test failures

* Hosts are strings

* Fix unit test

* host is a string

* update test assumption

* drop SharedPluginLoaderObj, pre-create a set, instead of 2 comparisons in the list comprehension

* Dedupe code

* Change to _hosts and _hosts_all in get_vars

* Add backwards compat for strategies that don't do set host caches

* Add deprecation message to SharedPluginLoaderObj

* Remove unused SharedPluginLoaderObj import

* Update docs/comments

* Remove debugging

* Indicate what patterh_hash is

* That won't work

* Re-fix tests

* Update _set_hosts_cache to accept the play directly, use without refresh in get_hosts_remaining and get_failed_hosts for backwards compat

* Rename variable to avoid confusion

* On add_host only manipulate _hosts_cache_all

* Add warning docs around _hosts and _hosts_all args
2019-07-22 14:25:20 -05:00
Jordan Borean
b6791e6ae3
ansible-galaxy: add collection sub command (#57106)
* ansible-galaxy: add collection init sub command

* Fix changelog and other sanity issues

* Slim down skeleton structure, fix encoding issue on template

* Fix doc generation code to include sub commands

* Added build step

* Tidy up the build action

* Fixed up doc changes and slight testing tweaks

* Re-organise tests to use pytest

* Added publish step and fixed up issues after working with Galaxy

* Unit test improvments

* Fix unit test on 3.5

* Add remaining build tests

* Test fixes, make the integration tests clearer to debug on failures

* Removed unicode name tests until I've got further clarification

* Added publish unit tests

* Change expected length value

* Added collection install steps, tests forthcoming

* Added unit tests for collection install entrypoint

* Added some more tests for collection install

* follow proper encoding rules and added more tests

* Add remaining tests

* tidied up tests and code based on review

* exclude pre-release versions from galaxy API
2019-07-10 05:47:25 +10:00
Sam Doran
b3ce3fc5eb
Restore ansible --version output (#55728)
* Add custom action class for version info
* Use args from CLI as prog for ArgumentParser object
* Make prog a required parameter of create_base_parser() and update all uses to pass in the newly required parameter.
* Add unit test for checking ansible --version
* Update other related unit tests
2019-04-29 16:38:31 -04:00
Matt Martz
db6cc60352
Migrate command line parsing to argparse (#50610)
* Start of migration to argparse

* various fixes and improvements

* Linting fixes

* Test fixes

* Fix vault_password_files

* Add PrependAction for argparse

* A bunch of additional tweak/fixes

* Fix ansible-config tests

* Fix man page generation

* linting fix

* More adhoc pattern fixes

* Add changelog fragment

* Add support for argcomplete

* Enable argcomplete global completion

* Rename PrependAction to PrependListAction to better describe what it does

* Add documentation for installing and configuring argcomplete

* Address rebase issues

* Fix display encoding for vault

* Fix line length

* Address rebase issues

* Handle rebase issues

* Use mutually exclusive group instead of handling manually

* Fix rebase issues

* Address rebase issue

* Update version added for argcomplete support

* -e must be given a value

* ci_complete
2019-04-23 13:54:39 -05:00
Matt Martz
d1116ef2d8
Remove unused internal -a argument and functionality (#52602) 2019-02-19 17:49:06 -06:00
Martin Krizek
84ba41e0e1 ansible-doc: fix traceback when passing -a arg (#52036)
Fixes #52034
2019-02-19 15:14:28 -05:00
Toshio Kuratomi
27c7d5bb01 Move the arguments module into cli/ and context_objects into utils
* Note: Python2 is not as intelligent at detecting false import loops as
  Python3.  context_objects.py cannot be added to cli/arguments because it
  would set up an import loop between cli/__init__.py,
  cli/arguments/context_objects.py, and context.py on Python2.

ci_complete
2019-01-03 18:12:23 -08:00
Toshio Kuratomi
7e92ff823e Split up the base_parser function
The goal of breaking apart the base_parser() function is to get rid of
a bunch of conditionals and parameters in the code and, instead, make
code look like simple composition.

When splitting, a choice had to be made as to whether this would operate
by side effect (modifying a passed in parser) or side effect-free
(returning a new parser everytime).

Making a version that's side-effect-free appears to be fighting with the
optparse API (it wants to work by creating a parser object, configuring
the object, and then parsing the arguments with it) so instead, make it
clear that our helper functions are modifying the passed in parser by
(1) not returning the parser and (2) changing the function names to be
more clear that it is operating by side-effect.

Also move all of the generic optparse code, along with the argument
context classes, into a new subdirectory.
2019-01-03 18:12:23 -08:00
Toshio Kuratomi
afdbb0d9d5 Save the command line arguments into a global context
* Once cli args are parsed, they're constant.  So, save the parsed args
  into the global context for everyone else to use them from now on.
* Port cli scripts to use the CLIARGS in the context
* Refactor call to parse cli args into the run() method
* Fix unittests for changes to the internals of CLI arg parsing
* Port callback plugins to use context.CLIARGS
  * Got rid of the private self._options attribute
  * Use context.CLIARGS in the individual callback plugins instead.
  * Also output positional arguments in default and unixy plugins
  * Code has been simplified since we're now dealing with a dict rather
    than Optparse.Value
2019-01-03 18:12:23 -08:00
Abhijeet Kasurde
162d9497ba
Ad-hoc command: fix 'any' call in play_ds (#49852)
* Ad-hoc command: fix 'any' call in play_ds
* Unit test for ad-hoc cli
* Review comments

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2018-12-16 11:07:06 +05:30
Matt Clay
3033fd96b0
Move unit test compat code out of lib/ansible/. (#46996)
* Move ansible.compat.tests to test/units/compat/.
* Fix unit test references to ansible.compat.tests.
* Move builtins compat to separate file.
* Fix classification of test/units/compat/ dir.
2018-10-12 20:01:14 -07:00
Matt Clay
0686450cae
Fix unit tests which modify the source tree. (#45763)
* Fix CNOS unit test log usage.
* Use temp dir for Galaxy unit tests.
* Write to temp files in interfaces_file unit test.
* Fix log placement in netapp_e_ldap unit test.
2018-09-17 20:27:14 -07:00
Aidan Feldman
fa18d45eb3 Link to the Galaxy platforms list from the meta file template (#34046)
* Link to the Galaxy platforms list from the meta file template
Fixes https://github.com/ansible/galaxy/issues/52.
* make ansible/ansible text match mazer text on platforms
2018-09-17 13:39:54 -05:00
Dylan Murray
90943a3362 Add APB role_type to ansible-galaxy init (#36789)
* Add APB role_type to ansible-galaxy init

* Add handlers scaffolding

* Fix shippable errors by setting default

* Add unit tests and fix shippable

* Updates init description
2018-03-09 06:50:17 -05:00
Pilou
26d20ec194 galaxy unit tests: add missing arg (#33766)
* galaxy unit tests: add missing program name

* galaxy unit tests: remove incorrect switchs

- 'install' action doesn't support '--offline' switch
- 'remove' action doesn't support '--init-path' switch
2018-01-04 09:16:56 -08:00
Pilou
7c187cae93 Fix playbook cli unit test (#33767)
* TestPlaybookCLI: '--flush-cache' were ignored

Check that using '--flush-cache' does something

* check that '_flush_cache' does something

There isn't any fact by default: the assertion was true even if
'cli._flush_cache()' isn't called.
2017-12-13 15:03:06 +00:00
Adrian Likins
86dc3c09ac
Fix vault --ask-vault-pass with no tty (#31493)
* Fix vault --ask-vault-pass with no tty

2.4.0 added a check for isatty() that would skip setting up interactive
vault password prompts if not running on a tty.

But... getpass.getpass() will fallback to reading from stdin if
it gets that far without a tty. Since 2.4.0 skipped the interactive
prompts / getpass.getpass() in that case, it would never get a chance
to fall back to stdin.

So if 'echo $VAULT_PASSWORD| ansible-playbook --ask-vault-pass site.yml'
was ran without a tty (ie, from a jenkins job or via the vagrant
ansible provisioner) the 2.4 behavior was different than 2.3. 2.4
would never read the password from stdin, resulting in a vault password
error like:

        ERROR! Attempting to decrypt but no vault secrets found

Fix is just to always call the interactive password prompts based
on getpass.getpass() on --ask-vault-pass or --vault-id @prompt and
let getpass sort it out.

* up test_prompt_no_tty to expect prompt with no tty

We do call the PromptSecret class if there is no tty, but
we are back to expecting it to read from stdin in that case.

* Fix logic for when to auto-prompt vault pass

If --ask-vault-pass is used, then pretty much always
prompt.

If it is not used, then prompt if there are no other
vault ids provided and 'auto_prompt==True'.

Fixes vagrant bug https://github.com/hashicorp/vagrant/issues/9033

Fixes #30993
2017-11-15 14:01:32 -05:00