Commit graph

2846 commits

Author SHA1 Message Date
Martin Krizek
b05e00e99a
Fix delegate_facts with interpreter not being set (#70293)
Fixes #70168

ci_complete

Co-authored-by: Brian Coca <bcoca@users.noreply.github.com>
Co-authored-by: Matt Clay <matt@mystile.com>
2020-06-30 15:49:09 +02:00
Hideki Saito
2d59e548f6
Improve ansible-galaxy STDOUT messages for collections (#70040)
- Fix issue #70010
- Add installation successful message
- This feature targets "collection" sub-command and does not affect "role" sub-command

Signed-off-by: Hideki Saito <saito@fgrep.org>
2020-06-29 15:18:30 -04:00
s-hamann
e39a9bf583
Add support for Parrot Linux, a Debian derivate (#69158)
Co-authored-by: black <invalid>
2020-06-29 20:22:44 +05:30
jctanner
b019029bf3
Add intentional unit tests for basic._set_cwd and common.dict_merge (#70283)
* Add unit tests for basic._set_cwd

* incidental coverage for dict_merge

* add test for async stderr inclusion
2020-06-25 14:45:13 -05:00
Abhijeet Kasurde
3fe48ecba2
Add collection path in CLI version info (#68633)
This will provide user default path of collection

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2020-06-25 01:58:56 -04:00
Rick Elrod
0073ab5fa6
Nuke strategy.SharedPluginLoaderObj, depr. 2.11 (#70235)
* Nuke strategy.SharedPluginLoaderObj, depr. 2.11

Change:
- Nuke SharedPluginLoaderObj class
- Update tests (which seemingly didn't use it anyway)
- Changelog

Test Plan:
CI, grep

Signed-off-by: Rick Elrod <rick@elrod.me>

* Nuke from ignore.txt

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-06-23 13:09:26 -04:00
Rick Elrod
7584e145a9
Nuke _get_item() from callback, deprecated 2.11 (#70233)
Change:
- Remove _get_item() alias as it has been deprecated
- Update tests
- Remove relevant sanity curtailment
- Add changelog

Test Plan:
CI, grep

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-06-23 12:03:27 -04:00
Rick Elrod
9cfc76a221
Nuke with_ squashing, deprecated for 2.11 (#70209)
Change:
Removes with_* loop squashing and tests for 2.11

Test Plan:
CI, and grepped for with_items in package manager integration targets.
There might be some test cases in collections which need to stop testing
this behavior.

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-06-23 11:53:25 -04:00
Matt Martz
1fedb95e4b
Use libc wcwidth to calculate print width in display (#66214)
* Use libc wcwidth to calculate print width in display. Fixes #63105

* Remove errantly added blank lines

* Fixes

* Move setlocale, adjust tests to work around py2 oddity with characters following null

* Don't change cli stub

* emojis

* Remove to_text call

* Special accounting for deletions

* Add initialization function, expand tests, ensure fallback to len

* get_text_width requires text, ensure banner deals with it

* Handle setlocale errors

* Move variable decrement

* Remove unused import
2020-06-22 16:57:01 -05:00
Matt Clay
98a0995fd0 Clean up unit test boilerplate. 2020-06-22 14:20:33 -07:00
Sam Doran
bc05415109
Only pass kwargs to our string checker not callable checkers (#70151)
Since only check_type_str() accepts extra param, only pass to our checker and
do not pass kwargs to custom checkers.

* Add unit tests
2020-06-19 09:52:05 -04:00
Matt Clay
c67efe0bd1 Avoid unnecessary comprehensions. 2020-06-18 11:50:31 -07:00
Matt Davis
de63cba7e8
transparent downstream vendoring (#69850)
* builtin downstream vendoring support

* allows downstream packagers to install packages to `ansible/_vendor` that will automatically be added to head of sys.path during `ansible` package load
* tests

* sort conflicting package names in warning text

* sanity fixes

* skip unnecessary comparison
2020-06-15 16:22:25 -07:00
Sam Doran
d45cb01b84
ansible-galaxy - fix collection installation with trailing slashes (#70016)
If we fail to find a member when extracting a directory, try adding a trailing
slash to the member name. In certain cases, the member in the tarfile will
contain a trailing slash but the file name in FILES.json will never contain
the trailing slash.

If unable to find the member, handle the KeyError and print a nicer error.

Also check if a directory exists before creating it since it may have been
extracted from the archive.

Fixes #70009

* Add unit tests
* Use loop for trying to get members
2020-06-15 17:36:07 -04:00
Matt Clay
8ffaed00f8 Add Azure Pipelines support to ansible-test. 2020-06-11 14:57:42 -07:00
James Cassell
47d14a33bd
config: singular ANSIBLE_COLLECTIONS_PATH (#70007)
* config: singular ANSIBLE_COLLECTIONS_PATH

Every other *_PATH setting in ansible is singular, and the traditional
$PATH variable is also singular despite containing a list of
directories.  Let's be consistent both internally and with POSIX
tradition.

* update all ANSIBLE_COLLECTIONS_PATHS env references to be singular

* deprecate plural ANSIBLE_COLLECTIONS_PATHS setting
2020-06-11 11:40:13 -07:00
Jordan Borean
d30fc6c0b3
galaxy - preserve symlinks on build/install (#69959)
* galaxy - preserve symlinks on build/install

* Handle directory symlinks

* py2 compat change

* Updated changelog fragment
2020-06-11 06:46:42 +10:00
Sloane Hertel
51f6d129cb
support hard coded module_defaults.yml groups for collections (#69919)
* Only allow groups which were hardcoded in module_defaults.yml

only load action groups from the collection if module_defaults contains a potential group for the action

* Fix tests using modules that override those whitelisted in lib/ansible/config/module_defaults.yml

Third party modules should not be using group/ - use the action name instead

* add externalized module_defaults tests

add the missing group and collections

ci_complete

Co-authored-by: Matt Davis <mrd@redhat.com>

* changelog

ci_complete

* Fix import in tests

ci_complete

* Update with requested changes

ci_complete

* don't traceback since we don't validate the contents of module_defaults

ci_complete

Co-authored-by: Matt Davis <mrd@redhat.com>
2020-06-09 15:38:57 -07:00
Felix Fontein
a862ff2d43
Deprecation revisited (#69926)
* Allow to specify collection_name separately for deprecation.

* Use new functionality in Ansible.

* Use new functionality in tests.

* Update tagging/untagging functions.

* Update pylint deprecated sanity test.

* Update validate-modules. Missing are basic checks for version_added (validate semantic version format for collections).

* Improve version validation. Re-add version_added validation.

* Make sure collection names are added to return docs before schema validation.

* Extra checks to avoid crashes on bad data.

* Make C# module utils code work, and update/extend tests.

* Add changelog fragment.

* Stop extracting collection name from potentially tagged versions/dates.

* Simplify C# code.

* Update Windows modules docs.

* Forgot semicolons.
2020-06-09 15:21:19 -07:00
Matt Davis
984216f52e
various deprecation, display, warning, error fixes for collections redirection (#69822)
* various deprecation, display, warning, error fixes

* Update lib/ansible/utils/display.py

Co-authored-by: Felix Fontein <felix@fontein.de>

* Update lib/ansible/utils/display.py

Co-authored-by: Felix Fontein <felix@fontein.de>

* Update lib/ansible/utils/display.py

Co-authored-by: Felix Fontein <felix@fontein.de>

* cleanup, test fixes

* add collection name to deprecated() calls

* clean up redirect entries from uncommitted tests

* fix dep warning/error header text to match previous

Co-authored-by: Felix Fontein <felix@fontein.de>
2020-06-05 00:28:40 -07:00
Brian Coca
062e780a68
starting metadata sunset (#69454)
* starting metadata sunset

 - purged metadata from any requirements
 - fix indent in generic handler for yaml content (whey metadata display was off)
 - make more resilient against bad formed docs
 - removed all metadata from docs template
 - remove metadata from schemas
 - removed mdata tests and from unrelated tests

Co-authored-by: Felix Fontein <felix@fontein.de>
Co-authored-by: Rick Elrod <rick@elrod.me>
2020-06-04 21:01:46 -04:00
Alex Schultz
247e43b252
Fix IncludedFile equality check (#69524)
In the case of a free style strategy, it is possible to end up with
multiple hosts trying to include from the same role, however the tasks
being included may be different with the use of tasks_from.  Previously
if you had two hosts that were included the same role when the
process_include_results function tries to determine if a included needs
to be run on a specific host, it would end up merging two different
tasks into which ever one was processed first.

This change updates the equality check to also check if the task uuid
associated with the IncludedFile is the same. The previous check only
checked if the task's parent uuid was the same. This breaks down when
both includes have the same parent.

    - hosts: all
      strategy: free
      gather_facts: false
      tasks:
        - include_role:
            name: random_sleep
        - block:
          - name: set a fact (1)
            include_role:
              name: set_a_fact
              tasks_from: fact1.yml
          - name: set a fact (2)
            include_role:
              name: set_a_fact
              tasks_from: fact2.yml
        - name: include didn't run
          fail:
            msg: >
              set_a_fact didn't run
              fact1: {{ fact1 | default('not defined')}}
              fact2: {{ fact2 | default('not defined') }}"
          when: (fact1 is not defined or fact2 is not defined)

Closes #69521
2020-06-04 10:41:46 -05:00
Dongsu Park
598e3392a9
Discover Flatcar Linux properly for hostname (#69627)
To avoid issues with Flatcar Container Linux being unable to be found,
detect Flatcar distro name especially for hostname, just like CoreOS
Container Linux was supported.

See also https://github.com/ansible/ansible/issues/69516
2020-06-02 18:41:53 +05:30
Sloane Hertel
e40889e711
Add support to install collections from git repositories (#69154)
* Enable installing collections from git repositories

* Add tests for installing individual and multiple collections from git repositories

* Test to make sure recursive dependencies with different syntax are deduplicated

* Add documentation

* add a changelog

* Skip Python 2.6

* Only fail if no collections are located in a git repository

Add support for a 'type' key for collections in requirement.yml files.
Update the changelog and document the supported keys and allowed values for the type.

Add a note that the collection(s) in the repo must contain a galaxy.yml

* Add a warning about embedding credentials in SCM URLs

* Update with review suggestions

* suppress sanity compile failure for Python 2.6
2020-05-29 13:33:32 -04:00
Karl Fogel
225ae65b0f
Make Python path warning say what it means to say (#69669) 2020-05-29 13:15:11 -04:00
Abhijeet Kasurde
7b3d9cbcb8
distribution: Added support for openEuler OS (#69324)
Detect os_family for openEuler OS as 'RedHat', instead of 'openEuler'.

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2020-05-29 12:43:16 +05:30
Felix Fontein
40f21dfd3c
Version source tagging (automatic and manual) for version_added and deprecation versions (#69680)
* Track collection for version_added.
Validate *all* version numbers in validate-modules.
For tagged version numbers (i.e. version_added), consider source collection to chose validation.

* Make tagging/untagging functions more flexible.

* Tag all versions in doc fragments.

* Tag all deprecation versions issued by code.

* Make Display.deprecated() understand tagged versions.

* Extend validation to enforce tagged version numbers.

* Tag versions in tests.

* Lint and fix test.

* Mention collection name in collection loader's deprecation/removal messages.

* Fix error IDs.

* Handle tagged dates in Display.deprecated().

* Also require that removed_at_date and deprecated_aliases.date are tagged.

* Also automatically tag/untag removed_at_date; fix sanity module removal version check.

* Improve error message when invalid version number is used (like '2.14' in collections).
2020-05-28 22:46:16 -07:00
Abhijeet Kasurde
564907d8ac
distribution: Refactor test_distribution_version testcases (#69300) 2020-05-27 14:01:31 +05:30
Felix Fontein
ea04e0048d
Allow to deprecate options and aliases by date (#68177)
* Allow to deprecate options and aliases by date instead of only by version.

* Update display.deprecate().

* Adjust behavior to conform to tested behavior, extend tests, and improve C# style.

* Parse date and fail on invalid date.

This is mainly to make sure that people start using invalid dates, and we eventually have a mess to clean up.

* C# code: improve validation and update/extend tests.

* Make sure that deprecate() is not called with both date and version.

* Forgot to remove no longer necessary formatting.

* Adjust order of warnings in C# code.

* Adjust unrelated test.

* Fix grammar (and make that test pass).

* Don't parse date, and adjust message to be same as in #67684.

* Sanity tests: disable date in past test.

* Validate-modules: validate ISO 8601 date format.

* Validate-modules: switch schema declaration for deprecated_aliases to improve error messages for invalid dates.

* Use DateTime instead of string for date deprecation.

* Validate that date in deprecated_aliases is actually a DateTime.

* Fix tests.

* Fix rebasing error.

* Adjust error codes for pylint, and add removed_at_date and deprecated_aliases.date checks to validate-modules.

* Make deprecation date in the past error codes optional.

* Make sure not both version and date are specified for AnsibleModule.deprecate() calls.

* Stop using Python 3.7+ API.

* Make sure errors are actually reported. Re-add 'ansible-' prefix.

* Avoid crashing when 'name' isn't there.

* Linting.

* Update lib/ansible/module_utils/csharp/Ansible.Basic.cs

Co-authored-by: Jordan Borean <jborean93@gmail.com>

* Adjust test to latest change.

* Prefer date over version if both end up in Display.deprecated().

Co-authored-by: Jordan Borean <jborean93@gmail.com>
2020-05-26 20:23:56 -04:00
Matt Davis
fa81cc6a0c
fix delegated loading when path is not a directory (#69713)
* find_module can't pop ImportError- we need to just translate to `None` since this is a normal condition with files on sys.path (eg `/usr/lib/python36.zip`)
* added test
2020-05-26 15:55:17 -07:00
Matt Davis
f7dfa817ae
collection routing (#67684)
* `meta/` directory in collections
* runtime metadata for redirection/deprecation/removal of plugin loads
* a compatibility layer to keep existing content working on ansible-base + collections
* a Python import redirection layer to keep collections-hosted (and otherwise moved) content importable by things that don't know better
* supported Ansible version validation on collection loads
2020-05-26 09:42:06 -07:00
Nilashish Chakraborty
dae3ba71a8
Add FQCN mappings for network facts modules (#69601)
* Add FQCN mappings for network facts modules

Signed-off-by: NilashishC <nilashishchakraborty8@gmail.com>

* Add unit tests

Signed-off-by: NilashishC <nilashishchakraborty8@gmail.com>

* update entry for ironware

Signed-off-by: NilashishC <nilashishchakraborty8@gmail.com>
2020-05-22 23:20:21 +05:30
Brian Coca
de3f7c7739
fix delegated interpreter discovery (#69604)
* fix delegated interpeter
* allow returning fact if it is 'the right host'
* added note for future fix/efficiency
 as it stands we rerun discovery for the delegated host
unless its saving facts to itself
 * fixed test lacking delegate_to mock
2020-05-22 09:31:34 -04:00
Matt Martz
e0f25a2b1f
Strip no log values from module response keys. Fixes #68400 (#69653) 2020-05-21 16:17:57 -04:00
Martin Krizek
1ee70fc272
ansible.utils.vars.isidentifier improvements (#58278)
ci_complete
2020-05-21 19:54:40 +02:00
Matt Martz
b748edea45
Add multipart/form-data functionality (#69376)
* Add multipart/form-data functionality

* Fix some linting issues

* Fix error message

* Allow filename to be provided with content

* Add integration test

* Update examples

* General improvements to multipart handling

* Use prepare_multipart for galaxy collection publish

* Properly account for py2 vs py3, ensuring no max header length

* Address test assumptions

* Add unit tests

* Add changelog

* Ensure to use CRLF instead of NL

* Ignore line-endings in fixture

* Consolidate code, add comment

* Bump fallaxy container version

* ci_complete
2020-05-20 15:44:01 -05:00
Sam Doran
f01de15d25
ansible-galaxy - remove warning during collection install (#69541)
* ansible-galaxy - remove warning during collection install

If existing collections do not contain a MANIFEST.json, which is common
for collections under development that were not installed from Ansible
Galaxy, fall back to inspecting galaxy.yml rather than displaying a
warning.

A warning will still be displayed in neither a MANIFEST.json nor
galaxy.yml are present.

* Update unit tests
2020-05-19 10:31:49 -05:00
Jordan Borean
ecea15c508
Unify ansible-galaxy install -r (#67843)
* Unify ansible-galaxy install -r

* Minor nit fixes for docs

* Re-align warnings

* Fix up integration test

* Fix up test where no roles/collections were in file
2020-05-19 05:09:42 +10:00
Matt Clay
e7c2eb519b
Add Shippable request signing to ansible-test. (#69526) 2020-05-15 15:38:02 -07:00
Lukas Pirl
34db57a47f
introduce fact "ansible_processor_nproc": number of usable vcpus (#66569)
This fact reflects the number of usable vcpus (which might be different
from ansible_processor_vcpus, e.g., in containers with limits). See
also #51504.

* Add fixture data and update unit tests

Co-authored-by: Sam Doran <sdoran@redhat.com>
2020-05-15 09:38:56 -04:00
Matt Clay
d0484fbf74 Cleanup for Python 3.9 and pycodestyle compat. 2020-05-14 21:06:16 -07:00
Sloane Hertel
343ffaa18b
better error for "ansible-galaxy collection verify" if there's no MANIFEST.json (#67498)
* Add a better error for "ansible-galaxy verify" if the MANIFEST.json has been deleted from the installed collection or if the collection hasn't been installed via normal means

* Fix unit tests for the remote collection

If there's something wrong with the local collection's version it will fail before the remote collection is sought

* Add a test for the new error msg

* Prevent the duplicate warning

Mock the new isfile call where needed

* Update lib/ansible/galaxy/collection.py

Co-Authored-By: Martin Krizek <martin.krizek@gmail.com>

Co-authored-by: Martin Krizek <martin.krizek@gmail.com>
2020-05-14 12:25:19 -04:00
Bob Weinand
f200487414
Fix filedescriptor out of range in select() when running commands (#65058)
* Fix filedescriptor out of range in select() when running commands

* Simplify the run_command() code

Now that we're using selectors in run_command(), we can simplify some of
the code.

* Use fileobj.read() instead of os.read()
* No longer use get_buffer_size() as we can just slurp all of the data
  instead.

Also use a simpler conditional check of whether the selector map is
empty

Co-authored-by: Toshio Kuratomi <a.badger@gmail.com>
2020-05-14 11:46:34 -04:00
Matt Martz
4c4406b2df
Flatten the directory hierarchy of modules (#68966)
* Flatten the directory hierarchy of modules

* Update ignore.txt, flatten units

* Update imports

* Completely flatten the modules directory

* Update sanity ignore

* Fix some sanity test ignores

* Fix relative import

* Fix docs builds without category

* ci_complete

* Clean up docs. ci_complete

* Adjust needs/file alias

* ci_complete

* fix hardcoded ping module paths

Co-authored-by: Matt Davis <mrd@redhat.com>
2020-05-13 19:14:53 -07:00
Felix Fontein
5b9418c06c
Fix SemanticVersion comparison (#69395)
* Fix SemanticVersion comparison.

* Complete tests for _Alpha and _Numeric comparators.

* Linting, and add comment.
2020-05-11 11:58:15 -05:00
Abhijeet Kasurde
794d269a4d
Distribution: Add PopOS as Debian OS_FAMILY (#69294)
PopOS is a Debian based OS distribution, added support to detect
ansible_os_family as 'debian' instead of 'Pop!_OS'

Fixes: #69286

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2020-05-05 11:43:28 -05:00
Clément Notin
9d48884e36
Fix support for Kali Linux detection (#69194)
* Fix support for Kali Linux detection

* Add test for Kali Linux detection

* Improve path matching with "in list"

Co-Authored-By: Abhijeet Kasurde <akasurde@redhat.com>

Co-authored-by: Abhijeet Kasurde <akasurde@redhat.com>
2020-05-05 10:31:32 -05:00
David Shrewsbury
ff47d3f766
Make sure collection is a list if a str is given (#69081)
* Make sure collection is a list if a str is given

* Call field validation early on collections

Because we are doing work on modifying the collections value before
it is actually validated, we can validate it ourselves early to make
sure the user supplies either a string or list. Dicts are not valid.

The new validation allows us to simplify the _ensure_default_collection()
function. And since the field is now static, we no longer need to specify
a default for it, which also allows us to simplify the function. Since
the default is now removed, we can also remove the sanity/ignore.txt entry
for collectionsearch.py.

New unit tests are added (and the existing one modified) that allow us to
make sure that we throw a parser error if a user specifies something other
than a string or list for the collections value everywhere it can be specified.

* Revert removing the collection default

The default is actually used, so restore it.

* Fix unit tests in test_helpers.py affected by early collection validation
2020-04-28 10:47:11 -05:00
Matt Martz
f27c417fc4
Use Templar for galaxy skeletons (#69106)
* Use Templar for galaxy skeletons. Fixes #69104

* Update checksum, our templar doesn't remove trailing newline, jinja2 seems to remove it
2020-04-23 10:36:14 -05:00
Jordan Borean
79fff7da69
Expose to_<str> as a public function (#68965)
* Expose to_<str> as a public function

* Fix sanity checks

* Move docstring to start of util
2020-04-17 07:54:00 +10:00