Commit graph

2213 commits

Author SHA1 Message Date
Matt Clay
fa48678a08
Rename pylint plugin and add tests. (#70225)
* Renamed custom pylint plugin for unwanted names.
* Add integration tests for sanity test failures.
2020-06-22 20:25:35 -07:00
Matt Clay
234994fc07 Fix boilerplate in setup.py and lib/ansible/ dir. 2020-06-22 16:29:16 -07:00
Matt Martz
1fedb95e4b
Use libc wcwidth to calculate print width in display (#66214)
* Use libc wcwidth to calculate print width in display. Fixes #63105

* Remove errantly added blank lines

* Fixes

* Move setlocale, adjust tests to work around py2 oddity with characters following null

* Don't change cli stub

* emojis

* Remove to_text call

* Special accounting for deletions

* Add initialization function, expand tests, ensure fallback to len

* get_text_width requires text, ensure banner deals with it

* Handle setlocale errors

* Move variable decrement

* Remove unused import
2020-06-22 16:57:01 -05:00
Martin Krizek
eaf6086eea
Fix storing local task_vars facts for the retry (#70171)
* Fix storing local task_vars facts for the retry

Fixes #70168
2020-06-22 10:07:47 -04:00
Jordan Borean
8b24a4c5ed
Fix up the redirection of the win setup module (#70158) 2020-06-19 09:24:35 -07:00
Sam Doran
bc05415109
Only pass kwargs to our string checker not callable checkers (#70151)
Since only check_type_str() accepts extra param, only pass to our checker and
do not pass kwargs to custom checkers.

* Add unit tests
2020-06-19 09:52:05 -04:00
Matt Clay
bccf6b85f2 Change comprehension to list() in plugin loader. 2020-06-18 12:29:52 -07:00
Matt Clay
1a0d8a51cd Fix yamllint sanity test line numbers.
Parse errors from libyaml now compensate for the offset of the documentation within a module.
2020-06-18 10:03:21 -07:00
Matt Clay
e3f3b4002e
Clean up ansible-test pylint configs. (#70129)
* Sort disabled pylint codes in config.
* Fix comments in pylint config.
* Add comments to pylint config.
* Prepare to sort pylint config.
* Fix pylint config indentation.
* Sort pylint good/bad names in config.
* Add changelog fragment.
2020-06-17 15:06:04 -07:00
Rick Elrod
cc071cdea6
Bump devel to 2.11.0 (#70121)
Signed-off-by: Rick Elrod <rick@elrod.me>
2020-06-17 14:22:38 -05:00
Matt Clay
8152d8bc1a Disabled inconsistent pylint checks. 2020-06-17 10:10:17 -07:00
Matt Clay
598786e16d Update ansible-test default test containers.
The main change is the upgrade to Python 3.9.0b3.
2020-06-16 16:50:04 -07:00
David Shrewsbury
fc448ea54f
Clarify blockinfile docs for insertafter/insertbefore (#69396)
* Clarify blockinfile docs for insertafter/insertbefore

It's not clear from the docs that these options take effect
only when no marker lines are found in the document.

* Add changelog fragment
2020-06-16 14:48:08 -04:00
Matt Clay
f69b1846f9 Add pyparsing constraint to ansible-test.
The upcoming pyparsing 3 release will require Python 3.5 or later, see:

https://github.com/pypa/packaging/issues/313

Unfortunately pip 8.x and earlier versions do not support python version requirements, which is why this constraint is needed.
2020-06-16 10:38:16 -07:00
Matt Davis
de63cba7e8
transparent downstream vendoring (#69850)
* builtin downstream vendoring support

* allows downstream packagers to install packages to `ansible/_vendor` that will automatically be added to head of sys.path during `ansible` package load
* tests

* sort conflicting package names in warning text

* sanity fixes

* skip unnecessary comparison
2020-06-15 16:22:25 -07:00
Matt Clay
7641d32f8e
Fix ansible-test import sanity test issues. (#70084) 2020-06-15 15:09:15 -07:00
Martin Krizek
843751a00d
yum/dnf: check type of elements in a name (#70072) 2020-06-15 11:01:42 -04:00
Matt Clay
5b3719805e Update ansible-test test containers. 2020-06-12 09:19:18 -07:00
John R Barker
37785255f2
validate ansible-base's and collections runtime.yml (#69742)
* Validate ansible-base & collection's runtime.yml

Add new test `runtime-metadata`

* Schema validation of file
* Error if a a legacy meta/routing.yml exist in a collection
* removal_date OR removal_version

* Add tombstone validation.

* Allow both ISO 8601 date strings and datetime.date objects (from YAML dates).

* Address review comments.

* Add metadata to test collection.

* Add requirements file.

Co-authored-by: Felix Fontein <felix@fontein.de>
Co-authored-by: Matt Clay <matt@mystile.com>
2020-06-12 15:34:34 +01:00
Matt Clay
9f49db1f99 Avoid use of deprecated junit-xml method. 2020-06-12 00:47:07 -07:00
Brian Coca
4c9d9dbb56
fix discovery on loop with delegation (#70013)
* fix discovery on loop with delegation

fixes #69963
2020-06-11 18:04:17 -04:00
Matt Clay
8ffaed00f8 Add Azure Pipelines support to ansible-test. 2020-06-11 14:57:42 -07:00
Brian Coca
8c3eb7ae4a
fix configurable pipelining (#69920)
* fix configurable pipelining

Co-authored-by: Sloane Hertel <shertel@redhat.com>
2020-06-11 16:17:58 -04:00
Matt Martz
4a4a11d282
Add mccabe complexity testing (#64623)
* Add mccabe complexity testing

* Make mccabe complexity an optional error

* Add mccabe to new sanity pylint requirements

* Add a changelog fragment.

Co-authored-by: Matt Clay <matt@mystile.com>
2020-06-11 13:02:06 -07:00
David Shrewsbury
0ae4dac65a
Fix copy module file perms with remote_src (#69993)
When using 'remote_src: yes' and 'mode: preserve', the code handling
the file modes has to be handled on the remote node because it's
the one that has access to the source files. This means that the
copy module itself must handle this, rather than the copy action
plugin (which is where all that logic exists). The copy module
handles this when we copy a single file over. But when it is a
directory as the src parameter value, the mode of the files
beneath it are not considered. Subdirectories are copied with
shutil.copytree() which will preserve permissions automatically.
Individual files are copied with shutil.copyfile() which does NOT
preserve permissions. We need to add some calls to shutil.copymode()
to correct that.

Note: This *always* retains individial file permissions. Specifying
a 'mode' other than 'preserve' when giving a source directory for
the 'src' param does not make sense so will be ignored in that case
only.

Fixes #69783

* Add changelog and test
2020-06-11 15:18:59 -04:00
James Cassell
47d14a33bd
config: singular ANSIBLE_COLLECTIONS_PATH (#70007)
* config: singular ANSIBLE_COLLECTIONS_PATH

Every other *_PATH setting in ansible is singular, and the traditional
$PATH variable is also singular despite containing a list of
directories.  Let's be consistent both internally and with POSIX
tradition.

* update all ANSIBLE_COLLECTIONS_PATHS env references to be singular

* deprecate plural ANSIBLE_COLLECTIONS_PATHS setting
2020-06-11 11:40:13 -07:00
Felix Fontein
a114da80ee
Use antsibull-changelog instead of packaged changelog generator (#69313)
Replace the ansible-base changelog linting and generation tool with antsibull-changelog and make it available for linting collections. Previously changelog linting was limited to ansible-base.
2020-06-11 11:21:21 -07:00
Felix Fontein
8d93ba9120
Plugin/module docs: parse return values, add collection names in them (version_added_collection), and format them nicely in ansible-doc (#69796)
* Tag return value docs if they are a dict (and not str/None).

* Try to parse return docs as YAML.

* Properly dump return values in ansible-doc.

* Adjust plugin formatter.

* Add changelog fragment.

* Don't add 'default' for return values.

* Fix plugin_formatter.

* Only try to parse return docs if they are still a string.

* Add tests.

* Warn if RETURN cannot be parsed.

* Adjust tests. Also test for warning.

* if -> elif (otherwise EXAMPLE will be parsed too).

* Always parse return documentation, and fail if it is invalid YAML.

* Polishing.

* Mostly re-enable ansible-doc tests.

Listing from the local collection seems to be somewhat broken. I assume this
is why the test was disabled.

* Lint and make tests work with Python 2.

* Keep FQCNs in plugins (not modules), i.e. restore previous state.
2020-06-11 11:03:43 -07:00
Alicia Cozine
f509a22f9d
add changelog categories, update CVE fragments to use security_fix category (#69968)
* use security_fix category in changelogs for CVEs

* these fragments do not say CVE but are security fixes

Co-authored-by: Alicia Cozine <acozine@users.noreply.github.com>
2020-06-11 10:24:01 -07:00
Nilashish Chakraborty
9d6b0f2b03
Support removed_at_date in ansible-doc (#70002)
* Support removed_at_date in ansible-doc

Signed-off-by: NilashishC <nilashishchakraborty8@gmail.com>

Changes:
  * ansible-doc does not support `removed_at_date` and assumes that
    deprecated dict will either have `removed_in` or `version`. This
    results in ansible-doc (and hence "sanity --test=ansible-doc")
    failing for modules having only `removed_at_date`.

  * This patch adds support for `removed_at_date` and also gives it
    precedence over `removed_in` or `version`.

* Add tests and changelog

Signed-off-by: NilashishC <nilashishchakraborty8@gmail.com>
2020-06-11 13:18:18 -04:00
Brian Coca
ebb22655e4
implemented 'prefix' for file based cache (#69872)
* implemented 'prefix' for file based cache

Co-authored-by: s-hertel <shertel@redhat.com>
2020-06-11 10:40:25 -04:00
Jordan Borean
d30fc6c0b3
galaxy - preserve symlinks on build/install (#69959)
* galaxy - preserve symlinks on build/install

* Handle directory symlinks

* py2 compat change

* Updated changelog fragment
2020-06-11 06:46:42 +10:00
Brian Coca
37e7191304
added more ways to config new options in url (#69950)
* added more ways to config new options in url
2020-06-10 15:48:20 -04:00
Martin Krizek
ac20466375
Do not keep empty blocks after tag filtering (#69987)
This prevents PlayIterator having to go through empty blocks
that were created in filter_tagged_tasks. This should
be a performance improvement for playbooks that mostly skip
tasks with tags.

ci_complete
2020-06-10 12:12:11 -05:00
Brian Coca
026bf4a9a8
do we really need update vars from pc? (#69952)
* don't we really need update vars from pc
2020-06-10 08:39:47 -04:00
Brian Coca
805dff4129
fixed missing default (#69972)
* fixed missing default

 also deprecated so new options wont have to go through this
2020-06-10 08:35:36 -04:00
Matt Clay
a81dd4f06a
Update ansible-test default containers. (#69819)
* Move ansible-test completion code.

* Fix a few type hints.

* Change docker completion based on context.

Collections now use version 2.0 of the default-test-container.
This is an updated version of the earlier 1.x default-test-container with ansible-base and cloud specific requirements removed.

Testing of ansible-base now uses version 1.0 of a new ansible-base-test-container.
This container is similar to the earlier 1.x default-test-container, but with unnecessary duplication of requirements across Python versions removed.

Collections which have tests that depend on requirements no longer present in the default test container should specify them in their test requirements files:

* tests/integration/requirements.txt
* tests/unit/requirements.txt

* Bump test container versions

Co-authored-by: Jordan Borean <jborean93@gmail.com>
2020-06-09 15:40:56 -07:00
Sloane Hertel
51f6d129cb
support hard coded module_defaults.yml groups for collections (#69919)
* Only allow groups which were hardcoded in module_defaults.yml

only load action groups from the collection if module_defaults contains a potential group for the action

* Fix tests using modules that override those whitelisted in lib/ansible/config/module_defaults.yml

Third party modules should not be using group/ - use the action name instead

* add externalized module_defaults tests

add the missing group and collections

ci_complete

Co-authored-by: Matt Davis <mrd@redhat.com>

* changelog

ci_complete

* Fix import in tests

ci_complete

* Update with requested changes

ci_complete

* don't traceback since we don't validate the contents of module_defaults

ci_complete

Co-authored-by: Matt Davis <mrd@redhat.com>
2020-06-09 15:38:57 -07:00
Felix Fontein
a862ff2d43
Deprecation revisited (#69926)
* Allow to specify collection_name separately for deprecation.

* Use new functionality in Ansible.

* Use new functionality in tests.

* Update tagging/untagging functions.

* Update pylint deprecated sanity test.

* Update validate-modules. Missing are basic checks for version_added (validate semantic version format for collections).

* Improve version validation. Re-add version_added validation.

* Make sure collection names are added to return docs before schema validation.

* Extra checks to avoid crashes on bad data.

* Make C# module utils code work, and update/extend tests.

* Add changelog fragment.

* Stop extracting collection name from potentially tagged versions/dates.

* Simplify C# code.

* Update Windows modules docs.

* Forgot semicolons.
2020-06-09 15:21:19 -07:00
Matt Clay
7bff3d312f
Split out and install sanity test requirements. (#69971)
* Split out sanity test requirements.

* Run each --venv test separately.

This provides verification that the requirements for each test are properly specified.

* Use a separate requirements file per sanity test.

* Skip setuptools/cryptography setup for sanity.

* Eliminate pyyaml missing warning.

* Eliminate more pip noise.

* Fix conflicting generate_pip_install commands.

* Add changelog fragment.
2020-06-09 13:38:36 -07:00
Brian Coca
ce199ef0e1
Add intg test to repro #36045 (add_host traceback) (#69912)
* Add intg test to repro #36045 (add_host traceback)

* fix raw_params usage in add_host

Co-authored-by: Adrian Likins <alikins@redhat.com>
2020-06-09 15:25:56 -04:00
Sam Doran
6065638e00
Display parameter name in string conversion warning (#57145)
Add prefix to error message for nested options. This is helpful if a subelement key has the same name as a top level key.
2020-06-09 13:05:53 -04:00
Brian Coca
a64418c2b3
avoid constant checking controlpersist (#69910)
* avoid constant checking controlpersist
2020-06-09 11:59:38 -04:00
Sam Doran
c41a160951
Update test images to version 1.16.0 (#66771) 2020-06-09 10:08:31 -04:00
Matt Martz
bfff95393c
Add custom globals to the environment, and not per template (#69278)
* Add custom globals to the environment, and not per template

* Add changelog fragment
2020-06-08 16:30:40 -05:00
Matt Martz
9667f221a5
Make AnsibleVaultEncryptedUnicode work more like a string (#67823)
* Make AnsibleVaultEncryptedUnicode work more like a string. Fixes #24425

* Remove debugging

* Wrap some things

* Reduce diff

* data should always result in text

* add tests

* Don't just copy and paste, kids

* Add eq and ne back

* Go full UserString copy/paste

* Various version related fixes

* Remove trailing newline

* py2v3

* Add a test that can evaluate whether a variable is vault encrypted

* map was introduces in jinja2 2.7

* moar jinja

* type fix

Co-Authored-By: Sam Doran <sdoran@redhat.com>

* Remove duplicate __hash__

* Fix typo

* Add changelog fragment

* ci_complete

Co-authored-by: Sam Doran <sdoran@redhat.com>
2020-06-08 16:30:14 -05:00
Sam Doran
8dd0356719
Update bundled version of distro from 1.4.0 to 1.5.0 (#69709)
Includes changes we made in our version that were merged upstream

ci_complete
2020-06-08 16:34:37 -04:00
Matt Martz
c1c6f61a18
Auto unroll generators produced by jinja filters (#68014)
* Auto unroll generators produced by jinja filters

* Unroll for native in finalize

* Fix indentation

Co-authored-by: Sam Doran <sdoran@redhat.com>

* Add changelog fragment

* ci_complete

* Always unroll regardless of jinja2

* ci_complete

Co-authored-by: Sam Doran <sdoran@redhat.com>
2020-06-08 12:58:03 -05:00
Felix Fontein
840d3a9dd7
ansible-doc: properly handle suboptions (#69795)
* Specifically handle suboptions, contains, etc in ansible-doc.
2020-06-08 11:51:33 -04:00
Matt Clay
06efc41fb0
Reduce pip output noise in ansible-test. (#69925) 2020-06-05 18:25:48 -07:00