Commit graph

111 commits

Author SHA1 Message Date
Matt Clay
8cd66ce95a [stable-2.10] Clean up unit test boilerplate.
(cherry picked from commit 98a0995fd0)

Co-authored-by: Matt Clay <matt@mystile.com>
2020-07-13 18:28:02 -07:00
Alex Schultz
247e43b252
Fix IncludedFile equality check (#69524)
In the case of a free style strategy, it is possible to end up with
multiple hosts trying to include from the same role, however the tasks
being included may be different with the use of tasks_from.  Previously
if you had two hosts that were included the same role when the
process_include_results function tries to determine if a included needs
to be run on a specific host, it would end up merging two different
tasks into which ever one was processed first.

This change updates the equality check to also check if the task uuid
associated with the IncludedFile is the same. The previous check only
checked if the task's parent uuid was the same. This breaks down when
both includes have the same parent.

    - hosts: all
      strategy: free
      gather_facts: false
      tasks:
        - include_role:
            name: random_sleep
        - block:
          - name: set a fact (1)
            include_role:
              name: set_a_fact
              tasks_from: fact1.yml
          - name: set a fact (2)
            include_role:
              name: set_a_fact
              tasks_from: fact2.yml
        - name: include didn't run
          fail:
            msg: >
              set_a_fact didn't run
              fact1: {{ fact1 | default('not defined')}}
              fact2: {{ fact2 | default('not defined') }}"
          when: (fact1 is not defined or fact2 is not defined)

Closes #69521
2020-06-04 10:41:46 -05:00
David Shrewsbury
ff47d3f766
Make sure collection is a list if a str is given (#69081)
* Make sure collection is a list if a str is given

* Call field validation early on collections

Because we are doing work on modifying the collections value before
it is actually validated, we can validate it ourselves early to make
sure the user supplies either a string or list. Dicts are not valid.

The new validation allows us to simplify the _ensure_default_collection()
function. And since the field is now static, we no longer need to specify
a default for it, which also allows us to simplify the function. Since
the default is now removed, we can also remove the sanity/ignore.txt entry
for collectionsearch.py.

New unit tests are added (and the existing one modified) that allow us to
make sure that we throw a parser error if a user specifies something other
than a string or list for the collections value everywhere it can be specified.

* Revert removing the collection default

The default is actually used, so restore it.

* Fix unit tests in test_helpers.py affected by early collection validation
2020-04-28 10:47:11 -05:00
David Shrewsbury
18a66e291d
Force collections to be static (#68723)
* Force collections to be static

Templating of collection names does not work at all. Force them to
be static so that a warning is generated for the user.

* Add collectionsearch unit test and fix for reviews

New unit test validates the new _load_collections() code and moves
the new check to the end of the method.

* Change unit test to pytest

* Adjust unit test to use capsys instead of monkeypatch

* Fix pep8 error

* Add changelog fragment

Closes #68704
2020-04-13 11:53:05 -05:00
Matt Clay
46fdbbdd18
Split test_play_context_make_become_cmd into files (#68026)
* Split test_play_context_make_become_cmd into files

For NWO migration. Split the become module assertions into distinct test
files and functions. For now, this is done naively - there is probably
room to abstract these tests out and remove some of the duplication
later on.

Signed-off-by: Rick Elrod <rick@elrod.me>

* use default_exe variable instead of hardcoding /bin/bash

Signed-off-by: Rick Elrod <rick@elrod.me>

* Move become plugin tests to their proper directory and rename them accordingly

Signed-off-by: Rick Elrod <rick@elrod.me>

* Fix up fixtures and imports.

* Remove stray file.

Co-authored-by: Matt Clay <matt@mystile.com>
2020-03-04 22:26:04 -08:00
Abhijeet Kasurde
683c467609
Check if task file is specified for import_tasks (#57572)
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2019-12-24 10:26:32 +05:30
Mads Jensen
0f491c0289 Replace TestCase.assertEquals with TestCase.assertEqual. 2019-11-11 10:24:21 -08:00
Mads Jensen
c5d61953b1 Remove empty setUp/tearDown/tearDownClass methods in test classes. 2019-11-06 08:14:29 -08:00
Mads Jensen
ecc8e51044 Remove empty overridden unittest.setUp and unittest.tearDown methods. 2019-11-05 09:12:11 -08:00
Matt Martz
7f4befdea7
Wrap CLI Passwords with AnsibleUnsafeText, ensure unsafe context is not lost during encode/decode (#63351)
* Wrap .encode and .decode on AnsibleUnsafe objects

* runme.sh needs to be executable

* ci_complete

* Update changelog with CVE
2019-10-11 09:17:10 -05:00
Matt Davis
d81ae27a4a
Collection role relative deps (#61517)
* default collection support

* playbooks run from inside a registered collection will set that collection as the first item in the search order (as will all non-collection roles)
* this allows easy migration of runme.sh style playbook/role integration tests to collections without the playbooks/roles needing to know the name of their enclosing collection

* disable default collection test under Windows

* enable collection search for role dependencies

* unqualified role deps in collection-hosted roles will first search the containing collection
* if the calling role has specified a collections search list in metadata, it will be appended to the search order for unqualified role deps

* disable cycle detection unit test

* failing on 3.7+, needs proper cycle detection
* see #61527
2019-08-29 02:25:44 -07:00
Andrey Klychkov
21863d48f3 unit tests: remove unused imports (#60462) 2019-08-13 19:21:43 +02:00
Matt Davis
a40baf22fa
various mod_args fixes (#60290)
* various mod_args fixes

* filter task keywords when parsing actions from task_ds- prevents repeatedly banging on the pluginloader for things we know aren't modules/actions
* clean up module/action error messaging. Death to `no action in task!`- actually list the candidate modules/actions from the task if present.

* remove shadowed_module test

* previous discussion was that this behavior isn't worth the complexity or performance costs in mod_args

* fix/add test, remove module shadow logic

* address review feedback
2019-08-13 09:57:49 +01:00
Matt Martz
697b566971
Update units to pass on macOS (#60435)
* Update units to pass on macOS. Fixes #27810

* raising=False
2019-08-12 15:13:07 -05:00
Andrey Klychkov
64e3cb0fa8 Unit tests: remove unused imports (#59812)
* Unit tests: remove unused imports

* Unit tests: remove unused imports, fix
2019-08-02 17:18:14 +02:00
Abhijeet Kasurde
79fdc2190a role: Fix role's hash_params (#55263)
* role: Fix role's hash_params

Fix based upon work done by alikins.

Fixes: #20596

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>

* review comments

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2019-07-08 09:25:59 -04:00
Sam Doran
b3ce3fc5eb
Restore ansible --version output (#55728)
* Add custom action class for version info
* Use args from CLI as prog for ArgumentParser object
* Make prog a required parameter of create_base_parser() and update all uses to pass in the newly required parameter.
* Add unit test for checking ansible --version
* Update other related unit tests
2019-04-29 16:38:31 -04:00
Matt Martz
db6cc60352
Migrate command line parsing to argparse (#50610)
* Start of migration to argparse

* various fixes and improvements

* Linting fixes

* Test fixes

* Fix vault_password_files

* Add PrependAction for argparse

* A bunch of additional tweak/fixes

* Fix ansible-config tests

* Fix man page generation

* linting fix

* More adhoc pattern fixes

* Add changelog fragment

* Add support for argcomplete

* Enable argcomplete global completion

* Rename PrependAction to PrependListAction to better describe what it does

* Add documentation for installing and configuring argcomplete

* Address rebase issues

* Fix display encoding for vault

* Fix line length

* Address rebase issues

* Handle rebase issues

* Use mutually exclusive group instead of handling manually

* Fix rebase issues

* Address rebase issue

* Update version added for argcomplete support

* -e must be given a value

* ci_complete
2019-04-23 13:54:39 -05:00
Brian Coca
e40832df84
become mixin is no more (#54002)
* become mixin is no more
  since sudo/su keywords are removed in 2.9 .. no need to keep this code around
 * also don't need test for code that is removed
 * made preprocess_data on base noop
   its not used by anything anymore, but kept for backwards compat since other methods of same name are used
2019-04-12 12:13:55 -04:00
Matt Martz
fbf2d5d2f4
Don't pollute include_variables (#54687)
* Don't pollute include_variables. Fixes #51667. Fixes #54618.

* Rename include_variables to include_args, so we can make the distinction about what they are

* Track args and vars separately

* oops

* oops again

* linting fix

* Add test
2019-04-09 10:14:42 -05:00
Matt Davis
f86345f777
Collection content loading (#52194)
* basic plugin loading working (with many hacks)

* task collections working

* play/block-level collection module/action working

* implement PEP302 loader

* implicit package support (no need for __init.py__ in collections)
* provides future options for secure loading of content that shouldn't execute inside controller (eg, actively ignore __init__.py on content/module paths)
* provide hook for synthetic collection setup (eg ansible.core pseudo-collection for specifying built-in plugins without legacy path, etc)

* synthetic package support

* ansible.core.plugins mapping works, others don't

* synthetic collections working for modules/actions

* fix direct-load legacy

* change base package name to ansible_collections

* note

* collection role loading

* expand paths from installed content root vars

* feature complete?

* rename ansible.core to ansible.builtin

* and various sanity fixes

* sanity tweaks

* unittest fixes

* less grabby error handler on has_plugin

* probably need to replace with a or harden callers

* fix win_ping test

* disable module test with explicit file extension; might be able to support in some scenarios, but can't see any other tests that verify that behavior...

* fix unicode conversion issues on py2

* attempt to keep things working-ish on py2.6

* python2.6 test fun round 2

* rename dirs/configs to "collections"

* add wrapper dir for content-adjacent

* fix pythoncheck to use localhost

* unicode tweaks, native/bytes string prefixing

* rename COLLECTION_PATHS to COLLECTIONS_PATHS

* switch to pathspec

* path handling cleanup

* change expensive `all` back to or chain

* unused import cleanup

* quotes tweak

* use wrapped iter/len in Jinja proxy

* var name expansion

* comment seemingly overcomplicated playbook_paths resolution

* drop unnecessary conditional nesting

* eliminate extraneous local

* zap superfluous validation function

* use slice for rolespec NS assembly

* misc naming/unicode fixes

* collection callback loader asks if valid FQ name instead of just '.'
* switch collection role resolution behavior to be internally `text` as much as possible

* misc fixmes

* to_native in exception constructor
* (slightly) detangle tuple accumulation mess in module_utils __init__ walker

* more misc fixmes

* tighten up action dispatch, add unqualified action test

* rename Collection mixin to CollectionSearch

* (attempt to) avoid potential confusion/conflict with builtin collections, etc

* stale fixmes

* tighten up pluginloader collections determination

* sanity test fixes

* ditch regex escape

* clarify comment

* update default collections paths config entry

* use PATH format instead of list

* skip integration tests on Python 2.6

ci_complete
2019-03-28 10:41:39 -07:00
Abhijeet Kasurde
83be129923
helper: raise Exception when ds is not dict type (#53936)
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2019-03-25 11:18:33 +05:30
Brian Coca
f73e03632f
remove obsolete set_play on play_context (#47934)
* remove redundant assignments from play

 - fix unit tests
2019-03-11 18:25:48 -04:00
Matt Martz
445ff39f94
Become plugins (#50991)
* [WIP] become plugins

Move from hardcoded method to plugins for ease of use, expansion and overrides
  - load into connection as it is going to be the main consumer
  - play_context will also use to keep backwards compat API
  - ensure shell is used to construct commands when needed
  - migrate settings remove from base config in favor of plugin specific configs
  - cleanup ansible-doc
  - add become plugin docs
  - remove deprecated sudo/su code and keywords
  - adjust become options for cli
  - set plugin options from context
  - ensure config defs are avaialbe before instance
  - refactored getting the shell plugin, fixed tests
     - changed into regex as they were string matching, which does not work with random string generation
     - explicitly set flags for play context tests
 - moved plugin loading up front
 - now loads for basedir also
 - allow pyc/o for non m modules
 - fixes to tests and some plugins
 - migrate to play objects fro play_context
 - simiplify gathering
 -  added utf8 headers
 - moved option setting
 - add fail msg to dzdo
 - use tuple for multiple options on fail/missing
 - fix relative plugin paths
 - shift from play context to play
 - all tasks already inherit this from play directly
 - remove obsolete 'set play'
 - correct environment handling
 - add wrap_exe option to pfexec
 - fix runas to noop
 - fixed setting play context
 - added password configs
 - removed required false
 - remove from doc building till they are ready

future development:
  - deal with 'enable' and 'runas' which are not 'command wrappers' but 'state flags' and currently hardcoded in diff subsystems

* cleanup

  remove callers to removed func
  removed --sudo cli doc refs
  remove runas become_exe
  ensure keyerorr on plugin
  also fix backwards compat, missing method is attributeerror, not ansible error
  get remote_user consistently
  ignore missing system_tmpdirs on plugin load
  correct config precedence
  add deprecation
  fix networking imports
  backwards compat for plugins using BECOME_METHODS

* Port become_plugins to context.CLIARGS

This is a work in progress:
* Stop passing options around everywhere as we can use context.CLIARGS
  instead

* Refactor make_become_commands as asked for by alikins

* Typo in comment fix

* Stop loading values from the cli in more than one place

Both play and play_context were saving default values from the cli
arguments directly.  This changes things so that the default values are
loaded into the play and then play_context takes them from there.

* Rename BECOME_PLUGIN_PATH to DEFAULT_BECOME_PLUGIN_PATH

As alikins said, all other plugin paths are named
DEFAULT_plugintype_PLUGIN_PATH.  If we're going to rename these, that
should be done all at one time rather than piecemeal.

* One to throw away

This is a set of hacks to get setting FieldAttribute defaults to command
line args to work.  It's not fully done yet.

After talking it over with sivel and jimi-c this should be done by
fixing FieldAttributeBase and _get_parent_attribute() calls to do the
right thing when there is a non-None default.

What we want to be able to do ideally is something like this:

class Base(FieldAttributeBase):
    _check_mode = FieldAttribute([..] default=lambda: context.CLIARGS['check'])

class Play(Base):
    # lambda so that we have a chance to parse the command line args
    # before we get here.  In the future we might be able to restructure
    # this so that the cli parsing code runs before these classes are
    # defined.

class Task(Base):
    pass

And still have a playbook like this function:

---
- hosts:
  tasks:
  - command: whoami
    check_mode: True

(The check_mode test that is added as a separate commit in this PR will
let you test variations on this case).

There's a few separate reasons that the code doesn't let us do this or
a non-ugly workaround for this as written right now.  The fix that
jimi-c, sivel, and I talked about may let us do this or it may still
require a workaround (but less ugly) (having one class that has the
FieldAttributes with default values and one class that inherits from
that but just overrides the FieldAttributes which now have defaults)

* Revert "One to throw away"

This reverts commit 23aa883cbed11429ef1be2a2d0ed18f83a3b8064.

* Set FieldAttr defaults directly from CLIARGS

* Remove dead code

* Move timeout directly to PlayContext, it's never needed on Play

* just for backwards compat, add a static version of BECOME_METHODS to constants

* Make the become attr on the connection public, since it's used outside of the connection

* Logic fix

* Nuke connection testing if it supports specific become methods

* Remove unused vars

* Address rebase issues

* Fix path encoding issue

* Remove unused import

* Various cleanups

* Restore network_cli check in _low_level_execute_command

* type improvements for cliargs_deferred_get and swap shallowcopy to default to False

* minor cleanups

* Allow the su plugin to work, since it doesn't define a prompt the same way

* Fix up ksu become plugin

* Only set prompt if build_become_command was called

* Add helper to assist connection plugins in knowing they need to wait for a prompt

* Fix tests and code expectations

* Doc updates

* Various additional minor cleanups

* Make doas functional

* Don't change connection signature, load become plugin from TaskExecutor

* Remove unused imports

* Add comment about setting the become plugin on the playcontext

* Fix up tests for recent changes

* Support 'Password:' natively for the doas plugin

* Make default prompts raw

* wording cleanups. ci_complete

* Remove unrelated changes

* Address spelling mistake

* Restore removed test, and udpate to use new functionality

* Add changelog fragment

* Don't hard fail in set_attributes_from_cli on missing CLI keys

* Remove unrelated change to loader

* Remove internal deprecated FieldAttributes now

* Emit deprecation warnings now
2019-02-11 11:27:44 -06:00
Toshio Kuratomi
27c7d5bb01 Move the arguments module into cli/ and context_objects into utils
* Note: Python2 is not as intelligent at detecting false import loops as
  Python3.  context_objects.py cannot be added to cli/arguments because it
  would set up an import loop between cli/__init__.py,
  cli/arguments/context_objects.py, and context.py on Python2.

ci_complete
2019-01-03 18:12:23 -08:00
Toshio Kuratomi
7e92ff823e Split up the base_parser function
The goal of breaking apart the base_parser() function is to get rid of
a bunch of conditionals and parameters in the code and, instead, make
code look like simple composition.

When splitting, a choice had to be made as to whether this would operate
by side effect (modifying a passed in parser) or side effect-free
(returning a new parser everytime).

Making a version that's side-effect-free appears to be fighting with the
optparse API (it wants to work by creating a parser object, configuring
the object, and then parsing the arguments with it) so instead, make it
clear that our helper functions are modifying the passed in parser by
(1) not returning the parser and (2) changing the function names to be
more clear that it is operating by side-effect.

Also move all of the generic optparse code, along with the argument
context classes, into a new subdirectory.
2019-01-03 18:12:23 -08:00
Toshio Kuratomi
afdbb0d9d5 Save the command line arguments into a global context
* Once cli args are parsed, they're constant.  So, save the parsed args
  into the global context for everyone else to use them from now on.
* Port cli scripts to use the CLIARGS in the context
* Refactor call to parse cli args into the run() method
* Fix unittests for changes to the internals of CLI arg parsing
* Port callback plugins to use context.CLIARGS
  * Got rid of the private self._options attribute
  * Use context.CLIARGS in the individual callback plugins instead.
  * Also output positional arguments in default and unixy plugins
  * Code has been simplified since we're now dealing with a dict rather
    than Optparse.Value
2019-01-03 18:12:23 -08:00
Matt Martz
3b49bbcfde Give IncludedFile more context via ansible_search_path (#50045)
* Give IncludedFile more context via ansible_search_path to template lookups. Fixes #49969

* Update units
2019-01-03 16:13:02 -05:00
Matt Martz
7eb1ab45a7
Allow tags to be templated from a variable (#49833)
* Allow tags to be templated from a variable. Fixes #49825

* Restore _load_tags to ensure we do csv tag splitting

* Add tests for csv tags and templated tags

* evaluate_tags doesn't need to accept strings, because _load_tags handles this
2018-12-17 15:40:26 -06:00
Matt Martz
2a469fd959 Consolidate handler tracking (#49338)
* Consolidate handler tracking
  -  Remove unused code. ci_complete
  - unit test fixes. ci_complete
  - Restore previous behavior of matching a single handler
  - when notifying a host for a handler, return True if it was added, False otherwise, to reduce copied logic
  - rename funcitons for clarity. ci_complete
  - Remove handler logic for static includes which was disabled previously
2018-12-11 10:58:19 -05:00
Sam Doran
40a5f7bfdf Add better error when k=v syntax is used with YAML in tasks (#41754)
* Add error message for k=v and YAML in a single task

Find the correct line, column, and position for k=v errors since they are different than the position reported initially.

Document bug in quoting syntax check.

* Change tense or error message

Since the error still exists, switch to present tense rather than past tense.

* Remove double spaces after periods in error messages.

http://www.slate.com/articles/technology/technology/2011/01/space_invaders.html

* Add changelog fragment

* Add tests for new error message

* Fix tests

* Add clarifying comments to unit test
2018-12-04 12:32:02 -05:00
Martin Krizek
421d67f1ee
conditional: fix direct boolean "shortcut" (#47941)
* conditional: fix direct boolean "shortcut"

* Add unit tests
2018-11-07 21:33:54 +01:00
Matt Clay
3033fd96b0
Move unit test compat code out of lib/ansible/. (#46996)
* Move ansible.compat.tests to test/units/compat/.
* Fix unit test references to ansible.compat.tests.
* Move builtins compat to separate file.
* Fix classification of test/units/compat/ dir.
2018-10-12 20:01:14 -07:00
Matt Martz
a06a5ded61
Do not use mutable defaults in FieldAttribute, instead allow supplying a callable for defaults of mutable types. Fixes #46824 (#46833) 2018-10-12 10:43:09 -05:00
Matt Clay
a11f631ee4 Python 3.8 collections compatibility fixes.
Includes a new pylint blacklist plugin to prevent regressions.
2018-10-08 11:26:37 -07:00
Matt Martz
dab975d302
Unit tests for ansible.playbook.become (#38541) 2018-08-23 13:47:20 -05:00
Matt Martz
617372f8c0
Mass nuke deprecated items that are easily removed. ci_complete (#44320) 2018-08-20 16:26:10 -05:00
Andrey Laguta
2e62e36590 Fix multiple var files combining (followup to #36357) 2018-07-20 09:29:20 -04:00
Andrew Gaffney
0ff04ad41b Fix wrong python keyword (followup to #36357) (#41067)
Also add tests around that code path
2018-06-07 19:48:07 +05:30
Jordan Borean
070a5557d1
always_run: removed deprecated always_run task option (#40470) 2018-05-23 07:10:36 +10:00
Jeffrey Forman
be3670f528 fix 'doas' become_method support, previously committed patch not submitted to devel branch (#37511)
* fix become_method 'doas' support by properly specifying becomecmd

a repatch of https://github.com/ansible/ansible/pull/13451/ which was never committed to 'devel' branch.

* fix play_context test for become_method doas to match new becomecmd
2018-05-07 21:30:47 -04:00
Matt Martz
cdb79b0e3a
Consider parent also when comparing IncludedFile (#37083)
* Consider parent also when comparing IncludedFile

* Add new tests for IncludedFile and convert to pytest
2018-04-18 17:02:14 -05:00
Mat Wilson
8eb94ccbd5 Add support for dzdo flags Fixes #38766 2018-04-16 23:28:51 -04:00
Andrew Gaffney
95ce00ff00 Allow loading dirs from role defaults/vars (#36357)
This commit moves code to look for vars files/dirs to a common place and
uses it for loading role defaults/vars. This allows things such as
'defaults/main' or 'vars/main' being a directory in a role, allowing
splitting of defaults/vars into multiple files. This commit also fixes
the role loading unit tests for py3 when bytestrings are used for paths
instead of utf8 strings.

This fixes #14248 and #11639.
2018-04-10 17:14:38 -04:00
Adrian Likins
e166946a0a Fix redundant yaml error blurbs on ModArgs parse errors (#36923)
* Fix redundant yaml error blurbs on ModArgs parse errors

Some of the AnsibleParserErrors from parsing.mod_args
are created with the obj=some_yaml_ds options but
some are not.

If they were, we don't want to add another yaml_ds to
it, because that will result in double yaml error blurbs.
And since we dont need to add info, we can just re raise it.

But if there is no ._obj, add it here so we get the extra
detail in the error message (see issue #14790) and raise
a new AnsibleParserError instance.

Fixes #36848

* cleanup existing test_tasks pep8/sanity issues
2018-03-19 10:55:29 -04:00
Brian Coca
d61742147f fixed test to new sig 2018-01-04 11:20:56 -05:00
Toshio Kuratomi
235bdfb996 Remove shell module specialcases
Shell is implemented via the command module.  There was a special case
in mod_args to do that.  Make shell into an action plugin to handle that
instead.

Also move the special case for the command nanny into a command module
action plugin.  This is more appropriate as we then do not have to send
a parameter that is only for the command module to every single module.
2017-12-21 20:42:44 -08:00
Matt Clay
e45c763b64 Fix invalid string escape sequences. 2017-11-21 10:03:34 -08:00
Pilou
43914b3837 Fix include_role unit tests (#31920)
* Ensure include_role unit tests check something

This is not the case: get_tasks_vars doesn't yield

* Fix include_role unit tests

Since e609618274, include_role are not
static anymore.
2017-11-02 10:36:15 -05:00
Brian Coca
fe3b4325c2 less confusing 'args' message (#29053)
* less confusing 'args' message

* fix test

* gone native
2017-09-06 18:20:06 -04:00