Commit graph

164 commits

Author SHA1 Message Date
Sloane Hertel
47b2cb748b Allow config boolean options to default to None (#55951)
* Allow plugin boolean options to default to None

* fix mistake

* changelog

* be more specific
2019-05-02 12:09:14 -07:00
Ken Partridge
7805be2449 Add WTI Serial Port Config module (#55102) 2019-04-24 09:53:40 -04:00
Zim Kalinowski
69ca89858c
Renaming azure_rm_managed_disk for consistency (#54825) 2019-04-04 12:24:03 +08:00
Matt Davis
f86345f777
Collection content loading (#52194)
* basic plugin loading working (with many hacks)

* task collections working

* play/block-level collection module/action working

* implement PEP302 loader

* implicit package support (no need for __init.py__ in collections)
* provides future options for secure loading of content that shouldn't execute inside controller (eg, actively ignore __init__.py on content/module paths)
* provide hook for synthetic collection setup (eg ansible.core pseudo-collection for specifying built-in plugins without legacy path, etc)

* synthetic package support

* ansible.core.plugins mapping works, others don't

* synthetic collections working for modules/actions

* fix direct-load legacy

* change base package name to ansible_collections

* note

* collection role loading

* expand paths from installed content root vars

* feature complete?

* rename ansible.core to ansible.builtin

* and various sanity fixes

* sanity tweaks

* unittest fixes

* less grabby error handler on has_plugin

* probably need to replace with a or harden callers

* fix win_ping test

* disable module test with explicit file extension; might be able to support in some scenarios, but can't see any other tests that verify that behavior...

* fix unicode conversion issues on py2

* attempt to keep things working-ish on py2.6

* python2.6 test fun round 2

* rename dirs/configs to "collections"

* add wrapper dir for content-adjacent

* fix pythoncheck to use localhost

* unicode tweaks, native/bytes string prefixing

* rename COLLECTION_PATHS to COLLECTIONS_PATHS

* switch to pathspec

* path handling cleanup

* change expensive `all` back to or chain

* unused import cleanup

* quotes tweak

* use wrapped iter/len in Jinja proxy

* var name expansion

* comment seemingly overcomplicated playbook_paths resolution

* drop unnecessary conditional nesting

* eliminate extraneous local

* zap superfluous validation function

* use slice for rolespec NS assembly

* misc naming/unicode fixes

* collection callback loader asks if valid FQ name instead of just '.'
* switch collection role resolution behavior to be internally `text` as much as possible

* misc fixmes

* to_native in exception constructor
* (slightly) detangle tuple accumulation mess in module_utils __init__ walker

* more misc fixmes

* tighten up action dispatch, add unqualified action test

* rename Collection mixin to CollectionSearch

* (attempt to) avoid potential confusion/conflict with builtin collections, etc

* stale fixmes

* tighten up pluginloader collections determination

* sanity test fixes

* ditch regex escape

* clarify comment

* update default collections paths config entry

* use PATH format instead of list

* skip integration tests on Python 2.6

ci_complete
2019-03-28 10:41:39 -07:00
James Cassell
51f2065df5 use ANSIBLE_ prefix on all ansible-defined env vars (#54272)
mark old version deprecated in all places referenced via docs
2019-03-26 17:43:48 -04:00
James Cassell
9040fa0235 openstack: add os module_defaults group (#54319) 2019-03-26 15:37:06 -04:00
Sloane Hertel
54be769e8d fix AWS plugin credential precedence for environment variables (#52945)
* fix AWS plugin credential precedence for environment variables

* Allow aliases in direct plugins options

Consolidate precedence fix just in the doc fragment using aliases for mismatched options

* Access options with the option name rather than alias

* fix indentation

* update unit tests

* Improve readability
2019-03-25 16:53:11 -04:00
Matt Clay
fa82188147 Fix name and default for become plugin config.
Also add version_added.
2019-03-22 17:05:51 -07:00
Pilou
dbb782a594 Conditional bare: fix typos (#53807) 2019-03-19 21:29:59 -04:00
Mariusz Mazur
bf629240bc More kubevirt_* modules in k8s mod defaults group (#53450) 2019-03-13 14:28:54 -04:00
Brian Coca
a8eb25ac14
clear all loader caches with new dir for plugin (#53413)
* clear all loader caches with new dir for plugin

  fixes #17078

* added toggle to revert behaviour
2019-03-12 16:19:56 -04:00
Brian Coca
9b67219096
Fine tune sanity (#53544)
* modify regex to use implicit charsets this should solve issues in py3 and unicode names
* fix issue with subgroups in yaml inventory
* clarify deprecation message
* separated per name warning from deprecation
* move noise to verbosity, simplify warnings
* fix docs to reflect actual 'good' practice
* change toggle to choice list to give users more options
2019-03-11 15:12:14 -04:00
Brian Coca
8940732b58
Configurable and parallel gather facts (#49399)
* Configurable list of facts modules (#31783)

 - allow for args dict for specific modules
 - add way to pass parameters
 - avoid facts poluting test
 - move to 'facts gathered' flag
 - add 'gathering' setting tests
 - allow parallel option in case serialization is too slow
 - added support to automatically map network facts
   uses "smart" connection mapping
2019-03-08 13:08:37 -05:00
Brian Coca
d241794daa
Add toggle to control invalid character substitution in group names (#52748)
* make add_group return proper name
* ensure central transform/check
* added 'silent' option to avoid spamming current users
  those already using the plugins were used to the transformations, so no need to alert them
* centralized valid var names
* dont display dupes
* comment on regex
* added regex tests
  ini and script will now warn about deprecation
* more complete errormsg
2019-03-06 11:49:40 -05:00
Brian Coca
7a0b4a5117
Allow user to control output of verbose messages (#52948)
to stdout (default) or stderr
2019-03-05 16:08:15 -05:00
Abhijeet Kasurde
ef8be6a599 Fix wrong number of arguments in string format (#53265)
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2019-03-05 15:32:20 -05:00
Chuck Douglas
0e9a79a589 Change the retry_files_enabled to False and modify the comments to reflect that this has been disabled (#52581)
* Change the retry_files_enabled to False and modify the comments to reflect that
this has been disabled.

* Change the default action of retry_files_enabled to False

* Update porting guide to reflect change in default state of retry_files_enabled variable

* Change log documenting a change in default behaviour of retry_files_enabled

* Revert config change to comment out the retry_files_enabled line to let the user decided what is best.
Comment above still states how to change.
2019-02-28 10:13:26 -05:00
Matt Davis
4d3a6123d5
Python interpreter discovery (#50163)
* Python interpreter discovery

* No longer blindly default to only `/usr/bin/python`
* `ansible_python_interpreter` defaults to `auto_legacy`, which will discover the platform Python interpreter on some platforms (but still favor `/usr/bin/python` if present for backward compatibility). Use `auto` to always use the discovered interpreter, append `_silent` to either value to suppress warnings.
* includes new doc utility method `get_versioned_doclink` to generate a major.minor versioned doclink against docs.ansible.com (or some other config-overridden URL)

* docs revisions for python interpreter discovery

(cherry picked from commit 5b53c0012ab7212304c28fdd24cb33fd8ff755c2)

* verify output on some distros, cleanup
2019-02-27 23:52:02 -08:00
Sam Doran
f52a088862
Add option to ignore, warn, or error when a module parameter is converted to a string (#51404)
* Add new module property to Windows modules
* Add brief pause to file tests to ensure the stat times are not equal, which was happening sometimes.
* Raise TypeError on error rather than fail_json()
* Rework error message to be less verbose
* Add porting guide entry
2019-02-22 16:44:32 -05:00
Mariusz Mazur
03c4cd757c Add k8s_auth and kubevirt_vm to the k8s group (#52320) 2019-02-19 16:01:11 -05:00
Brian Coca
5e9b089ec2
Avoid resolving module_paths for docs (#51453)
alternate to #51419
2019-02-18 15:51:49 -05:00
Matt Clay
f5c92f6bc1
Allow setting resource.RLIMIT_NOFILE in modules (#51989) 2019-02-15 17:52:35 -08:00
Abhijeet Kasurde
4e0e09d2de docs: update docs about pipelining (#51856)
Additional documentation about `keep_remote_files` and
`pipelining` configurations.

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2019-02-12 12:49:00 -05:00
Matt Martz
445ff39f94
Become plugins (#50991)
* [WIP] become plugins

Move from hardcoded method to plugins for ease of use, expansion and overrides
  - load into connection as it is going to be the main consumer
  - play_context will also use to keep backwards compat API
  - ensure shell is used to construct commands when needed
  - migrate settings remove from base config in favor of plugin specific configs
  - cleanup ansible-doc
  - add become plugin docs
  - remove deprecated sudo/su code and keywords
  - adjust become options for cli
  - set plugin options from context
  - ensure config defs are avaialbe before instance
  - refactored getting the shell plugin, fixed tests
     - changed into regex as they were string matching, which does not work with random string generation
     - explicitly set flags for play context tests
 - moved plugin loading up front
 - now loads for basedir also
 - allow pyc/o for non m modules
 - fixes to tests and some plugins
 - migrate to play objects fro play_context
 - simiplify gathering
 -  added utf8 headers
 - moved option setting
 - add fail msg to dzdo
 - use tuple for multiple options on fail/missing
 - fix relative plugin paths
 - shift from play context to play
 - all tasks already inherit this from play directly
 - remove obsolete 'set play'
 - correct environment handling
 - add wrap_exe option to pfexec
 - fix runas to noop
 - fixed setting play context
 - added password configs
 - removed required false
 - remove from doc building till they are ready

future development:
  - deal with 'enable' and 'runas' which are not 'command wrappers' but 'state flags' and currently hardcoded in diff subsystems

* cleanup

  remove callers to removed func
  removed --sudo cli doc refs
  remove runas become_exe
  ensure keyerorr on plugin
  also fix backwards compat, missing method is attributeerror, not ansible error
  get remote_user consistently
  ignore missing system_tmpdirs on plugin load
  correct config precedence
  add deprecation
  fix networking imports
  backwards compat for plugins using BECOME_METHODS

* Port become_plugins to context.CLIARGS

This is a work in progress:
* Stop passing options around everywhere as we can use context.CLIARGS
  instead

* Refactor make_become_commands as asked for by alikins

* Typo in comment fix

* Stop loading values from the cli in more than one place

Both play and play_context were saving default values from the cli
arguments directly.  This changes things so that the default values are
loaded into the play and then play_context takes them from there.

* Rename BECOME_PLUGIN_PATH to DEFAULT_BECOME_PLUGIN_PATH

As alikins said, all other plugin paths are named
DEFAULT_plugintype_PLUGIN_PATH.  If we're going to rename these, that
should be done all at one time rather than piecemeal.

* One to throw away

This is a set of hacks to get setting FieldAttribute defaults to command
line args to work.  It's not fully done yet.

After talking it over with sivel and jimi-c this should be done by
fixing FieldAttributeBase and _get_parent_attribute() calls to do the
right thing when there is a non-None default.

What we want to be able to do ideally is something like this:

class Base(FieldAttributeBase):
    _check_mode = FieldAttribute([..] default=lambda: context.CLIARGS['check'])

class Play(Base):
    # lambda so that we have a chance to parse the command line args
    # before we get here.  In the future we might be able to restructure
    # this so that the cli parsing code runs before these classes are
    # defined.

class Task(Base):
    pass

And still have a playbook like this function:

---
- hosts:
  tasks:
  - command: whoami
    check_mode: True

(The check_mode test that is added as a separate commit in this PR will
let you test variations on this case).

There's a few separate reasons that the code doesn't let us do this or
a non-ugly workaround for this as written right now.  The fix that
jimi-c, sivel, and I talked about may let us do this or it may still
require a workaround (but less ugly) (having one class that has the
FieldAttributes with default values and one class that inherits from
that but just overrides the FieldAttributes which now have defaults)

* Revert "One to throw away"

This reverts commit 23aa883cbed11429ef1be2a2d0ed18f83a3b8064.

* Set FieldAttr defaults directly from CLIARGS

* Remove dead code

* Move timeout directly to PlayContext, it's never needed on Play

* just for backwards compat, add a static version of BECOME_METHODS to constants

* Make the become attr on the connection public, since it's used outside of the connection

* Logic fix

* Nuke connection testing if it supports specific become methods

* Remove unused vars

* Address rebase issues

* Fix path encoding issue

* Remove unused import

* Various cleanups

* Restore network_cli check in _low_level_execute_command

* type improvements for cliargs_deferred_get and swap shallowcopy to default to False

* minor cleanups

* Allow the su plugin to work, since it doesn't define a prompt the same way

* Fix up ksu become plugin

* Only set prompt if build_become_command was called

* Add helper to assist connection plugins in knowing they need to wait for a prompt

* Fix tests and code expectations

* Doc updates

* Various additional minor cleanups

* Make doas functional

* Don't change connection signature, load become plugin from TaskExecutor

* Remove unused imports

* Add comment about setting the become plugin on the playcontext

* Fix up tests for recent changes

* Support 'Password:' natively for the doas plugin

* Make default prompts raw

* wording cleanups. ci_complete

* Remove unrelated changes

* Address spelling mistake

* Restore removed test, and udpate to use new functionality

* Add changelog fragment

* Don't hard fail in set_attributes_from_cli on missing CLI keys

* Remove unrelated change to loader

* Remove internal deprecated FieldAttributes now

* Emit deprecation warnings now
2019-02-11 11:27:44 -06:00
Brian Coca
f54be74869 added missing version_added 2019-02-01 09:34:14 -05:00
Brian Coca
4a0fceaa3b remove bare var handling in conditionals (#51030)
* remove bare var handling in conditionals

  this makes top level and multilevel vars (dicts keys) behave the same
  it will require adding |bool for 'string comparissons' in indirect templates

  - added new tests to ensure uniform handling
  - switched to 'is' testing for status
  - changed warning to 'conditional' as 'when:' is not only place it gets triggered

* updated to include toggle and deprecation

* fix deprecated

* updated tests to handle toggle

* fixed typo and added note about the future
2019-01-30 15:00:24 -05:00
Will Thames
23b5599244 Add k8s module_defaults group and document (#51093)
It makes sense to use module_defaults with k8s modules, and thus
have a k8s module_defaults group.
2019-01-29 14:23:27 -06:00
Brian Coca
9a1c291d01 add toggle for host pattern mismatch behaviour (#51199)
* add toggle for host pattern mismatch behaviour

	fixes #40030

* fix
2019-01-25 16:00:28 -05:00
Matt Clay
48bdb61870 Fix path encoding bug in config manager. 2019-01-24 22:29:08 -08:00
Brian Coca
96b3ef5553
Doc fragments to plugins (#50172)
* promote doc_fragments into actual plugins

  change tests hardcoded path to doc fragments
  avoid sanity in fragments
  avoid improper testing of doc_fragments
  also change runner paths
 fix botmeta
 updated comment for fragments
 updated docs
2019-01-23 20:03:47 -05:00
Ganesh Nalawade
1b6228fa10
Increase persistent command_timeout default value (#51056)
* Increase persistent command_timeout default value

*  Increase command_timeout default value from 10 to 30 sec
   to reduce frequent timeout issue for network connection
   types (netconf/network_cli/httpapi/napalm)

* Fix review comments
2019-01-21 10:50:52 +05:30
Matt Clay
465df0ef8d
Fix encoding issues with file paths. (#50830)
* Fix encoding issues with file paths.

Discovered while testing with ANSIBLE_CONFIG env var set to a path
that contained unicode characters while LC_ALL=C.

* Fix unit tests.

* Fix another path encoding issue.
2019-01-14 13:06:47 -08:00
Abhijeet Kasurde
44bd8fc0a8 Update unit tests for configmanager
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2018-12-18 12:20:35 -08:00
Abhijeet Kasurde
013c42b14f
Misc typo fixes (#49816)
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2018-12-14 15:12:58 +05:30
Pilou
1bad617f29 Inventory plugins move auto before ini (#44428)
* inventory plugins: try auto before ini

auto plugin should run before ini to avoid ini being able to parse
some plugin configuration YAML files successfully.

* Update comment

comment was added by 2ffe3c42bb but
related code was later removed by
506e6a0b2d.
2018-12-11 15:17:05 -05:00
Brian Coca
70ba960f6d
More specificity in errors for cfg mgr (#48995)
* More specificity in errors for cfg mgr
2018-12-10 11:38:42 -05:00
Sloane Hertel
ae12be6cd9 Add inventory cache options to config base (#46097) 2018-12-05 17:09:49 -05:00
Ryan Brown
a51eca364f New module: AWS EC2 Launch Template (#46972)
* Add launch template integration tests
2018-11-29 13:59:10 -05:00
Brian Coca
652da82558 fix bug in config mgr
- use function that creates config slots as needed
2018-11-19 11:46:17 -05:00
Brian Coca
87e44a7ed1
better handling of bad type in config (#48821)
* better handling of bad type in config

 fixes #22468, fixes #22476
2018-11-19 11:45:45 -05:00
Matt Martz
9949629e5a
Add toml inventory plugin (#41593)
* First pass at a toml inventory

* Make EXAMPLES yaml

* Remove unnecessary comment

* Small formatting changes

* Add ansible-inventory option to list as TOML

* TOML inventory improvements, to allow a more simple inventory, specifically related to children

* changelog

* Simplify logic

* Dedupe _expand_hostpattern, making it available to all inventory plugins

* Don't make the TOML inventory dependent on the YAML inventory

* Quote IP address values

* Add more TOML examples

* Further cleanups

* Enable the toml inventory to run by default

* Create toml specific dumper

* 2.8

* Clean up imports

* No toml pygments lexer

* Don't raise an exception early when toml isn't present, and move toml to the end, since it requires an external dep

* Require toml>=0.10.0

* Further clean up of empty data

* Don't require toml>=0.10.0, but prefer it, add code for fallback in older versions

* Ensure we actually pass an encoder to toml.dumps

* Simplify recursive data converter

* Appease tests, since we haven't limited controller testing to 2.7+

* Update docstring for convert_yaml_objects_to_native

* remove outdated catching of AttributeError

* We don't need to catch ImportError when import ansible.plugins.inventory.toml

* Add note about what self.dump_funcs.update is doing

* Address some things

* A little extra comment

* Fix toml availability check

* Don't create an intermediate list

* Require toml file extension

* Add metadata

* Remove TOML docs from intro_inventory to prevent people from getting the wrong idea

* It's in defaults, remove note

* core supported, indicate very clearly that this is preview status
2018-11-06 10:02:36 -06:00
Matt Martz
45ff6df1af Remove Deprecated configs 2.8 (#46825)
* Removed deprecated ANSIBLE_HOSTS

* Bump sudo/su configs to match deprecation version for cli and playbook args

* Bump include configs to match deprecation version for 'include'
2018-11-05 12:08:13 -05:00
Andrea Scarpino
c2cd132828 Cleanup INVENTORY_IGNORE_EXTS (#47880)
BLACKLIST_EXTS already contains tilde
2018-10-31 16:04:29 -04:00
Nathaniel Case
cdaf5bd806
Add configuration to override location of ansible-connection (#46128)
* Add configuration to override location of ansible-connection

* Check var or argv[0] before $PATH
2018-10-01 14:29:59 -04:00
Matt Martz
172137c3ca
Fix issues with PLUGIN_FILTERS_CFG config handling (#45994)
* Ensure that the value of PLUGIN_FILTERS_CFG is treated as type=path, and that we use the standard section of 'defaults' instead of 'default'

* deprecate the default section

* Don't add version_added for the corrected section
2018-09-24 14:34:59 -05:00
Matt Martz
829f48e3f5
Flip default for AGNOSTIC_BECOME_PROMPT (#45563)
* Flip default for AGNOSTIC_BECOME_PROMPT

* simplify AGNOSTIC_BECOME_PROMPT porting guide entry

* clarify AGNOSTIC_BECOME_PROMPT changelog entry

* fixes version typo

* uses alternate wording (rather than escaping *)
2018-09-14 11:16:13 -05:00
Ryan S. Brown
617b8df5e6 Add rds_instance to module_defaults groups 2018-09-04 08:17:58 -04:00
Toshio Kuratomi
f46c943d3d Fix another corner case of too many warnings for world readable current working directory
There should be no warning if there is no ansible.cfg file i nthe
current working directory.
2018-08-23 20:23:59 -07:00
Pilou
2603604fd6 gather facts improvements (#44008)
* gather_subset is a list of strings

When gather_subset is an integer, a message pointing out the problem, current tb error 
is replaced by this one:

    ERROR! the field 'gather_subset' should be a list of (<class 'str'>,), but the item '42' is a <class 'int'>
...
* gathering_facts test: ensure smart gathering is on
2018-08-23 11:36:06 -04:00
skylerbunny
d4e15d40e8 Update tags documentation to explain newer tags behavior (#44274) 2018-08-23 09:46:54 -04:00