Commit graph

319 commits

Author SHA1 Message Date
Vitalii Kostenko
2176b53a55 Add latest updates from FTD Ansible downstream repository. (#53638)
* Add latest updates from FTD Ansible downstream repository.
 - add a better implementation of the upsert operation;
 - add API version lookup functionality;
 - add filter which remove duplicated references from the list of references;
 - fix minor bugs.

* fix issues outlined by ansibot

* fix argument name for _check_enum_method
2019-04-01 18:08:01 +05:30
Matt Davis
f86345f777
Collection content loading (#52194)
* basic plugin loading working (with many hacks)

* task collections working

* play/block-level collection module/action working

* implement PEP302 loader

* implicit package support (no need for __init.py__ in collections)
* provides future options for secure loading of content that shouldn't execute inside controller (eg, actively ignore __init__.py on content/module paths)
* provide hook for synthetic collection setup (eg ansible.core pseudo-collection for specifying built-in plugins without legacy path, etc)

* synthetic package support

* ansible.core.plugins mapping works, others don't

* synthetic collections working for modules/actions

* fix direct-load legacy

* change base package name to ansible_collections

* note

* collection role loading

* expand paths from installed content root vars

* feature complete?

* rename ansible.core to ansible.builtin

* and various sanity fixes

* sanity tweaks

* unittest fixes

* less grabby error handler on has_plugin

* probably need to replace with a or harden callers

* fix win_ping test

* disable module test with explicit file extension; might be able to support in some scenarios, but can't see any other tests that verify that behavior...

* fix unicode conversion issues on py2

* attempt to keep things working-ish on py2.6

* python2.6 test fun round 2

* rename dirs/configs to "collections"

* add wrapper dir for content-adjacent

* fix pythoncheck to use localhost

* unicode tweaks, native/bytes string prefixing

* rename COLLECTION_PATHS to COLLECTIONS_PATHS

* switch to pathspec

* path handling cleanup

* change expensive `all` back to or chain

* unused import cleanup

* quotes tweak

* use wrapped iter/len in Jinja proxy

* var name expansion

* comment seemingly overcomplicated playbook_paths resolution

* drop unnecessary conditional nesting

* eliminate extraneous local

* zap superfluous validation function

* use slice for rolespec NS assembly

* misc naming/unicode fixes

* collection callback loader asks if valid FQ name instead of just '.'
* switch collection role resolution behavior to be internally `text` as much as possible

* misc fixmes

* to_native in exception constructor
* (slightly) detangle tuple accumulation mess in module_utils __init__ walker

* more misc fixmes

* tighten up action dispatch, add unqualified action test

* rename Collection mixin to CollectionSearch

* (attempt to) avoid potential confusion/conflict with builtin collections, etc

* stale fixmes

* tighten up pluginloader collections determination

* sanity test fixes

* ditch regex escape

* clarify comment

* update default collections paths config entry

* use PATH format instead of list

* skip integration tests on Python 2.6

ci_complete
2019-03-28 10:41:39 -07:00
Kyryl Galanov
4a5b9bd8eb New lookup module: manifold (#50435)
* New lookup module: manifold

Add Manifold.co integration. The plugin fetches resource credentials
from Manifold service.

* module manifold: fix ansible lint warnings

* module manifold: fix false warning - split test assertion

* manifold module: fix unittest import

* manifold module: fix unittest patch

* manifold module: fix python3 requests getheader error
2019-03-27 07:20:15 -04:00
Alan Rominger
f2f11341a3 Avoid erroring on host group confusion in keyed groups (#54099) 2019-03-25 16:57:46 -04:00
Sloane Hertel
54be769e8d fix AWS plugin credential precedence for environment variables (#52945)
* fix AWS plugin credential precedence for environment variables

* Allow aliases in direct plugins options

Consolidate precedence fix just in the doc fragment using aliases for mismatched options

* Access options with the option name rather than alias

* fix indentation

* update unit tests

* Improve readability
2019-03-25 16:53:11 -04:00
Matt Clay
3ea05f7757 Add pypsrp to unit tests and disable failing test. 2019-03-20 14:10:21 -07:00
Sloane Hertel
87ebc56de6 Allow parent groups to be variables or literal (#53649)
* Allow parent groups to be variables or literal, requires {{ }}
* Check strict before failing on templating errors
* Don't add a group if an invalid parent group was provided
2019-03-14 14:22:18 -04:00
Alan Rominger
8c2037aaa9 do not error in case of empty list constructed group (#53748) 2019-03-14 12:27:56 -04:00
Jordan Borean
8ef2e6da05 Add support for Windows hosts in the SSH connection plugin (#47732)
* Add support for Windows hosts in the SSH connection plugin

* fix Python 2.6 unit test and sanity issues

* fix up connection tests in CI, disable SCP for now

* ensure we don't pollute the existing environment during the test

* Add connection_windows_ssh to classifier

* use test dir for inventory file

* Required powershell as default shell and fix tests

* Remove exlicit become_methods on connection

* clarify console encoding comment

* ignore recent SCP errors in integration tests

* Add cmd shell type and added more tests

* Fix some doc issues

* revises windows faq

* add anchors for windows links

* revises windows setup page

* Update changelogs/fragments/windows-ssh.yaml

Co-Authored-By: jborean93 <jborean93@gmail.com>
2019-03-07 16:38:02 -08:00
Sloane Hertel
9687879840
Fix inventory cache interface (#50446)
* Replace InventoryFileCacheModule with a better developer-interface

Use new interface for inventory plugins with backwards compatibility

Auto-update the backing cache-plugin if the cache has changed after parsing the inventory plugin

* Update CacheModules to use the config system and add a deprecation warning if they are being imported directly rather than using cache_loader

* Fix foreman inventory caching

* Add tests

* Add integration test to check that fact caching works normally with cache plugins using ansible.constants and inventory caching provides a helpful error for non-compatible cache plugins

* Add some developer documentation for inventory and cache plugins

* Add user documentation for inventory caching

* Add deprecation docs

* Apply suggestions from docs review

* Add changelog
2019-03-06 12:12:35 -06:00
Alan Rominger
56e3597856 Add keyed_groups feature (#52045)
This implements:
- Allow creating keyed group parents
2019-03-05 12:34:34 -06:00
Matt Davis
4d3a6123d5
Python interpreter discovery (#50163)
* Python interpreter discovery

* No longer blindly default to only `/usr/bin/python`
* `ansible_python_interpreter` defaults to `auto_legacy`, which will discover the platform Python interpreter on some platforms (but still favor `/usr/bin/python` if present for backward compatibility). Use `auto` to always use the discovered interpreter, append `_silent` to either value to suppress warnings.
* includes new doc utility method `get_versioned_doclink` to generate a major.minor versioned doclink against docs.ansible.com (or some other config-overridden URL)

* docs revisions for python interpreter discovery

(cherry picked from commit 5b53c0012ab7212304c28fdd24cb33fd8ff755c2)

* verify output on some distros, cleanup
2019-02-27 23:52:02 -08:00
Jordan Borean
9717bc4bcf
Add laps_password lookup for retrieving a Windows LAPS Password (#52012)
* Add laps_password lookup for retrieving a Windows LAPS Password

* Remove python-ldap from requirements and fix doc typos

* enable unit tests without ldap dependency

* use config options for lookup plugin

* Update docs and tests based on further distro testing
2019-02-25 05:30:12 +10:00
Alan Rominger
62feba3d15 support deterministic host ordering from group ancestors (#44067)
Fixes #44065
2019-02-20 10:41:25 -05:00
Matt Clay
f5c92f6bc1
Allow setting resource.RLIMIT_NOFILE in modules (#51989) 2019-02-15 17:52:35 -08:00
Matt Martz
b34d141eed
Disallow use of remote home directories containing .. in their path (CVE-2019-3828) (#52133)
* Disallow use of remote home directories containing .. in their path

* Add CVE to changelog
2019-02-13 10:38:28 -06:00
Matt Martz
c55317a2bd
Update connection plugins to use correct, non-deprecated, methods (#52038)
* Update connection plugins to use correct, non-deprecated, methods

* Update tests to reflect calling become plugins
2019-02-11 13:44:54 -06:00
Matt Martz
445ff39f94
Become plugins (#50991)
* [WIP] become plugins

Move from hardcoded method to plugins for ease of use, expansion and overrides
  - load into connection as it is going to be the main consumer
  - play_context will also use to keep backwards compat API
  - ensure shell is used to construct commands when needed
  - migrate settings remove from base config in favor of plugin specific configs
  - cleanup ansible-doc
  - add become plugin docs
  - remove deprecated sudo/su code and keywords
  - adjust become options for cli
  - set plugin options from context
  - ensure config defs are avaialbe before instance
  - refactored getting the shell plugin, fixed tests
     - changed into regex as they were string matching, which does not work with random string generation
     - explicitly set flags for play context tests
 - moved plugin loading up front
 - now loads for basedir also
 - allow pyc/o for non m modules
 - fixes to tests and some plugins
 - migrate to play objects fro play_context
 - simiplify gathering
 -  added utf8 headers
 - moved option setting
 - add fail msg to dzdo
 - use tuple for multiple options on fail/missing
 - fix relative plugin paths
 - shift from play context to play
 - all tasks already inherit this from play directly
 - remove obsolete 'set play'
 - correct environment handling
 - add wrap_exe option to pfexec
 - fix runas to noop
 - fixed setting play context
 - added password configs
 - removed required false
 - remove from doc building till they are ready

future development:
  - deal with 'enable' and 'runas' which are not 'command wrappers' but 'state flags' and currently hardcoded in diff subsystems

* cleanup

  remove callers to removed func
  removed --sudo cli doc refs
  remove runas become_exe
  ensure keyerorr on plugin
  also fix backwards compat, missing method is attributeerror, not ansible error
  get remote_user consistently
  ignore missing system_tmpdirs on plugin load
  correct config precedence
  add deprecation
  fix networking imports
  backwards compat for plugins using BECOME_METHODS

* Port become_plugins to context.CLIARGS

This is a work in progress:
* Stop passing options around everywhere as we can use context.CLIARGS
  instead

* Refactor make_become_commands as asked for by alikins

* Typo in comment fix

* Stop loading values from the cli in more than one place

Both play and play_context were saving default values from the cli
arguments directly.  This changes things so that the default values are
loaded into the play and then play_context takes them from there.

* Rename BECOME_PLUGIN_PATH to DEFAULT_BECOME_PLUGIN_PATH

As alikins said, all other plugin paths are named
DEFAULT_plugintype_PLUGIN_PATH.  If we're going to rename these, that
should be done all at one time rather than piecemeal.

* One to throw away

This is a set of hacks to get setting FieldAttribute defaults to command
line args to work.  It's not fully done yet.

After talking it over with sivel and jimi-c this should be done by
fixing FieldAttributeBase and _get_parent_attribute() calls to do the
right thing when there is a non-None default.

What we want to be able to do ideally is something like this:

class Base(FieldAttributeBase):
    _check_mode = FieldAttribute([..] default=lambda: context.CLIARGS['check'])

class Play(Base):
    # lambda so that we have a chance to parse the command line args
    # before we get here.  In the future we might be able to restructure
    # this so that the cli parsing code runs before these classes are
    # defined.

class Task(Base):
    pass

And still have a playbook like this function:

---
- hosts:
  tasks:
  - command: whoami
    check_mode: True

(The check_mode test that is added as a separate commit in this PR will
let you test variations on this case).

There's a few separate reasons that the code doesn't let us do this or
a non-ugly workaround for this as written right now.  The fix that
jimi-c, sivel, and I talked about may let us do this or it may still
require a workaround (but less ugly) (having one class that has the
FieldAttributes with default values and one class that inherits from
that but just overrides the FieldAttributes which now have defaults)

* Revert "One to throw away"

This reverts commit 23aa883cbed11429ef1be2a2d0ed18f83a3b8064.

* Set FieldAttr defaults directly from CLIARGS

* Remove dead code

* Move timeout directly to PlayContext, it's never needed on Play

* just for backwards compat, add a static version of BECOME_METHODS to constants

* Make the become attr on the connection public, since it's used outside of the connection

* Logic fix

* Nuke connection testing if it supports specific become methods

* Remove unused vars

* Address rebase issues

* Fix path encoding issue

* Remove unused import

* Various cleanups

* Restore network_cli check in _low_level_execute_command

* type improvements for cliargs_deferred_get and swap shallowcopy to default to False

* minor cleanups

* Allow the su plugin to work, since it doesn't define a prompt the same way

* Fix up ksu become plugin

* Only set prompt if build_become_command was called

* Add helper to assist connection plugins in knowing they need to wait for a prompt

* Fix tests and code expectations

* Doc updates

* Various additional minor cleanups

* Make doas functional

* Don't change connection signature, load become plugin from TaskExecutor

* Remove unused imports

* Add comment about setting the become plugin on the playcontext

* Fix up tests for recent changes

* Support 'Password:' natively for the doas plugin

* Make default prompts raw

* wording cleanups. ci_complete

* Remove unrelated changes

* Address spelling mistake

* Restore removed test, and udpate to use new functionality

* Add changelog fragment

* Don't hard fail in set_attributes_from_cli on missing CLI keys

* Remove unrelated change to loader

* Remove internal deprecated FieldAttributes now

* Emit deprecation warnings now
2019-02-11 11:27:44 -06:00
Brian Coca
50b40c47df aws_ec2 Implement the missing 'region discovery' (#51333)
* aws_ec2 Implement the missing 'region discovery'

  fixes #45288

  tries to use api as documented (which seems to fail in latest boto3 versions)
  and fallback to boto3 'hardcoded' list of regions

* fixes and cleanup, add error for worst case scenario

* fix tests, remove more unused code

* add load_name

* acually load the plugin

* set plugin as required

* reverted test changes, removed options tests

* fixes as per feedback and cleanup
2019-01-29 14:59:38 -06:00
Sam Doran
9d4c0dc111 Catch sshpass authentication errors and don't retry multiple times to prevent account lockout (#50776)
* Catch SSH authentication errors and don't retry multiple times to prevent account lock out

Signed-off-by: Sam Doran <sdoran@redhat.com>

* Subclass AnsibleAuthenticationFailure from AnsibleConnectionFailure

Use comparison rather than range() because it's much more efficient.

Signed-off-by: Sam Doran <sdoran@redhat.com>

* Add tests

Signed-off-by: Sam Doran <sdoran@redhat.com>

* Make paramiko_ssh connection plugin behave the same way

Signed-off-by: Sam Doran <sdoran@redhat.com>

* Add changelog

Signed-off-by: Sam Doran <sdoran@redhat.com>
2019-01-23 11:32:25 -05:00
Martin Krizek
4c72ef124b tqm._initialize_notified_handlers() is noop now (#50713) 2019-01-09 10:15:11 -05:00
Ricardo Carrillo Cruz
f9079274e7
Checkpoint httpapi plugin (#49929)
* Add checkpoint httpapi plugin and access rule facts module

* WIP checkpoint_access_rule module

* Add publish and install policy, plus fix empty json object request for publish

* Refactor publish and install_policy onto module_utils

* Add update resource logic

* Add checkpoint_host_facts module

* Return code and response on get_acess_rule function

* Add checkpoint_host module

* Add checkpoint_run_script module

* Add checkpoint_task_facts module

* Show all tasks if no task id is passed

Note, this is only available on v1.3 of Checkpoint WS API

* Add update logic to checkpoint host

* Add full details on get task call

* Add checkpoint httpapi plugin

* Fix pep8

* Use auth instead of sid property and return False on handle_httperror method

* Fix version in docstring

* Remove constructor

* Remove Accept from base headers

* Do not override http error handler and assign Checkpoint sid to connection _auth

There is scaffolding in the base class to autoappend the token, given
it is assigned to connection _send

* Use new connection queue message method instead of display

* Remove unused display

* Catch ValueError, since it's a parent of JSONDecodeError

* Make static methods that are not used outside the class regular methods

* Add missing self to previously static methods

* Fix logout

Was carrying copy pasta from ftd plugin

* Remove send_auth_request

* Use BASE_HEADERS constant

* Simplify copyright header on httpapi plugin

* Remove access rule module

* Remove unused imports

* Add unit test

* Fix pep8

* Add test

* Add test

* Fix pep8
2019-01-07 14:02:29 +01:00
Toshio Kuratomi
afdbb0d9d5 Save the command line arguments into a global context
* Once cli args are parsed, they're constant.  So, save the parsed args
  into the global context for everyone else to use them from now on.
* Port cli scripts to use the CLIARGS in the context
* Refactor call to parse cli args into the run() method
* Fix unittests for changes to the internals of CLI arg parsing
* Port callback plugins to use context.CLIARGS
  * Got rid of the private self._options attribute
  * Use context.CLIARGS in the individual callback plugins instead.
  * Also output positional arguments in default and unixy plugins
  * Code has been simplified since we're now dealing with a dict rather
    than Optparse.Value
2019-01-03 18:12:23 -08:00
Yannig Perré
6b41588e93 Fix with_ini example and unittest
* Fix example in ini.py
* Fix unittest in test_ini.py to pass CI as latest ansible returns list in
  different order. To prevent such issues in future results are sorted
* PEP8 E501 styling improvements

Co-Authored-By: Sergii Golovatiuk <sgolovat@redhat.com>
2018-12-18 12:25:02 -08:00
Toshio Kuratomi
3fba006207 Update bare exceptions to specify Exception.
This will keep us from accidentally catching program-exiting exceptions
like KeyboardInterupt and SystemExit.
2018-12-16 15:03:19 -08:00
Abhijeet Kasurde
013c42b14f
Misc typo fixes (#49816)
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2018-12-14 15:12:58 +05:30
Matt Martz
2a469fd959 Consolidate handler tracking (#49338)
* Consolidate handler tracking
  -  Remove unused code. ci_complete
  - unit test fixes. ci_complete
  - Restore previous behavior of matching a single handler
  - when notifying a host for a handler, return True if it was added, False otherwise, to reduce copied logic
  - rename funcitons for clarity. ci_complete
  - Remove handler logic for static includes which was disabled previously
2018-12-11 10:58:19 -05:00
Toshio Kuratomi
c817bef3ae Fix for callback plugins on Python3 when a module returns dictionary keys that aren't strings
This fixes one of the problems reported in https://github.com/ansible/ansible/issues/49343

Upstream Python3 bug for the json traceback:
https://bugs.python.org/issue25457

and PR that may fix it:
https://github.com/python/cpython/pull/8011
2018-12-04 23:03:48 -08:00
Lars Kellogg-Stedman
bafc1f8a41 fix constructed functionality in openstack inventory plugin (#48833)
* test for openstack inventory constructed functionality

this adds unit tests for the compose, groups, and keyed_var features
of the openstack inventory plugin

* fix constructed functionality in openstack inventory plugin

The compose, groups, and keyed_groups functionality of the openstack
inventory plugin was broken:

- the plugin was not passing the correct variables to the
  Constructable methods for compose and groups
- the plugin was simply never calling the appropriate method for
  implementing keyed_groups

This commit fixes both issues.
2018-11-27 13:01:56 -05:00
Anton Nikulin
9770ac70f9 FTD modules: upsert functionality and bug fixes (#47747)
* FTD modules: bug fixes and upsert functionality

* Fix sanity checks

* Fix unit tests for Python 2.6

* Log status code for login/logout

* Use string formatting in logging
2018-11-16 11:55:36 +05:30
Jack Haden-Enneking
15e77ebfe8 ipaddr filter - return a network address when given an address with /32 subnet (#47539)
* fix #17872

* add changelog fragment and enable unit test
2018-11-08 11:49:42 -05:00
Michael Cassaniti
a2f3f16930 win_updates: Add post search category matching to support product matching (#45708)
* win_update: Add post search category matching to support product matching

* win_updates: Return categories of each update

* win_updates: Documentation fix-up

* win_updates: Adjusted documentation to reflect regex vs sub-string match of post-cat strings

* win_updates: Sped up post-category checking

* win_updates: Updated documentation to suggest querying post-category strings

* win_updates: Simplified saving and checking post-categories

* fixed some issues and added filtered categories to return value

* win_updates: Moved all category matching to occur after initial search

* win_updates: Adjustments to satisfy PowerShell lint checks

* win_updates: Dropped category validation from action plugin

* win_updates: Documentation updates

* win_updates: Fixed plugin unit tests
2018-11-07 19:32:07 +10:00
Ganesh Nalawade
17fea30841
Revert "enhance recv calls in network_cli (#47345)" (#47731)
This reverts commit c649d0ea32.

The change results in deadlock in network_cli while it is
waiting to check the return value of recv_ready() which
was added in this commit to improve performance
2018-10-29 09:50:30 +05:30
f-bor
c649d0ea32 enhance recv calls in network_cli (#47345)
* enhance recv calls in network_cli

* updated network_cli test unit

* enhance recv calls in network_cli

* fix mistake

* better timeout management

* remove exception trigger

* test

* test2

* restore exception and timeout

* ganeshrn's way

* correction

* timeout and exception return
2018-10-26 09:49:17 +05:30
Ganesh Nalawade
335a979f1d
Fix prompt mismatch issue for ios (#47004)
* Fix prompt mismatch issue for ios

Fixes #40884 #44463

*  If the command prompt is matched check if data is
   still pending to be read from buffer.
*  This fix adds a new timer `buffer_read_timeout`
   which will be trigerred after command prompt
   is matched and data is attempted to be read from channel.
   If not data is present of channel the timer will expire
   and response we be returned to calling function.

* Fix unit test failure

* Update to make buffer timeout float

* Update doc and fix review comment

* Fix CI issues

* Update doc

* Fix review comments

* Fix review comments
2018-10-22 21:05:15 +05:30
Matt Clay
37b013aca3
Enable additional pylint rules and resolve issues found. (#47221)
* Resolve unneeded-not.
* Resolve global-at-module-level.
* Resolve useless-import-alias.
* Resolve bad-whitespace.
* Resolve global-variable-not-assigned.
* Resolve logging-not-lazy.
* Resolve comparison-with-itself.
2018-10-18 13:38:08 -07:00
Steve Dodd
ee6ab5d5aa Add support for IOS vlan parsing filter. (#40555)
* Add support for IOS vlan parsing filter.
Example usage below:

{% set parsed_vlans = vlans | vlan_parser %}
switchport trunk allowed vlan {{ parsed_vlans[0] }}
{% for i in range (1, parsed_vlans | count) %}
switchport trunk allowed vlan add {{ parsed_vlans[i] }}

* Update test_network.py

Add import statement for filter

* Fixed PEP8 issues relating to comments

* Fix PEP8 issues related to blank lines

* Removed magic numbers for line lengths. This should generalize support
to other IOS-like NOS that use similar methods for listing vlans. The
default arguments for line lengths will still be specific to Cisco IOS.
The unit tests for line length are still specific to Cisco IOS.
2018-10-17 20:50:28 +05:30
Fran Fitzpatrick
ab8ed2f84d Add unit tests for junos terminal (#47103) 2018-10-16 15:14:38 +05:30
Matt Clay
3033fd96b0
Move unit test compat code out of lib/ansible/. (#46996)
* Move ansible.compat.tests to test/units/compat/.
* Fix unit test references to ansible.compat.tests.
* Move builtins compat to separate file.
* Fix classification of test/units/compat/ dir.
2018-10-12 20:01:14 -07:00
lwm
8d0f823de0 Add a Linode v4 dynamic inventory plugin. (#45902)
* Add a Linode v4 dynamic inventory plugin.

Closes https://github.com/ansible/ansible/issues/44721.

* Use the latest API for accessing host variables.

References:

  * https://github.com/linode/linode_api4-python/issues/141

* Minor docs formating
2018-10-09 12:54:31 +01:00
Brian Coca
d3d812b604 avoid tests when ncclient is not installed 2018-10-01 13:43:26 -04:00
Brian Coca
32ec69d827
allow jinja2 unique filter compat (#45637)
* allow jinja2 unique filter compat
* detect if unique is provided, fallback with warning
* handle j2 specific params
* now all filters using unique must pass environment
* added env to tests

also normalized on how we normally import and use exceptoins
2018-09-25 14:27:02 -04:00
Sam Doran
b60854357b
Unify terms and UI between 1Password lookups and facts module (#45427)
* Unify login behavior between 1Password lookup plugins and module

- Use the same names for all credential aspects
- Only require the minimal amount of information for each
- Add more examples

* Change parameter terms

- use terms in line with 1Password documentation.
- update examples
- update tests

* Improve error messages in lookup plugin

* Unify onepassword_facts with lookup plugins

- use same methods and logic for signing in or reusing existing session
- unify terms with lookup plugins

* Change rc test for determing login

An rc other than 1 can be returned when a current login session does not exist.

* Create AnsibleModuleError class

ansible.errors is not available to modules, so create an AnsibleModuleError class within the module

Do not user os.path.expanduser since this is already done by virtue of the type being "path" in the argument spec.

* Add note about risk with fact caching sensitive data

* Add note on op version that was used for testing
2018-09-21 14:26:05 -04:00
Matt Martz
95e77ac853
Ensure that the src file contents is converted to unicode in diff info (#45744)
* Ensure that the src file contents is converted to unicode in diff info. Fixes #45717

* Fix up and cleanup

* The diff functionality in the callback plugins should have the
  to_text() calls removed since we're now doing it in ActionBase
* catching of UnicodeError and warnings in the callback diff
  functionality from 61d01f549f haven't been
  needed since we switched to to_text so remove them.
* Add a note to ActionBase's diff function giving an example of when the
  diff function will be inaccurate and how to fix it

* Fix callback get_diff() tests

I believe the unittests of callback's get_diff() were wrong.  They were
sending in a list where strings were expected.  Because previous code
was transforming the lists into strings via their repr, the previous
tests did not fail but they would have formatted the test cases output
in an odd way if we had looked at it.
2018-09-20 12:31:48 -05:00
Nathaniel Case
406b59aeba
Move persistent connections to only use registered variables (#45616)
* Try to intuit proper plugins to send to ansible-connection

* Move sub-plugins to init so that vars will be populated in executor

* Fix connection unit tests
2018-09-20 09:56:43 -04:00
Anton Nikulin
b7263eab1d Store Authorization header inside HttpApi connection plugin (#45598) 2018-09-13 19:34:30 +05:30
Jon Dufresne
ef67d4074b Prefer io.BytesIO over six; available on all supported Pythons (#45388)
On all supported Pythons, the io.BytesIO is always a stream
implementation using an in-memory bytes buffer. Makes code slightly more
forward compatible by reducing use of the six module.
2018-09-10 15:19:16 -04:00
Matt Clay
f7979e4938 Skip ipaddr test that fails on Python 3.7+. 2018-09-04 09:23:31 -07:00
Ganesh Nalawade
c0326aea2f
Fix cli_command multiple prompt issue (#44922)
* Add check in network_cli to handle all prompts

* Add check_all flag to mandatory handle all the command prompt
  in prompts list. By default if any one prompt is handled
  remaining prompts are ignored.

* Fix cli_command multiple prompt issue

* If multiple prompt and answers are given as input network_cli
  handles only the first prompt that matched by default
* If a command execution results in muliple prompt the fix
  add support to set a boolean option C(check_all) to indicate
  network_cli to wait till all the prompts and answers are processed.

* Update cli_command

* Update api doc

* Fix unit test failure

* Fix CI failure

* Update network_cli

* Fix review comment
2018-08-31 20:04:12 +05:30
Deepak Agrawal
50c7702e46
cisco firepower : Make API endpoints configurable via hostvars (#44952)
* httpapi host vars

* Make configurable end-points for firepower

* pep8 fix
2018-08-31 18:38:16 +05:30
Sam Doran
8cd8d17980
Add ability to unlock 1Password vault to lookup plugins (#44923)
* Add ability to use login to 1Password vault to 1Password lookups

* Adjust unit tests

* Add changelog
2018-08-30 16:24:06 -04:00
pierremahot
d11e078366 adding an option for the filter ipsubnet and testing (#40670)
* adding an option for the filter ipsubnet
2018-08-30 13:21:22 +01:00
Anton Nikulin
40a97d43d1 New modules and updated HTTP API plugin for FTD devices (#44578)
* Add common and Swagger client utils for FTD modules

* Update FTD HTTP API plugin and add unit tests for it

* Add configuration layer handling object idempotency

* Add ftd_configuration module with unit tests

* Add ftd_file_download and ftd_file_upload modules with unit tests

* Validate operation data and parameters

* Fix ansible-doc, boilerplate and import errors

* Fix pip8 sanity errors

* Update object comparison to work recursively

* Add copyright
2018-08-29 15:48:10 +02:00
Matthias Fuchs
7871027c9d Share the implementation of hashing for both vars_prompt and password_hash (#21215)
* Share the implementation of hashing for both vars_prompt and password_hash.
* vars_prompt with encrypt does not require passlib for the algorithms
  supported by crypt.
* Additional checks ensure that there is always a result.
  This works around issues in the crypt.crypt python function that returns
  None for algorithms it does not know.
  Some modules (like user module) interprets None as no password at all,
  which is misleading.
* The password_hash filter supports all parameters of passlib.
  This allows users to provide a rounds parameter, fixing #15326.
* password_hash is not restricted to the subset provided by crypt.crypt,
  fixing one half of #17266.
* Updated documentation fixes other half of #17266.
* password_hash does not hard-code the salt-length, which fixes bcrypt
  in connection with passlib.
  bcrypt requires a salt with length 22, which fixes #25347
* Salts are only generated by ansible when using crypt.crypt.
  Otherwise passlib generates them.
* Avoids deprecated functionality of passlib with newer library versions.
* When no rounds are specified for sha256/sha256_crypt and sha512/sha512_crypt
  always uses the default values used by crypt, i.e. 5000 rounds.
  Before when installed passlibs' defaults were used.
  passlib changes its defaults with newer library versions, leading to non
  idempotent behavior.

  NOTE: This will lead to the recalculation of existing hashes generated
        with passlib and without a rounds parameter.
        Yet henceforth the hashes will remain the same.
        No matter the installed passlib version.
        Making these hashes idempotent.

Fixes #15326
Fixes #17266
Fixes #25347 except bcrypt still uses 2a, instead of the suggested 2b.

* random_salt is solely handled by encrypt.py.
  There is no _random_salt function there anymore.
  Also the test moved to test_encrypt.py.
* Uses pytest.skip when passlib is not available, instead of a silent return.
* More checks are executed when passlib is not available.

* Moves tests that require passlib into their own test-function.

* Uses the six library to reraise the exception.

* Fixes integration test.

When no rounds are provided the defaults of crypt are used.
In that case the rounds are not part of the resulting MCF output.
2018-08-27 08:40:41 -07:00
jctanner
653d9c0f87 New keyword: ignore_unreachable (#43857) 2018-08-23 11:41:02 -04:00
Jordan Borean
6982dfc756 psrp: Added new Windows connection plugin (#41729)
* psrp: Added new Windows connection plugin

* Tweaks to connection options from review
2018-08-21 16:43:13 -07:00
Artem Leshchev
8323f3e3b7 Fix module name in error text at parse_xml filter (#44313)
* Fix module name in error text at parse_xml filter

* Fix test that includes erroneous module name in error text at parse_xml filter
2018-08-20 10:19:16 +05:30
Zhikang Zhang
0971a342d8
Solve race condition in password lookup (#42529)
NOTE:
1. use os.open() with os.O_CREAT|os.O_EXCL to check existence
and create a lock file if not exists, it's an atomic operation
2. the fastest process will create the lock file and others will
wait until the lock file is removed
3. after the writer finished writing to the password file, all the reading
operations use built-in open so processes can read the file parallel
2018-08-15 15:10:52 -04:00
Lindsay Hill
5981a7489b new nos_command module (#43056) 2018-08-10 13:50:02 -07:00
Zhikang Zhang
3a3869f4c8
Clarify usage of encrypt arg (#43330)
According to the do_encrypt interface, encrypt arg should be the hash method name used for encrypting returning password. But in the doc and lookup code it's a boolean flag, correct it to string.
2018-08-02 14:03:49 -04:00
Joren Vrancken
b954917761 Surround top-level function and class definitions with two blank lines. 2018-07-31 12:06:56 -07:00
Fran Fitzpatrick
5928ec04ca Adds an ipmath filter (#41985) 2018-07-11 10:52:31 -07:00
Jordan Borean
940d4a0e89
win_reboot: fix 2.6 issues and better handle post reboot reboot (#42330)
* win_reboot: fix 2.6 issues and better handle post reboot reboot

* changed winrm _reset to reset

* Add handler to reset calls when .reset() throws an AnsibleError on older hosts

* Moving back to _reset to get the issue fixed
2018-07-11 09:12:29 +10:00
Jordan Borean
8bdd04c147 Fix remote_tmp when become with non admin user (#42396)
* Fix tmpdir on non root become

 - also avoid exception if tmpdir and remote_tmp are None
 - give 'None' on deescalation so tempfile will fallback to it's default behaviour
   and use system dirs
 - fix issue with bad tempdir (not existing/not createable/not writeable)
   i.e nobody and ~/.ansible/tmp
 - added tests for blockfile case

* Revert "Temporarily revert c119d54"

This reverts commit 5c614a59a6.

* changes based on PR feedback and changelog fragment

* changes based on the review

* Fix tmpdir when makedirs failed so we just use the system tmp

* Let missing remote_tmp fail

If remote_tmp is missing then there's something more basic wrong in the
communication from the controller to the module-side.  It's better to
be alerted in this case than to silently ignore it.

jborean and I have independently checked what happens if the user sets
ansible_remote_tmp to empty string and !!null and both cases work fine.
(null is turned into a default value controller-side.  empty string
triggers the warning because it is probably not a directory that the
become user is able to use).
2018-07-06 10:49:19 -07:00
Nathaniel Case
5850591d14
Base connection class for network-style connections (#41839)
* Create base class for network-style connections

* clean up some differences

* Move NetworkConnectionBase

* Tweak netconf for tests

* Tweak when network_os is checked to avoid failing tests

* Pull back exec_command
2018-07-02 08:41:00 -04:00
Jordan Borean
9b7b564d75
Stop displaying kinit pass input on a failure (#41882)
* Stop displaying kinit pass input on a failure

* Fixed up minor logic info and added tests
2018-06-26 05:49:19 +10:00
Brian Coca
27c43daab8
fix minor issues with debug and item labels (#41331)
* fix minor issues with debug and item labels

 - no more `item=None`, we always have a label now
 - debug should only show expected information, either msg= or the var in var=
 - also fixed method name, deprecated misleading _get_item
2018-06-21 16:14:57 -04:00
Jordan Borean
11bd3fd318 win_updates fix when win_updates is run with async (#41756) 2018-06-20 14:52:24 -07:00
Matt Davis
b01779ad18
skip winrm unit tests if winrm is not installed (#41596) 2018-06-15 10:23:15 -07:00
Jordan Borean
ad8e13e9f8
winrm: fix up unit tests (#41112) 2018-06-07 06:09:10 +08:00
Martin Krizek
77e2bd5f8e
Fix inserting noop tasks by using active state (#36056) 2018-06-06 12:34:25 +02:00
Ganesh Nalawade
ba4b12358c
Refactor ios cliconf plugin and ios_config module (#39695)
* Refactor ios cliconf plugin and ios_config module

*  Refactor ios cliconf plugin to support generic network_config module
*  Refactor ios_config module to work with cliconf api's
*  Enable command and response logging in cliconf pulgin
*  cliconf api documentation

* Fix unit test and other minor changes

* Doc update

* Fix CI failure

* Add default flag related changes

* Minor changes

* redact input command logging by default
2018-06-06 11:12:45 +05:30
Matt Martz
ad7ba91f75 Use _remote_is_local=True for local connection in synchronize (#40833)
* All instances of local connection should use _remote_is_local=True. Fixes #40551

* Switch to instance attribute for synchronize

* Add test that shows that synchronize _remote_is_local addresses tmpdir building
2018-05-30 13:28:11 -07:00
Sloane Hertel
cba64f5869 Fix inventory plugin cache + add tests (#38229)
* Fix setting the cache when refresh_cache or --flush-cache are used

* Use jsonify function that handles datetime objects in jsonfile cache plugin

* Don't access self._options directly

* Add initial integration tests for aws_ec2 inventory plugin

* Add CI alias

* Fix and add a few more unit tests

* Add integration tests for constructed

* Fix typo

* Use inventory config templates

* Collect all instances that are not terminated by default

* Create separate playbook for setting up the VPC, subnet, security group, and finding an image for the host

Create a separate playbook for removing the resources

* Allow easier grouping by region and add an example

* use a unified json encode/decode that can handle unsafe and vault
2018-05-24 15:34:54 -04:00
Jordan Borean
5e28e282a5 winrm: add better exception handling for krb5 auth with pexpect (#39930)
* winrm: add better exception handling for krb5 auth with pexpect

* Added changelog fragment

* Added exception handler in case kinit path isn't valid, added test cases

* fixed for Python 2 compatibility
2018-05-23 13:33:07 -07:00
Jordan Borean
457bccf540 win_updates: add scheduled tasks back in for older hosts (#38708)
* win_updates: add scheduled tasks back in for older hosts

* Fixed up typo in category name error message

* Fixed up some minor issues after merge

* added changelog fragment

* Default to become but add override to use scheduled tasks

* Added basic unit tests for win_updates

* fix minor typos
2018-05-23 13:21:01 -07:00
flowerysong
e2c1589201 Add cidr_merge filter (#36081) 2018-05-23 14:35:23 -05:00
Ken Celenza
dd02a4e943 create cisco type 5 filters (#39901) 2018-05-22 13:35:54 +01:00
Jordan Borean
1ac180c74d
winrm: source user from options than remote_user (#40467)
* winrm: source user from options than remote_user

* fixed up mock for kerberos import

* Added changelog fragment

* get hostname from option as well
2018-05-22 09:36:48 +10:00
Scott Buchanan
b12cf754f6 1Password lookup plugin (#37207)
* add pytest_cache to gitignore

* onepassword lookup plugin

* fix linter/style test complaints

* second pass at making pycodestyle happy

* use json module instead of jq

* update copyrights, license & version added

* fix python2 compatibility

* doh. fix spacing issue.

* use standard ansible exception

* remove potentially problematic stdin argument

* actually call assertion method

* add support for top-level fields

* make vault uuids pedantically consistent in fixture

* fix new style issues

* ability specify section & correct case handling

* improve error handling

* add onepassword_raw plugin

* Add maintainer info

* Move common code to module_utils/onepassword.py

* Load raw data JSON data for easier use in Ansible

* Put OnePass class back inside lookup plugin

There is no good place for sharing code across lookups currently.

* Remove debugging code in unit tests

* Patche proper module in raw unit tests

* Add changelog entry

Co-authored-by: Scott Buchanan <sbuchanan@ri.pn>
2018-05-17 17:29:47 -04:00
Ricardo Carrillo Cruz
62e1c14edc
Pull persistent connection parameters via get_option (#39367)
* WIP Pull persistent connection parameters via get_option

* Fix pep8

* Add use_persistent_connection setting to paramiko_ssh plugin

* Add vars section to persistent_command_timeout setting and prevail provider values over config manager

* Use persistent_command_timeout on network_cli instead of timeout

* Fix unit tests

If we don't call loader to get network_cli, then _load_name is never
set and we get KeyError.

* Pull persistent_command_timeout via config  manager for ios connection local

* Pull persistent_command_timeout via config manager on connection local
2018-05-16 14:59:01 +02:00
Jordan Borean
44ab948e5d
create module tmpdir based on remote_tmp (#39833)
* create module tmpdir based on remote_tmp

* Source remote_tmp from controller if possible

* Fixed sanity test and not use lambda

* Added expansion of env vars to the remote tmp

* Fixed sanity issues

* Added note around shell remote_tmp option

* Changed fallback tmp dir to ~/.ansible/tmp to make shell defaults
2018-05-15 09:31:21 +10:00
Brian Coca
4aac0f5f18
allow configuring sftp/scp executables (#36648)
* allow configuring sftp/scp executables

fixes #36616


also removed dupe test
2018-04-23 18:36:35 -04:00
Matthew Stone
5b1a8ee1e8 Adding slx_command module and supporting module_utils. (#38235)
* Adding slx_command module and supporting module_utils.

This commit adds the slx_command module and tests as well as the
required slxos module_utils.

* Update copyright in header

* Adding missing module init

* Cleaning up shebangs/licensing.

* Incorporating feedback

Removing reference to `waitfor` alias in `slxos_command` module.
Adding `Extreme Networks` to `short_description` of `slxos_command` module.

* Adding cliconf tests

* Fixing 3.X tests

* Adding docstrings to test methods for slxos cliconf tests

* Adding slxos terminal tests

* Adding slxos module_utils tests

* Adding Extreme Networks team members to BOTMETA.yml
2018-04-11 12:59:34 +01:00
AlanCoding
153c9bd539 Reduce recursion within group methods
This offers an optimization that allows loading larger
inventories of various structure by improving the
scaling laws involved for adding hosts and groups.

The primary speed benefit is the elimination of duplicate
recusion from traversing converging paths.
2018-04-05 18:38:44 -04:00
Izuke
e02c72d856 Fixes #32146 Fixes reconcile inventory to correctly remove hosts from ungrouped 2018-03-20 15:37:39 -04:00
Nathaniel Case
594840c1d6
Close & remove paramiko connection where appropriate (#37528)
* Close & remove paramiko connection where appropriate

* Update unit test
2018-03-16 13:28:04 -04:00
mikedlr
52f2057472 aws ssm parameter lookup - change to return Nones for missing variables (#36456)
* aws ssm parameter lookup - change to return Nones for missing variables

* aws ssm parameter lookup - fix error case message to dump response

* aws ssm parameter lookup - fix integration test cases
2018-02-21 06:50:00 -05:00
Brian Coca
a79378fccb remove extra fields from debug output
fixes #35493

updated tests
2018-02-20 08:46:05 -05:00
Toshio Kuratomi
06f73ad578
Normalize usage of temp and tmp on tmp (#36221)
* Normalize usage of temp and tmp on tmp
* Rename system_tmps system_tmpdirs
* Add ANSIBLE_REMOTE_TMP spelling of environment variables
2018-02-15 09:01:02 -08:00
Brian Coca
8743b20c8e skip aws inventory test if no boto 2018-02-13 11:04:37 -05:00
mikedlr
d31ded47fb Aws ssm multiple fixes (#35569)
* aws ssm parameter lookup test case - fails demonstrating no exception when parameter missing

* aws ssm parameter lookup - fail in case parameter doesn't exist

* aws ssm parameter lookup test case - failing case for nice return from path lookup

* aws ssm parameter lookup - convert incoming taglist to a key-value dictionary

* aws ssm parameter lookup - pep8 / style clean up

* aws_ssm lookup plugin rewrite for more standard interface

* aws_ssm module and lookup - introduce integration test and fix:

* aws_ssm module and lookup - error case integraton test and many PEP8 and other cleanups

* aws ssm parameter lookup - Various fixes in response to review + recursive fix & test

* aws ssm parameter lookup - more in response to review - shertel/abadger

* aws ssm parameter lookup unit test - move to mocker according to abadger

* aws ssm parameter lookup - integrate with new documentation fragment

* aws ssm parameter lookup - accept either aws_profile or boto_profile

* aws ssm parameter lookup - eliminate lookup document fragment until env vars are fixed later
2018-02-06 17:41:46 -05:00
Jason Vanderhoof
7c8e365dff Conjur Lookup Plugin (#34280)
* Imported lookup plugin from Role

* Plugin cleanup, including:
* Use existing Python YAML parsing
* Remove environment variables as connection options
* Added initial debugging information

* Reworked the lookup plugin using the Python Request library.  As it's available through Ansible, it makes communication with Conjur much more straight forward.

* Removed un-used libraries

* Fixed linting issues

* Standardized output on `format` and insure it works for 2.6, 2.7, and 3.x.

* Use quote_plus from the six library for improved python 2/3 behavior.

* Refactored identity & configuration to prefer user's file. This also includes a refactor to remove an un-needed dictionary merge method.

* Removed `requests` in favor of `ansible.module_utils.urls`.

* Refactored netrc loading to warn if host is not present.

* Tests and a refactor to support easier testing.

* Added reference to website

* Fixed two linting errors

* Fixed an extra line found by linting

* Updated file write to use binary to insure config files are written correctly

* Resolved linting issues

* Refactored config & identity loading to take advantage of plugin options

* Cleanup a bunch of small items caught by linting

* Removed extra line caught by linting

* Swapped in pytest and added some tests with mocked network responses

* Pushing to see if this approach works better...

* Refactored be open_url mocking based on feedback

* Fixed a couple linting issues & refactored mocking into each method to attempt to resolve a failing test

* Use a generic MagicMock for python 2.6

* Fixes doc typo

require -> required

* Use `type: path` in identity_file and config_file

Also removes `expanduser` calls below (which will now be called automatically on
paths.)

* Defines maintainers for conjur_variable plugin

* BOTMETA.yml:
** defines $team_cyberark_conjur as maintainers of Conjur Variable plugin
** adds myself and @jvanderhoof to that team

* Adds URLs to relevant documentation for Conjur Variable lookup plugin

* Clarifies "the server," "the machine" -> "controlling host"

The machine identity used is that of the Ansible controlling host, not any
server being provisioned or instructed. This documentation change aims to make
that relationship clear.

* Adds response code to exception message on authentication failure

* Enhances exception messages to specify the controlling host

These error messages are less likely to confuse a user as to which machine is
associated with the files, identities, and configurations being described.

* Adds ANSIBLE_METADATA for Conjur variable lookup plugin
2018-01-23 10:04:57 -06:00
Toshio Kuratomi
8c7bd8beb4 Set the tempdir on the shell mock object so that it is json serializable 2018-01-22 23:36:34 -08:00
Sloane Hertel
b94198f9de [cloud] New inventory plugin: aws_ec2 (#34416)
aws_ec2 is an inventory plugin intended to replace uses of the `contrib/inventory/ec2.py`
inventory script. 

For advanced grouping, please use the `constructed` inventory plugin.
2018-01-22 18:49:57 -05:00
Matt Martz
29c1d5cb5d Ensure that the become password is written on py3 in the ssh connection plugin. Fixes #34727 2018-01-18 09:32:22 -08:00
Brian Coca
bbd6b8bb42 Temporary (#31677)
* allow shells to have per host options, remote_tmp

added language to shell
removed module lang setting from general as  plugins have it now
use get to avoid bad powershell plugin
more resilient tmp discovery, fall back to `pwd`
add shell to docs
fixed options for when frags are only options
added shell set ops in t_e and fixed option frags
normalize tmp dir usag4e

- pass tmpdir/tmp/temp options as env var to commands, making it default for tempfile
- adjusted ansiballz tmpdir
- default local tempfile usage to the configured local tmp
- set env temp in action

add options to powershell
shift temporary to internal envvar/params
ensure tempdir is set if we pass var
ensure basic and url use expected tempdir
ensure localhost uses local tmp
give /var/tmp priority, less perms issues
more consistent tempfile mgmt for ansiballz
made async_dir configurable
better action handling, allow for finally rm tmp
fixed tmp issue and no more tempdir in ballz
hostvarize world readable and admin users
always set shell tempdir
added comment to discourage use of exception/flow control

* Mostly revert expand_user as it's not quite working.

This was an additional feature anyhow.

Kept the use of pwd as a fallback but moved it to a second ssh
connection.  This is not optimal but getting that to work in a single
ssh connection was part of the problem holding this up.

(cherry picked from commit 395b714120522f15e4c90a346f5e8e8d79213aca)

* fixed script and other action plugins

ensure tmpdir deletion
allow for connections that don't support new options (legacy, 3rd party)
fixed tests
2018-01-15 21:15:04 -08:00
James Cammarata
b107e397cb
Cache tasks as they are queued instead of en masse (#34752)
* Cache tasks as they are queued instead of en masse

This also moves the task caching from the PlayIterator to the
StrategyBase class, where it makes more sense (and makes it easier
to not have to change the strategy class methods leading to an API
change).

Fixes #31673

* Cleaning up unit tests due to 502ca780
2018-01-15 09:49:58 -06:00
Matt Martz
d1846425db
Provide a way to explicitly invoke the debugger (#34006)
* Provide a way to explicitly invoke the debugger with in the debug strategy

* Merge the debugger strategy into StrategyBase

* Fix some logic, pin to a single result

* Make redo also continue

* Make sure that if the debug closure doesn't need to process the result, that we still return it

* Fix failing tests for the strategy

* Clean up messages from debugger and exit code to match bin/ansible

* Move the FieldAttribute higher, to apply at different levels

* make debugger a string, expand logic

* Better host state rollbacks

* More explicit debugger prompt

* ENABLE_TASK_DEBUGGER should be boolean, and better docs

* No bare except, add pprint, alias h, vars to task_vars

* _validate_debugger can ignore non-string, that can be caught later

* Address issue if there were no previous tasks/state, and use the correct key

* Update docs for changes to the debugger

* Guard against a stat going negative through use of decrement

* Add a few notes about using the debugger on the free strategy

* Add changelog entry for task debugger

* Add a few versionadded indicators and a note about vars -> task_vars
2018-01-09 13:50:07 -06:00
Matt Martz
aece0818c4
Don't cache duplicate plugin names (#34420)
* Don't cache duplicate plugin names. Fixes #33484

* Add tests for duplicate plugin filenames, to showcase what happens
2018-01-04 16:47:23 -06:00