Commit graph

196 commits

Author SHA1 Message Date
Sloane Hertel
cba64f5869 Fix inventory plugin cache + add tests (#38229)
* Fix setting the cache when refresh_cache or --flush-cache are used

* Use jsonify function that handles datetime objects in jsonfile cache plugin

* Don't access self._options directly

* Add initial integration tests for aws_ec2 inventory plugin

* Add CI alias

* Fix and add a few more unit tests

* Add integration tests for constructed

* Fix typo

* Use inventory config templates

* Collect all instances that are not terminated by default

* Create separate playbook for setting up the VPC, subnet, security group, and finding an image for the host

Create a separate playbook for removing the resources

* Allow easier grouping by region and add an example

* use a unified json encode/decode that can handle unsafe and vault
2018-05-24 15:34:54 -04:00
Jordan Borean
5e28e282a5 winrm: add better exception handling for krb5 auth with pexpect (#39930)
* winrm: add better exception handling for krb5 auth with pexpect

* Added changelog fragment

* Added exception handler in case kinit path isn't valid, added test cases

* fixed for Python 2 compatibility
2018-05-23 13:33:07 -07:00
Jordan Borean
457bccf540 win_updates: add scheduled tasks back in for older hosts (#38708)
* win_updates: add scheduled tasks back in for older hosts

* Fixed up typo in category name error message

* Fixed up some minor issues after merge

* added changelog fragment

* Default to become but add override to use scheduled tasks

* Added basic unit tests for win_updates

* fix minor typos
2018-05-23 13:21:01 -07:00
flowerysong
e2c1589201 Add cidr_merge filter (#36081) 2018-05-23 14:35:23 -05:00
Ken Celenza
dd02a4e943 create cisco type 5 filters (#39901) 2018-05-22 13:35:54 +01:00
Jordan Borean
1ac180c74d
winrm: source user from options than remote_user (#40467)
* winrm: source user from options than remote_user

* fixed up mock for kerberos import

* Added changelog fragment

* get hostname from option as well
2018-05-22 09:36:48 +10:00
Scott Buchanan
b12cf754f6 1Password lookup plugin (#37207)
* add pytest_cache to gitignore

* onepassword lookup plugin

* fix linter/style test complaints

* second pass at making pycodestyle happy

* use json module instead of jq

* update copyrights, license & version added

* fix python2 compatibility

* doh. fix spacing issue.

* use standard ansible exception

* remove potentially problematic stdin argument

* actually call assertion method

* add support for top-level fields

* make vault uuids pedantically consistent in fixture

* fix new style issues

* ability specify section & correct case handling

* improve error handling

* add onepassword_raw plugin

* Add maintainer info

* Move common code to module_utils/onepassword.py

* Load raw data JSON data for easier use in Ansible

* Put OnePass class back inside lookup plugin

There is no good place for sharing code across lookups currently.

* Remove debugging code in unit tests

* Patche proper module in raw unit tests

* Add changelog entry

Co-authored-by: Scott Buchanan <sbuchanan@ri.pn>
2018-05-17 17:29:47 -04:00
Ricardo Carrillo Cruz
62e1c14edc
Pull persistent connection parameters via get_option (#39367)
* WIP Pull persistent connection parameters via get_option

* Fix pep8

* Add use_persistent_connection setting to paramiko_ssh plugin

* Add vars section to persistent_command_timeout setting and prevail provider values over config manager

* Use persistent_command_timeout on network_cli instead of timeout

* Fix unit tests

If we don't call loader to get network_cli, then _load_name is never
set and we get KeyError.

* Pull persistent_command_timeout via config  manager for ios connection local

* Pull persistent_command_timeout via config manager on connection local
2018-05-16 14:59:01 +02:00
Jordan Borean
44ab948e5d
create module tmpdir based on remote_tmp (#39833)
* create module tmpdir based on remote_tmp

* Source remote_tmp from controller if possible

* Fixed sanity test and not use lambda

* Added expansion of env vars to the remote tmp

* Fixed sanity issues

* Added note around shell remote_tmp option

* Changed fallback tmp dir to ~/.ansible/tmp to make shell defaults
2018-05-15 09:31:21 +10:00
Brian Coca
4aac0f5f18
allow configuring sftp/scp executables (#36648)
* allow configuring sftp/scp executables

fixes #36616


also removed dupe test
2018-04-23 18:36:35 -04:00
Matthew Stone
5b1a8ee1e8 Adding slx_command module and supporting module_utils. (#38235)
* Adding slx_command module and supporting module_utils.

This commit adds the slx_command module and tests as well as the
required slxos module_utils.

* Update copyright in header

* Adding missing module init

* Cleaning up shebangs/licensing.

* Incorporating feedback

Removing reference to `waitfor` alias in `slxos_command` module.
Adding `Extreme Networks` to `short_description` of `slxos_command` module.

* Adding cliconf tests

* Fixing 3.X tests

* Adding docstrings to test methods for slxos cliconf tests

* Adding slxos terminal tests

* Adding slxos module_utils tests

* Adding Extreme Networks team members to BOTMETA.yml
2018-04-11 12:59:34 +01:00
AlanCoding
153c9bd539 Reduce recursion within group methods
This offers an optimization that allows loading larger
inventories of various structure by improving the
scaling laws involved for adding hosts and groups.

The primary speed benefit is the elimination of duplicate
recusion from traversing converging paths.
2018-04-05 18:38:44 -04:00
Izuke
e02c72d856 Fixes #32146 Fixes reconcile inventory to correctly remove hosts from ungrouped 2018-03-20 15:37:39 -04:00
Nathaniel Case
594840c1d6
Close & remove paramiko connection where appropriate (#37528)
* Close & remove paramiko connection where appropriate

* Update unit test
2018-03-16 13:28:04 -04:00
mikedlr
52f2057472 aws ssm parameter lookup - change to return Nones for missing variables (#36456)
* aws ssm parameter lookup - change to return Nones for missing variables

* aws ssm parameter lookup - fix error case message to dump response

* aws ssm parameter lookup - fix integration test cases
2018-02-21 06:50:00 -05:00
Brian Coca
a79378fccb remove extra fields from debug output
fixes #35493

updated tests
2018-02-20 08:46:05 -05:00
Toshio Kuratomi
06f73ad578
Normalize usage of temp and tmp on tmp (#36221)
* Normalize usage of temp and tmp on tmp
* Rename system_tmps system_tmpdirs
* Add ANSIBLE_REMOTE_TMP spelling of environment variables
2018-02-15 09:01:02 -08:00
Brian Coca
8743b20c8e skip aws inventory test if no boto 2018-02-13 11:04:37 -05:00
mikedlr
d31ded47fb Aws ssm multiple fixes (#35569)
* aws ssm parameter lookup test case - fails demonstrating no exception when parameter missing

* aws ssm parameter lookup - fail in case parameter doesn't exist

* aws ssm parameter lookup test case - failing case for nice return from path lookup

* aws ssm parameter lookup - convert incoming taglist to a key-value dictionary

* aws ssm parameter lookup - pep8 / style clean up

* aws_ssm lookup plugin rewrite for more standard interface

* aws_ssm module and lookup - introduce integration test and fix:

* aws_ssm module and lookup - error case integraton test and many PEP8 and other cleanups

* aws ssm parameter lookup - Various fixes in response to review + recursive fix & test

* aws ssm parameter lookup - more in response to review - shertel/abadger

* aws ssm parameter lookup unit test - move to mocker according to abadger

* aws ssm parameter lookup - integrate with new documentation fragment

* aws ssm parameter lookup - accept either aws_profile or boto_profile

* aws ssm parameter lookup - eliminate lookup document fragment until env vars are fixed later
2018-02-06 17:41:46 -05:00
Jason Vanderhoof
7c8e365dff Conjur Lookup Plugin (#34280)
* Imported lookup plugin from Role

* Plugin cleanup, including:
* Use existing Python YAML parsing
* Remove environment variables as connection options
* Added initial debugging information

* Reworked the lookup plugin using the Python Request library.  As it's available through Ansible, it makes communication with Conjur much more straight forward.

* Removed un-used libraries

* Fixed linting issues

* Standardized output on `format` and insure it works for 2.6, 2.7, and 3.x.

* Use quote_plus from the six library for improved python 2/3 behavior.

* Refactored identity & configuration to prefer user's file. This also includes a refactor to remove an un-needed dictionary merge method.

* Removed `requests` in favor of `ansible.module_utils.urls`.

* Refactored netrc loading to warn if host is not present.

* Tests and a refactor to support easier testing.

* Added reference to website

* Fixed two linting errors

* Fixed an extra line found by linting

* Updated file write to use binary to insure config files are written correctly

* Resolved linting issues

* Refactored config & identity loading to take advantage of plugin options

* Cleanup a bunch of small items caught by linting

* Removed extra line caught by linting

* Swapped in pytest and added some tests with mocked network responses

* Pushing to see if this approach works better...

* Refactored be open_url mocking based on feedback

* Fixed a couple linting issues & refactored mocking into each method to attempt to resolve a failing test

* Use a generic MagicMock for python 2.6

* Fixes doc typo

require -> required

* Use `type: path` in identity_file and config_file

Also removes `expanduser` calls below (which will now be called automatically on
paths.)

* Defines maintainers for conjur_variable plugin

* BOTMETA.yml:
** defines $team_cyberark_conjur as maintainers of Conjur Variable plugin
** adds myself and @jvanderhoof to that team

* Adds URLs to relevant documentation for Conjur Variable lookup plugin

* Clarifies "the server," "the machine" -> "controlling host"

The machine identity used is that of the Ansible controlling host, not any
server being provisioned or instructed. This documentation change aims to make
that relationship clear.

* Adds response code to exception message on authentication failure

* Enhances exception messages to specify the controlling host

These error messages are less likely to confuse a user as to which machine is
associated with the files, identities, and configurations being described.

* Adds ANSIBLE_METADATA for Conjur variable lookup plugin
2018-01-23 10:04:57 -06:00
Toshio Kuratomi
8c7bd8beb4 Set the tempdir on the shell mock object so that it is json serializable 2018-01-22 23:36:34 -08:00
Sloane Hertel
b94198f9de [cloud] New inventory plugin: aws_ec2 (#34416)
aws_ec2 is an inventory plugin intended to replace uses of the `contrib/inventory/ec2.py`
inventory script. 

For advanced grouping, please use the `constructed` inventory plugin.
2018-01-22 18:49:57 -05:00
Matt Martz
29c1d5cb5d Ensure that the become password is written on py3 in the ssh connection plugin. Fixes #34727 2018-01-18 09:32:22 -08:00
Brian Coca
bbd6b8bb42 Temporary (#31677)
* allow shells to have per host options, remote_tmp

added language to shell
removed module lang setting from general as  plugins have it now
use get to avoid bad powershell plugin
more resilient tmp discovery, fall back to `pwd`
add shell to docs
fixed options for when frags are only options
added shell set ops in t_e and fixed option frags
normalize tmp dir usag4e

- pass tmpdir/tmp/temp options as env var to commands, making it default for tempfile
- adjusted ansiballz tmpdir
- default local tempfile usage to the configured local tmp
- set env temp in action

add options to powershell
shift temporary to internal envvar/params
ensure tempdir is set if we pass var
ensure basic and url use expected tempdir
ensure localhost uses local tmp
give /var/tmp priority, less perms issues
more consistent tempfile mgmt for ansiballz
made async_dir configurable
better action handling, allow for finally rm tmp
fixed tmp issue and no more tempdir in ballz
hostvarize world readable and admin users
always set shell tempdir
added comment to discourage use of exception/flow control

* Mostly revert expand_user as it's not quite working.

This was an additional feature anyhow.

Kept the use of pwd as a fallback but moved it to a second ssh
connection.  This is not optimal but getting that to work in a single
ssh connection was part of the problem holding this up.

(cherry picked from commit 395b714120522f15e4c90a346f5e8e8d79213aca)

* fixed script and other action plugins

ensure tmpdir deletion
allow for connections that don't support new options (legacy, 3rd party)
fixed tests
2018-01-15 21:15:04 -08:00
James Cammarata
b107e397cb
Cache tasks as they are queued instead of en masse (#34752)
* Cache tasks as they are queued instead of en masse

This also moves the task caching from the PlayIterator to the
StrategyBase class, where it makes more sense (and makes it easier
to not have to change the strategy class methods leading to an API
change).

Fixes #31673

* Cleaning up unit tests due to 502ca780
2018-01-15 09:49:58 -06:00
Matt Martz
d1846425db
Provide a way to explicitly invoke the debugger (#34006)
* Provide a way to explicitly invoke the debugger with in the debug strategy

* Merge the debugger strategy into StrategyBase

* Fix some logic, pin to a single result

* Make redo also continue

* Make sure that if the debug closure doesn't need to process the result, that we still return it

* Fix failing tests for the strategy

* Clean up messages from debugger and exit code to match bin/ansible

* Move the FieldAttribute higher, to apply at different levels

* make debugger a string, expand logic

* Better host state rollbacks

* More explicit debugger prompt

* ENABLE_TASK_DEBUGGER should be boolean, and better docs

* No bare except, add pprint, alias h, vars to task_vars

* _validate_debugger can ignore non-string, that can be caught later

* Address issue if there were no previous tasks/state, and use the correct key

* Update docs for changes to the debugger

* Guard against a stat going negative through use of decrement

* Add a few notes about using the debugger on the free strategy

* Add changelog entry for task debugger

* Add a few versionadded indicators and a note about vars -> task_vars
2018-01-09 13:50:07 -06:00
Matt Martz
aece0818c4
Don't cache duplicate plugin names (#34420)
* Don't cache duplicate plugin names. Fixes #33484

* Add tests for duplicate plugin filenames, to showcase what happens
2018-01-04 16:47:23 -06:00
Pierre-Louis Bonicoli
737920cd89 YAML inventory unit test: add tests
Relates: #33878
2017-12-20 16:59:53 -05:00
Pierre-Louis Bonicoli
35a4fe8546 YAML inventory unit test: add tests 2017-12-20 16:59:53 -05:00
Pierre-Louis Bonicoli
8dff0b091e Fix YAML inventory unit test 2017-12-20 16:59:53 -05:00
John R Barker
cfb15b0272 Revert "YAML inventory unit test: fix test inventory format (#33828)" (#33869)
This reverts commit dfb2f346d8.

https://github.com/ansible/ansible/pull/33828
2017-12-13 14:19:19 -05:00
Pilou
dfb2f346d8 YAML inventory unit test: fix test inventory format (#33828)
* Fix YAML inventory unit test

* YAML inventory unit test: add checks

* YAML: add hosts without any group to ungrouped
2017-12-13 15:30:59 +00:00
Toshio Kuratomi
155f36bbd8
Rekey on member (#33836)
* Change cast_list_to_dict to more generic rekey_on_member

cast_list_to_dict was taking an arbitrary data format in and returning
an arbitrary data format out.  Rework this to be a more generic function
which creates a dict of dicts based on a member of the dict.

Remove cast_dict_to_list since rekey_on_member handles the use cases we
know about and cast_dict_to_list suffers from the same problems as
cast_list_to_dict.  If this is still needed we could think about filters
we could add to do this in a short jinja2 pipeline.

* Fix bare excepts (bare excepts even catch sys.exit())
2017-12-12 19:02:15 -08:00
Adrian Likins
a57d6a4206 Use pop in cb clean results (#33779)
* Fix _clean_result for debug callback with 1 char var names

The check in _clean_results was removing any keys that happened
to be one of the chars in ('invocation') instead of the string
'invocation'. This was meant to be a tuple but there was no comma
so the for iterated the string instead of the tuple.

Introduced in 9dba580204

Update unit test to catch this.

Fixes #33723

* Use .pop() to remove invocation from results dict

In base callback _clean_results, simplify the way the
'invocation' item is removed.

Add some more unit tests.
2017-12-11 17:41:12 -05:00
Ganesh Nalawade
95044d69c7
Fix unit test failure for netconf connection plugin (#33547) 2017-12-05 12:04:48 +05:30
Toshio Kuratomi
e499bccbaa
Port some tests away from nose as examples (#33437)
We don't need to use both nose and pytest.  Once we get rid of all uses
of nose we can remove the extra dependency
2017-11-30 14:04:09 -08:00
Nathaniel Case
a1da3dc997
Auth to become (#33319)
* terminal.on_(de)?authorize -> terminal.on_(un)?become

* Update network_cli

* Readd on_(de)?authorize as wrappers

* Update tests
2017-11-28 16:28:58 -05:00
Ken Celenza
748107d369 Type manipulation (#28446)
* add type manipulation

* update for py3

* add tests for list to dict

* Add dict to list tests

* Update tests to search for regex in response

* pep8 clean up

* update Exception for py3

* update test to be py2/3 compat

* update for py26 compat

* potential fix for py3 and py26

* potential fix for py3 and py26 take2

* add new line to kick off shippable

* remove cache file created

* fix filter name

* add space for shipable
2017-11-24 15:26:29 -05:00
James Cammarata
d8ae4dfbf2 Adding aliases for field attributes and renaming async attribute (#33141)
* Adding aliases for field attributes and renaming async attribute

As of Python 3.7, the use of async raises an error, whereas before the use
of the reserved word was ignored. This adds an alias field for field attrs
so that both async and async_val (interally) work. This allows us to be
backwards-compatible with 3rd party plugins that may still reference Task.async,
but for the core engine to work on Py3.7+.

* Remove files fixed for 'async' usage from the python 3.7 skip list
2017-11-22 12:35:58 -08:00
jctanner
218987eac1
ANSIBLE_SSH_USETTY configuration option (#33148)
* Allow the user to circumvent adding -tt on ssh commands to help aid in
debugging ssh related problems.
* Move config to the plugin
* Set version_added
* Change yaml section to "connection"
* Fix ssh unit tests
2017-11-22 11:19:43 -05:00
Ganesh Nalawade
0ddf092ae3
Add new filter to parse xml output for network use cases (#31562)
* Add new filter to parse xml output for network use cases

Fixes #31026
*  Add parse_xml filter
*  Add documentation for parse_xml filter

* Edited for clarity.

* Fix review comment and add unit tests

* Fix unit test CI failure

* Fix CI issues

* Fix unit test failures

* Fix review comments

* More copy edits.
2017-11-21 12:16:18 +05:30
Brian Coca
23b1dbacaf
Config continued (#31024)
* included inventory and callback in new config

allow inventory to be configurable
updated connection options settings
also updated winrm to work with new configs
removed now obsolete set_host_overrides
added notes for future bcoca, current one is just punting, it's future's problem
updated docs per feedback
added remove group/host methods to inv data
moved fact cache from data to constructed
cleaner/better options
fix when vars are added
extended ignore list to config dicts
updated paramiko connection docs
removed options from base that paramiko already handles
left the look option as it is used by other plugin types
resolve delegation
updated cache doc options
fixed test_script
better fragment merge for options
fixed proxy command
restore ini for proxy
normalized options
moved pipelining to class
updates for host_key_checking
restructured mixins

* fix typo
2017-11-16 13:49:57 -05:00
Frederic Lepied
35f79370e1 mock ncclient import in test_connection.py (#32786) 2017-11-10 11:53:04 -05:00
Nathaniel Case
9c0275a879
Connection plugins network_cli and netconf (#32521)
* implements jsonrpc message passing for ansible-connection

* implements more generic mechanism for persistent connections
* starts persistent connection in task_executor if enabled and supported
* supports using network_cli as top level connection plugin
* enhances logging for persistent connection to stdout

* Update action plugins

* Fix Python3 RPC

* Fix Junos bytes<-->str issues

* supports using netconf as top level connection plugin

* Error message when running netconf on an unsupported platform
* Update tests

* Fix `authorize: yes` for `connection: local`

* Handle potentially JSON data in terminal

* Add clarifying detail if possible on ConnectionError
2017-11-09 15:04:40 -05:00
Brian Coca
db749de5b8 namespace facts
updated action plugins to use new guranteed facts
updated tests to new data clean
added cases for ansible_local and some docstrings
2017-11-09 09:48:14 -05:00
Chris Meyers
cf938e9992 tests for InventoryModule error conditions (#31381)
* tests for InventoryModule error conditions

* modified unicode in tests to ahear to Ansible best practices

* flake8 fixes
2017-10-16 18:52:44 -04:00
Brian Coca
01b6c7c9c6 better cleanup on task results display (#27175)
* better cleanup on task results display

callbacks get 'clean' copy of result objects
moved cleanup into result object itself
removed now redundant callback cleanup
moved no_log tests

* moved import as per feedback
2017-10-16 09:44:11 -04:00
Hervé Beraud
db70eeb913 Feature/test inventory (#30707)
* [test] move inventory test to right path
* [feat] add unit test for yaml inventory plugin
2017-10-02 23:01:25 -07:00
Matt Clay
dbf9634b1b Enable more pylint rules and fix reported issues. (#30543)
* Enable pylint invalid-encoded-data test.
* Enable pylint anomalous-unicode-escape-in-string test.
2017-09-19 00:45:35 -07:00
Toshio Kuratomi
75249e311e Fix set theory filters for python3
The set theory filters need to use lists rather than generators on python3.

Also add unit tests for most of the mathstuff filters

Fixes #26494
2017-09-09 17:38:39 -07:00