Commit graph

524 commits

Author SHA1 Message Date
Matt Clay 97a0810e0f
Move ansible-test entry point. (#60281) 2019-08-08 09:18:31 -07:00
Matt Clay d651bda123
Relocate ansible-test code. (#60147)
* Initial move of `test/runner/` content.

`test/runner/lib/` -> `test/lib/ansible_test/_internal/`
`test/runner/`     -> `test/lib/ansible_test/_internal/data/`

* Initial move of `test/sanity/` content.

`test/sanity/` -> `test/lib/ansible_test/_internal/data/sanity/` (except `test/sanity/ignore.txt`)

* Initial move of `test/units/pytest/` content.

`test/units/pytest/` -> `test/lib/ansible_test/_internal/data/pytest/`

* Follow-up move of `test/runner/unit/` content.

`test/lib/ansible_test/_internal/data/unit/` -> `test/lib/ansible_test/tests/unit/`

* Initial move of `ansible.cfg` content.

`test/units/ansible.cfg` -> `test/lib/ansible_test/_internal/data/units/ansible.cfg`
`test/env/ansible.cfg` -> `test/lib/ansible_test/_internal/data/env/ansible.cfg`

* Follow-up move of `data` directory.

`test/lib/ansible_test/_internal/data/` -> `test/lib/ansible_test/_data/`

* Update import statements.

* Add missing __init__.py for unit tests.

* Fix path references and miscellaneous issues.
2019-08-06 14:43:29 -07:00
Matt Davis 8d1f658ce4
move CLI entrypoints under ansible package (#60004)
* needed so ansible-test can always find the right ones to copy to a target
* renamed the underlying scripts to be properly accessible as Python modules
2019-08-02 11:27:02 -07:00
Sam Doran 6cf6f5a34b
Use atexit to cleanup tmp dirs (#56532)
* Wrap everything in try/except to avoid leaving files behind
* Add unit tests, integration tests, and changelog
* Do text the correct way
2019-06-28 16:19:27 -04:00
Matt Martz db6cc60352
Migrate command line parsing to argparse (#50610)
* Start of migration to argparse

* various fixes and improvements

* Linting fixes

* Test fixes

* Fix vault_password_files

* Add PrependAction for argparse

* A bunch of additional tweak/fixes

* Fix ansible-config tests

* Fix man page generation

* linting fix

* More adhoc pattern fixes

* Add changelog fragment

* Add support for argcomplete

* Enable argcomplete global completion

* Rename PrependAction to PrependListAction to better describe what it does

* Add documentation for installing and configuring argcomplete

* Address rebase issues

* Fix display encoding for vault

* Fix line length

* Address rebase issues

* Handle rebase issues

* Use mutually exclusive group instead of handling manually

* Fix rebase issues

* Address rebase issue

* Update version added for argcomplete support

* -e must be given a value

* ci_complete
2019-04-23 13:54:39 -05:00
Ganesh Nalawade d100faedb3
Add check to log jsonrpc request/response (#54606)
Fixes #54605

Add check in ansible-connection to check if persistent
log message is enabled, if yes log the jsonrpc request
in log file.

With the current approach the _messages queue in `NetworkConnectionBase`
is getting overwritten and response for `pop_message` api inovcation from
ansible-connection is not recevied at the module side.
2019-04-03 09:40:31 +05:30
Toshio Kuratomi afdbb0d9d5 Save the command line arguments into a global context
* Once cli args are parsed, they're constant.  So, save the parsed args
  into the global context for everyone else to use them from now on.
* Port cli scripts to use the CLIARGS in the context
* Refactor call to parse cli args into the run() method
* Fix unittests for changes to the internals of CLI arg parsing
* Port callback plugins to use context.CLIARGS
  * Got rid of the private self._options attribute
  * Use context.CLIARGS in the individual callback plugins instead.
  * Also output positional arguments in default and unixy plugins
  * Code has been simplified since we're now dealing with a dict rather
    than Optparse.Value
2019-01-03 18:12:23 -08:00
Nathaniel Case b2423e7602
Log device interaction and push labeled logs back to controller (#50028)
* Add session tracing support for network_cli, netconf and httapi connection

*  Add `persistent_log_messages` configuration option to log device inteaction
   in log file for network_cli, netconf and httapi connection
   type
*  Log jsonrpc request and response in log file is configuration option
   is enabled

* Update docs to talk about warning shown when persistent_log_messages is on
2018-12-21 10:31:43 -05:00
Nathaniel Case 62ec383232
Fix some handling of log messages from ansible-connection (#50214) 2018-12-20 16:38:59 -05:00
Nathaniel Case 1829a72885
Allow persistent connection plugins to queue messages back to ansible-connection (#49977)
* Connections can queue messages to be returned from ansible-connection

* Provide fallback for invalid display level

* Strip display from plugins

* Route messages through helper method to try to avoid improper appends
2018-12-19 10:54:42 -05:00
Nathaniel Case 61a649ce0f
Don't fail if a remote_addr with a '/' hits ansible_connection (#49781)
* Fail if a remote_addr with a '/' hist ansible_connection

This is _probably_ a CIDR block, but anything with a slash will fail,
so no need to try to parse to make sure

* Locks are now per-socket_path.

Locks use the same value as socket_path. Locks are also cleaned up in
shutdown like sockets.
2018-12-17 12:53:32 -05:00
Toshio Kuratomi 3fba006207 Update bare exceptions to specify Exception.
This will keep us from accidentally catching program-exiting exceptions
like KeyboardInterupt and SystemExit.
2018-12-16 15:03:19 -08:00
Brian Coca 3132266aa1 use ansible json encoding/decoding to avoid errors (#47214)
tackle known issues with (de)serializing certain objects
fixes #47200
2018-11-16 12:11:33 -05:00
Toshio Kuratomi 4d9504e775 Remove runtime usage of pkg_resources.
This should provide a startup time speed boost at the expense of making
it harder to get ansible to use newer versions of packages than are
provided by the platform.
2018-10-18 15:09:38 -07:00
Nathaniel Case 0d143ed7f6
Clean up after ansible-connection if failure occurred in start() (#45929) 2018-09-20 15:04:30 -04:00
Matt Clay f3d1f9544b
Make ansible-test available in the bin directory. (#45876) 2018-09-19 17:58:55 -07:00
Nathaniel Case 86f96d0212
Move plugin option settings. (#44774)
* Move plugin option settings.

* Add default value and clear after use.
2018-08-28 17:30:50 -04:00
Nathaniel Case f221105882
Prevent data being truncated over persistent connection socket (#43885)
* Change how data is sent to the persistent connection socket.

We can't rely on readline(), so send the size of the data first. We can
then read that many bytes from the stream on the recieving end.

* Set pty to noncanonical mode before sending

* Now that we send data length, we don't need a sentinel anymore

* Copy socket changes to persistent, too

* Use os.write instead of fdopen()ing and using that.

* Follow pickle with sha1sum of pickle

* Swap order of vars and init being passed to ansible-connection
2018-08-10 09:26:58 -04:00
Ganesh Nalawade 3f3101dfe5
Raise exception if command timeout is triggered (#43078)
* Raise exception if command timeout is triggered

Fixes #43076

If persistent connection timeout is triggered, riase
exception which will be send over socket to module code
instead of silently shutting down the socket.

* Fix CI failure

* Fix review comment

* Fix CI failure

* Fix review comment

* Fix review comment
2018-08-02 12:08:37 +05:30
Nathaniel Case 9169daa326
\r does not round-trip the local socket, escape and restore on the other side (#43507) 2018-08-01 08:44:39 -04:00
Ganesh Nalawade ea6ef3fae8
Enable setting options for cliconf and other implementation plugins (if set to configurable) (#43368)
* Enable setting setting cliconf plugin options

Fixes #43367

* Add support to set configuration options for implementation plugins (eg: cliconf)
  from `ansible-connection`

* Fix CI failure
2018-07-30 21:53:44 -07:00
joren485 d03b9eddf6 Remove unused imports from /bin and lib/ansible/cli (#43440)
* Remove unused imports from lib/ansible/cli

* Remove unused imports from bin/
2018-07-30 18:55:42 -07:00
Ganesh Nalawade 97d4e53131
Support setting persistent command timeout per task basis (#42847)
* Support setting persistent command timeout per task basis

Fixes #42200

*  Add variable `ansible_command_timeout` to `persistent_command_timeout`
   option for `network_cli` and `netconf` connection plugin so that the
   command_timeout can be set per task basis while using `connection=network_cli`
   or `connection=netconf`
   eg:
   ```
   - name: run copy command
     ios_command:
       commands:
       - show version
     vars:
      ansible_command_timeout: 40
   ```

* Modify `ansible-connection` to read command_timeout value from
  connection plugin options.

* Add `ansible_command_timeout` to `persistent_command_timeout`
  option in `persistent` to support `connection=local` so that
  it is backward compatibilty

* To support `connection=local` pass the timeout value as variables
  from persistent connection to `ansible-connection` instead of sending
  it in playcontext

* Fix CI failure

* Fix review comment
2018-07-20 10:04:53 +05:30
Nathaniel Case 483df13626 Fixing issues with httpapi (#40388)
* I seem to have forgotten the back half of tests

* Set http timeout from persistent_command_timeout

* Tweak URL generation and provide URL on error

* Push var_options to connection process

* Don't wait forever if coming from persistent

* Don't send the entire contents of variables to ansible-connection
2018-05-21 15:58:35 +01:00
Ricardo Carrillo Cruz 62e1c14edc
Pull persistent connection parameters via get_option (#39367)
* WIP Pull persistent connection parameters via get_option

* Fix pep8

* Add use_persistent_connection setting to paramiko_ssh plugin

* Add vars section to persistent_command_timeout setting and prevail provider values over config manager

* Use persistent_command_timeout on network_cli instead of timeout

* Fix unit tests

If we don't call loader to get network_cli, then _load_name is never
set and we get KeyError.

* Pull persistent_command_timeout via config  manager for ios connection local

* Pull persistent_command_timeout via config manager on connection local
2018-05-16 14:59:01 +02:00
James Cammarata 7ce9968ce1
Properly unlock the socket file lock in ansible-connection (#39223)
Also use a lock file per host, rather than one global file lock.

Commit 9c0275a879 introduced a bug where the lock file was only being
unlocked by the child PID of the resulting fork done in ansible-connection.
This causes delays when a large inventory causes a lot of contention on
that global lock. This patch fixes the problem by ensuring the lock is
released regardless of the fork condition, and also to use a lock file
based on the remote address of the target host, removing the global lock
bottleneck.

Fixes #38892
2018-04-25 15:00:15 -05:00
Toshio Kuratomi 06f73ad578
Normalize usage of temp and tmp on tmp (#36221)
* Normalize usage of temp and tmp on tmp
* Rename system_tmps system_tmpdirs
* Add ANSIBLE_REMOTE_TMP spelling of environment variables
2018-02-15 09:01:02 -08:00
Nathaniel Case 527fc492b9
Acquire displayed messages from forked process (#36064)
Also restore verbosity value from loaded `play_context`
2018-02-13 10:59:37 -05:00
Ganesh Nalawade 90cd87f950
Fix debug logs failing with persistent connection (#33049)
* Fix debug logs failing with persistent connection

Fixes #33047

*  As debug logs are written on stdout, it interrupts
   the communication between ansible-connection(background)
   process and main process. To avoid this add a string similar
   to exactly identify the response string.

*  Remove unwanted code in ansible-connection

*  Fix review comments

* Fix spurious log emitted on ansible-connection stdout issue

*  ansible-connection which runs as a background process sends a
   json string (contains response received from remote device)
   to foreground ansible-playbook process over stdout.

*  If in case debug flag is enabled the connection_loader api
   invoked from ansible-connection `ssh = connection_loader.get('ssh', class_only=True)`
   results in emitting debug logs on stdout. This  spurious log
   interfere with the actual response and results in failure while
   reading json string in ansible-playbook process

* To avoid this save stdout of ansible-connection and redirect it string
  buffer to accumulate all the logs emitted by core API's

* Add these logs in `result['messages']` which is send a json string after reinstating saved stdout

*  Remove unwanted code in ansible-connection

* Fix review comment
2018-01-25 02:48:45 +05:30
Matt Martz 55352bdda4
Error early if executing python version doesn't meet documented minimums (#34655)
* Error early if executing python version doesn't meet documented minimums. Fixes #34597

* Make recommended enhancements
2018-01-09 16:17:55 -06:00
Ganesh Nalawade 2f932d8767
Add parent pid to persistent connection socket path hash (#33518)
* Add parent pid to persistent connection socket path hash

Fixes #33192

*  Add parent pid in persistent connection socket path hash
   to avoid using same socket path for multiple simultaneous
   connection to same remote host.

* Ensure unique persistent socket path for each ansible-playbook run

* Fix CI failures
2017-12-15 10:21:56 +05:30
Julien Palard 53d9f9ffae Allow PDB to enter post mortem. fixes (#31086) 2017-11-28 13:10:08 -08:00
Peter Sprygada 69575e25d0 shuts down persistent connections at end of play run (#32825)
This change will now track any created persistent connection and shut it
down at the end of the play run.  This change also includes an update to
properly honor the reset_connection meta handler.
2017-11-22 15:30:06 +00:00
Brian Coca 23b1dbacaf
Config continued (#31024)
* included inventory and callback in new config

allow inventory to be configurable
updated connection options settings
also updated winrm to work with new configs
removed now obsolete set_host_overrides
added notes for future bcoca, current one is just punting, it's future's problem
updated docs per feedback
added remove group/host methods to inv data
moved fact cache from data to constructed
cleaner/better options
fix when vars are added
extended ignore list to config dicts
updated paramiko connection docs
removed options from base that paramiko already handles
left the look option as it is used by other plugin types
resolve delegation
updated cache doc options
fixed test_script
better fragment merge for options
fixed proxy command
restore ini for proxy
normalized options
moved pipelining to class
updates for host_key_checking
restructured mixins

* fix typo
2017-11-16 13:49:57 -05:00
Nathaniel Case 21c8ab537b
getattr() with no default is an AttributeError if missing (#32946) 2017-11-15 15:40:32 -05:00
Nathaniel Case a1517234f6
Fix authorize: yes and become_method: enable (#32864)
* Update connection play_context when socket exists

* Don't fail on connections other than network_cli

* Fix enable prompt detection on ios & eos

* Check against "Module not found" error code, defined in modules/jsonrpc.py
2017-11-14 15:51:14 -05:00
Nathaniel Case 9c0275a879
Connection plugins network_cli and netconf (#32521)
* implements jsonrpc message passing for ansible-connection

* implements more generic mechanism for persistent connections
* starts persistent connection in task_executor if enabled and supported
* supports using network_cli as top level connection plugin
* enhances logging for persistent connection to stdout

* Update action plugins

* Fix Python3 RPC

* Fix Junos bytes<-->str issues

* supports using netconf as top level connection plugin

* Error message when running netconf on an unsupported platform
* Update tests

* Fix `authorize: yes` for `connection: local`

* Handle potentially JSON data in terminal

* Add clarifying detail if possible on ConnectionError
2017-11-09 15:04:40 -05:00
Ganesh Nalawade 97d5e0d027 Remove log of command executed in persistent connection (#31581) 2017-10-11 19:03:21 +05:30
Brian Coca 9c6d7ddeb5 report extra info in verbose and if needed 2017-10-03 16:48:21 -04:00
Brian Coca 76aaaf127b nicer error on bad ansible config (#30461)
* nicer error on bad ansible config
2017-09-19 10:50:28 -04:00
Matt Clay 442af3744e Miscellaneous pylint fixes.
The following rules are no longer disabled:

- bad-format-string
- duplicate-key
- lost-exception
- trailing-newlines
- unexpected-keyword-arg
- useless-suppression
- using-constant-test
2017-09-13 01:53:08 -07:00
Ganesh Nalawade cd8c1c1108 Create persistent socket path using port and connection type (#28492)
* Create persistent socket path using port and connection type

*  Use remote address, port, connection type and remote user
   to create a socket path.

* Fix review comment
2017-08-23 18:28:44 -04:00
Matt Clay 8e0b5800b7 Miscellaneous ansible-test fixes. (#28355)
* Include .github in test targets. Fix BOTMETA.yml.
* Include bin in compile tests.
* Exclude links from test targets.
* Include bin in pep8 and pylint tests.
* Fix pep8 and pylint issues in bin dir.
2017-08-17 12:04:43 -07:00
Brian Coca c27248db60 ansible-inventory cli tool (#22477)
* ansible-inventory cli tool

added vars dump to graph
made yaml inventory dump actual yaml inventory format
cleaner dump
fixed graph, no dump needed
add pulling in host/group vars
pep indentation crime
added docstring for manpage autodoc
remove ansible_facts from output
added api compat layer
allow import from new and old APIs
better conditional for <2.4
pe4+p4

test stuck on OS X (seems popular today) but passes rest, merging anyways
2017-08-15 16:45:54 -04:00
Brian Coca f921369445 Ansible Config part2 (#27448)
* Ansible Config part2

- made dump_me nicer, added note this is not prod
- moved internal key removal function to vars
- carry tracebacks in errors we can now show tracebacks for plugins on vvv
- show inventory plugin tracebacks on vvv
- minor fixes to cg groups plugin
- draft config from plugin docs
- made search path warning 'saner' (top level dirs only)
- correctly display config entries and others
- removed unneeded code
- commented out some conn plugin specific from base.yml
- also deprecated sudo/su
- updated ssh conn docs
- shared get option method for connection plugins
- note about needing eval for defaults
- tailored yaml ext
- updated strategy entry
- for connection pliugins, options load on plugin load
- allow for long types in definitions
- better display in ansible-doc
- cleaned up/updated source docs and base.yml
- added many descriptions
- deprecated include toggles as include is
- draft backwards compat get_config
- fixes to ansible-config, added --only-changed
- some code reoorg
- small license headers
- show default in doc type
- pushed module utils details to 5vs
- work w/o config file
- PEPE ATE!
- moved loader to it's own file
- fixed rhn_register test
- fixed boto requirement in make tests
- I ate Pepe
- fixed dynamic eval of defaults
- better doc code

skip ipaddr filter tests when missing netaddr
removed devnull string from config
better becoem resolution

* killed extra space with extreeme prejudice

cause its an affront against all that is holy that 2 spaces touch each other!

shippable timing out on some images, but merging as it passes most
2017-08-15 16:38:59 -04:00
Nathaniel Case a3404418c0 Fix Python3 in ansible-connection (#27645)
* Fix Python3 in ansible-connection

* Add `errors='surrogate_or_strict'` to `to_bytes()`
2017-08-02 14:10:05 -04:00
Ganesh Nalawade 70ce394840 Persistent connection timer changes (#27272)
*  Add command_timeout timer that defines the amount
   of time to wait for a command or RPC call before
   timing out.
*  Remove connect_retries and connect_interval configuration
   varaible and replace it with connect_retry_timeout to control
   the timeout value of connection to local scoket.
*  Make required changes to netowrk action plugins and relevant
   network files in module_utils.
*  Required documentation changes.
2017-08-01 11:45:45 -06:00
Ganesh Nalawade 19fac707fa junos_netconf integration test failure fix (#27569)
*  Create socket using port value and not connection type
*  Correct error message in integration test task
2017-08-01 21:33:11 +05:30
Ganesh Nalawade bb998a3cd2 Add fix to read correct socket path recieved from ansible-connection (#27560)
Currently socket path is send from `ansible-connection` (running as background
process) over stdout. This can conflict with debug logs that are also send on
stdout resulting in incorrect socket path received by the main process.

To avoid this add a socket path delimiter string which is recevied by
main process and socket path is retrieved based on delimiter string.

This implementation will change in future when ansible-connection
framework is made more robust.
2017-08-01 19:32:25 +05:30
Nathaniel Case 28c6b226c7 ansible-connection Python3 fix (#26441)
* Fix a number of unicode <-> bytes mismatches

* Return socket_path as text, not bytes

* Docstring run()
2017-07-05 14:07:26 -04:00