Commit graph

5231 commits

Author SHA1 Message Date
Abhijit Menon-Sen
6e82df451a Clarify select() handling for ssh connections
This change is motivated by an ssh oddity: when ControlPersist is
enabled, the first (i.e. master) connection goes into the background; we
see EOF on its stdout and the process exits, but we never see EOF on its
stderr. So if we ran a command like this:

    ANSIBLE_SSH_PIPELINING=1 ansible -T 30 -vvv somehost -u someuser -m command -a whoami

We would first do select([stdout,stderr], timeout) and read the command
module output, then select([stdout,stderr], timeout) again and read EOF
on stdout, then select([stderr], timeout) AGAIN (though the process has
exited), and select() would wait for the full timeout before returning
rfd=[], and then we would exit. The use of a very short timeout in the
code masked the underlying problem (that we don't see EOF on stderr).

It's always preferable to call select() with a long timeout so that the
process doesn't use any CPU until one of the events it's interested in
happens (and then select will return independent of elapsed time).

(A long timeout value means "if nothing happens, sleep for up to <x>";
omitting the timeout value means "if nothing happens, sleep forever";
specifying a zero timeout means "don't sleep at all", i.e. poll for
events and return immediately.)

This commit uses a long timeout, but explicitly detects the condition
where we've seen EOF on stdout and the process has exited, but we have
not seen EOF on stderr. If and only if that happens, it reruns select()
with a short timeout (in practice it could just exit at that point, but
I chose to be extra cautious). As a result, we end up calling select()
far less often, and use less CPU while waiting, but don't sleep for a
long time waiting for something that will never happen.

Note that we don't omit the timeout to select() altogether because if
we're waiting for an escalation prompt, we DO want to give up with an
error after some time. We also don't set exceptfds, because we're not
actually acting on any notifications of exceptional conditions.
2015-09-24 12:10:16 -04:00
Toshio Kuratomi
03127dcfae remove the stdin return value from connection plugin exec_command() methods
The value was useless -- unused by the callers and always hardcoded to
the empty string.
2015-09-24 08:57:19 -07:00
James Cammarata
9d47eabfa4 Merge pull request #12506 from hyperized/devel
Add Weekday (0-6) as a number and add weeknumber (00-52)
2015-09-24 11:44:39 -04:00
Toshio Kuratomi
24b9e2e6d1 Update extras submodule ref 2015-09-24 07:18:23 -07:00
Gerben Geijteman
4c20964475 Add Weekday (0-6) as a number and add weeknumber (00-52) 2015-09-24 15:05:44 +02:00
Marius Gedminas
56f2a25bff Python 3: there's no 'unicode' 2015-09-24 12:53:48 +03:00
Marius Gedminas
6d4618f46f Python 3: there's no dict.iteritems() 2015-09-24 12:50:00 +03:00
Marius Gedminas
0624797375 Bugfix: if you define a custom __eq__, you must define a __hash__ too
Also, on Python 3 the stock object.__hash__ raises an error ("unhashable
type"), and we have code that uses Host instances as dict keys.
2015-09-24 12:46:06 +03:00
Marius Gedminas
a2bc6b4b26 Bugfix: if you define __eq__, you should define __ne__ too 2015-09-24 12:43:33 +03:00
Marius Gedminas
5d29a2eabd Python 3: shlex.split() wants unicode
On Python 2, shlex.split() raises if you pass it a unicode object with
non-ASCII characters in it.  The Ansible codebase copes by explicitly
converting the string using to_bytes() before passing it to
shlex.split().

On Python 3, shlex.split() raises ('bytes' object has no attribute 'read')
if you pass a bytes object.  Oops.

This commit introduces a new wrapper function, shlex_split, that
transparently performs the to_bytes/to_unicode conversions only on
Python 2.

Currently I've only converted one call site (the one that was causing a
unit test to fail on Python 3).  If this approach is deemed suitable,
I'll convert them all.
2015-09-24 12:36:05 +03:00
Toshio Kuratomi
5d3d9cfe0d Convert to byte strings to avoid UnicodeErrors
Fixes #12488
2015-09-23 15:24:17 -07:00
Brian Coca
de18bcb95f correct typo on error reporting
fixes #12495
2015-09-23 10:11:52 -04:00
Abhijit Menon-Sen
40f608a377 A bit more debugging output
We used to display input chunks earlier anyway, so this isn't making
things more verbose.
2015-09-23 22:35:14 +05:30
Abhijit Menon-Sen
9700d9c04f Fix typo in checking select results
It's possible for more than one fd to be set, so 'elif' is obviously not
the right thing to use.
2015-09-23 22:32:15 +05:30
James Cammarata
1164e83477 Remove unnecessary calls to save inventory restrictions since 81bf88b 2015-09-23 12:18:09 -04:00
Toshio Kuratomi
89a78ba16e Update submodule refs 2015-09-23 08:40:59 -07:00
James Cammarata
9e734df0ec Conditionally poll longer if we're still waiting for an auth prompt 2015-09-23 11:20:11 -04:00
Toshio Kuratomi
5f0f5363b6 Merge pull request #12487 from mgedmin/py3k
Fix one more failing test on Python 3
2015-09-23 08:18:17 -07:00
James Cammarata
2898e000a0 Don't use the connection timeout for the select poll timeout 2015-09-23 11:13:12 -04:00
Abhijit Menon-Sen
587054db2a Send initial data before calling select whenever possible
Without this, we could execute «ssh -q ...» and call select(), which
would timeout after the default 10s, and only then send initial data.
(This is a relic of the earlier change where we always ran ssh with
-vvv, so the situation where it would sit quietly never happened in
practice; but this would have been the right thing to do even then.)
2015-09-23 20:09:50 +05:30
James Cammarata
c9a004227e Improve error catching from malformed playbook data
Fixes #12478
2015-09-23 08:56:36 -04:00
James Cammarata
e8e1d9f6fb Apply --limit to inventory in adhoc commands
Fixes #12473
2015-09-23 08:28:38 -04:00
Marius Gedminas
95e655eb67 Python 3: there's no basestring
Fixes one failing test.

The long series of module_utils/basic.py fixes were all because
module_utils/basic is imported in ansible/inventory/script.py.
2015-09-23 10:04:26 +03:00
Marius Gedminas
2c4982b58d Python 3: there's no itertools.imap
Because the builtin map() acts like an iterator already.
2015-09-23 10:04:26 +03:00
Marius Gedminas
6708d56a21 Python 3: avoid long integer literals
Even Python 2.4 automatically promotes int to long.
2015-09-23 10:04:25 +03:00
Marius Gedminas
f5d4935197 Python 3: treat python as a function in module_utils/basic.py
NB: we can't use 'from __future__ import print_function', but luckily
print(one_thing) works fine on both Python 2 and Python 3 without that.
2015-09-23 10:04:25 +03:00
Marius Gedminas
e71a986e16 Python 3: avoid octal constants in module_utils/basic.py 2015-09-23 10:04:25 +03:00
Marius Gedminas
d2bec7f81f Python 3: avoid "except ..., e:" in module_utils/basic.py
Make the code compatible with Pythons 2.4 through 3.5 by using
sys.exc_info()[1] instead.

This is necessary but not sufficient for Python 3 compatibility.
2015-09-23 10:04:25 +03:00
James Cammarata
65630d2ce1 Fixing one more bug related to staticmethods in LookupBase 2015-09-23 02:33:32 -04:00
James Cammarata
cbbb270761 Cleanup bug from moving base lookup methods to staticmethods 2015-09-23 02:26:19 -04:00
Abhijit Menon-Sen
ac98fe9e89 Implement ssh connection handling as a state machine
The event loop (even after it was brought into one place in _run in the
previous commit) was hard to follow. The states and transitions weren't
clear or documented, and the privilege escalation code was non-blocking
while the rest was blocking.

Now we have a state machine with four states: awaiting_prompt,
awaiting_escalation, ready_to_send (initial data), and awaiting_exit.
The actions in each state and the transitions between then are clearly
documented.

The check_incorrect_password() method no longer checks for empty strings
(since they will always match), and check_become_success() uses equality
rather than a substring match to avoid thinking an echoed command is an
indication of successful escalation. Also adds a check_missing_password
connection method to detect the error from sudo -n/doas -n.
2015-09-23 01:55:00 -04:00
Abhijit Menon-Sen
840a32bc08 Reorganise ssh.py to cleanly separate responsibilities
The main exec_command/put_file/fetch_file methods now _build_command and
call _run to handle input from/output to the ssh process. The purpose is
to bring connection handling together in one place so that the locking
doesn't have to be split across functions.

Note that this doesn't change the privilege escalation and connection IO
code at all—just puts it all into one function.

Most of the changes are just moving code from one place to another (e.g.
from _connect to _build_command, from _exec_command and _communicate to
_run), but there are some other notable changes:

1. We test for the existence of sshpass the first time we need to use
   password authentication, and remember the result.
2. We set _persistent in _build_command if we're using ControlPersist,
   for later use in close(). (The detection could be smarter.)
3. Some apparently inadvertent inconsistencies between put_file and
   fetch_file (e.g. argument quoting, sftp -b use) have been removed.

Also reorders functions into a logical sequence, removes unused imports
and functions, etc.

Aside: the high-level EXEC/PUT/FETCH description should really be logged
from ConnectionBase, while individual subclasses log transport-specific
details.
2015-09-23 01:55:00 -04:00
James Cammarata
95c6fe88e4 Fix handling of conditional vars_files which contain variables
Fixes #12484
2015-09-23 01:26:24 -04:00
Toshio Kuratomi
c83f51b7f2 Some LookupBase cleanups:
* Make LookupBase an abc with required methods (run()) marked as an
  abstractmethod
* Mark methods that don't use self as @staticmethod
* Document how to implement the run method of a lookup plugin.
2015-09-22 16:19:36 -07:00
Toshio Kuratomi
049952fa50 Update submodule refs. 2015-09-22 14:01:53 -07:00
James Cammarata
1e7fd2196d Fixing synchronize + delegate_to user bug
Fixes #12464
2015-09-22 16:06:52 -04:00
James Cammarata
3ffc2783c4 Don't bomb out on handlers with undefined variables in their names 2015-09-22 12:42:02 -04:00
Toshio Kuratomi
4b0d52d2cb Merge pull request #12420 from ansible/win_prefix_modules
Fix for user defined modules not overriding modules from core.
2015-09-22 09:16:38 -07:00
James Cammarata
1076155d8d When failing because of vars_files templating, try and bubble up the file/line info 2015-09-22 12:13:55 -04:00
Toshio Kuratomi
18e2ee16ef Fix for user defined modules not overriding modules from core.
This fix takes into account that powershell modules are somewhat
different than regular modules and have to be kept separate.
2015-09-22 09:07:37 -07:00
Toshio Kuratomi
f61fb9787d Update submodule refs 2015-09-22 08:59:10 -07:00
Rene Moser
b43939dfd6 cloudstack: make tags handling idempotence
Credits to @jeffersongirao, who provided the patch.
2015-09-22 16:12:35 +02:00
James Cammarata
c30e464388 Additional tweaks to callback output for delegate_to 2015-09-22 09:25:38 -04:00
James Cammarata
513619867a Show delegated-to host in callback message
Fixes #12465
2015-09-22 08:45:04 -04:00
Rene Moser
efd122c2f0 cloudstack: add returns_to_int return handling into utils
It is not uncommon that the API returns string for int values e.g. ports in listFirewallRules or listPortForwardings,
2015-09-22 14:44:08 +02:00
Rene Moser
c9a3801a25 cloudstack: common argument_spec and requried_together to utils 2015-09-22 14:36:43 +02:00
James Cammarata
f563b22446 Merge pull request #12461 from mgedmin/py3k
Python 3: there's no basestring
2015-09-22 03:11:43 -04:00
James Cammarata
d2949f5449 Merge pull request #12463 from mgedmin/fix-ansible-doc
Fix option descriptions in ansible-doc output
2015-09-22 03:10:59 -04:00
James Cammarata
0fb4a6a67b Tweak to the way new host variables are created for delegated hosts 2015-09-22 03:03:21 -04:00
Marius Gedminas
339790adc4 Fix option descriptions in ansible-doc output
Fixes #12462.
2015-09-22 10:00:33 +03:00
James Cammarata
18adfc6d1a Set some default vars on hosts created for delegate_to connections 2015-09-22 02:35:01 -04:00
James Cammarata
a22f7b883d Restrict role param vars to tasks within that role
Fixes #12460
2015-09-22 02:14:18 -04:00
Marius Gedminas
fc0dcc3947 Python 3: there's no basestring
Fixes one failing test.

Now technically a filename can be a bytestring, even on Python 3.  I
hope this is unlikely for Ansible.
2015-09-22 08:42:33 +03:00
James Cammarata
ecf7d8c9ee Fix relative path bug in copy action
Without rebreaking #12055

Fixes #ansible-modules-core/2098
2015-09-22 01:31:03 -04:00
James Cammarata
53794b692c Merge pull request #12457 from ansible/warn-if-core-modules-not-present
Detect if core modules aren't installed and warn if that is the  case
2015-09-21 21:15:54 -04:00
Toshio Kuratomi
627f9d73ba Detect if core modules aren't installed and warn if that is the case
Fixes #11206
2015-09-21 12:31:51 -07:00
Brian Coca
d27b73e7b5 fixed case in which boto3 is present but module is not using it yet 2015-09-21 15:16:07 -04:00
James Cammarata
cb7060c9fe Allow undefined var errors to bubble up when templating vars_files in certain conditions
Follow up to 8769f03c, which allows the undefined var error to be raised
if we're getting vars with a full context (play/host/task) and the host
has already gathered facts. In this way, vars_files containing variables
that fail to be templated are not silently ignored.
2015-09-21 11:57:37 -04:00
Brian Coca
f96255f7fd fixed typo 2015-09-21 10:10:21 -04:00
James Cammarata
86566e691a Merge pull request #12447 from mgedmin/py3k
Python 3: there is no 'basestring'
2015-09-21 09:39:17 -04:00
James Cammarata
8769f03c16 Also catch AnsibleUndefinedVariable errors when templating vars_files names
Fixes #12449
2015-09-21 09:35:06 -04:00
Toshio Kuratomi
2a614c1c0f Close module file once we're done reading from it 2015-09-21 01:46:29 -07:00
Marius Gedminas
9cdb6ebae3 Python 3: there is no 'basestring'
This fixes a failing unit test.

In actual use (which is still quite far), I'm not sure if bytes ->
unicode conversion should be done here (in which case the code will fail
with an AttributeError: 'bytes' object has no attribute 'readlines'), or
inside self._connection.exec_command() (in which case my change is
correct).
2015-09-21 08:48:59 +03:00
Victor Salgado
20bbd66e85 Use dict.get() 2015-09-20 22:54:02 -03:00
James Cammarata
65bf14cbb0 Merge pull request #12431 from hslee16/devel
Stops using _groups_list to check for host when using 'add_host'
2015-09-20 18:17:02 -04:00
James Cammarata
88e1aa94fa Add play context vars to list of variables before post validation
Fixes #12437
2015-09-19 21:51:24 -04:00
James Cammarata
05f6e76407 Template handler names before checking to see if they need to run
Fixes #12427
2015-09-18 18:54:48 -04:00
James Cammarata
72769d1899 Merge pull request #12432 from mgedmin/py3k
Python 3: there's no xrange
2015-09-18 15:33:08 -04:00
James Cammarata
37f2cbc429 Merge pull request #12436 from amenonsen/ranges
Support «hosts: foo[1:]» and add tests for split/apply_subscript
2015-09-18 15:32:15 -04:00
James Cammarata
1f5584aa5b Refactoring delegate_to code
Now, instead of relying on hostvars on the executor side, we compile
the vars for the delegated to host in a special internal variable and
have the PlayContext object look for things there when applying task/
var overrides, which is much cleaner and takes advantage of the code
already dealing with all of the magic variable variations.

Fixes #12127
Fixes #12079
2015-09-18 15:25:10 -04:00
James Cammarata
fa69e8ebb8 Fixing some bugs in _compute_delegate() in TaskExecutor
* Clearing interpreter settings from variables, so those set for the
  original host aren't incorrectly applied to the delegated to host
* Fixed incorrect string for remote user in delegated hosts hostvars
* Properly looking for multiple possiblities in the delegated-to hosts
  hostvars (ansible_ssh_host vs. ansible_host)
2015-09-18 15:25:10 -04:00
Peter Sprygada
e45b71d6cd added new module object to vca 2015-09-18 13:45:37 -04:00
Abhijit Menon-Sen
2fcdb37e7b Support «hosts: groupname[1:]» notation (~= 'the rest of the group') 2015-09-18 22:28:34 +05:30
Brian Coca
aca83b15e5 Merge pull request #12425 from privateip/devel
initial add of vca common module
2015-09-18 11:24:54 -04:00
Brian Coca
11707576a3 fix for AIX network facts
fixes #12434
2015-09-18 10:43:53 -04:00
Brian Coca
d7b61db6dc made boto3 non mandatory 2015-09-18 08:05:33 -04:00
Toshio Kuratomi
fdb1c14004 Update the submodule pointers 2015-09-18 04:49:30 -07:00
Marius Gedminas
baf9320369 Python 3: there's no xrange
Use six.moves.range instead (aliased to xrange on Python 2, aliased to
range on Python 3).

Also I couldn't resist replacing the elaborate chr/ord/randrange dance
with the simpler random.choice(string.ascii_lowercase) that was already
used elsewhere in the Ansible codebase.
2015-09-18 08:52:26 +03:00
Alex Lee
cf289191e5 return None if group does not exist in inventory groups 2015-09-17 15:58:10 -07:00
James Cammarata
cc6627cdd6 Remove custom json encoder cleaner and strip proxy var stuff out before encoding
Fixes #12349
2015-09-17 16:04:47 -04:00
James Cammarata
c95d4ca05f Properly assign search path to environment.loader in template lookup
Same fix as was applied in f162990c to the action plugin

Fixes #12355
2015-09-17 14:47:20 -04:00
James Cammarata
bde5ed9672 Catch worker errors that may be subclassed on others that lead to incorrect exits 2015-09-17 14:46:53 -04:00
Peter Sprygada
8f43d222c0 initial add of vca common module 2015-09-17 14:41:53 -04:00
James Cammarata
053c41e79d Fix typo in ansible module_utils import from galaxy code 2015-09-17 14:25:48 -04:00
Chrrrles Paul
7b778d8a4b Merge pull request #12424 from amenonsen/ipv6-ranges
Tested.  Thanks for spotting and then fixing this -- Allow hexadecimal ranges in IPv6 addresses, not only 0-9
2015-09-17 14:21:03 -04:00
Abhijit Menon-Sen
2d420a9bb7 Allow hexadecimal ranges in IPv6 addresses, not only 0-9 2015-09-17 23:32:58 +05:30
James Cammarata
b386f0c64d Merge pull request #12423 from amenonsen/split-fixup
Remove deprecation warning; now handled in _split_patterns
2015-09-17 14:02:22 -04:00
James Cammarata
a1fffff92c Merge branch 'addresses' of https://github.com/amenonsen/ansible into amenonsen-addresses 2015-09-17 13:43:05 -04:00
Abhijit Menon-Sen
62ca6b5e35 Remove deprecation warning; now handled in _split_patterns 2015-09-17 23:09:56 +05:30
Abhijit Menon-Sen
575a8b8c77 Merge Host.ipv[46]_address into .address
The earlier distinction was never used; .ipv6_address was always a copy
of .ipv4_address, and the latter was always used to set the remote_addr
field in the PlayContext.

Also uses the canonical ansible_host/ansible_port names when setting the
address and port from variables.
2015-09-17 23:03:48 +05:30
James Cammarata
951128d7a6 Merge branch 'hostpatterns' of https://github.com/amenonsen/ansible into amenonsen-hostpatterns 2015-09-17 13:31:51 -04:00
Victor Salgado
14fefebaad Modify _split_pattern to use map when working with list input 2015-09-17 13:52:54 -03:00
James Cammarata
46baed864f Submodule update after correcting doc bug 2015-09-17 11:30:38 -04:00
James Cammarata
0beb37a121 Submodule update 2015-09-17 11:26:54 -04:00
James Cammarata
ba82e57445 Merge branch 'sudo_on_if_needed' of https://github.com/apollo13/ansible into apollo13-sudo_on_if_needed 2015-09-17 11:03:27 -04:00
Toshio Kuratomi
8532ddd5ed typo in formatting the warning message 2015-09-17 08:01:43 -07:00
Toshio Kuratomi
4ae5512fb0 Restore python3 compat fix and fix AnsibleError undefined before use 2015-09-17 07:45:21 -07:00
James Cammarata
c881b1acae Merge pull request #12407 from mgedmin/py3k
Python 3: basestring in play_context.py
2015-09-17 10:40:57 -04:00
James Cammarata
8acbc00853 Fixing groups variable structure to match v1
Fixes #12418
2015-09-17 10:24:03 -04:00
Abhijit Menon-Sen
349eec7855 Fix missing colon (typo) in IPv6 pattern 2015-09-17 19:34:33 +05:30