Commit graph

5051 commits

Author SHA1 Message Date
Toshio Kuratomi
46bf3d1bc1 Remove plugin._basedirs as we now use different data structures to achieve that functionality. 2015-09-10 10:41:31 -07:00
Jiri tyr
174f805fb3 Resurrection of the comment filter 2015-09-10 14:37:42 +02:00
Marius Gedminas
b95e3d18a7 Python 3: use the right PyYAML SafeRepresenter for unicode
PyYAML has a SafeRepresenter in lib/... that defines

    def represent_unicode(self, data):
        return self.represent_scalar(u'tag:yaml.org,2002:str', data)

and a different SafeRepresenter in lib3/... that defines

    def represent_str(self, data):
        return self.represent_scalar('tag:yaml.org,2002:str', data)

so the right thing to do on Python 3 is to use represent_str.

(AnsibleUnicode is a subclass of six.text_type, i.e. 'str' on Python 3.)
2015-09-10 08:57:53 +03:00
Benno Joy
6de13c3062 Merge pull request #11818 from Java1Guy/module_utils_ec2_for_ecs
change ec2 util to create only requested objects
2015-09-10 11:06:19 +05:30
James Cammarata
ee7905c7e1 Compare include params against deprecated param names too
Fixes #12282
2015-09-09 22:15:50 -04:00
James Cammarata
950e067d2f Merge pull request #12295 from napkindrawing/patch-3
Typo in comment: "bsaename" → "basename"
2015-09-09 21:41:13 -04:00
James Cammarata
f576e0f1af Merge pull request #12300 from robusto/devel
Vault missing DataLoader for pwd file #12293
2015-09-09 18:52:31 -04:00
James Cammarata
22639a2a22 Fix mistake in fact setting to nonpersistent cache in VariableManager
Fixes #12301
2015-09-09 18:28:39 -04:00
James Cammarata
aa3b290efd Fix indent on line in playbook executor 2015-09-09 18:10:38 -04:00
Toshio Kuratomi
4b895f04e3 Fix order of loading of modules.
Allows ANSIBLE_LIBRARY to overload core modules even if the module in
ANSIBLE_LIBRARY doesn't have a .py extension.
2015-09-09 15:00:26 -07:00
James Cammarata
e81947c3b4 Fix vars_prompt short form support (and mark deprecated) 2015-09-09 17:52:44 -04:00
Blake Atkinson
6062519986 Vault missing DataLoader for pwd file #12293 2015-09-09 15:33:52 -05:00
James Cammarata
be7bd392cb Fix unreachable host detection and have plays continue after unreachable failures
Fixes #12284
2015-09-09 15:27:26 -04:00
Toshio Kuratomi
25c97fff69 Merge pull request #12294 from ansible/fix-password-lookup
Fix problem with "=" in the initial file path.
2015-09-09 11:48:30 -07:00
Brian Coca
0dbebfddaa Revert "remove closing connections after every task, this goes against conneciton caching and was not expected behaviuor nor inhertited from v1"
needed for winrm, disabled closing connections in ssh to avoid issues with that persistance, need to normalize all this in future

This reverts commit 23a22397bf.
2015-09-09 14:21:56 -04:00
Brian Coca
3f0e14c2a0 disabled ssh closing connections 2015-09-09 14:21:46 -04:00
Toshio Kuratomi
e2c49b4ef4 Fix problem with "=" in the initial file path. 2015-09-09 11:14:31 -07:00
James Cammarata
56d7f3889d Add new level of vars to deal with set_fact/register precedence rules 2015-09-09 12:21:07 -04:00
napkindrawing
172bbac97a Typo in comment: "bsaename" → "basename" 2015-09-09 11:55:40 -04:00
James Cammarata
926f127245 set_fact should put facts in the fact_cache, not vars_cache
Also fixes a template path lookup issue when using an include

Fixes #12277
2015-09-09 11:10:18 -04:00
Toshio Kuratomi
e2e9319f06 Merge pull request #12287 from mgedmin/py3k
Python 3: use six.string_types instead of basestring
2015-09-09 01:35:46 -07:00
Toshio Kuratomi
fa2edfa1ef * Make sure we don't sometimes get byte strings instead of unicode strings
* Turn strings into byte strings before passing to shlex and turn them
  back into unicode strings after they are retyurned from there

  Fixes #12257
2015-09-09 01:24:11 -07:00
Toshio Kuratomi
e67459df86 Fix docstring of parse_value() to be informative 2015-09-09 01:24:11 -07:00
Marius Gedminas
ba0cf42ce9 Python 3: use six.string_types instead of basestring
Reduces the number of failing tests from 35 to 17.
2015-09-09 09:25:23 +03:00
James Cammarata
30399edada Use UnsafeProxy for lookup results too
Also fixes a couple of bugs that popped up when using the proxy class
2015-09-08 12:19:39 -04:00
Brian Coca
f1c72a55a5 Merge pull request #12261 from toshywoshy/12169
added Manjaro as an Archlinux family
2015-09-08 08:24:01 -04:00
Victor Costan
bd405169fb Fix #12258: don't wrap None in UnsafeProxy. 2015-09-08 04:02:04 -04:00
James Cammarata
ac6f61ec7f Properly handle group/host_var files with dots in them
Fixes #12245
2015-09-08 03:20:09 -04:00
James Cammarata
1eb5b42705 Don't double proxy unsafe variables
Potential fix for #12260
2015-09-08 02:50:07 -04:00
Gaurav Jain
6c107258fa Fix rsync connections to IPv6 addresses
Similar to https://github.com/ansible/ansible/pull/11816 we can unconditionally
wrap the host address in square brackets. This is required by rsync for IPv6
addresses.
2015-09-06 22:20:45 -07:00
h-hirokawa
cd478a76fc Add the new module replacer <<INCLUDE_ANSIBLE_MODULE_JSON_ARGS>>.
Fixes #12266
2015-09-07 12:01:26 +09:00
Toshio Kuratomi
718812d945 update submodule refs for documentation formatting 2015-09-05 08:56:15 -07:00
Toshio Kuratomi
26e109aa13 Update module refs 2015-09-05 08:27:48 -07:00
Toshaan Bharvani
4702d2be09 added Manjaro as an Archlinux family 2015-09-05 12:49:20 +02:00
James Cammarata
ff9f5d7dc8 Starting to add additional unit tests for VariableManager
Required some rewiring in inventory code to make sure we're using
the DataLoader class for some data file operations, which makes mocking
them much easier.

Also identified two corner cases not currently handled by the code, related
to inventory variable sources and which one "wins". Also noticed we weren't
properly merging variables from multiple group/host_var file locations
(inventory directory vs. playbook directory locations) so fixed as well.
2015-09-04 16:41:38 -04:00
James Cammarata
87f75a50ad Use proxy class to mark variables unsafe for templating
Fixes #12191
2015-09-04 16:38:06 -04:00
Toshio Kuratomi
2d251cba45 Merge pull request #12256 from ansible/password-lookup-fix
Use parse_kv to correctly parse parameters from password lookup
2015-09-04 12:27:18 -07:00
Toshio Kuratomi
c68e04c185 Use parse_kv to correctly parse parameters from password lookup
Fixes https://github.com/ansible/ansible-modules-core/issues/1968
2015-09-04 12:18:46 -07:00
Brian Coca
6e5f3db9de updated submodule refs 2015-09-04 12:17:34 -04:00
Brian Coca
c17fbf2f12 simplify become testing and handling, we had drifted and were doulbe checking prompt, become and become_pass
fixed tests to conform to new signature and now tests both with and w/o password
now we are more explicit about self.prompt
2015-09-04 10:36:16 -04:00
Brian Coca
eb0e7e198b remove closing connections after every task, this goes against conneciton caching and was not expected behaviuor nor inhertited from v1 2015-09-04 10:36:15 -04:00
Marius Gedminas
37be9539ff Python 3: use six.text_type instead of unicode
Replace 'unicode' with six.text_type, everywhere but in module_utils.
2015-09-04 08:40:10 +03:00
James Cammarata
1840906f74 Merge pull request #12236 from amenonsen/ssh-cpdir
Go back to using ~/.ansible/cp as the ControlPath
2015-09-03 21:12:45 -04:00
Toshio Kuratomi
e218bf8da9 Do not escape backslashes when processing a template file.
Fixes #12198
Fixes #12241
2015-09-03 17:45:17 -07:00
Abhijit Menon-Sen
7aa6cd3f63 Go back to using ~/.ansible/cp as the ControlPath
This was commented out earlier because of the lack of interprocess
locking and prepare_writeable_dir in v2.

The locking was not needed: it could only protect against other siblings
of this process (since they were all locking a temporary file that was
opened in the parent), and those would be running as the same user and
with the same umask. Also, os.makedirs() tolerates intermediate paths
being created by other processes. For any other kind of error, both
locking and non-locking code paths would fail in the same way.

So all we really need to do is make sure we have write permissions.

(We also move the cp_dir handling code to where we actually set the
ControlPath ourselves; if the user has set it via ssh_*args already,
we don't need to bother.)
2015-09-04 00:35:13 +05:30
James Cammarata
4ac2bafc4b Set hosts fact gathering flag based on fact cache entries
Fixes #12213
2015-09-03 14:11:19 -04:00
James Cammarata
6650ba7654 Squashed commit of the following:
commit 9921bb9d20
Author: Abhijit Menon-Sen <ams@2ndQuadrant.com>
Date:   Mon Aug 10 20:19:44 2015 +0530

    Document --ssh-extra-args command-line option

commit 8b25595e7b
Author: Abhijit Menon-Sen <ams@2ndQuadrant.com>
Date:   Thu Aug 13 13:24:57 2015 +0530

    Don't disable GSSAPI/Pubkey authentication when using --ask-pass

    This commit is based on a bug report and PR by kolbyjack (#6846) which
    was subsequently closed and rebased as #11690. The original problem was:

        «The password on the delegated host is different from the one I
        provided on the command line, so it had to use the pubkey, and the
        main host doesn't have a pubkey on it yet, so it had to use the
        password.»

    (This commit is revised and included here because #11690 would conflict
    with the changes in #11908 otherwise.)

    Closes #11690

commit 119d032389
Author: Abhijit Menon-Sen <ams@2ndQuadrant.com>
Date:   Thu Aug 13 11:16:42 2015 +0530

    Be more explicit about why SSH arguments are added

    This adds vvvvv log messages that spell out in detail where each SSH
    command-line argument is obtained from.

    Unfortunately, we can't be sure if, say, self._play_context.remote_user
    is obtained from ANSIBLE_REMOTE_USER in the environment, remote_user in
    ansible.cfg, -u on the command line, or an ansible_ssh_user setting in
    the inventory or on a task or play. In some cases, e.g. timeout, we
    can't even be sure if it was set by the user or just a default.

    Nevertheless, on the theory that at five v's you can use all the hints
    available, I've mentioned the possible sources in the log messages.

    Note that this caveat applies only to the arguments that ssh.py adds by
    itself. In the case of ssh_args and ssh_extra_args, we know where they
    are from, and say so, though we can't say WHERE in the inventory they
    may be set (e.g. in host_vars or group_vars etc.).

commit b605c285ba
Author: Abhijit Menon-Sen <ams@2ndQuadrant.com>
Date:   Tue Aug 11 15:19:43 2015 +0530

    Add a FAQ entry about ansible_ssh_extra_args

commit 49f8edd035
Author: Abhijit Menon-Sen <ams@2ndQuadrant.com>
Date:   Mon Aug 10 20:48:50 2015 +0530

    Allow ansible_ssh_args to be set as an inventory variable

    Before this change, ssh_args could be set only in the [ssh_connection]
    section of ansible.cfg, and was applied to all hosts. Now it's possible
    to set ansible_ssh_args as an inventory variable (directly, or through
    group_vars or host_vars) to selectively override the global setting.

    Note that the default ControlPath settings are applied only if ssh_args
    is not set, and this is true of ansible_ssh_args as well. So if you want
    to override ssh_args but continue to set ControlPath, you'll need to
    repeat the appropriate options when setting ansible_ssh_args.

    (If you only need to add options to the default ssh_args, you may be
    able to use the ansible_ssh_extra_args inventory variable instead.)

commit 37c1a5b679
Author: Abhijit Menon-Sen <ams@2ndQuadrant.com>
Date:   Mon Aug 10 19:42:30 2015 +0530

    Allow overriding ansible_ssh_extra_args on the command-line

    This patch makes it possible to do:

        ansible somehost -m setup \
            --ssh-extra-args '-o ProxyCommand="ssh -W %h:%p -q user@bouncer.example.com"'

    This overrides the inventory setting, if any, of ansible_ssh_extra_args.

    Based on a patch originally by @Richard2ndQuadrant.

commit b023ace8a8
Author: Abhijit Menon-Sen <ams@2ndQuadrant.com>
Date:   Mon Aug 10 19:06:19 2015 +0530

    Add an ansible_ssh_extra_args inventory variable

    This can be used to configure a per-host or per-group ProxyCommand to
    connect to hosts through a jumphost, e.g.:

        inventory:
            [gatewayed]
            foo ansible_ssh_host=192.0.2.1

        group_vars/gatewayed.yml:
            ansible_ssh_extra_args: '-o ProxyCommand="ssh -W %h:%p -q bounceuser@gateway.example.com"'

    Note that this variable is used in addition to any ssh_args configured
    in the [ssh_connection] section of ansible.cfg (so you don't need to
    repeat the ControlPath settings in ansible_ssh_extra_args).
2015-09-03 11:26:56 -04:00
James Cammarata
8774ff5f57 Make sure PlayContext is copied when iterating in a with_ loop 2015-09-03 08:11:30 -04:00
James Cammarata
c16961db60 Fix two PlayIterator bugs
* When iterating over a child state, a failure should be propagated
  up so parent blocks don't continue iterating
* Make sure a child state exists before trying to search it

Fixes #12210
2015-09-03 07:48:42 -04:00
Marius Gedminas
38a96d7e8f Python 3: avoid iteritems() in a template
I don't think six.iteritems is available here, but I also don't expect
there to be enough platforms to ever make the speed difference between
.items() and .iteritems() noticeable.
2015-09-03 10:11:32 +03:00
Toshio Kuratomi
364313c01f Simplify and speed up _split_role_params() by moving an invariant outside of a loop. 2015-09-03 00:06:34 -07:00
Toshio Kuratomi
1ae28f606b Refactor to replace filter and lambda with a generator expression 2015-09-03 00:06:34 -07:00
Marius Gedminas
a32bf1ec18 Simplify FactCache.copy()
Also fix the bug (missing from six import iteritems) I introduced in
823677b490.
2015-09-03 09:44:44 +03:00
Marius Gedminas
823677b490 Replace .iteritems() with six.iteritems()
Replace .iteritems() with six.iteritems() everywhere except in
module_utils (because there's no 'six' on the remote host).  And except
in lib/ansible/galaxy/data/metadata_template.j2, because I'm not sure
six is available there.
2015-09-03 09:23:27 +03:00
James Cammarata
9e1bc434c7 Further cleanup from locking changes 2015-09-03 01:07:29 -04:00
James Cammarata
b93f27e260 Move the lockfile back to tqm to make sure it stays unique 2015-09-03 00:45:42 -04:00
James Cammarata
b9afbf0ee4 Reorganizing the way the connection lockfile is created 2015-09-03 00:18:52 -04:00
James Cammarata
7034bbef30 Merge branch 'connection-locking' of https://github.com/amenonsen/ansible into amenonsen-connection-locking 2015-09-02 23:54:52 -04:00
Abhijit Menon-Sen
9378c8e2da Make the paramiko plugin use locking 2015-09-03 08:43:21 +05:30
Abhijit Menon-Sen
5887e96b27 Introduce a connection locking infrastructure
The lock file is (a temporary file) opened in the parent process, whose
open fd is inherited by the workers after fork, and passed down through
the PlayContext. Connection grows lock/unlock methods which can be used
by individual connection plugins.
2015-09-03 08:43:21 +05:30
Abhijit Menon-Sen
12d8f9132e Make lock_host_keys a real noop, instead of half-a-noop
Right now, we don't do any locking, but we still scan known_hosts files
twice per connection. That's completely unnecessary, and the proposed
solutions to the locking problem wouldn't need known_hosts scanning
anyway, so this code can go away.
2015-09-03 05:52:57 +05:30
Brian Coca
a006e85947 removed printing module args as it violates no_log settings 2015-09-02 16:14:40 -04:00
Toshio Kuratomi
1998eddf8d Keep newlines by default now.
In v1, a trailing newline was kept if the parameter was passed as key=value.  If
the parameter was passed as yaml dict the trailing newline was
discarded.  Since key-value and yaml dict were unified in v2 we have to
make a choice as to which behaviour we want.  Decided that keeping trailing
newlines by default made the most sense.

Fixes #12200
Fixes #12199
2015-09-02 11:55:45 -07:00
Toshio Kuratomi
41da8de094 Speedup for counting newlines 2015-09-02 10:59:51 -07:00
James Cammarata
843138e7d9 Merge pull request #12202 from mgedmin/py3k
Use rich comparisons for Attribute and FieldAttribute
2015-09-02 13:57:04 -04:00
James Cammarata
5434ed973b Make sure locals in safe_eval is a plain dict when running eval
Fixes #12206
2015-09-02 13:54:58 -04:00
Toshio Kuratomi
417bf1c805 Unittest the _count_trailing_newlines function 2015-09-02 08:59:31 -07:00
Toshio Kuratomi
7ed746ad45 Fix preserve_trailing_newlines (broken by 7f5080f64a )
Fix for one half of hte bug reported in #12198
2015-09-02 08:47:20 -07:00
Brian Coca
a7231c2203 actually implemented flags correctly for all priv escalation methods 2015-09-02 11:31:39 -04:00
Brian Coca
b9642585bd added missing become_exe from config 2015-09-02 09:52:26 -04:00
Brian Coca
e156d9b677 fixed and generalized privilege escalation exe settings 2015-09-02 09:29:34 -04:00
Marius Gedminas
33a374edde Use rich comparisons for Attribute and FieldAttribute
Because __cmp__ is not supported on Python 3.
2015-09-02 12:33:42 +03:00
Marius Gedminas
ca908f2e12 Fix indentation to be a multiple of 4 2015-09-02 12:22:18 +03:00
Abhijit Menon-Sen
147b04b682 Remove duplicated function definition (fallout from earlier repeated merge) 2015-09-02 07:47:48 +05:30
James Cammarata
f162990cb3 Properly assign the searchpath for templates to the environment loader
dbd755e0 previously assigned the value to self._templar.environment.searchpath,
which is incorrect - it needs to be assigned to the environment.loader.searchpath
value instead.

Fixes #11931
2015-09-01 17:33:14 -04:00
James Cammarata
2761df232e Small tweak to 7551b75 to correct the ordering of tests 2015-09-01 14:48:59 -04:00
James Cammarata
7551b75e61 Add ispath type for constants and make sure all local paths are ispath=True
Fixes #12180
2015-09-01 14:47:45 -04:00
Toshio Kuratomi
aeff960d02 Cleanup combine_vars
* Dedupe combine_vars() code (removed from VariableManager)
* Fix merge_hash algorithm to preserve the type
* unittest combine_vars and merge_hash
2015-09-01 11:23:12 -07:00
James Cammarata
7fe495d619 Merge pull request #12187 from caphrim007/add-utf-8-encoding-to-f5-module-utils
Adds utf-8 encoding to f5 module_utils
2015-09-01 14:15:42 -04:00
James Cammarata
d00c445d2c Merge pull request #12188 from amenonsen/easy-debug
Show module name and args with -vvv
2015-09-01 14:15:15 -04:00
Abhijit Menon-Sen
d6d523bcae Show module name and args with -vvv
This information was earlier shown only with ANSIBLE_DEBUG, but it's
extremely useful in a user context, especially with module invocations
with deeply nested args like the ec2_vpc/ec2 modules.

Closes #11680
2015-09-01 23:44:13 +05:30
James Cammarata
3e1a774ba5 Don't use iteritems() in templar to avoid (hostvars) dict size change errors 2015-09-01 14:11:23 -04:00
James Cammarata
974b69d236 looped includes need unique references to prevent information bleeding
Fixes #12155
2015-09-01 12:32:35 -04:00
James Cammarata
a341a8a093 Make sure tasks from a role see their defaults above all others
Fixes #12081
2015-09-01 11:27:35 -04:00
Tim Rupp
6f41a72158 Adds utf-8 encoding to f5 module_utils
The contributor's name on line 10 (originally line 7) includes a character
that the default Python encoding (ASCII) raises an error on when interpreting
the file.

Specifying the utf-8 encoding, as is done in other modules, resolves
the error.

The error being raised is

SyntaxError: Non-ASCII character '\xc3' in file /.../lib/ansible/module_utils/f5.py
on line 7, but no encoding declared; see http://www.python.org/peps/pep-0263.html
for details
2015-09-01 08:10:23 -07:00
James Cammarata
a7da25d48b Submodule update 2015-09-01 11:03:19 -04:00
James Cammarata
3d282cd1b0 Also always post_validate Handler class as well
Fixes #12154
2015-09-01 11:02:42 -04:00
Toshio Kuratomi
12edbfeb0b Merge pull request #12177 from mgedmin/py3k
Make combine_vars() compatible with Python 3
2015-09-01 08:00:08 -07:00
Brian Coca
66e2eb2231 removed as it does not fool setup.py to include data and it must be specified in setup.py 2015-09-01 10:58:58 -04:00
Marius Gedminas
54dbfba8f8 Make combine_vars() compatible with Python 3
Fixes

  TypeError: unsupported operand type(s) for +: 'dict_items' and 'dict_items'

on Python 3.
2015-09-01 09:39:59 +03:00
James Cammarata
5adcd7054b Allow setup to run when using --start-at-task
Fixes #12147
2015-08-31 22:10:34 -04:00
Brian Coca
7458331539 package and service now check that module exists before trying to execute it 2015-08-31 20:34:20 -04:00
Toshio Kuratomi
9d193d8fb4 Merge pull request #12173 from ansible/pr/10204
Rebase of 10204 - Add host key for ssh url only.
2015-08-31 13:50:40 -07:00
James Cammarata
fca27c29f4 Properly fail if slurp fails during fetch 2015-08-31 16:45:03 -04:00
Desmond O. Chang
58c3539196 Add host key for ssh url only.
Rewrite function `get_fqdn`.  It returns fqdn for all kinds of urls now.

`add_git_host_key` determines whether a url is ssh and whether its host
key should be added.
2015-08-31 13:42:32 -07:00
James Cammarata
66a2f2923e Look for proper values in slurp result during fetch with become enabled 2015-08-31 16:36:07 -04:00
Toshio Kuratomi
7f5080f64a Fix backslash escaping inside of jinja2 expressions
Fixes #11891
2015-08-31 13:17:26 -07:00
Toshio Kuratomi
5ae300a084 Update module refs 2015-08-31 08:05:45 -07:00
Brian Coca
6b73cac35a corrected skipped_hosts constant usage, thanks @benjixx 2015-08-30 20:21:15 -04:00
Brian Coca
5a65dce6fb fix issues with csh redirection 2015-08-30 17:49:16 -04:00