* Remove unnecessary copying of values from parents to role deps, as
this can cause problems when roles have multiple parents (or the same
parents with different params speficied through deps)
* Since we're already checking the dep chain in the block for role
things (which every task in a role should have), it is not necessary
to check the role directly in case it improperly grabs something
Fixes#14438
Our custom encoder for the to_json filter was simply returning the
object if it was not a HostVars object, leading in some cases to a
TypeError when the data contained an undefined variable. This lead
to an odd error message being propagated up, so we now properly catch
this as an undefined variable error.
Fixes#15610
Again, as we're carrying failed/unreachable hosts forward from play to play via
internal structures, we need to remember which ones had previously failed so that
unrelated host failures don't inflate the numbers for a given serial batch in the
PlaybookExecutor causing a premature exit.
Fixes#16364
The listen statement on handlers should have supported a list, however
it was broken in the revision of the pub/sub feature based on the handler
revamp. This patch corrects the bug, so this works again:
- name: some handler
...
listen:
- some target
- another target
Fixes#16378
* add new module network
* move EOS to NetworkModule
* shell.py Python 3.x compatibility
* implements the Command class through the connection for eos
This implements a new Command class that specifies the cli command
and output format. This removes the need to batch commands through
the connection
* initial add of netcmd module
Due to the fact that roles may be instantiated with different sets of
params (multiple inclusions of the same role or via role dependencies),
simply tracking notified handlers by name does not work. This patch
changes the way we track handler notifications by using the handler
object itself instead of just the name, allowing for multiple internal
instances. Normally this would be bad, but we also modify the way we
search for handlers by first looking at the notifying tasks dependency
chain (ensuring that roles find their own handlers first) and then at
the main list of handlers, using the first match it finds.
This patch also modifies the way we setup the internal list of handlers,
which should allow us to correctly identify if a notified handler exists
more easily.
Fixes#15084
This removes the extra layer of quotes around values in the 'args' file.
These quotes were there before the pipes.quote() call was added, but
were not removed, resulting in too much quoting.
Manifests as the following stack trace
File "/usr/local/Cellar/ansible/2.0.1.0/libexec/lib/python2.7/site-packages/ansible/utils/display.py", line 259, in error
new_msg = u"ERROR! " + msg
TypeError: coercing to Unicode: need string or buffer, AnsibleParserError found
This makes Ansible no longer set LC_ALL for remote systems. It is up to
the individual modules to set LC_ALL if they need it for screenscraping
the output from a program.
This is the 2.2 followup for #15138
Problem: When setting the file permissions on the remote server for
unprivileged users ansible expects that a chown will fail for unprivileged
users. For some systems (e.g. HP-UX) this is not the case.
Solution: Change the order how ansible sets the remote permissions.
* If the remote_user sudo's to an unprivileged user then we attempt to
grant the unprivileged user access via file system acls.
* If granting file system acls fails we try to change the owner of the
file with chown which only works in case the remote_user is privileged
or the remote systems allows chown calls by unprivileged users (e.g.
HP-UX)
* If the chown fails we can set the file to be world readable so that
the second unprivileged user can read the file. Since this could allow
other users to get access to private information we only do this
ansible is configured with "allow_world_readable_tmpfiles" in the
ansible.cfg
When the PYTHONPATH is an empty string python will treat it as though
the cwd is in the PYTHONPATH. This can be undesirable. So make sure we
delete PYTHONPATH from the environment altgether in this case.
Fixes#16195
Symlinks inside of the chroot were failng because we weren't able to
determine if they were pointing to a real file or not. We could write
some complicated code to walk the symlink path taking into account where
the root of the tree is but that could be fragile. Since this is just
a sanity check, instead we just assume that the chroot is fine if we
find that /bin/sh in the chroot is a symlink. Can revisit if it turns
out that many chroots have a /bin/sh that's a broken symlink.
Fixes#16097
The junos network module will now properly use the ssh key file if its
passed from the playbook to authenticate to the remote device. Prior
to this commit, the ssh keyfile was ignored.
When setuptools installs a python module (as is done via python setup.py
install) It puts the module into a subdirectory of site-packages and
then creates an entry in easy-install.pth to load that directory. This
makes it difficult for Ansiballz to function correctly as the .pth file
overrides the sys.path that the wrapper constructs. Using
sitecustomize.py fixes this because sitecustomize overrides the
directories handled in .pth files.
Fixes#16187
AIX ssh does not seem to like compression, moved it to ssh_args
to allow making it configurable. Note that those using ssh_args
already will need to add it explicitly to keep compression.
* Give a module the possibility to known its own name
This is useful for logging and reporting and fixes the longstanding problem with syslog-messages:
May 30 15:50:11 moria ansible-<stdin>: Invoked with ...
now becomes:
Jun 1 17:32:03 moria ansible-copy: Invoked with ...
This fixes#15830
* Rename the internal name from module.ansible_module_name to module._name
* Fix: create retry_files_save_path if it doesn't exist
Ansible documentation states that retry_files_save_path directory will be
created if it does not already exist. It currently doesn't, so this patch
fixes it :)
* Use makedirs_safe to ensure thread-safe dir creation
@bcoca suggested to use the makedirs_safe helper function :)
The changes to exclude implicit localhosts from group patterns exposed
the bug that we sometimes create multiple implicit localhosts, which
caused some bugs with things like includes, where the host was used as
an entry into a dict, so having multiple meant that the incorrect host
(with a different uuid) was found and includes were not executed for
implicit localhosts.
This allows the PlaybookExecutor to receive more information regarding
what happened internal to the TaskQueueManager and strategy, to determine
things like whether or not the play iteration should stop.
Fixes#15523
The nxos cli provider would not properly handle ssh key files passed
from the playbook task. The ssh_keyfile argument is now properly
passed to the ssh authentication method
This fix address the bug reported in #3862
Also updates doc on variable precedence, as it was incorrect for the
order of play vars/vars_prompt/vars_files in relation to set_fact and
registered variables.
Fixes#14702Fixes#14826
Since we now use the PlayIterator to carry forward failures from previous
play executions, in the event that some hosts which had previously failed
are not in the current inventory we now create a stub state instead of
raising an error.
Exception was raised when trying to use ssh-agent for authentication to
ios devices. This fix enables ssh-agent and enable use of password
protected ssh keys. There is one additional fix to capture authentication
exceptions nicely.
* Port urls.py to python3
Fixes (largely normalizing byte vs text strings) for python3
* Rework what we do with attributes that aren't set already.
* Comments
Has already been transferred as a tempfile.
This fixes the error in https://github.com/ansible/ansible/issues/16125
but there may be higher level issues that should be fixed as well (other
modules might be able to cause status fields like failed and changed to
return a censored string instead of a bool). So leaving 16125 open for
now.
If someone run:
ansible all -m file state=present
The error message is "Missing target hosts" which is misleading, since
the target hosts is here, the problem is the missing '-a'.
* In the VariableManager, we were not properly tracking if a file
had already been loaded, so we continuously append data to the end
of the list there for host and group vars, meaning large sets of data
are duplicated multiple times
* In the inventory, we were merging the host/group vars with the vars
local to the host needlessly, as the VariableManager already handles that.
This leads to needless duplication of the data and makes combining the
vars in VariableManager take even longer.
The output of 'ansible-galaxy info' was formatting the
'galaxy_info' key with one char per line.
Previously, when building the output string, items in
role_info that had a dict for value, the label for
it's key ('galaxy_info' for ex) was being added to
the text list in addition to being appended. Only
the append is needed.
Also added a unit test in test/units/cli/test_galaxy.py,
but skip it on py3 until galaxy is py3 compatible.
fixes#15177
Ansible excessively checks the file system for the potential presence of
`group_vars` and `host_vars` files.
For large numbers of groups this leads to combinatorial performance
issues.
This commit generates a set of group_vars and host_vars filenames using
`os.listdir()` in every possible location and then checks against the sets
before making a stat of the file system.
Also included in this commit is caching of the base directory lookup
for the inventory.
This makes it possible to use anything other than a list (e.g., a
tuple, or dict.keys() in py3k) for argument_spec choices. It also
improves the error messages if you don't use a list type.
Child blocks (whether nested or via includes) don't get a copy of the
dependency chain, so the above method should be used to ensure the block
looks at its parents dep chain.
Fixes#15996
* readd the service action plugin, was removed cause it created unexpected fact gathering and there are no split service plugins that would make this useful (yet)
Revert "removed action plugin as service facts and separate modules don't work yet and this forces gathering facts"
This reverts commit 7368030651.
* now only does minimal fact gathering
This class can be used by F5 modules for raising exceptions.
This should be used to handle known errors and raise them so
that they can be printed in the fail_json method.
The common Exception class built-in should not be used because
it hides tracebacks that are necessary to have when debugging
problems with the module.
* Catch DistributionNotFound when pycrypto is absent
On Solaris 11, module `pkg_resources` throws `DistributionNotFound` on import if `cryptography` is installed but `pycrypto` is not. This change causes that situation to be handled gracefully.
I'm not using Paramiko or Vault, so I my understanding is that I don't
need `pycrpto`. I could install `pycrypto` to make the error go away, but:
- The latest released version of `pycrypto` doesn't build cleanly on Solaris (https://github.com/dlitz/pycrypto/issues/184).
- Solaris includes an old version of GMP that triggers warnings every time Ansible runs (https://github.com/ansible/ansible/issues/6941). I notice that I can silence these warnings with `system_warnings` in `ansible.cfg`, but not installing `pycrypto` seems like a safer solution.
* Ignore only `pkg_resources.DistributionNotFound`, not other exceptions.
With some earlier changes, continuing to forward failed hosts on
to the iterator with each TQM run() call was causing plays with
max_fail_pct set to fail, as hosts which failed in previous plays
were counting those old failures against the % calculation.
Also changed the linear strategy's calculation to use the internal
failed list, rather than the iterator, as this now represents the
hosts failed during the current run only.
This change makes it so we know when it is safe to get rid of the module
(when we stop supporting python2.4) and makes it easier for us to find
code that is using the functions in there to update.
If needed, we'll create a pycompat26 and pycompat27 as well. These
files are for functions that are needed on that python version to write
portable code. So python-2.4 compatible modules may need code in
pycompat24, python26+ modules may need code in pycompat26, etc. If
a function is needed in multiple python versions, we should implement it
in an internal common file and use import to put it in the namespace for
each pycompatXY module.
As noted in the comment, the TQM may be used for more than one play. As such,
after creating the new PlayIterator object it is necessary to mark any failed
hosts from previous calls to run() as failed in the iterator, so they are
properly skipped during any future calls to run().
Since the pyrax website say that only python 2.7 is tested,
I do not think it is worth to aim for python 2.4 compatibility
for the various rackspace modules.
Since this is now the default package manager, it got moved
to another location on Netbsd :
netbsd# type pkgin
pkgin is a tracked alias for /usr/pkg/bin/pkgin
netbsd# uname -a
NetBSD netbsd.example.org 6.1.4 NetBSD 6.1.4 (GENERIC) amd64
But since the package manager is also used outside of NetBSD, we
have to keep the /opt/local path too.
The change is needed to support the multiple include statements
inside the jinja2 template file, as in '{% include ['another.j2'] %}'.
statement. I need this capability, as OpenSwitch `switch` role needs
to handle multiple *.j2 files and supporting the include statement
inside jinja2 file is essential, otherwise I need to combine multiple
template files into a single file, which easily causes conflicts
between developers working on different parts of the teamplate, ports
and interface.
Since it depend on libcloud and libcloud requirements include python 2.6
since libcloud 0.4.0 (https://libcloud.apache.org/about.html), which
was released in 2011 Q2, and GCE drivers were added in 2013,
we can't run a libcloud version with GCE support on 2.4.
Since the modules can use a paramiko transport (ergo
python 2.4 syntax), we need to keep compat with 2.4 and python 3,
so we need to use the get_exception trick, even if the various juniper
libraries are not compatible with 2.4.
It currently fail with
ansible/module_utils/facts.py\", line 357, in get_service_mgr_facts\r\nKeyError: 'distribution'\r\n"
Since self.facts['distribution'] is used after, we need to make sure
this is set by default and if needed, corrected somewhere for Linux.
* more robust hashi_vault module, and allow querying specific field in secret-dict
* allow fetching entire secret dict with trailing ':'
* process comment by bcoca for PR #13690
Initialize facts['distribution'] with self.system so that this fact does
not remain uninitialized on systems_platform_working platforms (FreeBSD,
OpenBSD).
Fixes#15841
Prior to this patch, the retry/until logic would fail any task that
succeeded if it took all of the alloted retries to succeed. This patch
reworks the retry/until logic to make things more simple and clear.
Fixes#15697
When using run_once, there is only one dict of facts so passing that
to the VariableManager results in the fact cache containing the same
dictionary reference for all hosts in inventory. This patch fixes that
by making sure we pass a copy of the facts dict to VariableManager.
Fixes#14279
* Update GCE module to use JSON credentials
* Ensure minimum libcloud version when using JSON crednetials for GCE
* Relax langauge around libcloud requirements
In the free strategy, we mark a host as blocked when it has work to do
(the PlayIterator returns a task) to prevent multiple tasks from being sent
to the host. However, we check for role duplicates after setting the blocked
flag, but were not clearing that when the task was skipped leading to an
infinite loop. This patch corrects that by clearing the blocked flag when
the task is skipped.
Fixes#15681
Previously the changed code was necessary, however it is now problematic
as we've started using the is_failed() method in other places in the code.
Additional changes at the strategy layer should make this safe to remove
now.
Fixes#15625
Fixes#15745
Applies conditional forwarding to all tasks/roles within the included playbook.
The existing line only applies forwarded conditionals to the main Task block, and misses pre_, post_, and roles.
Typo ::
Made a selection mistake when I copied over the one line change
In VariableManager, we fetch the params specifically in the next step,
so including them in the prior step is unnecessary and could lead to things
being overridden in an improper order.
In Block, we should not be getting the params for the role as they are
included earlier via the VariableManager.
Fixes#14411
This patch adds the port argument as a valid parameter to the f5_spec.
This argument is supported in bigsuds version 1.0.4 and greater, so
this patch uses the __version__ variable of the bigsuds module to
determine when the port value should be honored by the module.
By default the `Shell` class disables ssh agents. The `junos_netconf`
module uses this class, but doesn't re-enable agents.
Here it's explicitly enabled again, so an ssh agent can be used to
connect to and configure Junos devices.
* Don't filter hosts remaining based on their failed state. Instead rely
on the PlayIterator to return None/ITERATING_COMPLETE when the host is
failed.
* In the free strategy, make sure we wait outside the host loop for all
pending results to be processed.
* Use the internal _set_failed_state() instead of manually setting things
when a failed child state is hit
Fixes#15623
In `lib/ansible/executor/play_iterator.py`, ansible sets a host's
`_gathered_facts` property to `True` without checking to see if there
are any tasks to be executed. In the event that the entire play is
skipped, `_gathered_facts` will be `True` even though the `setup`
module was never run.
This patch modifies the logic to only set `_gathered_facts` to `True`
when there are tasks to execute.
Closes#15744.
This changeset addresses the issue reported here:
ansible/ansible-modules-core#1765
The yum module (at least) includes its task results as strings, rather than
dicts, and the code this changeset replaces assumed that in that instance the
task was skipped. The updated behaviour assumes that the task has been
skipped only if:
* results exist, and
* all results are dicts that include a truthy skipped value
Issue #15633 observes that a meta: inventory_refresh task causes the playbook
to exit. An inventory refresh flushes all caches and rebuilds all host
objects, assigning new UUIDs to each. These new host UUIDs currently fail to
match those on host objects stored for restrictions in the inventory, causing
the playbook to exit for having no hosts to run further tasks against.
This changeset attempts to address this issue by storing host restrictions
by name, and comparing inventory host names against these names when applying
restrictions in get_hosts.
* better fix for arch version detection
fixes #15696
* be extra safe about tracebacks in facts.py
* add comments to explain the setup
* make allowempty more conservative, ignore file content
* wrap function call in try/except
* should never happen, but if it happens the bug should be distribtion=N/A and not a traceback
* When var name is the same as var content, try to template it before reporting that var is not defined.
Add asserts in test_var_blending to check this special corner case.
* Fix integration tests when using debug with list or dict.
It wasn't doing anything that a literal couldn't do and used
unicode_escape which only understands latin1 (The author of the code
seems to have thought it took an encoding argument but it looks like
that was silently ignored.)
This was reinitialized every time we forked before so we weren't sharing
the same Locks. It also was not accounting for modules which were
directly invoked by an action plugin instead of going through the
strategy plguins.
* add tests for centos6, rhel6 and rhel7
* gen_distribution_version_testcase with python2.6
* remove unused imports
* fix redhat/vmware/... parsing
* add centos7 test case
* Make ziploader's ansible and ansible.module_utils libraries into
namespace packages.
* Move __version__ and __author__ from ansible/__init__ to
ansible/release.py. This is because namespace packages only load one
__init__.py. If that is not the __init__.py with the author and
version info then those won't be available.
* In ziplaoder, move the version ito ANSIBLE_CONSTANTS.
* Change PluginLoader to properly construct the path to the plugins even
when namespace packages are present.
* split code as separate class
* split different distributions as individual functions
* keep program logic mostly identical (for now)
* increase readability, reduce complexity/indentation
* make future testing/refactoring easier
* step towards making distribution parsing independent of the Facts class
* add some changes to make facts.py python3 parsable
* Add shared functionality to return list of security group IDs from list of security group names - this functionality can be used by nearly all ec2 modules
* Improved doc string for get_ec2_security_group_ids_from_names function
* Clarify exception handling in EOS
Also modify to EOS to standardize modules. It makes vimdiff a lot less angry
* Move IOS exception handling into Cli
* Move IOS-XR exception handling into Cli
* Move JUNOS exception handling into Cli
* Move NXOS exception handling into Cli
And reorganize to make it match the other modules
* Move OpenSwitch exception handling into Cli
More speculative restructuring here
added warnings for invalid kwargs
sniff supported authtypes (for new pywinrm)
use default authtypes (for old pywinrm)
error on unsupported authtype
allow no username/password to be specified (kerb SSO)
tested w/ old and new pywinrm
hacky CLIXML parsing of stderr
* HTTPError can also function as a non-exceptional file-like return value (the same thing that urlopen() returns)
* HTTPError - adding response to info dictionnary
* HTTPError - adding response to info dictionnary
* HTTPError - adding body response to info dictionnary
* use list of possible directories directly instead of checking distribution info
* this could fail if someone has keys in one of the other directories, but there could also be custom ssh key directories, which
are not checked at all
* this is work towards separating Facts from Distribution in facts.py
Ensure that initial setup in creating the key directory for ansible
accelerate mode keys, and generation/storage of the key for a
particular host are completed in a multiprocess safe manner.
Creating directories/files and then assigning permissions and contents
to them means that paths may exist and satisfy the os.path.exists()
method in python for other processes before they are usable.
Use a combination of locking around operations with use of unique named
files and an OS file system move to ensure that the conditions of
checking whether a file or directory exists, where it is potentially
created by another thread, will only succeed when the file has both the
correct contents and permissions.
Fixes#13850
Previously we were first checking the fail/run state of the child
state for tasks/rescue/always portions of the block. Instead we are now
always recursively iterating over the child state and then evaluating
whether the child state is failed or complete before changing the failed/
run state within the current block.
Fixes#14324
This update will attempt to guess the file format based on the template
extension if the format argument isnt set. It will also set the commit
comment to the task name if the comment isnt' explicitly defined.
This adds a check to validate the arugment for config_format kwarg in
get_config. If the specified format is not a valid option, the shared
module will call fail_json
Netconf support is provided using the junos-eznc library and the shared
module depends on junos-eznc to be installed on the local Ansible host. This
commit also adds changes to the netcfg library to handle receiving messages
over netconf.
To use netconf, specify transport=netconf for junos module. Be sure that
netconf has been enabled on the remote device.
Which we're use on a case-by-case basis if we find people were actually
using comma-separated strings for list values outside of hosts. Support
for doing so is now deprecated and users should instead use the full
YAML syntax for lists of values.
Fixes#15291
* Don't rely on username to check for root privileges
The SSH username isn't a reliable way to check if we've got root privileges on
the remote system (think "toor" on FreeBSD). Because of this check, Ansible
previously tried to use the fallback solutions for granting file access (ACLs,
world-readable files) even on systems where it had root privileges when the
remote username didn't match the literal string "root".
Instead of running checks on the username, just try using `chmod` in any case
and fall back to the previous "non-root" solution when that fails.
* Fail if we are root and changing ownership failed
Since this code is security sensitive we document exactly the expected
permissions of the temporary files once this function has run. That way
if a flaw is found in one end-result we know more precisely what scenarios
are affected and which are not.
Instead of using the old OpenSwitch runconfig, we'll use
Mir's new ops.dc declarative config for the DC interaction
with OpenSwitch. This gives us the clearer separation between
ansible and the OpenSwitch, as well as the performance
improvement done inside the ops.dc module itself.
Squashed the original Mir's change into single commit.
Tested-by: Kei Nohguchi <kei@nohguchi.com>
* Move zipcache temp dir creation into the locked section otherwise it
races with other workers.
* Catch IOError and turn it into an AnsibleError. IOErrors can hang
multiprocessng.
Updated python module wrapper explode method to drop 'args' file next to module.
Both execute() and excommunicate() debug methods now pass the module args via file to enable debuggers that are picky about stdin.
Updated unit tests to use a context manager for masking/restoring default streams and argv.
If add_host is performed with hostname existing in inventory, but
not yet accessed and put in inventory cache, additional host with
same hostname and different uuid is created, causing patterns to
misbehave.
- search entity by path
- search vm based on folder
- search for a vm based on given id ( uuid/name/dns_name/ip/inventory_path)
- search for a cluster by name, in a given datacenter (optionally)
- search for objects of a given type in a folder
Note: instance uuid is unique to a VM
The first filter is regex_search which adds the ability to do a regex
search on a fact. The filter supports returning either all capture groups
or a set of capture groups (either by index or named).
Example usage of regex_search filter
debug: msg="{{ out.stdout[0] | regex_search('image version. (?P<test>\d+).(\d+)', '\\g<test>', '\\2' ) }}"
The second filter is regex_findall with adds the capability to do a
regex findall on a fact or variable and return all occurances of a pattern.
debug: msg="{{ out.stdout[0] | regex_findall('vlan (\d+)' }}"
The filter supports two addtional keyword arguments, ignorecase=[true, false]
and multiline=[true, false]
rm _del_ as it might leak memory
renamed to tmp file cleanup
added exception handling when traversing file list, even if one fails try rest
added cleanup to finally to ensure removal in most cases
- get_real_file will decrypt vault encrypted files and return a path to
a temporary file.
- cleanup_real_file will remove a temporary file created previously with
get_real_file
Ansible when there was a percentage that was calculated to be less than
1.0 would run all hosts as the value for a rolling update.
The error is due to the fact that Python will round a
float that is under 1.0 to 0, which will trigger the case of
0 hosts. The 0 host case tells ansible to run all hosts.
The fix will see if the percentage calculation after int
conversion is 0 and will else to 1 host.
This makes our recursive, ast.parse performance measures as fast as
pre-ziploader baseline.
Since this unittest isn't testing that the returned module data is
correct we don't need to worry about os.rename not having any module
data. Should devise a separate test for the module and caching code
Since 2.0 made all includes dynamic, it is now possible to create and
include a file in the course of executing a playbook. However, with the
introduction of implicit static includes this can cause problems if an
include is thought to be static but does not yet exist. For now, we're
handling missing implicit static includes as a potential dynamic include
but also adding a deprecation message to show includes like this will
need to be marked as `static: no` in the future.
Fixes#15342
Due to an apparent race condition while using pty's on a heavily loaded
system, rarely a request to create a temp directory returns an empty
string rather than the newly created path, causing an error. Disabling
forced pty's appears to resolve the issue, so this patch modifies the
mkdtemp remote call not use -tt as we're not escalating privileges and
thus no pty is required.
Fixes#13876
* Could only have one alias before. Subsequent aliases overrode the
previous ones. Now multiple aliases work.
* Fix BLACKLISTED_MODULES. Previously, modules were listed in the
generated documentation despite being blacklisted
* Deprecated modules form extras were showing the (E) tag and not the
(D) tag. Reversed that now (Probably not necessary to also show the
E tag).
* Sort the deprecated modules alphabetically in the Category docs as
well as the list of all modules
* Optimization: Previously rendered the modules to rst twice once in all
group and once in individual categories. Fixed to only render them
once.
* Add fireball to blacklist and remove async_status (as people need to
use that).
