This completes the refactor of the iosxr 2.2 shared module. It also
includes the iosxr_config action plugin to be implemented by the
iosxr_config module for 2.2
ran task_executor through python-modernize and then made changes to the
code pointed out by it:
* Most places where we looped through dict.keys() changed to
for key in dict:
Using keys() in python2 creates a list() of keys. For iterating, we
can iterate over the dict itself and we'll be handed back each key.
In python3, doing it this way does not create a new list and thus is
more memory efficient.
* In one place, use:
for key in list(dict.keys()):
because we're deleting elements from the dictionary inside of the
loop. So we really do need to iterate over a separate list of the
keys to avoid modifying the dictionary that we're iterating over.
(Fixes Python3 bug)
* In one place, change the order of an if-elif-else tree so that the
most frequent cases are evaluated first. (Optimization)
Make !vault-encrypted create a AnsibleVaultUnicode
yaml object that can be used as a regular string object.
This allows a playbook to include a encrypted vault
blob for the value of a yaml variable. A 'secret_password'
variable can have it's value encrypted instead of having
to vault encrypt an entire vars file.
Add __ENCRYPTED__ to the vault yaml types so
template.Template can treat it similar
to __UNSAFE__ flags.
vault.VaultLib api changes:
- Split VaultLib.encrypt to encrypt and encrypt_bytestring
- VaultLib.encrypt() previously accepted the plaintext data
as either a byte string or a unicode string.
Doing the right thing based on the input type would fail
on py3 if given a arg of type 'bytes'. To simplify the
API, vaultlib.encrypt() now assumes input plaintext is a
py2 unicode or py3 str. It will encode to utf-8 then call
the new encrypt_bytestring(). The new methods are less
ambiguous.
- moved VaultLib.is_encrypted logic to vault module scope
and split to is_encrypted() and is_encrypted_file().
Add a test/unit/mock/yaml_helper.py
It has some helpers for testing parsing/yaml
Integration tests added as roles test_vault and test_vault_embedded
The 'import xmltodict' was causing import
errors when generating documentation. Since
xmltodict is a required but not stdlib module,
throw AnsibleError if unable to import.
Remove unused combine_vars.
Replace a use of 'stdin_iterator == None' with
idiomatic 'stdin_iterat is None'
Misc pep8 cleanups.
Make the plugin loading info displayed by callback plugins
match.
In debug mode (ANSIBLE_DEBUG=1 env), log all requests for
plugins including already cached plugins and class_only
requests.
Traceback (most recent call last):
File "/tmp/ansible_tpehdgt7/ansible_module_setup.py", line 134, in <module>
main()
File "/tmp/ansible_tpehdgt7/ansible_module_setup.py", line 124, in main
supports_check_mode = True,
File "/tmp/ansible_tpehdgt7/ansible_modlib.zip/ansible/module_utils/basic.py", line 696, in __init__
File "/tmp/ansible_tpehdgt7/ansible_modlib.zip/ansible/module_utils/basic.py", line 1670, in _log_invocation
File "/tmp/ansible_tpehdgt7/ansible_modlib.zip/ansible/module_utils/basic.py", line 469, in heuristic_log_sanitize
TypeError: 'str' does not support the buffer interface
This is enough to get minimal copy module working on python3
We have t omodify dataloader's path_dwim_relative_stack and everything
that calls it to use text paths instead of byte string paths
* Give native strings to selinux library functions.
SELinux takes pathnames as native strings. That means we need to
convert to bytes on python2 and convert to text on python3.
Fixes#17155
* Read kitchen documentation, make module_utils params more like kitchen API
* Remove none nonstring strategy and add strict
* Raise TypeError on invalid nonstring strategy
* Document to_native()
* Make unittests for testing module_utils.text
* Rm py2.7+ code in docker connection plugin
The docker connection plugin was using subprocess.check_output
which only exists in python 2.7 and later. Connection plugins
need to support python2.6 so this replaces it with Popen/communicate()
* Handle docker ver errors in docker connection
Add unit tests for DockerConnection
Fixes#16971
This commit updates the nxos transport shared plugins for
2.2. This includes updates to both Cli and Nxapi. This commit
also includes the nxos_config action plugin
* Cleanup basic.py code now that six is available
We had some hacks in basic.py to allow us python2 and python3
compatibility. Those can now be offloaded to the six library that we're
bundling.
* Cleanup basic.py code now that six is available
We had some hacks in basic.py to allow us python2 and python3
compatibility. Those can now be offloaded to the six library that we're
bundling.
This is part of the 2.2 refactor to extract the Cli class into a
separate module. This renames netcmd to netcli which is consistent
with the network shared modules implementations
This removes top level functions from the ios module and moves them
into the specific modules. This update also includes some clean up
of the Cli transport
This restructure moves the Cli object to netcmd and includes a roll up
of inor bugfix updates to CommandRunner
* CommandRunner now only allows one instance of a command in the stack and
raise an exception if a duplidate command is detected
* CommandRunner now caches returns based on command and output
* CommandRunner is not responsible for creating Command instances
test/units/plugins/action/test_action.py had code
for handling a bug in python 3.4's mock_open that
causes errors when reading binary data.
Moved to compat/tests/mock.py so other tests can
use it by default.
This update will now remove any keys from results that are created using
the private names. Private names are identified as double underscore (__)
on either side of the key name
* actions/unarchive: fix unarchive from remote url
Currently unarchive from remote url does not work because the core
unarchive module was updated to support 'remote_src' [1], but the
unarchive action plugin was not updated for this. This causes failures
because the action plugin assumes it needs to copy a file to the
remote server, but in the case of downloading a file from a remote
url a local file does not exist, so an error occurs when the file is
not found.
[1] https://github.com/ansible/ansible-modules-core/commit/467516e
* test_unarchive: fix test with wrong remote_src use
The non-ascii filenames test had improperly set remote_src=yes even
though it was actually copying the file from the local machine (i.e.
the file did not already exist remotely). This test was passing
until the remote_src behavior of unarchive was fixed in 276550f.
The calculation for max_fail_percentage was moved into the linear
strategy a while back, and works better there in the stategy layer
rather than at the PBE layer. This patch removes it from the PBE layer
and tweaks the logic controlling whether or not the next batch is run.
Fixes#15954
Fixes#10779
Refactor some of the block device, mount point, and
mtab/fstab facts collection for linux for better
performance on systems with lots of block devices.
Instead of invoking 'lsblk' for every entry in mtab,
invoke it once, then map the results to mtab entries.
Change the args used for invoking 'findmnt' since the
previous combination of args conflicts, so this would
always fail on some systems depending on version.
Add test cases for facts Hardware()/Network()/Virtual() classes
__new__ method and verify they create the proper subclass based
on the platform.system() results.
Split out all the 'invoke some command and grab it's output'
bits related to linux mount paths into their own methods so
it is easier to mock them in unit tests.
Fix the DragonFly* classes that did not defined a 'platform'
class attribute. This caused FreeBSD systems to potentially
get the DragonFly* subclasses incorrectly. In practice it
didnt matter much since the DragonFly* subclasses duplicated
the FreeBSD ones. Actual DragonFly systems would end up with
the generic Hardware() etc instead of the DragonFly* classes.
Fix Hardware.__new__() on PY3, passing args to __new__
would cause "object() takes no parameters" errors. So
check for PY3 and just call __new__ without the args
See
https://hg.python.org/cpython/file/44ed0cd3dc6d/Objects/typeobject.c#l2818
for some explaination.
The flag new_pb_basedir is not being utilized in Inventory._get_hostgroup_vars,
leading to the situation where an inventory with no playbook basedir set will
read host/group vars from the $CWD, regardless of the inventory and/or playbook
relative location. This patch corrects that by not using the playbook basedir
if it is unset (None).
This patch also corrects a bug in which the VariableManager would accumulate
host/group vars files, which could lead to incorrect vars files being used when
playbooks are run from different directories containing their own group/host vars
directories.
Fixes#16953
Copying the TaskInclude task (which is the parent) before loading the blocks
makes the code much more simple and clean, and fixes a bug introduced during
the performance improvement changes (and specifically the change which moved
things to a single-parent model).
Fixes#17064
Since we introduced static includes in 2.1, this broke the functionality
where a notify could be sent to a named include statement, triggering all
handlers contained within the include. This patch fixes that by adding a
search through the parents of a handler for any TaskIncludes which match.
Fixes#15915
We want to NOT consider the async task as failed if the result is
not parsed, which was the intent of:
https://github.com/ansible/ansible/pull/16458
However, the logic doesn't actually do that because we default
the 'parsed' value to True. It should default to False so that
we continue waiting, as intended.
Instead of immediately returning a failed code (indicating a break in
the play execution), we internally 'or' that failure code with the result
(now an integer flag instead of a boolean) so that we can properly handle
the rescue/always portions of blocks and still remember that the break
condition was hit.
Fixes#16937
* Introduce new 'filetree' lookup plugin
The new "filetree" lookup plugin makes it possible to recurse over a tree of files within the task loop. This makes it possible to e.g. template a complete tree of files to a target system with little effort while retaining permissions and ownership.
The module supports directories, files and symlinks.
The item dictionary consists of:
- src
- root
- path
- mode
- state
- owner
- group
- seuser
- serole
- setype
- selevel
- uid
- gid
- size
- mtime
- ctime
EXAMPLES:
Here is an example of how we use with_filetree within a role:
```yaml
- name: Create directories
file:
path: /web/{{ item.path }}
state: directory
mode: '{{ item.mode }}'
owner: '{{ item.owner }}'
group: '{{ item.group }}'
force: yes
with_filetree: web/
when: item.state == 'directory'
- name: Template complete tree
file:
src: '{{ item.src }}'
dest: /web/{{ item.path }}
state: 'link'
mode: '{{ item.mode }}'
owner: '{{ item.owner }}'
group: '{{ item.group }}'
with_filetree: web/
when: item.state == 'link'
- name: Template complete tree
template:
src: '{{ item.src }}'
dest: /web/{{ item.path }}
mode: '{{ item.mode }}'
owner: '{{ item.owner }}'
group: '{{ item.group }}'
force: yes
with_filetree: web/
when: item.state == 'file'
```
SPECIAL USE:
The following properties also have its special use:
- root: Makes it possible to filter by original location
- path: Is the relative path to root
- uid, gid: Makes it possible to force-create by exact id, rather than by name
- size, mtime, ctime: Makes it possible to filter out files by size, mtime or ctime
TODO:
- Add snippets to documentation
* Small fixes for Python 3
* Return the portion of the file’s mode that can be set by os.chmod()
And remove the exists=True, which is redundant.
* Use lstat() instead of stat() since we support symlinks
* Avoid a few possible stat() calls
* Bring in line with v1.9 and hybrid plugin
* Remove glob module since we no longer use it
* Included suggestions from @RussellLuo
- Two blank lines will be better. See PEP 8
- I think if props is not None is more conventional 😄
* Support failed pwd/grp lookups
* Implement first-found functionality in the path-order
* when including statically, make sure that all parents were also included
statically (issue #16990)
* properly resolve nested static include paths
* print a message when a file is statically included
Fixes#16990
When unittesting this we found that the platform selecting class
hierarchies weren't working in all cases. If the subclass was directly
created (ie: LinuxHardware()), then it would use its inherited __new__()
to try to create itself. The inherited __new__ would look for
subclasses and end up calling its own __new__() again. This would
recurse endlessly. The new code detects when we want to find a subclass
to create (when the base class is used, ie: Hardware()) vs when to
create the class itself (when the subclass is used, ie:
LinuxHardware()).
Rather than repeatedly searching for tasks by uuid via iterating over
all known blocks, cache the tasks when they are added to the PlayIterator
so the lookup becomes a simple key check in a dict.
It is possible that a block is copied prior to validation, in which case
some fields (like when) which should be something other than a string might
not be. Using validate() in copy() is relatively harmless and ensures the
blocks are in the proper structure.
This also cleans up some of the finalized logic from an earlier commit and
adds similar logic for validated.
Fixes#17018
