start of 'data manipulation' examples (#46979)

Co-authored-by: Klaus Frank <agowa338@users.noreply.github.com>
Co-authored-by: Felix Fontein <felix@fontein.de>
Co-authored-by: Abhijeet Kasurde <akasurde@redhat.com>
This commit is contained in:
Brian Coca 2020-08-10 00:48:41 -04:00 committed by GitHub
parent 4f54fb8d43
commit f46b124d65
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
3 changed files with 246 additions and 0 deletions

View file

@ -0,0 +1,243 @@
.. _complex_data_manipulation:
Data manipulation
#########################
In many cases, you need to do some complex operation with your variables, while Ansible is not recommended as a data processing/manipulation tool, you can use the existing Jinja2 templating in conjunction with the many added Ansible filters, lookups and tests to do some very complex transformations.
Let's start with a quick definition of each type of plugin:
- lookups: Mainly used to query 'external data', in Ansible these were the primary part of loops using the ``with_<lookup>`` construct, but they can be used independently to return data for processing. They normally return a list due to their primary function in loops as mentioned previously. Used with the ``lookup`` or ``query`` Jinja2 operators.
- filters: used to change/transform data, used with the ``|`` Jinja2 operator.
- tests: used to validate data, used with the ``is`` Jinja2 operator.
.. _note:
* Some tests and filters are provided directly by Jinja2, so their availability depends on the Jinja2 version, not Ansible.
.. _for_loops_or_list_comprehensions:
Loops and list comprehensions
=============================
Most programming languages have loops (``for``, ``while``, etc.) and list comprehensions to do transformations on lists including lists of objects. Jinja2 has a few filters that provide this functionality: ``map``, ``select``, ``reject``, ``selectattr``, ``rejectattr``.
- map: this is a basic for loop that just allows you to change every item in a list, using the 'attribute' keyword you can do the transformation based on attributes of the list elements.
- select/reject: this is a for loop with a condition, that allows you to create a subset of a list that matches (or not) based on the result of the condition.
- selectattr/rejectattr: very similar to the above but it uses a specific attribute of the list elements for the conditional statement.
.. _keys_from_dict_matching_list:
Extract keys from a dictionary matching elements from a list
------------------------------------------------------------
The Python equivalent code would be:
.. code-block:: python
chains = [1, 2]
for chain in chains:
for config in chains_config[chain]['configs']:
print(config['type'])
There are several ways to do it in Ansible, this is just one example:
.. code-block:: YAML+Jinja
:emphasize-lines: 3
:caption: Way to extract matching keys from a list of dictionaries
tasks:
- name: Show extracted list of keys from a list of dictionaries
debug: msg="{{ chains | map('extract', chains_config) | map(attribute='configs') | flatten | map(attribute='type') | flatten }}"
vars:
chains: [1, 2]
chains_config:
1:
foo: bar
configs:
- type: routed
version: 0.1
- type: bridged
version: 0.2
2:
foo: baz
configs:
- type: routed
version: 1.0
- type: bridged
version: 1.1
.. code-block:: ansible-output
:caption: Results of debug task, a list with the extracted keys
ok: [localhost] => {
"msg": [
"routed",
"bridged",
"routed",
"bridged"
]
}
.. _find_mount_point:
Find mount point
----------------
In this case, we want to find the mount point for a given path across our machines, since we already collect mount facts, we can use the following:
.. code-block:: YAML+Jinja
:caption: Use selectattr to filter mounts into list I can then sort and select the last from
:emphasize-lines: 7
- hosts: all
gather_facts: True
vars:
path: /var/lib/cache
tasks:
- name: The mount point for {{path}}, found using the Ansible mount facts, [-1] is the same as the 'last' filter
debug: msg="{{(ansible_facts.mounts | selectattr('mount', 'in', path) | list | sort(attribute='mount'))[-1]['mount']}}"
Omit elements from a list
-------------------------
The special ``omit`` variable ONLY works with module options, but we can still use it in other ways as an identifier to tailor a list of elements:
.. code-block:: YAML+Jinja
:caption: Inline list filtering when feeding a module option
:emphasize-lines: 3, 7
- name: enable a list of Windows features, by name
set_fact:
win_feature_list: "{{ namestuff | reject('equalto', omit) | list }}"
vars:
namestuff:
- "{{ (fs_installed_smb_v1 | default(False)) | ternary(omit, 'FS-SMB1') }}"
- "foo"
- "bar"
Another way is to avoid adding elements to the list in the first place, so you can just use it directly:
.. code-block:: YAML+Jinja
:caption: Using set_fact in a loop to increment a list conditionally
:emphasize-lines: 3, 4, 6
- name: build unique list with some items conditionally omittted
set_fact:
namestuff: ' {{ (namestuff | default([])) | union([item]) }}'
when: item != omit
loop:
- "{{ (fs_installed_smb_v1 | default(False)) | ternary(omit, 'FS-SMB1') }}"
- "foo"
- "bar"
.. _complex_type_transfomations:
Complex Type transformations
=============================
Jinja provides filters for simple data type transformations (``int``, ``bool``, etc), but when you want to transform data structures things are not as easy.
You can use loops and list comprehensions as shown above to help, also other filters and lookups can be chained and leveraged to achieve more complex transformations.
.. _create_dictionary_from_list:
Create dictionary from list
---------------------------
In most languages it is easy to create a dictionary (a.k.a. map/associative array/hash etc.) from a list of pairs, in Ansible there are a couple of ways to do it and the best one for you might depend on the source of your data.
These example produces ``{"a": "b", "c": "d"}``
.. code-block:: YAML+Jinja
:caption: Simple list to dict by assuming the list is [key, value , key, value, ...]
vars:
single_list: [ 'a', 'b', 'c', 'd' ]
mydict: "{{ dict(single_list) | slice(2) | list }}"
.. code-block:: YAML+Jinja
:caption: It is simpler when we have a list of pairs:
vars:
list_of_pairs: [ ['a', 'b'], ['c', 'd'] ]
mydict: "{{ dict(list_of_pairs) }}"
Both end up being the same thing, with the ``slice(2) | list`` transforming ``single_list`` to the same structure as ``list_of_pairs``.
A bit more complex, using ``set_fact`` and a ``loop`` to create/update a dictionary with key value pairs from 2 lists:
.. code-block:: YAML+Jinja
:caption: Using set_fact to create a dictionary from a set of lists
:emphasize-lines: 3, 4
- name: Uses 'combine' to update the dictionary and 'zip' to make pairs of both lists
set_fact:
mydict: "{{ mydict | default({}) | combine({item[0]: item[1]}) }}"
loop: "{{ (keys | zip(values)) | list }}"
vars:
keys:
- foo
- var
- bar
values:
- a
- b
- c
This results in ``{"foo": "a", "var": "b", "bar": "c"}``.
You can even combine these simple examples with other filters and lookups to create a dictionary dynamically by matching patterns to variable names:
.. code-block:: YAML+Jinja
:caption: Using 'vars' to define dictionary from a set of lists without needing a task
vars:
myvarnames: "{{ q('varnames', '^my') }}"
mydict: "{{ dict(myvarnames | zip(q('vars', *myvarnames))) }}"
A quick explanation, since there is a lot to unpack from these two lines:
- The ``varnames`` lookup returns a list of variables that match "begin with ``my``".
- Then feeding the list from the previous step into the ``vars`` lookup to get the list of values.
The ``*`` is used to 'dereference the list' (a pythonism that works in Jinja), otherwise it would take the list as a single argument.
- Both lists get passed to the ``zip`` filter to pair them off into a unified list (key, value, key2, value2, ...).
- The dict function then takes this 'list of pairs' to create the dictionary.
An example on how to use facts to find a host's data that meets condition X:
.. code-block:: YAML+Jinja
vars:
uptime_of_host_most_recently_rebooted: "{{ansible_play_hosts_all | map('extract', hostvars, 'ansible_uptime_seconds') | sort | first}}"
Using an example from @zoradache on reddit, to show the 'uptime in days/hours/minutes' (assumes facts where gathered).
https://www.reddit.com/r/ansible/comments/gj5a93/trying_to_get_uptime_from_seconds/fqj2qr3/
.. code-block:: YAML+Jinja
- debug:
msg: Timedelta {{ now() - now().fromtimestamp(now(fmt='%s') | int - ansible_uptime_seconds) }}
.. seealso::
:doc:`playbooks_filters`
Jinja2 filters included with Ansible
:doc:`playbooks_tests`
Jinja2 tests included with Ansible
`Jinja2 Docs <http://jinja.pocoo.org/docs/>`_
Jinja2 documentation, includes lists for core filters and tests

View file

@ -104,6 +104,8 @@ You've anchored the value of ``version`` with the ``&my_version`` anchor, and re
:ref:`playbooks_variables`
All about variables
:doc:`complex_data_manipulation`
Doing complex data manipulation in Ansible
`User Mailing List <https://groups.google.com/group/ansible-project>`_
Have a question? Stop by the google group!
`irc.freenode.net <http://irc.freenode.net>`_

View file

@ -16,6 +16,7 @@ As you write more playbooks and roles, you might have some special use cases. Fo
playbooks_environment
playbooks_error_handling
playbooks_advanced_syntax
complex_data_manipulation
../plugins/plugins
playbooks_prompts
playbooks_tags