From a361140680a980eeb63eed170cc57fff34c8d8b1 Mon Sep 17 00:00:00 2001 From: Alicia Cozine <879121+acozine@users.noreply.github.com> Date: Tue, 26 Feb 2019 09:40:13 -0600 Subject: [PATCH] Update the Loop docs (#47895) * first review of loops page * round two * round three * puts 'not deprecating' message in a note * fixes note syntax * clarifies loop_control sections * loops require list input * incorporates bcoca and sivel feedback * fixes links * more bcoca feedback * adds examples to loop update guidelines * adds explicit code-blocks * remove stray copypasta --- .../rst/user_guide/playbooks_loops.rst | 369 ++++++++++-------- 1 file changed, 208 insertions(+), 161 deletions(-) diff --git a/docs/docsite/rst/user_guide/playbooks_loops.rst b/docs/docsite/rst/user_guide/playbooks_loops.rst index d924d50695e..ea91d611dc5 100644 --- a/docs/docsite/rst/user_guide/playbooks_loops.rst +++ b/docs/docsite/rst/user_guide/playbooks_loops.rst @@ -1,21 +1,59 @@ .. _playbooks_loops: +***** Loops -===== +***** -Often you'll want to do many things in one task, such as create a lot of users, install a lot of packages, or -repeat a polling step until a certain result is reached. +Sometimes you want to repeat a task multiple times. In computer programming, this is called a loop. Common Ansible loops include changing ownership on several files and/or directories with the :ref:`file module `, creating multiple users with the :ref:`user module `, and +repeating a polling step until a certain result is reached. Ansible offers two keywords for creating loops: ``loop`` and ``with_``. -This chapter is all about how to use loops in playbooks. +.. note:: + * We added ``loop`` in Ansible 2.5. It is not yet a full replacement for ``with_``, but we recommend it for most use cases. + * We have not deprecated the use of ``with_`` - that syntax will still be valid for the foreseeable future. + * We are looking to improve ``loop`` syntax - watch this page and the `changelog `_ for updates. -.. contents:: Topics +.. contents:: + :local: + +Comparing ``loop`` and ``with_*`` +================================= + +* The ``with_`` keywords rely on :ref:`lookup_plugins` - even ``items`` is a lookup. +* The ``loop`` keyword is equivalent to ``with_list``, and is the best choice for simple loops. +* The ``loop`` keyword will not accept a string as input, see :ref:`query_vs_lookup`. +* Generally speaking, any use of ``with_*`` covered in :ref:`migrating_to_loop` can be updated to use ``loop``. +* Be careful when changing ``with_items`` to ``loop``, as ``with_items`` performed implicit single-level flattening. You may need to use ``flatten(1)`` with ``loop`` to match the exact outcome. For example, to get the same output as: + +.. code-block:: yaml + + with_items: + - 1 + - [2,3] + - 4 + +you would need:: + + loop: [1, [2,3] ,4] | flatten(1) + +* Any ``with_*`` statement that requires using ``lookup`` within a loop should not be converted to use the ``loop`` keyword. For example, instead of doing: + +.. code-block:: yaml + + loop: "{{ lookup('fileglob', '*.txt', wantlist=True) }}" + +it's cleaner to keep:: + + with_fileglob: '*.txt' .. _standard_loops: -Standard Loops -`````````````` +Standard loops +============== -To save some typing, repeated tasks can be written in short-hand like so:: +Iterating over a simple list +---------------------------- + +Repeated tasks can be written as standard loops over a simple list of strings. You can define the list directly in the task:: - name: add several users user: @@ -26,42 +64,43 @@ To save some typing, repeated tasks can be written in short-hand like so:: - testuser1 - testuser2 -If you have defined a YAML list in a variables file, or the 'vars' section, you can also do:: +You can define the list in a variables file, or in the 'vars' section of your play, then refer to the name of the list in the task:: loop: "{{ somelist }}" -The above would be the equivalent of:: +Either of these examples would be the equivalent of:: - name: add user testuser1 user: name: "testuser1" state: present groups: "wheel" + - name: add user testuser2 user: name: "testuser2" state: present groups: "wheel" -.. note:: Before 2.5 Ansible mainly used the ``with_`` keywords to create loops, the `loop` keyword is basically analogous to ``with_list``. - - -Some plugins like, the yum and apt modules can take lists directly to their options, this is more optimal than looping over the task. -See each action's documentation for details, for now here is an example:: +You can pass a list directly to a parameter for some plugins. Most of the packaging modules, like :ref:`yum_module` and :ref:`apt_module`, have this capability. When available, passing the list to a parameter is better than looping over the task. For example:: - name: optimal yum yum: - name: "{{list_of_packages}}" + name: "{{ list_of_packages }}" state: present - - name: non optimal yum, not only slower but might cause issues with interdependencies + - name: non-optimal yum, slower and may cause issues with interdependencies yum: - name: "{{item}}" + name: "{{ item }}" state: present - loop: "{{list_of_packages}}" + loop: "{{ list_of_packages }}" -Note that the types of items you iterate over do not have to be simple lists of strings. -If you have a list of hashes, you can reference subkeys using things like:: +Check the :ref:`module documentation ` to see if you can pass a list to any particular module's parameter(s). + +Iterating over a list of hashes +------------------------------- + +If you have a list of hashes, you can reference subkeys in a loop. For example:: - name: add several users user: @@ -72,8 +111,11 @@ If you have a list of hashes, you can reference subkeys using things like:: - { name: 'testuser1', groups: 'wheel' } - { name: 'testuser2', groups: 'root' } -Also be aware that when combining :doc:`playbooks_conditionals` with a loop, the ``when:`` statement is processed separately for each item. -See :ref:`the_when_statement` for an example. +When combining :ref:`playbooks_conditionals` with a loop, the ``when:`` statement is processed separately for each item. +See :ref:`the_when_statement` for examples. + +Iterating over a dictionary +--------------------------- To loop over a dict, use the ``dict2items`` :ref:`dict_filter`:: @@ -90,79 +132,18 @@ To loop over a dict, use the ``dict2items`` :ref:`dict_filter`:: Here, we don't want to set empty tags, so we create a dictionary containing only non-empty tags. +Registering variables with a loop +================================= -.. _complex_loops: +You can register the output of a loop as a variable. For example:: -Complex loops -````````````` + - shell: "echo {{ item }}" + loop: + - "one" + - "two" + register: echo -Sometimes you need more than what a simple list provides, you can use Jinja2 expressions to create complex lists: -For example, using the 'nested' lookup, you can combine lists:: - - - name: give users access to multiple databases - mysql_user: - name: "{{ item[0] }}" - priv: "{{ item[1] }}.*:ALL" - append_privs: yes - password: "foo" - loop: "{{ ['alice', 'bob'] |product(['clientdb', 'employeedb', 'providerdb'])|list }}" - -.. note:: ``with_`` loops are actually a combination of things ``with_`` + ``lookup()``, even ``items`` is a lookup. ``loop`` can be used in the same way as shown above. - - -Using lookup vs query with loop -``````````````````````````````` - -In Ansible 2.5 a new jinja2 function was introduced named :ref:`query`, that offers several benefits over ``lookup`` when using the new ``loop`` keyword. - -This is better described in the lookup documentation. However, ``query`` provides a simpler interface and a more predictable output from lookup plugins, ensuring better compatibility with ``loop``. - -In certain situations the ``lookup`` function may not return a list which ``loop`` requires. - -The following invocations are equivalent, using ``wantlist=True`` with ``lookup`` to ensure a return type of a list:: - - loop: "{{ query('inventory_hostnames', 'all') }}" - - loop: "{{ lookup('inventory_hostnames', 'all', wantlist=True) }}" - - -.. _do_until_loops: - -Do-Until Loops -`````````````` - -.. versionadded:: 1.4 - -Sometimes you would want to retry a task until a certain condition is met. Here's an example:: - - - shell: /usr/bin/foo - register: result - until: result.stdout.find("all systems go") != -1 - retries: 5 - delay: 10 - -The above example runs the shell module iteratively until the module's result has "all systems go" in its stdout or the task has -been retried for 5 times with a delay of 10 seconds. The default value for "retries" is 3 and "delay" is 5. - -The task returns the results returned by the last task run. The results of individual retries can be viewed by -vv option. -The registered variable will also have a new key "attempts" which will have the number of the retries for the task. - -.. note:: If the ``until`` parameter isn't defined, the value for the ``retries`` parameter is forced to 1. - -Using register with a loop -`````````````````````````` - -After using ``register`` with a loop, the data structure placed in the variable will contain a ``results`` attribute that is a list of all responses from the module. - -Here is an example of using ``register`` with ``loop``:: - - - shell: "echo {{ item }}" - loop: - - "one" - - "two" - register: echo - -This differs from the data structure returned when using ``register`` without a loop:: +When you use ``register`` with a loop, the data structure placed in the variable will contain a ``results`` attribute that is a list of all responses from the module. This differs from the data structure returned when using ``register`` without a loop:: { "changed": true, @@ -218,13 +199,52 @@ During iteration, the result of the current item will be placed in the variable: register: echo changed_when: echo.stdout != "one" +.. _complex_loops: + +Complex loops +============= + +Iterating over nested lists +--------------------------- + +You can use Jinja2 expressions to iterate over complex lists. For example, a loop can combine nested lists:: + + - name: give users access to multiple databases + mysql_user: + name: "{{ item[0] }}" + priv: "{{ item[1] }}.*:ALL" + append_privs: yes + password: "foo" + loop: "{{ ['alice', 'bob'] |product(['clientdb', 'employeedb', 'providerdb'])|list }}" -Looping over the inventory -`````````````````````````` +.. _do_until_loops: -If you wish to loop over the inventory, or just a subset of it, there are multiple ways. -One can use a regular ``loop`` with the ``ansible_play_batch`` or ``groups`` variables, like this:: +Retrying a task until a condition is met +---------------------------------------- + +.. versionadded:: 1.4 + +You can use the ``until`` keyword to retry a task until a certain condition is met. Here's an example:: + + - shell: /usr/bin/foo + register: result + until: result.stdout.find("all systems go") != -1 + retries: 5 + delay: 10 + +This task runs up to 5 times with a delay of 10 seconds between each attempt. If the result of any attempt has "all systems go" in its stdout, the task succeeds. The default value for "retries" is 3 and "delay" is 5. + +To see the results of individual retries, run the play with ``-vv``. + +When you run a task with ``until`` and register the result as a variable, the registered variable will include a key called "attempts", which records the number of the retries for the task. + +.. note:: You must set the ``until`` parameter if you want a task to retry. If ``until`` is not defined, the value for the ``retries`` parameter is forced to 1. + +Looping over inventory +---------------------- + +To loop over your inventory, or just a subset of it, you can use a regular ``loop`` with the ``ansible_play_batch`` or ``groups`` variables:: # show all the hosts in the inventory - debug: @@ -248,18 +268,94 @@ There is also a specific lookup plugin ``inventory_hostnames`` that can be used msg: "{{ item }}" loop: "{{ query('inventory_hostnames', 'all:!www') }}" -More information on the patterns can be found on :doc:`intro_patterns` +More information on the patterns can be found on :ref:`intro_patterns` + +.. _query_vs_lookup: + +Ensuring list input for ``loop``: ``query`` vs. ``lookup`` +========================================================== + +The ``loop`` keyword requires a list as input, but the ``lookup`` keyword returns a string of comma-separated values by default. Ansible 2.5 introduced a new Jinja2 function named :ref:`query` that always returns a list, offering a simpler interface and more predictable output from lookup plugins when using the ``loop`` keyword. + +You can force ``lookup`` to return a list to ``loop`` by using ``wantlist=True``, or you can use ``query`` instead. + +These examples do the same thing:: + + loop: "{{ query('inventory_hostnames', 'all') }}" + + loop: "{{ lookup('inventory_hostnames', 'all', wantlist=True) }}" + .. _loop_control: -Loop Control -```````````` - +Adding controls to loops +======================== .. versionadded:: 2.1 -In 2.0 you are again able to use loops and task includes (but not playbook includes). This adds the ability to loop over the set of tasks in one shot. -Ansible by default sets the loop variable ``item`` for each loop, which causes these nested loops to overwrite the value of ``item`` from the "outer" loops. -As of Ansible 2.1, the ``loop_control`` option can be used to specify the name of the variable to be used for the loop:: +The ``loop_control`` keyword lets you manage your loops in useful ways. + +Limiting loop output with ``label`` +----------------------------------- +.. versionadded:: 2.2 + +When looping over complex data structures, the console output of your task can be enormous. To limit the displayed output, use the ``label`` directive with ``loop_control``:: + + - name: create servers + digital_ocean: + name: "{{ item.name }}" + state: present + loop: + - name: server1 + disks: 3gb + ram: 15Gb + network: + nic01: 100Gb + nic02: 10Gb + ... + loop_control: + label: "{{ item.name }}" + +The output of this task will display just the ``name`` field for each ``item`` instead of the entire contents of the multi-line ``{{ item }}`` variable. + +Pausing within a loop +--------------------- +.. versionadded:: 2.2 + +To control the time (in seconds) between the execution of each item in a task loop, use the ``pause`` directive with ``loop_control``:: + + # main.yml + - name: create servers, pause 3s before creating next + digital_ocean: + name: "{{ item }}" + state: present + loop: + - server1 + - server2 + loop_control: + pause: 3 + +Tracking progress through a loop with ``index_var`` +--------------------------------------------------- +.. versionadded:: 2.5 + +To keep track of where you are in a loop, use the ``index_var`` directive with ``loop_control``. This directive specifies a variable name to contain the current loop index:: + + - name: count our fruit + debug: + msg: "{{ item }} with index {{ my_idx }}" + loop: + - apple + - banana + - pear + loop_control: + index_var: my_idx + +Defining inner and outer variable names with ``loop_var`` +--------------------------------------------------------- +.. versionadded:: 2.1 + +You can nest two looping tasks using ``include_tasks``. However, by default Ansible sets the loop variable ``item`` for each loop. This means the inner, nested loop will overwrite the value of ``item`` from the outer loop. +You can specify the name of the variable for each loop using ``loop_var`` with ``loop_control``:: # main.yml - include_tasks: inner.yml @@ -280,56 +376,6 @@ As of Ansible 2.1, the ``loop_control`` option can be used to specify the name o .. note:: If Ansible detects that the current loop is using a variable which has already been defined, it will raise an error to fail the task. -.. versionadded:: 2.2 - -When using complex data structures for looping the display might get a bit too "busy", this is where the ``label`` directive comes to help:: - - - name: create servers - digital_ocean: - name: "{{ item.name }}" - state: present - loop: - - name: server1 - disks: 3gb - ram: 15Gb - network: - nic01: 100Gb - nic02: 10Gb - ... - loop_control: - label: "{{ item.name }}" - -This will now display just the ``label`` field instead of the whole structure per ``item``, it defaults to ``{{ item }}`` to display things as usual. - -.. versionadded:: 2.2 - -Another option to loop control is ``pause``, which allows you to control the time (in seconds) between execution of items in a task loop.:: - - # main.yml - - name: create servers, pause 3s before creating next - digital_ocean: - name: "{{ item }}" - state: present - loop: - - server1 - - server2 - loop_control: - pause: 3 - -.. versionadded:: 2.5 - -If you need to keep track of where you are in a loop, you can use the ``index_var`` option to loop control to specify a variable name to contain the current loop index.:: - - - name: count our fruit - debug: - msg: "{{ item }} with index {{ my_idx }}" - loop: - - apple - - banana - - pear - loop_control: - index_var: my_idx - .. versionadded:: 2.8 As of Ansible 2.8 you can get extended loop information using the ``extended`` option to loop control. This option will expose the following information. @@ -354,23 +400,24 @@ Variable Description loop_control: extended: yes +.. _migrating_to_loop: + Migrating from with_X to loop -````````````````````````````` +============================= .. include:: shared_snippets/with2loop.txt - .. seealso:: - :doc:`playbooks` + :ref:`about_playbooks` An introduction to playbooks - :doc:`playbooks_reuse_roles` + :ref:`playbooks_reuse_roles` Playbook organization by roles - :doc:`playbooks_best_practices` + :ref:`playbooks_best_practices` Best practices in playbooks - :doc:`playbooks_conditionals` + :ref:`playbooks_conditionals` Conditional statements in playbooks - :doc:`playbooks_variables` + :ref:`playbooks_variables` All about variables `User Mailing List `_ Have a question? Stop by the google group!