[stable-2.10] Update module debugging docs (#70847) (#70972)

- Combine remote and local debugging instructions.
- Update the example code to match current AnsiballZ structure and behavior
- Change reference name and update references
- Clarify how PYTHON path is modified
- Also add note about other remote debugging tools.

Co-authored-by: Alicia Cozine <879121+acozine@users.noreply.github.com>
(cherry picked from commit 7f0c84ea15)

Co-authored-by: Sam Doran <sdoran@redhat.com>

docs/docsite/rst/dev_guide/debugging.rst

@@ -1,140 +1,112 @@
-.. _debugging:
+.. _debugging_modules:
 
 *****************
 Debugging modules
 *****************
 
-Debugging (local)
+.. contents::
-=================
+   :local:
 
-To break into a module running on ``localhost`` and step through with the debugger:
+.. _detailed_debugging:
 
-- Set a breakpoint in the module: ``import pdb; pdb.set_trace()``
+Detailed debugging steps
-- Run the module on the local machine: ``$ python -m pdb ./my_new_test_module.py ./args.json``
+========================
 
-Example
+Ansible modules are put together as a zip file consisting of the module file and the various Python module boilerplate inside of a wrapper script. To see what is actually happening in the module, you need to extract the file from the wrapper. The wrapper script provides helper methods that let you do that.
--------
 
-`echo '{"msg": "hello"}' | python ./my_new_test_module.py`
+The following steps use ``localhost`` as the target host, but you can use the same steps to debug against remote hosts as well. For a simpler approach to debugging without using the temporary files, see :ref:`simple debugging <simple_debugging>`.
-
-Debugging (remote)
-==================
-
-To debug a module running on a remote target (i.e. not ``localhost``):
-
-#. On your controller machine (running Ansible) set ``ANSIBLE_KEEP_REMOTE_FILES=1`` to tell Ansible to retain the modules it sends to the remote machine instead of removing them after you playbook runs.
-#. Run your playbook targeting the remote machine and specify ``-vvvv`` (verbose) to display the remote location Ansible is using for the modules (among many other things).
-#. Take note of the directory Ansible used to store modules on the remote host. This directory is usually under the home directory of your ``ansible_user``, in the form ``~/.ansible/tmp/ansible-tmp-...``.
-#. SSH into the remote target after the playbook runs.
-#. Navigate to the directory you noted in step 3.
-#. Extract the module you want to debug from the zipped file that Ansible sent to the remote host: ``$ python AnsiballZ_my_test_module.py explode``. Ansible will expand the module into ``./debug_dir``. You can optionally run the zipped file by specifying ``python AnsiballZ_my_test_module.py``.
-#. Navigate to the debug directory: ``$ cd debug_dir``.
-#. Modify or set a breakpoint in ``__main__.py``.
-#. Ensure that the unzipped module is executable: ``$ chmod 755 __main__.py``.
-#. Run the unzipped module directly, passing the ``args`` file that contains the params that were originally passed: ``$ ./__main__.py args``. This approach is good for reproducing behavior as well as modifying the parameters for debugging.
 
 
-.. _debugging_ansiblemodule_based_modules:
+#. Set :envvar:`ANSIBLE_KEEP_REMOTE_FILES` to ``1`` on the control host so Ansible will keep the remote module files instead of deleting them after the module finishes executing. Use the ``-vvv`` option to make Ansible more verbose. This will display the file name of the temporary module file.
 
-Debugging AnsibleModule-based modules
+    .. code-block:: shell-session
-=====================================
 
-.. tip::
+      $ ANSIBLE_KEEP_REMOTE_FILES=1 ansible localhost -m ping -a 'data=debugging_session' -vvv
+      <127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: badger
+      <127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1461434734.35-235318071810595 `" && echo "` echo $HOME/.ansible/tmp/ansible-tmp-1461434734.35-235318071810595 `" )'
+      <127.0.0.1> PUT /var/tmp/tmpjdbJ1w TO /home/badger/.ansible/tmp/ansible-tmp-1461434734.35-235318071810595/AnsiballZ_ping.py
+      <127.0.0.1> EXEC /bin/sh -c 'LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 /usr/bin/python /home/badger/.ansible/tmp/ansible-tmp-1461434734.35-235318071810595/AnsiballZ_ping.py && sleep 0'
+      localhost | SUCCESS => {
+          "changed": false,
+          "invocation": {
+              "module_args": {
+                  "data": "debugging_session"
+              },
+              "module_name": "ping"
+          },
+          "ping": "debugging_session"
+      }
 
-    If you're using the :file:`hacking/test-module.py` script then most of this
+#. Navigate to the temporary directory from the previous step. If the previous command was run against a remote host, connect to that host first before trying to navigate to the temporary directory.
-    is taken care of for you.  If you need to do some debugging of the module
-    on the remote machine that the module will actually run on or when the
-    module is used in a playbook then you may need to use this information
-    instead of relying on :file:`test-module.py`.
 
-Starting with Ansible 2.1, AnsibleModule-based modules are put together as
+   .. code-block:: shell-session
-a zip file consisting of the module file and the various python module
-boilerplate inside of a wrapper script instead of as a single file with all of
-the code concatenated together.  Without some help, this can be harder to
-debug as the file needs to be extracted from the wrapper in order to see
-what's actually going on in the module.  Luckily the wrapper script provides
-some helper methods to do just that.
 
-If you are using Ansible with the :envvar:`ANSIBLE_KEEP_REMOTE_FILES`
+      $ ssh remotehost  # only if not debugging against localhost
-environment variables to keep the remote module file, here's a sample of how
+      $ cd /home/badger/.ansible/tmp/ansible-tmp-1461434734.35-235318071810595
-your debugging session will start:
 
-.. code-block:: shell-session
+#. Run the wrapper's ``explode`` command to turn the string into some Python files that you can work with.
 
-    $ ANSIBLE_KEEP_REMOTE_FILES=1 ansible localhost -m ping -a 'data=debugging_session' -vvv
+    .. code-block:: shell-session
-    <127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: badger
-    <127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1461434734.35-235318071810595 `" && echo "` echo $HOME/.ansible/tmp/ansible-tmp-1461434734.35-235318071810595 `" )'
-    <127.0.0.1> PUT /var/tmp/tmpjdbJ1w TO /home/badger/.ansible/tmp/ansible-tmp-1461434734.35-235318071810595/ping
-    <127.0.0.1> EXEC /bin/sh -c 'LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 /usr/bin/python /home/badger/.ansible/tmp/ansible-tmp-1461434734.35-235318071810595/ping'
-    localhost | SUCCESS => {
-        "changed": false,
-        "invocation": {
-            "module_args": {
-                "data": "debugging_session"
-            },
-            "module_name": "ping"
-        },
-        "ping": "debugging_session"
-    }
 
-Setting :envvar:`ANSIBLE_KEEP_REMOTE_FILES` to ``1`` tells Ansible to keep the
+      $ python AnsiballZ_ping.py explode
-remote module files instead of deleting them after the module finishes
+      Module expanded into:
-executing.  Giving Ansible the ``-vvv`` option makes Ansible more verbose.
+      /home/badger/.ansible/tmp/ansible-tmp-1461434734.35-235318071810595/debug_dir
-That way it prints the file name of the temporary module file for you to see.
 
-If you want to examine the wrapper file you can.  It will show a small python
+    If you want to examine the wrapper file you can. It will show a small Python script with a large base64 encoded string. The string contains the module to execute.
-script with a large, base64 encoded string.  The string contains the module
-that is going to be executed.  Run the wrapper's explode command to turn the
-string into some python files that you can work with:
 
-.. code-block:: shell-session
+#. When you look into the temporary directory you'll see a structure like this:
 
-    $ python /home/badger/.ansible/tmp/ansible-tmp-1461434734.35-235318071810595/ping explode
+    .. code-block:: shell-session
-    Module expanded into:
-    /home/badger/.ansible/tmp/ansible-tmp-1461434734.35-235318071810595/debug_dir
 
-When you look into the debug_dir you'll see a directory structure like this::
+      ├── AnsiballZ_ping.py
+      └── debug_dir
+          ├── ansible
+          │   ├── __init__.py
+          │   ├── module_utils
+          │   │   ├── __init__.py
+          │   │   ├── _text.py
+          │   │   ├── basic.py
+          │   │   ├── common
+          │   │   ├── compat
+          │   │   ├── distro
+          │   │   ├── parsing
+          │   │   ├── pycompat24.py
+          │   │   └── six
+          │   └── modules
+          │       ├── __init__.py
+          │       └── ping.py
+          └── args
 
-    ├── AnsiballZ_ping.py
+    * ``AnsiballZ_ping.py`` is the Python script with the the module code stored in a base64 encoded string. It contains various helper functions for executing the module.
-    ├── args
-    └── ansible
-        ├── __init__.py
-        └── module_utils
-            ├── basic.py
-            └── __init__.py
 
-* :file:`AnsiballZ_ping.py` is the code for the module itself.  The name
+    * ``ping.py`` is the code for the module itself. You can modify this code to see what effect it would have on your module, or for debugging purposes.
-  is based on the name of the module with a prefix so that we don't clash with
-  any other python module names.  You can modify this code to see what effect
-  it would have on your module.
 
-* The :file:`args` file contains a JSON string.  The string is a dictionary
+    * The ``args`` file contains a JSON string. The string is a dictionary containing the module arguments and other variables that Ansible passes into the module to change its behavior. Modify this file to change the parameters passed to the module.
-  containing the module arguments and other variables that Ansible passes into
-  the module to change its behaviour.  If you want to modify the parameters
-  that are passed to the module, this is the file to do it in.
 
-* The :file:`ansible` directory contains code from
+    * The ``ansible`` directory contains the module code in ``modules`` as well as code from :mod:`ansible.module_utils` that is used by the module. Ansible includes files for any :mod:`ansible.module_utils` imports in the module but not any files from any other module. If your module uses :mod:`ansible.module_utils.url` Ansible will include it for you. But if your module includes `requests <https://requests.readthedocs.io/en/master/api/>`_, then you'll have to make sure that the Python `requests library <https://pypi.org/project/requests/>`_ is installed on the system before running the module.
-  :mod:`ansible.module_utils` that is used by the module.  Ansible includes
-  files for any :mod:`ansible.module_utils` imports in the module but not
-  any files from any other module.  So if your module uses
-  :mod:`ansible.module_utils.url` Ansible will include it for you, but if
-  your module includes `requests <https://requests.readthedocs.io/en/master/api/>`_ then you'll have to make sure that
-  the python `requests library <https://pypi.org/project/requests/>`_ is installed on the system before running the
-  module.  You can modify files in this directory if you suspect that the
-  module is having a problem in some of this boilerplate code rather than in
-  the module code you have written.
 
-Once you edit the code or arguments in the exploded tree you need some way to
+    You can modify files in this directory if you suspect that the module is having a problem in some of this boilerplate code rather than in the module code you have written.
-run it.  There's a separate wrapper subcommand for this:
 
-.. code-block:: shell-session
+#. Once you edit the code or arguments in the exploded tree, use the ``execute`` subcommand to run it:
 
-    $ python /home/badger/.ansible/tmp/ansible-tmp-1461434734.35-235318071810595/ping execute
+    .. code-block:: shell-session
-    {"invocation": {"module_args": {"data": "debugging_session"}}, "changed": false, "ping": "debugging_session"}
+
+      $ python AnsiballZ_ping.py execute
+      {"invocation": {"module_args": {"data": "debugging_session"}}, "changed": false, "ping": "debugging_session"}
+
+    This subcommand inserts the absolute path to ``debug_dir`` as the first item in ``sys.path`` and invokes the script using the arguments in the ``args`` file. You can continue to run the module like this until you understand the problem. Then you can copy the changes back into your real module file and test that the real module works via ``ansible`` or ``ansible-playbook``.
+
+
+.. _simple_debugging:
+
+Simple debugging
+================
+
+The easiest way to run a debugger in a module, either local or remote, is to use `epdb <https://pypi.org/project/epdb/>`_. Add ``import epdb; epdb.serve()`` in the module code on the control node at the desired break point. To connect to the debugger, run ``epdb.connect()``. See the `epdb documentation <https://pypi.org/project/epdb/>`_ for how to specify the ``host`` and ``port``. If connecting to a remote node, make sure to use a port that is allowed by any firewall between the control node and the remote node.
+
+This technique should work with any remote debugger, but we do not guarantee any particual remote debugging tool will work.
+
+The `q <https://pypi.org/project/q/>`_ library is another very useful debugging tool.
+
+Since ``print()`` statements do not work inside modules, raising an exception is a good approach if you just want to see some specific data. Put ``raise Exception(some_value)`` somewhere in the module and run it normally. Ansible will handle this exception, pass the message back to the control node, and display it.
 
-This subcommand takes care of setting the PYTHONPATH to use the exploded
-:file:`debug_dir/ansible/module_utils` directory and invoking the script using
-the arguments in the :file:`args` file.  You can continue to run it like this
-until you understand the problem.  Then you can copy it back into your real
-module file and test that the real module works via :command:`ansible` or
-:command:`ansible-playbook`.

docs/docsite/rst/dev_guide/developing_modules_general.rst

@@ -190,7 +190,7 @@ Exercising your module code
 ===========================
 
 Once you've modified the sample code above to do what you want, you can try out your module.
-Our :ref:`debugging tips <debugging>` will help if you run into bugs as you exercise your module code.
+Our :ref:`debugging tips <debugging_modules>` will help if you run into bugs as you exercise your module code.
 
 Exercising module code locally
 ------------------------------

docs/docsite/rst/dev_guide/index.rst

@@ -36,7 +36,7 @@ Find the task that best describes what you want to do:
 
 * I want to refine my code:
 
-   * I want to :ref:`debug my module code <debugging>`.
+   * I want to :ref:`debug my module code <debugging_modules>`.
    * I want to :ref:`add tests <developing_testing>`.
    * I want to :ref:`document my module <module_documenting>`.
    * I want to :ref:`document my set of modules for a network platform <documenting_modules_network>`.

docs/docsite/rst/dev_guide/testing_running_locally.rst

@@ -49,7 +49,7 @@ When using environment variables to manipulate tests there some limitations to k
     Example: ``ANSIBLE_KEEP_REMOTE_FILES=1`` can be set when running ``ansible-test integration --venv``. However, using the ``--docker`` option would
     require running ``ansible-test shell`` to gain access to the Docker environment. Once at the shell prompt, the environment variable could be set
     and the tests executed. This is useful for debugging tests inside a container by following the
-    :ref:`Debugging AnsibleModule-based modules <debugging_ansiblemodule_based_modules>` instructions.
+    :ref:`Debugging AnsibleModule-based modules <debugging_modules>` instructions.
 
 Interactive Shell
 =================