Add support to configure network_cli terminal related options (#60086)

* Add support for configurable terminal plugin options

Fixes #59404

*  Add terminal options to support platform specific login menu
*  Add terminal options to support configurable options for stdout
   and stderr regex list

* Fix CI failures

* Fix CI issues

* Fix review comments and add integration test

* Fix sanity test failures

* Fix review comments

* Fix integration test case

* Fix integration test failure

* Add support to configure terminal related options

Fixes https://github.com/ansible/ansible/issues/59404

*  Add network_cli configurable options to support platform specific login menu
*  Add network_cli configurable options to support configurable options for stdout
   and stderr regex list

* Fix review comment

* Fix review comment

docs/docsite/rst/network/user_guide/network_debug_troubleshooting.rst

@@ -561,6 +561,46 @@ To make this a permanent change, add the following to your ``ansible.cfg`` file:
    connect_retry_timeout = 30
 
 
+Timeout issue due to platform specific login menu with ``network_cli`` connection type
+--------------------------------------------------------------------------------------
+
+In Ansible 2.9 and later, the network_cli connection plugin configuration options are added
+to handle the platform specific login menu. These options can be set as group/host or tasks
+variables.
+
+Example: Handle single login menu prompts with host variables
+
+.. code-block:: console
+
+    $cat host_vars/<hostname>.yaml
+    ---
+    ansible_terminal_initial_prompt:
+      - "Connect to a host"
+    ansible_terminal_initial_answer:
+      - "3"
+
+Example: Handle remote host multiple login menu prompts with host variables
+
+.. code-block:: console
+
+    $cat host_vars/<inventory-hostname>.yaml
+    ---
+    ansible_terminal_initial_prompt:
+      - "Press any key to enter main menu"
+      - "Connect to a host"
+    ansible_terminal_initial_answer:
+      - "\\r"
+      - "3"
+    ansible_terminal_initial_prompt_checkall: True
+
+To handle multiple login menu prompts:
+
+* The values of ``ansible_terminal_initial_prompt`` and ``ansible_terminal_initial_answer`` should be a list.
+* The prompt sequence should match the answer sequence.
+* The value of ``ansible_terminal_initial_prompt_checkall`` should be set to ``True``.
+
+.. note:: If all the prompts in sequence are not received from remote host at the time connection initialization it will result in a timeout.
+
 
 Playbook issues
 ===============
@@ -757,3 +797,52 @@ To make this a global setting, add the following to your ``ansible.cfg`` file:
    buffer_read_timeout = 2
 
 This timer delay per command executed on remote host can be disabled by setting the value to zero.
+
+
+Task failure due to mismatched error regex within command response using ``network_cli`` connection type
+--------------------------------------------------------------------------------------------------------
+
+In Ansible 2.9 and later, the network_cli connection plugin configuration options are added
+to handle the stdout and stderr regex to identify if the command execution response consist
+of a normal response or an error response. These options can be set group/host variables or as
+tasks variables.
+
+Example: For mismatched error response
+
+.. code-block:: yaml
+
+  - name: fetch logs from remote host
+    ios_command:
+      commands:
+        - show logging
+
+
+Playbook run output:
+
+.. code-block:: console
+
+  TASK [first fetch logs] ********************************************************
+  fatal: [ios01]: FAILED! => {
+      "changed": false,
+      "msg": "RF Name:\r\n\r\n <--nsip-->
+             \"IPSEC-3-REPLAY_ERROR: Test log\"\r\n*Aug  1 08:36:18.483: %SYS-7-USERLOG_DEBUG:
+              Message from tty578(user id: ansible): test\r\nan-ios-02#"}
+
+Suggestions to resolve:
+
+Modify the error regex for individual task.
+
+.. code-block:: yaml
+
+  - name: fetch logs from remote host
+    ios_command:
+      commands:
+        - show logging
+    vars:
+      ansible_terminal_stderr_re:
+        - pattern: 'connection timed out'
+          flags: 're.I'
+
+The terminal plugin regex options ``ansible_terminal_stderr_re`` and ``ansible_terminal_stdout_re`` have
+``pattern`` and ``flags`` as keys. The value of the ``flags`` key should be a value that is accepted by
+the ``re.compile`` python method.

lib/ansible/plugins/connection/network_cli.py

@@ -191,6 +191,65 @@ options:
       - name: ANSIBLE_PERSISTENT_LOG_MESSAGES
     vars:
       - name: ansible_persistent_log_messages
+  terminal_stdout_re:
+    type: list
+    elements: dict
+    description:
+      - A single regex pattern or a sequence of patterns along with optional flags
+        to match the command prompt from the received response chunk. This option
+        accepts C(pattern) and C(flags) keys. The value of C(pattern) is a python
+        regex pattern to match the response and the value of C(flags) is the value
+        accepted by I(flags) argument of I(re.compile) python method to control
+        the way regex is matched with the response, for example I('re.I').
+    vars:
+      - name: ansible_terminal_stdout_re
+  terminal_stderr_re:
+    type: list
+    elements: dict
+    description:
+      - This option provides the regex pattern and optional flags to match the
+        error string from the received response chunk. This option
+        accepts C(pattern) and C(flags) keys. The value of C(pattern) is a python
+        regex pattern to match the response and the value of C(flags) is the value
+        accepted by I(flags) argument of I(re.compile) python method to control
+        the way regex is matched with the response, for example I('re.I').
+    vars:
+      - name: ansible_terminal_stderr_re
+  terminal_initial_prompt:
+    type: list
+    description:
+      - A single regex pattern or a sequence of patterns to evaluate the expected
+        prompt at the time of initial login to the remote host.
+    vars:
+      - name: ansible_terminal_initial_prompt
+  terminal_initial_answer:
+    type: list
+    description:
+      - The answer to reply with if the C(terminal_initial_prompt) is matched. The value can be a single answer
+        or a list of answers for multiple terminal_initial_prompt. In case the login menu has
+        multiple prompts the sequence of the prompt and excepted answer should be in same order and the value
+        of I(terminal_prompt_checkall) should be set to I(True) if all the values in C(terminal_initial_prompt) are
+        expected to be matched and set to I(False) if any one login prompt is to be matched.
+    vars:
+      - name: ansible_terminal_initial_answer
+  terminal_initial_prompt_checkall:
+    type: boolean
+    description:
+      - By default the value is set to I(False) and any one of the prompts mentioned in C(terminal_initial_prompt)
+        option is matched it won't check for other prompts. When set to I(True) it will check for all the prompts
+        mentioned in C(terminal_initial_prompt) option in the given order and all the prompts
+        should be received from remote host if not it will result in timeout.
+    default: False
+    vars:
+      - name: ansible_terminal_initial_prompt_checkall
+  terminal_inital_prompt_newline:
+    type: boolean
+    description:
+      - This boolean flag, that when set to I(True) will send newline in the response if any of values
+        in I(terminal_initial_prompt) is matched.
+    default: True
+    vars:
+      - name: ansible_terminal_initial_prompt_newline
 """
 
 import getpass
@@ -338,11 +397,12 @@ class Connection(NetworkConnectionBase):
 
             self.queue_message('vvvv', 'loaded terminal plugin for network_os %s' % self._network_os)
 
-            self.receive(
+            terminal_initial_prompt = self.get_option('terminal_initial_prompt') or self._terminal.terminal_initial_prompt
-                prompts=self._terminal.terminal_initial_prompt,
+            terminal_initial_answer = self.get_option('terminal_initial_answer') or self._terminal.terminal_initial_answer
-                answer=self._terminal.terminal_initial_answer,
+            newline = self.get_option('terminal_inital_prompt_newline') or self._terminal.terminal_inital_prompt_newline
-                newline=self._terminal.terminal_inital_prompt_newline
+            check_all = self.get_option('terminal_initial_prompt_checkall') or False
-            )
+
+            self.receive(prompts=terminal_initial_prompt, answer=terminal_initial_answer, newline=newline, check_all=check_all)
 
             self.queue_message('vvvv', 'firing event: on_open_shell()')
             self._terminal.on_open_shell()
@@ -386,6 +446,10 @@ class Connection(NetworkConnectionBase):
         command_prompt_matched = False
         matched_prompt_window = window_count = 0
 
+        # set terminal regex values for command prompt and errors in response
+        self._terminal_stderr_re = self._get_terminal_std_re('terminal_stderr_re')
+        self._terminal_stdout_re = self._get_terminal_std_re('terminal_stdout_re')
+
         cache_socket_timeout = self._ssh_shell.gettimeout()
         command_timeout = self.get_option('persistent_command_timeout')
         self._validate_timeout_value(command_timeout, "persistent_command_timeout")
@@ -463,8 +527,10 @@ class Connection(NetworkConnectionBase):
             if prompt_len != answer_len:
                 raise AnsibleConnectionFailure("Number of prompts (%s) is not same as that of answers (%s)" % (prompt_len, answer_len))
         try:
-            self._history.append(command)
+            cmd = b'%s\r' % command
-            self._ssh_shell.sendall(b'%s\r' % command)
+            self._history.append(cmd)
+            self._ssh_shell.sendall(cmd)
+            self._log_messages('send command: %s' % cmd)
             if sendonly:
                 return
             response = self.receive(command, prompt, answer, newline, prompt_retry_check, check_all)
@@ -513,7 +579,7 @@ class Connection(NetworkConnectionBase):
             single_prompt = True
         if not isinstance(answer, list):
             answer = [answer]
-        prompts_regex = [re.compile(r, re.I) for r in prompts]
+        prompts_regex = [re.compile(to_bytes(r), re.I) for r in prompts]
         for index, regex in enumerate(prompts_regex):
             match = regex.search(resp)
             if match:
@@ -557,13 +623,14 @@ class Connection(NetworkConnectionBase):
         '''
         errored_response = None
         is_error_message = False
-        for regex in self._terminal.terminal_stderr_re:
+
+        for regex in self._terminal_stderr_re:
             if regex.search(response):
                 is_error_message = True
 
                 # Check if error response ends with command prompt if not
                 # receive it buffered prompt
-                for regex in self._terminal.terminal_stdout_re:
+                for regex in self._terminal_stdout_re:
                     match = regex.search(response)
                     if match:
                         errored_response = response
@@ -573,7 +640,7 @@ class Connection(NetworkConnectionBase):
                         break
 
         if not is_error_message:
-            for regex in self._terminal.terminal_stdout_re:
+            for regex in self._terminal_stdout_re:
                 match = regex.search(response)
                 if match:
                     self._matched_pattern = regex.pattern
@@ -604,3 +671,23 @@ class Connection(NetworkConnectionBase):
         self.close()
         self._connect()
         self.close()
+
+    def _get_terminal_std_re(self, option):
+        terminal_std_option = self.get_option(option)
+        terminal_std_re = []
+
+        if terminal_std_option:
+            for item in terminal_std_option:
+                if "pattern" not in item:
+                    raise AnsibleConnectionFailure("'pattern' is a required key for option '%s',"
+                                                   " received option value is %s" % (option, item))
+                pattern = br"%s" % to_bytes(item['pattern'])
+                flag = item.get('flags', 0)
+                if flag:
+                    flag = getattr(re, flag.split('.')[1])
+                terminal_std_re.append(re.compile(pattern, flag))
+        else:
+            # To maintain backward compatibility
+            terminal_std_re = getattr(self._terminal, option)
+
+        return terminal_std_re

test/integration/targets/ios_command/tests/cli/error_regex.yaml

@@ -0,0 +1,59 @@
+---
+- debug: msg="START cli/error_regex.yaml on connection={{ ansible_connection }}"
+
+- block:
+  - name: clear logs 1
+    cli_command: &clear_logs
+      command: clear logging
+      prompt:
+        - Clear logging buffer
+      answer:
+        - "\r"
+    ignore_errors: True
+
+  - name: send log with error regex match 1
+    cli_command: &send_logs
+      command: "send log 'IPSEC-3-REPLAY_ERROR: test log_1'"
+    ignore_errors: True
+
+  - name: fetch logs without command specific error regex
+    ios_command:
+      commands:
+        - show logging
+    register: result
+    ignore_errors: True
+
+  - name: ensure task fails due to mismatched regex
+    assert:
+      that:
+        - "result.failed == true"
+
+  - name: pause to avoid rate limiting
+    pause:
+      seconds: 10
+
+  - name: clear logs 2
+    cli_command: *clear_logs
+    ignore_errors: True
+
+  - name: send log with error regex match 2
+    cli_command: *send_logs
+    ignore_errors: True
+
+  - name: fetch logs with command specific error regex
+    ios_command:
+      commands:
+        - show logging
+    register: result
+    vars:
+      ansible_terminal_stderr_re:
+        - pattern: 'connection timed out'
+          flags: 're.I'
+
+  - name: ensure task with modified error regex is success
+    assert:
+      that:
+        - "result.failed == false"
+  when: ansible_connection == 'network_cli'
+
+- debug: msg="END cli/error_regex.yaml on connection={{ ansible_connection }}"

test/units/plugins/connection/test_network_cli.py

@@ -111,15 +111,16 @@ class TestConnectionClass(unittest.TestCase):
         self.assertEqual(out, b'command response')
         mock_send.assert_called_with(command=b'command')
 
+    @patch("ansible.plugins.connection.network_cli.Connection._get_terminal_std_re")
     @patch("ansible.plugins.connection.network_cli.Connection._connect")
-    def test_network_cli_send(self, mocked_connect):
+    def test_network_cli_send(self, mocked_connect, mocked_terminal_re):
+
         pc = PlayContext()
         pc.network_os = 'ios'
         conn = connection_loader.get('network_cli', pc, '/dev/null')
 
         mock__terminal = MagicMock()
-        mock__terminal.terminal_stdout_re = [re.compile(b'device#')]
+        mocked_terminal_re.side_effect = [[re.compile(b'^ERROR')], [re.compile(b'device#')]]
-        mock__terminal.terminal_stderr_re = [re.compile(b'^ERROR')]
         conn._terminal = mock__terminal
 
         mock__shell = MagicMock()
@@ -139,7 +140,7 @@ class TestConnectionClass(unittest.TestCase):
 
         mock__shell.reset_mock()
         mock__shell.recv.side_effect = [b"ERROR: error message device#"]
-
+        mocked_terminal_re.side_effect = [[re.compile(b'^ERROR')], [re.compile(b'device#')]]
         with self.assertRaises(AnsibleConnectionFailure) as exc:
             conn.send(b'command')
         self.assertEqual(str(exc.exception), 'ERROR: error message device#')