Provide a way to explicitly invoke the debugger (#34006)

* Provide a way to explicitly invoke the debugger with in the debug strategy

* Merge the debugger strategy into StrategyBase

* Fix some logic, pin to a single result

* Make redo also continue

* Make sure that if the debug closure doesn't need to process the result, that we still return it

* Fix failing tests for the strategy

* Clean up messages from debugger and exit code to match bin/ansible

* Move the FieldAttribute higher, to apply at different levels

* make debugger a string, expand logic

* Better host state rollbacks

* More explicit debugger prompt

* ENABLE_TASK_DEBUGGER should be boolean, and better docs

* No bare except, add pprint, alias h, vars to task_vars

* _validate_debugger can ignore non-string, that can be caught later

* Address issue if there were no previous tasks/state, and use the correct key

* Update docs for changes to the debugger

* Guard against a stat going negative through use of decrement

* Add a few notes about using the debugger on the free strategy

* Add changelog entry for task debugger

* Add a few versionadded indicators and a note about vars -> task_vars
This commit is contained in:
Matt Martz 2018-01-09 13:50:07 -06:00 committed by GitHub
parent e802b769e6
commit d1846425db
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
9 changed files with 394 additions and 176 deletions

View file

@ -53,6 +53,8 @@ Ansible Changes By Release
2.9. use `get_checksum: True` with `checksum_algorithm: md5` to return an md5 hash of the file
under the `checksum` return value.
* `osx_say` module was renamed into `say`.
* Task debugger functionality was moved into `StrategyBase`, and extended to allow explicit invocation from use of the `debugger` keyword.
The `debug` strategy is still functional, and is now just a trigger to enable this functionality
#### Deprecated Modules (to be removed in 2.9):
* ec2_ami_find: replaced by ec2_ami_facts

View file

@ -3,8 +3,93 @@ Playbook Debugger
.. contents:: Topics
Ansible includes a ``debug`` strategy. This strategy enables you to invoke a debugger when a task has
failed. You have access to all of the features of the debugger in the context of the failed task. You can then, for example, check or set the value of variables, update module arguments, and re-run the failed task with the new variables and arguments to help resolve the cause of the failure.
Ansible includes a debugger as part of the strategy plugins. This debugger enables you to debug as task.
You have access to all of the features of the debugger in the context of the task. You can then, for example, check or set the value of variables, update module arguments, and re-run the task with the new variables and arguments to help resolve the cause of the failure.
There are multiple ways to invoke the debugger.
Using the debugger keyword
++++++++++++++++++++++++++
.. versionadded:: 2.5
The ``debugger`` keyword can be used on any block where you provide a ``name`` attribute, such as a play, role, block or task.
The ``debugger`` keyword accepts several values:
always
Always invoke the debugger, regardless of the outcome
never
Never invoke the debugger, regardless of the outcome
on_failed
Only invoke the debugger if a task fails
on_unreachable
Only invoke the debugger if the a host was unreachable
on_skipped
Only invoke the debugger if the task is skipped
These options override any global configuration to enable or disable the debugger.
On a task
`````````
::
- name: Execute a command
command: false
debugger: on_failed
On a play
`````````
::
- name: Play
hosts: all
debugger: on_skipped
tasks:
- name: Execute a command
command: true
when: False
When provided at a generic level and a more specific level, the more specific wins::
- name: Play
hosts: all
debugger: never
tasks:
- name: Execute a command
command: false
debugger: on_failed
Configuration or environment variable
+++++++++++++++++++++++++++++++++++++
.. versionadded:: 2.5
In ansible.cfg::
[defaults]
enable_task_debugger = True
As an environment variable::
ANSIBLE_ENABLE_TASK_DEBUGGER=True ansible-playbook -i hosts site.yml
When using this method, any failed or unreachable task will invoke the debugger,
unless otherwise explicitly disabled.
As a Strategy
+++++++++++++
.. note::
This is a backwards compatible method, to match Ansible versions before 2.5,
and may be remoevd in a future release
To use the ``debug`` strategy, change the ``strategy`` attribute like this::
@ -14,12 +99,19 @@ To use the ``debug`` strategy, change the ``strategy`` attribute like this::
...
If you don't want change the code, you can define ``ANSIBLE_STRATEGY=debug``
environment variable in order to enable the debugger.
environment variable in order to enable the debugger, or modify ``ansible.cfg`` such as::
[defaults]
strategy = debug
Examples
++++++++
For example, run the playbook below::
- hosts: test
strategy: debug
debugger: on_failed
gather_facts: no
vars:
var1: value1
@ -38,14 +130,27 @@ Let's change the module's arguments and run the task again
TASK [wrong variable] **********************************************************
fatal: [192.0.2.10]: FAILED! => {"failed": true, "msg": "ERROR! 'wrong_var' is undefined"}
Debugger invoked
(debug) p result
{'msg': u"ERROR! 'wrong_var' is undefined", 'failed': True}
(debug) p task.args
[192.0.2.10] TASK: wrong variable (debug)> p result._result
{'failed': True,
'msg': 'The task includes an option with an undefined variable. The error '
"was: 'wrong_var' is undefined\n"
'\n'
'The error appears to have been in '
"'playbooks/debugger.yml': line 7, "
'column 7, but may\n'
'be elsewhere in the file depending on the exact syntax problem.\n'
'\n'
'The offending line appears to be:\n'
'\n'
' tasks:\n'
' - name: wrong variable\n'
' ^ here\n'}
[192.0.2.10] TASK: wrong variable (debug)> p task.args
{u'data': u'{{ wrong_var }}'}
(debug) task.args['data'] = '{{ var1 }}'
(debug) p task.args
[192.0.2.10] TASK: wrong variable (debug)> task.args['data'] = '{{ var1 }}'
[192.0.2.10] TASK: wrong variable (debug)> p task.args
{u'data': '{{ var1 }}'}
(debug) redo
[192.0.2.10] TASK: wrong variable (debug)> redo
ok: [192.0.2.10]
PLAY RECAP *********************************************************************
@ -58,27 +163,27 @@ This time, the task runs successfully!
Available Commands
++++++++++++++++++
.. _p_command:
.. _pprint_command:
p *task/vars/host/result*
`````````````````````````
p(print) *task/task_vars/host/result*
`````````````````````````````````````
Print values used to execute a module::
(debug) p task
[192.0.2.10] TASK: install package (debug)> p task
TASK: install package
(debug) p task.args
[192.0.2.10] TASK: install package (debug)> p task.args
{u'name': u'{{ pkg_name }}'}
(debug) p vars
[192.0.2.10] TASK: install package (debug)> p task_vars
{u'ansible_all_ipv4_addresses': [u'192.0.2.10'],
u'ansible_architecture': u'x86_64',
...
}
(debug) p vars['pkg_name']
[192.0.2.10] TASK: install package (debug)> p task_vars['pkg_name']
u'bash'
(debug) p host
[192.0.2.10] TASK: install package (debug)> p host
192.0.2.10
(debug) p result
[192.0.2.10] TASK: install package (debug)> p result._result
{'_ansible_no_log': False,
'changed': False,
u'failed': True,
@ -105,32 +210,35 @@ If you run a playbook like this::
Debugger is invoked due to wrong package name, so let's fix the module's args::
(debug) p task.args
[192.0.2.10] TASK: install package (debug)> p task.args
{u'name': u'{{ pkg_name }}'}
(debug) task.args['name'] = 'bash'
(debug) p task.args
[192.0.2.10] TASK: install package (debug)> task.args['name'] = 'bash'
[192.0.2.10] TASK: install package (debug)> p task.args
{u'name': 'bash'}
(debug) redo
[192.0.2.10] TASK: install package (debug)> redo
Then the task runs again with new args.
.. _update_vars_command:
vars[*key*] = *value*
`````````````````````
task_vars[*key*] = *value*
``````````````````````````
Update vars.
Update ``task_vars``.
Let's use the same playbook above, but fix vars instead of args::
Let's use the same playbook above, but fix ``task_vars`` instead of args::
(debug) p vars['pkg_name']
[192.0.2.10] TASK: install package (debug)> p task_vars['pkg_name']
u'not_exist'
(debug) vars['pkg_name'] = 'bash'
(debug) p vars['pkg_name']
[192.0.2.10] TASK: install package (debug)> task_vars['pkg_name'] = 'bash'
[192.0.2.10] TASK: install package (debug)> p task_vars['pkg_name']
'bash'
(debug) redo
[192.0.2.10] TASK: install package (debug)> redo
Then the task runs again with new vars.
Then the task runs again with new ``task_vars``.
.. note::
In 2.5 this was updated from ``vars`` to ``task_vars`` to not conflict with the ``vars()`` python function.
.. _redo_command:
@ -153,6 +261,14 @@ q(uit)
Quit from the debugger. The playbook execution is aborted.
Use with the free strategy
++++++++++++++++++++++++++
Using the debugger on the ``free`` strategy will cause no further tasks to be queued or executed
while the debugger is active. Additionally, using ``redo`` on a task to schedule it for re-execution
may cause the rescheduled task to execute after subsequent tasks listed in your playbook.
.. seealso::
:doc:`playbooks`

View file

@ -932,6 +932,18 @@ DEFAULT_STDOUT_CALLBACK:
env: [{name: ANSIBLE_STDOUT_CALLBACK}]
ini:
- {key: stdout_callback, section: defaults}
ENABLE_TASK_DEBUGGER:
name: Whether to enable the task debugger
default: False
description:
- Whether or not to enable the task debugger, this previously was done as a strategy plugin.
- Now all strategy plugins can inherit this behavior. The debugger defaults to activating when
- a task is failed on unreachable. Use the debugger keyword for more flexibility.
type: boolean
env: [{name: ANSIBLE_ENABLE_TASK_DEBUGGER}]
ini:
- {key: enable_task_debugger, section: defaults}
version_added: "2.5"
DEFAULT_STRATEGY:
name: Implied strategy
default: 'linear'

View file

@ -46,6 +46,16 @@ class AggregateStats:
prev = (getattr(self, what)).get(host, 0)
getattr(self, what)[host] = prev + 1
def decrement(self, what, host):
_what = getattr(self, what)
try:
if _what[host] - 1 < 0:
# This should never happen, but let's be safe
raise KeyError("Don't be so negative")
_what[host] -= 1
except KeyError:
_what[host] = 0
def summarize(self, host):
''' return information about a particular host '''

View file

@ -64,6 +64,26 @@ class TaskResult:
def is_unreachable(self):
return self._check_key('unreachable')
def needs_debugger(self, globally_enabled=False):
_debugger = self._task_fields.get('debugger')
ret = False
if globally_enabled and (self.is_failed() or self.is_unreachable()):
ret = True
if _debugger in ('always',):
ret = True
elif _debugger in ('never',):
ret = False
elif _debugger in ('on_failed',) and self.is_failed():
ret = True
elif _debugger in ('on_unreachable',) and self.is_unreachable():
ret = True
elif _debugger in('on_skipped',) and self.is_skipped():
ret = True
return ret
def _check_key(self, key):
'''get a specific key from the result or its items'''

View file

@ -162,6 +162,9 @@ class Base(with_metaclass(BaseMeta, object)):
_diff = FieldAttribute(isa='bool')
_any_errors_fatal = FieldAttribute(isa='bool')
# explicitly invoke a debugger on tasks
_debugger = FieldAttribute(isa='string')
# param names which have been deprecated/removed
DEPRECATED_ATTRIBUTES = [
'sudo', 'sudo_user', 'sudo_pass', 'sudo_exe', 'sudo_flags',
@ -274,6 +277,12 @@ class Base(with_metaclass(BaseMeta, object)):
def get_variable_manager(self):
return self._variable_manager
def _validate_debugger(self, attr, name, value):
valid_values = frozenset(('always', 'on_failed', 'on_unreachable', 'on_skipped', 'never'))
if value and isinstance(value, string_types) and value not in valid_values:
raise AnsibleParserError("'%s' is not a valid value for debugger. Must be one of %s" % (value, ', '.join(valid_values)), obj=self.get_ds())
return value
def _validate_attributes(self, ds):
'''
Ensures that there are no keys in the datastructure which do

View file

@ -19,7 +19,11 @@
from __future__ import (absolute_import, division, print_function)
__metaclass__ = type
import cmd
import functools
import os
import pprint
import sys
import threading
import time
@ -76,6 +80,7 @@ class SharedPluginLoaderObj:
self.lookup_loader = lookup_loader
self.module_loader = module_loader
_sentinel = StrategySentinel()
@ -95,6 +100,67 @@ def results_thread_main(strategy):
pass
def debug_closure(func):
"""Closure to wrap ``StrategyBase._process_pending_results`` and invoke the task debugger"""
@functools.wraps(func)
def inner(self, iterator, one_pass=False, max_passes=None):
status_to_stats_map = (
('is_failed', 'failures'),
('is_unreachable', 'dark'),
('is_changed', 'changed'),
('is_skipped', 'skipped'),
)
# We don't know the host yet, copy the previous states, for lookup after we process new results
prev_host_states = iterator._host_states.copy()
results = func(self, iterator, one_pass=one_pass, max_passes=max_passes)
_processed_results = []
for result in results:
task = result._task
host = result._host
_queue_task_args = self._queue_task_args.pop('%s%s' % (host.name, task._uuid))
task_vars = _queue_task_args['task_vars']
play_context = _queue_task_args['play_context']
# Try to grab the previous host state, if it doesn't exist use get_host_state to generate an empty state
try:
prev_host_state = prev_host_states[host.name]
except KeyError:
prev_host_state = iterator.get_host_state(host)
while result.needs_debugger(globally_enabled=self.debugger_active):
next_action = NextAction()
dbg = Debugger(task, host, task_vars, play_context, result, next_action)
dbg.cmdloop()
if next_action.result == NextAction.REDO:
# rollback host state
self._tqm.clear_failed_hosts()
iterator._host_states[host.name] = prev_host_state
for method, what in status_to_stats_map:
if getattr(result, method)():
self._tqm._stats.decrement(what, host.name)
self._tqm._stats.decrement('ok', host.name)
# redo
self._queue_task(host, task, task_vars, play_context)
_processed_results.extend(debug_closure(func)(self, iterator, one_pass))
break
elif next_action.result == NextAction.CONTINUE:
_processed_results.append(result)
break
elif next_action.result == NextAction.EXIT:
# Matches KeyboardInterrupt from bin/ansible
sys.exit(99)
else:
_processed_results.append(result)
return _processed_results
return inner
class StrategyBase:
'''
@ -113,6 +179,7 @@ class StrategyBase:
self._final_q = tqm._final_q
self._step = getattr(tqm._options, 'step', False)
self._diff = getattr(tqm._options, 'diff', False)
self._queue_task_args = {}
# Backwards compat: self._display isn't really needed, just import the global display and use that.
self._display = display
@ -137,6 +204,8 @@ class StrategyBase:
# play completion
self._active_connections = dict()
self.debugger_active = C.ENABLE_TASK_DEBUGGER
def cleanup(self):
# close active persistent connections
for sock in itervalues(self._active_connections):
@ -201,6 +270,13 @@ class StrategyBase:
def _queue_task(self, host, task, task_vars, play_context):
''' handles queueing the task up to be sent to a worker '''
self._queue_task_args['%s%s' % (host.name, task._uuid)] = {
'host': host,
'task': task,
'task_vars': task_vars,
'play_context': play_context
}
display.debug("entering _queue_task() for %s/%s" % (host.name, task.action))
# Add a write lock for tasks.
@ -268,6 +344,7 @@ class StrategyBase:
return [actual_host]
@debug_closure
def _process_pending_results(self, iterator, one_pass=False, max_passes=None):
'''
Reads results off the final queue and takes appropriate action
@ -949,3 +1026,106 @@ class StrategyBase:
if socket_path:
if r._host not in self._active_connections:
self._active_connections[r._host] = socket_path
class NextAction(object):
""" The next action after an interpreter's exit. """
REDO = 1
CONTINUE = 2
EXIT = 3
def __init__(self, result=EXIT):
self.result = result
class Debugger(cmd.Cmd):
prompt_continuous = '> ' # multiple lines
def __init__(self, task, host, task_vars, play_context, result, next_action):
# cmd.Cmd is old-style class
cmd.Cmd.__init__(self)
self.prompt = '[%s] %s (debug)> ' % (host, task)
self.intro = None
self.scope = {}
self.scope['task'] = task
self.scope['task_vars'] = task_vars
self.scope['host'] = host
self.scope['play_context'] = play_context
self.scope['result'] = result
self.next_action = next_action
def cmdloop(self):
try:
cmd.Cmd.cmdloop(self)
except KeyboardInterrupt:
pass
do_h = cmd.Cmd.do_help
def do_EOF(self, args):
"""Quit"""
return self.do_quit(args)
def do_quit(self, args):
"""Quit"""
display.display('User interrupted execution')
self.next_action.result = NextAction.EXIT
return True
do_q = do_quit
def do_continue(self, args):
"""Continue to next result"""
self.next_action.result = NextAction.CONTINUE
return True
do_c = do_continue
def do_redo(self, args):
"""Schedule task for re-execution. The re-execution may not be the next result"""
self.next_action.result = NextAction.REDO
return True
do_r = do_redo
def evaluate(self, args):
try:
return eval(args, globals(), self.scope)
except Exception:
t, v = sys.exc_info()[:2]
if isinstance(t, str):
exc_type_name = t
else:
exc_type_name = t.__name__
display.display('***%s:%s' % (exc_type_name, repr(v)))
raise
def do_pprint(self, args):
"""Pretty Print"""
try:
result = self.evaluate(args)
display.display(pprint.pformat(result))
except Exception:
pass
do_p = do_pprint
def execute(self, args):
try:
code = compile(args + '\n', '<stdin>', 'single')
exec(code, globals(), self.scope)
except Exception:
t, v = sys.exc_info()[:2]
if isinstance(t, str):
exc_type_name = t
else:
exc_type_name = t.__name__
display.display('***%s:%s' % (exc_type_name, repr(v)))
raise
def default(self, line):
try:
self.execute(line)
except Exception:
pass

View file

@ -28,153 +28,10 @@ import cmd
import pprint
import sys
from ansible.module_utils.six.moves import reduce
from ansible.plugins.strategy.linear import StrategyModule as LinearStrategyModule
try:
from __main__ import display
except ImportError:
from ansible.utils.display import Display
display = Display()
class NextAction(object):
""" The next action after an interpreter's exit. """
REDO = 1
CONTINUE = 2
EXIT = 3
def __init__(self, result=EXIT):
self.result = result
class StrategyModule(LinearStrategyModule):
def __init__(self, tqm):
self.curr_tqm = tqm
super(StrategyModule, self).__init__(tqm)
def _queue_task(self, host, task, task_vars, play_context):
self.curr_host = host
self.curr_task = task
self.curr_task_vars = task_vars
self.curr_play_context = play_context
super(StrategyModule, self)._queue_task(host, task, task_vars, play_context)
def _process_pending_results(self, iterator, one_pass=False, max_passes=None):
if not hasattr(self, "curr_host"):
return super(StrategyModule, self)._process_pending_results(iterator, one_pass, max_passes)
prev_host_state = iterator.get_host_state(self.curr_host)
results = super(StrategyModule, self)._process_pending_results(iterator, one_pass)
while self._need_debug(results):
next_action = NextAction()
dbg = Debugger(self, results, next_action)
dbg.cmdloop()
if next_action.result == NextAction.REDO:
# rollback host state
self.curr_tqm.clear_failed_hosts()
iterator._host_states[self.curr_host.name] = prev_host_state
if reduce(lambda total, res: res.is_failed() or total, results, False):
self._tqm._stats.failures[self.curr_host.name] -= 1
elif reduce(lambda total, res: res.is_unreachable() or total, results, False):
self._tqm._stats.dark[self.curr_host.name] -= 1
# redo
super(StrategyModule, self)._queue_task(self.curr_host, self.curr_task, self.curr_task_vars, self.curr_play_context)
results = super(StrategyModule, self)._process_pending_results(iterator, one_pass)
elif next_action.result == NextAction.CONTINUE:
break
elif next_action.result == NextAction.EXIT:
exit(1)
return results
def _need_debug(self, results):
return reduce(lambda total, res: res.is_failed() or res.is_unreachable() or total, results, False)
class Debugger(cmd.Cmd):
prompt = '(debug) ' # debugger
prompt_continuous = '> ' # multiple lines
def __init__(self, strategy_module, results, next_action):
# cmd.Cmd is old-style class
cmd.Cmd.__init__(self)
self.intro = "Debugger invoked"
self.scope = {}
self.scope['task'] = strategy_module.curr_task
self.scope['vars'] = strategy_module.curr_task_vars
self.scope['host'] = strategy_module.curr_host
self.scope['result'] = results[0]._result
self.scope['results'] = results # for debug of this debugger
self.next_action = next_action
def cmdloop(self):
try:
cmd.Cmd.cmdloop(self)
except KeyboardInterrupt:
pass
def do_EOF(self, args):
return self.do_quit(args)
def do_quit(self, args):
display.display('aborted')
self.next_action.result = NextAction.EXIT
return True
do_q = do_quit
def do_continue(self, args):
self.next_action.result = NextAction.CONTINUE
return True
do_c = do_continue
def do_redo(self, args):
self.next_action.result = NextAction.REDO
return True
do_r = do_redo
def evaluate(self, args):
try:
return eval(args, globals(), self.scope)
except:
t, v = sys.exc_info()[:2]
if isinstance(t, str):
exc_type_name = t
else:
exc_type_name = t.__name__
display.display('***%s:%s' % (exc_type_name, repr(v)))
raise
def do_p(self, args):
try:
result = self.evaluate(args)
display.display(pprint.pformat(result))
except:
pass
def execute(self, args):
try:
code = compile(args + '\n', '<stdin>', 'single')
exec(code, globals(), self.scope)
except:
t, v = sys.exc_info()[:2]
if isinstance(t, str):
exc_type_name = t
else:
exc_type_name = t.__name__
display.display('***%s:%s' % (exc_type_name, repr(v)))
raise
def default(self, line):
try:
self.execute(line)
except:
pass
self.debugger_active = True

View file

@ -336,6 +336,13 @@ class TestStrategyBase(unittest.TestCase):
queue_items.append(task_result)
strategy_base._blocked_hosts['test01'] = True
strategy_base._pending_results = 1
strategy_base._queue_task_args = MagicMock()
strategy_base._queue_task_args.pop.return_value = {
'task_vars': {},
'play_context': {},
}
results = strategy_base._wait_on_pending_results(iterator=mock_iterator)
self.assertEqual(len(results), 1)
self.assertEqual(results[0], task_result)
@ -539,6 +546,11 @@ class TestStrategyBase(unittest.TestCase):
strategy_base._notified_handlers = {mock_handler_task._uuid: [mock_host]}
task_result = TaskResult(Host('host01'), Handler(), dict(changed=False))
strategy_base._queue_task_args = MagicMock()
strategy_base._queue_task_args.pop.return_value = {
'task_vars': {},
'play_context': mock_play_context
}
tqm._final_q.put(task_result)
result = strategy_base.run_handlers(iterator=mock_iterator, play_context=mock_play_context)