cmueller/ansible
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

starting metadata sunset ()

Browse source 
* starting metadata sunset

 - purged metadata from any requirements
 - fix indent in generic handler for yaml content (whey metadata display was off)
 - make more resilient against bad formed docs
 - removed all metadata from docs template
 - remove metadata from schemas
 - removed mdata tests and from unrelated tests

Co-authored-by: Felix Fontein <felix@fontein.de>
Co-authored-by: Rick Elrod <rick@elrod.me>
This commit is contained in:
Brian Coca 2020-06-04 21:01:46 -04:00 • committed by GitHub
parent f5718a354c
commit 062e780a68
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
91 changed files with 140 additions and 1761 deletions

60
docs/docsite/rst/dev_guide/developing_modules_documenting.rst
View file

@ -15,11 +15,11 @@ Every Ansible module written in Python must begin with seven standard sections i
.. note:: Why don't the imports go first?
Keen Python programmers may notice that contrary to PEP 8's advice we don't put ``imports`` at the top of the file. This is because the ``ANSIBLE_METADATA`` through ``RETURN`` sections are not used by the module code itself; they are essentially extra docstrings for the file. The imports are placed after these special variables for the same reason as PEP 8 puts the imports after the introductory comments and docstrings. This keeps the active parts of the code together and the pieces which are purely informational apart. The decision to exclude E402 is based on readability (which is what PEP 8 is about). Documentation strings in a module are much more similar to module level docstrings, than code, and are never utilized by the module itself. Placing the imports below this documentation and closer to the code, consolidates and groups all related code in a congruent manner to improve readability, debugging and understanding.
Keen Python programmers may notice that contrary to PEP 8's advice we don't put ``imports`` at the top of the file. This is because the ``DOCUMENTATION`` through ``RETURN`` sections are not used by the module code itself; they are essentially extra docstrings for the file. The imports are placed after these special variables for the same reason as PEP 8 puts the imports after the introductory comments and docstrings. This keeps the active parts of the code together and the pieces which are purely informational apart. The decision to exclude E402 is based on readability (which is what PEP 8 is about). Documentation strings in a module are much more similar to module level docstrings, than code, and are never utilized by the module itself. Placing the imports below this documentation and closer to the code, consolidates and groups all related code in a congruent manner to improve readability, debugging and understanding.
.. warning:: **Copy old modules with care!**
Some older modules in Ansible Core have ``imports`` at the bottom of the file, ``Copyright`` notices with the full GPL prefix, and/or ``ANSIBLE_METADATA`` fields in the wrong order. These are legacy files that need updating - do not copy them into new modules. Over time we're updating and correcting older modules. Please follow the guidelines on this page!
Some older modules in Ansible Core have ``imports`` at the bottom of the file, ``Copyright`` notices with the full GPL prefix, and/or ``DOCUMENTATION`` fields in the wrong order. These are legacy files that need updating - do not copy them into new modules. Over time we're updating and correcting older modules. Please follow the guidelines on this page!
.. _shebang:
@ -60,61 +60,15 @@ Major additions to the module (for instance, rewrites) may add additional copyri
ANSIBLE_METADATA block
======================
After the shebang, the UTF-8 coding, the copyright, and the license, your module file should contain an ``ANSIBLE_METADATA`` section. This section provides information about the module for use by other tools. For new modules, the following block can be simply added into your module:
Since we moved to collections we have deprecated the METADATA functionality, it is no longer required for modules, but it will not break anything if present.
.. code-block:: python
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['preview'],
'supported_by': 'community'}
.. warning::
* ``metadata_version`` is the version of the ``ANSIBLE_METADATA`` schema, *not* the version of the module.
* Promoting a module's ``status`` or ``supported_by`` status should only be done by members of the Ansible Core Team.
Ansible metadata fields
-----------------------
:metadata_version: An "X.Y" formatted string. X and Y are integers which
define the metadata format version. Modules shipped with Ansible are
tied to an Ansible release, so we will only ship with a single version
of the metadata. We'll increment Y if we add fields or legal values
to an existing field. We'll increment X if we remove fields or values
or change the type or meaning of a field.
Current metadata_version is "1.1"
:supported_by: Who supports the module.
Default value is ``community``. For information on what the support level values entail, please see
:ref:`Modules Support <modules_support>`. Values are:
* core
* network
* certified
* community
* curated (*deprecated value - modules in this category should be core or
certified instead*)
:status: List of strings describing how stable the module is likely to be. See also :ref:`module_lifecycle`.
The default value is a single element list ["preview"]. The following strings are valid
statuses and have the following meanings:
:stableinterface: The module's options (the parameters or arguments it accepts) are stable. Every effort will be made not to remove options or change
their meaning. **Not** a rating of the module's code quality.
:preview: The module is in tech preview. It may be
unstable, the options may change, or it may require libraries or
web services that are themselves subject to incompatible changes.
:deprecated: The module is deprecated and will be removed in a future release.
:removed: The module is not present in the release. A stub is
kept so that documentation can be built. The documentation helps
users port from the removed module to new modules.
.. _documentation_block:
DOCUMENTATION block
===================
After the shebang, the UTF-8 coding, the copyright line, the license, and the ``ANSIBLE_METADATA`` section comes the ``DOCUMENTATION`` block. Ansible's online module documentation is generated from the ``DOCUMENTATION`` blocks in each module's source code. The ``DOCUMENTATION`` block must be valid YAML. You may find it easier to start writing your ``DOCUMENTATION`` string in an :ref:`editor with YAML syntax highlighting <other_tools_and_programs>` before you include it in your Python file. You can start by copying our `example documentation string <https://github.com/ansible/ansible/blob/devel/examples/DOCUMENTATION.yml>`_ into your module file and modifying it. If you run into syntax issues in your YAML, you can validate it on the `YAML Lint <http://www.yamllint.com/>`_ website.
After the shebang, the UTF-8 coding, the copyright line, and the license section comes the ``DOCUMENTATION`` block. Ansible's online module documentation is generated from the ``DOCUMENTATION`` blocks in each module's source code. The ``DOCUMENTATION`` block must be valid YAML. You may find it easier to start writing your ``DOCUMENTATION`` string in an :ref:`editor with YAML syntax highlighting <other_tools_and_programs>` before you include it in your Python file. You can start by copying our `example documentation string <https://github.com/ansible/ansible/blob/devel/examples/DOCUMENTATION.yml>`_ into your module file and modifying it. If you run into syntax issues in your YAML, you can validate it on the `YAML Lint <http://www.yamllint.com/>`_ website.
Module documentation should briefly and accurately define what each module and option does, and how it works with others in the underlying system. Documentation should be written for broad audience--readable both by experts and non-experts.
* Descriptions should always start with a capital letter and end with a full stop. Consistency always helps.
@ -353,7 +307,7 @@ For example, all AWS modules should include:
EXAMPLES block
==============
After the shebang, the UTF-8 coding, the copyright line, the license, the ``ANSIBLE_METADATA`` section, and the ``DOCUMENTATION`` block comes the ``EXAMPLES`` block. Here you show users how your module works with real-world examples in multi-line plain-text YAML format. The best examples are ready for the user to copy and paste into a playbook. Review and update your examples with every change to your module.
After the shebang, the UTF-8 coding, the copyright line, the license section, and the ``DOCUMENTATION`` block comes the ``EXAMPLES`` block. Here you show users how your module works with real-world examples in multi-line plain-text YAML format. The best examples are ready for the user to copy and paste into a playbook. Review and update your examples with every change to your module.
Per playbook best practices, each example should include a ``name:`` line::
@ -375,7 +329,7 @@ If your module returns facts that are often needed, an example of how to use the
RETURN block
============
After the shebang, the UTF-8 coding, the copyright line, the license, the ``ANSIBLE_METADATA`` section, ``DOCUMENTATION`` and ``EXAMPLES`` blocks comes the ``RETURN`` block. This section documents the information the module returns for use by other modules.
After the shebang, the UTF-8 coding, the copyright line, the license section, ``DOCUMENTATION`` and ``EXAMPLES`` blocks comes the ``RETURN`` block. This section documents the information the module returns for use by other modules.
If your module doesn't return anything (apart from the standard returns), this section of your module should read: ``RETURN = r''' # '''``
Otherwise, for each value returned, provide the following fields. All fields are required unless specified otherwise.
@ -447,7 +401,7 @@ Here are two example ``RETURN`` sections, one with three simple fields and one w
Python imports
==============
After the shebang, the UTF-8 coding, the copyright line, the license, and the sections for ``ANSIBLE_METADATA``, ``DOCUMENTATION``, ``EXAMPLES``, and ``RETURN``, you can finally add the python imports. All modules must use Python imports in the form:
After the shebang, the UTF-8 coding, the copyright line, the license, and the sections for ``DOCUMENTATION``, ``EXAMPLES``, and ``RETURN``, you can finally add the python imports. All modules must use Python imports in the form:
.. code-block:: python

6
docs/docsite/rst/dev_guide/developing_modules_general.rst
View file

@ -63,12 +63,6 @@ To create a new module:
# Copyright: (c) 2018, Terry Jones <terry.jones@example.org>
# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
ANSIBLE_METADATA = {
'metadata_version': '1.1',
'status': ['preview'],
'supported_by': 'community'
}
DOCUMENTATION = '''
---
module: my_test

19
docs/docsite/rst/dev_guide/module_lifecycle.rst
View file

@ -4,15 +4,16 @@
The lifecycle of an Ansible module
**********************************
Modules in the main Ansible repo have a defined life cycle, from first introduction to final removal. The module life cycle is tied to the `Ansible release cycle <release_cycle>` and reflected in the :ref:`ansible_metadata_block`. A module may move through these four states:
Modules in the main Ansible repo have a defined life cycle, from first introduction to final removal. The module life cycle is tied to the `Ansible release cycle <release_cycle>`.
A module may move through these four states:
1. When a module is first accepted into Ansible, we consider it in tech preview and mark it ``preview``. Modules in ``preview`` are not stable. You may change the parameters or dependencies, expand or reduce the functionality of ``preview`` modules. Many modules remain ``preview`` for years.
1. When a module is first accepted into Ansible, we consider it in tech preview and will mark it as such in the documentation.
2. If a module matures, we may mark it ``stableinterface`` and commit to maintaining its parameters, dependencies, and functionality. We support (though we cannot guarantee) backwards compatibility for ``stableinterface`` modules, which means their parameters should be maintained with stable meanings.
2. If a module matures, we will remove the 'preview' mark in the documentation. We support (though we cannot guarantee) backwards compatibility for these modules, which means their parameters should be maintained with stable meanings.
3. If a module's target API changes radically, or if someone creates a better implementation of its functionality, we may mark it ``deprecated``. Modules that are ``deprecated`` are still available but they are reaching the end of their life cycle. We retain deprecated modules for 4 release cycles with deprecation warnings to help users update playbooks and roles that use them.
3. If a module's target API changes radically, or if someone creates a better implementation of its functionality, we may mark it deprecated. Modules that are deprecated are still available but they are reaching the end of their life cycle. We retain deprecated modules for 4 release cycles with deprecation warnings to help users update playbooks and roles that use them.
4. When a module has been deprecated for four release cycles, we remove the code and mark the stub file ``removed``. Modules that are ``removed`` are no longer shipped with Ansible. The stub file helps users find alternative modules.
4. When a module has been deprecated for four release cycles, we remove the code and mark the stub file removed. Modules that are removed are no longer shipped with Ansible. The stub file helps users find alternative modules.
.. _deprecating_modules:
@ -24,14 +25,16 @@ To deprecate a module, you must:
1. Rename the file so it starts with an ``_``, for example, rename ``old_cloud.py`` to ``_old_cloud.py``. This keeps the module available and marks it as deprecated on the module index pages.
2. Mention the deprecation in the relevant ``CHANGELOG``.
3. Reference the deprecation in the relevant ``porting_guide_x.y.rst``.
4. Update ``ANSIBLE_METADATA`` to contain ``status: ['deprecated']``.
5. Add ``deprecated:`` to the documentation with the following sub-values:
4. Add ``deprecated:`` to the documentation with the following sub-values:
:removed_in: A ``string``, such as ``"2.9"``; the version of Ansible where the module will be replaced with a docs-only module stub. Usually current release +4.
:removed_in: A ``string``, such as ``"2.10"``; the version of Ansible where the module will be replaced with a docs-only module stub. Usually current release +4.
:why: Optional string that used to detail why this has been removed.
:alternative: Inform users they should do instead, i.e. ``Use M(whatmoduletouseinstead) instead.``.
* note: with the advent of collections and ``routing.yml`` we might soon require another entry in this file to mark the deprecation.
* For an example of documenting deprecation, see this `PR that deprecates multiple modules <https://github.com/ansible/ansible/pull/43781/files>`_.
Some of the elements in the PR might now be out of date.
Changing a module name
======================

5
docs/docsite/rst/dev_guide/testing.rst
View file

@ -66,11 +66,10 @@ When Shippable detects an error and it can be linked back to a file that has bee
lib/ansible/modules/network/foo/bar.py:509:17: E265 block comment should start with '# '
The test `ansible-test sanity --test validate-modules` failed with the following errors:
The test `ansible-test sanity --test validate-modules` failed with the following error:
lib/ansible/modules/network/foo/bar.py:0:0: E307 version_added should be 2.4. Currently 2.3
lib/ansible/modules/network/foo/bar.py:0:0: E316 ANSIBLE_METADATA.metadata_version: required key not provided @ data['metadata_version']. Got None
From the above example we can see that ``--test pep8`` and ``--test validate-modules`` have identified issues. The commands given allow you to run the same tests locally to ensure you've fixed the issues without having to push your changed to GitHub and wait for Shippable, for example:
From the above example we can see that ``--test pep8`` and ``--test validate-modules`` have identified an issue. The commands given allow you to run the same tests locally to ensure you've fixed all issues without having to push your changes to GitHub and wait for Shippable, for example:
If you haven't already got Ansible available, use the local checkout by running::

11
docs/docsite/rst/dev_guide/testing_validate-modules.rst
View file

@ -80,10 +80,10 @@ Codes
documentation-error Documentation Error Unknown ``DOCUMENTATION`` error
documentation-syntax-error Documentation Error Invalid ``DOCUMENTATION`` schema
illegal-future-imports Imports Error Only the following ``from __future__`` imports are allowed: ``absolute_import``, ``division``, and ``print_function``.
import-before-documentation Imports Error Import found before documentation variables. All imports must appear below ``DOCUMENTATION``/``EXAMPLES``/``RETURN``/``ANSIBLE_METADATA``
import-before-documentation Imports Error Import found before documentation variables. All imports must appear below ``DOCUMENTATION``/``EXAMPLES``/``RETURN``
import-error Documentation Error ``Exception`` attempting to import module for ``argument_spec`` introspection
import-placement Locations Warning Imports should be directly below ``DOCUMENTATION``/``EXAMPLES``/``RETURN``/``ANSIBLE_METADATA`` for legacy modules
imports-improper-location Imports Error Imports should be directly below ``DOCUMENTATION``/``EXAMPLES``/``RETURN``/``ANSIBLE_METADATA``
import-placement Locations Warning Imports should be directly below ``DOCUMENTATION``/``EXAMPLES``/``RETURN``
imports-improper-location Imports Error Imports should be directly below ``DOCUMENTATION``/``EXAMPLES``/``RETURN``
incompatible-choices Documentation Error Choices value from the argument_spec is not compatible with type defined in the argument_spec
incompatible-default-type Documentation Error Default value from the argument_spec is not compatible with type defined in the argument_spec
invalid-argument-name Documentation Error Argument in argument_spec must not be one of 'message', 'syslog_facility' as it is used internally by Ansible Core Engine
@ -93,14 +93,10 @@ Codes
invalid-documentation-options Documentation Error ``DOCUMENTATION.options`` must be a dictionary/hash when used
invalid-examples Documentation Error ``EXAMPLES`` is not valid YAML
invalid-extension Naming Error Official Ansible modules must have a ``.py`` extension for python modules or a ``.ps1`` for powershell modules
invalid-metadata-status Documentation Error ``ANSIBLE_METADATA.status`` of deprecated or removed can't include other statuses
invalid-metadata-type Documentation Error ``ANSIBLE_METADATA`` was not provided as a dict, YAML not supported, Invalid ``ANSIBLE_METADATA`` schema
invalid-module-deprecation-source Documentation Error The deprecated version for the module must not be from a documentation fragment from another collection or Ansible-base
invalid-module-schema Documentation Error ``AnsibleModule`` schema validation error
invalid-requires-extension Naming Error Module ``#AnsibleRequires -CSharpUtil`` should not end in .cs, Module ``#Requires`` should not end in .psm1
invalid-tagged-version Documentation Error All version numbers specified in code have to be explicitly tagged with the collection name, i.e. ``community.general:1.2.3`` or ``ansible.builtin:2.10``
last-line-main-call Syntax Error Call to ``main()`` not the last line (or ``removed_module()`` in the case of deprecated & docs only modules)
metadata-changed Documentation Error ``ANSIBLE_METADATA`` cannot be changed in a point release for a stable branch
missing-doc-fragment Documentation Error ``DOCUMENTATION`` fragment missing
missing-existing-doc-fragment Documentation Warning Pre-existing ``DOCUMENTATION`` fragment missing
missing-documentation Documentation Error No ``DOCUMENTATION`` provided
@ -108,7 +104,6 @@ Codes
missing-gplv3-license Documentation Error GPLv3 license header not found
missing-if-name-main Syntax Error Next to last line is not ``if __name__ == "__main__":``
missing-main-call Syntax Error Did not find a call to ``main()`` (or ``removed_module()`` in the case of deprecated & docs only modules)
missing-metadata Documentation Error No ``ANSIBLE_METADATA`` provided
missing-module-utils-basic-import Imports Warning Did not find ``ansible.module_utils.basic`` import
missing-module-utils-import-csharp-requirements Imports Error No ``Ansible.ModuleUtils`` or C# Ansible util requirements/imports found
missing-powershell-interpreter Syntax Error Interpreter line is not ``#!powershell``

6
docs/docsite/rst/network/dev_guide/developing_resource_modules_network.rst
View file

@ -53,12 +53,6 @@ For example, the resource model builder includes the ``myos_interfaces.yml`` sam
---
GENERATOR_VERSION: '1.0'
ANSIBLE_METADATA: |
{
'metadata_version': '1.1',
'status': ['preview'],
'supported_by': '<support_group>'
}
NETWORK_OS: myos
RESOURCE: interfaces
COPYRIGHT: Copyright 2019 Red Hat

29
docs/templates/plugin.rst.j2 vendored
View file

@ -418,35 +418,6 @@ Status
- This @{ plugin_type }@ will be removed in version @{ deprecated['removed_in'] | default('') | string | rst_ify }@. *[deprecated]*
- For more information see `DEPRECATED`_.
{% else %}
{% set support = { 'core': 'the Ansible Core Team', 'network': 'the Ansible Network Team', 'certified': 'an Ansible Partner', 'community': 'the Ansible Community', 'curated': 'a Third Party'} %}
{% set module_states = { 'preview': 'not guaranteed to have a backwards compatible interface', 'stableinterface': 'guaranteed to have backward compatible interface changes going forward'} %}
{% if metadata %}
{% if metadata.status %}
{% for cur_state in metadata.status %}
- This @{ plugin_type }@ is @{ module_states[cur_state] }@. *[@{ cur_state }@]*
{% endfor %}
{% endif %}
{% if metadata.supported_by %}
{% set supported_by = support[metadata.supported_by] %}
- This @{ plugin_type }@ is :ref:`maintained by @{ supported_by }@ <modules_support>`. *[@{ metadata.supported_by }@]*
{% if metadata.supported_by in ('core', 'network') %}
Red Hat Support
~~~~~~~~~~~~~~~
More information about Red Hat's support of this @{ plugin_type }@ is available from this `Red Hat Knowledge Base article <https://access.redhat.com/articles/3166901>`_.
{% endif %}
{% endif %}
{% endif %}
{% endif %}
{% if author is defined -%}

537
hacking/metadata-tool.py
View file

@ -1,537 +0,0 @@
#!/usr/bin/env python
# (c) 2016-2017, Toshio Kuratomi <tkuratomi@ansible.com>
#
# This file is part of Ansible
#
# Ansible is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# Ansible is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with Ansible. If not, see <http://www.gnu.org/licenses/>.
# Make coding more python3-ish
from __future__ import (absolute_import, division, print_function)
__metaclass__ = type
import ast
import csv
import os
import sys
from collections import defaultdict
from distutils.version import StrictVersion
from pprint import pformat, pprint
from ansible.parsing.metadata import DEFAULT_METADATA, ParseError, extract_metadata
from ansible.plugins.loader import module_loader
# There's a few files that are not new-style modules. Have to blacklist them
NONMODULE_PY_FILES = frozenset(('async_wrapper.py',))
NONMODULE_MODULE_NAMES = frozenset(os.path.splitext(p)[0] for p in NONMODULE_PY_FILES)
class MissingModuleError(Exception):
"""Thrown when unable to find a plugin"""
pass
def usage():
print("""Usage:
metadata-tool.py report [--version X]
metadata-tool.py add [--version X] [--overwrite] CSVFILE
metadata-tool.py add-default [--version X] [--overwrite]
medatada-tool.py upgrade [--version X]""")
sys.exit(1)
def parse_args(arg_string):
if len(arg_string) < 1:
usage()
action = arg_string[0]
version = None
if '--version' in arg_string:
version_location = arg_string.index('--version')
arg_string.pop(version_location)
version = arg_string.pop(version_location)
overwrite = False
if '--overwrite' in arg_string:
overwrite = True
arg_string.remove('--overwrite')
csvfile = None
if len(arg_string) == 2:
csvfile = arg_string[1]
elif len(arg_string) > 2:
usage()
return action, {'version': version, 'overwrite': overwrite, 'csvfile': csvfile}
def find_documentation(module_data):
"""Find the DOCUMENTATION metadata for a module file"""
start_line = -1
mod_ast_tree = ast.parse(module_data)
for child in mod_ast_tree.body:
if isinstance(child, ast.Assign):
for target in child.targets:
if target.id == 'DOCUMENTATION':
start_line = child.lineno - 1
break
return start_line
def remove_metadata(module_data, start_line, start_col, end_line, end_col):
"""Remove a section of a module file"""
lines = module_data.split('\n')
new_lines = lines[:start_line]
if start_col != 0:
new_lines.append(lines[start_line][:start_col])
next_line = lines[end_line]
if len(next_line) - 1 != end_col:
new_lines.append(next_line[end_col:])
if len(lines) > end_line:
new_lines.extend(lines[end_line + 1:])
return '\n'.join(new_lines)
def insert_metadata(module_data, new_metadata, insertion_line, targets=('ANSIBLE_METADATA',)):
"""Insert a new set of metadata at a specified line"""
assignments = ' = '.join(targets)
pretty_metadata = pformat(new_metadata, width=1).split('\n')
new_lines = []
new_lines.append('{0} = {1}'.format(assignments, pretty_metadata[0]))
if len(pretty_metadata) > 1:
for line in pretty_metadata[1:]:
new_lines.append('{0}{1}'.format(' ' * (len(assignments) - 1 + len(' = {')), line))
old_lines = module_data.split('\n')
lines = old_lines[:insertion_line] + new_lines + old_lines[insertion_line:]
return '\n'.join(lines)
def parse_assigned_metadata_initial(csvfile):
"""
Fields:
:0: Module name
:1: Core (x if so)
:2: Extras (x if so)
:3: Category
:4: Supported/SLA
:5: Curated
:6: Stable
:7: Deprecated
:8: Notes
:9: Team Notes
:10: Notes 2
:11: final supported_by field
"""
with open(csvfile, 'rb') as f:
for record in csv.reader(f):
module = record[0]
if record[12] == 'core':
supported_by = 'core'
elif record[12] == 'curated':
supported_by = 'curated'
elif record[12] == 'community':
supported_by = 'community'
else:
print('Module %s has no supported_by field. Using community' % record[0])
supported_by = 'community'
supported_by = DEFAULT_METADATA['supported_by']
status = []
if record[6]:
status.append('stableinterface')
if record[7]:
status.append('deprecated')
if not status:
status.extend(DEFAULT_METADATA['status'])
yield (module, {'version': DEFAULT_METADATA['metadata_version'], 'supported_by': supported_by, 'status': status})
def parse_assigned_metadata(csvfile):
"""
Fields:
:0: Module name
:1: supported_by string. One of the valid support fields
core, community, certified, network
:2: stableinterface
:3: preview
:4: deprecated
:5: removed
https://docs.ansible.com/ansible/latest/dev_guide/developing_modules_documenting.html#ansible-metadata-block
"""
with open(csvfile, 'rb') as f:
for record in csv.reader(f):
module = record[0]
supported_by = record[1]
status = []
if record[2]:
status.append('stableinterface')
if record[4]:
status.append('deprecated')
if record[5]:
status.append('removed')
if not status or record[3]:
status.append('preview')
yield (module, {'metadata_version': '1.1', 'supported_by': supported_by, 'status': status})
def write_metadata(filename, new_metadata, version=None, overwrite=False):
with open(filename, 'rb') as f:
module_data = f.read()
try:
current_metadata, start_line, start_col, end_line, end_col, targets = \
extract_metadata(module_data=module_data, offsets=True)
except SyntaxError:
if filename.endswith('.py'):
raise
# Probably non-python modules. These should all have python
# documentation files where we can place the data
raise ParseError('Could not add metadata to {0}'.format(filename))
if current_metadata is None:
# No current metadata so we can just add it
start_line = find_documentation(module_data)
if start_line < 0:
if os.path.basename(filename) in NONMODULE_PY_FILES:
# These aren't new-style modules
return
raise Exception('Module file {0} had no ANSIBLE_METADATA or DOCUMENTATION'.format(filename))
module_data = insert_metadata(module_data, new_metadata, start_line, targets=('ANSIBLE_METADATA',))
elif overwrite or (version is not None and ('metadata_version' not in current_metadata or
StrictVersion(current_metadata['metadata_version']) < StrictVersion(version))):
# Current metadata that we do not want. Remove the current
# metadata and put the new version in its place
module_data = remove_metadata(module_data, start_line, start_col, end_line, end_col)
module_data = insert_metadata(module_data, new_metadata, start_line, targets=targets)
else:
# Current metadata and we don't want to overwrite it
return
# Save the new version of the module
with open(filename, 'wb') as f:
f.write(module_data)
def return_metadata(plugins):
"""Get the metadata for all modules
Handle duplicate module names
:arg plugins: List of plugins to look for
:returns: Mapping of plugin name to metadata dictionary
"""
metadata = {}
for name, filename in plugins:
# There may be several files for a module (if it is written in another
# language, for instance) but only one of them (the .py file) should
# contain the metadata.
if name not in metadata or metadata[name] is not None:
with open(filename, 'rb') as f:
module_data = f.read()
metadata[name] = extract_metadata(module_data=module_data, offsets=True)[0]
return metadata
def metadata_summary(plugins, version=None):
"""Compile information about the metadata status for a list of modules
:arg plugins: List of plugins to look for. Each entry in the list is
a tuple of (module name, full path to module)
:kwarg version: If given, make sure the modules have this version of
metadata or higher.
:returns: A tuple consisting of a list of modules with no metadata at the
required version and a list of files that have metadata at the
required version.
"""
no_metadata = {}
has_metadata = {}
supported_by = defaultdict(set)
status = defaultdict(set)
requested_version = StrictVersion(version)
all_mods_metadata = return_metadata(plugins)
for name, filename in plugins:
# Does the module have metadata?
if name not in no_metadata and name not in has_metadata:
metadata = all_mods_metadata[name]
if metadata is None:
no_metadata[name] = filename
elif version is not None and ('metadata_version' not in metadata or StrictVersion(metadata['metadata_version']) < requested_version):
no_metadata[name] = filename
else:
has_metadata[name] = filename
# What categories does the plugin belong in?
if all_mods_metadata[name] is None:
# No metadata for this module. Use the default metadata
supported_by[DEFAULT_METADATA['supported_by']].add(filename)
status[DEFAULT_METADATA['status'][0]].add(filename)
else:
supported_by[all_mods_metadata[name]['supported_by']].add(filename)
for one_status in all_mods_metadata[name]['status']:
status[one_status].add(filename)
return list(no_metadata.values()), list(has_metadata.values()), supported_by, status
# Filters to convert between metadata versions
def convert_metadata_pre_1_0_to_1_0(metadata):
"""
Convert pre-1.0 to 1.0 metadata format
:arg metadata: The old metadata
:returns: The new metadata
Changes from pre-1.0 to 1.0:
* ``version`` field renamed to ``metadata_version``
* ``supported_by`` field value ``unmaintained`` has been removed (change to
``community`` and let an external list track whether a module is unmaintained)
* ``supported_by`` field value ``committer`` has been renamed to ``curated``
"""
new_metadata = {'metadata_version': '1.0',
'supported_by': metadata['supported_by'],
'status': metadata['status']
}
if new_metadata['supported_by'] == 'unmaintained':
new_metadata['supported_by'] = 'community'
elif new_metadata['supported_by'] == 'committer':
new_metadata['supported_by'] = 'curated'
return new_metadata
def convert_metadata_1_0_to_1_1(metadata):
"""
Convert 1.0 to 1.1 metadata format
:arg metadata: The old metadata
:returns: The new metadata
Changes from 1.0 to 1.1:
* ``supported_by`` field value ``curated`` has been removed
* ``supported_by`` field value ``certified`` has been added
* ``supported_by`` field value ``network`` has been added
"""
new_metadata = {'metadata_version': '1.1',
'supported_by': metadata['supported_by'],
'status': metadata['status']
}
if new_metadata['supported_by'] == 'unmaintained':
new_metadata['supported_by'] = 'community'
elif new_metadata['supported_by'] == 'curated':
new_metadata['supported_by'] = 'certified'
return new_metadata
# Subcommands
def add_from_csv(csv_file, version=None, overwrite=False):
"""Implement the subcommand to add metadata from a csv file
"""
# Add metadata for everything from the CSV file
diagnostic_messages = []
for module_name, new_metadata in parse_assigned_metadata(csv_file):
filename = module_loader.find_plugin(module_name, mod_type='.py')
if filename is None:
diagnostic_messages.append('Unable to find the module file for {0}'.format(module_name))
continue
try:
write_metadata(filename, new_metadata, version, overwrite)
except ParseError as e:
diagnostic_messages.append(e.args[0])
continue
if diagnostic_messages:
pprint(diagnostic_messages)
return 0
def add_default(version=None, overwrite=False):
"""Implement the subcommand to add default metadata to modules
Add the default metadata to any plugin which lacks it.
:kwarg version: If given, the metadata must be at least this version.
Otherwise, treat the module as not having existing metadata.
:kwarg overwrite: If True, overwrite any existing metadata. Otherwise,
do not modify files which have metadata at an appropriate version
"""
# List of all plugins
plugins = module_loader.all(path_only=True)
plugins = ((os.path.splitext((os.path.basename(p)))[0], p) for p in plugins)
plugins = (p for p in plugins if p[0] not in NONMODULE_MODULE_NAMES)
# Iterate through each plugin
processed = set()
diagnostic_messages = []
for name, filename in (info for info in plugins if info[0] not in processed):
try:
write_metadata(filename, DEFAULT_METADATA, version, overwrite)
except ParseError as e:
diagnostic_messages.append(e.args[0])
continue
processed.add(name)
if diagnostic_messages:
pprint(diagnostic_messages)
return 0
def upgrade_metadata(version=None):
"""Implement the subcommand to upgrade the default metadata in modules.
:kwarg version: If given, the version of the metadata to upgrade to. If
not given, upgrade to the latest format version.
"""
if version is None:
# Number larger than any of the defined metadata formats.
version = 9999999
requested_version = StrictVersion(version)
# List all plugins
plugins = module_loader.all(path_only=True)
plugins = ((os.path.splitext((os.path.basename(p)))[0], p) for p in plugins)
plugins = (p for p in plugins if p[0] not in NONMODULE_MODULE_NAMES)
processed = set()
diagnostic_messages = []
for name, filename in (info for info in plugins if info[0] not in processed):
# For each plugin, read the existing metadata
with open(filename, 'rb') as f:
module_data = f.read()
metadata = extract_metadata(module_data=module_data, offsets=True)[0]
# If the metadata isn't the requested version, convert it to the new
# version
if 'metadata_version' not in metadata or metadata['metadata_version'] != version:
#
# With each iteration of metadata, add a new conditional to
# upgrade from the previous version
#
if 'metadata_version' not in metadata:
# First version, pre-1.0 final metadata
metadata = convert_metadata_pre_1_0_to_1_0(metadata)
if metadata['metadata_version'] == '1.0' and StrictVersion('1.0') < requested_version:
metadata = convert_metadata_1_0_to_1_1(metadata)
if metadata['metadata_version'] == '1.1' and StrictVersion('1.1') < requested_version:
# 1.1 version => XXX. We don't yet have anything beyond 1.1
# so there's nothing here
pass
# Replace the existing metadata with the new format
try:
write_metadata(filename, metadata, version, overwrite=True)
except ParseError as e:
diagnostic_messages.append(e.args[0])
continue
processed.add(name)
if diagnostic_messages:
pprint(diagnostic_messages)
return 0
def report(version=None):
"""Implement the report subcommand
Print out all the modules that have metadata and all the ones that do not.
:kwarg version: If given, the metadata must be at least this version.
Otherwise return it as not having metadata
"""
# List of all plugins
plugins = module_loader.all(path_only=True)
plugins = ((os.path.splitext((os.path.basename(p)))[0], p) for p in plugins)
plugins = (p for p in plugins if p[0] not in NONMODULE_MODULE_NAMES)
plugins = list(plugins)
no_metadata, has_metadata, support, status = metadata_summary(plugins, version=version)
print('== Has metadata ==')
pprint(sorted(has_metadata))
print('')
print('== Has no metadata ==')
pprint(sorted(no_metadata))
print('')
print('== Supported by core ==')
pprint(sorted(support['core']))
print('== Supported by value certified ==')
pprint(sorted(support['certified']))
print('== Supported by value network ==')
pprint(sorted(support['network']))
print('== Supported by community ==')
pprint(sorted(support['community']))
print('')
print('== Status: stableinterface ==')
pprint(sorted(status['stableinterface']))
print('== Status: preview ==')
pprint(sorted(status['preview']))
print('== Status: deprecated ==')
pprint(sorted(status['deprecated']))
print('== Status: removed ==')
pprint(sorted(status['removed']))
print('')
print('== Summary ==')
print('No Metadata: {0} Has Metadata: {1}'.format(len(no_metadata), len(has_metadata)))
print('Support level: core: {0} community: {1} certified: {2} network: {3}'.format(len(support['core']),
len(support['community']), len(support['certified']), len(support['network'])))
print('Status StableInterface: {0} Status Preview: {1} Status Deprecated: {2} Status Removed: {3}'.format(len(status['stableinterface']),
len(status['preview']), len(status['deprecated']), len(status['removed'])))
return 0
if __name__ == '__main__':
action, args = parse_args(sys.argv[1:])
if action == 'report':
rc = report(version=args['version'])
elif action == 'add':
rc = add_from_csv(args['csvfile'], version=args['version'], overwrite=args['overwrite'])
elif action == 'add-default':
rc = add_default(version=args['version'], overwrite=args['overwrite'])
elif action == 'upgrade':
rc = upgrade_metadata(version=args['version'])
sys.exit(rc)

23
hacking/report.py
View file

@ -9,7 +9,6 @@ import json
import os
import sqlite3
import sys
import yaml
DATABASE_PATH = os.path.expanduser('~/.ansible/report.db')
BASE_PATH = os.path.abspath(os.path.join(os.path.dirname(os.path.abspath(__file__)), '..')) + '/'
@ -81,7 +80,6 @@ def populate_modules():
module_dir = os.path.join(BASE_PATH, 'lib/ansible/modules/')
modules_rows = []
module_statuses_rows = []
for root, dir_names, file_names in os.walk(module_dir):
for file_name in file_names:
@ -99,29 +97,15 @@ def populate_modules():
result = read_docstring(path)
metadata = result['metadata']
doc = result['doc']
if not metadata:
if module == 'async_wrapper':
continue
raise Exception('no metadata for: %s' % path)
modules_rows.append(dict(
module=module,
namespace=namespace,
path=path.replace(BASE_PATH, ''),
supported_by=metadata['supported_by'],
version_added=str(doc.get('version_added', '')) if doc else '',
))
for status in metadata['status']:
module_statuses_rows.append(dict(
module=module,
status=status,
))
populate_data(dict(
modules=dict(
rows=modules_rows,
@ -129,15 +113,8 @@ def populate_modules():
('module', 'TEXT'),
('namespace', 'TEXT'),
('path', 'TEXT'),
('supported_by', 'TEXT'),
('version_added', 'TEXT'),
)),
module_statuses=dict(
rows=module_statuses_rows,
schema=(
('module', 'TEXT'),
('status', 'TEXT'),
)),
))

194
lib/ansible/cli/doc.py
View file

@ -24,7 +24,6 @@ from ansible.module_utils._text import to_native, to_text
from ansible.module_utils.common._collections_compat import Container, Sequence
from ansible.module_utils.common.json import AnsibleJSONEncoder
from ansible.module_utils.six import string_types
from ansible.parsing.metadata import extract_metadata
from ansible.parsing.plugin_docs import read_docstub
from ansible.parsing.yaml.dumper import AnsibleDumper
from ansible.plugins.loader import action_loader, fragment_loader
@ -54,10 +53,6 @@ def add_collection_plugins(plugin_list, plugin_type, coll_filter=None):
plugin_list.update(DocCLI.find_plugins(os.path.join(path, 'plugins', ptype), plugin_type, collection=collname))
class RemovedPlugin(Exception):
pass
class PluginNotFound(Exception):
pass
@ -112,12 +107,45 @@ class DocCLI(CLI):
return options
def display_plugin_list(self, results):
# format for user
displace = max(len(x) for x in self.plugin_list)
linelimit = display.columns - displace - 5
text = []
# format display per option
if context.CLIARGS['list_files']:
# list plugin file names
for plugin in results.keys():
filename = results[plugin]
text.append("%-*s %-*.*s" % (displace, plugin, linelimit, len(filename), filename))
else:
# list plugin names and short desc
deprecated = []
for plugin in results.keys():
desc = DocCLI.tty_ify(results[plugin])
if len(desc) > linelimit:
desc = desc[:linelimit] + '...'
if plugin.startswith('_'): # Handle deprecated # TODO: add mark for deprecated collection plugins
deprecated.append("%-*s %-*.*s" % (displace, plugin[1:], linelimit, len(desc), desc))
else:
text.append("%-*s %-*.*s" % (displace, plugin, linelimit, len(desc), desc))
if len(deprecated) > 0:
text.append("\nDEPRECATED:")
text.extend(deprecated)
# display results
DocCLI.pager("\n".join(text))
def run(self):
super(DocCLI, self).run()
plugin_type = context.CLIARGS['type']
do_json = context.CLIARGS['json_format']
if plugin_type in C.DOCUMENTABLE_PLUGINS:
@ -130,6 +158,7 @@ class DocCLI(CLI):
if basedir:
AnsibleCollectionConfig.playbook_paths = basedir
loader.add_directory(basedir, with_subdir=True)
if context.CLIARGS['module_path']:
for path in context.CLIARGS['module_path']:
if path:
@ -162,43 +191,10 @@ class DocCLI(CLI):
if do_json:
jdump(results)
elif self.plugin_list:
# format for user
displace = max(len(x) for x in self.plugin_list)
linelimit = display.columns - displace - 5
text = []
# format display per option
if context.CLIARGS['list_files']:
# list files
for plugin in results.keys():
filename = results[plugin]
text.append("%-*s %-*.*s" % (displace, plugin, linelimit, len(filename), filename))
else:
# list plugins
deprecated = []
for plugin in results.keys():
desc = DocCLI.tty_ify(results[plugin])
if len(desc) > linelimit:
desc = desc[:linelimit] + '...'
if plugin.startswith('_'): # Handle deprecated # TODO: add mark for deprecated collection plugins
deprecated.append("%-*s %-*.*s" % (displace, plugin[1:], linelimit, len(desc), desc))
else:
text.append("%-*s %-*.*s" % (displace, plugin, linelimit, len(desc), desc))
if len(deprecated) > 0:
text.append("\nDEPRECATED:")
text.extend(deprecated)
# display results
DocCLI.pager("\n".join(text))
self.display_plugin_list(results)
else:
display.warning("No plugins found.")
# dump plugin desc/metadata as JSON
# dump plugin desc/data as JSON
elif context.CLIARGS['dump']:
plugin_data = {}
plugin_names = DocCLI.get_all_plugins_of_type(plugin_type)
@ -208,7 +204,6 @@ class DocCLI(CLI):
plugin_data[plugin_name] = plugin_info
jdump(plugin_data)
else:
# display specific plugin docs
if len(context.CLIARGS['args']) == 0:
@ -222,9 +217,6 @@ class DocCLI(CLI):
except PluginNotFound:
display.warning("%s %s not found in:\n%s\n" % (plugin_type, plugin, search_paths))
continue
except RemovedPlugin:
display.warning("%s %s has been removed\n" % (plugin_type, plugin))
continue
except Exception as e:
display.vvv(traceback.format_exc())
raise AnsibleError("%s %s missing documentation (or could not parse"
@ -235,8 +227,7 @@ class DocCLI(CLI):
# The doc section existed but was empty
continue
plugin_docs[plugin] = {'doc': doc, 'examples': plainexamples,
'return': returndocs, 'metadata': metadata}
plugin_docs[plugin] = {'doc': doc, 'examples': plainexamples, 'return': returndocs, 'metadata': metadata}
if do_json:
# Some changes to how json docs are formatted
@ -257,6 +248,8 @@ class DocCLI(CLI):
doc_data['return'], doc_data['metadata'])
if textret:
text.append(textret)
else:
display.warning("No valid documentation was retrieved from '%s'" % plugin)
if text:
DocCLI.pager(''.join(text))
@ -286,20 +279,13 @@ class DocCLI(CLI):
collection_name = '.'.join(plugin_name.split('.')[1:3])
try:
doc, __, __, metadata = get_docstring(filename, fragment_loader, verbose=(context.CLIARGS['verbosity'] > 0),
collection_name=collection_name, is_module=(plugin_type == 'module'))
doc, __, __, __ = get_docstring(filename, fragment_loader, verbose=(context.CLIARGS['verbosity'] > 0),
collection_name=collection_name, is_module=(plugin_type == 'module'))
except Exception:
display.vvv(traceback.format_exc())
raise AnsibleError(
"%s %s at %s has a documentation error formatting or is missing documentation." %
(plugin_type, plugin_name, filename))
raise AnsibleError("%s %s at %s has a documentation formatting error or is missing documentation." % (plugin_type, plugin_name, filename))
if doc is None:
if 'removed' not in metadata.get('status', []):
raise AnsibleError(
"%s %s at %s has a documentation error formatting or is missing documentation." %
(plugin_type, plugin_name, filename))
# Removed plugins don't have any documentation
return None
@ -339,22 +325,8 @@ class DocCLI(CLI):
filename, fragment_loader, verbose=(context.CLIARGS['verbosity'] > 0),
collection_name=collection_name, is_module=(plugin_type == 'module'))
# If the plugin existed but did not have a DOCUMENTATION element and was not removed, it's
# an error
# If the plugin existed but did not have a DOCUMENTATION element and was not removed, it's an error
if doc is None:
# doc may be None when the module has been removed. Calling code may choose to
# handle that but we can't.
if 'status' in metadata and isinstance(metadata['status'], Container):
if 'removed' in metadata['status']:
raise RemovedPlugin('%s has been removed' % plugin)
# Backwards compat: no documentation but valid metadata (or no metadata, which results in using the default metadata).
# Probably should make this an error in 2.10
return {}, {}, {}, metadata
else:
# If metadata is invalid, warn but don't error
display.warning(u'%s has an invalid ANSIBLE_METADATA field' % plugin)
raise ValueError('%s did not contain a DOCUMENTATION attribute' % plugin)
doc['filename'] = filename
@ -383,7 +355,10 @@ class DocCLI(CLI):
if context.CLIARGS['show_snippet'] and plugin_type == 'module':
text = DocCLI.get_snippet_text(doc)
else:
text = DocCLI.get_man_text(doc)
try:
text = DocCLI.get_man_text(doc)
except Exception as e:
raise AnsibleError("Unable to retrieve documentation from '%s' due to: %s" % (plugin, to_native(e)))
return text
@ -450,13 +425,6 @@ class DocCLI(CLI):
continue
if not doc or not isinstance(doc, dict):
with open(filename) as f:
metadata = extract_metadata(module_data=f.read())
if metadata[0]:
if 'removed' not in metadata[0].get('status', []):
display.warning("%s parsing did not produce documentation." % plugin)
else:
continue
desc = 'UNDOCUMENTED'
else:
desc = doc.get('short_description', 'INVALID SHORT DESCRIPTION').strip()
@ -564,16 +532,16 @@ class DocCLI(CLI):
aliases = ''
if 'aliases' in opt:
if len(opt['aliases']) > 0:
aliases = "(Aliases: " + ", ".join(str(i) for i in opt['aliases']) + ")"
aliases = "(Aliases: " + ", ".join(to_text(i) for i in opt['aliases']) + ")"
del opt['aliases']
choices = ''
if 'choices' in opt:
if len(opt['choices']) > 0:
choices = "(Choices: " + ", ".join(str(i) for i in opt['choices']) + ")"
choices = "(Choices: " + ", ".join(to_text(i) for i in opt['choices']) + ")"
del opt['choices']
default = ''
if 'default' in opt or not required:
default = "[Default: %s" % str(opt.pop('default', '(null)')) + "]"
default = "[Default: %s" % to_text(opt.pop('default', '(null)')) + "]"
text.append(textwrap.fill(DocCLI.tty_ify(aliases + choices + default), limit,
initial_indent=opt_indent, subsequent_indent=opt_indent))
@ -612,31 +580,6 @@ class DocCLI(CLI):
text.append(DocCLI._dump_yaml({k: opt[k]}, opt_indent))
text.append('')
@staticmethod
def get_support_block(doc):
# Note: 'curated' is deprecated and not used in any of the modules we ship
support_level_msg = {'core': 'The Ansible Core Team',
'network': 'The Ansible Network Team',
'certified': 'an Ansible Partner',
'community': 'The Ansible Community',
'curated': 'A Third Party',
}
return [" * This module is maintained by %s" % support_level_msg[doc['metadata']['supported_by']]]
@staticmethod
def get_metadata_block(doc):
text = []
text.append("METADATA:")
text.append('\tSUPPORT LEVEL: %s' % doc['metadata']['supported_by'])
for k in (m for m in doc['metadata'] if m != 'supported_by'):
if isinstance(k, list):
text.append("\t%s: %s" % (k.capitalize(), ", ".join(doc['metadata'][k])))
else:
text.append("\t%s: %s" % (k.capitalize(), doc['metadata'][k]))
return text
@staticmethod
def get_man_text(doc):
@ -656,7 +599,7 @@ class DocCLI(CLI):
text.append("%s\n" % textwrap.fill(DocCLI.tty_ify(desc), limit, initial_indent=opt_indent,
subsequent_indent=opt_indent))
if 'deprecated' in doc and doc['deprecated'] is not None and len(doc['deprecated']) > 0:
if doc.get('deprecated', False):
text.append("DEPRECATED: \n")
if isinstance(doc['deprecated'], dict):
if 'version' in doc['deprecated'] and 'removed_in' not in doc['deprecated']:
@ -666,22 +609,15 @@ class DocCLI(CLI):
text.append("%s" % doc.pop('deprecated'))
text.append("\n")
try:
support_block = DocCLI.get_support_block(doc)
if support_block:
text.extend(support_block)
except Exception:
pass # FIXME: not suported by plugins
if doc.pop('action', False):
text.append(" * note: %s\n" % "This module has a corresponding action plugin.")
if 'options' in doc and doc['options']:
if doc.get('options', False):
text.append("OPTIONS (= is mandatory):\n")
DocCLI.add_fields(text, doc.pop('options'), limit, opt_indent)
text.append('')
if 'notes' in doc and doc['notes'] and len(doc['notes']) > 0:
if doc.get('notes', False):
text.append("NOTES:")
for note in doc['notes']:
text.append(textwrap.fill(DocCLI.tty_ify(note), limit - 6,
@ -690,7 +626,7 @@ class DocCLI(CLI):
text.append('')
del doc['notes']
if 'seealso' in doc and doc['seealso']:
if doc.get('seealso', False):
text.append("SEE ALSO:")
for item in doc['seealso']:
if 'module' in item:
@ -719,7 +655,7 @@ class DocCLI(CLI):
text.append('')
del doc['seealso']
if 'requirements' in doc and doc['requirements'] is not None and len(doc['requirements']) > 0:
if doc.get('requirements', False):
req = ", ".join(doc.pop('requirements'))
text.append("REQUIREMENTS:%s\n" % textwrap.fill(DocCLI.tty_ify(req), limit - 16, initial_indent=" ", subsequent_indent=opt_indent))
@ -732,11 +668,12 @@ class DocCLI(CLI):
elif isinstance(doc[k], (list, tuple)):
text.append('%s: %s' % (k.upper(), ', '.join(doc[k])))
else:
text.append(DocCLI._dump_yaml({k.upper(): doc[k]}, opt_indent))
# use empty indent since this affects the start of the yaml doc, not it's keys
text.append(DocCLI._dump_yaml({k.upper(): doc[k]}, ''))
del doc[k]
text.append('')
text.append('')
if 'plainexamples' in doc and doc['plainexamples'] is not None:
if doc.get('plainexamples', False):
text.append("EXAMPLES:")
text.append('')
if isinstance(doc['plainexamples'], string_types):
@ -746,20 +683,11 @@ class DocCLI(CLI):
text.append('')
text.append('')
if 'returndocs' in doc and doc['returndocs'] is not None:
if doc.get('returndocs', False):
text.append("RETURN VALUES:")
if isinstance(doc['returndocs'], string_types):
text.append(doc.pop('returndocs'))
else:
text.append(yaml.dump(doc.pop('returndocs'), indent=2, default_flow_style=False))
text.append('')
try:
metadata_block = DocCLI.get_metadata_block(doc)
if metadata_block:
text.extend(metadata_block)
text.append('')
except Exception:
pass # metadata is optional
return "\n".join(text)

5
lib/ansible/galaxy/data/network/library/example_command.py.j2
View file

@ -22,11 +22,6 @@ __metaclass__ = type
### Documentation
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['preview'],
'supported_by': 'community'}
DOCUMENTATION = """
Examples:
https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/network/iosxr/iosxr_command.py

5
lib/ansible/galaxy/data/network/library/example_config.py.j2
View file

@ -22,11 +22,6 @@ __metaclass__ = type
### Documentation
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['preview'],
'supported_by': 'community'}
DOCUMENTATION = """
Examples:
https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/network/iosxr/iosxr_config.py

5
lib/ansible/galaxy/data/network/library/example_facts.py.j2
View file

@ -22,11 +22,6 @@ __metaclass__ = type
### Documentation
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['preview'],
'supported_by': 'community'}
DOCUMENTATION = """
Examples:
https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/network/iosxr/iosxr_facts.py

3
lib/ansible/modules/add_host.py
View file

@ -7,9 +7,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'}
DOCUMENTATION = r'''
---

3
lib/ansible/modules/apt.py
View file

@ -10,9 +10,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'}
DOCUMENTATION = '''
---

3
lib/ansible/modules/apt_key.py
View file

@ -9,9 +9,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['preview'],
'supported_by': 'core'}
DOCUMENTATION = '''
---

3
lib/ansible/modules/apt_repository.py
View file

@ -10,9 +10,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['preview'],
'supported_by': 'core'}
DOCUMENTATION = '''
---

3
lib/ansible/modules/assemble.py
View file

@ -9,9 +9,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'}
DOCUMENTATION = r'''
---

3
lib/ansible/modules/assert.py
View file

@ -7,9 +7,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'}
DOCUMENTATION = r'''
---

3
lib/ansible/modules/async_status.py
View file

@ -7,9 +7,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'}
DOCUMENTATION = r'''
---

3
lib/ansible/modules/blockinfile.py
View file

@ -8,9 +8,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['preview'],
'supported_by': 'core'}
DOCUMENTATION = r'''
---

4
lib/ansible/modules/command.py
View file

@ -9,10 +9,6 @@ from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'}
DOCUMENTATION = r'''
---
module: command

3
lib/ansible/modules/copy.py
View file

@ -8,9 +8,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'}
DOCUMENTATION = r'''
---

3
lib/ansible/modules/cron.py
View file

@ -11,9 +11,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['preview'],
'supported_by': 'core'}
DOCUMENTATION = r'''
---

3
lib/ansible/modules/debconf.py
View file

@ -7,9 +7,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'}
DOCUMENTATION = r'''
---

3
lib/ansible/modules/debug.py
View file

@ -7,9 +7,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'}
DOCUMENTATION = r'''
---

5
lib/ansible/modules/dnf.py
View file

@ -11,11 +11,6 @@ from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'}
DOCUMENTATION = '''
---
module: dnf

5
lib/ansible/modules/dpkg_selections.py
View file

@ -8,11 +8,6 @@ from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['preview'],
'supported_by': 'core'}
DOCUMENTATION = '''
---
module: dpkg_selections

5
lib/ansible/modules/expect.py
View file

@ -8,11 +8,6 @@ from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['preview'],
'supported_by': 'core'}
DOCUMENTATION = r'''
---
module: expect

3
lib/ansible/modules/fail.py
View file

@ -7,9 +7,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'}
DOCUMENTATION = r'''
---

3
lib/ansible/modules/fetch.py
View file

@ -9,9 +9,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'}
DOCUMENTATION = r'''
---

3
lib/ansible/modules/file.py
View file

@ -8,9 +8,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'}
DOCUMENTATION = r'''
---

3
lib/ansible/modules/find.py
View file

@ -10,9 +10,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'}
DOCUMENTATION = r'''
---

5
lib/ansible/modules/gather_facts.py
View file

@ -7,11 +7,6 @@ from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['preview'],
'supported_by': 'core'}
DOCUMENTATION = '''
---
module: gather_facts

3
lib/ansible/modules/get_url.py
View file

@ -7,9 +7,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'}
DOCUMENTATION = r'''
---

3
lib/ansible/modules/getent.py
View file

@ -7,9 +7,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'}
DOCUMENTATION = '''
---

5
lib/ansible/modules/git.py
View file

@ -8,11 +8,6 @@ from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['preview'],
'supported_by': 'core'}
DOCUMENTATION = '''
---
module: git

3
lib/ansible/modules/group.py
View file

@ -7,9 +7,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'}
DOCUMENTATION = '''
---

3
lib/ansible/modules/group_by.py
View file

@ -7,9 +7,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'}
DOCUMENTATION = r'''
---

3
lib/ansible/modules/hostname.py
View file

@ -7,9 +7,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['preview'],
'supported_by': 'core'}
DOCUMENTATION = '''
---

5
lib/ansible/modules/import_playbook.py
View file

@ -7,11 +7,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {
'metadata_version': '1.1',
'status': ['preview'],
'supported_by': 'core'
}
DOCUMENTATION = r'''
---

5
lib/ansible/modules/import_role.py
View file

@ -6,11 +6,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {
'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'
}
DOCUMENTATION = r'''
---

5
lib/ansible/modules/import_tasks.py
View file

@ -7,11 +7,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {
'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'
}
DOCUMENTATION = r'''
---

5
lib/ansible/modules/include.py
View file

@ -7,11 +7,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {
'metadata_version': '1.1',
'status': ['preview'],
'supported_by': 'core'
}
DOCUMENTATION = r'''
---

5
lib/ansible/modules/include_role.py
View file

@ -7,11 +7,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {
'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'
}
DOCUMENTATION = r'''
---

5
lib/ansible/modules/include_tasks.py
View file

@ -7,11 +7,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {
'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'
}
DOCUMENTATION = r'''
---

5
lib/ansible/modules/include_vars.py
View file

@ -6,11 +6,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {
'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'
}
DOCUMENTATION = r'''
---

3
lib/ansible/modules/iptables.py
View file

@ -8,9 +8,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['preview'],
'supported_by': 'core'}
DOCUMENTATION = r'''
---

5
lib/ansible/modules/known_hosts.py
View file

@ -7,11 +7,6 @@ from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['preview'],
'supported_by': 'core'}
DOCUMENTATION = '''
---
module: known_hosts

4
lib/ansible/modules/lineinfile.py
View file

@ -9,10 +9,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['preview'],
'supported_by': 'core'}
DOCUMENTATION = r'''
---

3
lib/ansible/modules/meta.py
View file

@ -7,9 +7,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['preview'],
'supported_by': 'core'}
DOCUMENTATION = r'''
module: meta

5
lib/ansible/modules/package.py
View file

@ -9,11 +9,6 @@ from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'}
DOCUMENTATION = '''
---
module: package

3
lib/ansible/modules/package_facts.py
View file

@ -7,9 +7,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['preview'],
'supported_by': 'core'}
DOCUMENTATION = '''
module: package_facts

5
lib/ansible/modules/pause.py
View file

@ -6,11 +6,6 @@ from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'}
DOCUMENTATION = '''
---
module: pause

4
lib/ansible/modules/ping.py
View file

@ -9,10 +9,6 @@ from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'}
DOCUMENTATION = '''
---
module: ping

3
lib/ansible/modules/pip.py
View file

@ -7,9 +7,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['preview'],
'supported_by': 'core'}
DOCUMENTATION = '''
---

5
lib/ansible/modules/raw.py
View file

@ -7,11 +7,6 @@ from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'}
DOCUMENTATION = r'''
---
module: raw

3
lib/ansible/modules/reboot.py
View file

@ -6,9 +6,6 @@
from __future__ import (absolute_import, division, print_function)
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['preview'],
'supported_by': 'core'}
DOCUMENTATION = r'''
module: reboot

4
lib/ansible/modules/replace.py
View file

@ -8,10 +8,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'}
DOCUMENTATION = r'''
---

3
lib/ansible/modules/rpm_key.py
View file

@ -9,9 +9,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['preview'],
'supported_by': 'core'}
DOCUMENTATION = '''
---

3
lib/ansible/modules/script.py
View file

@ -4,9 +4,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'}
DOCUMENTATION = r'''
---

3
lib/ansible/modules/service.py
View file

@ -7,9 +7,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'}
DOCUMENTATION = r'''
---

4
lib/ansible/modules/service_facts.py
View file

@ -9,10 +9,6 @@ from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['preview'],
'supported_by': 'core'}
DOCUMENTATION = r'''
---
module: service_facts

3
lib/ansible/modules/set_fact.py
View file

@ -7,9 +7,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'}
DOCUMENTATION = r'''
---

3
lib/ansible/modules/set_stats.py
View file

@ -7,9 +7,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['preview'],
'supported_by': 'core'}
DOCUMENTATION = r'''
---

5
lib/ansible/modules/setup.py
View file

@ -8,11 +8,6 @@ from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'}
DOCUMENTATION = '''
---
module: setup

5
lib/ansible/modules/shell.py
View file

@ -12,11 +12,6 @@ from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'}
DOCUMENTATION = r'''
---
module: shell

4
lib/ansible/modules/slurp.py
View file

@ -8,10 +8,6 @@ from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'}
DOCUMENTATION = r'''
---
module: slurp

3
lib/ansible/modules/stat.py
View file

@ -6,9 +6,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'}
DOCUMENTATION = r'''
---

3
lib/ansible/modules/subversion.py
View file

@ -7,9 +7,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['preview'],
'supported_by': 'core'}
DOCUMENTATION = '''
---

3
lib/ansible/modules/systemd.py
View file

@ -7,9 +7,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'}
DOCUMENTATION = '''
module: systemd

5
lib/ansible/modules/sysvinit.py
View file

@ -8,11 +8,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {
'metadata_version': '1.1',
'status': ['preview'],
'supported_by': 'core'
}
DOCUMENTATION = '''
module: sysvinit

3
lib/ansible/modules/tempfile.py
View file

@ -8,9 +8,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['preview'],
'supported_by': 'core'}
DOCUMENTATION = '''
---

3
lib/ansible/modules/template.py
View file

@ -9,9 +9,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'}
DOCUMENTATION = r'''
---

3
lib/ansible/modules/unarchive.py
View file

@ -11,9 +11,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['preview'],
'supported_by': 'core'}
DOCUMENTATION = r'''
---

4
lib/ansible/modules/uri.py
View file

@ -8,10 +8,6 @@ from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'}
DOCUMENTATION = r'''
---
module: uri

3
lib/ansible/modules/user.py
View file

@ -7,9 +7,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'}
DOCUMENTATION = r'''
module: user

3
lib/ansible/modules/wait_for.py
View file

@ -7,9 +7,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'}
DOCUMENTATION = r'''
---

3
lib/ansible/modules/wait_for_connection.py
View file

@ -7,9 +7,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'}
DOCUMENTATION = r'''
---

3
lib/ansible/modules/yum.py
View file

@ -10,9 +10,6 @@
from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'}
DOCUMENTATION = '''
---

6
lib/ansible/modules/yum_repository.py
View file

@ -9,12 +9,6 @@ from __future__ import absolute_import, division, print_function
__metaclass__ = type
ANSIBLE_METADATA = {
'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'
}
DOCUMENTATION = '''
---
module: yum_repository

245
lib/ansible/parsing/metadata.py
View file

@ -1,245 +0,0 @@
# (c) 2017, Toshio Kuratomi <tkuratomi@ansible.com>
#
# This file is part of Ansible
#
# Ansible is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# Ansible is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with Ansible. If not, see <http://www.gnu.org/licenses/>.
# Make coding more python3-ish
from __future__ import (absolute_import, division, print_function)
__metaclass__ = type
import ast
import sys
import yaml
from ansible.module_utils._text import to_text
# There are currently defaults for all metadata fields so we can add it
# automatically if a file doesn't specify it
DEFAULT_METADATA = {'metadata_version': '1.1', 'status': ['preview'], 'supported_by': 'community'}
class ParseError(Exception):
"""Thrown when parsing a file fails"""
pass
def _seek_end_of_dict(module_data, start_line, start_col, next_node_line, next_node_col):
"""Look for the end of a dict in a set of lines
We know the starting position of the dict and we know the start of the
next code node but in between there may be multiple newlines and comments.
There may also be multiple python statements on the same line (separated
by semicolons)
Examples::
ANSIBLE_METADATA = {[..]}
DOCUMENTATION = [..]
ANSIBLE_METADATA = {[..]} # Optional comments with confusing junk => {}
# Optional comments {}
DOCUMENTATION = [..]
ANSIBLE_METADATA = {
[..]
}
# Optional comments {}
DOCUMENTATION = [..]
ANSIBLE_METADATA = {[..]} ; DOCUMENTATION = [..]
ANSIBLE_METADATA = {}EOF
"""
if next_node_line is None:
# The dict is the last statement in the file
snippet = module_data.splitlines()[start_line:]
next_node_col = 0
# Include the last line in the file
last_line_offset = 0
else:
# It's somewhere in the middle so we need to separate it from the rest
snippet = module_data.splitlines()[start_line:next_node_line]
# Do not include the last line because that's where the next node
# starts
last_line_offset = 1
if next_node_col == 0:
# This handles all variants where there are only comments and blank
# lines between the dict and the next code node
# Step backwards through all the lines in the snippet
for line_idx, line in tuple(reversed(tuple(enumerate(snippet))))[last_line_offset:]:
end_col = None
# Step backwards through all the characters in the line
for col_idx, char in reversed(tuple(enumerate(c for c in line))):
if not isinstance(char, bytes):
# Python3 wart. slicing a byte string yields integers
char = bytes((char,))
if char == b'}' and end_col is None:
# Potentially found the end of the dict
end_col = col_idx
elif char == b'#' and end_col is not None:
# The previous '}' was part of a comment. Keep trying
end_col = None
if end_col is not None:
# Found the end!
end_line = start_line + line_idx
break
else:
raise ParseError('Unable to find the end of dictionary')
else:
# Harder cases involving multiple statements on one line
# Good Ansible Module style doesn't do this so we're just going to
# treat this as an error for now:
raise ParseError('Multiple statements per line confuses the module metadata parser.')
return end_line, end_col
def _seek_end_of_string(module_data, start_line, start_col, next_node_line, next_node_col):
"""
This is much trickier than finding the end of a dict. A dict has only one
ending character, "}". Strings have four potential ending characters. We
have to parse the beginning of the string to determine what the ending
character will be.
Examples:
ANSIBLE_METADATA = '''[..]''' # Optional comment with confusing chars '''
# Optional comment with confusing chars '''
DOCUMENTATION = [..]
ANSIBLE_METADATA = '''
[..]
'''
DOCUMENTATIONS = [..]
ANSIBLE_METADATA = '''[..]''' ; DOCUMENTATION = [..]
SHORT_NAME = ANSIBLE_METADATA = '''[..]''' ; DOCUMENTATION = [..]
String marker variants:
* '[..]'
* "[..]"
* '''[..]'''
* \"\"\"[..]\"\"\"
Each of these come in u, r, and b variants:
* '[..]'
* u'[..]'
* b'[..]'
* r'[..]'
* ur'[..]'
* ru'[..]'
* br'[..]'
* b'[..]'
* rb'[..]'
"""
raise NotImplementedError('Finding end of string not yet implemented')
def extract_metadata(module_ast=None, module_data=None, offsets=False):
"""Extract the metadata from a module
:kwarg module_ast: ast representation of the module. At least one of this
or ``module_data`` must be given. If the code calling
:func:`extract_metadata` has already parsed the module_data into an ast,
giving the ast here will save reparsing it.
:kwarg module_data: Byte string containing a module's code. At least one
of this or ``module_ast`` must be given.
:kwarg offsets: If set to True, offests into the source code will be
returned. This requires that ``module_data`` be set.
:returns: a tuple of metadata (a dict), line the metadata starts on,
column the metadata starts on, line the metadata ends on, column the
metadata ends on, and the names the metadata is assigned to. One of
the names the metadata is assigned to will be ANSIBLE_METADATA. If no
metadata is found, the tuple will be (None, -1, -1, -1, -1, None).
If ``offsets`` is False then the tuple will consist of
(metadata, -1, -1, -1, -1, None).
:raises ansible.parsing.metadata.ParseError: if ``module_data`` does not parse
:raises SyntaxError: if ``module_data`` is needed but does not parse correctly
"""
if offsets and module_data is None:
raise TypeError('If offsets is True then module_data must also be given')
if module_ast is None and module_data is None:
raise TypeError('One of module_ast or module_data must be given')
metadata = None
start_line = -1
start_col = -1
end_line = -1
end_col = -1
targets = None
if module_ast is None:
module_ast = ast.parse(module_data)
for root_idx, child in reversed(list(enumerate(module_ast.body))):
if isinstance(child, ast.Assign):
for target in child.targets:
if isinstance(target, ast.Name) and target.id == 'ANSIBLE_METADATA':
metadata = ast.literal_eval(child.value)
if not offsets:
continue
try:
# Determine where the next node starts
next_node = module_ast.body[root_idx + 1]
next_lineno = next_node.lineno
next_col_offset = next_node.col_offset
except IndexError:
# Metadata is defined in the last node of the file
next_lineno = None
next_col_offset = None
if isinstance(child.value, ast.Dict):
# Determine where the current metadata ends
end_line, end_col = _seek_end_of_dict(module_data,
child.lineno - 1,
child.col_offset,
next_lineno,
next_col_offset)
elif isinstance(child.value, ast.Str):
metadata = yaml.safe_load(child.value.s)
end_line, end_col = _seek_end_of_string(module_data,
child.lineno - 1,
child.col_offset,
next_lineno,
next_col_offset)
elif isinstance(child.value, ast.Bytes):
metadata = yaml.safe_load(to_text(child.value.s, errors='surrogate_or_strict'))
end_line, end_col = _seek_end_of_string(module_data,
child.lineno - 1,
child.col_offset,
next_lineno,
next_col_offset)
else:
raise ParseError('Ansible plugin metadata must be a dict')
# Do these after the if-else so we don't pollute them in
# case this was a false positive
start_line = child.lineno - 1
start_col = child.col_offset
targets = [t.id for t in child.targets]
break
if metadata is not None:
# Once we've found the metadata we're done
break
return metadata, start_line, start_col, end_line, end_col, targets

39
lib/ansible/parsing/plugin_docs.py
View file

@ -5,17 +5,18 @@ from __future__ import (absolute_import, division, print_function)
__metaclass__ = type
import ast
import yaml
from ansible.module_utils._text import to_text
from ansible.parsing.metadata import extract_metadata
from ansible.parsing.yaml.loader import AnsibleLoader
from ansible.utils.display import Display
display = Display()
# NOTE: should move to just reading the variable as we do in plugin_loader since we already load as a 'module'
# which is much faster than ast parsing ourselves.
def read_docstring(filename, verbose=True, ignore_errors=True):
"""
Search for assignment of the DOCUMENTATION and EXAMPLES variables in the given file.
Parse DOCUMENTATION from YAML and return the YAML doc or None together with EXAMPLES, as plain text.
@ -25,7 +26,7 @@ def read_docstring(filename, verbose=True, ignore_errors=True):
'doc': None,
'plainexamples': None,
'returndocs': None,
'metadata': None,
'metadata': None, # NOTE: not used anymore, kept for compat
'seealso': None,
}
@ -33,6 +34,7 @@ def read_docstring(filename, verbose=True, ignore_errors=True):
'DOCUMENTATION': 'doc',
'EXAMPLES': 'plainexamples',
'RETURN': 'returndocs',
'ANSIBLE_METADATA': 'metadata', # NOTE: now unused, but kept for backwards compat
}
try:
@ -54,33 +56,16 @@ def read_docstring(filename, verbose=True, ignore_errors=True):
if isinstance(child.value, ast.Dict):
data[varkey] = ast.literal_eval(child.value)
else:
if theid == 'DOCUMENTATION':
# string should be yaml
data[varkey] = AnsibleLoader(child.value.s, file_name=filename).get_single_data()
else:
# not yaml, should be a simple string
if theid in ['EXAMPLES', 'RETURN']:
# examples 'can' be yaml, return must be, but even if so, we dont want to parse as such here
# as it can create undesired 'objects' that don't display well as docs.
data[varkey] = to_text(child.value.s)
display.debug('assigned :%s' % varkey)
else:
# string should be yaml if already not a dict
data[varkey] = AnsibleLoader(child.value.s, file_name=filename).get_single_data()
# Metadata is per-file and a dict rather than per-plugin/function and yaml
data['metadata'] = extract_metadata(module_ast=M)[0]
display.debug('assigned: %s' % varkey)
if data['metadata']:
# remove version
for field in ('version', 'metadata_version'):
if field in data['metadata']:
del data['metadata'][field]
if 'supported_by' not in data['metadata']:
data['metadata']['supported_by'] = 'community'
if 'status' not in data['metadata']:
data['metadata']['status'] = ['preview']
else:
# Add default metadata
data['metadata'] = {'supported_by': 'community',
'status': ['preview']}
except Exception:
if verbose:
display.error("unable to parse %s" % filename)

7
lib/ansible/plugins/filter/urlsplit.py
View file

@ -6,13 +6,6 @@ from __future__ import (absolute_import, division, print_function)
__metaclass__ = type
ANSIBLE_METADATA = {
'metadata_version': '1.1',
'status': ['preview'],
'supported_by': 'core'
}
from ansible.errors import AnsibleFilterError
from ansible.module_utils.six.moves.urllib.parse import urlsplit
from ansible.utils import helpers

4
lib/ansible/plugins/inventory/toml.py
View file

@ -4,10 +4,6 @@
from __future__ import (absolute_import, division, print_function)
__metaclass__ = type
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['preview'],
'supported_by': 'core'}
DOCUMENTATION = '''
inventory: toml
version_added: "2.8"

7
test/integration/targets/ansible-doc/fakemodule.output
View file

@ -2,7 +2,6 @@
this is a fake module
* This module is maintained by The Ansible Community
OPTIONS (= is mandatory):
- _notreal
@ -11,10 +10,6 @@ OPTIONS (= is mandatory):
AUTHOR: me
METADATA:
status:
- preview
supported_by: community
SHORT_DESCIPTOIN: fake module

35
test/integration/targets/ansible-doc/test.yml
View file

@ -49,6 +49,7 @@
- name: documented module with removed status
command: ansible-doc test_docs_removed_status
register: result
- assert:
that:
- '"WARNING" not in result.stderr'
@ -58,41 +59,17 @@
- name: empty module
command: ansible-doc test_empty
register: result
ignore_errors: true
- assert:
that:
- 'result.stdout == ""'
- 'result.stderr == ""'
- result is failed
- name: module with no documentation
command: ansible-doc test_no_docs
register: result
- assert:
that:
- 'result.stdout == ""'
- 'result.stderr == ""'
ignore_errors: true
- name: module with no documentation and no metadata
command: ansible-doc test_no_docs_no_metadata
register: result
- assert:
that:
- 'result.stdout == ""'
- 'result.stderr == ""'
- name: module with no documentation and no status in metadata
command: ansible-doc test_no_docs_no_status
ignore_errors: yes
register: result
- assert:
that:
- 'result.stdout == ""'
- 'result.stderr == ""'
- name: module with no documentation and non-iterable status in metadata
command: ansible-doc test_no_docs_non_iterable_status
ignore_errors: yes
register: result
- assert:
that:
- 'result is failed'
- '"ERROR! module test_no_docs_non_iterable_status missing documentation (or could not parse documentation): test_no_docs_non_iterable_status did not contain a DOCUMENTATION attribute" in result.stderr'
- result is failed

0
test/integration/targets/collections/inventory_cache/.keep
View file

142
test/lib/ansible_test/_data/sanity/validate-modules/validate_modules/main.py
View file

@ -41,7 +41,7 @@ import yaml
from ansible import __version__ as ansible_version
from ansible.executor.module_common import REPLACER_WINDOWS
from ansible.module_utils.common._collections_compat import Mapping
from ansible.module_utils._text import to_bytes, to_native
from ansible.module_utils._text import to_native
from ansible.plugins.loader import fragment_loader
from ansible.utils.collection_loader._collection_finder import _AnsibleCollectionFinder
from ansible.utils.plugin_docs import BLACKLIST, tag_versions_and_dates, add_fragments, get_docstring
@ -49,7 +49,7 @@ from ansible.utils.version import SemanticVersion
from .module_args import AnsibleModuleImportError, AnsibleModuleNotInitialized, get_argument_spec
from .schema import ansible_module_kwargs_schema, doc_schema, metadata_1_1_schema, return_schema
from .schema import ansible_module_kwargs_schema, doc_schema, return_schema
from .utils import CaptureStd, NoArgsAnsibleModule, compare_unordered_lists, is_empty, parse_yaml, parse_isodate
from voluptuous.humanize import humanize_error
@ -696,7 +696,7 @@ class ModuleValidator(Validator):
code='import-before-documentation',
msg=('Import found before documentation variables. '
'All imports must appear below '
'DOCUMENTATION/EXAMPLES/RETURN/ANSIBLE_METADATA.'),
'DOCUMENTATION/EXAMPLES/RETURN.'),
line=child.lineno
)
break
@ -713,8 +713,7 @@ class ModuleValidator(Validator):
code='import-before-documentation',
msg=('Import found before documentation '
'variables. All imports must appear below '
'DOCUMENTATION/EXAMPLES/RETURN/'
'ANSIBLE_METADATA.'),
'DOCUMENTATION/EXAMPLES/RETURN.'),
line=child.lineno
)
break
@ -724,7 +723,7 @@ class ModuleValidator(Validator):
msg = (
'import-placement',
('Imports should be directly below DOCUMENTATION/EXAMPLES/'
'RETURN/ANSIBLE_METADATA.')
'RETURN.')
)
if self._is_new_module():
self.reporter.error(
@ -829,11 +828,6 @@ class ModuleValidator(Validator):
'lineno': 0,
'end_lineno': 0,
},
'ANSIBLE_METADATA': {
'value': None,
'lineno': 0,
'end_lineno': 0,
}
}
for child in self.ast.body:
if isinstance(child, ast.Assign):
@ -859,17 +853,6 @@ class ModuleValidator(Validator):
docs['RETURN']['end_lineno'] = (
child.lineno + len(child.value.s.splitlines())
)
elif grandchild.id == 'ANSIBLE_METADATA':
docs['ANSIBLE_METADATA']['value'] = child.value
docs['ANSIBLE_METADATA']['lineno'] = child.lineno
try:
docs['ANSIBLE_METADATA']['end_lineno'] = (
child.lineno + len(child.value.s.splitlines())
)
except AttributeError:
docs['ANSIBLE_METADATA']['end_lineno'] = (
child.value.values[-1].lineno
)
return docs
@ -939,51 +922,13 @@ class ModuleValidator(Validator):
if self.object_name.startswith('_') and not os.path.islink(self.object_path):
filename_deprecated_or_removed = True
# Have to check the metadata first so that we know if the module is removed or deprecated
metadata = None
if not self.collection:
if not bool(doc_info['ANSIBLE_METADATA']['value']):
self.reporter.error(
path=self.object_path,
code='missing-metadata',
msg='No ANSIBLE_METADATA provided'
)
else:
if isinstance(doc_info['ANSIBLE_METADATA']['value'], ast.Dict):
metadata = ast.literal_eval(
doc_info['ANSIBLE_METADATA']['value']
)
else:
self.reporter.error(
path=self.object_path,
code='missing-metadata-format',
msg='ANSIBLE_METADATA was not provided as a dict, YAML not supported'
)
if metadata:
self._validate_docs_schema(metadata, metadata_1_1_schema(),
'ANSIBLE_METADATA', 'invalid-metadata-type')
# We could validate these via the schema if we knew what the values are ahead of
# time. We can figure that out for deprecated but we can't for removed. Only the
# metadata has that information.
if 'removed' in metadata['status']:
removed = True
if 'deprecated' in metadata['status']:
deprecated = True
if (deprecated or removed) and len(metadata['status']) > 1:
self.reporter.error(
path=self.object_path,
code='missing-metadata-status',
msg='ANSIBLE_METADATA.status must be exactly one of "deprecated" or "removed"'
)
else:
# We are testing a collection
if self.routing:
routing_deprecation = self.routing.get('plugin_routing', {}).get('modules', {}).get(self.name, {}).get('deprecation', {})
if routing_deprecation:
# meta/runtime.yml says this is deprecated
routing_says_deprecated = True
deprecated = True
# We are testing a collection
if self.routing:
routing_deprecation = self.routing.get('plugin_routing', {}).get('modules', {}).get(self.name, {}).get('deprecation', {})
if routing_deprecation:
# meta/runtime.yml says this is deprecated
routing_says_deprecated = True
deprecated = True
if not removed:
if not bool(doc_info['DOCUMENTATION']['value']):
@ -1080,7 +1025,7 @@ class ModuleValidator(Validator):
)
if not self.collection:
existing_doc = self._check_for_new_args(doc, metadata)
existing_doc = self._check_for_new_args(doc)
self._check_version_added(doc, existing_doc)
if not bool(doc_info['EXAMPLES']['value']):
@ -1152,7 +1097,7 @@ class ModuleValidator(Validator):
self.reporter.error(
path=self.object_path,
code='deprecation-mismatch',
msg='Module deprecation/removed must agree in Metadata, by prepending filename with'
msg='Module deprecation/removed must agree in documentaiton, by prepending filename with'
' "_", and setting DOCUMENTATION.deprecated for deprecation or by removing all'
' documentation for removed'
)
@ -2003,7 +1948,7 @@ class ModuleValidator(Validator):
msg=msg
)
def _check_for_new_args(self, doc, metadata):
def _check_for_new_args(self, doc):
if not self.base_branch or self._is_new_module():
return
@ -2038,16 +1983,6 @@ class ModuleValidator(Validator):
except ValueError:
mod_version_added = self._create_strict_version('0.0')
if self.base_branch and 'stable-' in self.base_branch:
metadata.pop('metadata_version', None)
metadata.pop('version', None)
if metadata != existing_metadata:
self.reporter.error(
path=self.object_path,
code='metadata-changed',
msg=('ANSIBLE_METADATA cannot be changed in a point release for a stable branch')
)
options = doc.get('options', {}) or {}
should_be = '.'.join(ansible_version.split('.')[:2])
@ -2146,24 +2081,10 @@ class ModuleValidator(Validator):
doc_info, docs = self._validate_docs()
# See if current version => deprecated.removed_in, ie, should be docs only
if isinstance(doc_info['ANSIBLE_METADATA']['value'], ast.Dict) and 'removed' in ast.literal_eval(doc_info['ANSIBLE_METADATA']['value'])['status']:
end_of_deprecation_should_be_removed_only = True
elif docs and 'deprecated' in docs and docs['deprecated'] is not None:
end_of_deprecation_should_be_removed_only = False
if 'removed_at_date' in docs['deprecated']:
try:
removed_at_date = docs['deprecated']['removed_at_date']
if parse_isodate(removed_at_date) < datetime.date.today():
msg = "Module's deprecated.removed_at_date date '%s' is before today" % removed_at_date
self.reporter.error(
path=self.object_path,
code='deprecated-date',
msg=msg,
)
except ValueError:
# Already checked during schema validation
pass
if docs and docs.get('deprecated', False):
if 'removed_in' in docs['deprecated']:
removed_in = None
try:
collection_name, version = self._split_tagged_version(docs['deprecated']['removed_in'])
if collection_name != self.collection_name:
@ -2172,20 +2093,31 @@ class ModuleValidator(Validator):
code='invalid-module-deprecation-source',
msg=('The deprecation version for a module must be added in this collection')
)
# Treat the module as not to be removed:
raise ValueError('')
removed_in = self._create_strict_version(str(version))
else:
removed_in = self.StrictVersion(str(version))
except ValueError:
end_of_deprecation_should_be_removed_only = False
else:
# ignore and hope we previouslly reported
pass
if removed_in:
if not self.collection:
strict_ansible_version = self._create_strict_version('.'.join(ansible_version.split('.')[:2]))
strict_ansible_version = self.StrictVersion('.'.join(ansible_version.split('.')[:2]))
end_of_deprecation_should_be_removed_only = strict_ansible_version >= removed_in
elif self.collection_version:
strict_ansible_version = self.collection_version
end_of_deprecation_should_be_removed_only = strict_ansible_version >= removed_in
else:
end_of_deprecation_should_be_removed_only = False
# handle deprecation by date
if 'removed_at_date' in docs['deprecated']:
try:
removed_at_date = docs['deprecated']['removed_at_date']
if parse_isodate(removed_at_date) < datetime.date.today():
msg = "Module's deprecated.removed_at_date date '%s' is before today" % removed_at_date
self.reporter.error(path=self.object_path, code='deprecated-date', msg=msg)
except ValueError:
# ignore and hope we previouslly reported
pass
if self._python_module() and not self._just_docs() and not end_of_deprecation_should_be_removed_only:
self._validate_ansible_module_call(docs)

26
test/lib/ansible_test/_data/sanity/validate-modules/validate_modules/schema.py
View file

@ -442,32 +442,6 @@ def doc_schema(module_name, for_collection=False, deprecated_module=False):
)
def metadata_1_0_schema(deprecated):
valid_status = Any('stableinterface', 'preview', 'deprecated', 'removed')
if deprecated:
valid_status = Any('deprecated')
return Schema(
{
Required('status'): [valid_status],
Required('metadata_version'): '1.0',
Required('supported_by'): Any('core', 'community', 'curated')
}
)
def metadata_1_1_schema():
valid_status = Any('stableinterface', 'preview', 'deprecated', 'removed')
return Schema(
{
Required('status'): [valid_status],
Required('metadata_version'): '1.1',
Required('supported_by'): Any('core', 'community', 'certified', 'network')
}
)
# Things to add soon
####################
# 1) Recursively validate `type: complex` fields

239
test/units/parsing/test_metadata.py
View file

@ -1,239 +0,0 @@
# coding: utf-8
# (c) 2017, Toshio Kuratomi <tkuratomi@ansible.com>
#
# This file is part of Ansible
#
# Ansible is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# Ansible is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with Ansible. If not, see <http://www.gnu.org/licenses/>.
# Make coding more python3-ish
from __future__ import (absolute_import, division, print_function)
__metaclass__ = type
import ast
import pytest
from ansible.parsing import metadata as md
LICENSE = b"""# some license text boilerplate
# That we have at the top of files
"""
FUTURE_IMPORTS = b"""
from __future__ import (absolute_import, division, print_function)
"""
REGULAR_IMPORTS = b"""
import test
from foo import bar
"""
STANDARD_METADATA = b"""
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'}
"""
TEXT_STD_METADATA = b"""
ANSIBLE_METADATA = u'''
metadata_version: '1.1'
status:
- 'stableinterface'
supported_by: 'core'
'''
"""
BYTES_STD_METADATA = b"""
ANSIBLE_METADATA = b'''
metadata_version: '1.1'
status:
- 'stableinterface'
supported_by: 'core'
'''
"""
TRAILING_COMMENT_METADATA = b"""
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'} # { Testing }
"""
MULTIPLE_STATEMENTS_METADATA = b"""
DOCUMENTATION = "" ; ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'} ; RETURNS = ""
"""
EMBEDDED_COMMENT_METADATA = b"""
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
# { Testing }
'supported_by': 'core'}
"""
HASH_SYMBOL_METADATA = b"""
ANSIBLE_METADATA = {'metadata_version': '1.1 # 4',
'status': ['stableinterface'],
'supported_by': 'core # Testing '}
"""
HASH_SYMBOL_METADATA = b"""
ANSIBLE_METADATA = {'metadata_version': '1.1 # 4',
'status': ['stableinterface'],
'supported_by': 'core # Testing '}
"""
HASH_COMBO_METADATA = b"""
ANSIBLE_METADATA = {'metadata_version': '1.1 # 4',
'status': ['stableinterface'],
# { Testing }
'supported_by': 'core'} # { Testing }
"""
METADATA = {'metadata_version': '1.1', 'status': ['stableinterface'], 'supported_by': 'core'}
HASH_SYMBOL_METADATA = {'metadata_version': '1.1 # 4', 'status': ['stableinterface'], 'supported_by': 'core'}
METADATA_EXAMPLES = (
# Standard import
(LICENSE + FUTURE_IMPORTS + STANDARD_METADATA + REGULAR_IMPORTS,
(METADATA, 5, 0, 7, 42, ['ANSIBLE_METADATA'])),
# Metadata at end of file
(LICENSE + FUTURE_IMPORTS + REGULAR_IMPORTS + STANDARD_METADATA.rstrip(),
(METADATA, 8, 0, 10, 42, ['ANSIBLE_METADATA'])),
# Metadata at beginning of file
(STANDARD_METADATA + LICENSE + REGULAR_IMPORTS,
(METADATA, 1, 0, 3, 42, ['ANSIBLE_METADATA'])),
# Standard import with a trailing comment
(LICENSE + FUTURE_IMPORTS + TRAILING_COMMENT_METADATA + REGULAR_IMPORTS,
(METADATA, 5, 0, 7, 42, ['ANSIBLE_METADATA'])),
# Metadata at end of file with a trailing comment
(LICENSE + FUTURE_IMPORTS + REGULAR_IMPORTS + TRAILING_COMMENT_METADATA.rstrip(),
(METADATA, 8, 0, 10, 42, ['ANSIBLE_METADATA'])),
# Metadata at beginning of file with a trailing comment
(TRAILING_COMMENT_METADATA + LICENSE + REGULAR_IMPORTS,
(METADATA, 1, 0, 3, 42, ['ANSIBLE_METADATA'])),
# FIXME: Current code cannot handle multiple statements on the same line.
# This is bad style so we're just going to ignore it for now
# Standard import with other statements on the same line
# (LICENSE + FUTURE_IMPORTS + MULTIPLE_STATEMENTS_METADATA + REGULAR_IMPORTS,
# (METADATA, 5, 0, 7, 42, ['ANSIBLE_METADATA'])),
# Metadata at end of file with other statements on the same line
# (LICENSE + FUTURE_IMPORTS + REGULAR_IMPORTS + MULTIPLE_STATEMENTS_METADATA.rstrip(),
# (METADATA, 8, 0, 10, 42, ['ANSIBLE_METADATA'])),
# Metadata at beginning of file with other statements on the same line
# (MULTIPLE_STATEMENTS_METADATA + LICENSE + REGULAR_IMPORTS,
# (METADATA, 1, 0, 3, 42, ['ANSIBLE_METADATA'])),
# Standard import with comment inside the metadata
(LICENSE + FUTURE_IMPORTS + EMBEDDED_COMMENT_METADATA + REGULAR_IMPORTS,
(METADATA, 5, 0, 8, 42, ['ANSIBLE_METADATA'])),
# Metadata at end of file with comment inside the metadata
(LICENSE + FUTURE_IMPORTS + REGULAR_IMPORTS + EMBEDDED_COMMENT_METADATA.rstrip(),
(METADATA, 8, 0, 11, 42, ['ANSIBLE_METADATA'])),
# Metadata at beginning of file with comment inside the metadata
(EMBEDDED_COMMENT_METADATA + LICENSE + REGULAR_IMPORTS,
(METADATA, 1, 0, 4, 42, ['ANSIBLE_METADATA'])),
# FIXME: Current code cannot handle hash symbols in the last element of
# the metadata. Fortunately, the metadata currently fully specifies all
# the strings inside of metadata and none of them can contain a hash.
# Need to fix this to future-proof it against strings containing hashes
# Standard import with hash symbol in metadata
# (LICENSE + FUTURE_IMPORTS + HASH_SYMBOL_METADATA + REGULAR_IMPORTS,
# (HASH_SYMBOL_METADATA, 5, 0, 7, 53, ['ANSIBLE_METADATA'])),
# Metadata at end of file with hash symbol in metadata
# (LICENSE + FUTURE_IMPORTS + REGULAR_IMPORTS + HASH_SYMBOL_HASH_SYMBOL_METADATA.rstrip(),
# (HASH_SYMBOL_METADATA, 8, 0, 10, 53, ['ANSIBLE_METADATA'])),
# Metadata at beginning of file with hash symbol in metadata
# (HASH_SYMBOL_HASH_SYMBOL_METADATA + LICENSE + REGULAR_IMPORTS,
# (HASH_SYMBOL_METADATA, 1, 0, 3, 53, ['ANSIBLE_METADATA'])),
# Standard import with a bunch of hashes everywhere
(LICENSE + FUTURE_IMPORTS + HASH_COMBO_METADATA + REGULAR_IMPORTS,
(HASH_SYMBOL_METADATA, 5, 0, 8, 42, ['ANSIBLE_METADATA'])),
# Metadata at end of file with a bunch of hashes everywhere
(LICENSE + FUTURE_IMPORTS + REGULAR_IMPORTS + HASH_COMBO_METADATA.rstrip(),
(HASH_SYMBOL_METADATA, 8, 0, 11, 42, ['ANSIBLE_METADATA'])),
# Metadata at beginning of file with a bunch of hashes everywhere
(HASH_COMBO_METADATA + LICENSE + REGULAR_IMPORTS,
(HASH_SYMBOL_METADATA, 1, 0, 4, 42, ['ANSIBLE_METADATA'])),
# Standard import with a junk ANSIBLE_METADATA as well
(LICENSE + FUTURE_IMPORTS + b"\nANSIBLE_METADATA = 10\n" + HASH_COMBO_METADATA + REGULAR_IMPORTS,
(HASH_SYMBOL_METADATA, 7, 0, 10, 42, ['ANSIBLE_METADATA'])),
)
# FIXME: String/yaml metadata is not implemented yet. Need more test cases once it is implemented
STRING_METADATA_EXAMPLES = (
# Standard import
(LICENSE + FUTURE_IMPORTS + TEXT_STD_METADATA + REGULAR_IMPORTS,
(METADATA, 5, 0, 10, 3, ['ANSIBLE_METADATA'])),
# Metadata at end of file
(LICENSE + FUTURE_IMPORTS + REGULAR_IMPORTS + TEXT_STD_METADATA.rstrip(),
(METADATA, 8, 0, 13, 3, ['ANSIBLE_METADATA'])),
# Metadata at beginning of file
(TEXT_STD_METADATA + LICENSE + REGULAR_IMPORTS,
(METADATA, 1, 0, 6, 3, ['ANSIBLE_METADATA'])),
# Standard import
(LICENSE + FUTURE_IMPORTS + BYTES_STD_METADATA + REGULAR_IMPORTS,
(METADATA, 5, 0, 10, 3, ['ANSIBLE_METADATA'])),
# Metadata at end of file
(LICENSE + FUTURE_IMPORTS + REGULAR_IMPORTS + BYTES_STD_METADATA.rstrip(),
(METADATA, 8, 0, 13, 3, ['ANSIBLE_METADATA'])),
# Metadata at beginning of file
(BYTES_STD_METADATA + LICENSE + REGULAR_IMPORTS,
(METADATA, 1, 0, 6, 3, ['ANSIBLE_METADATA'])),
)
@pytest.mark.parametrize("code, expected", METADATA_EXAMPLES)
def test_dict_metadata(code, expected):
assert md.extract_metadata(module_data=code, offsets=True) == expected
@pytest.mark.parametrize("code, expected", STRING_METADATA_EXAMPLES)
def test_string_metadata(code, expected):
# FIXME: String/yaml metadata is not implemented yet.
with pytest.raises(NotImplementedError):
assert md.extract_metadata(module_data=code, offsets=True) == expected
def test_required_params():
with pytest.raises(TypeError, match='One of module_ast or module_data must be given'):
assert md.extract_metadata()
def test_module_data_param_given_with_offset():
with pytest.raises(TypeError, match='If offsets is True then module_data must also be given'):
assert md.extract_metadata(module_ast='something', offsets=True)
def test_invalid_dict_metadata():
with pytest.raises(SyntaxError):
assert md.extract_metadata(module_data=LICENSE + FUTURE_IMPORTS + b'ANSIBLE_METADATA={"metadata_version": "1.1",\n' + REGULAR_IMPORTS)
with pytest.raises(md.ParseError, match='Unable to find the end of dictionary'):
assert md.extract_metadata(module_ast=ast.parse(LICENSE + FUTURE_IMPORTS + b'ANSIBLE_METADATA={"metadata_version": "1.1"}\n' + REGULAR_IMPORTS),
module_data=LICENSE + FUTURE_IMPORTS + b'ANSIBLE_METADATA={"metadata_version": "1.1",\n' + REGULAR_IMPORTS,
offsets=True)
def test_multiple_statements_limitation():
with pytest.raises(md.ParseError, match='Multiple statements per line confuses the module metadata parser.'):
assert md.extract_metadata(module_data=LICENSE + FUTURE_IMPORTS + b'ANSIBLE_METADATA={"metadata_version": "1.1"}; a=b\n' + REGULAR_IMPORTS,
offsets=True)