cmueller/ansible
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

metadata 1.1

Browse source 
* Add network value to support_by field.
* New support_by value, certified
* Deprecate curated in favor of certified
* Add conversion from 1.0 to 1.1 to metadata-tool
* Add supported by Red Hat field to ansible-doc output
This commit is contained in:
Toshio Kuratomi 2017-08-05 11:28:21 -07:00
parent d50d65d448
commit af2073d057
19 changed files with 175 additions and 71 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

3
docs/bin/plugin_formatter.py
View file

@ -255,9 +255,6 @@ def process_module(module, options, env, template, outputname, module_map, alias
if doc is None:
sys.exit("*** ERROR: MODULE MISSING DOCUMENTATION: %s, %s ***\n" % (fname, module))
if metadata is None:
sys.exit("*** ERROR: MODULE MISSING METADATA: %s, %s ***\n" % (fname, module))
if deprecated and 'deprecated' not in doc:
sys.exit("*** ERROR: DEPRECATED MODULE MISSING 'deprecated' DOCUMENTATION: %s, %s ***\n" % (fname, module))

2
docs/docsite/rst/dev_guide/developing_modules_checklist.rst
View file

@ -31,7 +31,7 @@ The following checklist items are important guidelines for people who want to c
ANSIBLE_METADATA = {'status': ['preview'],
'supported_by': 'community',
'metadata_version': '1.0'}
'metadata_version': '1.1'}
The complete module metadata specification is here: `Ansible metadata block <https://docs.ansible.com/ansible/dev_guide/developing_modules_documenting.html#ansible-metadata-block>`_

28
docs/docsite/rst/dev_guide/developing_modules_documenting.rst
View file

@ -75,7 +75,7 @@ For new modules, the following block can be simply added into your module
.. code-block:: python
ANSIBLE_METADATA = {'metadata_version': '1.0',
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['preview'],
'supported_by': 'community'}
@ -92,7 +92,7 @@ For new modules, the following block can be simply added into your module
ANSIBLE_METADATA doesn't look quite right because of this. Module
metadata should be fixed before checking it into the repository.
Version 1.0 of the metadata
Version 1.1 of the metadata
+++++++++++++++++++++++++++
Structure
@ -101,7 +101,7 @@ Structure
.. code-block:: python
ANSIBLE_METADATA = {
'metadata_version': '1.0',
'metadata_version': '1.1',
'supported_by': 'community',
'status': ['preview', 'deprecated']
}
@ -115,12 +115,16 @@ Fields
of the metadata. We’ll increment Y if we add fields or legal values
to an existing field. We’ll increment X if we remove fields or values
or change the type or meaning of a field.
Current metadata_version is "1.1"
:supported_by: This field records who supports the module.
Default value is ``community``. Values are:
:core:
:curated:
:community:
* core
* network
* certfied
* community
* curated (Deprecated. Modules in this category should probably be core or
certified instead)
For information on what the support level values entail, please see
`Modules Support <http://docs.ansible.com/ansible/modules_support.html>`_.
@ -142,6 +146,18 @@ Fields
kept so that documentation can be built. The documentation helps
users port from the removed module to new modules.
Changes from Version 1.0
++++++++++++++++++++++++
:metadata_version: Version updated from 1.0 to 1.1
:supported_by: All substantive changes were to potential values of the supported_by field
* Added the certified value
* Deprecated the curated value, modules shipped with Ansible will use
certified instead. Third party modules are encouraged not to use this as
it is meaningless within Ansible proper.
* Added the network value
DOCUMENTATION Block
-------------------

4
docs/docsite/rst/dev_guide/developing_modules_general.rst
View file

@ -56,9 +56,9 @@ working on a whole new file. Here is an example:
#!/usr/bin/python
ANSIBLE_METADATA = {
'metadata_version': '1.0',
'metadata_version': '1.1',
'status': ['preview'],
'supported_by': 'curated'
'supported_by': 'community'
}
DOCUMENTATION = '''

20
docs/docsite/rst/dev_guide/developing_modules_general_OLD.rst
View file

@ -88,9 +88,9 @@ working on a whole new file. Here is an example:
#!/usr/bin/python
ANSIBLE_METADATA = {
'metadata_version': '1.0',
'metadata_version': '1.1',
'status': ['preview'],
'supported_by': 'curated'
'supported_by': 'community'
}
DOCUMENTATION = '''
@ -401,9 +401,9 @@ working on a whole new file. Here is an example:
#!/usr/bin/python
ANSIBLE_METADATA = {
'metadata_version': '1.0',
'metadata_version': '1.1',
'status': ['preview'],
'supported_by': 'curated'
'supported_by': 'community'
}
DOCUMENTATION = '''
@ -714,9 +714,9 @@ working on a whole new file. Here is an example:
#!/usr/bin/python
ANSIBLE_METADATA = {
'metadata_version': '1.0',
'metadata_version': '1.1',
'status': ['preview'],
'supported_by': 'curated'
'supported_by': 'community'
}
DOCUMENTATION = '''
@ -1027,9 +1027,9 @@ working on a whole new file. Here is an example:
#!/usr/bin/python
ANSIBLE_METADATA = {
'metadata_version': '1.0',
'metadata_version': '1.1',
'status': ['preview'],
'supported_by': 'curated'
'supported_by': 'community'
}
DOCUMENTATION = '''
@ -1340,9 +1340,9 @@ working on a whole new file. Here is an example:
#!/usr/bin/python
ANSIBLE_METADATA = {
'metadata_version': '1.0',
'metadata_version': '1.1',
'status': ['preview'],
'supported_by': 'curated'
'supported_by': 'community'
}
DOCUMENTATION = '''

14
docs/docsite/rst/modules_support.rst
View file

@ -18,15 +18,21 @@ The modules are hosted on GitHub in a subdirectory of the `ansible <https://gith
Core
````
These are modules that the core ansible team maintains and will always ship with ansible itself.
These are modules that the Ansible Core Team maintains and will always ship with Ansible itself.
They will also receive slightly higher priority for all requests. Non-core modules are still fully usable.
Curated
Network
```````
Some examples of Curated modules are submitted by other companies or maintained by the community. Maintainers of these types of modules must watch for any issues reported or pull requests raised against the module.
These modules are supported by the Ansible Network Team in a relationship
similar to how the Ansible Core Team maintains the Core modules.
Core Committers will review all modules becoming Curated. Core Committers will review proposed changes to existing Curated modules once the community maintainers of the module have approved the changes. Core committers will also ensure that any issues that arise due to Ansible engine changes will be remediated.
Certified
`````````
Some examples of Certified modules are those submitted by other companies. Maintainers of these types of modules must watch for any issues reported or pull requests raised against the module.
Core Committers will review all modules becoming Certified. Core Committers will review proposed changes to existing Certified modules once the community maintainers of the module have approved the changes. Core committers will also ensure that any issues that arise due to Ansible engine changes will be remediated.
Also, it is strongly recommended (but not presently required) for these types of modules to have unit tests.
These modules are currently shipped with Ansible, but might be shipped separately in the future.

8
docs/templates/plugin.rst.j2 vendored
View file

@ -207,7 +207,7 @@ Notes
{% endif %}
{% if not deprecated %}
{% set support = { 'core': 'This module is maintained by those with core commit privileges', 'curated': 'This module is supported mainly by the community and is curated by core committers.', 'community': 'This module is community maintained without core committer oversight.'} %}
{% set support = { 'core': 'The Ansible Core Team', 'network': 'The Ansible Network Team', 'certified': 'an Ansible Partner', 'community': 'The Ansible Community', 'curated': 'A Third Party'} %}
{% set module_states = { 'preview': 'it is not guaranteed to have a backwards compatible interface', 'stableinterface': 'the maintainers for this module guarantee that no backward incompatible interface changes will be made'} %}
{% if metadata %}
@ -223,10 +223,10 @@ This module is flagged as **@{cur_state}@** which means that @{module_states[cur
{% if metadata.supported_by %}
Support
~~~~~~~
Support Level
~~~~~~~~~~~~~
@{ support[metadata.supported_by] }@
This module is maintained by @{ support[metadata.supported_by] }@
For more information on what this means please read :doc:`modules_support`

57
hacking/metadata-tool.py
View file

@ -28,17 +28,14 @@ from collections import defaultdict
from distutils.version import StrictVersion
from pprint import pformat, pprint
from ansible.parsing.metadata import ParseError, extract_metadata
from ansible.plugins import module_loader
from ansible.parsing.metadata import DEFAULT_METADATA, ParseError, extract_metadata
from ansible.plugins.loader import module_loader
# There's a few files that are not new-style modules. Have to blacklist them
NONMODULE_PY_FILES = frozenset(('async_wrapper.py',))
NONMODULE_MODULE_NAMES = frozenset(os.path.splitext(p)[0] for p in NONMODULE_PY_FILES)
# Default metadata
DEFAULT_METADATA = {'metadata_version': '1.0', 'status': ['preview'], 'supported_by': 'community'}
class MissingModuleError(Exception):
"""Thrown when unable to find a plugin"""
@ -123,7 +120,7 @@ def insert_metadata(module_data, new_metadata, insertion_line, targets=('ANSIBLE
new_lines.append('{}{}'.format(' ' * (len(assignments) - 1 + len(' = {')), line))
old_lines = module_data.split('\n')
lines = old_lines[:insertion_line] + new_lines + [''] + old_lines[insertion_line:]
lines = old_lines[:insertion_line] + new_lines + old_lines[insertion_line:]
return '\n'.join(lines)
@ -174,13 +171,13 @@ def parse_assigned_metadata(csvfile):
Fields:
:0: Module name
:1: supported_by string. One of the valid support fields
core, community, curated
core, community, certified, network
:2: stableinterface
:3: preview
:4: deprecated
:5: removed
https://github.com/ansible/proposals/issues/30
http://docs.ansible.com/ansible/latest/dev_guide/developing_modules_documenting.html#ansible-metadata-block
"""
with open(csvfile, 'rb') as f:
for record in csv.reader(f):
@ -197,7 +194,7 @@ def parse_assigned_metadata(csvfile):
if not status or record[3]:
status.append('preview')
yield (module, {'metadata_version': '1.0', 'supported_by': supported_by, 'status': status})
yield (module, {'metadata_version': '1.1', 'supported_by': supported_by, 'status': status})
def write_metadata(filename, new_metadata, version=None, overwrite=False):
@ -330,6 +327,31 @@ def convert_metadata_pre_1_0_to_1_0(metadata):
return new_metadata
def convert_metadata_1_0_to_1_1(metadata):
"""
Convert 1.0 to 1.1 metadata format
:arg metadata: The old metadata
:returns: The new metadata
Changes from 1.0 to 1.1:
* ``supported_by`` field value ``curated`` has been removed
* ``supported_by`` field value ``certified`` has been added
* ``supported_by`` field value ``network`` has been added
"""
new_metadata = {'metadata_version': '1.1',
'supported_by': metadata['supported_by'],
'status': metadata['status']
}
if new_metadata['supported_by'] == 'unmaintained':
new_metadata['supported_by'] = 'community'
elif new_metadata['supported_by'] == 'curated':
new_metadata['supported_by'] = 'certified'
return new_metadata
# Subcommands
@ -338,7 +360,7 @@ def add_from_csv(csv_file, version=None, overwrite=False):
"""
# Add metadata for everything from the CSV file
diagnostic_messages = []
for module_name, new_metadata in parse_assigned_metadata_initial(csv_file):
for module_name, new_metadata in parse_assigned_metadata(csv_file):
filename = module_loader.find_plugin(module_name, mod_type='.py')
if filename is None:
diagnostic_messages.append('Unable to find the module file for {}'.format(module_name))
@ -424,7 +446,10 @@ def upgrade_metadata(version=None):
metadata = convert_metadata_pre_1_0_to_1_0(metadata)
if metadata['metadata_version'] == '1.0' and StrictVersion('1.0') < requested_version:
# 1.0 version => XXX. We don't yet have anything beyond 1.0
metadata = convert_metadata_1_0_to_1_1(metadata)
if metadata['metadata_version'] == '1.1' and StrictVersion('1.1') < requested_version:
# 1.1 version => XXX. We don't yet have anything beyond 1.1
# so there's nothing here
pass
@ -469,8 +494,10 @@ def report(version=None):
print('== Supported by core ==')
pprint(sorted(support['core']))
print('== Supported by value curated ==')
pprint(sorted(support['curated']))
print('== Supported by value certified ==')
pprint(sorted(support['certified']))
print('== Supported by value network ==')
pprint(sorted(support['network']))
print('== Supported by community ==')
pprint(sorted(support['community']))
print('')
@ -487,8 +514,8 @@ def report(version=None):
print('== Summary ==')
print('No Metadata: {0} Has Metadata: {1}'.format(len(no_metadata), len(has_metadata)))
print('Supported by core: {0} Supported by community: {1} Supported by value curated: {2}'.format(len(support['core']),
len(support['community']), len(support['curated'])))
print('Support level: core: {0} community: {1} certified: {2} network: {3}'.format(len(support['core']),
len(support['community']), len(support['certified']), len(support['network'])))
print('Status StableInterface: {0} Status Preview: {1} Status Deprecated: {2} Status Removed: {3}'.format(len(status['stableinterface']),
len(status['preview']), len(status['deprecated']), len(status['removed'])))

39
lib/ansible/cli/doc.py
View file

@ -361,6 +361,36 @@ class DocCLI(CLI):
text.append(self._dump_yaml({k: opt[k]}, opt_indent))
text.append('')
@staticmethod
def get_support_block(doc):
# Note: 'curated' is deprecated and not used in any of the modules we ship
support_level_msg = {'core': 'The Ansible Core Team',
'network': 'The Ansible Network Team',
'certified': 'an Ansible Partner',
'community': 'The Ansible Community',
'curated': 'A Third Party',
}
if doc['metadata'].get('metadata_version') in ('1.0', '1.1'):
return [" * This module is maintained by %s" % support_level_msg[doc['metadata']['supported_by']]]
return []
@staticmethod
def get_metadata_block(doc):
text = []
if doc['metadata'].get('metadata_version') in ('1.0', '1.1'):
text.append("METADATA:")
text.append('\tSUPPORT LEVEL: %s' % doc['metadata']['supported_by'])
for k in (m for m in doc['metadata'] if m not in ('version', 'metadata_version', 'supported_by')):
if isinstance(k, list):
text.append("\t%s: %s" % (k.capitalize(), ", ".join(doc['metadata'][k])))
else:
text.append("\t%s: %s" % (k.capitalize(), doc['metadata'][k]))
return text
return []
def get_man_text(self, doc):
IGNORE = frozenset(['module', 'docuri', 'version_added', 'short_description', 'now_date'])
@ -381,6 +411,10 @@ class DocCLI(CLI):
if 'deprecated' in doc and doc['deprecated'] is not None and len(doc['deprecated']) > 0:
text.append("DEPRECATED: \n%s\n" % doc.pop('deprecated'))
support_block = self.get_support_block(doc)
if support_block:
text.extend(support_block)
if doc.pop('action', False):
text.append(" * note: %s\n" % "This module has a corresponding action plugin.")
@ -435,4 +469,9 @@ class DocCLI(CLI):
text.append(self._dump_yaml({k.upper(): doc[k]}, opt_indent))
text.append('')
metadata_block = self.get_metadata_block(doc)
if metadata_block:
text.extend(metadata_block)
text.append('')
return "\n".join(text)

5
lib/ansible/parsing/metadata.py
View file

@ -27,6 +27,11 @@ import yaml
from ansible.module_utils._text import to_text
# There are currently defaults for all metadata fields so we can add it
# automatically if a file doesn't specify it
DEFAULT_METADATA = {'metadata_version': '1.1', 'status': ['preview'], 'supported_by': 'community'}
class ParseError(Exception):
"""Thrown when parsing a file fails"""
pass

2
test/integration/targets/module_precedence/lib_no_extension/ping
View file

@ -20,7 +20,7 @@
# along with Ansible. If not, see <http://www.gnu.org/licenses/>.
ANSIBLE_METADATA = {'metadata_version': '1.0',
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'}

2
test/integration/targets/module_precedence/lib_with_extension/ping.py
View file

@ -20,7 +20,7 @@
# along with Ansible. If not, see <http://www.gnu.org/licenses/>.
ANSIBLE_METADATA = {'metadata_version': '1.0',
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'}

2
test/integration/targets/module_precedence/multiple_roles/bar/library/ping.py
View file

@ -20,7 +20,7 @@
# along with Ansible. If not, see <http://www.gnu.org/licenses/>.
ANSIBLE_METADATA = {'metadata_version': '1.0',
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'}

2
test/integration/targets/module_precedence/multiple_roles/foo/library/ping.py
View file

@ -20,7 +20,7 @@
# along with Ansible. If not, see <http://www.gnu.org/licenses/>.
ANSIBLE_METADATA = {'metadata_version': '1.0',
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'}

2
test/integration/targets/module_precedence/roles_no_extension/foo/library/ping
View file

@ -20,7 +20,7 @@
# along with Ansible. If not, see <http://www.gnu.org/licenses/>.
ANSIBLE_METADATA = {'metadata_version': '1.0',
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'}

2
test/integration/targets/module_precedence/roles_with_extension/foo/library/ping.py
View file

@ -20,7 +20,7 @@
# along with Ansible. If not, see <http://www.gnu.org/licenses/>.
ANSIBLE_METADATA = {'metadata_version': '1.0',
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'}

4
test/sanity/validate-modules/main.py
View file

@ -41,7 +41,7 @@ from ansible.utils.plugin_docs import BLACKLIST, get_docstring
from module_args import get_argument_spec
from schema import doc_schema, metadata_schema, return_schema
from schema import doc_schema, metadata_1_1_schema, return_schema
from utils import CaptureStd, parse_yaml
from voluptuous.humanize import humanize_error
@ -892,7 +892,7 @@ class ModuleValidator(Validator):
)
if metadata:
self._validate_docs_schema(metadata, metadata_schema(deprecated),
self._validate_docs_schema(metadata, metadata_1_1_schema(deprecated),
'ANSIBLE_METADATA', 316)
return doc_info

16
test/sanity/validate-modules/schema.py
View file

@ -104,7 +104,7 @@ def doc_schema(module_name):
)
def metadata_schema(deprecated):
def metadata_1_0_schema(deprecated):
valid_status = Any('stableinterface', 'preview', 'deprecated', 'removed')
if deprecated:
valid_status = Any('deprecated')
@ -118,6 +118,20 @@ def metadata_schema(deprecated):
)
def metadata_1_1_schema(deprecated):
valid_status = Any('stableinterface', 'preview', 'deprecated', 'removed')
if deprecated:
valid_status = Any('deprecated')
return Schema(
{
Required('status'): [valid_status],
Required('metadata_version'): '1.1',
Required('supported_by'): Any('core', 'community', 'certified', 'network')
}
)
# Things to add soon
####################
# 1) Recursively validate `type: complex` fields

30
test/units/parsing/test_metadata.py
View file

@ -41,14 +41,14 @@ from foo import bar
"""
STANDARD_METADATA = b"""
ANSIBLE_METADATA = {'metadata_version': '1.0',
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'}
"""
TEXT_STD_METADATA = b"""
ANSIBLE_METADATA = u'''
metadata_version: '1.0'
metadata_version: '1.1'
status:
- 'stableinterface'
supported_by: 'core'
@ -57,7 +57,7 @@ supported_by: 'core'
BYTES_STD_METADATA = b"""
ANSIBLE_METADATA = b'''
metadata_version: '1.0'
metadata_version: '1.1'
status:
- 'stableinterface'
supported_by: 'core'
@ -65,45 +65,45 @@ supported_by: 'core'
"""
TRAILING_COMMENT_METADATA = b"""
ANSIBLE_METADATA = {'metadata_version': '1.0',
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'} # { Testing }
"""
MULTIPLE_STATEMENTS_METADATA = b"""
DOCUMENTATION = "" ; ANSIBLE_METADATA = {'metadata_version': '1.0',
DOCUMENTATION = "" ; ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
'supported_by': 'core'} ; RETURNS = ""
"""
EMBEDDED_COMMENT_METADATA = b"""
ANSIBLE_METADATA = {'metadata_version': '1.0',
ANSIBLE_METADATA = {'metadata_version': '1.1',
'status': ['stableinterface'],
# { Testing }
'supported_by': 'core'}
"""
HASH_SYMBOL_METADATA = b"""
ANSIBLE_METADATA = {'metadata_version': '1.0 # 4',
ANSIBLE_METADATA = {'metadata_version': '1.1 # 4',
'status': ['stableinterface'],
'supported_by': 'core # Testing '}
"""
HASH_SYMBOL_METADATA = b"""
ANSIBLE_METADATA = {'metadata_version': '1.0 # 4',
ANSIBLE_METADATA = {'metadata_version': '1.1 # 4',
'status': ['stableinterface'],
'supported_by': 'core # Testing '}
"""
HASH_COMBO_METADATA = b"""
ANSIBLE_METADATA = {'metadata_version': '1.0 # 4',
ANSIBLE_METADATA = {'metadata_version': '1.1 # 4',
'status': ['stableinterface'],
# { Testing }
'supported_by': 'core'} # { Testing }
"""
METADATA = {'metadata_version': '1.0', 'status': ['stableinterface'], 'supported_by': 'core'}
HASH_SYMBOL_METADATA = {'metadata_version': '1.0 # 4', 'status': ['stableinterface'], 'supported_by': 'core'}
METADATA = {'metadata_version': '1.1', 'status': ['stableinterface'], 'supported_by': 'core'}
HASH_SYMBOL_METADATA = {'metadata_version': '1.1 # 4', 'status': ['stableinterface'], 'supported_by': 'core'}
METADATA_EXAMPLES = (
# Standard import
@ -225,15 +225,15 @@ def test_module_data_param_given_with_offset():
def test_invalid_dict_metadata():
with pytest.raises(SyntaxError):
assert md.extract_metadata(module_data=LICENSE + FUTURE_IMPORTS + b'ANSIBLE_METADATA={"metadata_version": "1.0",\n' + REGULAR_IMPORTS)
assert md.extract_metadata(module_data=LICENSE + FUTURE_IMPORTS + b'ANSIBLE_METADATA={"metadata_version": "1.1",\n' + REGULAR_IMPORTS)
with pytest.raises(md.ParseError, message='Unable to find the end of dictionary'):
assert md.extract_metadata(module_ast=ast.parse(LICENSE + FUTURE_IMPORTS + b'ANSIBLE_METADATA={"metadata_version": "1.0"}\n' + REGULAR_IMPORTS),
module_data=LICENSE + FUTURE_IMPORTS + b'ANSIBLE_METADATA={"metadata_version": "1.0",\n' + REGULAR_IMPORTS,
assert md.extract_metadata(module_ast=ast.parse(LICENSE + FUTURE_IMPORTS + b'ANSIBLE_METADATA={"metadata_version": "1.1"}\n' + REGULAR_IMPORTS),
module_data=LICENSE + FUTURE_IMPORTS + b'ANSIBLE_METADATA={"metadata_version": "1.1",\n' + REGULAR_IMPORTS,
offsets=True)
def test_multiple_statements_limitation():
with pytest.raises(md.ParseError, message='Multiple statements per line confuses the module metadata parser.'):
assert md.extract_metadata(module_data=LICENSE + FUTURE_IMPORTS + b'ANSIBLE_METADATA={"metadata_version": "1.0"}; a=b\n' + REGULAR_IMPORTS,
assert md.extract_metadata(module_data=LICENSE + FUTURE_IMPORTS + b'ANSIBLE_METADATA={"metadata_version": "1.1"}; a=b\n' + REGULAR_IMPORTS,
offsets=True)