cmueller/ansible
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
ansible/hacking/build_library/build_ansible/command_plugins/docs_build.py

169 lines
6.8 KiB
Python
Raw Normal View History

Collections docs generation (#59761) * Build documentation for Ansible-2.10 (formerly known as ACD). Builds plugin docs from collections whose source is on galaxy The new command downloads collections from galaxy, then finds the plugins inside of them to get the documentation for those plugins. * Update the python syntax checks * docs builds can now require python 3.6+. * Move plugin formatter code out to an external tool, antsibull-docs. Collection owners want to be able to extract docs for their own websites as well. * The jinja2 filters, tests, and other support code have moved to antsibull * Remove document_plugins as that has now been integrated into antsibull-docs * Cleanup and bugfix to other build script code: * The Commands class needed to have its metaclass set for abstractmethod to work correctly * Fix lint issues in some command plugins * Add the docs/docsite/rst/collections to .gitignore as everything in that directory will be generated so we don't want any of it saved in the git repository * gitignore the build dir and remove edit docs link on module pages * Add docs/rst/collections as a directory to remove on make clean * Split the collections docs from the main docs * remove version and edit on github * remove version banner for just collections * clarify examples need collection keyword defined * Remove references to plugin documentation locations that no longer exist. * Perhaps the pages in plugins/*.rst should be deprecated altogether and their content moved? * If not, perhaps we want to rephrase and link into the collection documentation? * Or perhaps we want to link to the plugins which are present in collections/ansible/builtin? * Remove PYTHONPATH from the build-ansible calls One of the design goals of the build-ansible.py script was for it to automatically set its library path to include the checkout of ansible and the library of code to implement itself. Because it automatically includes the checkout of ansible, we don't need to set PYTHONPATH in the Makefile any longer. * Create a command to only build ansible-base plugin docs * When building docs for devel, only build the ansible-base docs for now. This is because antsibull needs support for building a "devel tree" of docs. This can be changed once that is implemented * When building docs for the sanity tests, only build the ansible-base plugin docs for now. Those are the docs which are in this repo so that seems appropriate for now.
2020-07-17 22:07:35 +02:00
# coding: utf-8
# Copyright: (c) 2020, Ansible Project
# GNU General Public License v3.0+ (see COPYING or https://www.gnu.org/licenses/gpl-3.0.txt)
# Make coding more python3-ish
from __future__ import absolute_import, division, print_function
import glob
import os
import os.path
import pathlib
import shutil
from tempfile import TemporaryDirectory
import yaml
from ansible.release import __version__ as ansible_base__version__
# Pylint doesn't understand Python3 namespace modules.
# pylint: disable=relative-beyond-top-level
from ..commands import Command
# pylint: enable=relative-beyond-top-level
__metaclass__ = type
DEFAULT_TOP_DIR = pathlib.Path(__file__).parents[4]
DEFAULT_OUTPUT_DIR = pathlib.Path(__file__).parents[4] / 'docs/docsite'
#
# Subcommand base
#
def generate_base_docs(args):
"""Regenerate the documentation for all plugins listed in the plugin_to_collection_file."""
# imports here so that they don't cause unnecessary deps for all of the plugins
from antsibull.cli import antsibull_docs
with TemporaryDirectory() as tmp_dir:
#
# Construct a deps file with our version of ansible_base in it
#
modified_deps_file = os.path.join(tmp_dir, 'ansible.deps')
Updates to docs build for new antsibull CLI * --ansible-base-cache renamed to --ansible-base-source * _acd_version in the .deps file has been renamed to _ansible_version
2020-08-03 20:15:25 +02:00
# The _ansible_version doesn't matter since we're only building docs for base
deps_file_contents = {'_ansible_version': ansible_base__version__,
Collections docs generation (#59761) * Build documentation for Ansible-2.10 (formerly known as ACD). Builds plugin docs from collections whose source is on galaxy The new command downloads collections from galaxy, then finds the plugins inside of them to get the documentation for those plugins. * Update the python syntax checks * docs builds can now require python 3.6+. * Move plugin formatter code out to an external tool, antsibull-docs. Collection owners want to be able to extract docs for their own websites as well. * The jinja2 filters, tests, and other support code have moved to antsibull * Remove document_plugins as that has now been integrated into antsibull-docs * Cleanup and bugfix to other build script code: * The Commands class needed to have its metaclass set for abstractmethod to work correctly * Fix lint issues in some command plugins * Add the docs/docsite/rst/collections to .gitignore as everything in that directory will be generated so we don't want any of it saved in the git repository * gitignore the build dir and remove edit docs link on module pages * Add docs/rst/collections as a directory to remove on make clean * Split the collections docs from the main docs * remove version and edit on github * remove version banner for just collections * clarify examples need collection keyword defined * Remove references to plugin documentation locations that no longer exist. * Perhaps the pages in plugins/*.rst should be deprecated altogether and their content moved? * If not, perhaps we want to rephrase and link into the collection documentation? * Or perhaps we want to link to the plugins which are present in collections/ansible/builtin? * Remove PYTHONPATH from the build-ansible calls One of the design goals of the build-ansible.py script was for it to automatically set its library path to include the checkout of ansible and the library of code to implement itself. Because it automatically includes the checkout of ansible, we don't need to set PYTHONPATH in the Makefile any longer. * Create a command to only build ansible-base plugin docs * When building docs for devel, only build the ansible-base docs for now. This is because antsibull needs support for building a "devel tree" of docs. This can be changed once that is implemented * When building docs for the sanity tests, only build the ansible-base plugin docs for now. Those are the docs which are in this repo so that seems appropriate for now.
2020-07-17 22:07:35 +02:00
'_ansible_base_version': ansible_base__version__}
with open(modified_deps_file, 'w') as f:
f.write(yaml.dump(deps_file_contents))
# Generate the plugin rst
Need to return any error code from running antsibull-docs (#70763) This way we fail early if there's a problem
2020-07-20 23:05:01 +02:00
return antsibull_docs.run(['antsibull-docs', 'stable', '--deps-file', modified_deps_file,
Updates to docs build for new antsibull CLI * --ansible-base-cache renamed to --ansible-base-source * _acd_version in the .deps file has been renamed to _ansible_version
2020-08-03 20:15:25 +02:00
'--ansible-base-source', str(args.top_dir),
Need to return any error code from running antsibull-docs (#70763) This way we fail early if there's a problem
2020-07-20 23:05:01 +02:00
'--dest-dir', args.output_dir])
Collections docs generation (#59761) * Build documentation for Ansible-2.10 (formerly known as ACD). Builds plugin docs from collections whose source is on galaxy The new command downloads collections from galaxy, then finds the plugins inside of them to get the documentation for those plugins. * Update the python syntax checks * docs builds can now require python 3.6+. * Move plugin formatter code out to an external tool, antsibull-docs. Collection owners want to be able to extract docs for their own websites as well. * The jinja2 filters, tests, and other support code have moved to antsibull * Remove document_plugins as that has now been integrated into antsibull-docs * Cleanup and bugfix to other build script code: * The Commands class needed to have its metaclass set for abstractmethod to work correctly * Fix lint issues in some command plugins * Add the docs/docsite/rst/collections to .gitignore as everything in that directory will be generated so we don't want any of it saved in the git repository * gitignore the build dir and remove edit docs link on module pages * Add docs/rst/collections as a directory to remove on make clean * Split the collections docs from the main docs * remove version and edit on github * remove version banner for just collections * clarify examples need collection keyword defined * Remove references to plugin documentation locations that no longer exist. * Perhaps the pages in plugins/*.rst should be deprecated altogether and their content moved? * If not, perhaps we want to rephrase and link into the collection documentation? * Or perhaps we want to link to the plugins which are present in collections/ansible/builtin? * Remove PYTHONPATH from the build-ansible calls One of the design goals of the build-ansible.py script was for it to automatically set its library path to include the checkout of ansible and the library of code to implement itself. Because it automatically includes the checkout of ansible, we don't need to set PYTHONPATH in the Makefile any longer. * Create a command to only build ansible-base plugin docs * When building docs for devel, only build the ansible-base docs for now. This is because antsibull needs support for building a "devel tree" of docs. This can be changed once that is implemented * When building docs for the sanity tests, only build the ansible-base plugin docs for now. Those are the docs which are in this repo so that seems appropriate for now.
2020-07-17 22:07:35 +02:00
# If we make this more than just a driver for antsibull:
# Run other rst generation
# Run sphinx build
#
# Subcommand full
#
def generate_full_docs(args):
"""Regenerate the documentation for all plugins listed in the plugin_to_collection_file."""
# imports here so that they don't cause unnecessary deps for all of the plugins
import sh
from antsibull.cli import antsibull_docs
from packaging.version import Version
ansible_base_ver = Version(ansible_base__version__)
ansible_base_major_ver = '{0}.{1}'.format(ansible_base_ver.major, ansible_base_ver.minor)
with TemporaryDirectory() as tmp_dir:
sh.git(['clone', 'https://github.com/ansible-community/ansible-build-data'], _cwd=tmp_dir)
Fix version comparison for determining what ansible to build against * The version comparison for determining what ansible package to build docs against was comparing the version number for ansible-base but it needed to check the version number for ansible instead * add a comment about some bad logic than needs to be fixed after 2.10.0
2020-08-07 04:11:29 +02:00
# This is wrong. Once ansible and ansible-base major.minor versions get out of sync this
# will stop working. We probably need to walk all subdirectories in reverse version order
# looking for the latest ansible version which uses something compatible with
# ansible_base_major_ver.
Collections docs generation (#59761) * Build documentation for Ansible-2.10 (formerly known as ACD). Builds plugin docs from collections whose source is on galaxy The new command downloads collections from galaxy, then finds the plugins inside of them to get the documentation for those plugins. * Update the python syntax checks * docs builds can now require python 3.6+. * Move plugin formatter code out to an external tool, antsibull-docs. Collection owners want to be able to extract docs for their own websites as well. * The jinja2 filters, tests, and other support code have moved to antsibull * Remove document_plugins as that has now been integrated into antsibull-docs * Cleanup and bugfix to other build script code: * The Commands class needed to have its metaclass set for abstractmethod to work correctly * Fix lint issues in some command plugins * Add the docs/docsite/rst/collections to .gitignore as everything in that directory will be generated so we don't want any of it saved in the git repository * gitignore the build dir and remove edit docs link on module pages * Add docs/rst/collections as a directory to remove on make clean * Split the collections docs from the main docs * remove version and edit on github * remove version banner for just collections * clarify examples need collection keyword defined * Remove references to plugin documentation locations that no longer exist. * Perhaps the pages in plugins/*.rst should be deprecated altogether and their content moved? * If not, perhaps we want to rephrase and link into the collection documentation? * Or perhaps we want to link to the plugins which are present in collections/ansible/builtin? * Remove PYTHONPATH from the build-ansible calls One of the design goals of the build-ansible.py script was for it to automatically set its library path to include the checkout of ansible and the library of code to implement itself. Because it automatically includes the checkout of ansible, we don't need to set PYTHONPATH in the Makefile any longer. * Create a command to only build ansible-base plugin docs * When building docs for devel, only build the ansible-base docs for now. This is because antsibull needs support for building a "devel tree" of docs. This can be changed once that is implemented * When building docs for the sanity tests, only build the ansible-base plugin docs for now. Those are the docs which are in this repo so that seems appropriate for now.
2020-07-17 22:07:35 +02:00
deps_files = glob.glob(os.path.join(tmp_dir, 'ansible-build-data',
ansible_base_major_ver, '*.deps'))
if not deps_files:
raise Exception('No deps files exist for version {0}'.format(ansible_base_major_ver))
# Find the latest version of the deps file for this version
latest = None
latest_ver = Version('0')
for filename in deps_files:
with open(filename, 'r') as f:
deps_data = yaml.safe_load(f.read())
Fix version comparison for determining what ansible to build against * The version comparison for determining what ansible package to build docs against was comparing the version number for ansible-base but it needed to check the version number for ansible instead * add a comment about some bad logic than needs to be fixed after 2.10.0
2020-08-07 04:11:29 +02:00
new_version = Version(deps_data['_ansible_version'])
Collections docs generation (#59761) * Build documentation for Ansible-2.10 (formerly known as ACD). Builds plugin docs from collections whose source is on galaxy The new command downloads collections from galaxy, then finds the plugins inside of them to get the documentation for those plugins. * Update the python syntax checks * docs builds can now require python 3.6+. * Move plugin formatter code out to an external tool, antsibull-docs. Collection owners want to be able to extract docs for their own websites as well. * The jinja2 filters, tests, and other support code have moved to antsibull * Remove document_plugins as that has now been integrated into antsibull-docs * Cleanup and bugfix to other build script code: * The Commands class needed to have its metaclass set for abstractmethod to work correctly * Fix lint issues in some command plugins * Add the docs/docsite/rst/collections to .gitignore as everything in that directory will be generated so we don't want any of it saved in the git repository * gitignore the build dir and remove edit docs link on module pages * Add docs/rst/collections as a directory to remove on make clean * Split the collections docs from the main docs * remove version and edit on github * remove version banner for just collections * clarify examples need collection keyword defined * Remove references to plugin documentation locations that no longer exist. * Perhaps the pages in plugins/*.rst should be deprecated altogether and their content moved? * If not, perhaps we want to rephrase and link into the collection documentation? * Or perhaps we want to link to the plugins which are present in collections/ansible/builtin? * Remove PYTHONPATH from the build-ansible calls One of the design goals of the build-ansible.py script was for it to automatically set its library path to include the checkout of ansible and the library of code to implement itself. Because it automatically includes the checkout of ansible, we don't need to set PYTHONPATH in the Makefile any longer. * Create a command to only build ansible-base plugin docs * When building docs for devel, only build the ansible-base docs for now. This is because antsibull needs support for building a "devel tree" of docs. This can be changed once that is implemented * When building docs for the sanity tests, only build the ansible-base plugin docs for now. Those are the docs which are in this repo so that seems appropriate for now.
2020-07-17 22:07:35 +02:00
if new_version > latest_ver:
latest_ver = new_version
latest = filename
# Make a copy of the deps file so that we can set the ansible-base version to use
modified_deps_file = os.path.join(tmp_dir, 'ansible.deps')
shutil.copyfile(latest, modified_deps_file)
# Put our version of ansible-base into the deps file
with open(modified_deps_file, 'r') as f:
deps_data = yaml.safe_load(f.read())
deps_data['_ansible_base_version'] = ansible_base__version__
with open(modified_deps_file, 'w') as f:
f.write(yaml.dump(deps_data))
# Generate the plugin rst
Need to return any error code from running antsibull-docs (#70763) This way we fail early if there's a problem
2020-07-20 23:05:01 +02:00
return antsibull_docs.run(['antsibull-docs', 'stable', '--deps-file', modified_deps_file,
Updates to docs build for new antsibull CLI * --ansible-base-cache renamed to --ansible-base-source * _acd_version in the .deps file has been renamed to _ansible_version
2020-08-03 20:15:25 +02:00
'--ansible-base-source', str(args.top_dir),
Need to return any error code from running antsibull-docs (#70763) This way we fail early if there's a problem
2020-07-20 23:05:01 +02:00
'--dest-dir', args.output_dir])
Collections docs generation (#59761) * Build documentation for Ansible-2.10 (formerly known as ACD). Builds plugin docs from collections whose source is on galaxy The new command downloads collections from galaxy, then finds the plugins inside of them to get the documentation for those plugins. * Update the python syntax checks * docs builds can now require python 3.6+. * Move plugin formatter code out to an external tool, antsibull-docs. Collection owners want to be able to extract docs for their own websites as well. * The jinja2 filters, tests, and other support code have moved to antsibull * Remove document_plugins as that has now been integrated into antsibull-docs * Cleanup and bugfix to other build script code: * The Commands class needed to have its metaclass set for abstractmethod to work correctly * Fix lint issues in some command plugins * Add the docs/docsite/rst/collections to .gitignore as everything in that directory will be generated so we don't want any of it saved in the git repository * gitignore the build dir and remove edit docs link on module pages * Add docs/rst/collections as a directory to remove on make clean * Split the collections docs from the main docs * remove version and edit on github * remove version banner for just collections * clarify examples need collection keyword defined * Remove references to plugin documentation locations that no longer exist. * Perhaps the pages in plugins/*.rst should be deprecated altogether and their content moved? * If not, perhaps we want to rephrase and link into the collection documentation? * Or perhaps we want to link to the plugins which are present in collections/ansible/builtin? * Remove PYTHONPATH from the build-ansible calls One of the design goals of the build-ansible.py script was for it to automatically set its library path to include the checkout of ansible and the library of code to implement itself. Because it automatically includes the checkout of ansible, we don't need to set PYTHONPATH in the Makefile any longer. * Create a command to only build ansible-base plugin docs * When building docs for devel, only build the ansible-base docs for now. This is because antsibull needs support for building a "devel tree" of docs. This can be changed once that is implemented * When building docs for the sanity tests, only build the ansible-base plugin docs for now. Those are the docs which are in this repo so that seems appropriate for now.
2020-07-17 22:07:35 +02:00
# If we make this more than just a driver for antsibull:
# Run other rst generation
# Run sphinx build
class CollectionPluginDocs(Command):
name = 'docs-build'
_ACTION_HELP = """Action to perform.
full: Regenerate the rst for the full ansible website.
base: Regenerate the rst for plugins in ansible-base and then build the website.
named: Regenerate the rst for the named plugins and then build the website.
"""
@classmethod
def init_parser(cls, add_parser):
parser = add_parser(cls.name,
description='Generate documentation for plugins in collections.'
' Plugins in collections will have a stub file in the normal plugin'
' documentation location that says the module is in a collection and'
' point to generated plugin documentation under the collections/'
' hierarchy.')
parser.add_argument('action', action='store', choices=('full', 'base', 'named'),
default='full', help=cls._ACTION_HELP)
parser.add_argument("-o", "--output-dir", action="store", dest="output_dir",
default=DEFAULT_OUTPUT_DIR,
help="Output directory for generated doc files")
parser.add_argument("-t", "--top-dir", action="store", dest="top_dir",
default=DEFAULT_TOP_DIR,
help="Toplevel directory of this ansible-base checkout or expanded"
" tarball.")
parser.add_argument("-l", "--limit-to-modules", '--limit-to', action="store",
dest="limit_to", default=None,
help="Limit building module documentation to comma-separated list of"
" plugins. Specify non-existing plugin name for no plugins.")
@staticmethod
def main(args):
# normalize CLI args
if not args.output_dir:
args.output_dir = os.path.abspath(str(DEFAULT_OUTPUT_DIR))
if args.action == 'full':
return generate_full_docs(args)
if args.action == 'base':
return generate_base_docs(args)
# args.action == 'named' (Invalid actions are caught by argparse)
raise NotImplementedError('Building docs for specific files is not yet implemented')
# return 0
Reference in a new issue Copy permalink