The Python3 dev doc is more general than just modules (#22950)

The Python3 dev doc is more general than just modules

* Rename it to make that obvious.
* Move generally applicable Python3 information to the Controller section
* Add a Py3/Py2 section on formatting strings
* Fix code-blocks to highlight as python
* Enhance python3 support page
* Add Python3/Python2 compat note
This commit is contained in:
Toshio Kuratomi 2017-04-04 11:18:19 -07:00 committed by GitHub
parent 6f40cb9647
commit 3180b4757e
6 changed files with 195 additions and 76 deletions

View file

@ -59,7 +59,7 @@ The following topics will discuss how to develop and work with modules:
Best practices, recommendations, and things to avoid. Best practices, recommendations, and things to avoid.
:doc:`developing_modules_checklist` :doc:`developing_modules_checklist`
Checklist for contributing your module to Ansible. Checklist for contributing your module to Ansible.
:doc:`developing_modules_python3` :doc:`developing_python3`
Adding Python 3 support to modules (all new modules must be Python-2.6 and Python-3.5 compatible). Adding Python 3 support to modules (all new modules must be Python-2.6 and Python-3.5 compatible).
:doc:`developing_modules_in_groups` :doc:`developing_modules_in_groups`
A guide for partners wanting to submit multiple modules. A guide for partners wanting to submit multiple modules.

View file

@ -23,7 +23,7 @@ The following checklist items are important guidelines for people who want to c
* The shebang must always be ``#!/usr/bin/python``. This allows ``ansible_python_interpreter`` to work * The shebang must always be ``#!/usr/bin/python``. This allows ``ansible_python_interpreter`` to work
* Modules must be written to support Python 2.6. If this is not possible, required minimum Python version and rationale should be explained in the requirements section in ``DOCUMENTATION``. In Ansible-2.3 the minimum requirement for modules was Python-2.4. * Modules must be written to support Python 2.6. If this is not possible, required minimum Python version and rationale should be explained in the requirements section in ``DOCUMENTATION``. In Ansible-2.3 the minimum requirement for modules was Python-2.4.
* Modules must be written to use proper Python-3 syntax. At some point in the future we'll come up with rules for running on Python-3 but we're not there yet. See :doc:`developing_modules_python3` for help on how to do this. * Modules must be written to use proper Python-3 syntax. At some point in the future we'll come up with rules for running on Python-3 but we're not there yet. See :doc:`developing_python3` for help on how to do this.
* Modules must have a metadata section. For the vast majority of new modules, * Modules must have a metadata section. For the vast majority of new modules,
the metadata should look exactly like this: the metadata should look exactly like this:

View file

@ -91,7 +91,7 @@ And that's it.
Before pushing your PR to GitHub it's a good idea to review the :doc:`developing_modules_checklist` again. Before pushing your PR to GitHub it's a good idea to review the :doc:`developing_modules_checklist` again.
After publishing your PR to https://github.com/ansible/ansible, a Shippable CI test should run within a few minutes. Check the results (at the end of the PR page) to ensure that it's passing (green). If it's not passing, inspect each of the results. Most of the errors should be self-explanatory and are often related to badly formatted documentation (see :doc:`../YAMLSyntax`) or code that isn't valid Python 2.6 or valid Python 3.5 (see :doc:`developing_modules_python3`). If you aren't sure what a Shippable test message means, copy it into the PR along with a comment and we will review. After publishing your PR to https://github.com/ansible/ansible, a Shippable CI test should run within a few minutes. Check the results (at the end of the PR page) to ensure that it's passing (green). If it's not passing, inspect each of the results. Most of the errors should be self-explanatory and are often related to badly formatted documentation (see :doc:`../YAMLSyntax`) or code that isn't valid Python 2.6 or valid Python 3.5 (see :doc:`developing_python3`). If you aren't sure what a Shippable test message means, copy it into the PR along with a comment and we will review.
If you need further advice, consider join the ``#ansible-devel`` IRC channel (see how in the "Where to get support"). If you need further advice, consider join the ``#ansible-devel`` IRC channel (see how in the "Where to get support").

View file

@ -1,12 +1,18 @@
=============================== ====================
Ansible and Porting to Python 3 Ansible and Python 3
=============================== ====================
Ansible is pursuing a strategy of having one code base that runs on both
Python-2 and Python-3 because we want Ansible to be able to manage a wide
variety of machines. Contributors to Ansible should be aware of the tips in
this document so that they can write code that will run on the same versions
of Python as the rest of Ansible.
Ansible can be divided into three overlapping pieces for the purposes of Ansible can be divided into three overlapping pieces for the purposes of
porting: porting:
1. Controller-side code. This is the code which runs on the machine where you 1. Controller-side code. This is the code which runs on the machine where you
invoke /usr/bin/ansible invoke :command:`/usr/bin/ansible`
2. Modules. This is the code which Ansible transmits over the wire and 2. Modules. This is the code which Ansible transmits over the wire and
invokes on the managed machine. invokes on the managed machine.
3. module_utils code. This is code whose primary purpose is to be used by the 3. module_utils code. This is code whose primary purpose is to be used by the
@ -15,6 +21,8 @@ porting:
Much of the knowledge of porting code will be usable on all three of these Much of the knowledge of porting code will be usable on all three of these
pieces but there are some special considerations for some of it as well. pieces but there are some special considerations for some of it as well.
Information that is generally applicable to all three places is located in the
controller-side section.
-------------------------------------------- --------------------------------------------
Minimum Version of Python-3.x and Python-2.x Minimum Version of Python-3.x and Python-2.x
@ -108,7 +116,9 @@ Reading and writing to files
In Python-2, reading from files yields bytes. In Python-3, it can yield text. In Python-2, reading from files yields bytes. In Python-3, it can yield text.
To make code that's portable to both we don't make use of Python-3's ability To make code that's portable to both we don't make use of Python-3's ability
to yield text but instead do the conversion explicitly ourselves. For example:: to yield text but instead do the conversion explicitly ourselves. For example:
.. code-block:: python
from ansible.module_utils._text import to_text from ansible.module_utils._text import to_text
@ -125,7 +135,9 @@ to yield text but instead do the conversion explicitly ourselves. For example::
point, if there is demand for other encodings we may change that, but for point, if there is demand for other encodings we may change that, but for
now it is safe to assume that bytes are UTF-8. now it is safe to assume that bytes are UTF-8.
Writing to files is the opposite process:: Writing to files is the opposite process:
.. code-block:: python
from ansible.module_utils._text import to_bytes from ansible.module_utils._text import to_bytes
@ -145,7 +157,9 @@ functions, the text string will be converted to a byte string inside of the
function and a traceback will occur if non-ASCII characters are present. In function and a traceback will occur if non-ASCII characters are present. In
Python-3, a traceback will only occur if the text string can't be decoded in Python-3, a traceback will only occur if the text string can't be decoded in
the current locale, but it's still good to be explicit and have code which the current locale, but it's still good to be explicit and have code which
works on both versions:: works on both versions:
.. code-block:: python
import os.path import os.path
@ -160,7 +174,9 @@ works on both versions::
When you are only manipulating a filename as a string without talking to the When you are only manipulating a filename as a string without talking to the
filesystem (or a C library which talks to the filesystem) you can often get filesystem (or a C library which talks to the filesystem) you can often get
away without converting to bytes:: away without converting to bytes:
.. code-block:: python
import os.path import os.path
@ -188,7 +204,7 @@ subprocess library and byte strings should be expected back from it.
One of the main places in Ansible's controller code that we interact with One of the main places in Ansible's controller code that we interact with
other programs is the connection plugins' ``exec_command`` methods. These other programs is the connection plugins' ``exec_command`` methods. These
methods transform any text strings they receive in the command (and arugments methods transform any text strings they receive in the command (and arguments
to the command) to execute into bytes and return stdout and stderr as byte strings to the command) to execute into bytes and return stdout and stderr as byte strings
Higher level functions (like action plugins' ``_low_level_execute_command``) Higher level functions (like action plugins' ``_low_level_execute_command``)
transform the output into text strings. transform the output into text strings.
@ -200,7 +216,9 @@ Forwards Compatibility Boilerplate
---------------------------------- ----------------------------------
Use the following boilerplate code at the top of all controller-side modules Use the following boilerplate code at the top of all controller-side modules
to make certain constructs act the same way on Python-2 and Python-3:: to make certain constructs act the same way on Python-2 and Python-3:
.. code-block:: python
# Make coding more python3-ish # Make coding more python3-ish
from __future__ import (absolute_import, division, print_function) from __future__ import (absolute_import, division, print_function)
@ -229,7 +247,9 @@ Prefix byte strings with "b\_"
Since mixing text and bytes types leads to tracebacks we want to be clear Since mixing text and bytes types leads to tracebacks we want to be clear
about what variables hold text and what variables hold bytes. We do this by about what variables hold text and what variables hold bytes. We do this by
prefixing any variable holding bytes with ``b_``. For instance:: prefixing any variable holding bytes with ``b_``. For instance:
.. code-block:: python
filename = u'/var/tmp/café.txt' filename = u'/var/tmp/café.txt'
b_filename = to_bytes(filename) b_filename = to_bytes(filename)
@ -240,6 +260,100 @@ We do not prefix the text strings instead because we only operate
on byte strings at the borders, so there are fewer variables that need bytes on byte strings at the borders, so there are fewer variables that need bytes
than text. than text.
Bundled six
-----------
The third-party `python-six <https://pythonhosted.org/six/>`_ library exists
to help projects create code that runs on both Python-2 and Python-3. Ansible
includes a version of the library in module_utils so that other modules can use it
without requiring that it is installed on the remote system. To make use of
it, import it like this:
.. code-block:: python
from ansible.module_utils import six
.. note:: Ansible can also use a system copy of six
Ansible will use a system copy of six if the system copy is a later
version than the one Ansible bundles.
Exceptions
----------
In order for code to function on Python-2.6+ and Python-3, use the
new exception-catching syntax which uses the ``as`` keyword:
.. code-block:: python
try:
a = 2/0
except ValueError as e:
module.fail_json(msg="Tried to divide by zero: %s" % e)
Do **not** use the following syntax as it will fail on every version of Python-3:
.. code-block:: python
try:
a = 2/0
except ValueError, e:
module.fail_json(msg="Tried to divide by zero: %s" % e)
Octal numbers
-------------
In Python-2.x, octal literals could be specified as ``0755``. In Python-3,
octals must be specified as ``0o755``.
String formatting
-----------------
str.format() compatibility
~~~~~~~~~~~~~~~~~~~~~~~~~~
Starting in Python-2.6, strings gained a method called ``format()`` to put
strings together. However, one commonly used feature of ``format()`` wasn't
added until Python-2.7, so you need to remember not to use it in Ansible code:
.. code-block:: python
# Does not work in Python-2.6!
new_string = "Dear {}, Welcome to {}".format(username, location)
# Use this instead
new_string = "Dear {0}, Welcome to {1}".format(username, location)
Both of the format strings above map positional arguments of the ``format()``
method into the string. However, the first version doesn't work in
Python-2.6. Always remember to put numbers into the placeholders so the code
is compatible with Python-2.6.
.. seealso::
Python documentation on `format strings <https://docs.python.org/2/library/string.html#formatstrings>`_
Use percent format with byte strings
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In Python-3.x, byte strings do not have a ``format()`` method. However, it
does have support for the older, percent-formatting.
.. code-block:: python
b_command_line = b'ansible-playbook --become-user %s -K %s' % (user, playbook_file)
.. note:: Percent formatting added in Python-3.5
Percent formatting of byte strings was added back into Python3 in 3.5.
This isn't a problem for us because Python-3.5 is our minimum version.
However, if you happen to be testing Ansible code with Python-3.4 or
earlier, you will find that the byte string formatting here won't work.
Upgrade to Python-3.5 to test.
.. seealso::
Python documentation on `percent formatting <https://docs.python.org/2/library/stdtypes.html#string-formatting>`_
--------------------------- ---------------------------
Porting Modules to Python 3 Porting Modules to Python 3
--------------------------- ---------------------------
@ -274,67 +388,42 @@ coded to expect bytes on Python-2 and text on Python-3.
Tips, tricks, and idioms to adopt Tips, tricks, and idioms to adopt
================================= =================================
Exceptions Python-2.4 Compatible Exception Syntax
---------- --------------------------------------
In order for code to function on Python-2.6+ and Python-3, use the Until Ansible-2.4, modules needed to be compatible with Python-2.4 as
new exception-catching syntax which uses the ``as`` keyword:: well. Python-2.4 did not understand the new exception-catching syntax so
we had to write a compatibility function that could work with both
Python-2 and Python-3. You may still see this used in some modules:
.. code-block:: python
from ansible.module_utils.pycompat24 import get_exception
[...]
try: try:
a = 2/0 a = 2/0
except ValueError as e: except ValueError:
e = get_exception()
module.fail_json(msg="Tried to divide by zero: %s" % e) module.fail_json(msg="Tried to divide by zero: %s" % e)
Unless a change is going to be backported to Ansible-2.3, you should not
have to use this in new code.
.. note:: Old exception syntax Python 2.4 octal workaround
---------------------------
Until Ansible-2.4, modules needed to be compatible with Python-2.4 as Before Ansible-2.4, modules had to be compatible with Python-2.4.
well. Python-2.4 did not understand the new exception-catching syntax so Python-2.4 did not understand the new syntax for octal literals so we used
we had to write a compatibility function that could work with both the following workaround to specify octal values:
Python-2 and Python-3. You may still see this used in some modules::
from ansible.module_utils.pycompat24 import get_exception .. code-block:: python
[...]
try: # Can't use 0755 on Python-3 and can't use 0o755 on Python-2.4
a = 2/0 EXECUTABLE_PERMS = int('0755', 8)
except ValueError:
e = get_exception()
module.fail_json(msg="Tried to divide by zero: %s" % e)
Unless a change is going to be backported to Ansible-2.3, you should not Unless a change is going to be backported to Ansible-2.3, you should not
have to use this in new code. have to use this in new code.
Octal numbers
-------------
In Python-2.6, octal literals could be specified as ``0755``. In Python-3, that is
invalid and octals must be specified as ``0o755``.
.. note:: Workaround from Python-2.4 era
Before Ansible-2.4, modules had to be compatible with Python-2.4.
Python-2.4 did not understand the new syntax for octal literals so we used
the following workaround to specify octal values::
# Can't use 0755 on Python-3 and can't use 0o755 on Python-2.4
EXECUTABLE_PERMS = int('0755', 8)
Bundled six
-----------
The third-party `python-six <https://pythonhosted.org/six/>`_ library exists
to help projects create code that runs on both Python-2 and Python-3. Ansible
includes a version of the library in module_utils so that other modules can use it
without requiring that it is installed on the remote system. To make use of
it, import it like this::
from ansible.module_utils import six
.. note:: Ansible can also use a system copy of six
Ansible will use a system copy of six if the system copy is a later
version than the one Ansible bundles.
------------------------------------- -------------------------------------
Porting module_utils code to Python 3 Porting module_utils code to Python 3

View file

@ -16,7 +16,7 @@ To get started, select one of the following topics.
overview_architecture overview_architecture
developing_modules developing_modules
developing_modules_python3 developing_python3
developing_plugins developing_plugins
developing_inventory developing_inventory
developing_api developing_api

View file

@ -15,26 +15,56 @@ Testing Python 3 with commands and playbooks
* `Install Ansible 2.2+ <http://docs.ansible.com/ansible/intro_installation.html>`_ * `Install Ansible 2.2+ <http://docs.ansible.com/ansible/intro_installation.html>`_
* To test Python 3 on the controller, run your ansible command via * To test Python 3 on the controller, run your ansible command via
`python3`. For example:: `python3`. For example:
.. code-block:: shell
$ python3 /usr/bin/ansible localhost -m ping python3 /usr/bin/ansible localhost -m ping
$ python3 /usr/bin/ansible-playbook sample-playbook.yml python3 /usr/bin/ansible-playbook sample-playbook.yml
Testing Python 3 module support Testing Python 3 module support
-------------------------------- --------------------------------
* Set the ansible_python_interpreter configuration option to * Set the ansible_python_interpreter configuration option to
`/usr/bin/python3`. The `ansible_python_interpreter` configuration option is usually set per-host as inventory variable associated with a host or set of hosts. See the `inventory documentation <http://docs.ansible.com/ansible/intro_inventory.html>`_ for more information. :command:`/usr/bin/python3`. The ``ansible_python_interpreter`` configuration option is
* Run your command or playbook.:: usually set per-host as an inventory variable associated with a host or group of hosts:
$ ansible localhost -m ping .. code-block:: ini
$ ansible-playbook sample-playbook.yml
# Example inventory that makes an alias for localhost that uses python3
[py3-hosts]
localhost-py3 ansible_host=localhost ansible_connection=local
[py3-hosts:vars]
ansible_python_interpreter=/usr/bin/python3
See the :ref:`inventory documentation <inventory>` for more information.
* Run your command or playbook:
.. code-block:: shell
ansible localhost-py3 -m ping
ansible-playbook sample-playbook.yml
Note that you can also use the `-e` command line option to manually set the python interpreter when you run a command. For example:: Note that you can also use the :option:`-e` command line option to manually
set the python interpreter when you run a command. For example:
$ ansible localhost -m ping -e 'ansible_python_interpreter=/usr/bin/python3' .. code-block:: shell
$ ansible-playbook sample-playbook.yml -e 'ansible_python_interpreter=/usr/bin/python3'
ansible localhost -m ping -e 'ansible_python_interpreter=/usr/bin/python3'
ansible-playbook sample-playbook.yml -e 'ansible_python_interpreter=/usr/bin/python3'
What to do if an incompatibility is found
-----------------------------------------
If you find a bug while testing modules with Python3 you can submit a bug
report on `Ansible's GitHUb project
<https://github.com/ansible/ansible/issues/>_`. Be sure to mention Python3 in
the bug report so that the right people look at it.
If you would like to fix the code and submit a pull request on github, you can
refer to :doc:`dev_guide/developing_python3` for information on how we fix
common Python3 compatibility issues in the Ansible codebase.