cmueller/ansible
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[MERGE] AWS module Guidelines - multiple changes to bring up to date and start on aws.core (#26871)

Browse source 
* aws module Guidelines - rewrite - add AWS WG - add fail_json_aws - more modern
practices and fix some spelling

* aws module Guidelines - minor further changes with agressive fill to 99 width

* aws module Guidelines - formatting fix as per gundalow request

* aws module Guidelines - mark more keywords and code with backquotes

* AWS module guidelines - fixes from ryansb review + minor other changes
This commit is contained in:
mikedlr 2017-08-23 15:34:30 +01:00 committed by ansibot
parent 8ed0222eff
commit f93515adb9
1 changed files with 159 additions and 86 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

245
lib/ansible/modules/cloud/amazon/GUIDELINES.md
View file

@ -1,36 +1,46 @@
# Guidelines for AWS modules
## Getting Started
The Ansible AWS modules and these guidelines are maintained by the Ansible AWS Working Group. For
further information see
[the AWS working group community page](https://github.com/ansible/community/tree/master/group-aws).
If you are planning to contribute AWS modules to Ansible then getting in touch with the the working
group will be a good way to start, especially because a similar module may already be under
development.
Since Ansible 2.0, it is required that all new AWS modules are written to use boto3.
## Choose Boto3 not Boto
Prior to 2.0, modules may have been written in boto or boto3. The effort to port all modules to boto3 has begun.
Starting from Ansible 2.0, it has been required that all new AWS modules are written to use boto3.
Please do not add new dependencies on the old boto library.
Prior to 2.0, modules may have been written in boto or boto3. The effort to port all modules to
boto3 has begun. From Ansible 2.4 it is permissible for modules which previously required boto to
start to require boto3 in order to deliver the functionality previously supported with boto and the
boto dependency can be deleted.
## Bug fixing
Bug fixes to code that relies on boto will still be accepted. When possible, the code should be ported to use boto3.
Bug fixes to code that relies on boto will still be accepted. When possible, the code should be
ported to use boto3.
## Naming your module
Base the name of the module on the part of AWS that
you actually use. (A good rule of thumb is to take
whatever module you use with boto as a starting point).
Base the name of the module on the part of AWS that you actually use. (A good rule of thumb is to
take whatever module you use with boto as a starting point). Don't further abbreviate names - if
something is a well known abbreviation due to it being a major component of AWS, that's fine, but
don't create new ones independently (e.g. VPC, ELB, and similar abbreviations are fine).
Don't further abbreviate names - if something is a well
known abbreviation due to it being a major component of
AWS, that's fine, but don't create new ones independently
(e.g. VPC, ELB, etc. are fine)
Unless the name of your service is quite unique, please consider using "aws_" as a prefix. For
example "aws_lambda".
## Adding new features
Try to keep backward compatibility with relatively recent
versions of boto3. That means that if you want to implement some
functionality that uses a new feature of boto3, it should only
fail if that feature actually needs to be run, with a message
saying which version of boto3 is needed.
Try to keep backward compatibility with relatively recent versions of boto3. That means that if you
want to implement some functionality that uses a new feature of boto3, it should only fail if that
feature actually needs to be run, with a message stating the missing feature and minimum required
version of boto3.
Use feature testing (e.g. `hasattr('boto3.module', 'shiny_new_method')`)
to check whether boto3 supports a feature rather than version checking
Use feature testing (e.g. `hasattr('boto3.module', 'shiny_new_method')`) to check whether boto3
supports a feature rather than version checking
e.g. from the `ec2` module:
```python
@ -41,11 +51,18 @@ else:
module.fail_json(msg="instance_profile_name parameter requires boto version 2.5.0 or higher")
```
## Using boto and boto3
## Using botocore and boto3
### Importing
Wrap import statements in a try block and fail the module later if the import fails
The `ansible.module_utils.ec2` module and `ansible.module_utils.core.aws` modules will both
automatically import boto3 and botocore. If boto3 is missing from the system then the variable
HAS_BOTO3 will be set to false. Normally, this means that modules don't need to import either
botocore or boto3 directly.
If you want to import the modules anyway (for example `from botocore.exception import
ClientError`) Wrap import statements in a try block and fail the module later using HAS_BOTO3 if
the import fails.
#### boto
@ -66,37 +83,30 @@ def main():
#### boto3
```python
try:
import boto3
HAS_BOTO3 = True
except ImportError:
HAS_BOTO3 = False
from ansible.module_utils.aws.core import HAS_BOTO3
def main():
if not HAS_BOTO3:
module.fail_json(msg='boto3 required for this module')
module.fail_json(msg='boto3 and botocore are required for this module')
```
#### boto and boto3 combined
Ensure that you clearly document if a new parameter requires requires a specific version. Import boto3 at the top of the
module as normal and then use the HAS_BOTO3 bool when necessary, before the new feature.
Ensure that you clearly document if a new parameter requires requires a specific version. Import
boto3 at the top of the module as normal and then use the HAS_BOTO3 bool when necessary, before the
new feature.
```python
from ansible.module_utils.ec2 import HAS_BOTO3
try:
import boto
HAS_BOTO = True
except ImportError:
HAS_BOTO = False
try:
import boto3
HAS_BOTO3 = True
except ImportError:
HAS_BOTO3 = False
if my_new_feauture_Parameter_is_set:
if my_new_feature_Parameter_is_set:
if HAS_BOTO3:
# do feature
else:
@ -107,9 +117,11 @@ if my_new_feauture_Parameter_is_set:
To connect to AWS, you should use `get_aws_connection_info` and then `boto3_conn`.
These functions handle some of the more esoteric connection options, such as security tokens and boto profiles.
These functions handle some of the more esoteric connection options, such as security tokens and
boto profiles.
Some boto services require that the region is specified. You should check for the region parameter if required.
Some boto services require that the region is specified. You should check for the region parameter
if required.
#### boto
@ -128,8 +140,9 @@ else:
#### boto3
An example of connecting to ec2 is shown below. Note that there is no 'NoAuthHandlerFound' exception handling like in boto.
Instead, an AuthFailure exception will be thrown when you use 'connection'. See exception handling.
An example of connecting to ec2 is shown below. Note that there is no `NoAuthHandlerFound`
exception handling like in boto. Instead, an `AuthFailure` exception will be thrown when you use
'connection'. See exception handling.
```python
region, ec2_url, aws_connect_params = get_aws_connection_info(module, boto3=True)
@ -139,10 +152,11 @@ else:
module.fail_json(msg="region must be specified")
```
### Exception Handling
### Exception Handling for boto
You should wrap any boto call in a try block. If an exception is thrown, it is up to you decide how to handle it
but usually calling fail_json with the error or helpful message and traceback will suffice.
You should wrap any boto call in a try block. If an exception is thrown, it is up to you decide how
to handle it but usually calling fail_json with the error or helpful message and traceback will
suffice.
#### boto
@ -166,11 +180,64 @@ except BotoServerError as e:
**camel_dict_to_snake_dict(e.message))
```
#### boto3
### Exception Handling for boto3 and botocore
For more information on botocore exception handling see [http://botocore.readthedocs.org/en/latest/client_upgrades.html#error-handling]
You should wrap any boto3 or botocore call in a try block. If an exception is thrown, then there
are a number of possibilities for handling it.
Boto3 provides lots of useful info when an exception is thrown so pass this to the user along with the message.
* use aws_module.fail_json_aws() to report the module failure in a standard way
* retry using AWSRetry
* use fail_json() to report the failure without using `ansible.module_utils.aws.core`
* do something custom in the case where you know how to handle the exception
For more information on botocore exception handling see [the botocore error documentation](http://botocore.readthedocs.org/en/latest/client_upgrades.html#error-handling).
#### using fail_json_aws()
_fail_json_aws() is a new method and may be subject to change. You can use it in modules which are
being contributed back to Ansible, however if you are publishing your module separately please
don't use it before the start of 2018 / Ansible 2.4_
In the AnsibleAWSModule there is a special method, `module.fail_json.aws()` for nice reporting of
exceptions. Call this on your exception and it will report the error together with a traceback for
use in Ansible verbose mode.
```python
from ansible.module_utils.aws.core import HAS_BOTO3, AnsibleAWSModule
# Set up module parameters
...
# Connect to AWS
...
# Make a call to AWS
try:
result = connection.aws_call()
except Exception as e:
module.fail_json_aws(e, msg="trying to do aws_call")
```
Note that it should normally be acceptable to catch all normal exceptions here, however if you
expect anything other than botocore exceptions you should test everything works as expected.
If you need to perform an action based on the error boto3 returned, use the error code.
```python
# Make a call to AWS
try:
result = connection.aws_call()
except ClientError, e:
if e.response['Error']['Code'] == 'NoSuchEntity':
return None
else:
module.fail_json_aws(e, msg="trying to do aws_call")
```
#### using fail_json() and avoiding ansible.module_utils.aws.core
Boto3 provides lots of useful information when an exception is thrown so pass this to the user
along with the message.
```python
# Import ClientError from botocore
@ -207,17 +274,16 @@ except ClientError as e:
### Returning Values
When you make a call using boto3, you will probably get back some useful information that you should return in the module.
When you make a call using boto3, you will probably get back some useful information that you
should return in the module. As well as information related to the call itself, you will also have
some response metadata. It is OK to return this to the user as well as they may find it useful.
As well as information related to the call itself, you will also have some response metadata. It is OK to return this to
the user as well as they may find it useful.
Boto3 returns all values CamelCased. Ansible follows Python standards for variable names and uses
snake_case. There is a helper function in module_utils/ec2.py called `camel_dict_to_snake_dict`
that allows you to easily convert the boto3 response to snake_case.
Boto3 returns all values CamelCased. Ansible follows Python standards for variable names and uses snake_case. There is a
helper function in module_utils/ec2.py called `camel_dict_to_snake_dict` that allows you to easily convert the boto3
response to snake_case.
You should use this helper function and avoid changing the names of values returned by Boto3. E.g. if boto3 returns a
value called 'SecretAccessKey' do not change it to 'AccessKey'.
You should use this helper function and avoid changing the names of values returned by Boto3.
E.g. if boto3 returns a value called 'SecretAccessKey' do not change it to 'AccessKey'.
```python
# Make a call to AWS
@ -229,7 +295,8 @@ module.exit_json(changed=True, **camel_dict_to_snake_dict(result))
### Dealing with IAM JSON policy
If your module accepts IAM JSON policies then set the type to 'json' in the module spec. For example"
If your module accepts IAM JSON policies then set the type to 'json' in the module spec. For
example:
```python
argument_spec.update(
@ -239,8 +306,8 @@ argument_spec.update(
)
```
Note that AWS is unlikely to return the policy in the same order that is was submitted. Therefore, a helper
function has been created to order policies before comparison.
Note that AWS is unlikely to return the policy in the same order that is was submitted. Therefore,
a helper function has been created to order policies before comparison.
```python
# Get the policy from AWS
@ -260,60 +327,66 @@ else:
AWS has a concept of resource tags. Usually the boto3 API has separate calls for tagging and
untagging a resource. For example, the ec2 API has a create_tags and delete_tags call.
It is common practice in Ansible AWS modules to have a 'purge_tags' parameter that defaults to true.
It is common practice in Ansible AWS modules to have a `purge_tags` parameter that defaults to
true.
The purge_tags parameter means that existing tags will be deleted if they are not specified in
by the Ansible playbook.
The `purge_tags` parameter means that existing tags will be deleted if they are not specified by
the Ansible task.
There is a helper function 'compare_aws_tags' to ease dealing with tags. It can compare two dicts and
return the tags to set and the tags to delete. See the Helper function section below for more detail.
There is a helper function `compare_aws_tags` to ease dealing with tags. It can compare two dicts
and return the tags to set and the tags to delete. See the Helper function section below for more
detail.
### Helper functions
Along with the connection functions in Ansible ec2.py module_utils, there are some other useful functions detailed below.
Along with the connection functions in Ansible ec2.py module_utils, there are some other useful
functions detailed below.
#### camel_dict_to_snake_dict
boto3 returns results in a dict. The keys of the dict are in CamelCase format. In keeping
with Ansible format, this function will convert the keys to snake_case.
boto3 returns results in a dict. The keys of the dict are in CamelCase format. In keeping with
Ansible format, this function will convert the keys to snake_case.
#### ansible_dict_to_boto3_filter_list
Converts a an Ansible list of filters to a boto3 friendly list of dicts. This is useful for
any boto3 _facts modules.
Converts a an Ansible list of filters to a boto3 friendly list of dicts. This is useful for any
boto3 `_facts` modules.
#### boto3_tag_list_to_ansible_dict
Converts a boto3 tag list to an Ansible dict. Boto3 returns tags as a list of dicts containing keys called
'Key' and 'Value' by default. This key names can be overriden when calling the function. For example, if you have already
camel_cased your list of tags you may want to pass lowercase key names instead i.e. 'key' and 'value'.
Converts a boto3 tag list to an Ansible dict. Boto3 returns tags as a list of dicts containing keys
called 'Key' and 'Value' by default. This key names can be overridden when calling the function.
For example, if you have already camel_cased your list of tags you may want to pass lowercase key
names instead i.e. 'key' and 'value'.
This function converts the list in to a single dict where the dict key is the tag key and the dict value is the tag value.
This function converts the list in to a single dict where the dict key is the tag key and the dict
value is the tag value.
#### ansible_dict_to_boto3_tag_list
Opposite of above. Converts an Ansible dict to a boto3 tag list of dicts. You can again override the key names used if 'Key'
and 'Value' is not suitable.
Opposite of above. Converts an Ansible dict to a boto3 tag list of dicts. You can again override
the key names used if 'Key' and 'Value' is not suitable.
#### get_ec2_security_group_ids_from_names
Pass this function a list of security group names or combination of security group names and IDs and this function will
return a list of IDs. You should also pass the VPC ID if known because security group names are not necessarily unique
across VPCs.
Pass this function a list of security group names or combination of security group names and IDs
and this function will return a list of IDs. You should also pass the VPC ID if known because
security group names are not necessarily unique across VPCs.
#### sort_json_policy_dict
Pass any JSON policy dict to this function in order to sort any list contained therein. This is useful
because AWS rarely return lists in the same order that they were submitted so without this function, comparison
of identical policies returns false.
Pass any JSON policy dict to this function in order to sort any list contained therein. This is
useful because AWS rarely return lists in the same order that they were submitted so without this
function, comparison of identical policies returns false.
### compare_aws_tags
Pass two dicts of tags and an optional purge parameter and this function will return a dict containing key pairs you need
to modify and a list of tag key names that you need to remove. Purge is True by default. If purge is False then any
existing tags will not be modified.
Pass two dicts of tags and an optional purge parameter and this function will return a dict
containing key pairs you need to modify and a list of tag key names that you need to remove. Purge
is True by default. If purge is False then any existing tags will not be modified.
This function is useful when using boto3 'add_tags' and 'remove_tags' functions. Be sure to use the other helper function
'boto3_tag_list_to_ansible_dict' to get an appropriate tag dict before calling this function. Since the AWS APIs are not
uniform (e.g. EC2 versus Lambda) this will work without modification for some (Lambda) and others may need modification
before using these values (such as EC2, with requires the tags to unset to be in the form [{'Key': key1}, {'Key': key2}]).
This function is useful when using boto3 'add_tags' and 'remove_tags' functions. Be sure to use the
other helper function `boto3_tag_list_to_ansible_dict` to get an appropriate tag dict before
calling this function. Since the AWS APIs are not uniform (e.g. EC2 versus Lambda) this will work
without modification for some (Lambda) and others may need modification before using these values
(such as EC2, with requires the tags to unset to be in the form `[{'Key': key1}, {'Key': key2}]`).