Commit graph

415 commits

Author SHA1 Message Date
Matt Martz
c1c229c6d4
Remove use of simplejson throughout code base (#43548)
* Remove use of simplejson throughout code base. Fixes #42761

* Address failing tests

* Remove simplejson from contrib and other outlying files

* Add changelog fragment for simplejson removal
2018-08-10 11:13:29 -05:00
Joren Vrancken
b954917761 Surround top-level function and class definitions with two blank lines. 2018-07-31 12:06:56 -07:00
flowerysong
a08668cf00 Port ec2_tag to boto3 (#39712)
* Add volume manipulation to EC2 integration test policy

* Port ec2_tag to boto3
2018-07-27 15:45:18 -04:00
Toshio Kuratomi
52449cc01a AnsiballZ improvements
Now that we don't need to worry about python-2.4 and 2.5, we can make
some improvements to the way AnsiballZ handles modules.

* Change AnsiballZ wrapper to use import to invoke the module
  We need the module to think of itself as a script because it could be
  coded as:

      main()

  or as:

      if __name__ == '__main__':
          main()

  Or even as:

      if __name__ == '__main__':
          random_function_name()

  A script will invoke all of those.  Prior to this change, we invoked
  a second Python interpreter on the module so that it really was
  a script.  However, this means that we have to run python twice (once
  for the AnsiballZ wrapper and once for the module).  This change makes
  the module think that it is a script (because __name__ in the module ==
  '__main__') but it's actually being invoked by us importing the module
  code.

  There's three ways we've come up to do this.
  * The most elegant is to use zipimporter and tell the import mechanism
    that the module being loaded is __main__:
    * 5959f11c9d/lib/ansible/executor/module_common.py (L175)
    * zipimporter is nice because we do not have to extract the module from
      the zip file and save it to the disk when we do that.  The import
      machinery does it all for us.
    * The drawback is that modules do not have a __file__ which points
      to a real file when they do this.  Modules could be using __file__
      to for a variety of reasons, most of those probably have
      replacements (the most common one is to find a writable directory
      for temporary files.  AnsibleModule.tmpdir should be used instead)
      We can monkeypatch __file__ in fom AnsibleModule initialization
      but that's kind of gross.  There's no way I can see to do this
      from the wrapper.

  * Next, there's imp.load_module():
    * https://github.com/abadger/ansible/blob/340edf7489/lib/ansible/executor/module_common.py#L151
    * imp has the nice property of allowing us to set __name__ to
      __main__ without changing the name of the file itself
    * We also don't have to do anything special to set __file__ for
      backwards compatibility (although the reason for that is the
      drawback):
    * Its drawback is that it requires the file to exist on disk so we
      have to explicitly extract it from the zipfile and save it to
      a temporary file

  * The last choice is to use exec to execute the module:
    * https://github.com/abadger/ansible/blob/f47a4ccc76/lib/ansible/executor/module_common.py#L175
    * The code we would have to maintain for this looks pretty clean.
      In the wrapper we create a ModuleType, set __file__ on it, read
      the module's contents in from the zip file and then exec it.
    * Drawbacks: We still have to explicitly extract the file's contents
      from the zip archive instead of letting python's import mechanism
      handle it.
    * Exec also has hidden performance issues and breaks certain
      assumptions that modules could be making about their own code:
      http://lucumr.pocoo.org/2011/2/1/exec-in-python/

  Our plan is to use imp.load_module() for now, deprecate the use of
  __file__ in modules, and switch to zipimport once the deprecation
  period for __file__ is over (without monkeypatching a fake __file__ in
  via AnsibleModule).

* Rename the name of the AnsiBallZ wrapped module
  This makes it obvious that the wrapped module isn't the module file that
  we distribute.  It's part of trying to mitigate the fact that the module
  is now named __main)).py in tracebacks.

* Shield all wrapper symbols inside of a function
  With the new import code, all symbols in the wrapper become visible in
  the module.  To mitigate the chance of collisions, move most symbols
  into a toplevel function.  The only symbols left in the global namespace
  are now _ANSIBALLZ_WRAPPER and _ansiballz_main.

revised porting guide entry

Integrate code coverage collection into AnsiballZ.

ci_coverage
ci_complete
2018-07-26 20:07:25 -07:00
Calvin Wu
7e42e88cc1 ecs_taskdefinition can absent without containers argument (#41398)
* ecs_taskdefinition can absent without containers argument

* add regression test for absent with arn

* Add PassRole privilege for ecs_cluster to pass
2018-07-12 23:16:41 +10:00
Julien Vey
0f612d1b76 efs_facts: improve performance by reducing the number of api calls (#36520)
* efs_facts: improve performance by reducing the number of api calls

* Remove efs_facts tests from running in CI
2018-07-08 16:34:22 -04:00
Troy Murray
15ce7c5bab change OS X to macOS (#41294)
* change OS X to macOS

<!--- Your description here -->

+label: docsite_pr

* Update all Mac OS X references to be macOS

* Drop extra Mac
2018-06-26 14:09:23 -04:00
Jon Dufresne
dc7e50fa90 Update additional pypi.python.org URLs to pypi.org (#41373)
For details on the new PyPI, see the blog post:

https://pythoninsider.blogspot.ca/2018/04/new-pypi-launched-legacy-pypi-shutting.html
2018-06-17 14:01:18 +02:00
Will Thames
b235cb8734 aws_eks_cluster: New module for managing AWS EKS (#41183)
* aws_eks: New module for managing AWS EKS

aws_eks module is used for creating and removing EKS clusters.

Includes full test suite and updates to IAM policies to enable it.

* Clean up all security groups

* appease shippable

* Rename aws_eks module to aws_eks_cluster
2018-06-07 08:44:04 -04:00
Will Thames
a60fe1946c Remove ECS policies from AWS compute policy
The compute policy was exceeding maximum size and contained
policies that already exist in ecs-policy.

Look up suitable AMIs rather than hardcode

We don't want to maintain multiple image IDs for multiple regions
so use ec2_ami_facts to set a suitable image ID

Improve exception handling
2018-06-06 20:51:50 +10:00
Zhikang Zhang
b578bf9e20 Fix test-module failing to validate args (#41004)
* Fix test-module failing to validate args

The test-module pass a wrong argument _ansible_tmp cause the validation failed.
Change the argument _ansible_tmp to _ansible_tmpdir to fix this.

* Add a integration test for test-module.

Prior to this change, we don't have a test for test-module.

This change ensure the correctness of test-module script.
2018-06-01 12:02:56 -07:00
Will Thames
809c7404ab Add two missing VPC permissions (#37896)
Remove VPC permissions from network-policy.json as they mostly duplicate
compute-policy.json permissions - separating the VPC and compute permissions
would likely lead to further confusion.
2018-05-25 06:31:54 -04:00
Matt Clay
8deced3e04
Fix shebangs and file modes and update tests. (#40563)
* Add execute bit sanity test and apply fixes.
* Add shebang test for `lib` dirs and apply fixes.
* Shebang and execute bit cleanup.
2018-05-22 14:25:36 -07:00
Matt Davis
e4edb2842a 2.6 changelog gen/version/root dir cleanup (#40421)
* patched in changelog gen stuff from stable-2.5
* Makefile updates
* release.py as single-source-of-truth
* Remove obsolete ansible-core-sitemap.xml file.
* Move ROADMAP.rst into README.rst.
* dynamic rpm changelog, zap old deb/rpm changelogs
* fix changelog in MANIFEST.in
* Remove obsolete hacking/update.sh script.
* Remove ref to deleted authors script.
* Remove ref to removed module-formatter script.
* Update headings to match script names.
* MANIFEST.in cleanup
* removed RELEASES.txt and versions.yml
* removed obsolete release generation playbook/bits (not used since 2.5)
* misc Makefile cleanup
* speculative changes to DEB versioning
* allow override of DEB_VERSION/DEB_RELEASE
2018-05-21 16:14:53 -07:00
Toshio Kuratomi
5634dae290 Remove the cherrypick script
We only needed it for migrating cherrypicks between the unified repo and
the ansible-modules-* repos.  Now that we aren't supporting 2.3, we no
longer need this script.
2018-05-21 14:50:24 -05:00
Tim Rupp
548282139f
Fixes incorrect variable name (#40274)
Incorrect variable name was causing a NameError

  NameError: name 'comlpex_args' is not defined
2018-05-16 12:13:22 -07:00
Madhura-CSI
b85970b2b0 New Module: ec2_vpc_vpn_facts (#35983)
* New module: ec2_vpc_vpn_facts

* Add integration tests for ec2_vpc_vpn_facts and the IAM permissions

* Add retry to VPC removal

* Use unique name for VGW

* Always clean up after tests and add retries
2018-05-15 12:13:46 -04:00
Jordan Borean
44ab948e5d
create module tmpdir based on remote_tmp (#39833)
* create module tmpdir based on remote_tmp

* Source remote_tmp from controller if possible

* Fixed sanity test and not use lambda

* Added expansion of env vars to the remote tmp

* Fixed sanity issues

* Added note around shell remote_tmp option

* Changed fallback tmp dir to ~/.ansible/tmp to make shell defaults
2018-05-15 09:31:21 +10:00
Will Thames
29770a297a Fail with nice error message if elb target_type=ip not supported (#38313)
* Add helpful failure message if target_type=ip is not supported

Create test case for target_type=ip not supported

* Update elb_target_group module to latest standards

Use AnsibleAWSModule
Improve exception handling
Improve connection handling
2018-05-03 08:36:52 -04:00
mwpeterson
efdd92e1c0 Update test-module (#39331)
Update test-module To use C.DEFAULT_LOCAL_TMP
2018-04-26 07:16:33 -07:00
Will Thames
12f2b9506d [aws]Add VPC configuration to ECS modules (#34381)
Enable awsvpc network mode for ECS services and tasks and
their underlying task definitions

Improve test suite to thoroughly test the changes

Use runme.sh technique to run old and new versions of botocore to
ensure that the modules work with older botocore and older network modes
and fail gracefully if awsvpc network mode is used with older botocore
2018-04-25 15:41:04 -04:00
Matt Clay
c262dbfd30 Use https for links to ansible.com domains. 2018-04-23 11:33:56 -07:00
Ed Costello
0d31d1cd24 [cloud]Add aws_ses_identity_policy module for managing SES sending policies (#36623)
* Add aws_ses_identity_policy module for managing SES sending policies

* Add option to AnsibleAWSModule for applying a retry decorator to all calls.

* Add per-callsite opt in to retry behaviours in AnsibleAWSModule

* Update aws_ses_identity_policy module to opt in to retries at all callsites.

* Add test for aws_ses_identity_policy module with inline policy.

* Remove implicit retrys on boto resources since they're not working yet.
2018-04-05 15:11:12 -04:00
Will Thames
a1d3cf488d [cloud][test]Add missing IAM policy for cloudfront (#38248)
Cloudfront needs CreateOriginAccessIdentity

Add profile parameter to setup-iam.yml. Could arguably just use
AWS_PROFILE but given that other tasks are using profile, should
be consistent.
2018-04-05 14:06:04 -04:00
Pilou
7908f78fa6 module_common: handle None value for templar (#36651)
* module_common: set required parameter templar

Fix the following error (related to b455901):

  $ ./hacking/test-module -m ./lib/ansible/modules/system/ping.py -I ansible_python_interpreter=/usr/bin/python
  Traceback (most recent call last):
    File "./hacking/test-module", line 268, in <module>
      main()
    File "./hacking/test-module", line 249, in main
      (modfile, modname, module_style) = boilerplate_module(options.module_path, options.module_args, interpreters, options.check, options.filename)
    File "./hacking/test-module", line 152, in boilerplate_module
      task_vars=task_vars
    File "ansible/lib/ansible/executor/module_common.py", line 910, in modify_module
      environment=environment)
    File "ansible/lib/ansible/executor/module_common.py", line 736, in _find_module_utils
      shebang, interpreter = _get_shebang(u'/usr/bin/python', task_vars, templar)
    File "ansible/lib/ansible/executor/module_common.py", line 452, in _get_shebang
      interpreter = templar.template(task_vars[interpreter_config].strip())
  AttributeError: 'NoneType' object has no attribute 'template'

* module_common.modify_module: templar is required
2018-03-29 13:54:48 -04:00
Julien Vey
7c07877b1b s3_bucket: add integration tests (#36941)
Also update testing-policies/storage
2018-03-07 11:25:24 -05:00
Julien Vey
51d491f8f0 route53_zone: move to boto3, and enable comment update (#36641) 2018-03-07 11:16:04 -05:00
Ed Costello
645952c139 Add aws_caller_facts module and use it in setup-iam.yml (#36683)
* Add aws_caller_facts module and use it in setup-iam.yml

This removes the dependency on having the command line AWS tools
installed.
2018-02-28 16:30:34 +10:00
Will Thames
b5a1643e3d Add new aws_waf_condition module (#33110) 2018-02-01 18:16:27 -05:00
Ed Costello
d16bc1c3f4 New aws_ses_identity module to manage AWS Simple Email Service Identity (#31140)
* Add aws_ses_identity module

* Update CI alias, add BotoCoreError exception handling.

* Add SES and SNS permissions to hacking/aws_config to run aws_ses_identity integration tests
2018-01-29 15:35:49 -05:00
Marek
5fa29201a7 Port sts_assume_role to boto3 (#32569)
* Ported sts_assume_role to boto3

* Added integration tests
2018-01-22 17:46:08 -05:00
Adrian Likins
ec9769c82f
Facts distribution clear linux 31501 (#32453)
The search string used to look for Clear Linux
was changed in 45a9f96774 to
be more specific, but was too specific. Now finding
a substring match for 'Clear Linux' in /usr/lib/os-release
is enough to consider a match.

Since the details of the full name in os-release varies
('Clear Linux Software for Intel Architecture',
 'Clear Linux OS for Intel Architecture', etc) the
search string match was failing and would fall back to the
'first word in the release file' method resulting in
ansible_distribution='NAME="Clear'

Also add a meta fact indicating which search string
was matched.

Test case info from:
        https://github.com/ansible/ansible/issues/31501#issuecomment-340861535

Fixes #31501
2018-01-20 15:05:53 -05:00
Will Thames
4d58d16793 Add aws_s3 action plugin to find source files as expected (#35028)
People expect to be able to upload files to s3 using standard
locations for files.

Providing an action plugin that effectively rewrites the `src`
key to the result of finding such a file is a great help.

Tests added, and IAM permissions corrected
2018-01-19 17:11:42 -05:00
Will Thames
8d733dbdf0 [cloud] New module cloudfront_distribution (#31284)
* added cloudfont.py, modified cloudfront_facts.py class name and fixed a minor bug

* Improvements to cloudfront_distribution

* Reduce the scope of the cloudfront_distribution module
    * Remove presigning
    * Remove streaming distribution functionality
* Add full test suite for cloudfront distribution
* Meet Ansible AWS guidelines

* Make requested changes

Fix tests

Use built-in waiter

Update copyright
2018-01-17 11:03:23 -05:00
Matt Clay
797664d9cb Python 2.6 str.format() compatibility fixes. 2018-01-10 14:08:11 -08:00
Will Thames
866d7fdce9 [cloud] Create ECS integration test suite (#33757)
Tests for:
* ecs_cluster
* ecs_service
* ecs_service_facts
* ecs_taskdefinition
* ecs_taskdefinition_facts

* Add idempotency testing

Test ecs_cluster, ecs_service and ecs_taskdefinition for trivial
idempotency. Add FIXMEs to the tests because the latter two fail.

Remove unused dependencies
2017-12-15 08:15:01 -05:00
Abhijeet Kasurde
36f82ae8cc Replace exit() with sys.exit()
This fix adds replacement for exit() to sys.exit(), as
exit() is not recommended way to exit from the program.

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2017-12-14 22:03:08 -05:00
Sloane Hertel
f5471b3dcb [cloud] ec2_vpc_net integration tests (#33111)
* Add some integration tests for ec2_vpc_net module

* Add a couple tests for check mode

fix typo

ensure the DHCP option set is cleaned up

* Add permissions to test policy
2017-12-05 16:41:16 -05:00
Matt Martz
783a09d105
Fix failure to adjust my name when doing copy pasta (#33322) 2017-11-27 19:09:05 -06:00
Matt Martz
4fe08441be Deprecate tests used as filters (#32361)
* Warn on tests used as filters

* Update docs, add aliases for tests that fit more gramatically with test syntax

* Fix rst formatting

* Add successful filter, alias of success

* Remove renamed_deprecation, it was overkill

* Make directory alias for is_dir

* Update tests to use proper jinja test syntax

* Update additional documentation, living outside of YAML files, to reflect proper jinja test syntax

* Add conversion script, porting guide updates, and changelog updates

* Update newly added uses of tests as filters

* No underscore variable

* Convert recent tests as filter changes to win_stat

* Fix some changes related to rebasing a few integration tests

* Make tests_as_filters_warning explicitly accept the name of the test, instead of inferring the name

* Add test for tests_as_filters_warning

* Update tests as filters in newly added/modified tests

* Address recent changes to several integration tests

* Address recent changes in cs_vpc
2017-11-27 17:58:08 -05:00
Will Thames
1ca0c0e7f7 Consolidate IAM policies into fewer, larger policies (#33122)
Due to IAM limits allowing at most 10 policies per group,
need to reduce the number of total policies in use.
2017-11-21 17:15:31 -05:00
Will Thames
0b128e2e29 Add tests for ec2_vpc_route_table (#32036)
* Add tests for ec2_vpc_route_table

Initial tests for ec2_vpc_route_table ready for boto3 porting

* Turn AWS connection information into a YAML block
2017-11-20 17:31:08 -05:00
Pilou
a9afb1e19e hacking/test-module: fix Python 3 compatibility (#33069)
Exception was:
Traceback (most recent call last):
  File "./hacking/test-module", line 268, in <module>
    main()
  File "./hacking/test-module", line 249, in main
    (modfile, modname, module_style) = boilerplate_module(options.module_path, options.module_args, interpreters, options.check, options.filename)
  File "./hacking/test-module", line 155, in boilerplate_module
    if module_style == 'new' and 'ANSIBALLZ_WRAPPER = True' in module_data:
TypeError: a bytes-like object is required, not 'str'
2017-11-20 08:03:55 +01:00
Daniel Shepherd
cfbe9c8aee [cloud] Add IPv6 support for ec2_vpc_subnet module(#30444)
* Add integration test suite for ec2_vpc_subnet

* wrap boto3 connection in try/except

update module documentation and add RETURN docs

add IPv6 support to VPC subnet module

rename ipv6cidr to ipv6_cidr, use required_if for parameter testing, update some failure messages to be more descriptive

DryRun mode was removed from this function a while ago but exception handling was still checking for it, removed

add wait and timeout for subnet creation process

fixup the ipv6 cidr disassociation logic a bit per review

update RETURN values per review

added module parameter check

removed DryRun parameter from boto3 call since it would always be false here

fix subnet wait loop

add a purge_tags parameter, fix the ensure_tags function, update to use compare_aws_tags func

fix tags type error per review

remove **kwargs use in create_subnet function per review

* rebased on #31870, fixed merge conflicts, and updated error messages

* fixes to pass tests

* add test for failure on invalid ipv6 block and update tags test for purge_tags=true function

* fix pylint issue

* fix exception handling error when run with python3

* add ipv6 tests and fix module code

* Add permissions to hacking/aws_config/testing_policies/ec2-policy.json for adding IPv6 cidr blocks to VPC and subnets

* fix type in tests and update assert conditional to check entire returned value

* add AWS_SESSION_TOKEN into environment for aws cli commands to work in CI

* remove key and value options from call to boto3_tag_list_to_ansible_dict

* remove wait loop and use boto3 EC2 waiter

* remove unused register: result vars

* revert az argument default value to original setting default=None
2017-11-16 14:58:12 -05:00
Will Thames
46c4f6311a [cloud] Add retries/backoff to ec2_vpc_subnet module (#31870)
* Allow backoff for describe_subnets

Improve exception handling to latest standards

* Add integration test suite for ec2_vpc_subnet

* Add test for creating subnet without AZ

Fix bug identified by test

Fixes #31905
2017-11-16 13:09:42 -05:00
Sloane Hertel
1dd55acbc2 ec2_group: add rule description support - fixes #29040 (#30273)
* ec2_group: add support for rule descriptions.

* Document rule description feature and add an example using it.

* Fix removing rule descriptions.

* Add integration tests to verify adding/modifying/removing rule descriptions works as expected.

* Add permissions to hacking/aws_config/testing_policies/ec2-policy.json for updating ingress and egress rule descriptions.

* ec2_group: add backwards compatibility with older versions of botocore for rule descriptions.

* Add compatibility with older version of botocore for ec2_group integration tests.

* ec2_group: move HAS_RULE_DESCRIPTION to be checked first.

* Make requested change

* Pass around a variable instead of client

* Make sure has_rule_description defaults to None

* Fail if rule_desc is in any ingress/egress rules and the the botocore version < 1.7.2

* Remove unnecessary variable

* Fix indentation for changed=True when updating rule descriptions.

* minor refactor to remove duplicate code

* add missing parameter

* Fix pep8

* Update test policy.
2017-10-24 21:18:56 -04:00
Prasad Katti
d6737dcdd4 Added info about return_skeleton_generator.py
Added info about return_skeleton_generator.py to hacking/README.md
2017-10-02 15:58:57 -04:00
Will Thames
37736ee87e Allow AWS image and snapshot creation/deletion
Provide all necessary permissions for AMI tests
Allow tests to run in us-east-2
Ensure `always` section gets used
Update tests to ensure that cleanup works better, and add
deletion idempotency test
2017-09-19 23:12:50 -07:00
Will Thames
5e978b0f2b Use JSON returns values to create RETURN docs
After running hacking/test-module to generate some output,
the JSON output can be fed into the return skeletion generator
to create an excellent starting point for RETURN docs
2017-09-11 14:33:11 -07:00
Will Thames
63df0adc17 [cloud] Update RDS parameter group for boto3 (#25345)
* Update RDS parameter group for boto3

* Update to boto3
* Update to latest ansible standards
* Remove choices list for valid engines (See #19221 for context)
* Allow tagging
* Return some useful information, and document that information

* Add tests for rds_param_group

* Improve testing of rds_param_group

* Add purge_tags option for rds_param_group

* Fix remaining broken rds_param_group tests

* Ensure the group name is lowercased. Fixes integration tests when run on OSX
2017-08-28 12:52:22 -04:00