Commit graph

424 commits

Author SHA1 Message Date
Matt Clay
f3d1f9544b
Make ansible-test available in the bin directory. (#45876) 2018-09-19 17:58:55 -07:00
Will Thames
d2569a3f7d Improve iam_group exception handling (#45599)
* Improve iam_group exception handling

Use AnsibleAWSModule for iam_group and handle BotoCoreErrors
as well as ClientErrors. Use fail_json_aws to improve error messages

* Add minimal iam_group test suite

Update some of the read-only IAM permissions (this is not sufficient
to run the test suite but it gets further than it did until it tries
to add a (non-existent) user)

* Clean up after tests
2018-09-17 19:53:44 -04:00
Michael Scherer
83db157c35 Add a default man path (#45689)
It seems that on some Linux distribution (Fedora 28, Debian), man will
not fallback on a default path if MANPATH is set. So using the env-setup
script will prevent man from working.
2018-09-17 18:31:39 -04:00
Sviatoslav Sydorenko
e8731a1f5b
Add a ticket stub for guiding GitHub newcomers
PR #45497
2018-09-11 17:51:54 +02:00
Ed Costello
96c4efcd95 Add missing s3 permissions for s3 module testing. (#43243) 2018-09-10 17:21:36 -04:00
Matt Martz
7a61763fba
Don't require requests in hacking/report.py (#45350)
* Don't require requests in hacking/report.py

* move ansible import
2018-09-07 11:51:41 -05:00
Will Thames
60e3af42d5 sns_topic boto3 port (#39292)
* Port sns_topic to boto3 and add tests
2018-08-23 21:04:18 -04:00
sdubrul
061877d584 added account_alias in the response of module aws_caller_facts (#42345)
* added account_alias in the response of module aws_caller_facts

* added comment to explain list_account_aliases

* renamed caller_identity to caller_facts as the content is extended

* created changelog

* security-policy needs the iam:ListAccountAliases for this module to work

* test now checks for the added field account_alias

* gracefully handle missing iam:ListAccountAliases permission
2018-08-22 17:21:12 -04:00
Matt Clay
0392dbeba1 Fix path handling in hacking/env-setup. 2018-08-10 23:36:56 -07:00
Matt Martz
c1c229c6d4
Remove use of simplejson throughout code base (#43548)
* Remove use of simplejson throughout code base. Fixes #42761

* Address failing tests

* Remove simplejson from contrib and other outlying files

* Add changelog fragment for simplejson removal
2018-08-10 11:13:29 -05:00
Joren Vrancken
b954917761 Surround top-level function and class definitions with two blank lines. 2018-07-31 12:06:56 -07:00
flowerysong
a08668cf00 Port ec2_tag to boto3 (#39712)
* Add volume manipulation to EC2 integration test policy

* Port ec2_tag to boto3
2018-07-27 15:45:18 -04:00
Toshio Kuratomi
52449cc01a AnsiballZ improvements
Now that we don't need to worry about python-2.4 and 2.5, we can make
some improvements to the way AnsiballZ handles modules.

* Change AnsiballZ wrapper to use import to invoke the module
  We need the module to think of itself as a script because it could be
  coded as:

      main()

  or as:

      if __name__ == '__main__':
          main()

  Or even as:

      if __name__ == '__main__':
          random_function_name()

  A script will invoke all of those.  Prior to this change, we invoked
  a second Python interpreter on the module so that it really was
  a script.  However, this means that we have to run python twice (once
  for the AnsiballZ wrapper and once for the module).  This change makes
  the module think that it is a script (because __name__ in the module ==
  '__main__') but it's actually being invoked by us importing the module
  code.

  There's three ways we've come up to do this.
  * The most elegant is to use zipimporter and tell the import mechanism
    that the module being loaded is __main__:
    * 5959f11c9d/lib/ansible/executor/module_common.py (L175)
    * zipimporter is nice because we do not have to extract the module from
      the zip file and save it to the disk when we do that.  The import
      machinery does it all for us.
    * The drawback is that modules do not have a __file__ which points
      to a real file when they do this.  Modules could be using __file__
      to for a variety of reasons, most of those probably have
      replacements (the most common one is to find a writable directory
      for temporary files.  AnsibleModule.tmpdir should be used instead)
      We can monkeypatch __file__ in fom AnsibleModule initialization
      but that's kind of gross.  There's no way I can see to do this
      from the wrapper.

  * Next, there's imp.load_module():
    * https://github.com/abadger/ansible/blob/340edf7489/lib/ansible/executor/module_common.py#L151
    * imp has the nice property of allowing us to set __name__ to
      __main__ without changing the name of the file itself
    * We also don't have to do anything special to set __file__ for
      backwards compatibility (although the reason for that is the
      drawback):
    * Its drawback is that it requires the file to exist on disk so we
      have to explicitly extract it from the zipfile and save it to
      a temporary file

  * The last choice is to use exec to execute the module:
    * https://github.com/abadger/ansible/blob/f47a4ccc76/lib/ansible/executor/module_common.py#L175
    * The code we would have to maintain for this looks pretty clean.
      In the wrapper we create a ModuleType, set __file__ on it, read
      the module's contents in from the zip file and then exec it.
    * Drawbacks: We still have to explicitly extract the file's contents
      from the zip archive instead of letting python's import mechanism
      handle it.
    * Exec also has hidden performance issues and breaks certain
      assumptions that modules could be making about their own code:
      http://lucumr.pocoo.org/2011/2/1/exec-in-python/

  Our plan is to use imp.load_module() for now, deprecate the use of
  __file__ in modules, and switch to zipimport once the deprecation
  period for __file__ is over (without monkeypatching a fake __file__ in
  via AnsibleModule).

* Rename the name of the AnsiBallZ wrapped module
  This makes it obvious that the wrapped module isn't the module file that
  we distribute.  It's part of trying to mitigate the fact that the module
  is now named __main)).py in tracebacks.

* Shield all wrapper symbols inside of a function
  With the new import code, all symbols in the wrapper become visible in
  the module.  To mitigate the chance of collisions, move most symbols
  into a toplevel function.  The only symbols left in the global namespace
  are now _ANSIBALLZ_WRAPPER and _ansiballz_main.

revised porting guide entry

Integrate code coverage collection into AnsiballZ.

ci_coverage
ci_complete
2018-07-26 20:07:25 -07:00
Calvin Wu
7e42e88cc1 ecs_taskdefinition can absent without containers argument (#41398)
* ecs_taskdefinition can absent without containers argument

* add regression test for absent with arn

* Add PassRole privilege for ecs_cluster to pass
2018-07-12 23:16:41 +10:00
Julien Vey
0f612d1b76 efs_facts: improve performance by reducing the number of api calls (#36520)
* efs_facts: improve performance by reducing the number of api calls

* Remove efs_facts tests from running in CI
2018-07-08 16:34:22 -04:00
Troy Murray
15ce7c5bab change OS X to macOS (#41294)
* change OS X to macOS

<!--- Your description here -->

+label: docsite_pr

* Update all Mac OS X references to be macOS

* Drop extra Mac
2018-06-26 14:09:23 -04:00
Jon Dufresne
dc7e50fa90 Update additional pypi.python.org URLs to pypi.org (#41373)
For details on the new PyPI, see the blog post:

https://pythoninsider.blogspot.ca/2018/04/new-pypi-launched-legacy-pypi-shutting.html
2018-06-17 14:01:18 +02:00
Will Thames
b235cb8734 aws_eks_cluster: New module for managing AWS EKS (#41183)
* aws_eks: New module for managing AWS EKS

aws_eks module is used for creating and removing EKS clusters.

Includes full test suite and updates to IAM policies to enable it.

* Clean up all security groups

* appease shippable

* Rename aws_eks module to aws_eks_cluster
2018-06-07 08:44:04 -04:00
Will Thames
a60fe1946c Remove ECS policies from AWS compute policy
The compute policy was exceeding maximum size and contained
policies that already exist in ecs-policy.

Look up suitable AMIs rather than hardcode

We don't want to maintain multiple image IDs for multiple regions
so use ec2_ami_facts to set a suitable image ID

Improve exception handling
2018-06-06 20:51:50 +10:00
Zhikang Zhang
b578bf9e20 Fix test-module failing to validate args (#41004)
* Fix test-module failing to validate args

The test-module pass a wrong argument _ansible_tmp cause the validation failed.
Change the argument _ansible_tmp to _ansible_tmpdir to fix this.

* Add a integration test for test-module.

Prior to this change, we don't have a test for test-module.

This change ensure the correctness of test-module script.
2018-06-01 12:02:56 -07:00
Will Thames
809c7404ab Add two missing VPC permissions (#37896)
Remove VPC permissions from network-policy.json as they mostly duplicate
compute-policy.json permissions - separating the VPC and compute permissions
would likely lead to further confusion.
2018-05-25 06:31:54 -04:00
Matt Clay
8deced3e04
Fix shebangs and file modes and update tests. (#40563)
* Add execute bit sanity test and apply fixes.
* Add shebang test for `lib` dirs and apply fixes.
* Shebang and execute bit cleanup.
2018-05-22 14:25:36 -07:00
Matt Davis
e4edb2842a 2.6 changelog gen/version/root dir cleanup (#40421)
* patched in changelog gen stuff from stable-2.5
* Makefile updates
* release.py as single-source-of-truth
* Remove obsolete ansible-core-sitemap.xml file.
* Move ROADMAP.rst into README.rst.
* dynamic rpm changelog, zap old deb/rpm changelogs
* fix changelog in MANIFEST.in
* Remove obsolete hacking/update.sh script.
* Remove ref to deleted authors script.
* Remove ref to removed module-formatter script.
* Update headings to match script names.
* MANIFEST.in cleanup
* removed RELEASES.txt and versions.yml
* removed obsolete release generation playbook/bits (not used since 2.5)
* misc Makefile cleanup
* speculative changes to DEB versioning
* allow override of DEB_VERSION/DEB_RELEASE
2018-05-21 16:14:53 -07:00
Toshio Kuratomi
5634dae290 Remove the cherrypick script
We only needed it for migrating cherrypicks between the unified repo and
the ansible-modules-* repos.  Now that we aren't supporting 2.3, we no
longer need this script.
2018-05-21 14:50:24 -05:00
Tim Rupp
548282139f
Fixes incorrect variable name (#40274)
Incorrect variable name was causing a NameError

  NameError: name 'comlpex_args' is not defined
2018-05-16 12:13:22 -07:00
Madhura-CSI
b85970b2b0 New Module: ec2_vpc_vpn_facts (#35983)
* New module: ec2_vpc_vpn_facts

* Add integration tests for ec2_vpc_vpn_facts and the IAM permissions

* Add retry to VPC removal

* Use unique name for VGW

* Always clean up after tests and add retries
2018-05-15 12:13:46 -04:00
Jordan Borean
44ab948e5d
create module tmpdir based on remote_tmp (#39833)
* create module tmpdir based on remote_tmp

* Source remote_tmp from controller if possible

* Fixed sanity test and not use lambda

* Added expansion of env vars to the remote tmp

* Fixed sanity issues

* Added note around shell remote_tmp option

* Changed fallback tmp dir to ~/.ansible/tmp to make shell defaults
2018-05-15 09:31:21 +10:00
Will Thames
29770a297a Fail with nice error message if elb target_type=ip not supported (#38313)
* Add helpful failure message if target_type=ip is not supported

Create test case for target_type=ip not supported

* Update elb_target_group module to latest standards

Use AnsibleAWSModule
Improve exception handling
Improve connection handling
2018-05-03 08:36:52 -04:00
mwpeterson
efdd92e1c0 Update test-module (#39331)
Update test-module To use C.DEFAULT_LOCAL_TMP
2018-04-26 07:16:33 -07:00
Will Thames
12f2b9506d [aws]Add VPC configuration to ECS modules (#34381)
Enable awsvpc network mode for ECS services and tasks and
their underlying task definitions

Improve test suite to thoroughly test the changes

Use runme.sh technique to run old and new versions of botocore to
ensure that the modules work with older botocore and older network modes
and fail gracefully if awsvpc network mode is used with older botocore
2018-04-25 15:41:04 -04:00
Matt Clay
c262dbfd30 Use https for links to ansible.com domains. 2018-04-23 11:33:56 -07:00
Ed Costello
0d31d1cd24 [cloud]Add aws_ses_identity_policy module for managing SES sending policies (#36623)
* Add aws_ses_identity_policy module for managing SES sending policies

* Add option to AnsibleAWSModule for applying a retry decorator to all calls.

* Add per-callsite opt in to retry behaviours in AnsibleAWSModule

* Update aws_ses_identity_policy module to opt in to retries at all callsites.

* Add test for aws_ses_identity_policy module with inline policy.

* Remove implicit retrys on boto resources since they're not working yet.
2018-04-05 15:11:12 -04:00
Will Thames
a1d3cf488d [cloud][test]Add missing IAM policy for cloudfront (#38248)
Cloudfront needs CreateOriginAccessIdentity

Add profile parameter to setup-iam.yml. Could arguably just use
AWS_PROFILE but given that other tasks are using profile, should
be consistent.
2018-04-05 14:06:04 -04:00
Pilou
7908f78fa6 module_common: handle None value for templar (#36651)
* module_common: set required parameter templar

Fix the following error (related to b455901):

  $ ./hacking/test-module -m ./lib/ansible/modules/system/ping.py -I ansible_python_interpreter=/usr/bin/python
  Traceback (most recent call last):
    File "./hacking/test-module", line 268, in <module>
      main()
    File "./hacking/test-module", line 249, in main
      (modfile, modname, module_style) = boilerplate_module(options.module_path, options.module_args, interpreters, options.check, options.filename)
    File "./hacking/test-module", line 152, in boilerplate_module
      task_vars=task_vars
    File "ansible/lib/ansible/executor/module_common.py", line 910, in modify_module
      environment=environment)
    File "ansible/lib/ansible/executor/module_common.py", line 736, in _find_module_utils
      shebang, interpreter = _get_shebang(u'/usr/bin/python', task_vars, templar)
    File "ansible/lib/ansible/executor/module_common.py", line 452, in _get_shebang
      interpreter = templar.template(task_vars[interpreter_config].strip())
  AttributeError: 'NoneType' object has no attribute 'template'

* module_common.modify_module: templar is required
2018-03-29 13:54:48 -04:00
Julien Vey
7c07877b1b s3_bucket: add integration tests (#36941)
Also update testing-policies/storage
2018-03-07 11:25:24 -05:00
Julien Vey
51d491f8f0 route53_zone: move to boto3, and enable comment update (#36641) 2018-03-07 11:16:04 -05:00
Ed Costello
645952c139 Add aws_caller_facts module and use it in setup-iam.yml (#36683)
* Add aws_caller_facts module and use it in setup-iam.yml

This removes the dependency on having the command line AWS tools
installed.
2018-02-28 16:30:34 +10:00
Will Thames
b5a1643e3d Add new aws_waf_condition module (#33110) 2018-02-01 18:16:27 -05:00
Ed Costello
d16bc1c3f4 New aws_ses_identity module to manage AWS Simple Email Service Identity (#31140)
* Add aws_ses_identity module

* Update CI alias, add BotoCoreError exception handling.

* Add SES and SNS permissions to hacking/aws_config to run aws_ses_identity integration tests
2018-01-29 15:35:49 -05:00
Marek
5fa29201a7 Port sts_assume_role to boto3 (#32569)
* Ported sts_assume_role to boto3

* Added integration tests
2018-01-22 17:46:08 -05:00
Adrian Likins
ec9769c82f
Facts distribution clear linux 31501 (#32453)
The search string used to look for Clear Linux
was changed in 45a9f96774 to
be more specific, but was too specific. Now finding
a substring match for 'Clear Linux' in /usr/lib/os-release
is enough to consider a match.

Since the details of the full name in os-release varies
('Clear Linux Software for Intel Architecture',
 'Clear Linux OS for Intel Architecture', etc) the
search string match was failing and would fall back to the
'first word in the release file' method resulting in
ansible_distribution='NAME="Clear'

Also add a meta fact indicating which search string
was matched.

Test case info from:
        https://github.com/ansible/ansible/issues/31501#issuecomment-340861535

Fixes #31501
2018-01-20 15:05:53 -05:00
Will Thames
4d58d16793 Add aws_s3 action plugin to find source files as expected (#35028)
People expect to be able to upload files to s3 using standard
locations for files.

Providing an action plugin that effectively rewrites the `src`
key to the result of finding such a file is a great help.

Tests added, and IAM permissions corrected
2018-01-19 17:11:42 -05:00
Will Thames
8d733dbdf0 [cloud] New module cloudfront_distribution (#31284)
* added cloudfont.py, modified cloudfront_facts.py class name and fixed a minor bug

* Improvements to cloudfront_distribution

* Reduce the scope of the cloudfront_distribution module
    * Remove presigning
    * Remove streaming distribution functionality
* Add full test suite for cloudfront distribution
* Meet Ansible AWS guidelines

* Make requested changes

Fix tests

Use built-in waiter

Update copyright
2018-01-17 11:03:23 -05:00
Matt Clay
797664d9cb Python 2.6 str.format() compatibility fixes. 2018-01-10 14:08:11 -08:00
Will Thames
866d7fdce9 [cloud] Create ECS integration test suite (#33757)
Tests for:
* ecs_cluster
* ecs_service
* ecs_service_facts
* ecs_taskdefinition
* ecs_taskdefinition_facts

* Add idempotency testing

Test ecs_cluster, ecs_service and ecs_taskdefinition for trivial
idempotency. Add FIXMEs to the tests because the latter two fail.

Remove unused dependencies
2017-12-15 08:15:01 -05:00
Abhijeet Kasurde
36f82ae8cc Replace exit() with sys.exit()
This fix adds replacement for exit() to sys.exit(), as
exit() is not recommended way to exit from the program.

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2017-12-14 22:03:08 -05:00
Sloane Hertel
f5471b3dcb [cloud] ec2_vpc_net integration tests (#33111)
* Add some integration tests for ec2_vpc_net module

* Add a couple tests for check mode

fix typo

ensure the DHCP option set is cleaned up

* Add permissions to test policy
2017-12-05 16:41:16 -05:00
Matt Martz
783a09d105
Fix failure to adjust my name when doing copy pasta (#33322) 2017-11-27 19:09:05 -06:00
Matt Martz
4fe08441be Deprecate tests used as filters (#32361)
* Warn on tests used as filters

* Update docs, add aliases for tests that fit more gramatically with test syntax

* Fix rst formatting

* Add successful filter, alias of success

* Remove renamed_deprecation, it was overkill

* Make directory alias for is_dir

* Update tests to use proper jinja test syntax

* Update additional documentation, living outside of YAML files, to reflect proper jinja test syntax

* Add conversion script, porting guide updates, and changelog updates

* Update newly added uses of tests as filters

* No underscore variable

* Convert recent tests as filter changes to win_stat

* Fix some changes related to rebasing a few integration tests

* Make tests_as_filters_warning explicitly accept the name of the test, instead of inferring the name

* Add test for tests_as_filters_warning

* Update tests as filters in newly added/modified tests

* Address recent changes to several integration tests

* Address recent changes in cs_vpc
2017-11-27 17:58:08 -05:00
Will Thames
1ca0c0e7f7 Consolidate IAM policies into fewer, larger policies (#33122)
Due to IAM limits allowing at most 10 policies per group,
need to reduce the number of total policies in use.
2017-11-21 17:15:31 -05:00