Commit graph

431 commits

Author SHA1 Message Date
Matt Martz
7a89d373ac Perf graphing (#46346)
* csv of memory usage

* Fix var

* Configurable output file

* Add cpu profiling

* Valdiate the existence of cgroup files

* Add guard to prevent exception when trying to reset max memory value

* to_bytes/to_text and docs updates

* Add support for CPU results

* Just track the max, don't log all results, and then calculate max

* Restore cgroup_memory_recap, and move new functionality into cgroup_perf_recap

* Add pid count tracking, restructure to support more profilers

* Add cli tool for graphing cgroup_perf_recap data

* csv_output_dir is a path

* Correct CALLBACK_NAME

* Include uuid in csv data

* fix linting errors

* Bump version_added

* Create helper funciton to create dict from list of keys, with callable default

* Updated notes to include pids

* Print a newline after each section

* Plugin improvements

* Add option to supporess recap display
* Add default for output directory
* Add option to dictate whether or not to write files
* Add JSON-seq output option

* s/uuid/task_uuid

* Use bytes for paths

* Increase polling interval length for pids/memory

* Reduce instance attrs, change how we invoke profilers

* Shorten some line lengths

* Remove more instance attrs

* Fix some typos

* document directory creation, and catch exceptions

* Enable per task file outputs, and filename customization

* s/per_task_file/file_per_task/g
2019-01-08 13:29:22 -05:00
Toshio Kuratomi
61b1daa65f Port from plaform.dist to ansible.module_utils.distro.linux_distribution
ci_complete
2018-12-17 11:01:01 -08:00
Toshio Kuratomi
3fba006207 Update bare exceptions to specify Exception.
This will keep us from accidentally catching program-exiting exceptions
like KeyboardInterupt and SystemExit.
2018-12-16 15:03:19 -08:00
Ed Costello
b70d5d9aee [AWS] ses rule set module for inbound email processing (#42781)
* Add module ses_rule_set for Amazon SES

* Update behaviours and naming to be consistent with other aws_ses_ modules.

* Add global lock around tests using active rule sets to prevent intermittent test failures.

* Fix deletion of rule sets so that we don't inactivate the active rule set
when force deleting an inactive rule set.
2018-11-14 12:15:24 -05:00
Pierre-Louis Bonicoli
92103bf5d0 test-module: define ansible_version attribute
Executed command:

    ./hacking/test-module -m lib/ansible/modules/cloud/scaleway/scaleway_security_group.py -a ...

Fix this exception found while testing scaleway_security_group module:

    Traceback (most recent call last):
      File "~/debug_dir/__main__.py", line 240, in <module>
        main()
      File "~/debug_dir/__main__.py", line 236, in main
        core(module)
      File "~/debug_dir/__main__.py", line 209, in core
        api = Scaleway(module=module)
      File "~/debug_dir/ansible/module_utils/scaleway.py", line 58, in __init__
        'User-Agent': self.get_user_agent_string(module),
      File "~/debug_dir/ansible/module_utils/scaleway.py", line 99, in get_user_agent_string
        return "ansible %s Python %s" % (module.ansible_version, sys.version.split(' ')[0])
    AttributeError: 'AnsibleModule' object has no attribute 'ansible_version'
2018-10-18 09:45:56 +02:00
Shuang Wang
0c6513e9b1 add module aws_codecommit to represent AWS CodeCommit (#46161)
* kick off

* done for the day

* beta code and test

* fix a typo

* boto3_conn and boto_exception aren't used in this code, ec2_argument_spec is used but unneeded.

* Returning when find a match avoids doing extra work, especially when pagination is involved

* add new permissions for test

* (output is changed) is preferred over accessing the attribute directly.

* pass the result through camel_dict_to_snake_dict() before returning it.

* AnsibleAWSModule automatically merges the argument_spec.

* deletes the created resources even if a test fails.

* AnsibleAWSModule automatically merges the argument_spec.

* fix typo

* fix pep8

* paginate list_repositories

* specify permissions for test

* cut the unnecessary code.

* add return doc string

* add missed ':'

* fix syntax error: mapping values are not allowed here

* add description for return

* fix syntax error

* rename module name and turn off automated integration test.
2018-10-18 14:32:06 +10:00
Matt Martz
49eb53b44d
pylint plugin to catch due/past-due deprecated calls (#44143)
* Start of work on pylint plugin to catch due/past-due deprecated calls

* Improve deprecated pylint plugin

* Catch call to AnsibleModule.deprecate also

* Skip splatted kwargs, we can't infer that info

* Add error for invalid version in deprecation

* Skip version if it's a reference to a var

* Disable ansible-deprecated-no-version for displaying deprecated module info

* fix comments

* is None

* Force specifying a version, this can be disabled on a per case basis

* Disable ansible-deprecated-version by default

* Remove to look for 2.8 deprecated

* Revert "Remove to look for 2.8 deprecated"

This reverts commit 4e84034fd1.

* Add script and template used for creating issues for deprecated issues

* Fix underscore var
2018-09-25 10:31:41 -05:00
Matt Clay
f3d1f9544b
Make ansible-test available in the bin directory. (#45876) 2018-09-19 17:58:55 -07:00
Will Thames
d2569a3f7d Improve iam_group exception handling (#45599)
* Improve iam_group exception handling

Use AnsibleAWSModule for iam_group and handle BotoCoreErrors
as well as ClientErrors. Use fail_json_aws to improve error messages

* Add minimal iam_group test suite

Update some of the read-only IAM permissions (this is not sufficient
to run the test suite but it gets further than it did until it tries
to add a (non-existent) user)

* Clean up after tests
2018-09-17 19:53:44 -04:00
Michael Scherer
83db157c35 Add a default man path (#45689)
It seems that on some Linux distribution (Fedora 28, Debian), man will
not fallback on a default path if MANPATH is set. So using the env-setup
script will prevent man from working.
2018-09-17 18:31:39 -04:00
Sviatoslav Sydorenko
e8731a1f5b
Add a ticket stub for guiding GitHub newcomers
PR #45497
2018-09-11 17:51:54 +02:00
Ed Costello
96c4efcd95 Add missing s3 permissions for s3 module testing. (#43243) 2018-09-10 17:21:36 -04:00
Matt Martz
7a61763fba
Don't require requests in hacking/report.py (#45350)
* Don't require requests in hacking/report.py

* move ansible import
2018-09-07 11:51:41 -05:00
Will Thames
60e3af42d5 sns_topic boto3 port (#39292)
* Port sns_topic to boto3 and add tests
2018-08-23 21:04:18 -04:00
sdubrul
061877d584 added account_alias in the response of module aws_caller_facts (#42345)
* added account_alias in the response of module aws_caller_facts

* added comment to explain list_account_aliases

* renamed caller_identity to caller_facts as the content is extended

* created changelog

* security-policy needs the iam:ListAccountAliases for this module to work

* test now checks for the added field account_alias

* gracefully handle missing iam:ListAccountAliases permission
2018-08-22 17:21:12 -04:00
Matt Clay
0392dbeba1 Fix path handling in hacking/env-setup. 2018-08-10 23:36:56 -07:00
Matt Martz
c1c229c6d4
Remove use of simplejson throughout code base (#43548)
* Remove use of simplejson throughout code base. Fixes #42761

* Address failing tests

* Remove simplejson from contrib and other outlying files

* Add changelog fragment for simplejson removal
2018-08-10 11:13:29 -05:00
Joren Vrancken
b954917761 Surround top-level function and class definitions with two blank lines. 2018-07-31 12:06:56 -07:00
flowerysong
a08668cf00 Port ec2_tag to boto3 (#39712)
* Add volume manipulation to EC2 integration test policy

* Port ec2_tag to boto3
2018-07-27 15:45:18 -04:00
Toshio Kuratomi
52449cc01a AnsiballZ improvements
Now that we don't need to worry about python-2.4 and 2.5, we can make
some improvements to the way AnsiballZ handles modules.

* Change AnsiballZ wrapper to use import to invoke the module
  We need the module to think of itself as a script because it could be
  coded as:

      main()

  or as:

      if __name__ == '__main__':
          main()

  Or even as:

      if __name__ == '__main__':
          random_function_name()

  A script will invoke all of those.  Prior to this change, we invoked
  a second Python interpreter on the module so that it really was
  a script.  However, this means that we have to run python twice (once
  for the AnsiballZ wrapper and once for the module).  This change makes
  the module think that it is a script (because __name__ in the module ==
  '__main__') but it's actually being invoked by us importing the module
  code.

  There's three ways we've come up to do this.
  * The most elegant is to use zipimporter and tell the import mechanism
    that the module being loaded is __main__:
    * 5959f11c9d/lib/ansible/executor/module_common.py (L175)
    * zipimporter is nice because we do not have to extract the module from
      the zip file and save it to the disk when we do that.  The import
      machinery does it all for us.
    * The drawback is that modules do not have a __file__ which points
      to a real file when they do this.  Modules could be using __file__
      to for a variety of reasons, most of those probably have
      replacements (the most common one is to find a writable directory
      for temporary files.  AnsibleModule.tmpdir should be used instead)
      We can monkeypatch __file__ in fom AnsibleModule initialization
      but that's kind of gross.  There's no way I can see to do this
      from the wrapper.

  * Next, there's imp.load_module():
    * https://github.com/abadger/ansible/blob/340edf7489/lib/ansible/executor/module_common.py#L151
    * imp has the nice property of allowing us to set __name__ to
      __main__ without changing the name of the file itself
    * We also don't have to do anything special to set __file__ for
      backwards compatibility (although the reason for that is the
      drawback):
    * Its drawback is that it requires the file to exist on disk so we
      have to explicitly extract it from the zipfile and save it to
      a temporary file

  * The last choice is to use exec to execute the module:
    * https://github.com/abadger/ansible/blob/f47a4ccc76/lib/ansible/executor/module_common.py#L175
    * The code we would have to maintain for this looks pretty clean.
      In the wrapper we create a ModuleType, set __file__ on it, read
      the module's contents in from the zip file and then exec it.
    * Drawbacks: We still have to explicitly extract the file's contents
      from the zip archive instead of letting python's import mechanism
      handle it.
    * Exec also has hidden performance issues and breaks certain
      assumptions that modules could be making about their own code:
      http://lucumr.pocoo.org/2011/2/1/exec-in-python/

  Our plan is to use imp.load_module() for now, deprecate the use of
  __file__ in modules, and switch to zipimport once the deprecation
  period for __file__ is over (without monkeypatching a fake __file__ in
  via AnsibleModule).

* Rename the name of the AnsiBallZ wrapped module
  This makes it obvious that the wrapped module isn't the module file that
  we distribute.  It's part of trying to mitigate the fact that the module
  is now named __main)).py in tracebacks.

* Shield all wrapper symbols inside of a function
  With the new import code, all symbols in the wrapper become visible in
  the module.  To mitigate the chance of collisions, move most symbols
  into a toplevel function.  The only symbols left in the global namespace
  are now _ANSIBALLZ_WRAPPER and _ansiballz_main.

revised porting guide entry

Integrate code coverage collection into AnsiballZ.

ci_coverage
ci_complete
2018-07-26 20:07:25 -07:00
Calvin Wu
7e42e88cc1 ecs_taskdefinition can absent without containers argument (#41398)
* ecs_taskdefinition can absent without containers argument

* add regression test for absent with arn

* Add PassRole privilege for ecs_cluster to pass
2018-07-12 23:16:41 +10:00
Julien Vey
0f612d1b76 efs_facts: improve performance by reducing the number of api calls (#36520)
* efs_facts: improve performance by reducing the number of api calls

* Remove efs_facts tests from running in CI
2018-07-08 16:34:22 -04:00
Troy Murray
15ce7c5bab change OS X to macOS (#41294)
* change OS X to macOS

<!--- Your description here -->

+label: docsite_pr

* Update all Mac OS X references to be macOS

* Drop extra Mac
2018-06-26 14:09:23 -04:00
Jon Dufresne
dc7e50fa90 Update additional pypi.python.org URLs to pypi.org (#41373)
For details on the new PyPI, see the blog post:

https://pythoninsider.blogspot.ca/2018/04/new-pypi-launched-legacy-pypi-shutting.html
2018-06-17 14:01:18 +02:00
Will Thames
b235cb8734 aws_eks_cluster: New module for managing AWS EKS (#41183)
* aws_eks: New module for managing AWS EKS

aws_eks module is used for creating and removing EKS clusters.

Includes full test suite and updates to IAM policies to enable it.

* Clean up all security groups

* appease shippable

* Rename aws_eks module to aws_eks_cluster
2018-06-07 08:44:04 -04:00
Will Thames
a60fe1946c Remove ECS policies from AWS compute policy
The compute policy was exceeding maximum size and contained
policies that already exist in ecs-policy.

Look up suitable AMIs rather than hardcode

We don't want to maintain multiple image IDs for multiple regions
so use ec2_ami_facts to set a suitable image ID

Improve exception handling
2018-06-06 20:51:50 +10:00
Zhikang Zhang
b578bf9e20 Fix test-module failing to validate args (#41004)
* Fix test-module failing to validate args

The test-module pass a wrong argument _ansible_tmp cause the validation failed.
Change the argument _ansible_tmp to _ansible_tmpdir to fix this.

* Add a integration test for test-module.

Prior to this change, we don't have a test for test-module.

This change ensure the correctness of test-module script.
2018-06-01 12:02:56 -07:00
Will Thames
809c7404ab Add two missing VPC permissions (#37896)
Remove VPC permissions from network-policy.json as they mostly duplicate
compute-policy.json permissions - separating the VPC and compute permissions
would likely lead to further confusion.
2018-05-25 06:31:54 -04:00
Matt Clay
8deced3e04
Fix shebangs and file modes and update tests. (#40563)
* Add execute bit sanity test and apply fixes.
* Add shebang test for `lib` dirs and apply fixes.
* Shebang and execute bit cleanup.
2018-05-22 14:25:36 -07:00
Matt Davis
e4edb2842a 2.6 changelog gen/version/root dir cleanup (#40421)
* patched in changelog gen stuff from stable-2.5
* Makefile updates
* release.py as single-source-of-truth
* Remove obsolete ansible-core-sitemap.xml file.
* Move ROADMAP.rst into README.rst.
* dynamic rpm changelog, zap old deb/rpm changelogs
* fix changelog in MANIFEST.in
* Remove obsolete hacking/update.sh script.
* Remove ref to deleted authors script.
* Remove ref to removed module-formatter script.
* Update headings to match script names.
* MANIFEST.in cleanup
* removed RELEASES.txt and versions.yml
* removed obsolete release generation playbook/bits (not used since 2.5)
* misc Makefile cleanup
* speculative changes to DEB versioning
* allow override of DEB_VERSION/DEB_RELEASE
2018-05-21 16:14:53 -07:00
Toshio Kuratomi
5634dae290 Remove the cherrypick script
We only needed it for migrating cherrypicks between the unified repo and
the ansible-modules-* repos.  Now that we aren't supporting 2.3, we no
longer need this script.
2018-05-21 14:50:24 -05:00
Tim Rupp
548282139f
Fixes incorrect variable name (#40274)
Incorrect variable name was causing a NameError

  NameError: name 'comlpex_args' is not defined
2018-05-16 12:13:22 -07:00
Madhura-CSI
b85970b2b0 New Module: ec2_vpc_vpn_facts (#35983)
* New module: ec2_vpc_vpn_facts

* Add integration tests for ec2_vpc_vpn_facts and the IAM permissions

* Add retry to VPC removal

* Use unique name for VGW

* Always clean up after tests and add retries
2018-05-15 12:13:46 -04:00
Jordan Borean
44ab948e5d
create module tmpdir based on remote_tmp (#39833)
* create module tmpdir based on remote_tmp

* Source remote_tmp from controller if possible

* Fixed sanity test and not use lambda

* Added expansion of env vars to the remote tmp

* Fixed sanity issues

* Added note around shell remote_tmp option

* Changed fallback tmp dir to ~/.ansible/tmp to make shell defaults
2018-05-15 09:31:21 +10:00
Will Thames
29770a297a Fail with nice error message if elb target_type=ip not supported (#38313)
* Add helpful failure message if target_type=ip is not supported

Create test case for target_type=ip not supported

* Update elb_target_group module to latest standards

Use AnsibleAWSModule
Improve exception handling
Improve connection handling
2018-05-03 08:36:52 -04:00
mwpeterson
efdd92e1c0 Update test-module (#39331)
Update test-module To use C.DEFAULT_LOCAL_TMP
2018-04-26 07:16:33 -07:00
Will Thames
12f2b9506d [aws]Add VPC configuration to ECS modules (#34381)
Enable awsvpc network mode for ECS services and tasks and
their underlying task definitions

Improve test suite to thoroughly test the changes

Use runme.sh technique to run old and new versions of botocore to
ensure that the modules work with older botocore and older network modes
and fail gracefully if awsvpc network mode is used with older botocore
2018-04-25 15:41:04 -04:00
Matt Clay
c262dbfd30 Use https for links to ansible.com domains. 2018-04-23 11:33:56 -07:00
Ed Costello
0d31d1cd24 [cloud]Add aws_ses_identity_policy module for managing SES sending policies (#36623)
* Add aws_ses_identity_policy module for managing SES sending policies

* Add option to AnsibleAWSModule for applying a retry decorator to all calls.

* Add per-callsite opt in to retry behaviours in AnsibleAWSModule

* Update aws_ses_identity_policy module to opt in to retries at all callsites.

* Add test for aws_ses_identity_policy module with inline policy.

* Remove implicit retrys on boto resources since they're not working yet.
2018-04-05 15:11:12 -04:00
Will Thames
a1d3cf488d [cloud][test]Add missing IAM policy for cloudfront (#38248)
Cloudfront needs CreateOriginAccessIdentity

Add profile parameter to setup-iam.yml. Could arguably just use
AWS_PROFILE but given that other tasks are using profile, should
be consistent.
2018-04-05 14:06:04 -04:00
Pilou
7908f78fa6 module_common: handle None value for templar (#36651)
* module_common: set required parameter templar

Fix the following error (related to b455901):

  $ ./hacking/test-module -m ./lib/ansible/modules/system/ping.py -I ansible_python_interpreter=/usr/bin/python
  Traceback (most recent call last):
    File "./hacking/test-module", line 268, in <module>
      main()
    File "./hacking/test-module", line 249, in main
      (modfile, modname, module_style) = boilerplate_module(options.module_path, options.module_args, interpreters, options.check, options.filename)
    File "./hacking/test-module", line 152, in boilerplate_module
      task_vars=task_vars
    File "ansible/lib/ansible/executor/module_common.py", line 910, in modify_module
      environment=environment)
    File "ansible/lib/ansible/executor/module_common.py", line 736, in _find_module_utils
      shebang, interpreter = _get_shebang(u'/usr/bin/python', task_vars, templar)
    File "ansible/lib/ansible/executor/module_common.py", line 452, in _get_shebang
      interpreter = templar.template(task_vars[interpreter_config].strip())
  AttributeError: 'NoneType' object has no attribute 'template'

* module_common.modify_module: templar is required
2018-03-29 13:54:48 -04:00
Julien Vey
7c07877b1b s3_bucket: add integration tests (#36941)
Also update testing-policies/storage
2018-03-07 11:25:24 -05:00
Julien Vey
51d491f8f0 route53_zone: move to boto3, and enable comment update (#36641) 2018-03-07 11:16:04 -05:00
Ed Costello
645952c139 Add aws_caller_facts module and use it in setup-iam.yml (#36683)
* Add aws_caller_facts module and use it in setup-iam.yml

This removes the dependency on having the command line AWS tools
installed.
2018-02-28 16:30:34 +10:00
Will Thames
b5a1643e3d Add new aws_waf_condition module (#33110) 2018-02-01 18:16:27 -05:00
Ed Costello
d16bc1c3f4 New aws_ses_identity module to manage AWS Simple Email Service Identity (#31140)
* Add aws_ses_identity module

* Update CI alias, add BotoCoreError exception handling.

* Add SES and SNS permissions to hacking/aws_config to run aws_ses_identity integration tests
2018-01-29 15:35:49 -05:00
Marek
5fa29201a7 Port sts_assume_role to boto3 (#32569)
* Ported sts_assume_role to boto3

* Added integration tests
2018-01-22 17:46:08 -05:00
Adrian Likins
ec9769c82f
Facts distribution clear linux 31501 (#32453)
The search string used to look for Clear Linux
was changed in 45a9f96774 to
be more specific, but was too specific. Now finding
a substring match for 'Clear Linux' in /usr/lib/os-release
is enough to consider a match.

Since the details of the full name in os-release varies
('Clear Linux Software for Intel Architecture',
 'Clear Linux OS for Intel Architecture', etc) the
search string match was failing and would fall back to the
'first word in the release file' method resulting in
ansible_distribution='NAME="Clear'

Also add a meta fact indicating which search string
was matched.

Test case info from:
        https://github.com/ansible/ansible/issues/31501#issuecomment-340861535

Fixes #31501
2018-01-20 15:05:53 -05:00
Will Thames
4d58d16793 Add aws_s3 action plugin to find source files as expected (#35028)
People expect to be able to upload files to s3 using standard
locations for files.

Providing an action plugin that effectively rewrites the `src`
key to the result of finding such a file is a great help.

Tests added, and IAM permissions corrected
2018-01-19 17:11:42 -05:00
Will Thames
8d733dbdf0 [cloud] New module cloudfront_distribution (#31284)
* added cloudfont.py, modified cloudfront_facts.py class name and fixed a minor bug

* Improvements to cloudfront_distribution

* Reduce the scope of the cloudfront_distribution module
    * Remove presigning
    * Remove streaming distribution functionality
* Add full test suite for cloudfront distribution
* Meet Ansible AWS guidelines

* Make requested changes

Fix tests

Use built-in waiter

Update copyright
2018-01-17 11:03:23 -05:00