Commit graph

468 commits

Author SHA1 Message Date
Mark Chappell
77e4371460 aws_kms: Update policy on existing keys (when passed) (#60059)
* aws_kms: (integration tests) Use module_defaults to reduce the copy and paste

* aws_kms: (integration tests) make sure policy option functions.

* aws_kms: (integration tests) Move iam_role creation to start of playbook.

iam_roles aren't fully created when iam_role completes, there's a delay on the Amazon side before they're fully recognised.

* aws_kms: Update policy on existing keys (when passed)
2019-08-23 20:38:38 +10:00
Mark Chappell
70777020c4 Fix iam_password_policy integration tests (#60930)
* iam_password_policy: (integration tests) Use module defaults for AWS connection details

* iam_password_policy: (integration tests) Ensure the policy is removed when tests fail

* iam_password_policy: (integration tests) Add regression test for #59102

* iam_password_policy: Only return changed when the policy changes.

* iam_password_policy: PasswordReusePrevention must be omitted to remove/set to 0

* #60930 add changelog

* Update hacking AWS security policy to allow testing of Password Policy Management
2019-08-22 23:25:25 +10:00
Toshio Kuratomi
d9b3af523b Galaxy meta docs table (#60171)
* Use an rst table instead of a raw html table

* Rst is easier to read so we want to use it wherever possible
* Fix the jinja2 filters which create links so that they do not include
  extraneous whitespace in the URL

* Normalize description data before sending them to the templates
2019-08-13 15:00:13 -05:00
Matt Clay
d651bda123
Relocate ansible-test code. (#60147)
* Initial move of `test/runner/` content.

`test/runner/lib/` -> `test/lib/ansible_test/_internal/`
`test/runner/`     -> `test/lib/ansible_test/_internal/data/`

* Initial move of `test/sanity/` content.

`test/sanity/` -> `test/lib/ansible_test/_internal/data/sanity/` (except `test/sanity/ignore.txt`)

* Initial move of `test/units/pytest/` content.

`test/units/pytest/` -> `test/lib/ansible_test/_internal/data/pytest/`

* Follow-up move of `test/runner/unit/` content.

`test/lib/ansible_test/_internal/data/unit/` -> `test/lib/ansible_test/tests/unit/`

* Initial move of `ansible.cfg` content.

`test/units/ansible.cfg` -> `test/lib/ansible_test/_internal/data/units/ansible.cfg`
`test/env/ansible.cfg` -> `test/lib/ansible_test/_internal/data/env/ansible.cfg`

* Follow-up move of `data` directory.

`test/lib/ansible_test/_internal/data/` -> `test/lib/ansible_test/_data/`

* Update import statements.

* Add missing __init__.py for unit tests.

* Fix path references and miscellaneous issues.
2019-08-06 14:43:29 -07:00
Jesse Evers
e410dcbfed Add logic to handle multiple actions in an ALB listener rule, Fixes #41861 (#41975)
* added logic to handle multiple actions in an ALB listener rule (#41861)

* fix linting and pep8 issues

* added test for multiple actions using OIDC authentication

* added error messages related to old versions of botocore and multiple actions

* fix action validation error checks (need to check the exception string)

* added logic to make oidc configs idempotent (remove clientsecret for check)

* modified TargetGroupName to TargetGroupArn substitution to account for multiple rule actions

* refactored tests so that it can be run against different versions of botocore

* fix runme.sh to refelct changes to cloud testsuite

* add UseExistingClientSecret to oidc config (AWS api change)

* remove tests for OIDC auth action; add tests for redirect and fixed-response

* add in fixes from markuman and mjmayer

* remove documentation for cognito integration (not sure how to test); added example config for fixed-response and redirect actions

* renamed oidc/multiple action tests; leaving commented due to some AWS API changes

* pep8 fix

* more pep8 fixes

* Restructure elb_application_lb test suite

Move from runme.sh to virtualenv based roles

Update policies to fix tests

Don't log temp dir deletion, so many files in the diff!
2019-08-02 10:40:53 +10:00
Toshio Kuratomi
0e9f002073 Fix build-ansible.py release-announcement
* When the release scripts were changed, this file to implement release
  announcements was left out

* Add announce.py to the compile skip lists as only the release engineers
  run this during package build.
2019-07-26 16:39:56 -07:00
Will Thames
60c76be03c rds_instance: add point-in-time instance restore test (#59411) 2019-07-25 09:49:08 -04:00
Jordan Borean
65049620ee
Generate galaxy.yml based on single source of truth (#59170)
* Generate galaxy.yml based on single source of truth

* Fix up tests and align file names

* Minor Makefile tweak

* Remove link in galaxy.yml file and make it a template file

* Moved collections docs to dev_guide

* change Makefile clean path

* Added readme to example meta file

* review fixes

* Use newer style for doc generation script

* Fix mistake in dev_guide index

* removed uneeded file, fixed links and added preview banner

* Moved banner for sanity test
2019-07-23 06:50:46 +10:00
Aljaž Košir
42073b6331 Add lambda_bucket_event module (#58059) 2019-07-18 10:54:01 -07:00
Toshio Kuratomi
5392d8697d
Docs scripts cleanup (#59169)
* Fix dump_keywords to require attribute docs

* Change the documentation commands to make definition files positional

Since the definition files are mandatory, make them positional
parameters instead of options.
2019-07-17 12:01:07 -07:00
Toshio Kuratomi
019d078a5a
Move common build code from _build_helpers (#55986)
We have some common code used by several docs scripts.  Migrate that
into the build-only shared code repository.

* Move lib/ansible/utils/_build_helpers.py to the directory for common
  build code
* Migrate docs/bin/dump_config.py to a build-ansible subcommand
* Migrate dump_keywords to the build-ansible framework
  * Make the script more maintainable by using functions and good
    variable names
  * Port to Python3 idioms
  * Fix bug so that private attributes will be undocumented
* Move generate_man to a build-ansible subcommand
* Port plugin_formatter to a build-ansible subcommand
* Rework command_plugins so that docs scripts can target Python-3.4+ and
  releng-only subcommands can use more recent versions of Python.
  The architecture is now that command_plugins/* need to be importable
  on Python-3.4.  The init_parsers() method needs to run on Python-3.4.
  But the main() method can utilize features of more recent Python as
  long as it fits within those parameters.
* Update docs build requirements

Port the plugin_formatter to build-ansible framework
2019-07-16 12:19:01 -07:00
Toshio Kuratomi
33d2728879 Rename python files in hacking/ directory to have .py suffix
ansible-test only passes files which have the .py suffix for sanity
tests on python files.  This change will allow sanity tests to run on
the Python files in hacking/

* Rename test-module to test-module.py
* Symlink test-module for backwards compat since end users may be using
  test-module
* Fix test-module sanity errors that are now triggered
* Rename ansible_profile to ansible-profile.py
* Rename build-ansible
2019-07-10 22:17:35 -07:00
Will Thames
60fb9fc208 Fix EC2 test suite to work with testing policies (#44387)
* Update testing policies to ensure all required permissions are present
* Tidy up security policies to reduce duplicate permissions
* Make roles static so that they can be present before CI is run,
  meaning that role creation permission is not required by the CI
  itself, only by someone setting up the roles prior to testing
* Move contents to cloudfront policy to network policy to ensure policy
  count (maximum of 10) stays low
* Maintain compute policy below 6144 bytes
2019-07-04 15:25:19 -04:00
Will Thames
a6d757e074 Ensure block device instance creation outputs instances
Not waiting outputs results in a format that will never
be matched by the tests

Ensure instances get tidied up

Allow ec2:ReportInstanceStatus

ec2_instance: Improve test cleanup on failing tests

Improve describe/modify attribute error handling

Address feedback on PR
2019-07-02 17:48:07 -04:00
Sam Doran
ab6a9ef130
Get minor version number for CentOS and Debian (#57814)
* Get the most detailed version number from distro.version() for CentOS and Debian
* Update tests and fixtures
* Update fixture generation script to gather distro info and work with Python 3
* Update LinuxMint fixtures
* Cleanup fixture formatting
* Improvements based on feedback from abadger:
    - use unicode since that is what distro returns
    - use frozenset with a tuple
    - include link Debian to bug
2019-06-28 19:21:35 -04:00
Elijah Lynn
19c6448459 Fix "no matches for wildcard" error in env-setup.fish script. (#58345)
See https://github.com/ansible/ansible/pull/58345#discussion_r297288979
2019-06-27 18:15:11 -04:00
Will Thames
eda5dd826f Add rds_snapshot module (#39994)
* new module uses modern ansible AWS standards
* adds additional tests for snapshots
* Update return_skeleton_generator for python3 - should
  set type to `str`, not `string`.
2019-06-26 16:27:11 -07:00
Will Thames
a09aa205e1 Fix RDS test suite and minor bugs revealed (#57940)
* Update testing policy to be correct for RDS test suite
* Create read replica in same region to avoid more permissions being
  required
* Ensure modifying DB doesn't try to downgrade engine version
* Add tags to main test suite to limit number of tests run for problem
  solving
2019-06-25 09:54:03 -04:00
Kohei Asano
ddf6d096c5 Support the new TLS termination on NLBs (#51327) (#58031) 2019-06-20 17:03:04 -07:00
Will Thames
924352a051 ecs_cluster test suite refactor (#57716)
* Combine testing policies

Because of the maximum of 10 policies per group, need to
consolidate testing policies as best we can.

* Tidy put-account-setting tasks and add permission

Using `environment` and `command` rather than `shell` avoids the
need for `no_log` and means that people can fix the problem

* refactor ecs_cluster test suite

move from runme.sh technique to virtualenv

use ec2_instance rather than ec2 module to
avoid need for boto
2019-06-17 11:41:20 -07:00
Stefan Horning
77ec0549b0 New module for AWS CodeBuild (#47187)
* New AWS module for the CodeBuild service, called aws_codebuild

* Integration test for new aws_codebuild module
2019-06-12 10:25:53 -04:00
Ed Costello
2013d4abc4 Update setup-iam playbook to use aws_caller_info rather than deprecated (#57675)
aws_caller_facts
2019-06-11 09:23:56 -04:00
Stefan Horning
ddfaa83ccf s3_bucket: add encryption capabilities to the module (#55985)
* s3_bucket: add encryption capabilities to the module
2019-06-03 11:25:09 -07:00
mjmayer
c8e179fbf1 Aws waf region (#48953)
* Add waiter for AWSRegional

* Add support for WAF Regional

* Add support for regional waf web acl

* Remove set_trace, pep formatting

* Add paginator for regional_waf

* Change name of param for waf_regional

This is more in line with how AWS refers to the service. Additional
 changes made to how client is called. Used ternary to reduce if
 statements

* Change parameter name to waf_regional

* Add support for removal waf regional condition

* Change parameter from cloudfront to waf_regional

* Added state: absent waf rule

* Remove set_trace

* Add integration tests for waf regional

* WIP: adding region parameter to tests

* Add support for waf facts module

* Add region to waf regional integration tests

* Update security policy for waf regional testing

* Add type to documentation for waf_regional param
2019-05-17 10:36:14 +10:00
Toshio Kuratomi
3161a91d7e
Implement a framework for having common code for release scripts (#55893)
* Implement a framework for having common code for release scripts

* Release scripts will go through hacking/build-ansible.  build-ansible is
  a pluggable script which will set a directory that has common code for
  non-enduser scripts.  It will then invoke the plugin which implements
  that subcommand.  Uses straight.plugin for loading each sub-command.

* We're going to add tools which are needed to test ansible (the changelog
  generation, for instance) so we need to include the pieces relevant to
  that in the tarball.

* Add straight.plugin to the sanity test requirements for the same
  reason

* Skip compile test just for build-ansible plugins which won't be run as
  part of sanity tests.
2019-05-01 13:57:03 -05:00
Matt Martz
65dcb4242a
Fix deprecated issue creator (#55327) 2019-04-15 15:28:25 -05:00
Toshio Kuratomi
e5a31e81b6 Add a script to generate a new porting guide
* porting-guide.py is allowed to be Python-3.6+ as it will only be run
  by release engineers
2019-04-12 12:57:35 -07:00
Karolis Tamutis
54384e7a12 Make test-module use default value for interpreter (#54053)
* Make test-module use default value for interpreter

* Changing from static interpreter path to sys.executable as per #54053

* A little ntegration test for #54053
2019-04-10 09:04:49 -05:00
Toshio Kuratomi
8dd46d6f95
Fix punctuation around issue reporting link (#54295)
Figure out how to format the release announcement so a link isn't
directly followed by a period which would hinder cutand paste but uses
proper grammar and punctuation.
2019-03-25 07:35:45 -07:00
Toshio Kuratomi
5fb416ae34 Add a script to generate twitter and mailing list announcements
Announcements taken from https://github.com/ansible/community/wiki/RelEng:-ReleaseProcess
and then cleaned up:

* Update issue reporting blurb from feedback from acozine and gundalow
* Add a subject and to line for email output
* Ignore long line tests on the jinja templates (as jinja doesn't give
  enough control to get rid of newlines when text wrapping)
* Skip shebang and compile tests for older pythons since this is a
  release engineer-only script.  (ok'd by mattclay)
2019-03-22 11:23:54 -07:00
Sloane Hertel
7da565b3ae
parse botocore.endpoint logs into a list of AWS actions (#49312)
* Add an option to parse botocore.endpoint logs for the AWS actions performed during a task

Add a callback to consolidate all AWS actions used by modules

Added some documentation to the AWS guidelines

* Enable aws_resource_actions callback only for AWS tests

* Add script to help generate policies

* Set debug_botocore_endpoint_logs via environment variable for all AWS integration tests

Ensure AWS tests inherit environment

(also remove AWS CLI in aws_rds inventory tests and use the module)
2019-03-18 08:29:03 -05:00
Bob Boldin
b67505d271 AWS: new module ec2_transit_gateway fixes #49376 (#53651)
* AWS: new module ec2_transit_gateway fixes #49376

* Add permissions neeeded for integration tests

* uncomment nolog on creds

* add unsupported to integration test aliases

* remove the shippable/aws/group alias so doesn't conflict with unsupported
2019-03-14 09:42:33 +10:00
Andrea Tartaglia
5c6b16edc3 Fix ec2_instance eventual consistency when wait: false (#51885)
* Do not return 'instances' when wait is false

* Added integration tests for wait: false

* Added changelog fragment

* Fix test suite to work with ec2_instance

* Additional permissions
* Enforce boto3 version
* Fix broken tests
* Improve error messages

* fix linter issues
2019-03-06 22:46:37 +10:00
Tad Merchant
b979b26a74 Add launch type to ecs task (#49081)
* adds fargate launch_type to ecs_task module

* White space changes

* fix documentation for running ecs task on fargate

* remove extraneous example from ecs_task

* White space changes

* Adds changelog fragment

* Pluralize minor_changes in changelog fragment

* Add Stop and Start task permissions
2019-03-06 22:40:32 +10:00
Rafael Driutti
c68838fb13 AWS Redshift: port module to boto3 and fix parameters check (#37052)
* fix parameters check and port module to boto3

* begin with integration tests

* allow redshift iam policy

* Wait for cluster to be created before moving on to delete it

* Allow sts credentials so this can be run in CI

Don't log credentials

ensure cluster can be removed

* - Replace DIY waiters with boto3 waiters
- test multi node cluster

* catch specific boto3 error codes

* remove wait from test

* add missing alias for shippable

* - Rework modify function.
- Default unavailable parameters to none.
- Add cluster modify test

* Ensure resources are cleaned up if tests fail

* Ensure all botocore ClientError and BotoCoreError exceptions are handled
2019-02-21 17:04:42 -06:00
Will Thames
46fbcf08bc
aws_kms enhancements (#31960)
* Allow creation and deletion of keys (deletion just schedules for
  deletion, recreating an old key is just cancelling its deletion)
* Allow grants to be set, thus enabling encryption contexts to be
  used with keys
* Allow tags to be added and modified
* Add testing for KMS module
* Tidy up aws_kms module to latest standards
2019-02-13 13:06:58 +10:00
Toshio Kuratomi
dfd8b659c0 Have update-bundled check for updates to all bundled code
Add a test for whether we have bundled code inside of ansible that needs to be updated
2019-02-06 13:59:55 -08:00
Matt Martz
7a89d373ac Perf graphing (#46346)
* csv of memory usage

* Fix var

* Configurable output file

* Add cpu profiling

* Valdiate the existence of cgroup files

* Add guard to prevent exception when trying to reset max memory value

* to_bytes/to_text and docs updates

* Add support for CPU results

* Just track the max, don't log all results, and then calculate max

* Restore cgroup_memory_recap, and move new functionality into cgroup_perf_recap

* Add pid count tracking, restructure to support more profilers

* Add cli tool for graphing cgroup_perf_recap data

* csv_output_dir is a path

* Correct CALLBACK_NAME

* Include uuid in csv data

* fix linting errors

* Bump version_added

* Create helper funciton to create dict from list of keys, with callable default

* Updated notes to include pids

* Print a newline after each section

* Plugin improvements

* Add option to supporess recap display
* Add default for output directory
* Add option to dictate whether or not to write files
* Add JSON-seq output option

* s/uuid/task_uuid

* Use bytes for paths

* Increase polling interval length for pids/memory

* Reduce instance attrs, change how we invoke profilers

* Shorten some line lengths

* Remove more instance attrs

* Fix some typos

* document directory creation, and catch exceptions

* Enable per task file outputs, and filename customization

* s/per_task_file/file_per_task/g
2019-01-08 13:29:22 -05:00
Toshio Kuratomi
61b1daa65f Port from plaform.dist to ansible.module_utils.distro.linux_distribution
ci_complete
2018-12-17 11:01:01 -08:00
Toshio Kuratomi
3fba006207 Update bare exceptions to specify Exception.
This will keep us from accidentally catching program-exiting exceptions
like KeyboardInterupt and SystemExit.
2018-12-16 15:03:19 -08:00
Ed Costello
b70d5d9aee [AWS] ses rule set module for inbound email processing (#42781)
* Add module ses_rule_set for Amazon SES

* Update behaviours and naming to be consistent with other aws_ses_ modules.

* Add global lock around tests using active rule sets to prevent intermittent test failures.

* Fix deletion of rule sets so that we don't inactivate the active rule set
when force deleting an inactive rule set.
2018-11-14 12:15:24 -05:00
Pierre-Louis Bonicoli
92103bf5d0 test-module: define ansible_version attribute
Executed command:

    ./hacking/test-module -m lib/ansible/modules/cloud/scaleway/scaleway_security_group.py -a ...

Fix this exception found while testing scaleway_security_group module:

    Traceback (most recent call last):
      File "~/debug_dir/__main__.py", line 240, in <module>
        main()
      File "~/debug_dir/__main__.py", line 236, in main
        core(module)
      File "~/debug_dir/__main__.py", line 209, in core
        api = Scaleway(module=module)
      File "~/debug_dir/ansible/module_utils/scaleway.py", line 58, in __init__
        'User-Agent': self.get_user_agent_string(module),
      File "~/debug_dir/ansible/module_utils/scaleway.py", line 99, in get_user_agent_string
        return "ansible %s Python %s" % (module.ansible_version, sys.version.split(' ')[0])
    AttributeError: 'AnsibleModule' object has no attribute 'ansible_version'
2018-10-18 09:45:56 +02:00
Shuang Wang
0c6513e9b1 add module aws_codecommit to represent AWS CodeCommit (#46161)
* kick off

* done for the day

* beta code and test

* fix a typo

* boto3_conn and boto_exception aren't used in this code, ec2_argument_spec is used but unneeded.

* Returning when find a match avoids doing extra work, especially when pagination is involved

* add new permissions for test

* (output is changed) is preferred over accessing the attribute directly.

* pass the result through camel_dict_to_snake_dict() before returning it.

* AnsibleAWSModule automatically merges the argument_spec.

* deletes the created resources even if a test fails.

* AnsibleAWSModule automatically merges the argument_spec.

* fix typo

* fix pep8

* paginate list_repositories

* specify permissions for test

* cut the unnecessary code.

* add return doc string

* add missed ':'

* fix syntax error: mapping values are not allowed here

* add description for return

* fix syntax error

* rename module name and turn off automated integration test.
2018-10-18 14:32:06 +10:00
Matt Martz
49eb53b44d
pylint plugin to catch due/past-due deprecated calls (#44143)
* Start of work on pylint plugin to catch due/past-due deprecated calls

* Improve deprecated pylint plugin

* Catch call to AnsibleModule.deprecate also

* Skip splatted kwargs, we can't infer that info

* Add error for invalid version in deprecation

* Skip version if it's a reference to a var

* Disable ansible-deprecated-no-version for displaying deprecated module info

* fix comments

* is None

* Force specifying a version, this can be disabled on a per case basis

* Disable ansible-deprecated-version by default

* Remove to look for 2.8 deprecated

* Revert "Remove to look for 2.8 deprecated"

This reverts commit 4e84034fd1.

* Add script and template used for creating issues for deprecated issues

* Fix underscore var
2018-09-25 10:31:41 -05:00
Matt Clay
f3d1f9544b
Make ansible-test available in the bin directory. (#45876) 2018-09-19 17:58:55 -07:00
Will Thames
d2569a3f7d Improve iam_group exception handling (#45599)
* Improve iam_group exception handling

Use AnsibleAWSModule for iam_group and handle BotoCoreErrors
as well as ClientErrors. Use fail_json_aws to improve error messages

* Add minimal iam_group test suite

Update some of the read-only IAM permissions (this is not sufficient
to run the test suite but it gets further than it did until it tries
to add a (non-existent) user)

* Clean up after tests
2018-09-17 19:53:44 -04:00
Michael Scherer
83db157c35 Add a default man path (#45689)
It seems that on some Linux distribution (Fedora 28, Debian), man will
not fallback on a default path if MANPATH is set. So using the env-setup
script will prevent man from working.
2018-09-17 18:31:39 -04:00
Sviatoslav Sydorenko
e8731a1f5b
Add a ticket stub for guiding GitHub newcomers
PR #45497
2018-09-11 17:51:54 +02:00
Ed Costello
96c4efcd95 Add missing s3 permissions for s3 module testing. (#43243) 2018-09-10 17:21:36 -04:00
Matt Martz
7a61763fba
Don't require requests in hacking/report.py (#45350)
* Don't require requests in hacking/report.py

* move ansible import
2018-09-07 11:51:41 -05:00