Commit graph

539 commits

Author SHA1 Message Date
Toshio Kuratomi
9dda393d70
Collections docs generation (#59761)
* Build documentation for Ansible-2.10 (formerly known as ACD).

Builds plugin docs from collections whose source is on galaxy

The new command downloads collections from galaxy, then finds the
plugins inside of them to get the documentation for those plugins.

* Update the python syntax checks
  * docs builds can now require python 3.6+.

* Move plugin formatter code out to an external tool, antsibull-docs.
  Collection owners want to be able to extract docs for their own
  websites as well.
* The jinja2 filters, tests, and other support code have moved to antsibull
* Remove document_plugins as that has now been integrated into antsibull-docs

* Cleanup and bugfix to other build script code:
  * The Commands class needed to have its metaclass set for abstractmethod
    to work correctly
  * Fix lint issues in some command plugins

* Add the docs/docsite/rst/collections to .gitignore as
  everything in that directory will be generated so we don't want any of
  it saved in the git repository
* gitignore the build dir and remove edit docs link on module pages

* Add docs/rst/collections as a directory to remove on make clean
* Split the collections docs from the main docs

* remove version and edit on github
* remove version banner for just collections
* clarify examples need collection keyword defined

* Remove references to plugin documentation locations that no longer exist.
  * Perhaps the pages in plugins/*.rst should be deprecated
    altogether and their content moved?
  * If not, perhaps we want to rephrase and link into the collection
    documentation?
  * Or perhaps we want to link to the plugins which are present in
    collections/ansible/builtin?

* Remove PYTHONPATH from the build-ansible calls
  One of the design goals of the build-ansible.py script was for it to
  automatically set its library path to include the checkout of ansible
  and the library of code to implement itself.  Because it automatically
  includes the checkout of ansible, we don't need to set PYTHONPATH in
  the Makefile any longer.

* Create a command to only build ansible-base plugin docs
  * When building docs for devel, only build the ansible-base docs for
    now.  This is because antsibull needs support for building a "devel
    tree" of docs.  This can be changed once that is implemented
  * When building docs for the sanity tests, only build the ansible-base
    plugin docs for now.  Those are the docs which are in this repo so
    that seems appropriate for now.
2020-07-17 13:07:35 -07:00
Matt Clay
fb7740ae3b Update hacking/shippable docs.
The `--all` option downloads more than is needed for analyzing code coverage.
2020-07-07 10:22:45 -07:00
Matt Clay
4816bb4f43
More boilerplate fixes. (#70224)
* Fix boilerplate in hacking dir.
* Fix boilerplate in docs dir.
* Fix boilerplate in integration tests.
* Fix boilerplate in examples.
2020-06-22 19:05:30 -07:00
Sam Doran
fd882e0e18
Get test data file directly in rebalance script (#70107)
Rather than looking through tests.json to find the data file, look for it explicitly
within a given target to avoid problems processing data in tests.json.
2020-06-19 09:57:50 -04:00
Matt Clay
c67efe0bd1 Avoid unnecessary comprehensions. 2020-06-18 11:50:31 -07:00
Felix Fontein
8d93ba9120
Plugin/module docs: parse return values, add collection names in them (version_added_collection), and format them nicely in ansible-doc (#69796)
* Tag return value docs if they are a dict (and not str/None).

* Try to parse return docs as YAML.

* Properly dump return values in ansible-doc.

* Adjust plugin formatter.

* Add changelog fragment.

* Don't add 'default' for return values.

* Fix plugin_formatter.

* Only try to parse return docs if they are still a string.

* Add tests.

* Warn if RETURN cannot be parsed.

* Adjust tests. Also test for warning.

* if -> elif (otherwise EXAMPLE will be parsed too).

* Always parse return documentation, and fail if it is invalid YAML.

* Polishing.

* Mostly re-enable ansible-doc tests.

Listing from the local collection seems to be somewhat broken. I assume this
is why the test was disabled.

* Lint and make tests work with Python 2.

* Keep FQCNs in plugins (not modules), i.e. restore previous state.
2020-06-11 11:03:43 -07:00
Brian Coca
062e780a68
starting metadata sunset (#69454)
* starting metadata sunset

 - purged metadata from any requirements
 - fix indent in generic handler for yaml content (whey metadata display was off)
 - make more resilient against bad formed docs
 - removed all metadata from docs template
 - remove metadata from schemas
 - removed mdata tests and from unrelated tests

Co-authored-by: Felix Fontein <felix@fontein.de>
Co-authored-by: Rick Elrod <rick@elrod.me>
2020-06-04 21:01:46 -04:00
Felix Fontein
6a4455ff54
Update plugin_formatter to understand tagged version_added (#69797)
* Update version_added handling for plugin_formatter w.r.t. #69680.

* Update hacking/build_library/build_ansible/command_plugins/plugin_formatter.py

Co-authored-by: Abhijeet Kasurde <akasurde@redhat.com>

Co-authored-by: Abhijeet Kasurde <akasurde@redhat.com>
2020-06-02 08:07:34 -07:00
Dongsu Park
598e3392a9
Discover Flatcar Linux properly for hostname (#69627)
To avoid issues with Flatcar Container Linux being unable to be found,
detect Flatcar distro name especially for hostname, just like CoreOS
Container Linux was supported.

See also https://github.com/ansible/ansible/issues/69516
2020-06-02 18:41:53 +05:30
Rick Elrod
3ec18ccb77
Allow announce script to work for base and older (#69768)
Change:
- Generalize the announce script changes from 4dd0f4127

Test Plan:
Harcoded 'hashes' to [] and then:
- Ran with --version 2.9.9 --version 2.10.0.dev1
- Ran with --version 2.9.9
- Ran with --version 2.10.1

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-05-29 16:20:30 -05:00
Rick Elrod
4dd0f41270
Update announce script for ansible-base (#69735)
Change:
- s/Ansible/ansible-base/
- Update URLs

Test Plan:
- Hope for the best. (No real way to test this until release day because
  there are no files on releases.a.c for it to hash right now. A staging
  releng environment would fix this.)

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-05-27 23:04:42 -05:00
Toshio Kuratomi
efff35a7cd Remove unused imports in the porting_guide command plugin 2020-05-14 09:14:37 -07:00
Toshio Kuratomi
26704b915b Turn pathlib paths into strs
Some APIs do not take a pathlib.  They need to have a string
representation of a path.  Transform the default path to a str
so those APIs will work with the default value.
2020-05-14 09:14:37 -07:00
Toshio Kuratomi
957ad8e769 Fix the command plugin to use the ABCMeta metaclass
The abstract* decorators don't have any effect unless the class has an
ABCMeta metaclass.
2020-05-14 09:14:37 -07:00
Toshio Kuratomi
a3c400d0ce Remove outdated docs and unneeded import from change_detection build file 2020-05-14 09:14:37 -07:00
Rick Elrod
96c56b119d Add a script for adding backport references
Change:
- Add a place for adding backport-related scripts in the future
- Add a script for adding backport references

Test Plan:
- Used it for this latest batch of PR reference-adding.

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-05-07 21:44:13 -05:00
Evgeni Golov
cdad594b16
plugin_formatter: sys.exit does not take a file argument (#68016)
Cleanup of leftover from bcdfdc0cc3.

sys.exit does not take any named argument.
2020-05-05 08:38:23 +05:30
Matt Martz
a513a62536
Remove cgroup_perf_recap_graph.py, since that callback no longer exists here (#68779) 2020-04-08 14:04:53 -05:00
Matt Clay
7323d5dd0d Fix references to old egg-info directory. 2020-03-30 13:56:43 -07:00
Rick Elrod
bd9e31fcb0
Add a custom policy for hackers using ansible-test (#68535)
Signed-off-by: Rick Elrod <rick@elrod.me>
2020-03-28 17:07:29 -05:00
Jill Rouleau
0bf7a08eb6 Remove hacking/aws_config
AWS hacking config has been moved to the AWS CI terminator repo, and is no
longer required in base.
https://github.com/mattclay/aws-terminator/pull/91
2020-03-25 18:47:34 -07:00
Jordan Borean
36fc5840b7
hacking - Add script to easily rebalance Shippable groups (#67976)
* hacking - Add script to easily rebalance Shippable groups

* Fix py26 compile issue

* Add option to automatically change test target aliases
2020-03-24 08:14:09 +10:00
Matt Clay
d049888a92 Update Shippable hacking scripts default branch. 2020-03-23 13:04:56 -07:00
Matt Clay
707eea3afa Add new options to hacking/shippable/incidental.py (#68384)
* Add `--plugin-path` option to `incidental.py`.

* Report on plugins with no test target.

* Add `--verbose` option to script.
2020-03-23 11:14:21 -05:00
Rick Elrod
649f657f91 Fix coverage script to handle ongoing runs (#68380)
* Fix get_recent_coverage_runs.py to handle ongoing runs

Signed-off-by: Rick Elrod <rick@elrod.me>

* Color code test status to make it easier to see at a glance

Signed-off-by: Rick Elrod <rick@elrod.me>

* fix lint

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-03-23 11:14:21 -05:00
Matt Clay
82516a424e Fix hacking/shippable/incidental.py source search. 2020-03-23 11:14:21 -05:00
Rick Elrod
d87d96077f Add script for getting URLs of coverage runs
Signed-off-by: Rick Elrod <rick@elrod.me>
2020-03-23 11:14:21 -05:00
Matt Clay
f18f480a3c Add hacking/shippable/incidental.py script. (#68182)
* Add hacking/shippable/incidental.py script.

* Add README.md.
2020-03-23 11:14:21 -05:00
Rick Elrod
787089cba2 First attempt at making the shippable downloader recurse to pull parent results of reruns (#68154)
* First attempt at making the shippable downloader recurse to pull parent results of reruns

Signed-off-by: Rick Elrod <rick@elrod.me>

* Update message for previous run download.

Co-authored-by: Matt Clay <matt@mystile.com>
2020-03-23 11:14:21 -05:00
Mark Chappell
da30e6d2e1
sqs_queue - Move to boto3 and add support for various extra features (#66795)
* reworked sqs_queue

* Switch default purge_tags behaviour to false.

This matches the behaviour of ec2_tag and ecs_tag.

* Minor lint / review fixups

* Add missing AWS IAM policy for SQS tests

* Move integration tests to using module_defaults: group/aws:...

* add changelog

* Break out the 'compatability' map from our spec definition (gets flagged by the schema validation)

* Tweaks based on review

* add basic examples

* Lint fixups

* Switch out NonExistentQueue logic so it's easier to follow

* Reorder name argument options for consistency

Co-authored-by: Dennis Podkovyrin <dennis.podkovyrin@gmail.com>
2020-02-27 15:26:20 -07:00
mmoyle
cfe96b2092
add module cloudformation_exports (#67349)
* add module cloudformation_exports

* add RETURN, add aliases group, clean up yaml

* update return value. uncomment security_token. remove cloudformation shortcut

* fix typo

* try to delete test stack

* rename stack

* add cleanup and assert. try to set stack name with variable

* create s3 bucket instead

* set bucket name

* add tests, remove unsed key and import, add iam role, add to module_defaults

* import exceptions, fix assert syntax

* fix assert

* Update test/integration/targets/cloudformation_exports/tasks/main.yml

Co-Authored-By: Jill R <4121322+jillr@users.noreply.github.com>

* fix export name

* renamed module

Co-authored-by: Jill R <4121322+jillr@users.noreply.github.com>
2020-02-27 11:39:59 -07:00
Mark Chappell
9c6495d4d4
elb_target / elb_target_info : Integration test fixups (#61256)
* Update AWS policy to enable management of TargetGroups

* elb_target: (integration tests) migrate to using module_defaults

* elb_target: (integration tests) lookup the AMI by name rather than hard coding AMI IDs

* elb_target_info: (integration tests) finish rename of integration test role

* elb_target: (integration tests) rename various resources to consistently use {{ resource_prefix }}

* elb_target_info: (integration tests) Migrate to using module_defaults

* elb_target_info: (integration tests) Lookup AMI by name rather than hard coding AMI IDs

* Apply suggestions from code review

Co-Authored-By: Jill R <4121322+jillr@users.noreply.github.com>

* elb_target: (integration tests) Remove the 'unsupported' alias

* Try bumping up the timeout

* Rules don't permit 'shippable' (resource_prefix uses this when run in shippable)

* Try bumping up more timeouts :/

* Avoid double evaluation of target_health assertion

* Simplify target_type usage a little (rather than constantly performing a lookup)

* mark elb_target tests 'unstable' for now, they're slow

Co-authored-by: Jill R <4121322+jillr@users.noreply.github.com>
2020-02-20 17:13:09 -07:00
Matt Clay
19f6fc68d4
Relocate Shippable tools. (#67556)
* Move Shippable tools to hacking directory.

These limits the `test/utils/shippable/` directory to scripts required for CI.

* Fix `test/utils/shippable/` file classification.

* Update package-data sanity test.
2020-02-19 09:55:34 -08:00
Mark Chappell
b3db41e6d8
Fix hacking policy (#67579) 2020-02-19 12:05:43 -05:00
Mark Chappell
9edcda7ef7
Refactor iam_role to bring down the complexity score (#66027)
* Simplify BotoCore- / Client- Error try/except loops where we don't need different behaviour

* Refactor IAM Role manipulation to reduce complexity scores

* Missing permissions

* Add retry decorator by default

* compare_attached_role_policies is dead code, remove it
2020-02-15 12:57:52 +00:00
Mark Chappell
8d574c3770
AnsibleAWSModule related cleanup - redshift (#66779)
* AnsibleAWSModule related cleanup - redshift

* Apply a backoff on modify_cluster to cope with concurrent operations

* Add AWS 'hacking' policy to allow creation of Redshift ServiceRole

* Adding the retry policies makes the redshift test suite more reliable
2020-02-12 11:17:18 -07:00
Rick Elrod
236773bfea
Improve distribution unit tests (#67300)
* Include distro.{os,lsb}_release_info() in distro fixture generation script
* Add distro release information to existing fixtures
* Add some new fixtures for newer distribution versions
* Add fixture for Arch Linux with lsb-release
2020-02-11 16:46:48 -05:00
Dick Visser
5b93a14a0f
Add anchor to each parameter row (#66895)
* Add anchor to each paramater row

* Update docs/templates/plugin.rst.j2

Co-Authored-By: Felix Fontein <felix@fontein.de>

* Insert full keys into plugin docs.

* Added visible links.

Co-authored-by: Felix Fontein <felix@fontein.de>
2020-02-03 20:11:48 +01:00
Prasad Katti
b16525b841 [aws] integration tests for ec2_vol (#66193) 2020-01-23 15:34:06 -07:00
Toshio Kuratomi
0da13469d3 Add a script to update the intersphinx inventory (#66646)
* Add a script to update the intersphinx inventory files

* We're about to add intersphinx inventories for separate ansible docs
  so we need an easy way to update them.  Also, we should be updating
  these cache files for other upstreams occassionally as well.  With a
  script, we can add updating them to a release process.
* Now that we don't know what the version of the cache is, change the
  filenames to not contain versions.

* Update the intersphinx cache files with the latest upstream versions

Results of running:

hacking/build-ansible.py update-intersphinx-cache -o docs/docsite -c docs/docsite/rst/conf.py

* Add a comment to the configuration file which says how to structure the intersphinx mapping and why.

* Update docs/docsite/rst/conf.py

Co-Authored-By: Sandra McCann <samccann@redhat.com>

Co-authored-by: Sandra McCann <samccann@redhat.com>
2020-01-22 17:20:02 -05:00
Clint Byrum
284f26303c Add support for ECR Lifecycle Policies to ecs_ecr (#48997)
* Fix copy/pasta for ecs_ecr test names

* Add support for lifecycle policies to ecs_ecr

New feature for ecs_ecr to support [ECR Lifecycle Policies][].

Fixes #32003

 [ECR Lifecycle Policies]: https://docs.aws.amazon.com/AmazonECR/latest/userguide/LifecyclePolicies.html

* Improve error message for ecs_ecr parsing errors

Replaces the exception and stack trace with a description of what's
actually going wrong from a user perspective.

* Rename delete policy to purge policy

Marks the `delete_policy` parameter as deprecated, to be removed in
Ansible 2.6.

* Add version_added to purge_policy

* Remove changing results based on verbosity

What I really want is --diff support, and changing results based on
verbosity is abnormal.

* Ensure repository name is lowercase

* Fix deprecation cycle to 4 releases

* Use a YAML anchor for credentials

* Remove filters from assertions

* Add minimal permissions needed

* Updating version_added and deprecation cycle

The original PR sat while a few releases happened.

* Bumping version added and deprecation version

We missed the 2.8 release.

* Removing bare except:

This is not allowed and is generally bad practice.

* Fix lint errors

* update ansible release metadata

* Use the new alias deprecation scheme

This was added in the time the PR has been in development, so rework
things to use it.

* Add test coverage

This makes sure that lifecycle_policy is produced when passed in.

*Also a minor suggestion for simplification from PR.

* Restore changes from 62871 lost in rebase

* Add changelog

* Remove version_added for new purge_policy option

Per sanity test fail.
2019-12-19 11:41:41 -07:00
Prasad Katti
b8729b2544 cloudformation integration tests (#65643) 2019-12-17 15:27:52 -07:00
Prasad Katti
056b035c98 add module aws_step_functions_state_machine_execution (#64431)
* add module aws_step_functions_state_machine_execution

* AWS step functions tests - Use module defaults

* Return all attributes from aws api calls as ansible task output

* aws_sfn - make start and stop execution idempotent and fix check mode

* aws sfn - use build_full_result method of the paginator

* aws sfn - remove changes made to help with local debugging
2019-12-10 14:08:49 -05:00
Prasad Katti
37ce55fd79 lightsail - Use AnsibleAWSModule (#65275)
* lightsail - Use AnsibleAWSModule

- Use AnsibleAWSModule
- Refactor the logic for wait into a separate function (Fixes #63869)
- Handle exceptions in find_instance_info and add a fail_if_not_found parameter
- Add a new state `rebooted` as an alias for `restarted`. AWS calls the action Reboot.
- Add required_if clause for when state is present

* lightsail - Use the default keypair if one is not provided

* lightsail - add a required_if for when state=present

* Update short description for lightsail module
2019-12-02 13:12:44 -07:00
Prasad Katti
95bd92da04 Add integration tests for aws lightsail (#63770)
* Add integration tests for aws lightsail

* lightsail - use module_defaults instead of aws_connection_info

* lightsail tests - assert instance state on create

* Fix yaml syntax error

Co-Authored-By: Jill R <4121322+jillr@users.noreply.github.com>

* [lightsail] create keypair as part of the testsuite

* Fix lightsail actions in compute-policy

* Add ability to delete keypair in lightsail_keypair
2019-11-21 16:53:36 -07:00
Mark Chappell
a815fdf8bb Update Route53 IAM policy so the Route53 tests run (#64886) 2019-11-18 08:48:58 -05:00
Mark Chappell
551b17b8a2 ec2_vpc_net_info: integration tests (#62649)
* ec2_vpc_net: (integration tests) migrate to using module_defaults

* ec2_vpc_net: (integration tests) use a private subnet for the tests

* ec2_vpc_net_info: Add integration tests

* ec2_vpc_net_info: add cidr_block_association_set to documentation

* Update AWS hacking test policy to allow VPC CIDR disassociation

* Update test/integration/targets/ec2_vpc_net/tasks/main.yml

Co-Authored-By: Jill R <4121322+jillr@users.noreply.github.com>

* Store vpc2 ID to make it clearer which VPC we're changing

* Be more consistent with our quoting

* Explicitly test that the VPC IDs haven't changed
2019-11-14 13:36:42 -07:00
Mark Chappell
b5f484dcc3 ec2_eip Don't throw an exception when re-releasing an EIP (idempotency) (#62332)
* ec2_eip: (integration tests) move to using module_defaults

* ec2_eip: (integration tests) expand integration tests

Also clean up a little
- Delete EIPs when we finish testing them (reduce the chance of hitting limits)
- Rejig deletion so that it works when runs fail
- Add tests for ec2_eip_info

* ec2_eip: Minor doc tweaks

* ec2_eip: Don't throw an exception when we try to disassociate an already disassociated EIP

* ec2_eip: Add missing IAM policy (manage IGWs)

* ec2_eip: (integration tests) Use the VPC as a crude lock to avoid running parallel tests

We test that untagged EIPs come and go as we expect, if multiple tests are
running in parallel this confuses things

* Fix ec2_eip association
2019-11-13 13:27:35 -07:00
Matthew Davis
4ee9f40e62 Add aws_acm module (#60552)
* convert aws_acm_facts to AnsibleAWSModule

* factor aws_acm_facts into module_utils

* add more filtering options for aws_acm_info

* add aws_acm module and tests

* uncomment aws_acm test

* fix linting for aws_acm

* fix __future__ linting for aws_acm

* fix linting for aws_acm

* fix linting for aws_acm

* fix linting for aws_acm

* fix linting for aws_acm

* fix aws_acm_info arg type

* remove test for old module name aws_acm_facts

* simplify AWS ACM client creation

* fix indent typo in aws_acm test

* catch BotoCoreError in aws_acm

* fix indent typo in aws_acm test

* tighten AWS ACM test policy resource

* move aws acm int test to venv

* remove errant file

* fix AWS ACM int test perms

* undo copyright addition to wrong file

* fix invalid log message in aws_acm

Co-Authored-By: Jill R <4121322+jillr@users.noreply.github.com>

* rephrase aws_acm_info doc from facts to information

Co-Authored-By: Jill R <4121322+jillr@users.noreply.github.com>

* rename aws_facts var to aws_info

* remove case insensitivity for aws_acm pem compare

* add no_log for aws_acm credential setting

* add per-test prefix to aws_acm test resource names

* make aws_acm use crypto module_util

* clarify copyright for aws_acm

* make aws_acm int test clearer

* add explicit crypto dependency to aws_acm

* change requests for aws_acm pr

* fix wrong copyright owner aws_acm test

* fix wrong copyright owner aws_acm test

* rewrite aws_acm cert chain compare with regex, no dependency

* fix linting for aws_acm unit test

* fix linting for aws_acm unit test

* fix linting and duplicate ignore

* fix failed cert chain split in aws_acm, add more tests

* remove errant file

* more linting fixes for aws_acm

* fix sanity ignore

* rewrite cert compare in aws_acm to use base64 decode

* improve regex for pem cert chain split in aws_acm

* undo changes to crypto module util for aws_acm

* increment ansible version for new aws_acm module

* convert aws_acm return(x) to return x

* increment version added for aws_acm_info new features

* fix linting

* fix bugs with AWS ACM

* fix bad rebase

* disable AWS ACM integration test, due to AWS account limit issue

* remove aws acm integration test from shippable group
2019-11-05 11:57:08 -07:00
Sandra McCann
c63ef6d911 update too old version to 2.4 (#64167) 2019-10-31 19:44:53 -05:00