Commit graph

505 commits

Author SHA1 Message Date
Zhikang Zhang
b578bf9e20 Fix test-module failing to validate args (#41004)
* Fix test-module failing to validate args

The test-module pass a wrong argument _ansible_tmp cause the validation failed.
Change the argument _ansible_tmp to _ansible_tmpdir to fix this.

* Add a integration test for test-module.

Prior to this change, we don't have a test for test-module.

This change ensure the correctness of test-module script.
2018-06-01 12:02:56 -07:00
Will Thames
809c7404ab Add two missing VPC permissions (#37896)
Remove VPC permissions from network-policy.json as they mostly duplicate
compute-policy.json permissions - separating the VPC and compute permissions
would likely lead to further confusion.
2018-05-25 06:31:54 -04:00
Matt Clay
8deced3e04
Fix shebangs and file modes and update tests. (#40563)
* Add execute bit sanity test and apply fixes.
* Add shebang test for `lib` dirs and apply fixes.
* Shebang and execute bit cleanup.
2018-05-22 14:25:36 -07:00
Matt Davis
e4edb2842a 2.6 changelog gen/version/root dir cleanup (#40421)
* patched in changelog gen stuff from stable-2.5
* Makefile updates
* release.py as single-source-of-truth
* Remove obsolete ansible-core-sitemap.xml file.
* Move ROADMAP.rst into README.rst.
* dynamic rpm changelog, zap old deb/rpm changelogs
* fix changelog in MANIFEST.in
* Remove obsolete hacking/update.sh script.
* Remove ref to deleted authors script.
* Remove ref to removed module-formatter script.
* Update headings to match script names.
* MANIFEST.in cleanup
* removed RELEASES.txt and versions.yml
* removed obsolete release generation playbook/bits (not used since 2.5)
* misc Makefile cleanup
* speculative changes to DEB versioning
* allow override of DEB_VERSION/DEB_RELEASE
2018-05-21 16:14:53 -07:00
Toshio Kuratomi
5634dae290 Remove the cherrypick script
We only needed it for migrating cherrypicks between the unified repo and
the ansible-modules-* repos.  Now that we aren't supporting 2.3, we no
longer need this script.
2018-05-21 14:50:24 -05:00
Tim Rupp
548282139f
Fixes incorrect variable name (#40274)
Incorrect variable name was causing a NameError

  NameError: name 'comlpex_args' is not defined
2018-05-16 12:13:22 -07:00
Madhura-CSI
b85970b2b0 New Module: ec2_vpc_vpn_facts (#35983)
* New module: ec2_vpc_vpn_facts

* Add integration tests for ec2_vpc_vpn_facts and the IAM permissions

* Add retry to VPC removal

* Use unique name for VGW

* Always clean up after tests and add retries
2018-05-15 12:13:46 -04:00
Jordan Borean
44ab948e5d
create module tmpdir based on remote_tmp (#39833)
* create module tmpdir based on remote_tmp

* Source remote_tmp from controller if possible

* Fixed sanity test and not use lambda

* Added expansion of env vars to the remote tmp

* Fixed sanity issues

* Added note around shell remote_tmp option

* Changed fallback tmp dir to ~/.ansible/tmp to make shell defaults
2018-05-15 09:31:21 +10:00
Will Thames
29770a297a Fail with nice error message if elb target_type=ip not supported (#38313)
* Add helpful failure message if target_type=ip is not supported

Create test case for target_type=ip not supported

* Update elb_target_group module to latest standards

Use AnsibleAWSModule
Improve exception handling
Improve connection handling
2018-05-03 08:36:52 -04:00
mwpeterson
efdd92e1c0 Update test-module (#39331)
Update test-module To use C.DEFAULT_LOCAL_TMP
2018-04-26 07:16:33 -07:00
Will Thames
12f2b9506d [aws]Add VPC configuration to ECS modules (#34381)
Enable awsvpc network mode for ECS services and tasks and
their underlying task definitions

Improve test suite to thoroughly test the changes

Use runme.sh technique to run old and new versions of botocore to
ensure that the modules work with older botocore and older network modes
and fail gracefully if awsvpc network mode is used with older botocore
2018-04-25 15:41:04 -04:00
Matt Clay
c262dbfd30 Use https for links to ansible.com domains. 2018-04-23 11:33:56 -07:00
Ed Costello
0d31d1cd24 [cloud]Add aws_ses_identity_policy module for managing SES sending policies (#36623)
* Add aws_ses_identity_policy module for managing SES sending policies

* Add option to AnsibleAWSModule for applying a retry decorator to all calls.

* Add per-callsite opt in to retry behaviours in AnsibleAWSModule

* Update aws_ses_identity_policy module to opt in to retries at all callsites.

* Add test for aws_ses_identity_policy module with inline policy.

* Remove implicit retrys on boto resources since they're not working yet.
2018-04-05 15:11:12 -04:00
Will Thames
a1d3cf488d [cloud][test]Add missing IAM policy for cloudfront (#38248)
Cloudfront needs CreateOriginAccessIdentity

Add profile parameter to setup-iam.yml. Could arguably just use
AWS_PROFILE but given that other tasks are using profile, should
be consistent.
2018-04-05 14:06:04 -04:00
Pilou
7908f78fa6 module_common: handle None value for templar (#36651)
* module_common: set required parameter templar

Fix the following error (related to b455901):

  $ ./hacking/test-module -m ./lib/ansible/modules/system/ping.py -I ansible_python_interpreter=/usr/bin/python
  Traceback (most recent call last):
    File "./hacking/test-module", line 268, in <module>
      main()
    File "./hacking/test-module", line 249, in main
      (modfile, modname, module_style) = boilerplate_module(options.module_path, options.module_args, interpreters, options.check, options.filename)
    File "./hacking/test-module", line 152, in boilerplate_module
      task_vars=task_vars
    File "ansible/lib/ansible/executor/module_common.py", line 910, in modify_module
      environment=environment)
    File "ansible/lib/ansible/executor/module_common.py", line 736, in _find_module_utils
      shebang, interpreter = _get_shebang(u'/usr/bin/python', task_vars, templar)
    File "ansible/lib/ansible/executor/module_common.py", line 452, in _get_shebang
      interpreter = templar.template(task_vars[interpreter_config].strip())
  AttributeError: 'NoneType' object has no attribute 'template'

* module_common.modify_module: templar is required
2018-03-29 13:54:48 -04:00
Julien Vey
7c07877b1b s3_bucket: add integration tests (#36941)
Also update testing-policies/storage
2018-03-07 11:25:24 -05:00
Julien Vey
51d491f8f0 route53_zone: move to boto3, and enable comment update (#36641) 2018-03-07 11:16:04 -05:00
Ed Costello
645952c139 Add aws_caller_facts module and use it in setup-iam.yml (#36683)
* Add aws_caller_facts module and use it in setup-iam.yml

This removes the dependency on having the command line AWS tools
installed.
2018-02-28 16:30:34 +10:00
Will Thames
b5a1643e3d Add new aws_waf_condition module (#33110) 2018-02-01 18:16:27 -05:00
Ed Costello
d16bc1c3f4 New aws_ses_identity module to manage AWS Simple Email Service Identity (#31140)
* Add aws_ses_identity module

* Update CI alias, add BotoCoreError exception handling.

* Add SES and SNS permissions to hacking/aws_config to run aws_ses_identity integration tests
2018-01-29 15:35:49 -05:00
Marek
5fa29201a7 Port sts_assume_role to boto3 (#32569)
* Ported sts_assume_role to boto3

* Added integration tests
2018-01-22 17:46:08 -05:00
Adrian Likins
ec9769c82f
Facts distribution clear linux 31501 (#32453)
The search string used to look for Clear Linux
was changed in 45a9f96774 to
be more specific, but was too specific. Now finding
a substring match for 'Clear Linux' in /usr/lib/os-release
is enough to consider a match.

Since the details of the full name in os-release varies
('Clear Linux Software for Intel Architecture',
 'Clear Linux OS for Intel Architecture', etc) the
search string match was failing and would fall back to the
'first word in the release file' method resulting in
ansible_distribution='NAME="Clear'

Also add a meta fact indicating which search string
was matched.

Test case info from:
        https://github.com/ansible/ansible/issues/31501#issuecomment-340861535

Fixes #31501
2018-01-20 15:05:53 -05:00
Will Thames
4d58d16793 Add aws_s3 action plugin to find source files as expected (#35028)
People expect to be able to upload files to s3 using standard
locations for files.

Providing an action plugin that effectively rewrites the `src`
key to the result of finding such a file is a great help.

Tests added, and IAM permissions corrected
2018-01-19 17:11:42 -05:00
Will Thames
8d733dbdf0 [cloud] New module cloudfront_distribution (#31284)
* added cloudfont.py, modified cloudfront_facts.py class name and fixed a minor bug

* Improvements to cloudfront_distribution

* Reduce the scope of the cloudfront_distribution module
    * Remove presigning
    * Remove streaming distribution functionality
* Add full test suite for cloudfront distribution
* Meet Ansible AWS guidelines

* Make requested changes

Fix tests

Use built-in waiter

Update copyright
2018-01-17 11:03:23 -05:00
Matt Clay
797664d9cb Python 2.6 str.format() compatibility fixes. 2018-01-10 14:08:11 -08:00
Will Thames
866d7fdce9 [cloud] Create ECS integration test suite (#33757)
Tests for:
* ecs_cluster
* ecs_service
* ecs_service_facts
* ecs_taskdefinition
* ecs_taskdefinition_facts

* Add idempotency testing

Test ecs_cluster, ecs_service and ecs_taskdefinition for trivial
idempotency. Add FIXMEs to the tests because the latter two fail.

Remove unused dependencies
2017-12-15 08:15:01 -05:00
Abhijeet Kasurde
36f82ae8cc Replace exit() with sys.exit()
This fix adds replacement for exit() to sys.exit(), as
exit() is not recommended way to exit from the program.

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2017-12-14 22:03:08 -05:00
Sloane Hertel
f5471b3dcb [cloud] ec2_vpc_net integration tests (#33111)
* Add some integration tests for ec2_vpc_net module

* Add a couple tests for check mode

fix typo

ensure the DHCP option set is cleaned up

* Add permissions to test policy
2017-12-05 16:41:16 -05:00
Matt Martz
783a09d105
Fix failure to adjust my name when doing copy pasta (#33322) 2017-11-27 19:09:05 -06:00
Matt Martz
4fe08441be Deprecate tests used as filters (#32361)
* Warn on tests used as filters

* Update docs, add aliases for tests that fit more gramatically with test syntax

* Fix rst formatting

* Add successful filter, alias of success

* Remove renamed_deprecation, it was overkill

* Make directory alias for is_dir

* Update tests to use proper jinja test syntax

* Update additional documentation, living outside of YAML files, to reflect proper jinja test syntax

* Add conversion script, porting guide updates, and changelog updates

* Update newly added uses of tests as filters

* No underscore variable

* Convert recent tests as filter changes to win_stat

* Fix some changes related to rebasing a few integration tests

* Make tests_as_filters_warning explicitly accept the name of the test, instead of inferring the name

* Add test for tests_as_filters_warning

* Update tests as filters in newly added/modified tests

* Address recent changes to several integration tests

* Address recent changes in cs_vpc
2017-11-27 17:58:08 -05:00
Will Thames
1ca0c0e7f7 Consolidate IAM policies into fewer, larger policies (#33122)
Due to IAM limits allowing at most 10 policies per group,
need to reduce the number of total policies in use.
2017-11-21 17:15:31 -05:00
Will Thames
0b128e2e29 Add tests for ec2_vpc_route_table (#32036)
* Add tests for ec2_vpc_route_table

Initial tests for ec2_vpc_route_table ready for boto3 porting

* Turn AWS connection information into a YAML block
2017-11-20 17:31:08 -05:00
Pilou
a9afb1e19e hacking/test-module: fix Python 3 compatibility (#33069)
Exception was:
Traceback (most recent call last):
  File "./hacking/test-module", line 268, in <module>
    main()
  File "./hacking/test-module", line 249, in main
    (modfile, modname, module_style) = boilerplate_module(options.module_path, options.module_args, interpreters, options.check, options.filename)
  File "./hacking/test-module", line 155, in boilerplate_module
    if module_style == 'new' and 'ANSIBALLZ_WRAPPER = True' in module_data:
TypeError: a bytes-like object is required, not 'str'
2017-11-20 08:03:55 +01:00
Daniel Shepherd
cfbe9c8aee [cloud] Add IPv6 support for ec2_vpc_subnet module(#30444)
* Add integration test suite for ec2_vpc_subnet

* wrap boto3 connection in try/except

update module documentation and add RETURN docs

add IPv6 support to VPC subnet module

rename ipv6cidr to ipv6_cidr, use required_if for parameter testing, update some failure messages to be more descriptive

DryRun mode was removed from this function a while ago but exception handling was still checking for it, removed

add wait and timeout for subnet creation process

fixup the ipv6 cidr disassociation logic a bit per review

update RETURN values per review

added module parameter check

removed DryRun parameter from boto3 call since it would always be false here

fix subnet wait loop

add a purge_tags parameter, fix the ensure_tags function, update to use compare_aws_tags func

fix tags type error per review

remove **kwargs use in create_subnet function per review

* rebased on #31870, fixed merge conflicts, and updated error messages

* fixes to pass tests

* add test for failure on invalid ipv6 block and update tags test for purge_tags=true function

* fix pylint issue

* fix exception handling error when run with python3

* add ipv6 tests and fix module code

* Add permissions to hacking/aws_config/testing_policies/ec2-policy.json for adding IPv6 cidr blocks to VPC and subnets

* fix type in tests and update assert conditional to check entire returned value

* add AWS_SESSION_TOKEN into environment for aws cli commands to work in CI

* remove key and value options from call to boto3_tag_list_to_ansible_dict

* remove wait loop and use boto3 EC2 waiter

* remove unused register: result vars

* revert az argument default value to original setting default=None
2017-11-16 14:58:12 -05:00
Will Thames
46c4f6311a [cloud] Add retries/backoff to ec2_vpc_subnet module (#31870)
* Allow backoff for describe_subnets

Improve exception handling to latest standards

* Add integration test suite for ec2_vpc_subnet

* Add test for creating subnet without AZ

Fix bug identified by test

Fixes #31905
2017-11-16 13:09:42 -05:00
Sloane Hertel
1dd55acbc2 ec2_group: add rule description support - fixes #29040 (#30273)
* ec2_group: add support for rule descriptions.

* Document rule description feature and add an example using it.

* Fix removing rule descriptions.

* Add integration tests to verify adding/modifying/removing rule descriptions works as expected.

* Add permissions to hacking/aws_config/testing_policies/ec2-policy.json for updating ingress and egress rule descriptions.

* ec2_group: add backwards compatibility with older versions of botocore for rule descriptions.

* Add compatibility with older version of botocore for ec2_group integration tests.

* ec2_group: move HAS_RULE_DESCRIPTION to be checked first.

* Make requested change

* Pass around a variable instead of client

* Make sure has_rule_description defaults to None

* Fail if rule_desc is in any ingress/egress rules and the the botocore version < 1.7.2

* Remove unnecessary variable

* Fix indentation for changed=True when updating rule descriptions.

* minor refactor to remove duplicate code

* add missing parameter

* Fix pep8

* Update test policy.
2017-10-24 21:18:56 -04:00
Prasad Katti
d6737dcdd4 Added info about return_skeleton_generator.py
Added info about return_skeleton_generator.py to hacking/README.md
2017-10-02 15:58:57 -04:00
Will Thames
37736ee87e Allow AWS image and snapshot creation/deletion
Provide all necessary permissions for AMI tests
Allow tests to run in us-east-2
Ensure `always` section gets used
Update tests to ensure that cleanup works better, and add
deletion idempotency test
2017-09-19 23:12:50 -07:00
Will Thames
5e978b0f2b Use JSON returns values to create RETURN docs
After running hacking/test-module to generate some output,
the JSON output can be fed into the return skeletion generator
to create an excellent starting point for RETURN docs
2017-09-11 14:33:11 -07:00
Will Thames
63df0adc17 [cloud] Update RDS parameter group for boto3 (#25345)
* Update RDS parameter group for boto3

* Update to boto3
* Update to latest ansible standards
* Remove choices list for valid engines (See #19221 for context)
* Allow tagging
* Return some useful information, and document that information

* Add tests for rds_param_group

* Improve testing of rds_param_group

* Add purge_tags option for rds_param_group

* Fix remaining broken rds_param_group tests

* Ensure the group name is lowercased. Fixes integration tests when run on OSX
2017-08-28 12:52:22 -04:00
welchwilmerck
af40d04247 bring comments and docs up-to-date for invoking hacking/test-module (#20940)
inside test-module, rundebug also needs to know interpreters
2017-08-18 09:46:15 -04:00
Toshio Kuratomi
af2073d057 metadata 1.1
* Add network value to support_by field.
* New support_by value, certified
* Deprecate curated in favor of certified
* Add conversion from 1.0 to 1.1 to metadata-tool
* Add supported by Red Hat field to ansible-doc output
2017-08-15 23:12:08 -07:00
Sam Doran
aa19563388 Improve fish environment setup (#26151)
* Add test runner to PATH

* Add Python3 support

* Updating env-setup.fish to use more portable '-exec' rather than '-delete'

* Create gen_egg_info function

Move code into a function similar to env-setup.

Silence all output when run with -q
2017-08-01 09:41:21 -04:00
Will Thames
f972994662 [cloud] fix VPC behavior for ec2_group module, improve integration tests (#27038)
* Add tests for group in a VPC

* Improve ec2_group output and documentation

Update ec2_group to provide full security group information
Add RETURN documentation to match

* Fix ec2_group creation within a VPC

Ensure VPC ID gets passed when creating security group

* Add test for auto creating SG

* Fix ec2_group auto group creation

* Add backoff to describe_security_groups

Getting LimitExceeded from describe_security_groups is definitely
possible (source: me) so add backoff to increase likelihood of
success.

To ensure that all `describe_security_group` calls are backed off,
remove implicit ones that use `ec2.SecurityGroup`. From there,
the decision to remove the `ec2` boto3 resource and rely on the client
alone makes good sense.

* Tidy up auto created security group

Add resource_prefix to auto created security group and delete
it in the `always` section.
Use YAML argument form for all module parameters
2017-08-01 06:53:43 -04:00
Will Thames
f3bc8b84b6 [cloud] Fix setup_iam to use policy_name, not PolicyName (#26880)
Update setup_iam.yml to work with latest iam_managed_policies
module, which correctly snakifies the results
2017-07-28 07:50:07 -04:00
Toshio Kuratomi
225fa5d092 Fix undefined variables, basestring usage, and some associated python3 issues 2017-07-25 15:58:23 -07:00
sramakr
b980a5c02a Use Boto3 for ec2_group Fixes #23507 (#25340)
* Use Boto3 for ec2_group

Currently boto doesn't support ipv6. To support ipv6 in ec2_group, we need boto3.
boto3 has significant API changes, which caused more re-factoring for ec2_group module.
Added additional integration test to test_ec2_group role.

* Follow the standard for boto3 ansible

Fixed imports. Use boto3 ansible exception with camel_dict_to_snake_dict.
Refactored the call to authorize/revoke  ingress and egress.

* Removed dependancy with module ipaddress

Added new parameter called cidr_ipv6 for specifying
ipv6 addresses inline with how boto3 handles ipv6 addresses.

* Updated integration test

* Added ipv6 integration test for ec2_group

* Set purge_rules to false for integration test

* Fixed import statements

Added example for ipv6.
Removed defining HAS_BOTO3 variable and import HAS_BOTO3 from ec2.
Cleaned up import statements.

* Fixed exception handling

* Add IAM permissions for ec2_group tests

Missing AuthorizeSecurityGroupEgress necessary for latest tests

* Wrapped botocore import in try/except block

Import just botocore to be more similar to other modules
2017-07-17 12:03:31 +10:00
Matt Clay
1c1527c73f Initial version of source/testing report tool. 2017-07-15 00:15:04 -07:00
Will Thames
0ed1c3ba9c Split up testing IAM policies and automate creating them (#26223)
* Split up testing IAM policies and automate creating them

Move to managed policies to avoid the 5KB limit on policies
for an IAM entity.

The policy file is templated, so need to make sure that there
is an easy mechanism to populate the templates and push the
new policies.

* Update IAM policies for ec2_scaling_policy tests

* Fix RouteTable policies

DescribeRouteTable should be plural
ModifyRouteTable does not exist, but ReplaceRouteTableAssociation
does.

* Some IAM policies do not allow specified Resources

Various IAM policies do not allow Resources
to be specified and should just use `*`. This differs
per service

* [Autoscaling](http://docs.aws.amazon.com/autoscaling/latest/userguide/control-access-using-iam.html#policy-auto-scaling-resources)
* [EC2](http://docs.aws.amazon.com/AWSEC2/latest/APIReference/ec2-api-permissions.html#ec2-api-unsupported-resource-permissions)
* [ECR](http://docs.aws.amazon.com/AmazonECR/latest/userguide/ecr-supported-iam-actions-resources.html)
* [ELB](http://docs.aws.amazon.com/elasticloadbalancing/latest/userguide/load-balancer-authentication-access-control.html)

* Finish fixing AWS IAM resource specifications for testing

Update Lambda and RDS policies
2017-07-14 14:50:55 +10:00
hondak
db5ec11758 Replace double-quote with single-quote at example (#26667) 2017-07-12 11:22:50 -04:00
Sloane Hertel
113f92548a hacking/test-module: fix for python3 (#26194)
* make hacking/test-module compatible with python3
2017-07-11 13:21:51 -07:00
Toshio Kuratomi
3ee997b720 metadata and doc generator optimization and fixes
* Fix ansible-doc traceback when a plugin doesn't parse correctly
* Change extract_metadata ivocation to take either an ast or source
  code.  When given source code, it can find file offsets for the start
  and end of dict.  When given the ast, it is quicker as it doesn't have
  to reparse the source.  Requires changing the call to the function to
  use a keyword arg.
* Fix reading of metadata to find the last occurrence of
  ANSIBLE_METADATA instead of the first.
* Add some more unittests to get closer to complete coverage
2017-07-04 01:12:43 -07:00
Toshio Kuratomi
e27c2860e7 Move metadata extraction into a library function (#26198)
* Move metadata extraction into a library function
* fix the string cases to throw NotImplementedError
* Some python3 fixes
2017-06-28 18:10:26 -07:00
Michael De La Rue
56d33a2967 Fix hacking/test-module to allow running modules with pdb (#23339)
* Fix hacking/test-module to allow running modules with pdb

* add emacs autosave files to gitignore
2017-06-28 10:58:22 -07:00
Yujun Zhang
1f3755f86b Include error message when ansiballz_setup fails (#26127)
It is quite difficult to pinpoint what is wrong without it
2017-06-28 12:45:38 -04:00
Toshio Kuratomi
e238ae999b Cyptography pr 20566 rebase (#25560)
Make pyca/cryptography the preferred backend for cryptographic needs (mainly vault) falling back to pycrypto

pyca/cryptography is already implicitly a dependency in many cases
through paramiko (2.0+) as well as the new openssl_publickey module,
which requires pyOpenSSL 16.0+. Additionally, pyca/cryptography is
an optional dep for better performance with vault already.

This commit leverages cryptography's padding, constant time comparisons,
and CBC/CTR modes to reduce the amount of code ansible needs to
maintain.

* Handle wrong password given for VaultAES format

* Do not display deprecation warning for cryptography on python-2.6

* Namespace all of the pycrypto imports and always import them

  Makes unittests better and the code less likely to get stupid mistakes
  (like using HMAC from cryptogrpahy when the one from pycrypto is needed)

* Add back in atfork since we need pycrypto to reinitialize its RNG just in case we're being used with old paramiko

* contrib/inventory/gce: Remove spurious require on pycrypto

(cherry picked from commit 9e16b9db275263b3ea8d1b124966fdebfc9ab271)

* Add cryptography to ec2_win_password module requirements
  * Fix python3 bug which would pass text strings to a function which
    requires byte strings.

* Attempt to add pycrypto version to setup deps

* Change hacking README for dual pycrypto/cryptography

* update dependencies for various CI scripts

* additional CI dockerfile/script updates

* add paramiko to the windows and sanity requirement set

  This is needed because ansible lists it as a requirement. Previously
  the missing dep wasn't enforced, but cryptography imports pkg_resources
  so you can't ignore a requirement any more

* Add integration test cases for old vault and for wrong passwords

* helper script for manual testing of pycrypto/cryptography

* Skip the pycrypto tests so that users without it installed can still run the unittests

* Run unittests for vault with both cryptography and pycrypto backend
2017-06-27 06:00:15 -07:00
Brian Coca
74842adc07 1st part of ansible config, adds ansible-config to view/manage configs (#12797)
* Start of ansible config project

moved configuration definitions to external yaml file vs hardcoded
 * updated constants to be a data strcutures that are looped over and also return origin of setting
changed to manager/data scheme for base classes
new cli ansible-config to view/manage ansible configuration settings
 * prints green for default/unchanged and yellow for those that have been overriden
 * added list action to show all configurable settings and their associated ini and env var names
 * allows specifying config file to see what result would look like
 * TBD update, edit and view options

removed test for functions that have been removed

env_Vars are now list of dicts
allows for version_added and deprecation in future
added a couple of descriptions for future doc autogeneration
ensure test does not fail if delete_me exists
normalized 'path expansion'
added yaml config to setup packaging
removed unused imports
better encoding handling

updated as per feedback

* pep8
2017-06-14 11:08:34 -04:00
Abhijeet Kasurde
b89cb95609 Fix spelling mistakes (comments only) (#25564)
Original Author : klemens <ka7@github.com>

Taking over previous PR as per
https://github.com/ansible/ansible/pull/23644#issuecomment-307334525

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2017-06-12 07:55:19 +01:00
Brian Coca
d03b16e88b start of 'profiling utils' 2017-05-31 14:00:12 -04:00
Martyn Ranyard
014f33655c Allows for testing binary modules (#24857) 2017-05-22 14:00:06 -04:00
Michael De La Rue
e9e661ebbb aws integration tests - provide an IAM policy for running (#24725)
* aws integration tests - provide an IAM policy that can be used for running them

* move documentation of aws policies into main integration testing documentation + some updates there

* Update testing_integration.rst

Edits
2017-05-19 14:37:39 -07:00
Dag Wieers
9a5a61bdca hacking/: PEP8 compliancy (#24683)
- Make PEP8 compliant
2017-05-16 18:52:07 +01:00
Matt Martz
8e523089ef Speed up env-setup (#24133) 2017-05-01 09:54:50 -05:00
Brian Coca
424e1946f4 moved docs generation and templates to docs/ 2017-03-24 15:52:36 -04:00
Brian Coca
7839f70e36 Enable documentation in plugins
Made ansible-doc more plugin agnostic
We can have docs in lookup, callback, connectionm strategy, etc
Use first docstring and make pepizis happy
generalized module_docs to plugin_docs
documented cartesian, ssh, default, jsonfile, etc as examples
changed lack of docs to warning when listing
made smarter about bad docstrings
better blacklisting
added handling of options/config/envs/etc
move blacklist to find_plugins, only need once
2017-03-23 01:27:19 -04:00
Brian Coca
18cbc3b16b minor doc gen fixes 2017-03-20 15:47:24 -04:00
Brian Coca
b606bcec04 Doc directives (#22524)
* draft docs for directives

* updated to document directives
2017-03-15 11:29:58 -04:00
Toshio Kuratomi
eb1214baad New metadata 1.0 (#22587)
Changes to the metadata format were approved here:
https://github.com/ansible/proposals/issues/54
* Update documentation to the new metadata format
* Changes to metadata-tool to account for new metadata
  * Add GPL license header
  * Add upgrade subcommand to upgrade metadata version
  * Change default metadata to the new format
  * Fix exclusion of non-modules from the metadata report
* Fix ansible-doc for new module metadata
* Exclude metadata version from ansible-doc output
* Fix website docs generation for the new metadata
* Update metadata schema in valiate-modules test
* Update the metadata in all modules to the new version
2017-03-14 09:07:22 -07:00
John R Barker
04e816e13b Stricter module documentation validation (#22353)
Raise the bar for module `DOCUMENTAION`
This validator update was used to find the issues in https://github.com/ansible/ansible/pull/22297/files

**Validation**
* Updated Validation and docs to enforce more (items fixed in https://github.com/ansible/ansible/pull/22297/files)
* Use `suboptions` to document complex options 
* Validate module name
* Validate deprecated modules have correct ANSIBLE_METADATA

**Module Documentation Generation**
* Document `suboptions:` Example https://gist.github.com/gundalow/4bdc3669d696268328ccc18528cc6718
* Tidy up HTML generation (valid HTML, no empty lists, etc)
 
**Documentation**
* Clarify the steps for deprecating a module
* Use correct RST headings
* Document `suboptions:` (options)
* Document `contains:` (returns)


**Details**
The aim is to get this (and corresponding module updates) complete by the time `devel` becomes `2.4`, as this allows us to raise the bar for new modules

Example `suboptions` https://gist.github.com/gundalow/4bdc3669d696268328ccc18528cc6718

The aim is to get this PR integrated into `devel` *before* we branch `stable-2.3`, this will allows us to:
* Raise the bar for new modules in 2.4
* Ensure the generated module documentation for 2.3 and higher is improved, important as we will be doing versioned docs moving forward.
2017-03-13 19:49:27 +00:00
Matt Clay
48e82b4ca8 Add test/runner to $PATH in hacking/env-setup. (#22387) 2017-03-07 16:31:53 -08:00
Brian Coca
b714f1fc4b made module notes render nicer on docsite 2017-02-24 15:15:18 -05:00
Matt Martz
5942de603d Legacy pep8 clean fixes for contrib and hacking (#21081) 2017-02-07 09:49:55 -06:00
Allan
ba39d1158c Update test-module (#20737)
* Update test-module

Ensuring invoke is assigned

Traceback (most recent call last):
  File "ansible/hacking/test-module", line 267, in <module>
    main()
  File "ansible/hacking/test-module", line 263, in main
    runtest(modfile, argspath, modname, module_style, interpreters)
  File "ansible/hacking/test-module", line 207, in runtest
    invoke = "%s%s" % (invoke, modfile)
UnboundLocalError: local variable 'invoke' referenced before assignment

* Update test-module

Made the change to only require a single if, making the function more 'DRY'.
2017-01-31 20:05:53 -08:00
Matt Clay
10d9318de7 PEP 8 indent cleanup. (#20800)
* PEP 8 E121 cleanup.

* PEP 8 E126 cleanup.

* PEP 8 E122 cleanup.
2017-01-29 07:28:53 +00:00
Matt Clay
d0d1158c5e PEP 8 cleanup. (#20789)
* PEP 8 E703 cleanup.
* PEP 8 E701 cleanup.
* PEP 8 E711 cleanup.
* PEP 8 W191 and E101 cleanup.
2017-01-28 00:12:11 -08:00
Matt Clay
95789f3949 PEP 8 whitespace cleanup. (#20783)
* PEP 8 E271 whitespace cleanup.
* PEP 8 W293 whitespace cleanup.
* Fix whitespace issue from recent PR.
2017-01-27 15:45:23 -08:00
Matt Clay
63b1e0c277 Fix infrequent PEP 8 issues. 2017-01-27 14:06:21 -08:00
Brian Coca
bcdfdc0cc3 minor cleanups
removed redundant print/exits
removed unused import and var
2017-01-19 19:37:36 -05:00
Adam Johnson
fbfe66967d Doc module meta grammar fixes (#20446)
* Remove unnecessary 'the'
* End sentences with a period
2017-01-19 12:01:55 -08:00
Will Thames
082082857d Use the python used with test-module to run modules (#19591)
* Use ansible_python_interpreter to run modules

Use ansible_python_interpreter to run modules if
`-I ansible_python_interpreter` is set.

Remove unused default from `-I` help text.

* Update test-module to pep8 standards
2017-01-17 17:25:56 -08:00
Matt Clay
cd3fdca540 Switch tests to pytest and ansible-test.
- Replace nose usage with pytest.
- Remove legacy Shippable integration.sh.
- Update Makefile to use pytest and ansible-test.
- Convert most yield unit tests to pytest parametrize.
2017-01-11 12:34:59 -08:00
Brian Coca
3c4afd0470 Doc module meta (#20093)
* updated module docs to reflect unification/meta

* new doc page

* also remove module refs to old core/extra

* Update modules_support.rst

Light edit.
2017-01-10 12:58:22 -08:00
Brian Coca
d138132bc2 show status/support in toc 2017-01-10 12:26:46 -05:00
Toshio Kuratomi
08d6990e67 Fix code-blocks to use correct syntax highlighting 2017-01-08 09:18:45 -08:00
Brian Coca
f5c1aefca9 added symbol conversion to deprecated
fixes #19697
2017-01-05 17:38:30 -05:00
Toshio Kuratomi
d001d250f0 Whitelist the hacking/cherrypick script's use of python3 2017-01-05 10:20:58 -08:00
Toshio Kuratomi
8f6009de0f Add initial script to aid in cherrypicking from devel to stable-2.2 2017-01-05 10:01:22 -08:00
Lujeni
021756a691 Update the module path from the hacking documentation (#19683)
- New path since the modules merge
2016-12-26 18:24:01 +00:00
Brian Coca
0f4ca877ac many doc fixes
removed a bunch of warnings
made nicer formatting of new module info
2016-12-22 12:27:18 -05:00
Sam Doran
73fc4e42f1 Correct type-o on support statement 2016-12-15 11:32:55 -05:00
Matt Clay
bea8a3201d Remove obsolete files and instructions. (#19079)
* Remove obsolete `unify_repos.sh` script.
* Remove submodule update after checkout reference.
2016-12-09 12:42:29 -05:00
Matt Clay
c709b22e5c Fix differences with devel. 2016-12-08 11:35:21 -05:00
Matt Davis
0b47c908eb quick and dirty module doc updates (#18944)
clean up core/extras, plumb in basic metadata display
2016-12-08 11:35:20 -05:00
Matt Davis
cb1888125d add metadata to doc support (#18802)
fix broken module docs
change doc AST id extraction to use == instead of in
2016-12-08 11:35:20 -05:00
Matt Clay
75c281debc Fix compile errors in scripts. 2016-12-08 11:35:20 -05:00
Toshio Kuratomi
ab43a26915 Additional reports (#18779)
* Additional reports
* Fix default status to be a list
2016-12-08 11:35:19 -05:00
Adrian Likins
b5a63a556d fix docs on related to remove of core/extras 2016-12-08 11:35:19 -05:00
Toshio Kuratomi
c53b73228f Update the metadata tool (#18765)
* Latest spreadsheet version uses different fields for support.
* Format the metadata to be a little nicer (multiple lines)
2016-12-06 06:27:10 -08:00
Toshio Kuratomi
d3dd11b21c Update for latest spreadsheet fields 2016-12-05 06:44:37 -08:00
Toshio Kuratomi
39c2249e49 Update metadata for the rename of curated to committer 2016-12-05 04:27:41 -08:00
Brian Coca
bfb43ed0b1 yes, im using bashisms 2016-12-01 09:20:20 -05:00
Brian Coca
2b13ad0ae8 dont skip github modules... 2016-11-30 22:21:29 -05:00
Brian Coca
eef7b8b5a5 now also unifies plugins in same dir structure 2016-11-30 22:18:35 -05:00
Toshio Kuratomi
c22dde615d Add a tool to work with module metadata (#18488)
* Add a tool to work with module metadata

* Don't hardcode async_wrapper
2016-11-14 15:31:02 -08:00
Toshio Kuratomi
b8321c74c8 Fix indentation of multiline descriptions 2016-11-02 17:55:21 -07:00
Matt Clay
0d46805979 Clean up shebangs for various files.
- Remove shebangs from:
  - ini files
  - unit tests
  - module_utils
  - plugins
  - module_docs_fragments
  - non-executable Makefiles
- Change non-modules from '/usr/bin/python' to '/usr/bin/env python'.
- Change '/bin/env' to '/usr/bin/env'.

Also removed main functions from unit tests (since they no longer
have a shebang) and fixed a python 3 compatibility issue with
update_bundled.py so it does not need to specify a python 2 shebang.

A script was added to check for unexpected shebangs in files.
This script is run during CI on Shippable.
2016-11-02 17:00:27 -07:00
Brian Coca
c16a88cac4 keep list as list in synopsis 2016-11-01 16:00:05 -04:00
Sam Doran
cf8639ff62 Fish hacking setup fix (#18084)
* Remove old egg-info files before creating new ones

Currently, setup.py generates egg files then they are deleted. This change
fixes this behavior and matches that in env-setup.

* Do not try to move ansible*egg-info to lib/

setup.py creates the ansible.egg-info in lib/ so this step is unnecessary. Matches env-setup behavior.

* Better test for number of arguments in argv

This prevents an erronous error message from being thrown since set -q returns an error code with the number of variables not defined, resulting in a non-zero exit if no arguments are passed.

Indent case statement within switch statement.
2016-10-24 10:35:00 -07:00
Adrian Likins
cf39a1abab test-module _ansible_selinux_special_fs arg added
modules need to have _ansible_selinux_special_fs passed in
as an arg, so add the default to the args.
2016-10-23 16:34:53 -07:00
Brian Coca
72e6ffad47 added script to unify repos
in case i forget how to do this, can be deleted afterwards
2016-10-03 22:55:38 -04:00
cinerama70
db276373e5 gen_distribution_version_testcase.py should fail if ansible run fails (#17693)
Currently, "ansible localhost -m setup" can fail silently during the
run of gen_distribution_version_testcase.py, resulting in incorrect
output. Use check_output() rather than communicate() and handle
the exception if we get a nonzero return value.
2016-09-26 15:35:36 -07:00
cinerama70
02cec7dca9 Add os_family to test_distribution_version (#17620)
As suggested in feedback on
https://github.com/ansible/ansible/pull/17575, add
os_family to test_distribution_version. Add the
correct os_family to the existing testcase data
entries.

Also add os_family to the output of
gen_distribution_version_testcase.py so any new
generated entries will contain this data.
2016-09-17 23:27:35 +02:00
Toshio Kuratomi
4ed88512e4 Move uses of to_bytes, to_text, to_native to use the module_utils version (#17423)
We couldn't copy to_unicode, to_bytes, to_str into module_utils because
of licensing.  So once created it we had two sets of functions that did
the same things but had different implementations.  To remedy that, this
change removes the ansible.utils.unicode versions of those functions.
2016-09-06 22:54:17 -07:00
Michael Scherer
619f3d13ef Add detection of python3 to hacking/env-setup (#17357)
People can still override the detection using PYTHON_BIN
2016-09-02 12:11:38 -07:00
scottb
235eab6609 Merge pull request #16316 from alikins/alikins_docsite_make
Fewer deps for 'make webdocs' from top level
2016-08-19 12:06:56 -07:00
Josh Smift
81d7e132f9 force deletion of .pyc files (#17109) 2016-08-17 00:43:10 +02:00
Matt Davis
525c0469d2 tweak hacking/env-setup to work under ash (#17054) 2016-08-11 16:58:08 -07:00
Toshio Kuratomi
48a2773463 Find places where ziploader is used and change them to ansiballz so that people aren't confused when they google for information.information (#16715) 2016-07-21 10:58:24 -07:00
Adrian Likins
a1c21f8400 Fewer deps for 'make webdocs' from top level
Previously it also built the MANPAGES target
requiring asciidoc and libxml, before starting
a 'make docs' in docsite.

Also change the #! line in
hacking/dump_playbook_attributes.py to not specify
python2... yet.
2016-06-28 11:01:42 -04:00
André Pinto
fa4011c6c3 Do not corrupt $MANPATH when it's not present (#16300)
Fixes #16299
2016-06-15 11:41:14 -04:00
Matt Davis
ec2cb07988 Make 'required' optional in module docs (#15906)
Updated module dev docs, doc build, ansible-doc to match
2016-05-18 16:57:36 -07:00
Brian Coca
71a707fba5 quick yaml syntax checker 2016-05-18 18:15:12 -04:00
Brian Coca
478674cc57 typo fix 2016-05-05 17:05:46 -04:00
Brian Coca
b7c874f81a enforce required 'required' in docs 2016-05-05 16:55:18 -04:00
Robin Roth
1d6608e84f Dist version fix for Red Hat and more tests (#15663)
* add tests for centos6, rhel6 and rhel7

* gen_distribution_version_testcase with python2.6

* remove unused imports

* fix redhat/vmware/... parsing

* add centos7 test case
2016-04-29 13:18:50 -07:00
Brett Inman
1f89a46b52 Make -q flag totally quiet for env-setup.fish
The first echo and setup.py steps are not quiet and result in output that is annoying if your Fish is configured to source the file.
2016-04-27 11:34:51 -07:00
Toshio Kuratomi
2762f12f7f Remove the duplicate modstyle parameter 2016-04-27 10:04:16 -07:00
Toshio Kuratomi
bdd73e31dc Have test-module clean up the local temp dir when it exits
Get test-module's debugger switch to do something useful with ziploader modules
2016-04-24 20:44:42 -07:00
Toshio Kuratomi
7833b5bec4 Merge pull request #15420 from robinro/distribution_version_tests
Distribution version tests
2016-04-18 10:05:55 -07:00
Robin Roth
692bf51fde move gen_testcase to hacking/tests
* also use json instead of pprint
2016-04-15 17:14:43 +02:00
Toshio Kuratomi
669f3dc3a8 Cleanup. Since we no longer pass a lock, we no longer need to create it 2016-04-14 11:59:06 -07:00
Toshio Kuratomi
d78ba34cf0 We switched away from passing the lock via the arguments to modify_module
Need to fix test-module to not pass the lock either
2016-04-13 13:35:51 -07:00
Toshio Kuratomi
dcc5dfdf81 Controller-side module caching.
This makes our recursive, ast.parse performance measures as fast as
pre-ziploader baseline.

Since this unittest isn't testing that the returned module data is
correct we don't need to worry about os.rename not having any module
data.  Should devise a separate test for the module and caching code
2016-04-12 08:01:07 -07:00
Toshio Kuratomi
b27c424fa1 Fixes to the documentation build (#15356)
* Could only have one alias before.  Subsequent aliases overrode the
  previous ones.  Now multiple aliases work.
* Fix BLACKLISTED_MODULES.   Previously, modules were listed in the
  generated documentation despite being blacklisted
* Deprecated modules form extras were showing the (E) tag and not the
  (D) tag. Reversed that now (Probably not necessary to also show the
  E tag).
* Sort the deprecated modules alphabetically in the Category docs as
  well as the list of all modules
* Optimization: Previously rendered the modules to rst twice once in all
  group and once in individual categories.  Fixed to only render them
  once.
* Add fireball to blacklist and remove async_status (as people need to
  use that).
2016-04-11 17:11:55 -07:00
Toshio Kuratomi
a330a24ccc Python2.6 fix for test-module 2016-04-11 12:13:31 -07:00
Toshio Kuratomi
4b0aa1214c Ziploader
* Ziploader proof of concept (jimi-c)

* Cleanups to proof of concept ziploader branch:

* python3 compatible base64 encoding
* zipfile compression (still need to enable toggling this off for
  systems without zlib support in python)
* Allow non-wildcard imports (still need to make this recusrsive so that
  we can have module_utils code that imports other module_utils code.)
* Better tracebacks: module filename is kept and module_utils directory
  is kept so that tracebacks show the real filenames that the errors
  appear in.

* Make sure we import modules that are used into the module_utils files that they are used in.

* Set ansible version in a more pythonic way for ziploader than we were doing in module replacer

* Make it possible to set the module compression as an inventory var

This may be necessary on systems where python has been compiled without
zlib compression.

* Refactoring of module_common code:

* module replacer only replaces values that make sense for that type of
  file (example: don't attempt to replace python imports if we're in
  a powershell module).
* Implement configurable shebang support for ziploader wrapper
* Implement client-side constants (for SELINUX_SPECIAL_FS and SYSLOG)
  via environment variable.
* Remove strip_comments param as we're never going to use it (ruins line
  numbering)

* Don't repeat ourselves about detecting REPLACER

* Add an easy way to debug

* Port test-module to the ziploader-aware modify_module()

* strip comments and blank lines from the wrapper so we send less over the wire.

* Comments cleanup

* Remember to output write the module line itself in powershell modules

* for line in lines strips the newlines so we have to add them back in
2016-04-05 11:06:17 -07:00
Brian Coca
6a85da7e0c avoid private attributes
hardcode adding with_ for tasks
2016-03-10 01:00:33 -05:00
Brian Coca
e1faa78722 fixed typo 2016-02-29 17:52:55 -05:00
Brian Coca
cbc797a7ed added missing : 2016-02-26 16:27:10 -05:00
Brian Coca
a3489408a5 fixes to playbooks_directives generation
order is now predictable
now correctly substitutes loop for with_
adds local_action to action
2016-02-26 16:18:55 -05:00
Brian Coca
3e28ee0fd4 avoid printing internal loop and loop_args 2016-02-25 19:58:44 -05:00
Brian Coca
fbdcb22e36 now generate list of playbook ojbect directives
TODO: needs links/info and conditionals added
2016-02-25 16:48:37 -05:00
Brian Coca
fe09f7ee49 clarify that requirements are on host that runs it
fixes http://github.com/ansible/ansible-modules-core/issues/3061
2016-02-18 05:09:42 -08:00
Brian Coca
6414c967e4 now check for description and listify if needed
fixes #14371
2016-02-08 12:34:26 -05:00
Toshio Kuratomi
5cd3f71792 Handle utf-8 in module short desc 2016-02-02 13:00:24 -08:00
Jeremy Audet
5cac8efd73 Make "make webdocs" compatible with Python 3
The `webdocs` make target fails under Python 3. It fails due to a variety of
syntax errors, such as the use of `except Foo, e` and `print 'foo'`. Fix #13463
by making code compatible with both Python 2 and 3.
2015-12-08 12:00:53 -05:00
Patrik Lundin
9f9944e355 Replace -delete with portable -exec rm {} \;
Needed on OpenBSD which does not support -delete.
2015-12-02 11:06:09 +01:00
Alberto Gireud
52ded67db5 Update check mode argument 2015-11-08 20:55:10 -06:00
Brian Coca
4a206cdde9 force egg deletion 2015-11-02 11:58:20 -05:00
Brian Coca
5719912e7f added a skip for the test dir in module repos 2015-10-27 19:59:43 -04:00