Make pyca/cryptography the preferred backend for cryptographic needs (mainly vault) falling back to pycrypto
pyca/cryptography is already implicitly a dependency in many cases
through paramiko (2.0+) as well as the new openssl_publickey module,
which requires pyOpenSSL 16.0+. Additionally, pyca/cryptography is
an optional dep for better performance with vault already.
This commit leverages cryptography's padding, constant time comparisons,
and CBC/CTR modes to reduce the amount of code ansible needs to
maintain.
* Handle wrong password given for VaultAES format
* Do not display deprecation warning for cryptography on python-2.6
* Namespace all of the pycrypto imports and always import them
Makes unittests better and the code less likely to get stupid mistakes
(like using HMAC from cryptogrpahy when the one from pycrypto is needed)
* Add back in atfork since we need pycrypto to reinitialize its RNG just in case we're being used with old paramiko
* contrib/inventory/gce: Remove spurious require on pycrypto
(cherry picked from commit 9e16b9db275263b3ea8d1b124966fdebfc9ab271)
* Add cryptography to ec2_win_password module requirements
* Fix python3 bug which would pass text strings to a function which
requires byte strings.
* Attempt to add pycrypto version to setup deps
* Change hacking README for dual pycrypto/cryptography
* update dependencies for various CI scripts
* additional CI dockerfile/script updates
* add paramiko to the windows and sanity requirement set
This is needed because ansible lists it as a requirement. Previously
the missing dep wasn't enforced, but cryptography imports pkg_resources
so you can't ignore a requirement any more
* Add integration test cases for old vault and for wrong passwords
* helper script for manual testing of pycrypto/cryptography
* Skip the pycrypto tests so that users without it installed can still run the unittests
* Run unittests for vault with both cryptography and pycrypto backend
* Start of ansible config project
moved configuration definitions to external yaml file vs hardcoded
* updated constants to be a data strcutures that are looped over and also return origin of setting
changed to manager/data scheme for base classes
new cli ansible-config to view/manage ansible configuration settings
* prints green for default/unchanged and yellow for those that have been overriden
* added list action to show all configurable settings and their associated ini and env var names
* allows specifying config file to see what result would look like
* TBD update, edit and view options
removed test for functions that have been removed
env_Vars are now list of dicts
allows for version_added and deprecation in future
added a couple of descriptions for future doc autogeneration
ensure test does not fail if delete_me exists
normalized 'path expansion'
added yaml config to setup packaging
removed unused imports
better encoding handling
updated as per feedback
* pep8
* aws integration tests - provide an IAM policy that can be used for running them
* move documentation of aws policies into main integration testing documentation + some updates there
* Update testing_integration.rst
Edits
Made ansible-doc more plugin agnostic
We can have docs in lookup, callback, connectionm strategy, etc
Use first docstring and make pepizis happy
generalized module_docs to plugin_docs
documented cartesian, ssh, default, jsonfile, etc as examples
changed lack of docs to warning when listing
made smarter about bad docstrings
better blacklisting
added handling of options/config/envs/etc
move blacklist to find_plugins, only need once
Changes to the metadata format were approved here:
https://github.com/ansible/proposals/issues/54
* Update documentation to the new metadata format
* Changes to metadata-tool to account for new metadata
* Add GPL license header
* Add upgrade subcommand to upgrade metadata version
* Change default metadata to the new format
* Fix exclusion of non-modules from the metadata report
* Fix ansible-doc for new module metadata
* Exclude metadata version from ansible-doc output
* Fix website docs generation for the new metadata
* Update metadata schema in valiate-modules test
* Update the metadata in all modules to the new version
Raise the bar for module `DOCUMENTAION`
This validator update was used to find the issues in https://github.com/ansible/ansible/pull/22297/files
**Validation**
* Updated Validation and docs to enforce more (items fixed in https://github.com/ansible/ansible/pull/22297/files)
* Use `suboptions` to document complex options
* Validate module name
* Validate deprecated modules have correct ANSIBLE_METADATA
**Module Documentation Generation**
* Document `suboptions:` Example https://gist.github.com/gundalow/4bdc3669d696268328ccc18528cc6718
* Tidy up HTML generation (valid HTML, no empty lists, etc)
**Documentation**
* Clarify the steps for deprecating a module
* Use correct RST headings
* Document `suboptions:` (options)
* Document `contains:` (returns)
**Details**
The aim is to get this (and corresponding module updates) complete by the time `devel` becomes `2.4`, as this allows us to raise the bar for new modules
Example `suboptions` https://gist.github.com/gundalow/4bdc3669d696268328ccc18528cc6718
The aim is to get this PR integrated into `devel` *before* we branch `stable-2.3`, this will allows us to:
* Raise the bar for new modules in 2.4
* Ensure the generated module documentation for 2.3 and higher is improved, important as we will be doing versioned docs moving forward.
* Update test-module
Ensuring invoke is assigned
Traceback (most recent call last):
File "ansible/hacking/test-module", line 267, in <module>
main()
File "ansible/hacking/test-module", line 263, in main
runtest(modfile, argspath, modname, module_style, interpreters)
File "ansible/hacking/test-module", line 207, in runtest
invoke = "%s%s" % (invoke, modfile)
UnboundLocalError: local variable 'invoke' referenced before assignment
* Update test-module
Made the change to only require a single if, making the function more 'DRY'.
* Use ansible_python_interpreter to run modules
Use ansible_python_interpreter to run modules if
`-I ansible_python_interpreter` is set.
Remove unused default from `-I` help text.
* Update test-module to pep8 standards
- Replace nose usage with pytest.
- Remove legacy Shippable integration.sh.
- Update Makefile to use pytest and ansible-test.
- Convert most yield unit tests to pytest parametrize.
- Remove shebangs from:
- ini files
- unit tests
- module_utils
- plugins
- module_docs_fragments
- non-executable Makefiles
- Change non-modules from '/usr/bin/python' to '/usr/bin/env python'.
- Change '/bin/env' to '/usr/bin/env'.
Also removed main functions from unit tests (since they no longer
have a shebang) and fixed a python 3 compatibility issue with
update_bundled.py so it does not need to specify a python 2 shebang.
A script was added to check for unexpected shebangs in files.
This script is run during CI on Shippable.
* Remove old egg-info files before creating new ones
Currently, setup.py generates egg files then they are deleted. This change
fixes this behavior and matches that in env-setup.
* Do not try to move ansible*egg-info to lib/
setup.py creates the ansible.egg-info in lib/ so this step is unnecessary. Matches env-setup behavior.
* Better test for number of arguments in argv
This prevents an erronous error message from being thrown since set -q returns an error code with the number of variables not defined, resulting in a non-zero exit if no arguments are passed.
Indent case statement within switch statement.
Currently, "ansible localhost -m setup" can fail silently during the
run of gen_distribution_version_testcase.py, resulting in incorrect
output. Use check_output() rather than communicate() and handle
the exception if we get a nonzero return value.
As suggested in feedback on
https://github.com/ansible/ansible/pull/17575, add
os_family to test_distribution_version. Add the
correct os_family to the existing testcase data
entries.
Also add os_family to the output of
gen_distribution_version_testcase.py so any new
generated entries will contain this data.
We couldn't copy to_unicode, to_bytes, to_str into module_utils because
of licensing. So once created it we had two sets of functions that did
the same things but had different implementations. To remedy that, this
change removes the ansible.utils.unicode versions of those functions.
Previously it also built the MANPAGES target
requiring asciidoc and libxml, before starting
a 'make docs' in docsite.
Also change the #! line in
hacking/dump_playbook_attributes.py to not specify
python2... yet.
* add tests for centos6, rhel6 and rhel7
* gen_distribution_version_testcase with python2.6
* remove unused imports
* fix redhat/vmware/... parsing
* add centos7 test case
This makes our recursive, ast.parse performance measures as fast as
pre-ziploader baseline.
Since this unittest isn't testing that the returned module data is
correct we don't need to worry about os.rename not having any module
data. Should devise a separate test for the module and caching code
* Could only have one alias before. Subsequent aliases overrode the
previous ones. Now multiple aliases work.
* Fix BLACKLISTED_MODULES. Previously, modules were listed in the
generated documentation despite being blacklisted
* Deprecated modules form extras were showing the (E) tag and not the
(D) tag. Reversed that now (Probably not necessary to also show the
E tag).
* Sort the deprecated modules alphabetically in the Category docs as
well as the list of all modules
* Optimization: Previously rendered the modules to rst twice once in all
group and once in individual categories. Fixed to only render them
once.
* Add fireball to blacklist and remove async_status (as people need to
use that).
* Ziploader proof of concept (jimi-c)
* Cleanups to proof of concept ziploader branch:
* python3 compatible base64 encoding
* zipfile compression (still need to enable toggling this off for
systems without zlib support in python)
* Allow non-wildcard imports (still need to make this recusrsive so that
we can have module_utils code that imports other module_utils code.)
* Better tracebacks: module filename is kept and module_utils directory
is kept so that tracebacks show the real filenames that the errors
appear in.
* Make sure we import modules that are used into the module_utils files that they are used in.
* Set ansible version in a more pythonic way for ziploader than we were doing in module replacer
* Make it possible to set the module compression as an inventory var
This may be necessary on systems where python has been compiled without
zlib compression.
* Refactoring of module_common code:
* module replacer only replaces values that make sense for that type of
file (example: don't attempt to replace python imports if we're in
a powershell module).
* Implement configurable shebang support for ziploader wrapper
* Implement client-side constants (for SELINUX_SPECIAL_FS and SYSLOG)
via environment variable.
* Remove strip_comments param as we're never going to use it (ruins line
numbering)
* Don't repeat ourselves about detecting REPLACER
* Add an easy way to debug
* Port test-module to the ziploader-aware modify_module()
* strip comments and blank lines from the wrapper so we send less over the wire.
* Comments cleanup
* Remember to output write the module line itself in powershell modules
* for line in lines strips the newlines so we have to add them back in
The `webdocs` make target fails under Python 3. It fails due to a variety of
syntax errors, such as the use of `except Foo, e` and `print 'foo'`. Fix#13463
by making code compatible with both Python 2 and 3.
Replace .iteritems() with six.iteritems() everywhere except in
module_utils (because there's no 'six' on the remote host). And except
in lib/ansible/galaxy/data/metadata_template.j2, because I'm not sure
six is available there.
test-module is useful but sometimes you want to edit the
result before running it to e.g. set a debug point.
Added a noexecute option (i.e. just create the module script, don't
run it) and an output option to choose the filename of the result.
added docs for new v2 features
restructures modules bar and core/extras info into their own pages
changed templates to refer to the new pages
added some missing ansible.cfg options
more info on gathering setting
now version_added < 1.3 does not get shown, up from 1.0
option's version_added is also now filterd against this threshold
module version_added is more prominent
exaples now uses pure rst instead of intermingled with html formatting
aliases now shown in description for options
bad version fields now throw warnings instead of exceptions
ansible-doc errors now show traceback in very very verbose mode, for easier debugging
- clearer 'version added' for module options, now it sits under the option name
- made notes a section, so it now appears in toc
- moved requirements and made it a list, more prominent and more readable
Default python interpreter to the same interpreter the test-module
script is executed with. This is so that the interpreter doesn't have
to be specified twice in the command when using non-default python
(e.g. ``/path/to/python ./hacking/test-module -I python=/path/to/python ...``)
The default ksh in OpenBSD throws the following error:
===
$ . hacking/env-setup
ksh: hacking/env-setup[23]: ${.sh.file}": bad substitution
[...]
===
The same error can be seen on Linux if pdksh is used.
