Now that we don't need to worry about python-2.4 and 2.5, we can make
some improvements to the way AnsiballZ handles modules.
* Change AnsiballZ wrapper to use import to invoke the module
We need the module to think of itself as a script because it could be
coded as:
main()
or as:
if __name__ == '__main__':
main()
Or even as:
if __name__ == '__main__':
random_function_name()
A script will invoke all of those. Prior to this change, we invoked
a second Python interpreter on the module so that it really was
a script. However, this means that we have to run python twice (once
for the AnsiballZ wrapper and once for the module). This change makes
the module think that it is a script (because __name__ in the module ==
'__main__') but it's actually being invoked by us importing the module
code.
There's three ways we've come up to do this.
* The most elegant is to use zipimporter and tell the import mechanism
that the module being loaded is __main__:
* 5959f11c9d/lib/ansible/executor/module_common.py (L175)
* zipimporter is nice because we do not have to extract the module from
the zip file and save it to the disk when we do that. The import
machinery does it all for us.
* The drawback is that modules do not have a __file__ which points
to a real file when they do this. Modules could be using __file__
to for a variety of reasons, most of those probably have
replacements (the most common one is to find a writable directory
for temporary files. AnsibleModule.tmpdir should be used instead)
We can monkeypatch __file__ in fom AnsibleModule initialization
but that's kind of gross. There's no way I can see to do this
from the wrapper.
* Next, there's imp.load_module():
* https://github.com/abadger/ansible/blob/340edf7489/lib/ansible/executor/module_common.py#L151
* imp has the nice property of allowing us to set __name__ to
__main__ without changing the name of the file itself
* We also don't have to do anything special to set __file__ for
backwards compatibility (although the reason for that is the
drawback):
* Its drawback is that it requires the file to exist on disk so we
have to explicitly extract it from the zipfile and save it to
a temporary file
* The last choice is to use exec to execute the module:
* https://github.com/abadger/ansible/blob/f47a4ccc76/lib/ansible/executor/module_common.py#L175
* The code we would have to maintain for this looks pretty clean.
In the wrapper we create a ModuleType, set __file__ on it, read
the module's contents in from the zip file and then exec it.
* Drawbacks: We still have to explicitly extract the file's contents
from the zip archive instead of letting python's import mechanism
handle it.
* Exec also has hidden performance issues and breaks certain
assumptions that modules could be making about their own code:
http://lucumr.pocoo.org/2011/2/1/exec-in-python/
Our plan is to use imp.load_module() for now, deprecate the use of
__file__ in modules, and switch to zipimport once the deprecation
period for __file__ is over (without monkeypatching a fake __file__ in
via AnsibleModule).
* Rename the name of the AnsiBallZ wrapped module
This makes it obvious that the wrapped module isn't the module file that
we distribute. It's part of trying to mitigate the fact that the module
is now named __main)).py in tracebacks.
* Shield all wrapper symbols inside of a function
With the new import code, all symbols in the wrapper become visible in
the module. To mitigate the chance of collisions, move most symbols
into a toplevel function. The only symbols left in the global namespace
are now _ANSIBALLZ_WRAPPER and _ansiballz_main.
revised porting guide entry
Integrate code coverage collection into AnsiballZ.
ci_coverage
ci_complete
Allow specifying the source and destination files' encodings in the template module
* Added output_encoding to the template module, default to utf-8
* Added documentation for the new variables
* Leveraged the encoding argument on to_text() and to_bytes() to keep the implementation as simple as possible
* Added integration tests with files in utf-8 and windows-1252 encodings, testing all combinations
* fix bad smell test by excluding windows-1252 files from the utf8 checks
* fix bad smell test by excluding valid files from the smart quote test
* Documentation change for resizefs
Changed documentation to match the default value of resizefs set in the code.
Added a note on the resizefs use on the example utilizing it.
* Remove test now it validates fine
* Update Shippable integration test groups.
* Update integration test group aliases.
* Rebalance AWS and Azure tests with extra group.
* Rebalance Windows tests with another group.
* win_chocolatey: refactor module to fix bugs and add new features
* Fix some typos and only emit install warning not in check mode
* Fixes when testing out installing chocolatey from a server
* Added changelog fragment
* Update dnsimple-python minimum version to 1.0.0 as it supports API v2 and API v1 is deprecated.
* Update examples.
* Update documentation.
Fixes: #42495
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* Windows: Add special parameter types
Adding explicit parameter types now exposes this information in the
module documentation, and proves really helpful.
We only do this for non-string types as strings, mostly because strings
are implicit.
PS We also make copyright statements consistent and use #Requires for
explicit library imports
* Type "string" must be type "str"
* A few more Copyright corrections
* More fixes
* Don't add file encoding to Powershell files
* Don't add missing interfacetypes parameter
Otherwise CI demands an incorrect version_added
* Small fix
This library is a backport (to 2.x versions of Python) of the code
that is found in the mainline versions of Python 3.x. It is being
included so that networking vendors, and others, can make use of it
without needing to add a python module dependency to their own modules.
A separate dependency would add to user burden of satisfying those
dependencies before using some Ansible modules.
In a previous core meeting, this was approved. Naming of the directory
it is found in was up for debate, but "compat" was the first directory
to have some sort of concensus.
* Make the code more PowerShell compliant
* Correction of the mistaken comment
* Revert style changes to If/Else/Elseif
* Remove an ignored PSScriptAnalyzer rule due to the code correction
* Decreasing of an entropy ;-) and replacing the next to aliases
* minor whitespace changes
* Adds requests.Session like class
* py2 syntax fix
* Add a few examples to the Request docstrings
* Add helper methods and docs
* Fix test failures
* Switch tests to test Request instead of open_url, add simple open_url test to validate funcitonality
* Fix filename in replace-urlopen code smell test
* First pass at vmware_deploy_ovf functionality
* Add OVA file support, re-structure code
* Move some useful functions to module_utils.vmware, and perform a little DRY too
* Better handling of errors during spec validation and import
* Properly calculate the lease progress percentage for all vmdk files
* Make warnings and errors a little better
* Add an allow_duplicates argument, that defaults to true, to allow users to have name based idempotency
* Add fail_on_spec_warnings to cause the module to treat warnings as errors
* Support non-vmdk uploads
* Add ova alias for ovf
* Rename vmdk_post_url to device_upload_url so it does not sound to specific to VMDK files
* Safer handling of * hostname in urls
* Add default Content-Type, remove unused headers var
* Add deploymentOptions and propertyMapping functionalities
* Add basic check_mode support
* Add vmware_deploy_ovf to list of use-argspec-type-path ignores
* Update version_added and fix path for use-argspec-type-path
* Add configurable folder
* Doc changes
* The module now correctly sets the timezone in both the config file and
in /etc/localtime; while hwclock is set in both the config and
/etc/adjtime.
* Module checks if the timezone is actually set by checking
/etc/localtime. Before it only checked if it was set in the config file.
* Fixed module not setting the timezone on RedHat systems if
/etc/localtime was a symbolic link.
* Fixed module failures in case of missing config files or incorrect data
in them.
* Added a lot of integrations tests to cover most of these situations.
* cs_instance: implement host migration support
* fix build
* fail fast on update if user is not admin
* improve tests a bit
* expunge it
* fix typo
* disable temporarly verify for host on starting instance.
* Refactor ec2_group
Replace nested for loops with list comprehensions
Purge rules before adding new ones in case sg has maximum permitted rules
* Add check mode tests for ec2_group
* add tests
* Remove dead code
* Fix integration test assertions for old boto versions
* Add waiter for security group that is autocreated
* Add support for in-account group rules
* Add common util to get AWS account ID
Fixes#31383
* Fix protocol number and add separate tests for egress rule handling
* Return egress rule treatment to be backwards compatible
* Remove functions that were obsoleted by `Rule` namedtuple
* IP tests
* Move description updates to a function
* Fix string formatting missing index
* Add tests for auto-creation of the same group in quick succession
* Resolve use of brand-new group in a rule without a description
* Clean up duplicated get-security-group function
* Add reverse cleanup in case of dependency issues
* Add crossaccount ELB group support
* Deal with non-STS calls to account API
* Add filtering of owner IDs that match the current account
Change the command to get the interface in a vlan "show vlan" => "show vlan brief"
Change the parsing of the return command of the switch.
The return of the ios command is fixed so i cut with fix number of carracter.
Adding looking for the next line to add the forgeted interfaces.
* fixing azure sanity tests
* fixing sanity
* fixing some tags issues
* removed unnecessary things from managed disk
* fixed location problem
* more sanity fixes
* sanity test fixes
* final sanity fixes?
* final fixes again
* undo changes related to container instance
* removed container instance
* readd again
* fixed stupid mistake
* removed _azure from changes
* one more mistake
* Set encrypted as default and fix empty password reporting changed
* Starting with Postgres 10 `UNENCRYPTED` passwords are removed and
because of that this module fails with the default `encrypted=no`.
Also encrypted passwords are suported since version 7.2
(https://www.postgresql.org/docs/7.2/static/sql-createuser.html) which
went EOL in 2007 and since 7.3 it is the default. Because of this it
makes a lot more sense to make `encrypted=yes` the default. This won't
break backward compatibility, the module would just update the user's
password in the DB in the hashed format and everything else will work
like before. It's also a security bad practice to store passwords in
plain text. fixes#25823
* There was also a bug with `encrypted=yes` and an empty password always
reported as changed.
* Improved documentation for `encrypted`/`password` parameters, and
removed some obsolete notes about passlib.
* Fix clearing user's password to work with all versions of Postgres
* Add tests for clearing the user password
* Fix documentation atfer rebase
* Add changelog fragment
* Modifying cnos-facts, cnos_command and cnos-config in line with the design followed in Ansible. Adding unit test cases for these modules. Added plugins to support them.
* Removing doc fragment conflicts with other modules
* Replacing show with display
* fix problem with documentation and param definition difference
* removed some E324 from ignore.txt
* fixed mistake
* remove one more E324
* removed function app
* fixing append tags
* leaving append tags for later
This PR includes:
- Fixes to the majority of module validation issues
(deliberate inconsistencies between docs and arg_spec)
- Removal of deprecated parameters 'method' and 'protocols'
- A few typos in the documentation
There are still some left-over validation errors, some are deliberate
(like doc strings as default to indicate ranges, etc.)
* Allow subspec defaults to be processed when the parent argument is not supplied
* Allow this to be configurable via apply_defaults on the parent
* Document attributes of arguments in argument_spec
* Switch manageiq_connection to use apply_defaults
* add choices to api_version in argument_spec
* vdirect modules: fix 'import' sanity test
* Remove passing file from import skip list.
* vdirect modules: fix validate-modules warnings
- Arguments with a default should not be marked as required
- add choices in doc
* vdirect_runnable: use formatting function
* should not need <>, but fails without
* adds anchor to keywords page, uses it on plugins pages
* fixes envvar link errors
* harmonize file name and ref name as python_3
* removes undefined-lable from ignore list
* a refactor of pool member and node modules to be inline with current f5 conventions
* Added priority_group_activation to pools
* various other small convention fixes and bug fixes
* Adds gnat provisioning to bigip_provision
* Adds special handling for AFM in bigip_provision
* Add device rebooting for provisioning as necessary
* Refactored route domain module to be inline with current f5 conventions
* Minor refactors across modules
* Remove raw byte-strings from cliconf plugins of supported platforms + edgeos
Remove uses of to_bytes, too
* Update CliConfBase docstring to reflect current position on byte strings
* Include change classification data in metadata.
* Add support for disabled tests.
* Add support for unstable tests.
* Add support for unsupported tests.
* Overhaul integration aliases sanity test.
* Update Shippable scripts to handle unstable tests.
* Mark unstable Azure tests.
* Mark unstable Windows tests.
* Mark disabled tests.
* Start of tests for ansible.module_utils.urls
* Start adding file for generic functions throughout urls
* Add tests for maybe_add_ssl_handler
* Remove commented out line
* Improve coverage of maybe_add_ssl_handler, test basic_auth_header
* Start tests for open_url
* pep8 and ignore urlopen in test_url_open.py tests
* Extend auth tests, add test for validate_certs=False
* Finish tests for open_url
* Add tests for fetch_url
* Add fetch_url tests to replace-urlopen ignore
* dummy instead of _
* Add BadStatusLine test
* Reorganize/rename tests
* Add tests for RedirectHandlerFactory
* Add POST test to confirm behavior is to convert to GET
* Update tests to handle recent changes to RedirectHandlerFactory
* Special test, just to confirm that aliasing http_error_308 to http_error_307 does not cause issues with urllib2 type redirects
* tolerate windows line endings when loading windows module utils. Helpful for old custom windows modules.
* add test modules to demonstrate win line ending module load behaviour.
* attempt to fix sanity check failures
* pep8 fix
* explict skip of test modules from shebang check (core modules must still have expected unix style line endings)
* switch to rstrip() following core team meeting feedback
* Use arg_spec type for comparisons on default and choices
* Further improve type casting
* Make sure to capture output in more places
* Individually report invalid choices
* Update ignore.txt after resolving merge conflicts
* Clean up Grafana docs a bit
But more is needed, default values, missing choices, proper
descriptions, actual sentences :-)
* Remove validate-modules entries
Remove module from ignore lists and some documentation fix in
digital_ocean_block_storage and digital_ocean module.
Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
* Ensure we merge doc fragments early, for comparisons
* Perform boolean conversion from arg_spec data too
* Update valdiate-modules ignore.txt due to recent changes
* Wildcard imports should be taken care of. Enable the pylint check for them
* Remove wildcard import code-smell test as we're now checking via pylint
* Add unused-wildcard-import as ignored in our compat code.
These three files use wildcard imports so that they can export
symbols in a compatible location. The real code lives elsewhere.
So disable the pylint tests for the relevant sections of code.
Enforce module deprecation.
After module has reached the end of it's deprecation cycle we will replace it with a docs stub.
* Replace deprecated modules with docs-only sub
* Use of deprecated past deprecation cycle gives meaningful message (see examples below)
* Enforce documentation.deprecation dict via `schema.py`
* Update `ansible-doc` and web docs to display documentation.deprecation
* Document that structure in `dev_guide`
* Ensure that all modules starting with `_` have a `deprecation:` block
* Ensure `deprecation:` block is only used on modules that start with `_`
* `removed_in` A string which represents when this module needs **deleting**
* CHANGELOG.md and porting_guide_2.5.rst list removed modules as well as alternatives
* CHANGELOG.md links to porting guide index
To ensure that meaningful messages are given to the user if they try to use a module at the end of it's deprecation cycle we enforce the module to contain:
```python
if __name__ == '__main__':
removed_module()
```
* fix broken import
Error was:
lib/ansible/modules/cloud/azure/azure_rm_dnsrecordset.py:223:0: NameError: name 'ARecord' is not defined
* doc: use formatting functions
* remove unused dict
* Skip parameters that are being deprecated
So in most cases when parameters are deprecated, you may not want to
advertise them in the documentation. One reason for this is because
these parameters were not introduced within Ansible but predate
upstreaming them.
So this change avoids reporting parameters that are not documented and
are deprecated. It's only a small subset of the existing E322 errors.
* Add support for E323 with these changes
* Unsure about the other issues now
* Add better error message with E321
* Restore some erroneously remove files
* Readd false positives to ignore.txt
* compare arg+aliases between docs and argument_spec
* Add some special handling for the network modules provider options that also appear in the top level arg spec
* Fix error code for bigip_hostname
* Address merge conflicts due to changes in f5 modules
* Update validate-modules ignore based off a clean execution
* Address merge conflicts
* Address renamed module
* Address recent changes to modules
* Add ignore for ucs_ip_pool
* Update aci modules to get more reliable documentation comparison, but not mutating the module_utils aci_argument_spec
* Update ignore.txt after recent aci updates
* Add extra guard to ensure we handle provider special only for network modules
* Address additional changes to modules
* allow shells to have per host options, remote_tmp
added language to shell
removed module lang setting from general as plugins have it now
use get to avoid bad powershell plugin
more resilient tmp discovery, fall back to `pwd`
add shell to docs
fixed options for when frags are only options
added shell set ops in t_e and fixed option frags
normalize tmp dir usag4e
- pass tmpdir/tmp/temp options as env var to commands, making it default for tempfile
- adjusted ansiballz tmpdir
- default local tempfile usage to the configured local tmp
- set env temp in action
add options to powershell
shift temporary to internal envvar/params
ensure tempdir is set if we pass var
ensure basic and url use expected tempdir
ensure localhost uses local tmp
give /var/tmp priority, less perms issues
more consistent tempfile mgmt for ansiballz
made async_dir configurable
better action handling, allow for finally rm tmp
fixed tmp issue and no more tempdir in ballz
hostvarize world readable and admin users
always set shell tempdir
added comment to discourage use of exception/flow control
* Mostly revert expand_user as it's not quite working.
This was an additional feature anyhow.
Kept the use of pwd as a fallback but moved it to a second ssh
connection. This is not optimal but getting that to work in a single
ssh connection was part of the problem holding this up.
(cherry picked from commit 395b714120522f15e4c90a346f5e8e8d79213aca)
* fixed script and other action plugins
ensure tmpdir deletion
allow for connections that don't support new options (legacy, 3rd party)
fixed tests
* Add validation for the next to last line of a module
* Fix last error code
* Reduce to a single conditional
* Fix conditionals
* Move the final warnings statement to main() in mysql_replication
This patch is primarily a refactor to make the validate-modules arg-spec
no longer generate a traceback. It additionally includes removal of deprecated
code in the virtual server module.
The main patch is to remove the traceback generating code. There are
other small fixes that were made in addition to doing that.
* Removed re-def of cleanup_tokens.
* Changed parameter args to be keywords.
* Changed imports to include new module_util locations.
* Imports also include developing (sideband) module_util locations.
* Changed to using F5Client and plain AnsibleModule to prevent tracebacks caused by missing libraries.
* Removed init and update methods from most Parameter classes (optimization) as its now included in module_utils.
* Changed module and module param references to take into account the new self.module arg.
* Minor bug fixes made during this refactor.
The main purpose of this patch is to do the refactor that
supports replacing tracebacks with fail_json. Additionally, the
following was done.
* Removed re-def of cleanup_tokens.
* Changed parameter args to be keywords.
* Changed imports to include new module_util locations.
* Imports also include developing (sideband) module_util locations.
* Changed to using F5Client and plain AnsibleModule to prevent tracebacks caused by missing libraries.
* Removed init and update methods from most Parameter classes (optimization) as its now included in module_utils.
* Changed module and module param references to take into account the new self.module arg.
* Minor bug fixes made during this refactor.
Primarily, this patch contains refactors to remove tracebacks that
are generated when libraries are missing. There is also,
* Removed re-def of cleanup_tokens.
* Changed parameter args to be keywords.
* Changed imports to include new module_util locations.
* Imports also include developing (sideband) module_util locations.
* Changed to using F5Client and plain AnsibleModule to prevent tracebacks caused by missing libraries.
* Removed init and update methods from most Parameter classes (optimization) as its now included in module_utils.
* Changed module and module param references to take into account the new self.module arg. Minor bug fixes made during this refactor.
This patch was primarily an effort to reduce traceback errors for
work that sivel was doing. Part of (and in some cases in addition to)
that, the following was done.
* Removed re-def of cleanup_tokens.
* Changed parameter args to be keywords.
* Changed imports to include new module_util locations.
* Imports also include developing (sideband) module_util locations.
* Changed to using F5Client and plain AnsibleModule to prevent tracebacks caused by missing libraries.
* Removed init and update methods from most Parameter classes (optimization) as its now included in module_utils.
* Changed module and module param references to take into account the new self.module arg.
* Minor bug fixes made during this refactor.
* fix broken import
* ansible.module_utils.ec2.HAS_BOTO is already used
* wait_timeout parameter: use 'int' type
* wait is always True there
* doc: use formatting function
* Update validate-modules arg_spec introspection to be faster, by only mocking the imports we explicitly list
* The use of types.MethodType in redhat_subscription wasn't py3 compatible, use partial instead
* Remove argument_spec import hacks, make them errors, we can ignore them with ansible-test
* Enable the --arg-spec flag for validate-modules
* Resolve newly added tests as filters
* Add code smell to test for ansible provided jinja tests as filters syntax
* Add docs for no-tests-as-filters code smell test
* Address tests as filters in new integration tests
* Address feedback
* Address feedback 2
* Update validation test for new copyright
Ensure new modules without the new copyright header fail
validation
Ensure existing modules without copyright in top 20 lines fail
* Add documentation of 108 error
Create label in developing modules documentation so that
the validation page can point to it
* Ensure new style copyright header passes test!
* Bulk pep8 fixes - hand crafted
Fix by hand the remaining issues that autopep8 couldn't
* Next batch of hand crafted pep8 fixes
* Ignore W503
https://github.com/PyCQA/pycodestyle/pull/499
* Revert more of W503
* - Netconf plugin addition for iosxr
- Utilities refactoring to support netconf and cliconf
- iosx_banner refactoring for netconf and cliconf
- Integration testcases changes to accomodate above changes
* Fix sanity failures, shippable errors and review comments
* fix pep8 issue
* changes run_command method to send specific command args
* - Review comment fixes
- iosxr_command changes to remove ComplexDict based command_spec
* - Move namespaces removal method from utils to netconf plugin
* Minor refactoring in utils and change in deprecation message
* rewrite build_xml logic and import changes for new utils dir structure
* - Review comment changes and minor changes to documentation
* * refactor common code and docs updates
