Commit graph

106 commits

Author SHA1 Message Date
Abhijeet Kasurde
055871cbb8
api: time.clock compatible code (#70650)
time.clock is removed in Python 3.8. Add time.clock
compatible code.

Fixes: #70649

Signed-off-by: Abhijeet Kasurde <akasurde@redhat.com>
2020-07-16 12:24:22 +05:30
jabdr
0690b68bd3
Support datetime.date object in module result (#70595)
* Support datetime.date object in module result

Fixes #70583

* change blank lines for pep8 sanity test
2020-07-14 11:42:40 -04:00
Mark Chappell
8d160b1881
Remove remaining examples of 1.1.1.1 (#70552)
* Remove remaining examples of 1.1.1.1

* Update ec2_group.py
2020-07-14 11:19:59 -04:00
jctanner
b019029bf3
Add intentional unit tests for basic._set_cwd and common.dict_merge (#70283)
* Add unit tests for basic._set_cwd

* incidental coverage for dict_merge

* add test for async stderr inclusion
2020-06-25 14:45:13 -05:00
Matt Clay
98a0995fd0 Clean up unit test boilerplate. 2020-06-22 14:20:33 -07:00
Sam Doran
bc05415109
Only pass kwargs to our string checker not callable checkers (#70151)
Since only check_type_str() accepts extra param, only pass to our checker and
do not pass kwargs to custom checkers.

* Add unit tests
2020-06-19 09:52:05 -04:00
Felix Fontein
a862ff2d43
Deprecation revisited (#69926)
* Allow to specify collection_name separately for deprecation.

* Use new functionality in Ansible.

* Use new functionality in tests.

* Update tagging/untagging functions.

* Update pylint deprecated sanity test.

* Update validate-modules. Missing are basic checks for version_added (validate semantic version format for collections).

* Improve version validation. Re-add version_added validation.

* Make sure collection names are added to return docs before schema validation.

* Extra checks to avoid crashes on bad data.

* Make C# module utils code work, and update/extend tests.

* Add changelog fragment.

* Stop extracting collection name from potentially tagged versions/dates.

* Simplify C# code.

* Update Windows modules docs.

* Forgot semicolons.
2020-06-09 15:21:19 -07:00
Dongsu Park
598e3392a9
Discover Flatcar Linux properly for hostname (#69627)
To avoid issues with Flatcar Container Linux being unable to be found,
detect Flatcar distro name especially for hostname, just like CoreOS
Container Linux was supported.

See also https://github.com/ansible/ansible/issues/69516
2020-06-02 18:41:53 +05:30
Felix Fontein
40f21dfd3c
Version source tagging (automatic and manual) for version_added and deprecation versions (#69680)
* Track collection for version_added.
Validate *all* version numbers in validate-modules.
For tagged version numbers (i.e. version_added), consider source collection to chose validation.

* Make tagging/untagging functions more flexible.

* Tag all versions in doc fragments.

* Tag all deprecation versions issued by code.

* Make Display.deprecated() understand tagged versions.

* Extend validation to enforce tagged version numbers.

* Tag versions in tests.

* Lint and fix test.

* Mention collection name in collection loader's deprecation/removal messages.

* Fix error IDs.

* Handle tagged dates in Display.deprecated().

* Also require that removed_at_date and deprecated_aliases.date are tagged.

* Also automatically tag/untag removed_at_date; fix sanity module removal version check.

* Improve error message when invalid version number is used (like '2.14' in collections).
2020-05-28 22:46:16 -07:00
Felix Fontein
ea04e0048d
Allow to deprecate options and aliases by date (#68177)
* Allow to deprecate options and aliases by date instead of only by version.

* Update display.deprecate().

* Adjust behavior to conform to tested behavior, extend tests, and improve C# style.

* Parse date and fail on invalid date.

This is mainly to make sure that people start using invalid dates, and we eventually have a mess to clean up.

* C# code: improve validation and update/extend tests.

* Make sure that deprecate() is not called with both date and version.

* Forgot to remove no longer necessary formatting.

* Adjust order of warnings in C# code.

* Adjust unrelated test.

* Fix grammar (and make that test pass).

* Don't parse date, and adjust message to be same as in #67684.

* Sanity tests: disable date in past test.

* Validate-modules: validate ISO 8601 date format.

* Validate-modules: switch schema declaration for deprecated_aliases to improve error messages for invalid dates.

* Use DateTime instead of string for date deprecation.

* Validate that date in deprecated_aliases is actually a DateTime.

* Fix tests.

* Fix rebasing error.

* Adjust error codes for pylint, and add removed_at_date and deprecated_aliases.date checks to validate-modules.

* Make deprecation date in the past error codes optional.

* Make sure not both version and date are specified for AnsibleModule.deprecate() calls.

* Stop using Python 3.7+ API.

* Make sure errors are actually reported. Re-add 'ansible-' prefix.

* Avoid crashing when 'name' isn't there.

* Linting.

* Update lib/ansible/module_utils/csharp/Ansible.Basic.cs

Co-authored-by: Jordan Borean <jborean93@gmail.com>

* Adjust test to latest change.

* Prefer date over version if both end up in Display.deprecated().

Co-authored-by: Jordan Borean <jborean93@gmail.com>
2020-05-26 20:23:56 -04:00
Matt Martz
e0f25a2b1f
Strip no log values from module response keys. Fixes #68400 (#69653) 2020-05-21 16:17:57 -04:00
Bob Weinand
f200487414
Fix filedescriptor out of range in select() when running commands (#65058)
* Fix filedescriptor out of range in select() when running commands

* Simplify the run_command() code

Now that we're using selectors in run_command(), we can simplify some of
the code.

* Use fileobj.read() instead of os.read()
* No longer use get_buffer_size() as we can just slurp all of the data
  instead.

Also use a simpler conditional check of whether the selector map is
empty

Co-authored-by: Toshio Kuratomi <a.badger@gmail.com>
2020-05-14 11:46:34 -04:00
Brian Coca
ac509d489b
Revert "stricter permissions on atomic_move when creating new file (#68970)" (#68983)
This reverts commit 566f2467f6.
2020-04-16 12:52:15 -04:00
Brian Coca
566f2467f6
stricter permissions on atomic_move when creating new file (#68970)
fixes #67794
  updated some tests that expected previous defaults
  CVE-2020-1736
2020-04-16 09:06:18 -04:00
Toshio Kuratomi
184f540056 Add some more unittests for fail_json
Test that order of msg as a keyword arg doesn't matter
2020-04-08 07:56:11 -07:00
Toshio Kuratomi
6531ba38f8 Allow the msg argument to fail_json() to be a positional argument
fial_json() requires a message be given to it to inform the end user of
why the module failed.  Prior to this commit, the message had to be a
keyword argument:

    module.fail_json(msg='Failed due to error')

Since this is a required parameter, this commit allows the message to be
given as a positional argument instead:

   module.fail_json('Failed due to an error')
2020-04-08 07:56:11 -07:00
Sam Doran
3461c682c3
Add mechanism for storing warnings and deprecations outside of AnsibleModule (#58993)
* Move warn() and deprecate() methods out of basic.py
* Use _global_warnings and _global_deprications and create accessor functions
    - This lays the foundation for future functions being moved outside of AnsibleModule
       that need an interface to warnings and deprecations without modifying them.
* Add unit tests for new warn and deprecate functions
2020-01-28 12:12:14 -05:00
Sam Doran
617fbad743
synchronize - fix password authentication (#66542)
On Python 2, leave all fds open since there is no mechanism to close specific fds with subprocess.Popen() on Python 2

Add unit tests.

Co-authored-by: Matt Martz <matt@sivel.net>
2020-01-21 10:43:56 -05:00
kaorihinata
3ca4580cb4 Allow no_log=False to silence the no_log warnings for module parameters (#64733)
As AnsibleModule._log_invocation is currently implemented, any parameter
with a name that matches PASSWORD_MATCH triggers the no_log warning as a
precaution against parameters that may contain sensitive data, but have not
been marked as sensitive by the module author.

This patch would allow module authors to explicitly mark the aforementioned
parameters as not sensitive thereby bypassing an erroneous warning message,
while still catching parameters which have not been marked at all by the
author.

Adds tests for various no_log states including True, False, and None (as
extracted by AnsibleModule._log_invocation) when applied to an argument with
a name that matches PASSWORD_MATCH.

Fixes: #49465 #64656
2020-01-09 16:47:57 -05:00
Mads Jensen
0f491c0289 Replace TestCase.assertEquals with TestCase.assertEqual. 2019-11-11 10:24:21 -08:00
Jill R
d49d52eb5f
Add tests for new alias deprecation functionality (#61476)
* Add tests for the alias deprecation added in #61245
2019-09-05 15:47:42 -07:00
Andrey Klychkov
21863d48f3 unit tests: remove unused imports (#60462) 2019-08-13 19:21:43 +02:00
Matt Martz
697b566971
Update units to pass on macOS (#60435)
* Update units to pass on macOS. Fixes #27810

* raising=False
2019-08-12 15:13:07 -05:00
Andrey Klychkov
4e8df9a4b8 unit tests: remove unused imports (#59636) 2019-07-26 10:53:31 -04:00
Felix Fontein
4a574c4d0c Option parsing: warn if both an option and its alias are specified for a module (#53698)
* Print warning when both an option and its alias is specified.

* Improve output.

* Put warnings into self._warnings directly, resp. use self.warn() when handling subspecs.

* Add changelog.

* Add unit test.
2019-07-19 13:11:41 -04:00
Brian Coca
ee4cba1b85 ensure bytes in run_command (#58411)
* ensure we pass bytes to run_command

* changed tests for new behaviour

* dont b the pytest

* fixes by sivel
2019-07-02 17:07:48 -05:00
Sam Doran
ff88bd82b5
Move type checking methods out of basic.py and add unit tests (#53687)
* Move check_type_str() out of basic.py

* Move check_type_list() out of basic.py

* Move safe_eval() out of basic.py

* Move check_type_dict() out of basic.py

* Move json importing code to common location

* Move check_type_bool() out of basic.py

* Move _check_type_int() out of basic.py

* Move _check_type_float() out of basic.py

* Move _check_type_path() out of basic.py

* Move _check_type_raw() out of basic.py

* Move _check_type_bytes() out of basic.py

* Move _check_type_bits() out of basic.py

* Create text.formatters.py

Move human_to_bytes, bytes_to_human, and _lenient_lowercase out of basic.py into text.formatters.py
Change references in modules to point to function at new location

* Move _check_type_jsonarg() out of basic.py

* Rename json related functions and put them in common.text.converters

Move formatters.py to common.text.formatters.py and update references in modules.

* Rework check_type_str()

Add allow_conversion option to make the function more self-contained.
Move the messaging back to basic.py since those error messages are more relevant to using this function in the context of AnsibleModule and not when using the function in isolation.

* Add unit tests for type checking functions

* Change _lenient_lowercase to lenient_lowercase per feedback
2019-03-21 09:40:19 -04:00
Sam Doran
43a44e6f35
Move utility functions out of basic.py (#51715)
Move the following methods to lib/anisble/module_utils/common/validation.py:

- _count_terms()
- _check_mutually_exclusive()
- _check_required_one_of()
- _check_required_together()
- _check_required_by()
- _check_required_arguments()
- _check_required_if
- fail_on_missing_params() --> create check_missing_parameters()
2019-03-14 21:29:55 -04:00
Brian Coca
e280f2f7b0
Try to get correct buffer size to avoid races (#53547)
* Try to get correct buffer size to avoid races

  fixes #51393

* fix test, mock buffer function since all is mocked
2019-03-14 11:04:56 -04:00
Sam Doran
bf28b5ceca
Use distro.id() instead of distro.name() (#52199)
* Update sys_info unit tests

* Correct distribution values in hostname.py

* Normalize Rhel to Redhat
2019-03-07 13:25:59 -05:00
Felix Fontein
07fcb60d55 Python 2: accept both long and int for type=int (module options) (#53289)
* Added unit test
2019-03-05 09:39:03 -05:00
Sam Doran
aba4bed803
Move _handle_no_log_values() out of basic.py (#48628)
* Rename method and make private
* Use is_iterable, combine transformations
* Remove unused return_values from network modules
* Improve docstrings in new functions
* Add new PASS_VAR
* Add unit tests for list_no_log_values
* Fix unit tests for Python 2.6
2019-03-04 17:02:44 -05:00
Sam Doran
2a98faee2b
Move _handle_aliases() out of basic.py (#48578)
Refinements:
- return legal_inputs and update class properties
- remove redundant arguments from method and handle in caller
- add better exception types to method

* Add unit tests for handle_aliases
2019-02-28 16:43:19 -05:00
Ganesh Nalawade
41e2bd1df5
Add support for elements validation in argspec (#50335)
* Add support for elements validation in argspec

Fixes #48473

*  Add support to validate the elements value in argspec
   when type is `list`

* Fix unit test failures

* Add unit test for elements validation

* Fix CI failures

* Fix review comments

* Fix unit test and CI failures after rebase
2019-02-19 17:35:20 +05:30
Sviatoslav Sydorenko
f1e67c3328 Bugfix/ fix mocker patch in tests (#52372)
* 🐛 Fix invalid os.stat mock in tests

* 🐛 Fix leaking mock patch in tests

Closes #52347
2019-02-15 13:58:26 -08:00
Dag Wieers
cd9471ef17 Introduce new 'required_by' argument_spec option (#28662)
* Introduce new "required_by' argument_spec option

This PR introduces a new **required_by** argument_spec option which allows you to say *"if parameter A is set, parameter B and C are required as well"*.

- The difference with **required_if** is that it can only add dependencies if a parameter is set to a specific value, not when it is just defined.
- The difference with **required_together** is that it has a commutative property, so: *"Parameter A and B are required together, if one of them has been defined"*.

As an example, we need this for the complex options that the xml module provides. One of the issues we often see is that users are not using the correct combination of options, and then are surprised that the module does not perform the requested action(s).

This would be solved by adding the correct dependencies, and mutual exclusives. For us this is important to get this shipped together with the new xml module in Ansible v2.4. (This is related to bugfix https://github.com/ansible/ansible/pull/28657)

```python
    module = AnsibleModule(
        argument_spec=dict(
            path=dict(type='path', aliases=['dest', 'file']),
            xmlstring=dict(type='str'),
            xpath=dict(type='str'),
            namespaces=dict(type='dict', default={}),
            state=dict(type='str', default='present', choices=['absent',
'present'], aliases=['ensure']),
            value=dict(type='raw'),
            attribute=dict(type='raw'),
            add_children=dict(type='list'),
            set_children=dict(type='list'),
            count=dict(type='bool', default=False),
            print_match=dict(type='bool', default=False),
            pretty_print=dict(type='bool', default=False),
            content=dict(type='str', choices=['attribute', 'text']),
            input_type=dict(type='str', default='yaml', choices=['xml',
'yaml']),
            backup=dict(type='bool', default=False),
        ),
        supports_check_mode=True,
        required_by=dict(
            add_children=['xpath'],
            attribute=['value', 'xpath'],
            content=['xpath'],
            set_children=['xpath'],
            value=['xpath'],
        ),
        required_if=[
            ['count', True, ['xpath']],
            ['print_match', True, ['xpath']],
        ],
        required_one_of=[
            ['path', 'xmlstring'],
            ['add_children', 'content', 'count', 'pretty_print', 'print_match', 'set_children', 'value'],
        ],
        mutually_exclusive=[
            ['add_children', 'content', 'count', 'print_match','set_children', 'value'],
            ['path', 'xmlstring'],
        ],
    )
```

* Rebase and fix conflict

* Add modules that use required_by functionality

* Update required_by schema

* Fix rebase issue
2019-02-15 10:57:45 +10:00
Toshio Kuratomi
5844c8c7f0 Cleanups to the common.sys_info API
* Move get_all_subclasses out of sys_info as it is unrelated to system
  information.
* get_all_subclasses now returns a set() instead of a list.
* Don't port get_platform to sys_info as it is deprecated.  Code using
  the common API should just use platform.system() directly.
* Rename load_platform_subclass() to get_platform_subclass and do not
  instantiate the rturned class.
* Test the compat shims in module_utils/basic.py separately from the new
  API in module_utils/common/sys_info.py and module_utils/common/_utils.py
2019-01-03 16:21:09 -08:00
Toshio Kuratomi
61b1daa65f Port from plaform.dist to ansible.module_utils.distro.linux_distribution
ci_complete
2018-12-17 11:01:01 -08:00
Andreas Calminder
876b637208 move some of basic into common (#48078)
* move file functions into common.file

* move *_PERM_BITS and mark as private (_*_PERM_BITS)

* move get_{platform, distribution, distribution_version} get_all_subclasses and load_platform_subclass into common.sys_info

* forgot get_distribution_version, properly rename get_all_subclasses

* add common/sys_info.py to recursive finder test

* update module paths in test_platform_distribution.py

* update docstrings, _get_all_subclasses -> get_all_subclasses

* forgot to update names

* remove trailing whitespace
2018-12-07 10:21:11 -08:00
Matt Clay
feb5b0b299 Fix unit test issues with pytest >= 4.0.0. 2018-11-15 10:06:07 -08:00
Matt Clay
3033fd96b0
Move unit test compat code out of lib/ansible/. (#46996)
* Move ansible.compat.tests to test/units/compat/.
* Fix unit test references to ansible.compat.tests.
* Move builtins compat to separate file.
* Fix classification of test/units/compat/ dir.
2018-10-12 20:01:14 -07:00
Matt Clay
53b230ca74 Fix unit test parametrize order on Python 3.5. 2018-09-17 22:45:42 -07:00
Matt Martz
c1c229c6d4
Remove use of simplejson throughout code base (#43548)
* Remove use of simplejson throughout code base. Fixes #42761

* Address failing tests

* Remove simplejson from contrib and other outlying files

* Add changelog fragment for simplejson removal
2018-08-10 11:13:29 -05:00
Toshio Kuratomi
52449cc01a AnsiballZ improvements
Now that we don't need to worry about python-2.4 and 2.5, we can make
some improvements to the way AnsiballZ handles modules.

* Change AnsiballZ wrapper to use import to invoke the module
  We need the module to think of itself as a script because it could be
  coded as:

      main()

  or as:

      if __name__ == '__main__':
          main()

  Or even as:

      if __name__ == '__main__':
          random_function_name()

  A script will invoke all of those.  Prior to this change, we invoked
  a second Python interpreter on the module so that it really was
  a script.  However, this means that we have to run python twice (once
  for the AnsiballZ wrapper and once for the module).  This change makes
  the module think that it is a script (because __name__ in the module ==
  '__main__') but it's actually being invoked by us importing the module
  code.

  There's three ways we've come up to do this.
  * The most elegant is to use zipimporter and tell the import mechanism
    that the module being loaded is __main__:
    * 5959f11c9d/lib/ansible/executor/module_common.py (L175)
    * zipimporter is nice because we do not have to extract the module from
      the zip file and save it to the disk when we do that.  The import
      machinery does it all for us.
    * The drawback is that modules do not have a __file__ which points
      to a real file when they do this.  Modules could be using __file__
      to for a variety of reasons, most of those probably have
      replacements (the most common one is to find a writable directory
      for temporary files.  AnsibleModule.tmpdir should be used instead)
      We can monkeypatch __file__ in fom AnsibleModule initialization
      but that's kind of gross.  There's no way I can see to do this
      from the wrapper.

  * Next, there's imp.load_module():
    * https://github.com/abadger/ansible/blob/340edf7489/lib/ansible/executor/module_common.py#L151
    * imp has the nice property of allowing us to set __name__ to
      __main__ without changing the name of the file itself
    * We also don't have to do anything special to set __file__ for
      backwards compatibility (although the reason for that is the
      drawback):
    * Its drawback is that it requires the file to exist on disk so we
      have to explicitly extract it from the zipfile and save it to
      a temporary file

  * The last choice is to use exec to execute the module:
    * https://github.com/abadger/ansible/blob/f47a4ccc76/lib/ansible/executor/module_common.py#L175
    * The code we would have to maintain for this looks pretty clean.
      In the wrapper we create a ModuleType, set __file__ on it, read
      the module's contents in from the zip file and then exec it.
    * Drawbacks: We still have to explicitly extract the file's contents
      from the zip archive instead of letting python's import mechanism
      handle it.
    * Exec also has hidden performance issues and breaks certain
      assumptions that modules could be making about their own code:
      http://lucumr.pocoo.org/2011/2/1/exec-in-python/

  Our plan is to use imp.load_module() for now, deprecate the use of
  __file__ in modules, and switch to zipimport once the deprecation
  period for __file__ is over (without monkeypatching a fake __file__ in
  via AnsibleModule).

* Rename the name of the AnsiBallZ wrapped module
  This makes it obvious that the wrapped module isn't the module file that
  we distribute.  It's part of trying to mitigate the fact that the module
  is now named __main)).py in tracebacks.

* Shield all wrapper symbols inside of a function
  With the new import code, all symbols in the wrapper become visible in
  the module.  To mitigate the chance of collisions, move most symbols
  into a toplevel function.  The only symbols left in the global namespace
  are now _ANSIBALLZ_WRAPPER and _ansiballz_main.

revised porting guide entry

Integrate code coverage collection into AnsiballZ.

ci_coverage
ci_complete
2018-07-26 20:07:25 -07:00
Toshio Kuratomi
5b4a7cc283 Fix journald unittests
d7df072b96 changed how we call
journal.send() from positional arguments to keyword arguments.  So we
need to update the test to check for the arguments it was called with in
the keyword args, not in the positional args.
2018-07-25 11:20:24 -07:00
Jordan Borean
8bdd04c147 Fix remote_tmp when become with non admin user (#42396)
* Fix tmpdir on non root become

 - also avoid exception if tmpdir and remote_tmp are None
 - give 'None' on deescalation so tempfile will fallback to it's default behaviour
   and use system dirs
 - fix issue with bad tempdir (not existing/not createable/not writeable)
   i.e nobody and ~/.ansible/tmp
 - added tests for blockfile case

* Revert "Temporarily revert c119d54"

This reverts commit 5c614a59a6.

* changes based on PR feedback and changelog fragment

* changes based on the review

* Fix tmpdir when makedirs failed so we just use the system tmp

* Let missing remote_tmp fail

If remote_tmp is missing then there's something more basic wrong in the
communication from the controller to the module-side.  It's better to
be alerted in this case than to silently ignore it.

jborean and I have independently checked what happens if the user sets
ansible_remote_tmp to empty string and !!null and both cases work fine.
(null is turned into a default value controller-side.  empty string
triggers the warning because it is probably not a directory that the
become user is able to use).
2018-07-06 10:49:19 -07:00
Jordan Borean
5c39c3b2d1
Module basic.py to create parent dirs of tmpdir if needed (#40201)
* Module basic.py to create parent dirs of tmpdir if needed

* Added warning to dir creation

* Assert if make_dirs was called or not in unit tests
2018-05-17 09:52:46 +10:00
Jordan Borean
44ab948e5d
create module tmpdir based on remote_tmp (#39833)
* create module tmpdir based on remote_tmp

* Source remote_tmp from controller if possible

* Fixed sanity test and not use lambda

* Added expansion of env vars to the remote tmp

* Fixed sanity issues

* Added note around shell remote_tmp option

* Changed fallback tmp dir to ~/.ansible/tmp to make shell defaults
2018-05-15 09:31:21 +10:00
Matt Martz
1663b64e18
Allow subspec defaults to be processed when the parent argument is not supplied (#38967)
* Allow subspec defaults to be processed when the parent argument is not supplied

* Allow this to be configurable via apply_defaults on the parent

* Document attributes of arguments in argument_spec

* Switch manageiq_connection to use apply_defaults

* add choices to api_version in argument_spec
2018-05-07 11:23:13 -05:00
René Moser
2f36b9e5ce basic: allow one or more when param list having choices (#34537)
* basic: allow one or more when param list having choices

* add unit tests

* optimize a bit

* re-add get_exception import

* a number of existing modules expect to be able to get it from basic.py
2018-02-07 22:59:21 -08:00