Commit graph

346 commits

Author SHA1 Message Date
Matt Davis
cc8d180801
fix internal cases of actions calling unqualified module names (#70818) (#70840)
* fix internal cases of actions calling unqualified module names

* add porting_guide entry
* misc other fixes around action/module resolution broken by redirection

ci_complete

* Update docs/docsite/rst/porting_guides/porting_guide_2.10.rst

Co-authored-by: Rick Elrod <rick@elrod.me>

* Update docs/docsite/rst/porting_guides/porting_guide_2.10.rst

Co-authored-by: Rick Elrod <rick@elrod.me>

* address review feedback

* pep8

* unit test fixes

* win fixes

* gather_facts fix module args ignores

* docs sanity

* pep8

* fix timeout test

* fix win name rewrites

Co-authored-by: Rick Elrod <rick@elrod.me>
(cherry picked from commit 4c0af6c808)
2020-07-23 10:29:09 -07:00
Brian Coca
6cd015d7e2
Make filter type errors 'loop friendly' (#70417) (#70574)
- ensure we preserve the typeerror part of the exception so loop defereed error handling
 can postpone those caused by undefined variables until the when check is done.
 - fix tests to comply with the 'new normal'

 - human_to_bytes and others can issue TypeError not only on 'non string'
 but also bad string that is not convertable.

Co-authored-by: Sloane Hertel <shertel@redhat.com>

Co-authored-by: Sloane Hertel <shertel@redhat.com>
(cherry picked from commit cf89ca8a03)
2020-07-17 12:51:18 -07:00
Felix Fontein
15355ed059
[2.10] ansible-doc: include collection name in text output / plugin loader: return collection name; ansible-doc: handle ansible.builtin correctly (#70572)
* ansible-doc: include collection name in text output (#70401)

* ansible-doc: include collection name in text output

* Be more careful to not accidentally pass ansible.builtin for user-supplied modules.

(cherry picked from commit f4c89eab23)

* plugin loader: return collection name; ansible-doc: handle ansible.builtin correctly (#70026)

* Determine collection in plugin loader.

* Fix test.

* Use PluginPathContext objects in PluginLoader._plugin_path_cache instead of tuples.

(cherry picked from commit 24dcaf8974)
2020-07-17 12:50:23 -07:00
Martin Krizek
7dfda4026e
Fix delegate_facts with interpreter not being set (#70293) (#70384)
Fixes #70168

ci_complete

Co-authored-by: Brian Coca <bcoca@users.noreply.github.com>
Co-authored-by: Matt Clay <matt@mystile.com>
(cherry picked from commit b05e00e99a)
2020-07-17 12:40:00 -07:00
Matt Clay
8cd66ce95a [stable-2.10] Clean up unit test boilerplate.
(cherry picked from commit 98a0995fd0)

Co-authored-by: Matt Clay <matt@mystile.com>
2020-07-13 18:28:02 -07:00
Sloane Hertel
51f6d129cb
support hard coded module_defaults.yml groups for collections (#69919)
* Only allow groups which were hardcoded in module_defaults.yml

only load action groups from the collection if module_defaults contains a potential group for the action

* Fix tests using modules that override those whitelisted in lib/ansible/config/module_defaults.yml

Third party modules should not be using group/ - use the action name instead

* add externalized module_defaults tests

add the missing group and collections

ci_complete

Co-authored-by: Matt Davis <mrd@redhat.com>

* changelog

ci_complete

* Fix import in tests

ci_complete

* Update with requested changes

ci_complete

* don't traceback since we don't validate the contents of module_defaults

ci_complete

Co-authored-by: Matt Davis <mrd@redhat.com>
2020-06-09 15:38:57 -07:00
Matt Davis
984216f52e
various deprecation, display, warning, error fixes for collections redirection (#69822)
* various deprecation, display, warning, error fixes

* Update lib/ansible/utils/display.py

Co-authored-by: Felix Fontein <felix@fontein.de>

* Update lib/ansible/utils/display.py

Co-authored-by: Felix Fontein <felix@fontein.de>

* Update lib/ansible/utils/display.py

Co-authored-by: Felix Fontein <felix@fontein.de>

* cleanup, test fixes

* add collection name to deprecated() calls

* clean up redirect entries from uncommitted tests

* fix dep warning/error header text to match previous

Co-authored-by: Felix Fontein <felix@fontein.de>
2020-06-05 00:28:40 -07:00
Nilashish Chakraborty
dae3ba71a8
Add FQCN mappings for network facts modules (#69601)
* Add FQCN mappings for network facts modules

Signed-off-by: NilashishC <nilashishchakraborty8@gmail.com>

* Add unit tests

Signed-off-by: NilashishC <nilashishchakraborty8@gmail.com>

* update entry for ironware

Signed-off-by: NilashishC <nilashishchakraborty8@gmail.com>
2020-05-22 23:20:21 +05:30
Brian Coca
de3f7c7739
fix delegated interpreter discovery (#69604)
* fix delegated interpeter
* allow returning fact if it is 'the right host'
* added note for future fix/efficiency
 as it stands we rerun discovery for the delegated host
unless its saving facts to itself
 * fixed test lacking delegate_to mock
2020-05-22 09:31:34 -04:00
Bob Weinand
f200487414
Fix filedescriptor out of range in select() when running commands (#65058)
* Fix filedescriptor out of range in select() when running commands

* Simplify the run_command() code

Now that we're using selectors in run_command(), we can simplify some of
the code.

* Use fileobj.read() instead of os.read()
* No longer use get_buffer_size() as we can just slurp all of the data
  instead.

Also use a simpler conditional check of whether the selector map is
empty

Co-authored-by: Toshio Kuratomi <a.badger@gmail.com>
2020-05-14 11:46:34 -04:00
Jordan Borean
702949e64c
windows - fix up env var quoting for unicode single quotes (#68968)
* windows - fix up env var quoting for unicode single quotes

* Add sanity ignore check for smart quotes
2020-04-16 20:24:23 +10:00
Ansible Core Team
e0e57b4beb migration final cleanup 2020-03-23 11:14:21 -05:00
Ansible Core Team
35063de590 Migrated to junipernetworks.junos 2020-03-23 11:14:21 -05:00
Ansible Core Team
8ee0187388 Migrated to openstack.cloud 2020-03-23 11:14:21 -05:00
Ansible Core Team
7e04b5ba8b Migrated to cisco.ios 2020-03-23 11:14:21 -05:00
Ansible Core Team
6da1ec1d8c Migrated to check_point.mgmt 2020-03-23 11:14:21 -05:00
Ansible Core Team
48d4e9370e Migrated to ansible.windows 2020-03-23 11:14:21 -05:00
Ansible Core Team
829eb0f1d1 Migrated to ansible.posix 2020-03-23 11:14:21 -05:00
Ansible Core Team
9a7a574ffc Migrated to ansible.netcommon 2020-03-23 11:14:21 -05:00
Ansible Core Team
6d910034a0 Migrated to ansible.amazon 2020-03-23 11:14:21 -05:00
Ansible Core Team
bb3494356a Migrated to community.windows 2020-03-23 11:14:21 -05:00
Ansible Core Team
4e488d8435 Migrated to community.general 2020-03-23 11:14:21 -05:00
Ansible Core Team
58e8a91f4d Migrated to community.amazon 2020-03-23 11:14:21 -05:00
Rick Elrod
21dfec6c4f make test_raw use setUp instead of global definitions
Signed-off-by: Rick Elrod <rick@elrod.me>
2020-03-06 09:34:21 -08:00
Rick Elrod
65edd5e37f Split test_connection into individual files
Signed-off-by: Rick Elrod <rick@elrod.me>
2020-03-05 14:08:38 -08:00
Matt Clay
46fdbbdd18
Split test_play_context_make_become_cmd into files (#68026)
* Split test_play_context_make_become_cmd into files

For NWO migration. Split the become module assertions into distinct test
files and functions. For now, this is done naively - there is probably
room to abstract these tests out and remove some of the duplication
later on.

Signed-off-by: Rick Elrod <rick@elrod.me>

* use default_exe variable instead of hardcoding /bin/bash

Signed-off-by: Rick Elrod <rick@elrod.me>

* Move become plugin tests to their proper directory and rename them accordingly

Signed-off-by: Rick Elrod <rick@elrod.me>

* Fix up fixtures and imports.

* Remove stray file.

Co-authored-by: Matt Clay <matt@mystile.com>
2020-03-04 22:26:04 -08:00
Matt Clay
04666c9fa1
Clean up unit tests to assist with migration. (#67875)
* Split out cache plugin unit tests.

* Rename unit tests to match code under test.

* Relocate unit test code to match code under test.

* Another rename.

* Update sanity ignores.
2020-02-28 20:29:41 -08:00
Brian Scholer
fc7980af9a
Fix UNC path support in the powershell shell plugin (#66604)
* Fix UNC path joining in the powershell shell plugin, add test

* Remove testy bits and a redundant line

* Fix style nits

* Update to use os.ntpath

* Add changelog for #66604
2020-02-04 16:34:11 +10:00
Rick Elrod
e2a57414f4 Remove with statement for pytest-mock unit tests
As per:
https://github.com/pytest-dev/pytest-mock#note-about-usage-as-context-manager
pytest-mock is not meant to be used within a `with` context or as a
decorator. Instead, pytest-mock will automatically unpatch the mocked
methods when each test is complete.

In newer pytest-mock, this use actually throws an exception and causes
the tests to fail.

This hasn't been hit in Ansible's CI yet, because the docker image
that the tests run in uses an older version of pytest-mock. However,
there is no constraint on the upper bound of pytest-mock in
test/lib/ansible_test/_data/requirements/constraints.txt which means
that when running the tests locally, outside of that docker image, the
tests never pass.

This patch removes the `with` context in each such case.

Signed-off-by: Rick Elrod <rick@elrod.me>
2020-01-29 13:29:40 -08:00
Pilou
bbbdc1c25c throttle: fix linear based strategies (#65422)
* throttle tests: fix detection of parallel execution

The test wasn't able to detect if too many workers were running.

On my laptop:
- without this change, the 'throttle' target takes ~20 seconds
- with this change, the 'throttle' target takes ~70 seconds
- 1 second isn't long enough to encounter the issue

* Fix throttle test when strategy is 'free' based

'free' strategy allows multiple tasks to be executed in parallel: use
one 'throttledir' per task.

Use 'linear' strategy with a dedicated play for cleanup/setup tasks

* throttle: reset worker idx before queuing a new task

* TestStrategyBase: define task.throttle

otherwise '1' will be used instead of the default value due to the
following expression being equal to '1':

    int(templar.template(task_mock.throttle))

Co-authored-by: James Cammarata <jimi@sngx.net>
2020-01-24 08:18:08 -06:00
psharkey
f8fb391548 new connection plugin aws_ssm (#49652)
* new connection plugin aws_ssm

Return code may be at the end of the last command output line.

Marking regex.

Ensure command status code is on it's owm line - last 3 lines are not part of command stdout.

* Adding timeout parameter - aws_ssm_timeout

Default 10 second timeout (https://docs.ansible.com/ansible/2.4/intro_configuration.html#timeout) is marginal.
This avoids changing this default and allowing the SSM timeout to be controlled via inventory.
This change wraps commands so commands which may never return do timeout.

* Added integration tests

Added AWS SSM Executor, target and config functions

Fixed more code for integration tests

Improved execution

Added S3 bucket name

Fixed pylint

Reverted lib changes

Reverted few more changes

Improved support for integration test execution

added ansible role for aws_ssm_integration_test setup and teardown and modifiled runme.sh

Reset to 17fa565 commit

inventory file location changed

change inventory file location

deleted meta and handlers folder as it is not required

deleted main.yml inside vars, removed extra space from tasks/main.yml, Added appropriate tags for ec2 and delete test folder as it is not required

deleted main.yml inside vars, removed extra space from tasks/main.yml, Added appropriate tags for ec2 and delete test folder as it is not required

modified task/main.yml

added region variable and fixed pattern for using variable

modified policy for IAM role

moved to first line of scrip set -eux

Updated Session Manager plugin installation

edited custme policy

Included tags for Session Manager plugin installation

Added README.md

Upddated README and added support for ssm-plugin for Amazon-Linux

Added Windows Integration test support

Improved user data for Linux

Added random value generation for the role and policy,delete vars_to_delete.yml

upadte README in vars

fixed typo

update policy

Updated IAM policy file

update playbook

Updated playbook to include ssm-agent userdata

modified jing2 template

modified jing2 template

modified jing2 template and fixed role deletion

fixed role name issue while deleting and task name

Updated playbook to include wait_connection for ec2

Corrected Synatx changes and updated ssm-plugin debian file

Changed region variable to us-east-1

Removed vars file and updated  to /tmp dir

fixed typo

Improved setup

Fixed boto3 dependency

Fixed missing tag

Added boto as dependency as well

Improved execution workflow

Trying other way of defining tags

Fixed undefined var

Changed AMI ID to Amazon Linux

Improved Tags

Ok, created different directory for WIndows test execution

Fixed IAM Role Name for Windows

Fixed inventory not found

Improved integration test execution

Fixed Windows Inventory path

Fixed wrong Windows AMI ID

Fixes issue for windows test execution

* Don't attempt to terminate sessions without a session id.

* Added Unit test cases file for AWS SSM Connection plugin

updated test file with close

updated unit test file with start_session

updated test files

* Eliminate AWS CLI dependency for terminal session.

* Removing unused code, cleanup logic.

Reduce mark length - 52^26 should be plenty
Be explicit about subprocess.Popen options
Simplify if/else for mark end
_stdin_readline is not used now

* updated test file

Added exec command and fixed close session unit tests

updated test files

Improved ansible ssm test command

updated file for lint checks

updated for pylint checks

New Unit_testcases for pre-signed URL file

removing additonal spaces and white spaces

remaning error changes

fixed changes

fixed spaces issues

python 2.7 version and whitespaces

python 2.7 version and whitespaces

python 2.7 skip if

space issue with 16:1

Unit test cases for windows and linux

Unit test cases for windows and linux with issues fixed issues

Unit test cases for windows and linux with issues fixed issues1

* Added support for S3 Pre-signed URLs

* Updated documentation and comments

* Documentation and curl dependency removal for controller machine

Fixing lint errors and removing requirements.

* Adding support for Windows remote EC2 instances.

* Added Encoding fixes

* Updating author section and adding obvious requirement for the SSM agent.

* Refactor stdout post processing
Attempt to get real return code on error (test using ansible -m raw -a 'cmd /c exit 99'.
Fixes problem at terminal width (ansible windows -i ./hosts.yml -m setup).

* Refactor back to a single module.

* Fixed fetch file for windows

* ssm usage examples for linux and windows

* Update aws_ssm.py

Service state corrected.

* Strip line continuation when at terminal width - otherwise replace.
Strip ANSI control sequences only for Windows.

Test playbook:

---
- name: test
  hosts: windows
  gather_facts: False
  vars:
    small: 'abc'
  tasks:
  - name:
    set_fact:
      large: "{{ lookup('password', '/dev/null length=2000 chars=ascii_letters,digits,hexdigits,punctuation') }}"
  - name: small fixed
    raw: echo '{{ small }}'
    register: small_result
    changed_when: False
  - name: check
    assert:
      that:
        - "(item | length) == (small | length)"
        - "item == small"
      msg: "'{{ item | length }} must equal '{{ small | length }}' and '{{ item }}' must match '{{ small }}'"
    with_items:
      - "{{ small_result.stdout_lines[0] }}"
  - name: large random
    raw: echo '{{ large }}'
    register: large_result
    changed_when: False
  - name: check
    assert:
      that:
        - "(item | length) == (large | length)"
        - "item == large"
      msg: "'{{ item | length }} must equal '{{ large | length }}' and '{{ item }}' must match '{{ large }}'"
    with_items:
      - "{{ large_result.stdout_lines[0] }}"
  - name: gather facts
    setup:

* Correct module parameter names.

* Updated Windows Executable variable.

Updated Windows Executable variable to "ansible_shell_type".

Fixing Examples with raw declaration

Updated the plugin timeout variable.

* Fix to work with dynamic inventory plug-in

* IntegrationTest template updates

* Removing unsupported flag for integration tests.
Fixing unit test.

* Adding shippable group.

* SSM Usage examples with dynamic inventory plugin

* Fixing yamllint errors.

* Fixed Integration tests

* Fixed Integration tests

* Updates for python3.

Removing python3 restriction.

* Remove python3 restriction.
Change block from retry to always

* Fixed Integration tests with Python 3

* Fixed shellcheck

* Fix for Windows which could pick up end mark prematurely
Move debug to _wrap_command and use a single return point

Single-quotes not needed around linux marks
Fix typo in comment

End mark to new command.

* Unit test cases now works on Python2 and Python3

* Skip tests on Python 2.6

* Fix for wait_for_connection module for windows.

* Updated changes as per review comments

* Fixing broken pipe error seen with session-manager-plugin version 1.1.17.0.
Eliminating sleep as this looks to be fixed in session-manager-plugin version 1.1.17.0.

* Adding back delays for Windows with session-manager-plugin 1.1.17.0.

* Updating Windows AMI ID for integration Test

* Upgrading windows ssm agent to the latest

* Adding boilerplate code.

* Windows ami and integration test updates

* Revert "Windows ami and integration test updates"

This reverts commit cd6ca3579b7cda584bd9c065f9c0835bddb23627.

* Updating windows ami for Integration tests

* Integration test suite updates and fixes.

* Updates and fixes

* Eliminate duplicate processing for exit code on failed command.

* Add powershell wrap.

* Refactor windows post_process.

* AMI Lookup, aliases, OSC filter, test suite updates

Co-authored-by: Gaurav Ashtikar <gau1991@gmail.com>
Co-authored-by: Deepak Choudhary <40276333+deepsvc@users.noreply.github.com>
Co-authored-by: Hanumanth <46720371+hanumantharaomvl@users.noreply.github.com>
Co-authored-by: KUMAR MAYANK <mayank@flux7.com>
2020-01-11 15:22:01 +10:00
Abhay Kadam
2fa8f9cfd8 Fix env lookup plugin error on utf8 values (#65541)
This commit fixes issue #65297.

The env lookup plugin used to fail when environment variable value
contained any UTF-8 characters (e.g., δ, ζ).
2019-12-25 17:24:38 +05:30
Paul Belanger
093d1500b9
Clean up flake8 for ansible.netcommon collection (#65842)
We noticed the following flake8 errors when we ran the migration script
for ansible.netcommon.

Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2019-12-16 08:24:38 -05:00
chkp-orso
b1e6667664 fix all modules to be able to get parameter with value false (very important fix) (#65040)
* add import of future and __metaclass__

* fix all modules to be able to get parameter with value false (very importent fix)

* Update ignore.txt
2019-11-20 12:09:40 +05:30
Jordan Borean
7092c196ed
config manager str type vault source (#65023)
* config manager str type vault source

* Convert vault text to_text earlier and add tests
2019-11-20 12:02:28 +10:00
Jordan Borean
480b106d65
become - stop using play context in more places (#62373)
* become - stop using play context in more places - ci_complete

* Fix up review points
2019-11-18 05:32:56 +10:00
Vincent Bernat
bf19060683 ipaddr: add an option to return peer of a point-to-point link (#59144)
* ipaddr: add an option to return peer of a point-to-point link

Signed-off-by: Vincent Bernat <vincent@bernat.ch>

* ipaddr: extend "peer" to also work with /30
2019-11-13 13:47:05 -05:00
Mads Jensen
0f491c0289 Replace TestCase.assertEquals with TestCase.assertEqual. 2019-11-11 10:24:21 -08:00
Mads Jensen
c5d61953b1 Remove empty setUp/tearDown/tearDownClass methods in test classes. 2019-11-06 08:14:29 -08:00
chkp-orso
3a0c6454a9 Enable logging to any domain in the check point machine (#63976)
* enable using any domain in the check point machine

* Update checkpoint.py

* trying to checge `test_chrckpoint` according to `test_ftd` in order to pass the tests

* Update test_checkpoint.py
2019-11-05 17:44:34 +05:30
Pierre Prinetti
2444fae208 Add a named parameter to the filter to_uuid (#64057)
Let the caller choose a namespace for `to_uuid` and document the
behaviour of both the default case, and the new explicit case.

This PR does not change the existing behaviour of the `to_uuid` UUIDv5
filter.
2019-11-01 13:11:34 -04:00
Mark Chappell
ef7d060a3f AWS module_utils: Clear out Sanity Test issues (#63991) 2019-10-28 09:21:40 +00:00
Sam Doran
987265a6ef
Account for empty strings when splitting the host pattern (#62442)
Improve tests
- add more unit test cases
- add specific integration test with more cases

Testing shows no major downside to calling .strip() twice in a comprehension vs. using a regular for loop and only calling .strip() once. Going with the comprehension for ease of maintenance and because comprehensions are optimized in CPython.
2019-09-20 16:03:51 -04:00
Evgeni Golov
8d0c193b25 allow before/after diff to be NoneType (#62582)
when creating or deleting an object (e.g. via an API), before/after can
be `None` (or at least represented as such by the used library). to
avoid modules havig to do

    diff={'before': before or '', 'after': after or ''}

let's just convert `None` to an empty string that can be diffed properly
2019-09-20 15:47:18 -04:00
Trishna Guha
5cd3be9129
gather_facts action plugin: Fix loading network facts modules for smart gathering (#59856)
* fix smart gathering for network_os in gather_facts action plugin

Signed-off-by: Trishna Guha <trishnaguha17@gmail.com>

* Fix detection of network_os for smart gathering

Signed-off-by: Trishna Guha <trishnaguha17@gmail.com>

* Add unittest

Signed-off-by: Trishna Guha <trishnaguha17@gmail.com>

* make pep8 happy

Signed-off-by: Trishna Guha <trishnaguha17@gmail.com>
2019-09-17 18:35:44 +05:30
Nathaniel Case
50e09be14f
Allow the use of _paramiko_conn even if the connection hasn't been started. (#61570)
* Allow the use of _paramiko_conn even if the connection hasn't been started.

I'm not sure what the benefit is of Noneing paramiko_conn on close, but will keep for now

* Fix test

* Try to fix up net_put & net_get

* Add changelog
2019-09-09 16:59:20 -04:00
Vitalii Kostenko
6f7cd8370a Ansible FTD Module improvements and tests update. (#60640)
* Ansible FTD Module improvements and tests update.

* Fix sanity tests

* Fix formatting
2019-09-03 08:38:11 +05:30
Nathaniel Case
41e4ece44b
Defer passlib dependency from plugins/filter/network.py (#59304)
* Defer passlib dependency from plugins/filter/network.py to what utils/encrypt.py provides.

* Revert test changes now that passlib is optional
2019-08-29 09:01:30 -04:00
James Cammarata
51b33b79c0
T woerner max concurrent (#60702)
* play, block, task: New attribute forks

With this it is possible to limit the number of concurrent task runs.
forks can now be used in play, block and task. If forks is set in different
levels in the chain, then the smallest value will be used for the task.

The attribute has been added to the Base class as a list to easily provide
all the values that have been set in the different levels of the chain.

A warning has been added because of the conflict with run_once. forks will
be ignored in this case.

The forks limitation in StrategyBase._queue_task is not used for the free
strategy.

Signed-off-by: Thomas Woerner <twoerner@redhat.com>

* Handle forks in free strategy

The forks attribute for the free strategy is handled in run in the free
StrategyModule. This is dony by counting the amount of tasks where the uuid
is the same as the current task, that should be queued next. If this amount
is bigger or equal to the forks attribute from the chain (task, block,
play), then it will be skipped to the next host. Like it is also done with
blocked_hosts.

Signed-off-by: Thomas Woerner <twoerner@redhat.com>

* Test cases for forks with linear and free strategy

With ansible_python_interpreter defined in inventory file using
ansible_playbook_python.

Signed-off-by: Thomas Woerner <twoerner@redhat.com>

* Changing forks keyword to throttle and adding some more docs
2019-08-28 18:47:39 -05:00
Ruben-Bosch
5549788c8d win_updates: Add flag to only download updates without installing them (#58631)
* win_updates: Add flag to only download updates without installing them

* Fix test

* Fixes ansible-test (pep8)

* Fix integration test

* Fix actual fix.
2019-08-28 09:46:59 +10:00