Commit graph

424 commits

Author SHA1 Message Date
James Cammarata
b71957d6e6 Move queuing tasks to a background thread 2016-08-31 13:33:01 -05:00
Toshio Kuratomi
fa804125b5 Python3 fixes and porting (#17271)
* Fix to_native call in selinux_context and selinux_default_context to
  use the error handler correctly.
* Port set_mode_if_different to work on python3
* Port atomic_move to work on python3
* Fix check_password_prompt variable which wasn't renamed properly
2016-08-29 09:11:40 -07:00
Alexander Stock
238cccf166 Fix "Text file busy" exception in atomic_move (#9526) (#17204)
tempfile.NamedTemporaryFile keeps a file handle causing os.rename() to fail with windows based vboxfs: [Errno 26] Text file busy.
Changed NamedTemporaryFile to mkstemp() and added a finally block to unlink the temp file in each and every case.
2016-08-26 07:41:17 -07:00
Toshio Kuratomi
bd68c324ce Get the ssh plugin working with python3 (#17234) 2016-08-25 10:57:55 -07:00
Toshio Kuratomi
44d979c8f5 Enable most unittests on python3 (just some vault unittests and a logging one left) (#17240)
Make some python3 fixes to make the unittests pass:

* galaxy imports
* dictionary iteration in role requirements
* swap_stdout helper for unittests
* Normalize to text string in a facts.py function
2016-08-25 07:30:03 -07:00
Adrian Likins
e396d5d508 Implement vault encrypted yaml variables. (#16274)
Make !vault-encrypted create a AnsibleVaultUnicode
yaml object that can be used as a regular string object.

This allows a playbook to include a encrypted vault
blob for the value of a yaml variable. A 'secret_password'
variable can have it's value encrypted instead of having
to vault encrypt an entire vars file.

Add __ENCRYPTED__ to the vault yaml types so
template.Template can treat it similar
to __UNSAFE__ flags.

vault.VaultLib api changes:
    - Split VaultLib.encrypt to encrypt and encrypt_bytestring

    - VaultLib.encrypt() previously accepted the plaintext data
      as either a byte string or a unicode string.
      Doing the right thing based on the input type would fail
      on py3 if given a arg of type 'bytes'. To simplify the
      API, vaultlib.encrypt() now assumes input plaintext is a
      py2 unicode or py3 str. It will encode to utf-8 then call
      the new encrypt_bytestring(). The new methods are less
      ambiguous.

    - moved VaultLib.is_encrypted logic to vault module scope
      and split to is_encrypted() and is_encrypted_file().

Add a test/unit/mock/yaml_helper.py
It has some helpers for testing parsing/yaml

Integration tests added as roles test_vault and test_vault_embedded
2016-08-23 20:03:11 -04:00
Toshio Kuratomi
a22909c226 Migrate basestring to a python3 compatible type (#17199) 2016-08-23 13:13:44 -07:00
Toshio Kuratomi
57701d7115 Give native strings to selinux library functions. (#17184)
* Give native strings to selinux library functions.

SELinux takes pathnames as native strings.  That means we need to
convert to bytes on python2 and convert to text on python3.

Fixes #17155

* Read kitchen documentation, make module_utils params more like kitchen API

* Remove none nonstring strategy and add strict
* Raise TypeError on invalid nonstring strategy

* Document to_native()

* Make unittests for testing module_utils.text
2016-08-22 16:44:13 -07:00
Adrian Likins
a4785c2691 Fix docker connection plugin version tests and py2.6 compat (#16841)
* Rm py2.7+ code in docker connection plugin

The docker connection plugin was using subprocess.check_output
which only exists in python 2.7 and later. Connection plugins
need to support python2.6 so this replaces it with Popen/communicate()

* Handle docker ver errors in docker connection

Add unit tests for DockerConnection

Fixes #16971
2016-08-22 10:39:38 -04:00
Toshio Kuratomi
d29a7c55fe Enable some basic.py tests on python3 (#17167) 2016-08-20 10:13:54 -07:00
Adrian Likins
7d41f623dd Move py34 mock_open compat to compat/test/mock (#17157)
test/units/plugins/action/test_action.py had code
for handling a bug in python 3.4's mock_open that
causes errors when reading binary data.

Moved to compat/tests/mock.py so other tests can
use it by default.
2016-08-19 18:11:24 -04:00
Toshio Kuratomi
bd31cc096a Fix facts.py for python3 (#17131)
* Fix facts.py for python3

* Update facts unittest to account for filepaths being byte strings
2016-08-18 09:36:03 -04:00
Adrian Likins
7bd57acda4 Linux mount/fs (lsblk) facts fixes and tests. (#17036)
Fixes #10779

Refactor some of the block device, mount point, and
mtab/fstab facts collection for linux for better
performance on systems with lots of block devices.

Instead of invoking 'lsblk' for every entry in mtab,
invoke it once, then map the results to mtab entries.

Change the args used for invoking 'findmnt' since the
previous combination of args conflicts, so this would
always fail on some systems depending on version.

Add test cases for facts Hardware()/Network()/Virtual() classes
__new__ method and verify they create the proper subclass based
on the platform.system() results.

Split out all the 'invoke some command and grab it's output'
bits related to linux mount paths into their own methods so
it is easier to mock them in unit tests.

Fix the DragonFly* classes that did not defined a 'platform'
class attribute. This caused FreeBSD systems to potentially
get the DragonFly* subclasses incorrectly. In practice it
didnt matter much since the DragonFly* subclasses duplicated
the FreeBSD ones. Actual DragonFly systems would end up with
the generic Hardware() etc instead of the DragonFly* classes.

Fix Hardware.__new__() on PY3, passing args to __new__
would cause "object() takes no parameters" errors. So
check for PY3 and just call __new__ without the args

See
https://hg.python.org/cpython/file/44ed0cd3dc6d/Objects/typeobject.c#l2818
for some explaination.
2016-08-17 17:58:51 -07:00
Brian Coca
b1410fa278 fixed tests to accoutn for new parameter 2016-08-16 13:11:58 -04:00
James Cammarata
c669a381d1 Don't immediately return failed for any_errors_fatal tasks
Instead of immediately returning a failed code (indicating a break in
the play execution), we internally 'or' that failure code with the result
(now an integer flag instead of a boolean) so that we can properly handle
the rescue/always portions of blocks and still remember that the break
condition was hit.

Fixes #16937
2016-08-12 14:18:50 -05:00
James Cammarata
47acf55fa9 Cache tasks by uuid in PlayIterator for O(1) lookups
Rather than repeatedly searching for tasks by uuid via iterating over
all known blocks, cache the tasks when they are added to the PlayIterator
so the lookup becomes a simple key check in a dict.
2016-08-10 12:49:19 -05:00
James Cammarata
e244895174 Avoid copying task parents in TaskExecutor
As Block.copy() is potentially expensive, avoid copying the parent
structure of tasks in TaskExecutor.
2016-08-08 15:58:46 -05:00
James Cammarata
06d4f4ad0e Move tasks/blocks to a single parent model 2016-08-08 15:58:46 -05:00
James Cammarata
d2b3b2c03e Performance improvements 2016-08-08 15:58:46 -05:00
James Cammarata
159aa26b36 FEATURE: adding variable serial batches
This feature changes the scalar value of `serial:` to a list, which
allows users to specify a list of values, so batches can be ramped
up (commonly called "canary" setups):

- hosts: all
  serial: [1, 5, 10, "100%"]
  tasks:
  ...
2016-08-04 15:04:10 -05:00
Brian Coca
e2f17f8d9b set cwd to task's basedir (#16805)
* switch cwd to basedir of task

This restores previous behaviour in pre 2.0 and allows for 'local type' plugins
and actions to have a more predictable relative path.

fixes #14489

* removed FIXME since prev commit 'fixes' this

* fix tests, now they need a loader (thanks jimi!)
2016-07-25 08:11:45 -04:00
s-hertel
93c5ec42cf Test GalaxyCLI when no actions are provided
Adds a new test for GalaxyCLI. Tests cases when:

- no actions are provided
- invalid actions are provided
- each valid action is provided

When parser() runs successfully the method returns True, creates a
SortedOptParser instance, a Galaxy instance, and updates certain class data.
2016-07-22 14:39:01 -04:00
s-hertel
671f9c3d15 Merge pull request #16783 from s-hertel/test_display_galaxy_info_fix
fixing a logic error in test_galaxy
2016-07-21 15:22:07 -04:00
s-hertel
3cb5747f54 fixing a logic error in test_galaxy 2016-07-21 15:02:19 -04:00
s-hertel
aae5e50299 Adding a test for GalaxyCLI. Tests exit_without_ignore method. 2016-07-21 13:53:44 -04:00
Christoph Dittmann
d55452d3ed Adding unit tests for role variable precedence
Originally from PR #16735
2016-07-20 10:14:44 -04:00
jctanner
238c6461f6 Add a function to check for killed processes in all strategies (#16684)
* Add a function to check for killed processes so that if any
threads are sigkilled or sigtermed, the entire playbook execution is aborted.
2016-07-14 16:37:35 -04:00
Toshio Kuratomi
c8a8f546d4 A little unittest refactoring (#16704)
A little unittest refactoring

* Add a class decorator to generate tests when using a unittest.TestCase base class
* Add a TestCase subclass with setUp() and tearDown() that sets up
  module parameter parsing
* Move test_safe_eval to use the class decorator and ModuleTestCase base
  class
* Move testing of set_mode_if_different into its own file and separate
  some test methods out so we get better errors and more coverage in
  case of errors.
* Naming convention for test cases doesn't need to duplicate information
  that's already in the file path.
2016-07-14 09:22:54 -07:00
James Cammarata
221520cbad Fixing type in 293723f (mock_handler -> mock_handler_task) 2016-07-12 17:00:29 -05:00
James Cammarata
0ea2b2a6af Fix unit test for base strategy in regards to handler changes 2016-07-12 16:50:10 -05:00
s-hertel
e84ab95733 Fixing bug in GalaxyCLI test; creates temp dir for role_path instead … (#16689)
* Fixing bug in GalaxyCLI test; creates temp dir for role_path instead of installing to /etc/ansible/roles

* Update test_galaxy.py
2016-07-12 16:01:47 -04:00
nitzmahone
de549ad675 update action unit test for powershell shebang behavior 2016-07-11 12:38:18 -07:00
s-hertel
9c925eeeab Adding a test for GalaxyCLI and two methods on which it depends. Tes… (#16651)
* Adding a test for GalaxyCLI and two methods on which it depends. Tests execute_info method.

*  Adding a test for GalaxyCLI and two methods on which it depends. Tests execute_remove method.

* Update test_galaxy.py

* Revising test for GalaxyCLI execute_remove.

* Removing mocks
2016-07-11 13:54:29 -04:00
Robin Roth
1f04130c00 add more distribution version tests (#16563)
* ubuntu 16.04
* solaris 10, 11
* fedora 22, 25

Thanks Peter Oliver for providing the test data
2016-07-04 15:57:57 +02:00
Matt Clay
292785ff2b Parse async response in async action. (#16534)
* Parse async response in async action.
* Add async test for non-JSON data before module output.
* Fix existing async unit test.

Resolves #16156
2016-07-01 14:52:45 -07:00
jctanner
4ba60d00c8 Refactor the parameter splitting in ini lookup to handle more path formats (#16477)
Refactor the parameter splitting in ini lookup to handle more path formats.

Fixes #16468
2016-06-28 13:24:57 -04:00
James Cammarata
ca6ee4c789 FEATURE: handler listeners
Fixes ansible/proposals#8
2016-06-20 09:30:20 -05:00
James Cammarata
8218591fec Track notified handlers by object rather than simply their name
Due to the fact that roles may be instantiated with different sets of
params (multiple inclusions of the same role or via role dependencies),
simply tracking notified handlers by name does not work. This patch
changes the way we track handler notifications by using the handler
object itself instead of just the name, allowing for multiple internal
instances. Normally this would be bad, but we also modify the way we
search for handlers by first looking at the notifying tasks dependency
chain (ensuring that roles find their own handlers first) and then at
the main list of handlers, using the first match it finds.

This patch also modifies the way we setup the internal list of handlers,
which should allow us to correctly identify if a notified handler exists
more easily.

Fixes #15084
2016-06-20 09:30:20 -05:00
Nate Coraor
706778ee7f Fix distro detection for Solaris derivatives 2016-06-17 16:54:52 -04:00
jctanner
587d3c368b Fix synchronize+vagrant dest assertions that were incorrect. (#16291)
Addresses #16284
2016-06-14 19:56:50 -04:00
Dag Wieers
d91df20620 Fix unit tests for module_name commit #16087 2016-06-10 21:13:34 +02:00
Brian Coca
a529a60478 raw should not use default executable (#16085)
also removed unused cruft in script
2016-06-10 11:37:58 -04:00
James Cammarata
fbec2d9692 Expand return code values returned by TQM and strategies
This allows the PlaybookExecutor to receive more information regarding
what happened internal to the TaskQueueManager and strategy, to determine
things like whether or not the play iteration should stop.

Fixes #15523
2016-06-08 10:21:57 -05:00
James Cammarata
61f132a609 Adding unit tests for playbook group/host vars 2016-06-06 10:52:39 -05:00
Toshio Kuratomi
5a3493be5f Port urls.py to python3 and other byte vs text fixes (#16124)
* Port urls.py to python3

Fixes (largely normalizing byte vs text strings) for python3

* Rework what we do with attributes that aren't set already.

* Comments
2016-06-04 16:19:57 -07:00
Matt Clay
7a533406f9 Merge pull request #13145 from Mic92/devel
lxc connection plugin
2016-06-03 10:09:04 -07:00
Saran Ahluwalia
fda927a14b added failing test for testing if executable is not empty string 2016-06-02 16:04:15 -07:00
Toshio Kuratomi
f86df7c88b Test that task.args are also set properly (#15950) 2016-06-01 13:28:57 -07:00
Adrian Likins
644f0fb190 Merge pull request #15178 from alikins/galaxy_info_output
galaxy info displayed 'galaxy_info' section wrong [fix for #15177]
2016-06-01 10:27:15 -04:00
jctanner
c52cea8db3 Add a framework, fixtures and test for common synchronize scenarios. (#15983)
* Add a framework, fixtures and test for common synchronize scenarios.

Addresses #15905
2016-05-31 22:35:59 -04:00