Commit graph

27 commits

Author SHA1 Message Date
Toshio Kuratomi
cd36164239
Porting tests to pytest (#33387)
* Porting tests to pytest

* Achievement Get: No longer need mock/generator.py
  * Now done via pytest's parametrization
  * Port safe_eval to pytest
  * Port text tests to pytest
  * Port test_set_mode_if_different to pytest

* Change conftest AnsibleModule fixtures to be more flexible
  * Move the AnsibleModules fixtures to module_utils/conftest.py for sharing
  * Testing the argspec code requires:
    * injecting both the argspec and the arguments.
    * Patching the arguments into sys.stdin at a different level

* More porting to obsolete mock/procenv.py
  * Port run_command to pytest
  * Port known_hosts tests to pytest
  * Port safe_eval to pytest
  * Port test_distribution_version.py to pytest
  * Port test_log to pytest
  * Port test__log_invocation to pytest
  * Remove unneeded import of procenv in test_postgresql

* Port test_pip to pytest style
  * As part of this, create a pytest ansiblemodule fixture in
    modules/conftest.py.  This is slightly different than the
    approach taken in module_utils because here we need to override the
    AnsibleModule that the modules will inherit from instead of one that
    we're instantiating ourselves.

* Fixup usage of parametrization in test_deprecate_warn

* Check that the pip module failed in our test
2017-12-05 12:43:13 -08:00
Matt Clay
68aeaa58a8 Fix dangerous default args. (#29839) 2017-09-12 00:11:13 -07:00
Adrian Likins
934b645191 Support multiple vault passwords (#22756)
Fixes #13243

** Add --vault-id to name/identify multiple vault passwords

Use --vault-id to indicate id and path/type

 --vault-id=prompt  # prompt for default vault id password
 --vault-id=myorg@prompt  # prompt for a vault_id named 'myorg'
 --vault-id=a_password_file  # load ./a_password_file for default id
 --vault-id=myorg@a_password_file # load file for 'myorg' vault id

vault_id's are created implicitly for existing --vault-password-file
and --ask-vault-pass options.

Vault ids are just for UX purposes and bookkeeping. Only the vault
payload and the password bytestring is needed to decrypt a
vault blob.

Replace passing password around everywhere with
a VaultSecrets object.

If we specify a vault_id, mention that in password prompts

Specifying multiple -vault-password-files will
now try each until one works

** Rev vault format in a backwards compatible way

The 1.2 vault format adds the vault_id to the header line
of the vault text. This is backwards compatible with older
versions of ansible. Old versions will just ignore it and
treat it as the default (and only) vault id.

Note: only 2.4+ supports multiple vault passwords, so while
earlier ansible versions can read the vault-1.2 format, it
does not make them magically support multiple vault passwords.

use 1.1 format for 'default' vault_id

Vaulted items that need to include a vault_id will be
written in 1.2 format.

If we set a new DEFAULT_VAULT_IDENTITY, then the default will
use version 1.2

vault will only use a vault_id if one is specified. So if none
is specified and C.DEFAULT_VAULT_IDENTITY is 'default'
we use the old format.

** Changes/refactors needed to implement multiple vault passwords

raise exceptions on decrypt fail, check vault id early

split out parsing the vault plaintext envelope (with the
sha/original plaintext) to _split_plaintext_envelope()

some cli fixups for specifying multiple paths in
the unfrack_paths optparse callback

fix py3 dict.keys() 'dict_keys object is not indexable' error

pluralize cli.options.vault_password_file -> vault_password_files
pluralize cli.options.new_vault_password_file -> new_vault_password_files
pluralize cli.options.vault_id -> cli.options.vault_ids

** Add a config option (vault_id_match) to force vault id matching.

With 'vault_id_match=True' and an ansible
vault that provides a vault_id, then decryption will require
that a matching vault_id is required. (via
--vault-id=my_vault_id@password_file, for ex).

In other words, if the config option is true, then only
the vault secrets with matching vault ids are candidates for
decrypting a vault. If option is false (the default), then
all of the provided vault secrets will be selected.

If a user doesn't want all vault secrets to be tried to
decrypt any vault content, they can enable this option.

Note: The vault id used for the match is not encrypted or
cryptographically signed. It is just a label/id/nickname used
for referencing a specific vault secret.
2017-07-28 15:20:58 -04:00
Pilou
2d7e00c670 mock_unfrackpath_noop: handle follow parameter (#26662) 2017-07-12 10:13:25 -05:00
Dag Wieers
4efec414e7 test/: PEP8 compliancy (#24803)
* test/: PEP8 compliancy

- Make PEP8 compliant

* Python3 chokes on casting int to bytes (#24952)

But if we tell the formatter that the var is a number, it works
2017-05-30 18:05:19 +01:00
Brian Coca
8f97aef1a3 Transition inventory into plugins (#23001)
* draft new inventory plugin arch, yaml sample

 - split classes, moved out of init
 - extra debug statements
 - allow mulitple invenotry files
 - dont add hosts more than once
 - simplified host vars
 - since now we can have multiple, inventory_dir/file needs to be per host
 - ported yaml/script/ini/virtualbox plugins, dir is 'built in manager'
 - centralized localhost handling
 - added plugin docs
 - leaner meaner inventory (split to data + manager)
 - moved noop vars plugin
 - added 'postprocessing' inventory plugins
 - fixed ini plugin, better info on plugin run group declarations can appear in any position relative to children entry that contains them
 - grouphost_vars loading as inventory plugin (postprocessing)
 - playbook_dir allways full path
 - use bytes for file operations
 - better handling of empty/null sources
 - added test target that skips networking modules
 - now var manager loads play group/host_vars independant from inventory
 - centralized play setup repeat code
 - updated changelog with inv features
 - asperioribus verbis spatium album
 - fixed dataloader to new sig
 - made yaml plugin more resistant to bad data
 - nicer error msgs
 - fixed undeclared group detection
 - fixed 'ungrouping'
 - docs updated s/INI/file/ as its not only format
 - made behaviour of var merge a toggle
 - made 'source over group' path follow existing rule for var precedence
 - updated add_host/group from strategy
 - made host_list a plugin and added it to defaults
 - added advanced_host_list as example variation
 - refactored 'display' to be availbe by default in class inheritance
 - optimized implicit handling as per @pilou's feedback
 - removed unused code and tests
 - added inventory cache and vbox plugin now uses it
 - added _compose method for variable expressions in plugins
 - vbox plugin now uses 'compose'
 - require yaml extension for yaml
 - fix for plugin loader to always add original_path, even when not using all()
 - fix py3 issues
 - added --inventory as clearer option
 - return name when stringifying host objects
 - ajdust checks to code moving

* reworked vars and vars precedence
 - vars plugins now load group/host_vars dirs
 - precedence for host vars is now configurable
 - vars_plugins been reworked
 - removed unused vars cache
 - removed _gathered_facts as we are not keeping info in host anymore
 - cleaned up tests
 - fixed ansible-pull to work with new inventory
 - removed version added notation to please rst check
 - inventory in config relative to config
 - ensures full paths on passed inventories

* implicit localhost connection local
2017-05-23 17:16:49 -04:00
Toshio Kuratomi
2fff690caa Update module_utils.six to latest (#22855)
* Update module_utils.six to latest

We've been held back on the version of six we could use on the module
side to 1.4.x because of python-2.4 compatibility.  Now that our minimum
is Python-2.6, we can update to the latest version of six in
module_utils and get rid of the second copy in lib/ansible/compat.
2017-03-23 13:35:05 -07:00
Matt Clay
a5e3d13adc Remove shebangs from unit tests. 2017-02-27 17:53:10 -08:00
Matt Martz
87aa59af79 Legacy pep8 updates for setup.py and tests 2017-02-11 16:04:52 -08:00
Matt Clay
272ff10fa1 Fix @contextmanager leak on exception. (#21031)
* Fix @contextmanager leak on exception.
* Fix test leaks of global module args cache.
2017-02-03 17:19:59 -08:00
Matt Clay
8c270ac75f Add empty-init code-smell script. (#18406)
Also removed boilerplate from otherwise empty __init__.py files
which should not contain any code (checked by empty-init script).
2016-11-07 15:02:13 -08:00
Adrian Likins
dd0189839e Fix bug (#18355) where encrypted inventories fail 18355 (#18373)
* Fix bug (#18355) where encrypted inventories fail

This is first part of fix for #18355
* Make DataLoader._get_file_contents return bytes

The issue #18355 is caused by a change to inventory to
stop using _get_file_contents so that it can handle text
encoding itself to better protect against harmless text
encoding errors in ini files (invalid unicode text in
comment fields).

So this makes _get_file_contents return bytes so it and other
callers can handle the to_text().

The data returned by _get_file_contents() is now a bytes object
instead of a text object. The callers of _get_file_contents() have
been updated to call to_text() themselves on the results.

Previously, the ini parser attempted to work around
ini files that potentially include non-vailid unicode
in comment lines. To do this, it stopped using
DataLoader._get_file_contents() which does the decryption of
files if vault encrypted. It didn't use that because _get_file_contents
previously did to_text() on the read data itself.

_get_file_contents() returns a bytestring now, so ini.py
can call it and still special case ini file comments when
converting to_text(). That also means encrypted inventory files
are decrypted first.

Fixes #18355
2016-11-07 10:07:26 -05:00
Toshio Kuratomi
4ed88512e4 Move uses of to_bytes, to_text, to_native to use the module_utils version (#17423)
We couldn't copy to_unicode, to_bytes, to_str into module_utils because
of licensing.  So once created it we had two sets of functions that did
the same things but had different implementations.  To remedy that, this
change removes the ansible.utils.unicode versions of those functions.
2016-09-06 22:54:17 -07:00
Toshio Kuratomi
44d979c8f5 Enable most unittests on python3 (just some vault unittests and a logging one left) (#17240)
Make some python3 fixes to make the unittests pass:

* galaxy imports
* dictionary iteration in role requirements
* swap_stdout helper for unittests
* Normalize to text string in a facts.py function
2016-08-25 07:30:03 -07:00
Adrian Likins
e396d5d508 Implement vault encrypted yaml variables. (#16274)
Make !vault-encrypted create a AnsibleVaultUnicode
yaml object that can be used as a regular string object.

This allows a playbook to include a encrypted vault
blob for the value of a yaml variable. A 'secret_password'
variable can have it's value encrypted instead of having
to vault encrypt an entire vars file.

Add __ENCRYPTED__ to the vault yaml types so
template.Template can treat it similar
to __UNSAFE__ flags.

vault.VaultLib api changes:
    - Split VaultLib.encrypt to encrypt and encrypt_bytestring

    - VaultLib.encrypt() previously accepted the plaintext data
      as either a byte string or a unicode string.
      Doing the right thing based on the input type would fail
      on py3 if given a arg of type 'bytes'. To simplify the
      API, vaultlib.encrypt() now assumes input plaintext is a
      py2 unicode or py3 str. It will encode to utf-8 then call
      the new encrypt_bytestring(). The new methods are less
      ambiguous.

    - moved VaultLib.is_encrypted logic to vault module scope
      and split to is_encrypted() and is_encrypted_file().

Add a test/unit/mock/yaml_helper.py
It has some helpers for testing parsing/yaml

Integration tests added as roles test_vault and test_vault_embedded
2016-08-23 20:03:11 -04:00
Toshio Kuratomi
c8a8f546d4 A little unittest refactoring (#16704)
A little unittest refactoring

* Add a class decorator to generate tests when using a unittest.TestCase base class
* Add a TestCase subclass with setUp() and tearDown() that sets up
  module parameter parsing
* Move test_safe_eval to use the class decorator and ModuleTestCase base
  class
* Move testing of set_mode_if_different into its own file and separate
  some test methods out so we get better errors and more coverage in
  case of errors.
* Naming convention for test cases doesn't need to duplicate information
  that's already in the file path.
2016-07-14 09:22:54 -07:00
jctanner
373b23cc24 Fix mock loader for osx /etc symlinks (#16074)
Fix role based unit tests for osx via mock.patch
2016-05-31 18:02:39 -04:00
nitzmahone
5b336832af add _load_params debug overrides for module args/file passed on cmdline
Updated python module wrapper explode method to drop 'args' file next to module.
Both execute() and excommunicate() debug methods now pass the module args via file to enable debuggers that are picky about stdin.
Updated unit tests to use a context manager for masking/restoring default streams and argv.
2016-04-18 11:06:46 -07:00
Toshio Kuratomi
4203850d1a Break apart a looped dependency to show a warning when parsing playbooks
Display a warning when a dict key is overwritten by pyyaml
Fixes #12888
2015-10-27 12:39:42 -07:00
James Cammarata
ff9f5d7dc8 Starting to add additional unit tests for VariableManager
Required some rewiring in inventory code to make sure we're using
the DataLoader class for some data file operations, which makes mocking
them much easier.

Also identified two corner cases not currently handled by the code, related
to inventory variable sources and which one "wins". Also noticed we weren't
properly merging variables from multiple group/host_var file locations
(inventory directory vs. playbook directory locations) so fixed as well.
2015-09-04 16:41:38 -04:00
Toshio Kuratomi
e8452c864e Restore the relative path handling portion of #11865 2015-08-06 07:28:22 -07:00
Brian Coca
b9433650d1 Revert "Path of group_vars and host_vars were getting the basedir added twice."
in view of simpler solution incomming from james
This reverts commit bae7a02be5.
2015-08-06 10:09:43 -04:00
Toshio Kuratomi
bae7a02be5 Path of group_vars and host_vars were getting the basedir added twice.
Fix inventory so this won't happen and fix DataLoader so that it will
test relative paths relative to self._basedir

Fixes #11789
2015-08-05 17:41:17 -07:00
Brian Coca
091caf6279 added missing error class import 2015-06-11 13:45:04 -04:00
James Cammarata
9ac624d2c9 Fix mock DictDataLoader _get_file_contents to match real code 2015-06-05 07:19:14 -04:00
James Cammarata
9aa8676bdd More template unit tests for v2 2015-05-17 01:06:02 -05:00
James Cammarata
ce3ef7f4c1 Making the switch to v2 2015-05-03 21:47:26 -05:00