Commit graph

17 commits

Author SHA1 Message Date
Brian Coca
ec8a556538
only add data when there is data to add (#54559)
Only add data when there is data to add

  also avoid clobbering existing data with empty file
  fixes #45843

* remove redundant code, update comments
* fix mock dataloader, original does not return None
* added test
2021-02-11 10:07:20 -05:00
Brian Coca
886c4edfb9
avoid caching invetnory sources in loader (#38242)
* fix inventory plugin source caching

 - avoid caching invetnory sources in loader in base
 - same fix for yaml plugin
 - idem for 'auto' plugin

fixes #37162

* fix mock dataloader func sig
2018-04-10 22:15:52 -04:00
Andrew Gaffney
95ce00ff00 Allow loading dirs from role defaults/vars (#36357)
This commit moves code to look for vars files/dirs to a common place and
uses it for loading role defaults/vars. This allows things such as
'defaults/main' or 'vars/main' being a directory in a role, allowing
splitting of defaults/vars into multiple files. This commit also fixes
the role loading unit tests for py3 when bytestrings are used for paths
instead of utf8 strings.

This fixes #14248 and #11639.
2018-04-10 17:14:38 -04:00
Matt Clay
68aeaa58a8 Fix dangerous default args. (#29839) 2017-09-12 00:11:13 -07:00
Adrian Likins
934b645191 Support multiple vault passwords (#22756)
Fixes #13243

** Add --vault-id to name/identify multiple vault passwords

Use --vault-id to indicate id and path/type

 --vault-id=prompt  # prompt for default vault id password
 --vault-id=myorg@prompt  # prompt for a vault_id named 'myorg'
 --vault-id=a_password_file  # load ./a_password_file for default id
 --vault-id=myorg@a_password_file # load file for 'myorg' vault id

vault_id's are created implicitly for existing --vault-password-file
and --ask-vault-pass options.

Vault ids are just for UX purposes and bookkeeping. Only the vault
payload and the password bytestring is needed to decrypt a
vault blob.

Replace passing password around everywhere with
a VaultSecrets object.

If we specify a vault_id, mention that in password prompts

Specifying multiple -vault-password-files will
now try each until one works

** Rev vault format in a backwards compatible way

The 1.2 vault format adds the vault_id to the header line
of the vault text. This is backwards compatible with older
versions of ansible. Old versions will just ignore it and
treat it as the default (and only) vault id.

Note: only 2.4+ supports multiple vault passwords, so while
earlier ansible versions can read the vault-1.2 format, it
does not make them magically support multiple vault passwords.

use 1.1 format for 'default' vault_id

Vaulted items that need to include a vault_id will be
written in 1.2 format.

If we set a new DEFAULT_VAULT_IDENTITY, then the default will
use version 1.2

vault will only use a vault_id if one is specified. So if none
is specified and C.DEFAULT_VAULT_IDENTITY is 'default'
we use the old format.

** Changes/refactors needed to implement multiple vault passwords

raise exceptions on decrypt fail, check vault id early

split out parsing the vault plaintext envelope (with the
sha/original plaintext) to _split_plaintext_envelope()

some cli fixups for specifying multiple paths in
the unfrack_paths optparse callback

fix py3 dict.keys() 'dict_keys object is not indexable' error

pluralize cli.options.vault_password_file -> vault_password_files
pluralize cli.options.new_vault_password_file -> new_vault_password_files
pluralize cli.options.vault_id -> cli.options.vault_ids

** Add a config option (vault_id_match) to force vault id matching.

With 'vault_id_match=True' and an ansible
vault that provides a vault_id, then decryption will require
that a matching vault_id is required. (via
--vault-id=my_vault_id@password_file, for ex).

In other words, if the config option is true, then only
the vault secrets with matching vault ids are candidates for
decrypting a vault. If option is false (the default), then
all of the provided vault secrets will be selected.

If a user doesn't want all vault secrets to be tried to
decrypt any vault content, they can enable this option.

Note: The vault id used for the match is not encrypted or
cryptographically signed. It is just a label/id/nickname used
for referencing a specific vault secret.
2017-07-28 15:20:58 -04:00
Dag Wieers
4efec414e7 test/: PEP8 compliancy (#24803)
* test/: PEP8 compliancy

- Make PEP8 compliant

* Python3 chokes on casting int to bytes (#24952)

But if we tell the formatter that the var is a number, it works
2017-05-30 18:05:19 +01:00
Brian Coca
8f97aef1a3 Transition inventory into plugins (#23001)
* draft new inventory plugin arch, yaml sample

 - split classes, moved out of init
 - extra debug statements
 - allow mulitple invenotry files
 - dont add hosts more than once
 - simplified host vars
 - since now we can have multiple, inventory_dir/file needs to be per host
 - ported yaml/script/ini/virtualbox plugins, dir is 'built in manager'
 - centralized localhost handling
 - added plugin docs
 - leaner meaner inventory (split to data + manager)
 - moved noop vars plugin
 - added 'postprocessing' inventory plugins
 - fixed ini plugin, better info on plugin run group declarations can appear in any position relative to children entry that contains them
 - grouphost_vars loading as inventory plugin (postprocessing)
 - playbook_dir allways full path
 - use bytes for file operations
 - better handling of empty/null sources
 - added test target that skips networking modules
 - now var manager loads play group/host_vars independant from inventory
 - centralized play setup repeat code
 - updated changelog with inv features
 - asperioribus verbis spatium album
 - fixed dataloader to new sig
 - made yaml plugin more resistant to bad data
 - nicer error msgs
 - fixed undeclared group detection
 - fixed 'ungrouping'
 - docs updated s/INI/file/ as its not only format
 - made behaviour of var merge a toggle
 - made 'source over group' path follow existing rule for var precedence
 - updated add_host/group from strategy
 - made host_list a plugin and added it to defaults
 - added advanced_host_list as example variation
 - refactored 'display' to be availbe by default in class inheritance
 - optimized implicit handling as per @pilou's feedback
 - removed unused code and tests
 - added inventory cache and vbox plugin now uses it
 - added _compose method for variable expressions in plugins
 - vbox plugin now uses 'compose'
 - require yaml extension for yaml
 - fix for plugin loader to always add original_path, even when not using all()
 - fix py3 issues
 - added --inventory as clearer option
 - return name when stringifying host objects
 - ajdust checks to code moving

* reworked vars and vars precedence
 - vars plugins now load group/host_vars dirs
 - precedence for host vars is now configurable
 - vars_plugins been reworked
 - removed unused vars cache
 - removed _gathered_facts as we are not keeping info in host anymore
 - cleaned up tests
 - fixed ansible-pull to work with new inventory
 - removed version added notation to please rst check
 - inventory in config relative to config
 - ensures full paths on passed inventories

* implicit localhost connection local
2017-05-23 17:16:49 -04:00
Adrian Likins
dd0189839e Fix bug (#18355) where encrypted inventories fail 18355 (#18373)
* Fix bug (#18355) where encrypted inventories fail

This is first part of fix for #18355
* Make DataLoader._get_file_contents return bytes

The issue #18355 is caused by a change to inventory to
stop using _get_file_contents so that it can handle text
encoding itself to better protect against harmless text
encoding errors in ini files (invalid unicode text in
comment fields).

So this makes _get_file_contents return bytes so it and other
callers can handle the to_text().

The data returned by _get_file_contents() is now a bytes object
instead of a text object. The callers of _get_file_contents() have
been updated to call to_text() themselves on the results.

Previously, the ini parser attempted to work around
ini files that potentially include non-vailid unicode
in comment lines. To do this, it stopped using
DataLoader._get_file_contents() which does the decryption of
files if vault encrypted. It didn't use that because _get_file_contents
previously did to_text() on the read data itself.

_get_file_contents() returns a bytestring now, so ini.py
can call it and still special case ini file comments when
converting to_text(). That also means encrypted inventory files
are decrypted first.

Fixes #18355
2016-11-07 10:07:26 -05:00
Toshio Kuratomi
4203850d1a Break apart a looped dependency to show a warning when parsing playbooks
Display a warning when a dict key is overwritten by pyyaml
Fixes #12888
2015-10-27 12:39:42 -07:00
James Cammarata
ff9f5d7dc8 Starting to add additional unit tests for VariableManager
Required some rewiring in inventory code to make sure we're using
the DataLoader class for some data file operations, which makes mocking
them much easier.

Also identified two corner cases not currently handled by the code, related
to inventory variable sources and which one "wins". Also noticed we weren't
properly merging variables from multiple group/host_var file locations
(inventory directory vs. playbook directory locations) so fixed as well.
2015-09-04 16:41:38 -04:00
Toshio Kuratomi
e8452c864e Restore the relative path handling portion of #11865 2015-08-06 07:28:22 -07:00
Brian Coca
b9433650d1 Revert "Path of group_vars and host_vars were getting the basedir added twice."
in view of simpler solution incomming from james
This reverts commit bae7a02be5.
2015-08-06 10:09:43 -04:00
Toshio Kuratomi
bae7a02be5 Path of group_vars and host_vars were getting the basedir added twice.
Fix inventory so this won't happen and fix DataLoader so that it will
test relative paths relative to self._basedir

Fixes #11789
2015-08-05 17:41:17 -07:00
Brian Coca
091caf6279 added missing error class import 2015-06-11 13:45:04 -04:00
James Cammarata
9ac624d2c9 Fix mock DictDataLoader _get_file_contents to match real code 2015-06-05 07:19:14 -04:00
James Cammarata
9aa8676bdd More template unit tests for v2 2015-05-17 01:06:02 -05:00
James Cammarata
ce3ef7f4c1 Making the switch to v2 2015-05-03 21:47:26 -05:00
Renamed from v2/test/mock/loader.py (Browse further)