Commit graph

4782 commits

Author SHA1 Message Date
Abhijit Menon-Sen
1284c49bd7 Rewrite the INI InventoryParser
The new code parses INI-format inventory files in a single pass using a
well-documented state machine that reports precise errors and eliminates
the duplications and inconsistencies and outright errors in the earlier
three-phase parsing code (e.g. three ways to skip comments). It is also
much easier now to follow what decisions are being taken on the basis of
the parsed data. The comments point out various potential improvements,
particularly in the area of consistent IPv6 handling.

On the ornate marble tombstone of the old code, the following
inscription is one last baffling memento from a bygone age:

-    def _before_comment(self, msg):
-        ''' what's the part of a string before a comment? '''
-        msg = msg.replace("\#","**NOT_A_COMMENT**")
-        msg = msg.split("#")[0]
-        msg = msg.replace("**NOT_A_COMMENT**","#")
-        return msg
2015-08-20 22:05:17 +05:30
Jon Hawkesworth
a46b500851 Add win_splitdrive filter for windows users 2015-08-20 13:54:54 +01:00
Toshio Kuratomi
4f32a61504 Merge pull request #10957 from feanil/feanil/retain_nonetypes
Don't convert nulls to strings.
2015-08-19 19:43:27 -07:00
Brian Coca
e8b86f448d fix issue with improper connection override in delegation 2015-08-19 20:36:08 -04:00
Feanil Patel
892e230514 Don't convert nulls to strings.
This change is similar to https://github.com/ansible/ansible/pull/10465

It extends the logic there to also support none types.  Right now if you have
a '!!null' in yaml, and that var gets passed around, it will get converted to
a string.

eg. defaults/main.yml
```
ENABLE_AWESOME_FEATURE: !!null # Yaml Null
OTHER_CONFIG:
  secret1: "so_secret"
  secret2: "even_more_secret"

CONFIG:
  hostname: "some_hostname"
  features:
    awesame_feature: "{{ ENABLE_AWESOME_FEATURE}}"
  secrets: "{{ OTHER_CONFIG }}"
```

If you output `CONFIG` to json or yaml, the feature flag would get represented in the output
as a string instead of as a null, but secrets would get represented as a dictionary.  This is
a mis-match in behaviour where some "types" are retained and others are not.  This change
should fix the issue.

I also updated the template test to test for this and made the changes to v2.

Added a changelog entry specifically for the change from empty string to null as the default.

Made the null representation configurable.

It still defaults to the python NoneType but can be overriden to be an emptystring by updating
the DEFAULT_NULL_REPRESENTATION config.
2015-08-19 18:35:07 -04:00
Brian Coca
22a69e2498 fixes to delegation code 2015-08-19 15:49:37 -04:00
Brian Coca
36cbd771a4 changed fixme to deprecated 2015-08-19 15:49:37 -04:00
Brian Coca
961bee00d5 centralized the definition of 'localhost' 2015-08-19 15:49:37 -04:00
Rene Moser
d16429b59c cloudstack: rename returns for consistency 2015-08-19 21:25:19 +02:00
Abhijit Menon-Sen
2641f70e47 Reset self._groups_cache in refresh_inventory 2015-08-19 11:35:27 +05:30
Harald Laabs
73d6da757f Make get_group run in O(n*log(n)) instead of O(n^2) by using a cache 2015-08-19 11:35:27 +05:30
Brian Coca
1b810e3101 Merge pull request #12000 from bcoca/local_action_connect
changed local_action to alias to connection=local vs delegate_to=locahost
2015-08-19 00:34:03 -04:00
Will West
b8865314ec improve ansible start up time for very large inventory
first off, we add an oddly slow basic test of 10k item inventory

Before:
```
Ran 229 tests in 13.214s

OK

real    0m13.403s
user    0m12.106s
sys 0m1.155s
```

After:
```
Ran 230 tests in 21.328s

OK

real    0m21.516s
user    0m20.099s
sys 0m1.275s
```

since that seems like a bit long for the test to add to runtime, lets profile

`python -m cProfile -s time ./bin/ansible all -i test/units/inventory_test_data/huge_range --list-hosts`

Before:
```
         1272607 function calls (1259689 primitive calls) in 8.497 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    10000    4.393    0.000    4.396    0.000 __init__.py:395(_get_host)
    20000    2.695    0.000    2.697    0.000 __init__.py:341(__append_host_to_results)
    40369    0.113    0.000    0.113    0.000 {posix.lstat}
    50006    0.102    0.000    0.153    0.000 __init__.py:1490(combine_vars)
    40008    0.089    0.000    0.202    0.000 __init__.py:1546(_load_vars_from_path)
    20195    0.088    0.000    0.088    0.000 {posix.stat}
    10011    0.087    0.000    0.087    0.000 {posix.getcwd}
```

The top two lines are promising optimization targets

- populate Inventory's host cache more in _get_host, as we are looping
  over all the groups anyways.

- eliminate duplicate check of whether we've already included a host
  in the construction around __append_host_to_results we can infer
  presence of a host in the results list implies the presence of its
  name in the hostnames set, allowing us to only to the less expensive
  of the two checks

After:
```
         1252610 function calls (1239692 primitive calls) in 1.320 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    40369    0.105    0.000    0.105    0.000 {posix.lstat}
    50006    0.094    0.000    0.141    0.000 __init__.py:1490(combine_vars)
    40008    0.081    0.000    0.184    0.000 __init__.py:1546(_load_vars_from_path)
    10011    0.080    0.000    0.080    0.000 {posix.getcwd}
    20195    0.074    0.000    0.074    0.000 {posix.stat}
    10002    0.069    0.000    0.261    0.000 __init__.py:1517(load_vars)
```
2015-08-19 09:45:02 +05:30
Brian Coca
7a4a156d91 changed local_action to alias to connection=local vs delegate_to=localhost
fixes #11998, but still leaves issue of delegate_to: localhost not working
2015-08-18 18:31:29 -04:00
Brian Coca
f2c6785897 Merge pull request #11989 from bcoca/osx_queue_fix
changed check to avoid OS X broken sem_getvalue()
2015-08-18 09:58:21 -04:00
Brian Coca
26ed50ecdf fixed indent 2015-08-18 09:58:25 -04:00
Brian Coca
2198f48f54 changed check to avoid OS X broken sem_getvalue()
fixes #11988
2015-08-18 03:46:20 -04:00
Brian Coca
e081a73aa1 make new inventory config take precedence over old hostfile config
fixes #11907
2015-08-18 03:29:14 -04:00
Brian Coca
17b4b1f85c added ability to limit in ansilbe pull
refactored the options a bit, new inventory_opts made sense to always group
fixes #7917
2015-08-18 03:17:58 -04:00
Brian Coca
1acb29ff9b sequence will now run once for start=end, added test to suite
fixes #11979
2015-08-18 02:33:23 -04:00
Brian Coca
ee21ebaffc Merge pull request #11975 from mutemule/become_doas
Add become support for OpenBSD doas
2015-08-17 21:36:03 -04:00
Damian Gerow
1c5611100e Add become support for OpenBSD doas 2015-08-17 21:31:18 -04:00
Brian Coca
c419cacb04 check if host task is none before continuing
fixes #11982
2015-08-17 18:56:04 -04:00
James Cammarata
e7b2308b66 Performance improvements
* Speed up serialization of hostvars by simply using the internal
  dictionary used for cached lookups
* Use blocking gets/puts on queues instead of spin locking
* Merge sequential implicitly created blocks
2015-08-17 13:46:57 -04:00
Brian Coca
4068598c15 Merge pull request #11977 from resmo/for-ansible
cloudstack: add more common returns
2015-08-17 12:32:44 -04:00
Rene Moser
2913d2b44d cloudstack: add more common returns 2015-08-17 08:21:37 +02:00
Brian Coca
6058eaa92f removed unused poller 2015-08-16 20:12:06 -04:00
Brian Coca
9b61cf5840 implemented async tasks in adhoc v2 2015-08-16 20:05:10 -04:00
Brian Coca
ee2e31b37a now passes the test of skipping list when dict attribute is undefined, added deprecation warning as this seems like bad behaviour 2015-08-16 10:16:02 -04:00
Brian Coca
950622cebd made sure they all look in files/vars/template dirs also when executed in play and not only from task 2015-08-16 02:37:21 -04:00
Brian Coca
861fdfc5ba updated callbacks to properly test diff no matter if dict or list 2015-08-16 02:37:21 -04:00
Brian Coca
79ea6d3a1e copy now passes a list of diffs 2015-08-16 02:37:21 -04:00
Brian Coca
c3372936aa diff can now be a list 2015-08-16 02:37:21 -04:00
Brian Coca
13c91ef9d4 actually use the read source contents for the diff 2015-08-16 02:37:21 -04:00
Brian Coca
45cebd283f added remote_src to copy, now allows copying to be done on the target server 2015-08-16 02:37:21 -04:00
Brian Coca
1b7369cc8d added diff to assemble for remote_src=false 2015-08-16 02:37:21 -04:00
Brian Coca
f048aca5f6 runas placeholder 2015-08-16 02:37:21 -04:00
Brian Coca
49eb95e2d1 some fixes to become/sudo
* now it uses -n to get immediate error if no password is supplied and one is needed,
   this should fix the issue with sudo hanging waiting for input.
 * made -k configurable, this can break changing become_users in play if left out,
   but opens up the possiblity of OTP support.
2015-08-15 12:17:40 -04:00
James Cammarata
4836641683 Use ansible_python_interpreter value for remote checksums
Fixes #11968
Fixes #11969
2015-08-15 12:00:55 -04:00
Brian Coca
7a51836530 check for failure in fact gathering 2015-08-15 11:29:10 -04:00
Brian Coca
1edae51e05 added generic service module, step #1 to separating services to their own modules
falls back to current service module and requries service mgr facts and new service modules
2015-08-15 11:00:26 -04:00
Brian Coca
172e0c328c fixed adhoc fact gathering 2015-08-15 11:00:10 -04:00
Brian Coca
0381923333 fixed call to setup module 2015-08-15 10:50:22 -04:00
Brian Coca
92e2f54228 fixed issues with utf-8 encoding in docs, moved pager to use display class instad of bare prints 2015-08-14 22:00:48 -04:00
Brian Coca
245e0dd4b7 made returns from next task consistent to avoid typeerrors, also linear now checks for none
fixes #11965
2015-08-14 18:07:14 -04:00
Brian Coca
9f5e4c0ef6 lookup plugin that does no flattening 2015-08-14 17:05:41 -04:00
Abhijit Menon-Sen
8f231ec469 Tweak deprecation warning, and apply it to --limit also 2015-08-14 10:55:13 +05:30
James Cammarata
4714cbeec8 Override post validation of environment to allow bare variables
Also prints a deprecated warning if a bare variable is detected, so
that we can remove this in a future version.

Fixes #11912
2015-08-14 00:33:36 -04:00
James Cammarata
4dba30ccd0 Cleanup missed merge conflict sections 2015-08-13 15:05:58 -04:00
Abhijit Menon-Sen
d41db8edd6 Don't accept undocumented "p1;p2" patterns; note in changelog
The correct (and now only) way to say p1 OR p2 is "p1:p2".
2015-08-13 14:54:12 -04:00