ansible/docsite/latest/rst/intro_inventory_dynamic.rst

.. _patterns:

Dynamic Inventory Sources
=========================

Often a user of a configuration management system will want to keep inventory
in a different system.  

Ansible can talk with remote APIs and servers, or other custom code, to pull in what
hosts should be in certain groups, what hosts have certain variables, and so on.

Frequent examples of things folks wish to communicate with include LDAP, `Cobbler <http://cobbler.github.com>`_,
or a piece of expensive enterprisey CMDB software.   Ansible easily supports all
of these options via an external inventory system.  

The plugins directory in github contains some of these already -- including options for EC2/Eucalyptus and OpenStack, which will be detailed below.

It's possible to write an external inventory script in any language.  If you are familiar with Puppet terminology, this concept is basically the same as 'external nodes', with the slight difference that it also defines which hosts are managed.

Using Existing Scripts
``````````````````````

Looking at `the Ansible inventory plugins directory on github <https://github.com/ansible/ansible/tree/devel/plugins/inventory>_`, you will find inventory sources already available for:

   Amazon Ec2
   BSD Jails
   Cobbler
   Digital Ocean
   OpenStack Nova
   Linode
   RackSpace Cloud
   Spacewalk
   Vagrant
   Zabbix

Read the comments in each file for usage notes as some require configuration or credentials files.

It is recommended you copy the script you are using to /etc/ansible and chmod +x the file.  Specifying the file with "-i" will use the
script as a dynamic inventory file in Ansible.  Also in ansible, if your inventory location given to "-i" is a directory, you can use
a mixture of inventory scripts and static files at the same time.

Example: The Cobbler External Inventory Script
``````````````````````````````````````````````

Cobbler is a neat datacenter management tool that assists with PXE, DHCP/DNS management, packaging mirroring, and other things.
Cobbler was one of Michael's earlier projects, and is now maintained by James Cammarata, who also works on Ansible for AnsibleWorks.

It is expected that many Ansible may also be `Cobbler <http://cobbler.github.com>`_ users, and those users might like to use their 
Cobbler inventory to define what hosts Ansible should talk to.

How does this work?  
Cobbler has a generic layer that allows it to represent data for multiple configuration management systems (even at the same time), and has
been referred to as a 'lightweight CMDB' by some admins.   This particular script will communicate with Cobbler
lists of hosts in Cobbler to define groups in Ansible -- though this is neither required or suggested, we're just showing you how!  

The example script will talk to Cobbler using Cobbler's XMLRPC API.

To tie Ansible's inventory to Cobbler (optional), copy `this script <https://raw.github.com/ansible/ansible/devel/plugins/inventory/cobbler.py>`_ to /etc/ansible/cobbler and `chmod +x` the file.  cobblerd will now nee to be running when you are using Ansible.

Test the file by running `./etc/ansible/cobbler` directly.   You should see some JSON data output, but it may not have
anything in it just yet.

Let's explore what this does.  In cobbler, assume a scenario somewhat like the following::

    cobbler profile add --name=webserver --distro=CentOS6-x86_64
    cobbler profile edit --name=webserver --mgmt-classes="webserver" --ksmeta="a=2 b=3"
    cobbler system edit --name=foo --dns-name="foo.example.com" --mgmt-classes="atlanta" --ksmeta="c=4"
    cobbler system edit --name=bar --dns-name="bar.example.com" --mgmt-classes="atlanta" --ksmeta="c=5"

In the example above, the system 'foo.example.com' will be addressable by ansible directly, but will also be addressable when using the group names 'webserver' or 'atlanta'.  Since Ansible uses SSH, we'll try to contract system foo over 'foo.example.com', only, never just 'foo'.  Similarly, if you try "ansible foo" it wouldn't find the system... but "ansible 'foo*'" would, because the system DNS name starts with 'foo'.

The script doesn't just provide host and group info.  In addition, as a bonus, when the 'setup' module is run (which happens automatically when using playbooks), the variables 'a', 'b', and 'c' will all be auto-populated in the templates::

    # file: /srv/motd.j2
    Welcome, I am templated with a value of a={{ a }}, b={{ b }}, and c={{ c }}

Which could be executed just like this::

    ansible webserver -m setup
    ansible webserver -m template -a "src=/tmp/motd.j2 dest=/etc/motd"

.. note::
   The name 'webserver' came from cobbler, as did the variables for
   the config file.  You can still pass in your own variables like
   normal in Ansible, but variables from the external inventory script
   will override any that have the same name.

So, with the template above (motd.j2), this would result in the following data being written to /etc/motd for system 'foo'::

    Welcome, I am templated with a value of a=2, b=3, and c=4

And on system 'bar' (bar.example.com)::

    Welcome, I am templated with a value of a=2, b=3, and c=5

And technically, though there is no major good reason to do it, this also works too::

    ansible webserver -m shell -a "echo {{ a }}"

So in other words, you can use those variables in arguments/actions as well.  You might use this to name
a conf.d file appropriately or something similar.  Who knows?

Example: AWS EC2 External Inventory Script
``````````````````````````````````````````

If you use Amazon Web Services EC2, maintaining an inventory file staticlaly may not be the best approach as you'll likely want to make
cloud instances come and go frequently, or might even be taking advantage of AWS autoscaling. 

For this reason, you can use the `EC2 external inventory  <https://raw.github.com/ansible/ansible/devel/plugins/inventory/ec2.py>`_ script.

As with other inventroy scripts, you'll want to specify the path with the "-i" parameter, and will also need to need to copy the `ec2.ini  <https://raw.github.com/ansible/ansible/devel/plugins/inventory/ec2.ini>`_ file alongside the ec2 inventory script. Then you can run ansible as you would normally.

    ansible -i ec2.py -u ubuntu us-east-1d -m ping

Note that to successfully make an API call to AWS, you will need to configure Boto (the Python interface to AWS). There are a `variety of methods <http://docs.pythonboto.org/en/latest/boto_config_tut.html>`_ available, but the simplest is just to export two environment variables:

    export AWS_ACCESS_KEY_ID='AK123'
    export AWS_SECRET_ACCESS_KEY='abc123'

You can test the script by itself to make sure your config is correct

    cd plugins/inventory
    ./ec2.py --list

After a few moments, you should see your entire EC2 inventory across all regions in JSON.

Since each region requires its own API call, if you are only using a small set of regions, feel free to edit ``ec2.ini`` and list only the regions you are interested in. There are other config options in ``ec2.ini`` including cache control, and destination variables.

At their heart, inventory files are simply a mapping from some name to a destination address. The default ``ec2.ini`` settings are configured for running Ansible from outside EC2 (from your laptop for example). If you are running Ansible from within EC2 (which will be faster), internal DNS names and IP addresses may make more sense than public DNS names. In this case, you should modify the ``destination_variable`` in ``ec2.ini`` to be the private DNS name of an instance. This is particularly important when running Ansible within a private subnet inside a VPC, where the only way to access an instance is via its private IP address. For VPC instances, `vpc_destination_variable` in ``ec2.ini`` provides a means of using which ever `boto.ec2.instance variable <http://docs.pythonboto.org/en/latest/ref/ec2.html#module-boto.ec2.instance>`_ makes the most sense for your use case.

The EC2 external inventory provides mappings to instances from several groups:

Instance ID
  These are groups of one since instance IDs are unique, and therefore function as aliases to the hostnames.
  e.g.
  ``i-00112233``
  ``i-a1b1c1d1``

Region
  A group of all instances in an AWS region.
  e.g.
  ``us-east-1``
  ``us-west-2``

Availability Zone
  A group of all instances in an availability zone.
  e.g.
  ``us-east-1a``
  ``us-east-1b``

Security Group
  Instances belong to one or more security groups. A group is created for each security group, with all characters except alphanumerics, dashes (-) converted to underscores (_). Each group is prefixed by ``security_group_``
  e.g.
  ``security_group_default``
  ``security_group_webservers``
  ``security_group_Pete_s_Fancy_Group``

Tags
  Each instance can have a variety of key/value pairs associated with it called Tags. The most common tag key is 'Name', though anything is possible. Each key/value pair is its own group of instances, again with special characters converted to underscores, in the format ``tag_KEY_VALUE``
  e.g.
  ``tag_Name_Web``
  ``tag_Name_redis-master-001``
  ``tag_aws_cloudformation_logical-id_WebServerGroup``

When the Ansible is interacting with a specific server, the EC2 inventory script is called again with the ``--host HOST`` option. This looks up the HOST in the index cache to get the instance ID, and then makes an API call to AWS to get information about that specific instance. It then makes information about that instance available as variables to your playbooks. Each variable is prefixed by ``ec2_``. Here are some of the variables available:

- ec2_architecture
- ec2_description
- ec2_dns_name
- ec2_id
- ec2_image_id
- ec2_instance_type
- ec2_ip_address
- ec2_kernel
- ec2_key_name
- ec2_launch_time
- ec2_monitored
- ec2_ownerId
- ec2_placement
- ec2_platform
- ec2_previous_state
- ec2_private_dns_name
- ec2_private_ip_address
- ec2_public_dns_name
- ec2_ramdisk
- ec2_region
- ec2_root_device_name
- ec2_root_device_type
- ec2_security_group_ids
- ec2_security_group_names
- ec2_spot_instance_request_id
- ec2_state
- ec2_state_code
- ec2_state_reason
- ec2_status
- ec2_subnet_id
- ec2_tag_Name
- ec2_tenancy
- ec2_virtualization_type
- ec2_vpc_id

Both ``ec2_security_group_ids`` and ``ec2_security_group_names`` are comma-separated lists of all security groups. Each EC2 tag is a variable in the format ``ec2_tag_KEY``.

To see the complete list of variables available for an instance, run the script by itself::

    cd plugins/inventory
    ./ec2.py --host ec2-12-12-12-12.compute-1.amazonaws.com

The EC2 inventory script caches information to avoid repeated calls.  To clear this cache, run the ec2 inventory script with the '--refresh-cache' parameter.  The maximum age for the cache is set in the ec2.ini file.

Further Examples
````````````````

Examples of using the inventory scripts for other cloud providers (such as Rackspace Cloud or OpenStack Nova) will be added to this documentation over time, though you should be able to easily tell how to get them operational by reading the scripts in github.  Feel free to browse them for details about how they function.

Writing Your Own
````````````````

To learn how to write your own inventory script, for example, to interface with our own custom database or web service implementation,
see the `developing_inventory` page.

.. seealso::

   :doc:`intro_inventory`
       Basic (non-dynamic) inventory information
   :doc:`developers_inventory`
       How to write your own inventory script
   :doc:`modules`
       List of built-in modules
   `Mailing List <http://groups.google.com/group/ansible-project>`_
       Questions? Help? Ideas?  Stop by the list on Google Groups
   `irc.freenode.net <http://irc.freenode.net>`_
       #ansible IRC chat channel