Merge pull request #13489 from chouseknecht/galaxy-2.0

Galaxy 2.0

docsite/rst/galaxy.rst

@@ -8,7 +8,7 @@ Ansible Galaxy
 The Website
 ```````````
 
-The website `Ansible Galaxy <https://galaxy.ansible.com>`_, is a free site for finding, downloading, rating, and reviewing all kinds of community developed Ansible roles and can be a great way to get a jumpstart on your automation projects.
+The website `Ansible Galaxy <https://galaxy.ansible.com>`_, is a free site for finding, downloading, and sharing community developed Ansible roles. Downloading roles from Galaxy is a great way to jumpstart your automation projects.
 
 You can sign up with social auth and use the download client 'ansible-galaxy' which is included in Ansible 1.4.2 and later.
 
@@ -24,7 +24,7 @@ Installing Roles
 
 The most obvious is downloading roles from the Ansible Galaxy website::
 
-   ansible-galaxy install username.rolename
+   $ ansible-galaxy install username.rolename
 
 .. _galaxy_cli_roles_path:
 
@@ -33,23 +33,16 @@ roles_path
 
 You can specify a particular directory where you want the downloaded roles to be placed::
 
-   ansible-galaxy install username.role -p ~/Code/ansible_roles/
+   $ ansible-galaxy install username.role -p ~/Code/ansible_roles/
    
 This can be useful if you have a master folder that contains ansible galaxy roles shared across several projects. The default is the roles_path configured in your ansible.cfg file (/etc/ansible/roles if not configured).
 
-Building out Role Scaffolding
------------------------------
-
-It can also be used to initialize the base structure of a new role, saving time on creating the various directories and main.yml files a role requires::
-
-   ansible-galaxy init rolename
-
 Installing Multiple Roles From A File
--------------------------------------
+=====================================
 
 To install multiple roles, the ansible-galaxy CLI can be fed a requirements file.  All versions of ansible allow the following syntax for installing roles from the Ansible Galaxy website::
 
-   ansible-galaxy install -r requirements.txt
+   $ ansible-galaxy install -r requirements.txt
 
 Where the requirements.txt looks like::
 
@@ -64,7 +57,7 @@ To request specific versions (tags) of a role, use this syntax in the roles file
 Available versions will be listed on the Ansible Galaxy webpage for that role.
 
 Advanced Control over Role Requirements Files
----------------------------------------------
+=============================================
 
 For more advanced control over where to download roles from, including support for remote repositories, Ansible 1.8 and later support a new YAML format for the role requirements file, which must end in a 'yml' extension.  It works like this::
 
@@ -121,3 +114,275 @@ Roles pulled from galaxy work as with other SCM sourced roles above. To download
    `irc.freenode.net <http://irc.freenode.net>`_
        #ansible IRC chat channel
 
+Building Role Scaffolding
+-------------------------
+
+Use the init command to initialize the base structure of a new role, saving time on creating the various directories and main.yml files a role requires::
+
+   $ ansible-galaxy init rolename
+
+The above will create the following directory structure in the current working directory:
+  
+::
+
+   README.md
+   .travsis.yml
+   defaults/
+       main.yml
+   files/
+   handlers/
+       main.yml
+   meta/
+       main.yml
+   templates/
+   tests/
+       inventory
+       test.yml
+   vars/
+       main.yml
+
+.. note::
+
+    .travis.yml and tests/ are new in Ansible 2.0
+
+If a directory matching the name of the role already exists in the current working directory, the init command will result in an error. To ignore the error use the --force option. Force will create the above subdirectories and files, replacing anything that matches.
+
+Search for Roles
+----------------
+
+The search command provides for querying the Galaxy database, allowing for searching by tags, platforms, author and multiple keywords. For example:
+
+::
+
+    $ ansible-galaxy search elasticsearch --author geerlingguy
+
+The search command will return a list of the first 1000 results matching your search:
+
+::
+    
+    Found 2 roles matching your search:
+
+    Name                              Description
+    ----                              -----------
+    geerlingguy.elasticsearch         Elasticsearch for Linux.
+    geerlingguy.elasticsearch-curator Elasticsearch curator for Linux.
+
+.. note::
+
+    The format of results pictured here is new in Ansible 2.0.
+
+Get More Information About a Role
+---------------------------------
+
+Use the info command To view more detail about a specific role:
+
+::
+
+    $ ansible-galaxy info username.role_name
+
+This returns everything found in Galaxy for the role:
+
+::
+
+    Role: username.rolename
+        description: Installs and configures a thing, a distributed, highly available NoSQL thing.
+        active: True
+        commit: c01947b7bc89ebc0b8a2e298b87ab416aed9dd57
+        commit_message: Adding travis
+        commit_url: https://github.com/username/repo_name/commit/c01947b7bc89ebc0b8a2e298b87ab
+        company: My Company, Inc.
+        created: 2015-12-08T14:17:52.773Z
+        download_count: 1
+        forks_count: 0
+        github_branch:
+        github_repo: repo_name
+        github_user: username
+        id: 6381
+        is_valid: True
+        issue_tracker_url:
+        license: Apache
+        min_ansible_version: 1.4
+        modified: 2015-12-08T18:43:49.085Z
+        namespace: username
+        open_issues_count: 0
+        path: /Users/username/projects/roles
+        scm: None
+        src: username.repo_name
+        stargazers_count: 0
+        travis_status_url: https://travis-ci.org/username/repo_name.svg?branch=master
+        version:
+        watchers_count: 1
+
+.. note::
+
+    The format of results pictured here is new in Ansible 2.0.
+
+
+List Installed Roles
+--------------------
+
+The list command shows the name and version of each role installed in roles_path.
+
+::
+
+    $ ansible-galaxy list
+
+    - chouseknecht.role-install_mongod, master
+    - chouseknecht.test-role-1, v1.0.2
+    - chrismeyersfsu.role-iptables, master
+    - chrismeyersfsu.role-required_vars, master
+
+Remove an Installed Role
+------------------------
+
+The remove command will delete a role from roles_path:
+
+::
+
+    $ ansible-galaxy remove username.rolename
+
+Authenticate with Galaxy
+------------------------
+
+To use the import, delete and setup commands authentication with Galaxy is required. The login command will authenticate the user,retrieve a token from Galaxy, and store it in the user's home directory.
+
+::
+
+    $ ansible-galaxy login
+
+    We need your Github login to identify you.
+    This information will not be sent to Galaxy, only to api.github.com.
+    The password will not be displayed.
+
+    Use --github-token if you do not want to enter your password.
+
+    Github Username: dsmith
+    Password for dsmith:
+    Succesfully logged into Galaxy as dsmith
+
+As depicted above, the login command prompts for a GitHub username and password. It does NOT send your password to Galaxy. It actually authenticates with GitHub and creates a personal access token. It then sends the personal access token to Galaxy, which in turn verifies that you are you and returns a Galaxy access token. After authentication completes the GitHub personal access token is destroyed. 
+
+If you do not wish to use your GitHub password, or if you have two-factor authentication enabled with GitHub, use the --github-token option to pass a personal access token that you create. Log into GitHub, go to Settings and click on Personal Access Token to create a token. 
+
+Import a Role
+-------------
+
+Roles can be imported using ansible-galaxy. The import command expects that the user previously authenticated with Galaxy using the login command.
+
+Import any GitHub repo you have access to:
+
+::
+
+    $ ansible-galaxy import github_user github_repo
+
+By default the command will wait for the role to be imported by Galaxy, displaying the results as the import progresses:
+
+::
+
+    Successfully submitted import request 41
+    Starting import 41: role_name=myrole repo=githubuser/ansible-role-repo ref=
+    Retrieving Github repo githubuser/ansible-role-repo
+    Accessing branch: master
+    Parsing and validating meta/main.yml
+    Parsing galaxy_tags
+    Parsing platforms
+    Adding dependencies
+    Parsing and validating README.md
+    Adding repo tags as role versions
+    Import completed
+    Status SUCCESS : warnings=0 errors=0
+
+Use the --branch option to import a specific branch. If not specified, the default branch for the repo will be used.
+
+If the --no-wait option is present, the command will not wait for results. Results of the most recent import for any of your roles is available on the Galaxy web site under My Imports.
+
+.. note::
+
+    The import command is only available in Ansible 2.0.
+
+Delete a Role
+-------------
+
+Remove a role from the Galaxy web site using the delete command.  You can delete any role that you have access to in GitHub. The delete command expects that the user previously authenticated with Galaxy using the login command.
+
+::
+
+    ansible-galaxy delete github_user github_repo
+
+This only removes the role from Galaxy. It does not impact the actual GitHub repo.
+
+.. note::
+
+    The delete command is only available in Ansible 2.0.
+
+Setup Travis Integerations
+--------------------------
+
+Using the setup command you can enable notifications from `travis <http://travis-ci.org>`_. The setup command expects that the user previously authenticated with Galaxy using the login command.
+
+::
+
+    $ ansible-galaxy setup travis github_user github_repo xxxtravistokenxxx
+
+    Added integration for travis chouseknecht/ansible-role-sendmail 
+
+The setup command requires your Travis token. The Travis token is not stored in Galaxy. It is used along with the GitHub username and repo to create a hash as described in `the Travis documentation <https://docs.travis-ci.com/user/notifications/>`_. The calculated hash is stored in Galaxy and used to verify notifications received from Travis.
+
+The setup command enables Galaxy to respond to notifications. Follow the `Travis getting started guide <https://docs.travis-ci.com/user/getting-started/>`_ to enable the Travis build process for the role repository.
+
+When you create your .travis.yml file add the following to cause Travis to notify Galaxy when a build completes:
+
+::
+
+    notifications:
+        webhooks: https://galaxy.ansible.com/api/v1/notifications/
+
+.. note::
+
+    The setup command is only available in Ansible 2.0.
+
+
+List Travis Integrtions
+=======================
+
+Use the --list option to display your Travis integrations:
+
+::
+
+    $ ansible-galaxy setup --list
+
+
+    ID         Source     Repo
+    ---------- ---------- ----------
+    2          travis     github_user/github_repo
+    1          travis     github_user/github_repo
+
+
+Remove Travis Integrations
+==========================
+
+Use the --remove option to disable a Travis integration:
+
+::
+
+    $ ansible-galaxy setup --remove ID
+
+Provide the ID of the integration you want disabled. Use the --list option to get the ID.
+
+
+
+
+
+
+
+
+
+
+
+ 
+
+
+
+
+
+

lib/ansible/cli/galaxy.py

@@ -22,10 +22,11 @@
 from __future__ import (absolute_import, division, print_function)
 __metaclass__ = type
 
-import os
 import os.path
 import sys
 import yaml
+import json
+import time
 
 from collections import defaultdict
 from jinja2 import Environment
@@ -36,7 +37,10 @@ from ansible.errors import AnsibleError, AnsibleOptionsError
 from ansible.galaxy import Galaxy
 from ansible.galaxy.api import GalaxyAPI
 from ansible.galaxy.role import GalaxyRole
+from ansible.galaxy.login import GalaxyLogin
+from ansible.galaxy.token import GalaxyToken
 from ansible.playbook.role.requirement import RoleRequirement
+from ansible.module_utils.urls import open_url
 
 try:
     from __main__ import display
@@ -44,18 +48,52 @@ except ImportError:
     from ansible.utils.display import Display
     display = Display()
 
-
 class GalaxyCLI(CLI):
 
-    VALID_ACTIONS = ("init", "info", "install", "list", "remove", "search")
+    available_commands = {
+        "delete":    "remove a role from Galaxy",
+        "import":    "add a role contained in a GitHub repo to Galaxy",
+        "info":      "display details about a particular role",
+        "init":      "create a role directory structure in your roles path",
+        "install":   "download a role into your roles path",
+        "list":      "enumerate roles found in your roles path",
+        "login":     "authenticate with Galaxy API and store the token",
+        "remove":    "delete a role from your roles path",
+        "search":    "query the Galaxy API",
+        "setup":     "add a TravisCI integration to Galaxy",
+    } 
+
     SKIP_INFO_KEYS = ("name", "description", "readme_html", "related", "summary_fields", "average_aw_composite", "average_aw_score", "url" )
     
     def __init__(self, args):
-
+        self.VALID_ACTIONS = self.available_commands.keys()
+        self.VALID_ACTIONS.sort()
         self.api = None
         self.galaxy = None
         super(GalaxyCLI, self).__init__(args)
 
+    def set_action(self):
+        """
+        Get the action the user wants to execute from the sys argv list.
+        """
+        for i in range(0,len(self.args)):
+            arg = self.args[i]
+            if arg in self.VALID_ACTIONS:
+                self.action = arg
+                del self.args[i]
+                break
+
+        if not self.action:
+            self.show_available_actions()
+
+    def show_available_actions(self):
+        # list available commands
+        display.display(u'\n' + "usage: ansible-galaxy COMMAND [--help] [options] ...")
+        display.display(u'\n' + "availabe commands:" + u'\n\n')
+        for key in self.VALID_ACTIONS:
+            display.display(u'\t' + "%-12s %s" % (key, self.available_commands[key]))
+        display.display(' ')
+
     def parse(self):
         ''' create an options parser for bin/ansible '''
 
@@ -67,7 +105,17 @@ class GalaxyCLI(CLI):
         self.set_action()
 
         # options specific to actions
-        if self.action == "info":
+        if self.action == "delete":
+            self.parser.set_usage("usage: %prog delete [options] github_user github_repo")
+        elif self.action == "import":
+            self.parser.set_usage("usage: %prog import [options] github_user github_repo")
+            self.parser.add_option('-n', '--no-wait', dest='wait', action='store_false', default=True,
+                help='Don\'t wait for import results.')
+            self.parser.add_option('-b', '--branch', dest='reference',
+                help='The name of a branch to import. Defaults to the repository\'s default branch (usually master)')
+            self.parser.add_option('-t', '--status', dest='check_status', action='store_true', default=False,
+                help='Check the status of the most recent import request for given github_user/github_repo.')
+        elif self.action == "info":
             self.parser.set_usage("usage: %prog info [options] role_name[,version]")
         elif self.action == "init":
             self.parser.set_usage("usage: %prog init [options] role_name")
@@ -88,22 +136,35 @@ class GalaxyCLI(CLI):
             self.parser.set_usage("usage: %prog remove role1 role2 ...")
         elif self.action == "list":
             self.parser.set_usage("usage: %prog list [role_name]")
+        elif self.action == "login":
+            self.parser.set_usage("usage: %prog login [options]")
+            self.parser.add_option('-g','--github-token', dest='token', default=None,
+                help='Identify with github token rather than username and password.')
         elif self.action == "search":
             self.parser.add_option('--platforms', dest='platforms',
                 help='list of OS platforms to filter by')
             self.parser.add_option('--galaxy-tags', dest='tags',
                 help='list of galaxy tags to filter by')
-            self.parser.set_usage("usage: %prog search [<search_term>] [--galaxy-tags <galaxy_tag1,galaxy_tag2>] [--platforms platform]")
+            self.parser.add_option('--author', dest='author',
+                help='GitHub username')
+            self.parser.set_usage("usage: %prog search [searchterm1 searchterm2] [--galaxy-tags galaxy_tag1,galaxy_tag2] [--platforms platform1,platform2] [--author username]")
+        elif self.action == "setup":
+            self.parser.set_usage("usage: %prog setup [options] source github_user github_repo secret" +
+                u'\n\n' + "Create an integration with travis.")
+            self.parser.add_option('-r', '--remove', dest='remove_id', default=None,
+                help='Remove the integration matching the provided ID value. Use --list to see ID values.')
+            self.parser.add_option('-l', '--list', dest="setup_list", action='store_true', default=False,
+                help='List all of your integrations.')
 
         # options that apply to more than one action
-        if self.action != "init":
+        if not self.action in ("config","import","init","login","setup"):
             self.parser.add_option('-p', '--roles-path', dest='roles_path', default=C.DEFAULT_ROLES_PATH,
                 help='The path to the directory containing your roles. '
                      'The default is the roles_path configured in your '
                      'ansible.cfg file (/etc/ansible/roles if not configured)')
 
-        if self.action in ("info","init","install","search"):
+        if self.action in ("import","info","init","install","login","search","setup","delete"):
-            self.parser.add_option('-s', '--server', dest='api_server', default="https://galaxy.ansible.com",
+            self.parser.add_option('-s', '--server', dest='api_server', default=C.GALAXY_SERVER,
                 help='The API server destination')
             self.parser.add_option('-c', '--ignore-certs', action='store_false', dest='validate_certs', default=True,
                 help='Ignore SSL certificate validation errors.')
@@ -112,8 +173,9 @@ class GalaxyCLI(CLI):
             self.parser.add_option('-f', '--force', dest='force', action='store_true', default=False,
                 help='Force overwriting an existing role')
 
+        if self.action:
             # get options, args and galaxy object
-        self.options, self.args =self.parser.parse_args(self.args[1:])
+            self.options, self.args =self.parser.parse_args()
             display.verbosity = self.options.verbosity
             self.galaxy = Galaxy(self.options)
 
@@ -121,14 +183,15 @@ class GalaxyCLI(CLI):
 
     def run(self):
 
+        if not self.action:
+            return True
+
         super(GalaxyCLI, self).run()
 
         # if not offline, get connect to galaxy api
-        if self.action in ("info","install", "search") or (self.action == 'init' and not self.options.offline):
+        if self.action in ("import","info","install","search","login","setup","delete") or \
-            api_server = self.options.api_server
+            (self.action == 'init' and not self.options.offline):            
-            self.api = GalaxyAPI(self.galaxy, api_server)
+            self.api = GalaxyAPI(self.galaxy)
-            if not self.api:
-                raise AnsibleError("The API server (%s) is not responding, please try again later." % api_server)
 
         self.execute()
 
@@ -188,7 +251,7 @@ class GalaxyCLI(CLI):
                             "however it will reset any main.yml files that may have\n"
                             "been modified there already." % role_path)
 
-        # create the default README.md
+        # create default README.md
         if not os.path.exists(role_path):
             os.makedirs(role_path)
         readme_path = os.path.join(role_path, "README.md")
@@ -196,9 +259,16 @@ class GalaxyCLI(CLI):
         f.write(self.galaxy.default_readme)
         f.close()
 
+        # create default .travis.yml
+        travis = Environment().from_string(self.galaxy.default_travis).render()
+        f = open(os.path.join(role_path, '.travis.yml'), 'w')
+        f.write(travis)
+        f.close()
+
         for dir in GalaxyRole.ROLE_DIRS:
             dir_path = os.path.join(init_path, role_name, dir)
             main_yml_path = os.path.join(dir_path, 'main.yml')
+
             # create the directory if it doesn't exist already
             if not os.path.exists(dir_path):
                 os.makedirs(dir_path)
@@ -234,6 +304,20 @@ class GalaxyCLI(CLI):
                 f.write(rendered_meta)
                 f.close()
                 pass
+            elif dir == "tests":
+                # create tests/test.yml
+                inject = dict(
+                    role_name = role_name
+                )
+                playbook = Environment().from_string(self.galaxy.default_test).render(inject)
+                f = open(os.path.join(dir_path, 'test.yml'), 'w')
+                f.write(playbook)
+                f.close()
+
+                # create tests/inventory
+                f = open(os.path.join(dir_path, 'inventory'), 'w')
+                f.write('localhost')
+                f.close()
             elif dir not in ('files','templates'):
                 # just write a (mostly) empty YAML file for main.yml
                 f = open(main_yml_path, 'w')
@@ -325,7 +409,7 @@ class GalaxyCLI(CLI):
 
                     for role in required_roles:
                         role = RoleRequirement.role_yaml_parse(role)
-                        display.debug('found role %s in yaml file' % str(role))
+                        display.vvv('found role %s in yaml file' % str(role))
                         if 'name' not in role and 'scm' not in role:
                             raise AnsibleError("Must specify name or src for role")
                         roles_left.append(GalaxyRole(self.galaxy, **role))
@@ -348,7 +432,7 @@ class GalaxyCLI(CLI):
                 roles_left.append(GalaxyRole(self.galaxy, rname.strip()))
 
         for role in roles_left:
-            display.debug('Installing role %s ' % role.name)
+            display.vvv('Installing role %s ' % role.name)
             # query the galaxy API for the role data
 
             if role.install_info is not None and not force:
@@ -458,21 +542,189 @@ class GalaxyCLI(CLI):
         return 0
 
     def execute_search(self):
-
+        page_size = 1000
         search = None
-        if len(self.args) > 1:
-            raise AnsibleOptionsError("At most a single search term is allowed.")
-        elif len(self.args) == 1:
-            search = self.args.pop()
         
-        response = self.api.search_roles(search, self.options.platforms, self.options.tags)
+        if len(self.args):
+            terms = []
+            for i in range(len(self.args)):
+               terms.append(self.args.pop())
+            search = '+'.join(terms)
 
-        if 'count' in response:
+        if not search and not self.options.platforms and not self.options.tags and not self.options.author:
-            display.display("Found %d roles matching your search:\n" % response['count'])
+            raise AnsibleError("Invalid query. At least one search term, platform, galaxy tag or author must be provided.")
+
+        response = self.api.search_roles(search, platforms=self.options.platforms,
+            tags=self.options.tags, author=self.options.author, page_size=page_size)
+    
+        if response['count'] == 0:
+            display.display("No roles match your search.", color="yellow")
+            return True
 
         data = ''
-        if 'results' in response:
+
+        if response['count'] > page_size:
+            data += ("Found %d roles matching your search. Showing first %s.\n" % (response['count'], page_size))
+        else:
+            data += ("Found %d roles matching your search:\n" % response['count'])
+
+        max_len = []
         for role in response['results']:
-                data += self._display_role_info(role)
+            max_len.append(len(role['username'] + '.' + role['name']))
+        name_len = max(max_len)
+        format_str = " %%-%ds %%s\n" % name_len
+        data +='\n'
+        data += (format_str % ("Name", "Description"))
+        data += (format_str % ("----", "-----------"))
+        for role in response['results']:
+            data += (format_str % (role['username'] + '.' + role['name'],role['description']))
             
         self.pager(data)
+
+        return True
+
+    def execute_login(self):
+        """
+        Verify user's identify via Github and retreive an auth token from Galaxy.
+        """
+        # Authenticate with github and retrieve a token
+        if self.options.token is None:
+            login = GalaxyLogin(self.galaxy)
+            github_token = login.create_github_token()
+        else:
+            github_token = self.options.token
+
+        galaxy_response = self.api.authenticate(github_token)
+        
+        if self.options.token is None:
+            # Remove the token we created
+            login.remove_github_token()
+        
+        # Store the Galaxy token 
+        token = GalaxyToken()
+        token.set(galaxy_response['token'])
+
+        display.display("Succesfully logged into Galaxy as %s" % galaxy_response['username'])
+        return 0
+
+    def execute_import(self):
+        """
+        Import a role into Galaxy
+        """
+        
+        colors = {
+            'INFO':    'normal',
+            'WARNING': 'yellow',
+            'ERROR':   'red',
+            'SUCCESS': 'green',
+            'FAILED':  'red'
+        }
+
+        if len(self.args) < 2:
+            raise AnsibleError("Expected a github_username and github_repository. Use --help.")
+
+        github_repo = self.args.pop()
+        github_user = self.args.pop()
+
+        if self.options.check_status:
+            task = self.api.get_import_task(github_user=github_user, github_repo=github_repo)
+        else:
+            # Submit an import request
+            task = self.api.create_import_task(github_user, github_repo, reference=self.options.reference)
+            
+            if len(task) > 1:
+                # found multiple roles associated with github_user/github_repo
+                display.display("WARNING: More than one Galaxy role associated with Github repo %s/%s." % (github_user,github_repo),
+                    color='yellow')
+                display.display("The following Galaxy roles are being updated:" + u'\n', color='yellow')
+                for t in task:
+                    display.display('%s.%s' % (t['summary_fields']['role']['namespace'],t['summary_fields']['role']['name']), color='yellow')
+                display.display(u'\n' + "To properly namespace this role, remove each of the above and re-import %s/%s from scratch" % (github_user,github_repo),
+                    color='yellow')
+                return 0
+            # found a single role as expected
+            display.display("Successfully submitted import request %d" % task[0]['id'])
+            if not self.options.wait:
+                display.display("Role name: %s" % task[0]['summary_fields']['role']['name'])
+                display.display("Repo: %s/%s" % (task[0]['github_user'],task[0]['github_repo']))
+
+        if self.options.check_status or self.options.wait:
+            # Get the status of the import
+            msg_list = []
+            finished = False
+            while not finished:
+                task = self.api.get_import_task(task_id=task[0]['id'])
+                for msg in task[0]['summary_fields']['task_messages']:
+                    if msg['id'] not in msg_list:
+                        display.display(msg['message_text'], color=colors[msg['message_type']])
+                        msg_list.append(msg['id'])
+                if task[0]['state'] in ['SUCCESS', 'FAILED']:
+                    finished = True
+                else:
+                    time.sleep(10)
+
+        return 0
+
+    def execute_setup(self):
+        """
+        Setup an integration from Github or Travis
+        """
+
+        if self.options.setup_list:
+            # List existing integration secrets
+            secrets = self.api.list_secrets()
+            if len(secrets) == 0:
+                # None found
+                display.display("No integrations found.")
+                return 0
+            display.display(u'\n' + "ID         Source     Repo", color="green")
+            display.display("---------- ---------- ----------", color="green")
+            for secret in secrets:
+                display.display("%-10s %-10s %s/%s" % (secret['id'], secret['source'], secret['github_user'],
+                    secret['github_repo']),color="green")
+            return 0
+
+        if self.options.remove_id:
+            # Remove a secret
+            self.api.remove_secret(self.options.remove_id)
+            display.display("Secret removed. Integrations using this secret will not longer work.", color="green")
+            return 0
+
+        if len(self.args) < 4:
+            raise AnsibleError("Missing one or more arguments. Expecting: source github_user github_repo secret")
+            return 0
+            
+        secret = self.args.pop()
+        github_repo = self.args.pop()
+        github_user = self.args.pop()
+        source = self.args.pop()
+
+        resp = self.api.add_secret(source, github_user, github_repo, secret)
+        display.display("Added integration for %s %s/%s" % (resp['source'], resp['github_user'], resp['github_repo']))
+
+        return 0
+
+    def execute_delete(self):
+        """
+        Delete a role from galaxy.ansible.com
+        """
+
+        if len(self.args) < 2:
+            raise AnsibleError("Missing one or more arguments. Expected: github_user github_repo")
+        
+        github_repo = self.args.pop()
+        github_user = self.args.pop()
+        resp = self.api.delete_role(github_user, github_repo)
+
+        if len(resp['deleted_roles']) > 1:
+            display.display("Deleted the following roles:")
+            display.display("ID     User            Name")
+            display.display("------ --------------- ----------")
+            for role in resp['deleted_roles']:
+                display.display("%-8s %-15s %s" % (role.id,role.namespace,role.name))
+        
+        display.display(resp['status'])
+
+        return True
+
+        

lib/ansible/constants.py

@@ -255,7 +255,8 @@ ACCELERATE_MULTI_KEY           = get_config(p, 'accelerate', 'accelerate_multi_k
 PARAMIKO_PTY                   = get_config(p, 'paramiko_connection', 'pty', 'ANSIBLE_PARAMIKO_PTY', True, boolean=True)
 
 # galaxy related
-DEFAULT_GALAXY_URI             = get_config(p, 'galaxy', 'server_uri', 'ANSIBLE_GALAXY_SERVER_URI', 'https://galaxy.ansible.com')
+GALAXY_SERVER                  = get_config(p, 'galaxy', 'server', 'ANSIBLE_GALAXY_SERVER', 'https://galaxy.ansible.com')
+GALAXY_IGNORE_CERTS            = get_config(p, 'galaxy', 'ignore_certs', 'ANSIBLE_GALAXY_IGNORE', False, boolean=True)
 # this can be configured to blacklist SCMS but cannot add new ones unless the code is also updated
 GALAXY_SCMS                    = get_config(p, 'galaxy', 'scms', 'ANSIBLE_GALAXY_SCMS', 'git, hg', islist=True)
 

lib/ansible/galaxy/__init__.py

@@ -52,6 +52,8 @@ class Galaxy(object):
         #TODO: move to getter for lazy loading
         self.default_readme = self._str_from_data_file('readme')
         self.default_meta = self._str_from_data_file('metadata_template.j2')
+        self.default_test = self._str_from_data_file('test_playbook.j2')
+        self.default_travis = self._str_from_data_file('travis.j2')
 
     def add_role(self, role):
         self.roles[role.name] = role

lib/ansible/galaxy/api.py

@@ -25,11 +25,15 @@ from __future__ import (absolute_import, division, print_function)
 __metaclass__ = type
 
 import json
+import urllib
+
 from urllib2 import quote as urlquote, HTTPError
 from urlparse import urlparse
 
+import ansible.constants as C
 from ansible.errors import AnsibleError
 from ansible.module_utils.urls import open_url
+from ansible.galaxy.token import GalaxyToken
 
 try:
     from __main__ import display
@@ -43,45 +47,113 @@ class GalaxyAPI(object):
 
     SUPPORTED_VERSIONS = ['v1']
 
-    def __init__(self, galaxy, api_server):
+    def __init__(self, galaxy):
 
         self.galaxy = galaxy
+        self.token = GalaxyToken()
+        self._api_server = C.GALAXY_SERVER
+        self._validate_certs = C.GALAXY_IGNORE_CERTS
 
-        try:
+        # set validate_certs
-            urlparse(api_server, scheme='https')
+        if galaxy.options.validate_certs == False:
-        except:
+            self._validate_certs = False
-            raise AnsibleError("Invalid server API url passed: %s" % api_server)
+        display.vvv('Check for valid certs: %s' % self._validate_certs)
 
-        server_version = self.get_server_api_version('%s/api/' % (api_server))
+        # set the API server
-        if not server_version:
+        if galaxy.options.api_server != C.GALAXY_SERVER:
-            raise AnsibleError("Could not retrieve server API version: %s" % api_server)
+            self._api_server = galaxy.options.api_server
+        display.vvv("Connecting to galaxy_server: %s" % self._api_server)
+
+        server_version = self.get_server_api_version()
        
         if server_version in self.SUPPORTED_VERSIONS:
-            self.baseurl = '%s/api/%s' % (api_server, server_version)
+            self.baseurl = '%s/api/%s' % (self._api_server, server_version)
             self.version = server_version # for future use
-            display.vvvvv("Base API: %s" % self.baseurl)
+            display.vvv("Base API: %s" % self.baseurl)
         else:
             raise AnsibleError("Unsupported Galaxy server API version: %s" % server_version)
 
-    def get_server_api_version(self, api_server):
+    def __auth_header(self):
+        token = self.token.get()
+        if token is None:
+            raise AnsibleError("No access token. You must first use login to authenticate and obtain an access token.")
+        return {'Authorization': 'Token ' + token}
+
+    def __call_galaxy(self, url, args=None, headers=None, method=None):
+        if args and not headers:
+            headers = self.__auth_header()
+        try:
+            display.vvv(url)
+            resp = open_url(url, data=args, validate_certs=self._validate_certs, headers=headers, method=method)
+            data = json.load(resp)
+        except HTTPError as e:
+            res = json.load(e)
+            raise AnsibleError(res['detail'])
+        return data
+
+    @property
+    def api_server(self):
+        return self._api_server    
+    
+    @property
+    def validate_certs(self):
+        return self._validate_certs
+
+    def get_server_api_version(self):
         """
         Fetches the Galaxy API current version to ensure
         the API server is up and reachable.
         """
-        #TODO: fix galaxy server which returns current_version path (/api/v1) vs actual version (v1)
-        # also should set baseurl using supported_versions which has path
-        return 'v1'
-
         try:
-            data = json.load(open_url(api_server, validate_certs=self.galaxy.options.validate_certs))
+            url = '%s/api/' % self._api_server
-            return data.get("current_version", 'v1')
+            data = json.load(open_url(url, validate_certs=self._validate_certs))
-        except Exception:
+            return data['current_version']
-            # TODO: report error
+        except Exception as e:
-            return None
+            raise AnsibleError("The API server (%s) is not responding, please try again later." % url)
+        
+    def authenticate(self, github_token):
+        """
+        Retrieve an authentication token
+        """
+        url = '%s/tokens/' % self.baseurl
+        args = urllib.urlencode({"github_token": github_token})
+        resp = open_url(url, data=args, validate_certs=self._validate_certs, method="POST")
+        data = json.load(resp)
+        return data
+
+    def create_import_task(self, github_user, github_repo, reference=None):
+        """
+        Post an import request
+        """
+        url = '%s/imports/' % self.baseurl
+        args = urllib.urlencode({
+            "github_user": github_user,
+            "github_repo": github_repo,
+            "github_reference": reference if reference else ""
+        })
+        data = self.__call_galaxy(url, args=args)
+        if data.get('results', None):
+            return data['results']
+        return data
+
+    def get_import_task(self, task_id=None, github_user=None, github_repo=None):
+        """
+        Check the status of an import task.
+        """
+        url = '%s/imports/' % self.baseurl
+        if not task_id is None:
+            url = "%s?id=%d" % (url,task_id)
+        elif not github_user is None and not github_repo is None:
+            url = "%s?github_user=%s&github_repo=%s" % (url,github_user,github_repo)
+        else:
+            raise AnsibleError("Expected task_id or github_user and github_repo")
+        
+        data = self.__call_galaxy(url)
+        return data['results']
        
     def lookup_role_by_name(self, role_name, notify=True):
         """
-        Find a role by name
+        Find a role by name.
         """
         role_name = urlquote(role_name)
 
@@ -92,18 +164,12 @@ class GalaxyAPI(object):
             if notify:
                 display.display("- downloading role '%s', owned by %s" % (role_name, user_name))
         except:
-            raise AnsibleError("- invalid role name (%s). Specify role as format: username.rolename" % role_name)
+            raise AnsibleError("Invalid role name (%s). Specify role as format: username.rolename" % role_name)
 
         url = '%s/roles/?owner__username=%s&name=%s' % (self.baseurl, user_name, role_name)
-        display.vvvv("- %s" % (url))
+        data = self.__call_galaxy(url)
-        try:
-            data = json.load(open_url(url, validate_certs=self.galaxy.options.validate_certs))
         if len(data["results"]) != 0:
             return data["results"][0]
-        except:
-            # TODO: report on connection/availability errors
-            pass
-
         return None
 
     def fetch_role_related(self, related, role_id):
@@ -114,13 +180,12 @@ class GalaxyAPI(object):
 
         try:
             url = '%s/roles/%d/%s/?page_size=50' % (self.baseurl, int(role_id), related)
-            data = json.load(open_url(url, validate_certs=self.galaxy.options.validate_certs))
+            data = self.__call_galaxy(url)
             results = data['results']
             done = (data.get('next', None) is None)
             while not done:
                 url = '%s%s' % (self.baseurl, data['next'])
-                display.display(url)
+                data = self.__call_galaxy(url)
-                data = json.load(open_url(url, validate_certs=self.galaxy.options.validate_certs))
                 results += data['results']
                 done = (data.get('next', None) is None)
             return results
@@ -131,10 +196,9 @@ class GalaxyAPI(object):
         """
         Fetch the list of items specified.
         """
-
         try:
             url = '%s/%s/?page_size' % (self.baseurl, what)
-            data = json.load(open_url(url, validate_certs=self.galaxy.options.validate_certs))
+            data = self.__call_galaxy(url)
             if "results" in data:
                 results = data['results']
             else:
@@ -144,41 +208,64 @@ class GalaxyAPI(object):
                 done = (data.get('next', None) is None)
             while not done:
                 url = '%s%s' % (self.baseurl, data['next'])
-                display.display(url)
+                data = self.__call_galaxy(url)
-                data = json.load(open_url(url, validate_certs=self.galaxy.options.validate_certs))
                 results += data['results']
                 done = (data.get('next', None) is None)
             return results
         except Exception as error:
             raise AnsibleError("Failed to download the %s list: %s" % (what, str(error)))
 
-    def search_roles(self, search, platforms=None, tags=None):
+    def search_roles(self, search, **kwargs):
 
-        search_url = self.baseurl + '/roles/?page=1'
+        search_url = self.baseurl + '/search/roles/?'
 
         if search:
-            search_url += '&search=' + urlquote(search)
+            search_url += '&autocomplete=' + urlquote(search)
 
-        if tags is None:
+        tags = kwargs.get('tags',None)
-            tags = []
+        platforms = kwargs.get('platforms', None)
-        elif isinstance(tags, basestring):
+        page_size = kwargs.get('page_size', None)
+        author = kwargs.get('author', None)
+
+        if tags and isinstance(tags, basestring):
             tags = tags.split(',')
+            search_url += '&tags_autocomplete=' + '+'.join(tags)
         
-        for tag in tags:
+        if platforms and isinstance(platforms, basestring):
-            search_url += '&chain__tags__name=' + urlquote(tag)
-
-        if platforms is None:
-            platforms = []
-        elif isinstance(platforms, basestring):
             platforms = platforms.split(',')
+            search_url += '&platforms_autocomplete=' + '+'.join(platforms)
 
-        for plat in platforms:
+        if page_size:
-            search_url += '&chain__platforms__name=' + urlquote(plat)
+            search_url += '&page_size=%s' % page_size
 
-        display.debug("Executing query: %s" % search_url)
+        if author:
-        try:
+            search_url += '&username_autocomplete=%s' % author
-            data = json.load(open_url(search_url, validate_certs=self.galaxy.options.validate_certs))
-        except HTTPError as e:
-            raise AnsibleError("Unsuccessful request to server: %s" % str(e))
    
+        data = self.__call_galaxy(search_url)
+        return data
+
+    def add_secret(self, source, github_user, github_repo, secret):
+        url = "%s/notification_secrets/" % self.baseurl
+        args = urllib.urlencode({
+            "source": source,
+            "github_user": github_user,
+            "github_repo": github_repo,
+            "secret": secret
+        })
+        data = self.__call_galaxy(url, args=args)
+        return data
+
+    def list_secrets(self):
+        url = "%s/notification_secrets" % self.baseurl
+        data = self.__call_galaxy(url, headers=self.__auth_header())
+        return data
+
+    def remove_secret(self, secret_id):
+        url = "%s/notification_secrets/%s/" % (self.baseurl, secret_id)
+        data = self.__call_galaxy(url, headers=self.__auth_header(), method='DELETE')
+        return data
+
+    def delete_role(self, github_user, github_repo):
+        url = "%s/removerole/?github_user=%s&github_repo=%s" % (self.baseurl,github_user,github_repo)
+        data = self.__call_galaxy(url, headers=self.__auth_header(), method='DELETE')
         return data

lib/ansible/galaxy/data/metadata_template.j2

@@ -2,9 +2,11 @@ galaxy_info:
   author: {{ author }}
   description: {{description}}
   company: {{ company }}
+  
   # If the issue tracker for your role is not on github, uncomment the
   # next line and provide a value
   # issue_tracker_url: {{ issue_tracker_url }}
+  
   # Some suggested licenses:
   # - BSD (default)
   # - MIT
@@ -13,7 +15,17 @@ galaxy_info:
   # - Apache
   # - CC-BY
   license: {{ license }}
+  
   min_ansible_version: {{ min_ansible_version }}
+
+  # Optionally specify the branch Galaxy will use when accessing the GitHub
+  # repo for this role. During role install, if no tags are available,
+  # Galaxy will use this branch. During import Galaxy will access files on
+  # this branch. If travis integration is cofigured, only notification for this
+  # branch will be accepted. Otherwise, in all cases, the repo's default branch
+  # (usually master) will be used.
+  #github_branch:
+  
   #
   # Below are all platforms currently available. Just uncomment
   # the ones that apply to your role. If you don't see your
@@ -28,6 +40,7 @@ galaxy_info:
   #  - {{ version }}
     {%- endfor %}
   {%- endfor %}
+  
   galaxy_tags: []
     # List tags for your role here, one per line. A tag is
     # a keyword that describes and categorizes the role.
@@ -36,6 +49,7 @@ galaxy_info:
     #
     # NOTE: A tag is limited to a single word comprised of
     # alphanumeric characters. Maximum 20 tags per role.
+
 dependencies: []
   # List your role dependencies here, one per line.
   # Be sure to remove the '[]' above if you add dependencies

lib/ansible/galaxy/data/test_playbook.j2

@@ -0,0 +1,5 @@
+---
+- hosts: localhost
+  remote_user: root
+  roles:
+    - {{ role_name }}

lib/ansible/galaxy/data/travis.j2

@@ -0,0 +1,29 @@
+---
+language: python
+python: "2.7"
+
+# Use the new container infrastructure
+sudo: false
+
+# Install ansible
+addons:
+  apt:
+    packages:
+    - python-pip
+
+install:
+  # Install ansible
+  - pip install ansible
+
+  # Check ansible version
+  - ansible --version
+
+  # Create ansible.cfg with correct roles_path
+  - printf '[defaults]\nroles_path=../' >ansible.cfg
+
+script:
+  # Basic role syntax check
+  - ansible-playbook tests/test.yml -i tests/inventory --syntax-check
+
+notifications:
+  webhooks: https://galaxy.ansible.com/api/v1/notifications/

lib/ansible/galaxy/login.py

@@ -0,0 +1,113 @@
+#!/usr/bin/env python
+
+########################################################################
+#
+# (C) 2015, Chris Houseknecht <chouse@ansible.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+#
+########################################################################
+
+from __future__ import (absolute_import, division, print_function)
+__metaclass__ = type
+
+import getpass
+import json
+import urllib
+
+from urllib2 import quote as urlquote, HTTPError
+from urlparse import urlparse
+
+from ansible.errors import AnsibleError, AnsibleOptionsError
+from ansible.module_utils.urls import open_url
+from ansible.utils.color import stringc
+
+try:
+    from __main__ import display
+except ImportError:
+    from ansible.utils.display import Display
+    display = Display()
+
+class GalaxyLogin(object):
+    ''' Class to handle authenticating user with Galaxy API prior to performing CUD operations '''
+
+    GITHUB_AUTH = 'https://api.github.com/authorizations'
+    
+    def __init__(self, galaxy, github_token=None):
+        self.galaxy = galaxy
+        self.github_username = None
+        self.github_password = None
+        
+        if github_token == None:
+            self.get_credentials()
+
+    def get_credentials(self):
+        display.display(u'\n\n' + "We need your " + stringc("Github login",'bright cyan') + 
+            " to identify you.", screen_only=True)
+        display.display("This information will " + stringc("not be sent to Galaxy",'bright cyan') + 
+            ", only to " + stringc("api.github.com.","yellow"), screen_only=True)
+        display.display("The password will not be displayed." + u'\n\n', screen_only=True)
+        display.display("Use " + stringc("--github-token",'yellow') + 
+            " if you do not want to enter your password." + u'\n\n', screen_only=True)
+        
+        try:
+            self.github_username = raw_input("Github Username: ")
+        except:
+            pass
+
+        try:
+            self.github_password = getpass.getpass("Password for %s: " % self.github_username)
+        except:
+            pass
+
+        if not self.github_username or not self.github_password:
+            raise AnsibleError("Invalid Github credentials. Username and password are required.")
+
+    def remove_github_token(self):
+        '''
+        If for some reason an ansible-galaxy token was left from a prior login, remove it. We cannot
+        retrieve the token after creation, so we are forced to create a new one.
+        '''
+        try:
+            tokens = json.load(open_url(self.GITHUB_AUTH, url_username=self.github_username,
+                url_password=self.github_password, force_basic_auth=True,))
+        except HTTPError as e:
+            res = json.load(e)
+            raise AnsibleError(res['message'])
+
+        for token in tokens:
+            if token['note'] == 'ansible-galaxy login':
+                display.vvvvv('removing token: %s' % token['token_last_eight'])
+                try: 
+                    open_url('https://api.github.com/authorizations/%d' % token['id'], url_username=self.github_username,
+                        url_password=self.github_password, method='DELETE', force_basic_auth=True,)
+                except HTTPError as e:
+                    res = json.load(e)
+                    raise AnsibleError(res['message'])
+
+    def create_github_token(self):
+        '''
+        Create a personal authorization token with a note of 'ansible-galaxy login'
+        '''
+        self.remove_github_token()
+        args = json.dumps({"scopes":["public_repo"], "note":"ansible-galaxy login"})
+        try:
+            data = json.load(open_url(self.GITHUB_AUTH, url_username=self.github_username,
+                url_password=self.github_password, force_basic_auth=True, data=args))
+        except HTTPError as e:
+            res = json.load(e)
+            raise AnsibleError(res['message'])
+        return data['token']

lib/ansible/galaxy/role.py

@@ -46,7 +46,7 @@ class GalaxyRole(object):
     SUPPORTED_SCMS = set(['git', 'hg'])
     META_MAIN = os.path.join('meta', 'main.yml')
     META_INSTALL = os.path.join('meta', '.galaxy_install_info')
-    ROLE_DIRS = ('defaults','files','handlers','meta','tasks','templates','vars')
+    ROLE_DIRS = ('defaults','files','handlers','meta','tasks','templates','vars','tests')
 
 
     def __init__(self, galaxy, name, src=None, version=None, scm=None, path=None):
@@ -198,10 +198,10 @@ class GalaxyRole(object):
                 role_data = self.src
                 tmp_file = self.fetch(role_data)
             else:
-                api = GalaxyAPI(self.galaxy, self.options.api_server)
+                api = GalaxyAPI(self.galaxy)
                 role_data = api.lookup_role_by_name(self.src)
                 if not role_data:
-                    raise AnsibleError("- sorry, %s was not found on %s." % (self.src, self.options.api_server))
+                    raise AnsibleError("- sorry, %s was not found on %s." % (self.src, api.api_server))
 
                 role_versions = api.fetch_role_related('versions', role_data['id'])
                 if not self.version:
@@ -213,6 +213,8 @@ class GalaxyRole(object):
                         loose_versions = [LooseVersion(a.get('name',None)) for a in role_versions]
                         loose_versions.sort()
                         self.version = str(loose_versions[-1])
+                    elif role_data.get('github_branch', None):
+                        self.version = role_data['github_branch']
                     else:
                         self.version = 'master' 
                 elif self.version != 'master':

lib/ansible/galaxy/token.py

@@ -0,0 +1,67 @@
+#!/usr/bin/env python
+
+########################################################################
+#
+# (C) 2015, Chris Houseknecht <chouse@ansible.com>
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+#
+########################################################################
+from __future__ import (absolute_import, division, print_function)
+__metaclass__ = type
+
+import os
+import yaml
+from stat import *
+
+try:
+    from __main__ import display
+except ImportError:
+    from ansible.utils.display import Display
+    display = Display()
+
+
+class GalaxyToken(object):
+    ''' Class to storing and retrieving token in ~/.ansible_galaxy '''
+
+    def __init__(self):
+        self.file = os.path.expanduser("~") + '/.ansible_galaxy'
+        self.config = yaml.safe_load(self.__open_config_for_read())
+        if not self.config:
+            self.config = {}
+        
+    def __open_config_for_read(self):
+        if os.path.isfile(self.file):
+            display.vvv('Opened %s' % self.file)
+            return open(self.file, 'r')
+        # config.yml not found, create and chomd u+rw
+        f = open(self.file,'w')
+        f.close()
+        os.chmod(self.file,S_IRUSR|S_IWUSR) # owner has +rw
+        display.vvv('Created %s' % self.file) 
+        return open(self.file, 'r')
+
+    def set(self, token): 
+        self.config['token'] = token
+        self.save()
+    
+    def get(self):
+        return self.config.get('token', None)
+    
+    def save(self):
+        with open(self.file,'w') as f:
+            yaml.safe_dump(self.config,f,default_flow_style=False)
+