Update netconf_config module (#44379)

Fixes #40650
Fixes #40245
Fixes #41541

*  Refactor netconf_config module as per proposal #104
*  Update netconf_config module metadata to core network supported
*  Refactor local connection to use persistent connection framework
   for backward compatibility
*  Update netconf connection plugin configuration varaibles (Fixes #40245)
*  Add support for optional lock feature to Fixes #41541
*  Add integration test for netconf_config module
*  Documentation update
* Move deprecated options in netconf_config module

docs/docsite/rst/network/user_guide/platform_index.rst

@@ -20,6 +20,7 @@ Some Ansible Network platforms support multiple connection types, privilege esca
    platform_routeros
    platform_slxos
    platform_voss
+   platform_netconf_enabled
 
 .. _settings_by_platform:
 
@@ -63,5 +64,8 @@ Settings by Platform
 +-------------------+-------------------------+----------------------+----------------------+------------------+------------------+
 | VyOS*             | ``vyos``                | in v. >=2.5          | N/A                  | N/A              |  in v. >=2.4     |
 +-------------------+-------------------------+----------------------+----------------------+------------------+------------------+
+| OS that supports  | ``<network-os>``        | N/A                  | in v. >=2.6          | N/A              | in v. >=2.2      |
+| Netconf*          |                         |                      |                      |                  |                  |
++-------------------+-------------------------+----------------------+----------------------+------------------+------------------+
 
 `*` Maintained by Ansible Network Team

docs/docsite/rst/network/user_guide/platform_netconf_enabled.rst

@@ -0,0 +1,89 @@
+.. _netconf_enabled_platform_options:
+
+***************************************
+Netconf enabled Platform Options
+***************************************
+
+This page offers details on how the netconf connection works in Ansible 2.7 and how to use it.
+
+.. contents:: Topics
+
+Connections Available
+================================================================================
+
++----------------------------+----------------------------------------------------------------------------------------------------+
+| |                          | | NETCONF                                                                                          |
+| |                          | | * all modules except ``junos_netconf``, which enables NETCONF                                    |
++============================+====================================================================================================+
+| **Protocol**               | XML over SSH                                                                                       |
++----------------------------+----------------------------------------------------------------------------------------------------+
+| | **Credentials**          | | uses SSH keys / SSH-agent if present                                                             |
+| |                          | | accepts ``-u myuser -k`` if using password                                                       |
++----------------------------+----------------------------------------------------------------------------------------------------+
+| **Indirect Access**        | via a bastion (jump host)                                                                          |
++----------------------------+----------------------------------------------------------------------------------------------------+
+| **Connection Settings**    |   ``ansible_connection: netconf``                                                                  |
++----------------------------+----------------------------------------------------------------------------------------------------+
+
+For legacy playbooks, Ansible still supports ``ansible_connection=local`` for the netconf_config module only. We recommend modernizing to use ``ansible_connection=netconf`` as soon as possible.
+
+Using NETCONF in Ansible 2.6 onwards
+================================================================================
+
+Enabling NETCONF
+----------------
+
+Before you can use NETCONF to connect to a switch, you must:
+
+- install the ``ncclient`` Python package on your control node(s) with ``pip install ncclient``
+- enable NETCONF on the Junos OS device(s)
+
+To enable NETCONF on a new switch via Ansible, use the platform specific module via the CLI connection or set it manually.
+For example set up your platform-level variables just like in the CLI example above, then run a playbook task like this:
+
+.. code-block:: yaml
+
+   - name: Enable NETCONF
+     connection: network_cli
+     junos_netconf:
+     when: ansible_network_os == 'junos'
+
+Once NETCONF is enabled, change your variables to use the NETCONF connection.
+
+Example NETCONF inventory ``[junos:vars]``
+------------------------------------------
+
+.. code-block:: yaml
+
+   [junos:vars]
+   ansible_connection=netconf
+   ansible_network_os=junos
+   ansible_user=myuser
+   ansible_ssh_pass=!vault |
+
+
+Example NETCONF Task
+--------------------
+
+.. code-block:: yaml
+
+   - name: Backup current switch config
+     netconf_config:
+       backup: yes
+     register: backup_junos_location
+
+Example NETCONF Task with configurable variables
+------------------------------------------------
+
+.. code-block:: yaml
+
+   - name: configure interface while providing different private key file path
+     netconf_config:
+       backup: yes
+     register: backup_junos_location
+     vars:
+       ansible_private_key_file: /home/admin/.ssh/newprivatekeyfile
+
+Note: For nectonf connection plugin configurable variables .. _Refer: https://docs.ansible.com/ansible/latest/plugins/connection/netconf.html
+
+.. include:: shared_snippets/SSH_warning.txt

lib/ansible/config/base.yml

@@ -1473,7 +1473,7 @@ MERGE_MULTIPLE_CLI_TAGS:
   version_added: "2.3"
 NETWORK_GROUP_MODULES:
   name: Network module families
-  default: [eos, nxos, ios, iosxr, junos, enos, ce, vyos, sros, dellos9, dellos10, dellos6, asa, aruba, aireos, bigip, ironware, onyx]
+  default: [eos, nxos, ios, iosxr, junos, enos, ce, vyos, sros, dellos9, dellos10, dellos6, asa, aruba, aireos, bigip, ironware, onyx, netconf]
   description: 'TODO: write it'
   env: [{name: NETWORK_GROUP_MODULES}]
   ini:

lib/ansible/module_utils/network/netconf/netconf.py

@@ -18,13 +18,22 @@
 #
 import json
 
+from copy import deepcopy
 from contextlib import contextmanager
 
-from ansible.module_utils._text import to_text
+try:
+    from lxml.etree import fromstring, tostring
+except ImportError:
+    from xml.etree.ElementTree import fromstring, tostring
+
+from ansible.module_utils._text import to_text, to_bytes
 from ansible.module_utils.connection import Connection, ConnectionError
 from ansible.module_utils.network.common.netconf import NetconfConnection
 
 
+IGNORE_XML_ATTRIBUTE = ()
+
+
 def get_connection(module):
     if hasattr(module, '_netconf_connection'):
         return module._netconf_connection
@@ -114,3 +123,15 @@ def dispatch(module, request):
         module.fail_json(msg=to_text(e, errors='surrogate_then_replace').strip())
 
     return response
+
+
+def sanitize_xml(data):
+    tree = fromstring(to_bytes(deepcopy(data), errors='surrogate_then_replace'))
+    for element in tree.getiterator():
+        # remove attributes
+        attribute = element.attrib
+        if attribute:
+            for key in attribute:
+                if key not in IGNORE_XML_ATTRIBUTE:
+                    attribute.pop(key)
+    return to_text(tostring(tree), errors='surrogate_then_replace').strip()

lib/ansible/modules/network/netconf/netconf_config.py

@@ -9,112 +9,157 @@ __metaclass__ = type
 
 ANSIBLE_METADATA = {'metadata_version': '1.1',
                     'status': ['preview'],
-                    'supported_by': 'community'}
+                    'supported_by': 'network'}
 
 
 DOCUMENTATION = '''
 ---
 module: netconf_config
+version_added: "2.2"
 author: "Leandro Lisboa Penz (@lpenz)"
 short_description: netconf device configuration
 description:
     - Netconf is a network management protocol developed and standardized by
       the IETF. It is documented in RFC 6241.
-
     - This module allows the user to send a configuration XML file to a netconf
       device, and detects if there was a configuration change.
-notes:
+extends_documentation_fragment: netconf
-    - This module supports devices with and without the candidate and
-      confirmed-commit capabilities. It always use the safer feature.
-    - This module supports the use of connection=netconf
-version_added: "2.2"
 options:
-  host:
+  content:
     description:
-     - the hostname or ip address of the netconf device
+     - The configuration data as defined by the device's data models, the value can be either in
-    required: true
+       xml string format or text format. The format of the configuration should be supported by remote
-  port:
+       Netconf server
+    aliases: ['xml']
+  target:
     description:
-     - the netconf port
+      Name of the configuration datastore to be edited.
-    default: 830
+      - auto, uses candidate and fallback to running
-    required: false
+      - candidate, edit <candidate/> datastore and then commit
-  hostkey_verify:
+      - running, edit <running/> datastore directly
-    description:
-     - if true, the ssh host key of the device must match a ssh key present on the host
-     - if false, the ssh host key of the device is not checked
-    default: true
-    required: false
-  look_for_keys:
-    description:
-     - if true, enables looking in the usual locations for ssh keys (e.g. ~/.ssh/id_*)
-     - if false, disables looking for ssh keys
-    default: true
-    required: false
-    version_added: "2.4"
-  allow_agent:
-    description:
-     - if true, enables querying SSH agent (if found) for keys
-     - if false, disables querying the SSH agent for ssh keys
-    default: true
-    required: false
-    version_added: "2.4"
-  datastore:
-    description:
-     - auto, uses candidate and fallback to running
-     - candidate, edit <candidate/> datastore and then commit
-     - running, edit <running/> datastore directly
     default: auto
-    required: false
     version_added: "2.4"
+    aliases: ['datastore']
+  source_datastore:
+    description:
+    - Name of the configuration datastore to use as the source to copy the configuration
+      to the datastore mentioned by C(target) option. The values can be either I(running), I(candidate),
+      I(startup) or a remote URL
+    version_added: "2.7"
+    aliases: ['source']
+  format:
+    description:
+    - The format of the configuration provided as value of C(content). Accepted values are I(xml) and I(test) and
+      the given configuration format should be supported by remote Netconf server.
+    default: xml
+    choices: ['xml', 'text']
+    version_added: "2.7"
+  lock:
+    description:
+      - Instructs the module to explicitly lock the datastore specified as C(target). By setting the option
+        value I(always) is will explicitly lock the datastore mentioned in C(target) option. It the value
+        is I(never) it will not lock the C(target) datastore. The value I(if-supported) lock the C(target)
+        datastore only if it is supported by the remote Netconf server.
+    default: always
+    choices: ['never', 'always', 'if-supported']
+    version_added: "2.7"
+  default_operation:
+    description:
+    - The default operation for <edit-config> rpc, valid values are I(merge), I(replace) and I(none).
+      If the default value is merge, the configuration data in the C(content) option is merged at the
+      corresponding level in the C(target) datastore. If the value is replace the data in the C(content)
+      option completely replaces the configuration in the C(target) datastore. If the value is none the C(target)
+      datastore is unaffected by the configuration in the config option, unless and until the incoming configuration
+      data uses the C(operation) operation to request a different operation.
+    default: merge
+    choices: ['merge', 'replace', 'none']
+    version_added: "2.7"
+  confirm:
+    description:
+    - This argument will configure a timeout value for the commit to be confirmed before it is automatically
+      rolled back. If the C(confirm_commit) argument is set to False, this argument is silently ignored. If the
+      value of this argument is set to 0, the commit is confirmed immediately. The remote host should
+      support :candidate and :confirmed-commit capability for this option to .
+    default: 0
+    version_added: "2.7"
+  confirm_commit:
+    description:
+    - This argument will execute commit operation on remote device. It can be used to confirm a previous commit.
+    type: bool
+    default: 'no'
+    version_added: "2.7"
+  error_option:
+    description:
+    - This option control the netconf server action after a error is occured while editing the configuration.
+      If the value is I(stop-on-error) abort the config edit on first error, if value is I(continue-on-error)
+      it continues to process configuration data on erro, error is recorded and negative response is generated
+      if any errors occur. If value is C(rollback-on-error) it rollback to the original configuration in case
+      any error occurs, this requires the remote Netconf server to support the :rollback-on-error capability.
+    default: stop-on-error
+    choices: ['stop-on-error', 'continue-on-error', 'rollback-on-error']
+    version_added: "2.7"
   save:
     description:
-      - The C(save) argument instructs the module to save the running-
+      - The C(save) argument instructs the module to save the running-config to the startup-config if changed.
-        config to the startup-config if changed.
-    required: false
     default: false
     version_added: "2.4"
-  username:
+  backup:
     description:
-     - the username to authenticate with
+      - This argument will cause the module to create a full backup of
-    required: true
+        the current C(running-config) from the remote device before any
-  password:
+        changes are made.  The backup file is written to the C(backup)
+        folder in the playbook root directory or role root directory, if
+        playbook is part of an ansible role. If the directory does not exist,
+        it is created.
+    type: bool
+    default: 'no'
+    version_added: "2.7"
+  delete:
     description:
-     - password of the user to authenticate with
+      - It instructs the module to delete the configuration from value mentioned in C(target) datastore.
-    required: true
+    type: bool
-  xml:
+    default: 'no'
+    version_added: "2.7"
+  commit:
     description:
-     - the XML content to send to the device
+      - This boolean flag controls if the configuration changes should be committed or not after editing the
-    required: false
+        candidate datastore. This oprion is supported only if remote Netconf server supports :candidate
+        capability. If the value is set to I(False) commit won't be issued after edit-config operation
+        and user needs to handle commit or discard-changes explicitly.
+    type: bool
+    default: True
+    version_added: "2.7"
+  validate:
+    description:
+      - This boolean flag if set validates the content of datastore given in C(target) option.
+        For this option to work remote Netconf server shoule support :validate capability.
+    type: bool
+    default: False
+    version_added: "2.7"
   src:
     description:
-      - Specifies the source path to the xml file that contains the configuration
+    - Specifies the source path to the xml file that contains the configuration or configuration template
-        or configuration template to load.  The path to the source file can
+      to load. The path to the source file can either be the full path on the Ansible control host or
-        either be the full path on the Ansible control host or a relative
+      a relative path from the playbook or role root directory. This argument is mutually exclusive with I(xml).
-        path from the playbook or role root directory.  This argument is mutually
-        exclusive with I(xml).
-    required: false
     version_added: "2.4"
-
-
 requirements:
-  - "python >= 2.6"
   - "ncclient"
+notes:
+  - This module requires the netconf system service be enabled on
+    the remote device being managed.
+  - This module supports devices with and without the candidate and
+    confirmed-commit capabilities. It will always use the safer feature.
+  - This module supports the use of connection=netconf
 '''
 
 EXAMPLES = '''
 - name: use lookup filter to provide xml configuration
   netconf_config:
-    xml: "{{ lookup('file', './config.xml') }}"
+    content: "{{ lookup('file', './config.xml') }}"
-    host: 10.0.0.1
-    username: admin
-    password: admin
 
 - name: set ntp server in the device
   netconf_config:
-    host: 10.0.0.1
+    content: |
-    username: admin
-    password: admin
-    xml: |
         <config xmlns:xc="urn:ietf:params:xml:ns:netconf:base:1.0">
             <system xmlns="urn:ietf:params:xml:ns:yang:ietf-system">
                 <ntp>
@@ -129,10 +174,7 @@ EXAMPLES = '''
 
 - name: wipe ntp configuration
   netconf_config:
-    host: 10.0.0.1
+    content: |
-    username: admin
-    password: admin
-    xml: |
         <config xmlns:xc="urn:ietf:params:xml:ns:netconf:base:1.0">
             <system xmlns="urn:ietf:params:xml:ns:yang:ietf-system">
                 <ntp>
@@ -144,6 +186,12 @@ EXAMPLES = '''
             </system>
         </config>
 
+- name: configure interface while providing different private key file path (for connection=netconf)
+  netconf_config:
+    backup: yes
+  register: backup_junos_location
+  vars:
+    ansible_private_key_file: /home/admin/.ssh/newprivatekeyfile
 '''
 
 RETURN = '''
@@ -152,191 +200,189 @@ server_capabilities:
     returned: success
     type: list
     sample: ['urn:ietf:params:netconf:base:1.1','urn:ietf:params:netconf:capability:confirmed-commit:1.0','urn:ietf:params:netconf:capability:candidate:1.0']
-
+backup_path:
+  description: The full path to the backup file
+  returned: when backup is yes
+  type: string
+  sample: /playbooks/ansible/backup/config.2016-07-16@22:28:34
+diff:
+  description: If --diff option in enabled while running, the before and after configration change are
+               returned as part of before and after key.
+  returned: when diff is enabled
+  type: string
+  sample: /playbooks/ansible/backup/config.2016-07-16@22:28:34
 '''
 
-import traceback
+from ansible.module_utils._text import to_text
-import xml.dom.minidom
+from ansible.module_utils.basic import AnsibleModule, env_fallback
-
-from xml.etree.ElementTree import fromstring, tostring
-
-try:
-    import ncclient.manager
-    HAS_NCCLIENT = True
-except ImportError:
-    HAS_NCCLIENT = False
-
-from ansible.module_utils.basic import AnsibleModule
-from ansible.module_utils._text import to_native
 from ansible.module_utils.connection import Connection, ConnectionError
-
+from ansible.module_utils.network.netconf.netconf import get_capabilities, get_config, sanitize_xml
-
-def netconf_edit_config(m, xml, commit, retkwargs, datastore, capabilities, local_connection):
-    m.lock(target=datastore)
-    try:
-        if datastore == "candidate":
-            m.discard_changes()
-
-        config_before = m.get_config(source=datastore)
-        m.edit_config(target=datastore, config=xml)
-        config_after = m.get_config(source=datastore)
-
-        if local_connection:
-            changed = config_before.data_xml != config_after.data_xml
-        else:
-            changed = config_before != config_after
-
-        if changed and commit and datastore == "candidate":
-            if ":confirmed-commit" in capabilities:
-                m.commit(confirmed=True)
-                m.commit()
-            else:
-                m.commit()
-
-        return changed
-    finally:
-        m.unlock(target=datastore)
-
-
-# ------------------------------------------------------------------- #
-# Main
 
 
 def main():
-
+    """ main entry point for module execution
-    module = AnsibleModule(
+    """
-        argument_spec=dict(
+    argument_spec = dict(
-            xml=dict(type='str', required=False),
+        content=dict(aliases=['xml']),
-            src=dict(type='path', required=False),
+        target=dict(choices=['auto', 'candidate', 'running'], default='auto', aliases=['datastore']),
-
+        source_datastore=dict(aliases=['source']),
-            datastore=dict(choices=['auto', 'candidate', 'running'], default='auto'),
+        format=dict(choices=['xml', 'text'], default='xml'),
-            save=dict(type='bool', default=False),
+        lock=dict(choices=['never', 'always', 'if-supported'], default='always'),
-
+        default_operation=dict(choices=['merge', 'replace', 'none'], default='merge'),
-            # connection arguments
+        confirm=dict(type='int', default=0),
-            host=dict(type='str'),
+        confirm_commit=dict(type='bool', default=False),
-            port=dict(type='int', default=830),
+        error_option=dict(choices=['stop-on-error', 'continue-on-error', 'rollback-on-error'], default='stop-on-error'),
-
+        backup=dict(type='bool', default=False),
-            username=dict(type='str', no_log=True),
+        save=dict(type='bool', default=False),
-            password=dict(type='str', no_log=True),
+        delete=dict(type='bool', default=False),
-
+        commit=dict(type='bool', default=True),
-            hostkey_verify=dict(type='bool', default=True),
+        validate=dict(type='bool', default=False),
-            look_for_keys=dict(type='bool', default=True),
-
-            allow_agent=dict(type='bool', default=True),
-        ),
-        mutually_exclusive=[('xml', 'src')]
     )
 
-    if not module._socket_path and not HAS_NCCLIENT:
+    # deprecated options
-        module.fail_json(msg='could not import the python library '
+    netconf_top_spec = {
-                         'ncclient required by this module')
+        'src': dict(type='path', removed_in_version=2.11),
+        'host': dict(removed_in_version=2.11),
+        'port': dict(removed_in_version=2.11, type='int', default=830),
+        'username': dict(fallback=(env_fallback, ['ANSIBLE_NET_USERNAME']), removed_in_version=2.11, no_log=True),
+        'password': dict(fallback=(env_fallback, ['ANSIBLE_NET_PASSWORD']), removed_in_version=2.11, no_log=True),
+        'ssh_keyfile': dict(fallback=(env_fallback, ['ANSIBLE_NET_SSH_KEYFILE']), removed_in_version=2.11, type='path'),
+        'hostkey_verify': dict(removed_in_version=2.11, type='bool', default=True),
+        'look_for_keys': dict(removed_in_version=2.11, type='bool', default=True),
+        'timeout': dict(removed_in_version=2.11, type='int', default=10),
+    }
+    argument_spec.update(netconf_top_spec)
 
-    if (module.params['src']):
+    mutually_exclusive = [('content', 'src', 'source', 'delete', 'confirm_commit')]
-        config_xml = str(module.params['src'])
+    required_one_of = [('content', 'src', 'source', 'delete', 'confirm_commit')]
-    elif module.params['xml']:
-        config_xml = str(module.params['xml'])
-    else:
-        module.fail_json(msg='Option src or xml must be provided')
 
-    local_connection = module._socket_path is None
+    module = AnsibleModule(argument_spec=argument_spec,
+                           required_one_of=required_one_of,
+                           mutually_exclusive=mutually_exclusive,
+                           supports_check_mode=True)
 
-    if not local_connection:
+    if module.params['src']:
-        m = Connection(module._socket_path)
+        module.deprecate(msg="argument 'src' has been deprecated. Use file lookup plugin instead to read file contents.",
-        capabilities = module.from_json(m.get_capabilities())
+                         version="4 releases from v2.7")
-        server_capabilities = capabilities.get('server_capabilities')
 
-    else:
+    config = module.params['content'] or module.params['src']
-        nckwargs = dict(
+    target = module.params['target']
-            host=module.params['host'],
+    lock = module.params['lock']
-            port=module.params['port'],
+    source = module.params['source']
-            hostkey_verify=module.params['hostkey_verify'],
+    delete = module.params['delete']
-            allow_agent=module.params['allow_agent'],
+    confirm_commit = module.params['confirm_commit']
-            look_for_keys=module.params['look_for_keys'],
+    confirm = module.params['confirm']
-            username=module.params['username'],
+    validate = module.params['validate']
-            password=module.params['password'],
-        )
 
-        try:
+    conn = Connection(module._socket_path)
-            m = ncclient.manager.connect(**nckwargs)
+    capabilities = get_capabilities(module)
-            server_capabilities = list(m.server_capabilities)
+    operations = capabilities['device_operations']
-        except ncclient.transport.errors.AuthenticationError:
-            module.fail_json(
-                msg='authentication failed while connecting to device'
-            )
-        except Exception as e:
-            module.fail_json(msg='error connecting to the device: %s' % to_native(e), exception=traceback.format_exc())
 
-        try:
+    supports_commit = operations.get('supports_commit', False)
-            xml.dom.minidom.parseString(config_xml)
+    supports_writable_running = operations.get('supports_writable_running', False)
-        except Exception as e:
+    supports_startup = operations.get('supports_startup', False)
-            module.fail_json(msg='error parsing XML: %s' % to_native(e), exception=traceback.format_exc())
 
-    retkwargs = dict()
+    # identify target datastore
-    retkwargs['server_capabilities'] = server_capabilities
+    if target == 'candidate' and not supports_commit:
-
+        module.fail_json(msg=':candidate is not supported by this netconf server')
-    server_capabilities = '\n'.join(server_capabilities)
+    elif target == 'running' and not supports_writable_running:
-
+        module.fail_json(msg=':writable-running is not supported by this netconf server')
-    if module.params['datastore'] == 'candidate':
+    elif target == 'auto':
-        if ':candidate' in server_capabilities:
+        if supports_commit:
-            datastore = 'candidate'
+            target = 'candidate'
+        elif supports_writable_running:
+            target = 'running'
         else:
-            if local_connection:
+            module.fail_json(msg='neither :candidate nor :writable-running are supported by this netconf server')
-                m.close_session()
+
-            module.fail_json(
+    # Netconf server capability validation against input options
-                msg=':candidate is not supported by this netconf server'
+    if module.params['save'] and not supports_startup:
-            )
+        module.fail_json(msg='cannot copy <running/> to <startup/>, while :startup is not supported')
-    elif module.params['datastore'] == 'running':
+
-        if ':writable-running' in server_capabilities:
+    if module.params['confirm_commit'] and not operations.get('supports_confirm_commit', False):
-            datastore = 'running'
+        module.fail_json(msg='confirm commit is not supported by Netconf server')
-        else:
+
-            if local_connection:
+    if confirm_commit or (confirm > 0) and not operations.get('supports_confirm_commit', False):
-                m.close_session()
+        module.fail_json(msg='confirm commit is not supported by this netconf server')
-            module.fail_json(
+
-                msg=':writable-running is not supported by this netconf server'
+    if validate and not operations.get('supports_validate', False):
-            )
+        module.fail_json(msg='validate is not supported by this netconf server')
-    elif module.params['datastore'] == 'auto':
+
-        if ':candidate' in server_capabilities:
+    if lock == 'never':
-            datastore = 'candidate'
+        execute_lock = False
-        elif ':writable-running' in server_capabilities:
+    elif target in operations.get('lock_datastore', []):
-            datastore = 'running'
+        # lock is requested (always/if-support) and supported => lets do it
-        else:
+        execute_lock = True
-            if local_connection:
-                m.close_session()
-            module.fail_json(
-                msg='neither :candidate nor :writable-running are supported by this netconf server'
-            )
     else:
-        if local_connection:
+        # lock is requested (always/if-supported) but not supported => issue warning
-            m.close_session()
+        module.warn("lock operation on '%s' source is not supported on this device" % target)
-        module.fail_json(
+        execute_lock = (lock == 'always')
-            msg=module.params['datastore'] + ' datastore is not supported by this ansible module'
-        )
-
-    if module.params['save']:
-        if ':startup' not in server_capabilities:
-            module.fail_json(
-                msg='cannot copy <running/> to <startup/>, while :startup is not supported'
-            )
 
+    result = {'changed': False, 'server_capabilities': capabilities.get('server_capabilities', [])}
+    before = None
+    locked = False
     try:
-        changed = netconf_edit_config(
+        if module.params['backup']:
-            m=m,
+            response = get_config(module, target, lock=execute_lock)
-            xml=config_xml,
+            before = to_text(response, errors='surrogate_then_replace').strip()
-            commit=True,
+            result['__backup__'] = before.strip()
-            retkwargs=retkwargs,
+        if validate:
-            datastore=datastore,
+            if not module.check_mode:
-            capabilities=server_capabilities,
+                conn.validate(target)
-            local_connection=local_connection
+        if source:
-        )
+            if not module.check_mode:
-        if changed and module.params['save']:
+                conn.copy(source, target)
-            m.copy_config(source="running", target="startup")
+            result['changed'] = True
-    except Exception as e:
+        elif delete:
-        module.fail_json(msg='error editing configuration: %s' % to_native(e), exception=traceback.format_exc())
+            if not module.check_mode:
-    finally:
+                conn.delete(target)
-        if local_connection:
+            result['changed'] = True
-            m.close_session()
+        elif confirm_commit:
+            if not module.check_mode:
+                conn.commit()
+            result['changed'] = True
+        else:
+            if module.check_mode and not supports_commit:
+                module.warn("check mode not supported as Netconf server doesn't support candidate capability")
+                result['changed'] = True
+                module.exit_json(**result)
 
-    module.exit_json(changed=changed, **retkwargs)
+            if lock:
+                conn.lock(target=target)
+                locked = True
+            if before is None:
+                before = to_text(conn.get_config(source=target), errors='surrogate_then_replace').strip()
+
+            kwargs = {
+                'target': target,
+                'default_operation': module.params['default_operation'],
+                'error_option': module.params['error_option'],
+                'format': module.params['format'],
+            }
+            conn.edit_config(config, **kwargs)
+            if supports_commit and module.params['commit']:
+                if not module.check_mode:
+                    timeout = confirm if confirm > 0 else None
+                    conn.commit(confirmed=confirm_commit, timeout=timeout)
+                else:
+                    conn.discard_changes()
+
+            after = to_text(conn.get_config(source='running'), errors='surrogate_then_replace').strip()
+
+            if sanitize_xml(before) != sanitize_xml(after):
+                result['changed'] = True
+
+            if module._diff:
+                if result['changed']:
+                    result['diff'] = {'before': before, 'after': after}
+
+    except ConnectionError as e:
+        module.fail_json(msg=to_text(e, errors='surrogate_then_replace').strip())
+    finally:
+        if locked:
+            conn.unlock(target=target)
+
+    module.exit_json(**result)
 
 
 if __name__ == '__main__':

lib/ansible/plugins/action/netconf.py

@@ -0,0 +1,74 @@
+#
+# Copyright 2018 Red Hat Inc.
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+#
+from __future__ import (absolute_import, division, print_function)
+__metaclass__ = type
+
+import copy
+import sys
+
+from ansible.plugins.action.normal import ActionModule as _ActionModule
+
+try:
+    from __main__ import display
+except ImportError:
+    from ansible.utils.display import Display
+    display = Display()
+
+
+class ActionModule(_ActionModule):
+
+    def run(self, tmp=None, task_vars=None):
+        del tmp  # tmp no longer has any effect
+
+        if self._play_context.connection not in ['netconf', 'local'] and self._task.action == 'netconf_config':
+            return {'failed': True, 'msg': 'Connection type %s is not valid for netconf_config module. '
+                                           'Valid connection type is netconf or local (deprecated)' % self._play_context.connection}
+        elif self._play_context.connection not in ['netconf'] and self._task.action != 'netconf_config':
+            return {'failed': True, 'msg': 'Connection type %s is not valid for %s module. '
+                                           'Valid connection type is netconf.' % (self._play_context.connection, self._task.action)}
+
+        if self._play_context.connection == 'local' and self._task.action == 'netconf_config':
+            args = self._task.args
+            pc = copy.deepcopy(self._play_context)
+            pc.connection = 'netconf'
+            pc.port = int(args.get('port') or self._play_context.port or 830)
+
+            pc.remote_user = args.get('username') or self._play_context.connection_user
+            pc.password = args.get('password') or self._play_context.password
+            pc.private_key_file = args.get('ssh_keyfile') or self._play_context.private_key_file
+
+            display.vvv('using connection plugin %s (was local)' % pc.connection, pc.remote_addr)
+            connection = self._shared_loader_obj.connection_loader.get('persistent', pc, sys.stdin)
+
+            timeout = args.get('timeout')
+            command_timeout = int(timeout) if timeout else connection.get_option('persistent_command_timeout')
+            connection.set_options(direct={'persistent_command_timeout': command_timeout, 'look_for_keys': args.get('look_for_keys'),
+                                           'hostkey_verify': args.get('hostkey_verify'),
+                                           'allow_agent': args.get('allow_agent')})
+
+            socket_path = connection.run()
+            display.vvvv('socket_path: %s' % socket_path, pc.remote_addr)
+            if not socket_path:
+                return {'failed': True,
+                        'msg': 'unable to open shell. Please see: ' +
+                               'https://docs.ansible.com/ansible/network_debug_troubleshooting.html#unable-to-open-shell'}
+
+            task_vars['ansible_socket'] = socket_path
+
+        return super(ActionModule, self).run(task_vars=task_vars)

lib/ansible/plugins/action/netconf_config.py

@@ -19,9 +19,94 @@
 from __future__ import (absolute_import, division, print_function)
 __metaclass__ = type
 
-from ansible.plugins.action import ActionBase
+import os
-from ansible.plugins.action.net_config import ActionModule as NetActionModule
+import re
+import time
+import glob
+
+from ansible.plugins.action.netconf import ActionModule as _ActionModule
+from ansible.module_utils._text import to_text, to_bytes
+from ansible.module_utils.six.moves.urllib.parse import urlsplit
+
+PRIVATE_KEYS_RE = re.compile('__.+__')
 
 
-class ActionModule(NetActionModule, ActionBase):
+class ActionModule(_ActionModule):
-    pass
+
+    def run(self, tmp=None, task_vars=None):
+        if self._task.args.get('src'):
+            try:
+                self._handle_template()
+            except ValueError as exc:
+                return dict(failed=True, msg=to_text(exc))
+
+        result = super(ActionModule, self).run(tmp, task_vars)
+        del tmp  # tmp no longer has any effect
+
+        if self._task.args.get('backup') and result.get('__backup__'):
+            # User requested backup and no error occurred in module.
+            # NOTE: If there is a parameter error, _backup key may not be in results.
+            filepath = self._write_backup(task_vars['inventory_hostname'],
+                                          result['__backup__'])
+
+            result['backup_path'] = filepath
+
+        # strip out any keys that have two leading and two trailing
+        # underscore characters
+        for key in list(result):
+            if PRIVATE_KEYS_RE.match(key):
+                del result[key]
+
+        return result
+
+    def _get_working_path(self):
+        cwd = self._loader.get_basedir()
+        if self._task._role is not None:
+            cwd = self._task._role._role_path
+        return cwd
+
+    def _write_backup(self, host, contents):
+        backup_path = self._get_working_path() + '/backup'
+        if not os.path.exists(backup_path):
+            os.mkdir(backup_path)
+        for fn in glob.glob('%s/%s*' % (backup_path, host)):
+            os.remove(fn)
+        tstamp = time.strftime("%Y-%m-%d@%H:%M:%S", time.localtime(time.time()))
+        filename = '%s/%s_config.%s' % (backup_path, host, tstamp)
+        with open(filename, 'wb') as f:
+            f.write(to_bytes(to_text(contents, encoding='latin-1'), encoding='utf-8'))
+        return filename
+
+    def _handle_template(self):
+        src = self._task.args.get('src')
+        working_path = self._get_working_path()
+
+        if os.path.isabs(src) or urlsplit('src').scheme:
+            source = src
+        else:
+            source = self._loader.path_dwim_relative(working_path, 'templates', src)
+            if not source:
+                source = self._loader.path_dwim_relative(working_path, src)
+
+        if not os.path.exists(source):
+            raise ValueError('path specified in src not found')
+
+        try:
+            with open(source, 'r') as f:
+                template_data = to_text(f.read())
+        except IOError:
+            return dict(failed=True, msg='unable to load src file')
+
+        # Create a template search path in the following order:
+        # [working_path, self_role_path, dependent_role_paths, dirname(source)]
+        searchpath = [working_path]
+        if self._task._role is not None:
+            searchpath.append(self._task._role._role_path)
+            if hasattr(self._task, "_block:"):
+                dep_chain = self._task._block.get_dep_chain()
+                if dep_chain is not None:
+                    for role in dep_chain:
+                        searchpath.append(role._role_path)
+        searchpath.append(os.path.dirname(source))
+        self._templar.environment.loader.searchpath = searchpath
+        self._task.args['src'] = self._templar.template(template_data)

lib/ansible/plugins/connection/netconf.py

@@ -102,7 +102,8 @@ options:
       - name: ANSIBLE_HOST_KEY_AUTO_ADD
   look_for_keys:
     default: True
-    description: 'TODO: write it'
+    description:
+      -  enables looking for ssh keys in the usual locations for ssh keys (e.g. :file:`~/.ssh/id_*`)
     env:
       - name: ANSIBLE_PARAMIKO_LOOK_FOR_KEYS
     ini:
@@ -218,6 +219,7 @@ class Connection(NetworkConnectionBase):
         display.display('network_os is set to %s' % self._network_os, log_only=True)
 
         self._manager = None
+        self.key_filename = None
 
     def exec_command(self, cmd, in_data=None, sudoable=True):
         """Sends the request to the node and returns the reply
@@ -252,9 +254,9 @@ class Connection(NetworkConnectionBase):
             allow_agent = False
         setattr(self._play_context, 'allow_agent', allow_agent)
 
-        key_filename = None
+        self.key_filename = self._play_context.private_key_file or self.get_option('private_key_file')
-        if self._play_context.private_key_file:
+        if self.key_filename:
-            key_filename = os.path.expanduser(self._play_context.private_key_file)
+            self.key_filename = os.path.expanduser(self.key_filename)
 
         if self._network_os == 'default':
             for cls in netconf_loader.all(class_only=True):
@@ -277,7 +279,7 @@ class Connection(NetworkConnectionBase):
                 port=self._play_context.port or 830,
                 username=self._play_context.remote_user,
                 password=self._play_context.password,
-                key_filename=str(key_filename),
+                key_filename=str(self.key_filename),
                 hostkey_verify=self.get_option('host_key_checking'),
                 look_for_keys=self.get_option('look_for_keys'),
                 device_params=device_params,

lib/ansible/plugins/netconf/__init__.py

@@ -111,8 +111,6 @@ class NetconfBase(AnsiblePlugin):
         :param name: Name of rpc in string format
         :return: Received rpc response from remote host
         """
-        """RPC to be execute on remote device
-           :name: Name of rpc in string format"""
         try:
             obj = to_ele(name)
             resp = self.m.rpc(obj)
@@ -275,13 +273,19 @@ class NetconfBase(AnsiblePlugin):
         return resp.data_xml if hasattr(resp, 'data_xml') else resp.xml
 
     @ensure_connected
-    def locked(self, target):
+    def delete_config(self, target):
         """
-        Returns a context manager for a lock on a datastore
+        delete a configuration datastore
-        :param target: Name of the configuration datastore to lock
+        :param target: specifies the  name or URL of configuration datastore to delete
-        :return: Locked context object
+        :return: Returns xml string containing the RPC response received from remote host
         """
-        return self.m.locked(target)
+        resp = self.m.delete_config(target)
+        return resp.data_xml if hasattr(resp, 'data_xml') else resp.xml
+
+    @ensure_connected
+    def locked(self, *args, **kwargs):
+        resp = self.m.locked(*args, **kwargs)
+        return resp.data_xml if hasattr(resp, 'data_xml') else resp.xml
 
     @abstractmethod
     def get_capabilities(self):
@@ -341,6 +345,7 @@ class NetconfBase(AnsiblePlugin):
         operations['supports_startup'] = ':startup' in capabilities
         operations['supports_xpath'] = ':xpath' in capabilities
         operations['supports_writable_running'] = ':writable-running' in capabilities
+        operations['supports_validate'] = ':writable-validate' in capabilities
 
         operations['lock_datastore'] = []
         if operations['supports_writable_running']:

lib/ansible/plugins/netconf/ce.py

@@ -109,9 +109,9 @@ class Netconf(NetconfBase):
                 port=obj._play_context.port or 830,
                 username=obj._play_context.remote_user,
                 password=obj._play_context.password,
-                key_filename=obj._play_context.private_key_file,
+                key_filename=obj.key_filename,
-                hostkey_verify=C.HOST_KEY_CHECKING,
+                hostkey_verify=obj.get_option('host_key_checking'),
-                look_for_keys=C.PARAMIKO_LOOK_FOR_KEYS,
+                look_for_keys=obj.get_option('look_for_keys'),
                 allow_agent=obj._play_context.allow_agent,
                 timeout=obj._play_context.timeout
             )

lib/ansible/plugins/netconf/iosxr.py

@@ -104,9 +104,9 @@ class Netconf(NetconfBase):
                 port=obj._play_context.port or 830,
                 username=obj._play_context.remote_user,
                 password=obj._play_context.password,
-                key_filename=obj._play_context.private_key_file,
+                key_filename=obj.key_filename,
-                hostkey_verify=C.HOST_KEY_CHECKING,
+                hostkey_verify=obj.get_option('host_key_checking'),
-                look_for_keys=C.PARAMIKO_LOOK_FOR_KEYS,
+                look_for_keys=obj.get_option('look_for_keys'),
                 allow_agent=obj._play_context.allow_agent,
                 timeout=obj._play_context.timeout
             )

lib/ansible/plugins/netconf/junos.py

@@ -113,9 +113,9 @@ class Netconf(NetconfBase):
                 port=obj._play_context.port or 830,
                 username=obj._play_context.remote_user,
                 password=obj._play_context.password,
-                key_filename=obj._play_context.private_key_file,
+                key_filename=obj.key_filename,
-                hostkey_verify=C.HOST_KEY_CHECKING,
+                hostkey_verify=obj.get_option('host_key_checking'),
-                look_for_keys=C.PARAMIKO_LOOK_FOR_KEYS,
+                look_for_keys=obj.get_option('look_for_keys'),
                 allow_agent=obj._play_context.allow_agent,
                 timeout=obj._play_context.timeout
             )

lib/ansible/plugins/netconf/sros.py

@@ -82,9 +82,9 @@ class Netconf(NetconfBase):
                 port=obj._play_context.port or 830,
                 username=obj._play_context.remote_user,
                 password=obj._play_context.password,
-                key_filename=obj._play_context.private_key_file,
+                key_filename=obj.key_filename,
-                hostkey_verify=C.HOST_KEY_CHECKING,
+                hostkey_verify=obj.get_option('host_key_checking'),
-                look_for_keys=C.PARAMIKO_LOOK_FOR_KEYS,
+                look_for_keys=obj.get_option('look_for_keys'),
                 allow_agent=obj._play_context.allow_agent,
                 timeout=obj._play_context.timeout
             )

lib/ansible/utils/module_docs_fragments/netconf.py

@@ -0,0 +1,77 @@
+#
+#
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+
+class ModuleDocFragment(object):
+
+    # Standard files documentation fragment
+    DOCUMENTATION = """
+options:
+  host:
+    description:
+      - Specifies the DNS host name or address for connecting to the remote
+        device over the specified transport.  The value of host is used as
+        the destination address for the transport.
+    required: true
+  port:
+    description:
+      - Specifies the port to use when building the connection to the remote
+        device.  The port value will default to port 830.
+    type: int
+    default: 830
+  username:
+    description:
+      - Configures the username to use to authenticate the connection to
+        the remote device.  This value is used to authenticate
+        the SSH session. If the value is not specified in the task, the
+        value of environment variable C(ANSIBLE_NET_USERNAME) will be used instead.
+  password:
+    description:
+      - Specifies the password to use to authenticate the connection to
+        the remote device.   This value is used to authenticate
+        the SSH session. If the value is not specified in the task, the
+        value of environment variable C(ANSIBLE_NET_PASSWORD) will be used instead.
+  timeout:
+    description:
+      - Specifies the timeout in seconds for communicating with the network device
+        for either connecting or sending commands.  If the timeout is
+        exceeded before the operation is completed, the module will error.
+    type: int
+    default: 10
+  ssh_keyfile:
+    description:
+      - Specifies the SSH key to use to authenticate the connection to
+        the remote device.   This value is the path to the key
+        used to authenticate the SSH session. If the value is not specified in
+        the task, the value of environment variable C(ANSIBLE_NET_SSH_KEYFILE)
+        will be used instead.
+    type: path
+  hostkey_verify:
+    description:
+      - If set to true, the ssh host key of the device must match a ssh key present on
+        the host if false, the ssh host key of the device is not checked.
+    type: bool
+    default: True
+  look_for_keys:
+    description:
+      - Enables looking in the usual locations for the ssh keys (e.g. :file:`~/.ssh/id_*`)
+    type: bool
+    default: True
+notes:
+  - For information on using netconf see the :ref:`Platform Options guide using Netconf<netconf_platform_enabled_options>`
+  - For more information on using Ansible to manage network devices see the :ref:`Ansible Network Guide <network_guide>`
+"""

test/integration/targets/netconf_config/defaults/main.yaml

@@ -0,0 +1,2 @@
+---
+testcase: "*"

test/integration/targets/netconf_config/meta/main.yml

@@ -0,0 +1,4 @@
+---
+dependencies:
+  - { role: prepare_junos_tests, when: ansible_network_os == 'junos' }
+  - { role: prepare_iosxr_tests, when: ansible_network_os == 'iosxr' }

test/integration/targets/netconf_config/tasks/iosxr.yaml

@@ -0,0 +1,16 @@
+---
+- name: collect all netconf test cases
+  find:
+    paths: "{{ role_path }}/tests/iosxr"
+    patterns: "{{ testcase }}.yaml"
+  register: test_cases
+  connection: local
+
+- name: set test_items
+  set_fact: test_items="{{ test_cases.files | map(attribute='path') | list }}"
+
+- name: run test case (connection=netconf)
+  include: "{{ test_case_to_run }}"
+  with_items: "{{ test_items }}"
+  loop_control:
+    loop_var: test_case_to_run

test/integration/targets/netconf_config/tasks/junos.yaml

@@ -0,0 +1,16 @@
+---
+- name: collect all netconf test cases
+  find:
+    paths: "{{ role_path }}/tests/junos"
+    patterns: "{{ testcase }}.yaml"
+  register: test_cases
+  connection: local
+
+- name: set test_items
+  set_fact: test_items="{{ test_cases.files | map(attribute='path') | list }}"
+
+- name: run test case (connection=netconf)
+  include: "{{ test_case_to_run }} ansible_connection=netconf"
+  with_items: "{{ test_items }}"
+  loop_control:
+    loop_var: test_case_to_run

test/integration/targets/netconf_config/tasks/main.yaml

@@ -0,0 +1,3 @@
+---
+- { include: junos.yaml, when: ansible_network_os == 'junos', tags: ['netconf'] }
+- { include: iosxr.yaml, when: ansible_network_os == 'iosxr', tags: ['netconf'] }

test/integration/targets/netconf_config/tests/junos/basic.yaml

@@ -0,0 +1,55 @@
+---
+- debug: msg="START netconf_config junos/basic.yaml on connection={{ ansible_connection }}"
+
+- include_vars: "{{playbook_dir }}/targets/netconf_config/tests/junos/fixtures/config.yml"
+
+- name: syslog file config- setup
+  junos_config:
+    lines:
+    - delete system syslog file test_netconf_config
+
+- name: configure syslog file
+  netconf_config:
+    content: "{{ syslog_config }}"
+  register: result
+
+- assert:
+    that:
+      - "result.changed == true"
+      - "'<name>test_netconf_config</name>' in result.diff.after"
+
+- name: configure syslog file (idempotent)
+  netconf_config:
+    content: "{{ syslog_config }}"
+  register: result
+
+- assert:
+    that:
+      - "result.changed == false"
+
+- name: configure syslog file replace
+  netconf_config:
+    content: "{{ syslog_config_replace }}"
+    default_operation: 'replace'
+  register: result
+
+- assert:
+    that:
+      - "result.changed == true"
+
+- name: test backup
+  netconf_config:
+    content: "{{ syslog_config }}"
+    backup: True
+  register: result
+
+- assert:
+    that:
+      - "'backup_path' in result"
+
+- name: syslog file config- teardown
+  junos_config:
+    lines:
+    - delete system syslog file test_netconf_config
+
+- debug: msg="END netconf_config junos/basic.yaml on connection={{ ansible_connection }}"

test/integration/targets/netconf_config/tests/junos/fixtures/config.yml

@@ -0,0 +1,38 @@
+---
+syslog_config: |
+            <config xmlns:xc="urn:ietf:params:xml:ns:netconf:base:1.0">
+              <configuration>
+                <system>
+                  <syslog>
+                    <file>
+                      <name>test_netconf_config</name>
+                      <contents>
+                        <name>any</name>
+                        <any/>
+                      </contents>
+                      <contents>
+                        <name>kernel</name>
+                        <critical/>
+                      </contents>
+                    </file>
+                  </syslog>
+                </system>
+              </configuration>
+            </config>
+
+syslog_config_replace: |
+            <config xmlns:xc="urn:ietf:params:xml:ns:netconf:base:1.0">
+              <configuration>
+                <system>
+                  <syslog>
+                    <file>
+                      <name>test_netconf_config</name>
+                      <contents>
+                        <name>any</name>
+                        <any/>
+                      </contents>
+                    </file>
+                  </syslog>
+                </system>
+              </configuration>
+            </config>