cmueller/ansible
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

get_url: PEP8, imports, cosmetics (#24676)

Browse source 
- Make PEP8 compliant
- Ensure imports are specific
- Few cosmetic changes (sort lists, casing, punctuation)
This commit is contained in:
Dag Wieers 2017-05-30 19:24:07 +02:00 committed by John R Barker
parent 630185cb20
commit b3b11fbce2
2 changed files with 88 additions and 103 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

190
lib/ansible/modules/net_tools/basics/get_url.py
View file

@ -24,8 +24,7 @@ ANSIBLE_METADATA = {'metadata_version': '1.0',
'status': ['stableinterface'], 'status': ['stableinterface'],
'supported_by': 'core'} 'supported_by': 'core'}
DOCUMENTATION = r'''
DOCUMENTATION = '''
--- ---
module: get_url module: get_url
short_description: Downloads files from HTTP, HTTPS, or FTP to node short_description: Downloads files from HTTP, HTTPS, or FTP to node
@ -42,7 +41,7 @@ description:
your proxy environment for both protocols is correct. your proxy environment for both protocols is correct.
- From Ansible 2.4 when run with C(--check), it will do a HEAD request to validate the URL but - From Ansible 2.4 when run with C(--check), it will do a HEAD request to validate the URL but
will not download the entire file or verify it against hashes. will not download the entire file or verify it against hashes.
version_added: "0.6" version_added: '0.6'
options: options:
url: url:
description: description:
@ -50,103 +49,91 @@ options:
required: true required: true
dest: dest:
description: description:
- absolute path of where to download the file to. - Absolute path of where to download the file to.
- If C(dest) is a directory, either the server provided filename or, if - If C(dest) is a directory, either the server provided filename or, if
none provided, the base name of the URL on the remote server will be none provided, the base name of the URL on the remote server will be
used. If a directory, C(force) has no effect. used. If a directory, C(force) has no effect.
If C(dest) is a directory, the file will always be - If C(dest) is a directory, the file will always be downloaded
downloaded (regardless of the force option), but replaced only if the contents changed. (regardless of the C(force) option), but replaced only if the contents changed..
required: true required: true
tmp_dest: tmp_dest:
description: description:
- absolute path of where temporary file is downloaded to. - Absolute path of where temporary file is downloaded to.
- Defaults to TMPDIR, TEMP or TMP env variables or a platform specific value - Defaults to C(TMPDIR), C(TEMP) or C(TMP) env variables or a platform specific value.
- https://docs.python.org/2/library/tempfile.html#tempfile.tempdir - U(https://docs.python.org/2/library/tempfile.html#tempfile.tempdir)
required: false
default: ''
version_added: '2.1' version_added: '2.1'
force: force:
description: description:
- If C(yes) and C(dest) is not a directory, will download the file every - If C(yes) and C(dest) is not a directory, will download the file every
time and replace the file if the contents change. If C(no), the file time and replace the file if the contents change. If C(no), the file
will only be downloaded if the destination does not exist. Generally will only be downloaded if the destination does not exist. Generally
should be C(yes) only for small local files. Prior to 0.6, this module should be C(yes) only for small local files.
behaved as if C(yes) was the default. - Prior to 0.6, this module behaved as if C(yes) was the default.
version_added: "0.7" version_added: '0.7'
required: false default: 'no'
choices: [ "yes", "no" ] type: bool
default: "no" aliases: [ thirsty ]
aliases: [ "thirsty" ]
backup: backup:
description: description:
- Create a backup file including the timestamp information so you can get - Create a backup file including the timestamp information so you can get
the original file back if you somehow clobbered it incorrectly. the original file back if you somehow clobbered it incorrectly.
required: false required: false
choices: [ "yes", "no" ] default: 'no'
default: "no" type: bool
version_added: '2.1' version_added: '2.1'
sha256sum: sha256sum:
description: description:
- If a SHA-256 checksum is passed to this parameter, the digest of the - If a SHA-256 checksum is passed to this parameter, the digest of the
destination file will be calculated after it is downloaded to ensure destination file will be calculated after it is downloaded to ensure
its integrity and verify that the transfer completed successfully. its integrity and verify that the transfer completed successfully.
This option is deprecated. Use 'checksum'. This option is deprecated. Use C(checksum) instead.
default: ''
version_added: "1.3" version_added: "1.3"
required: false
default: null
checksum: checksum:
description: description:
- 'If a checksum is passed to this parameter, the digest of the - 'If a checksum is passed to this parameter, the digest of the
destination file will be calculated after it is downloaded to ensure destination file will be calculated after it is downloaded to ensure
its integrity and verify that the transfer completed successfully. its integrity and verify that the transfer completed successfully.
Format: <algorithm>:<checksum>, e.g.: checksum="sha256:D98291AC[...]B6DC7B97" Format: <algorithm>:<checksum>, e.g. checksum="sha256:D98291AC[...]B6DC7B97"'
If you worry about portability, only the sha1 algorithm is available - If you worry about portability, only the sha1 algorithm is available
on all platforms and python versions. The third party hashlib on all platforms and python versions.
library can be installed for access to additional algorithms. - The third party hashlib library can be installed for access to additional algorithms.
Additionally, if a checksum is passed to this parameter, and the file exist under - Additionally, if a checksum is passed to this parameter, and the file exist under
the C(dest) location, the destination_checksum would be calculated, and if the C(dest) location, the I(destination_checksum) would be calculated, and if
checksum equals destination_checksum, the file download would be skipped checksum equals I(destination_checksum), the file download would be skipped
(unless C(force) is true). ' (unless C(force) is true).
default: ''
version_added: "2.0" version_added: "2.0"
required: false
default: null
use_proxy: use_proxy:
description: description:
- if C(no), it will not use a proxy, even if one is defined in - if C(no), it will not use a proxy, even if one is defined in
an environment variable on the target hosts. an environment variable on the target hosts.
required: false
default: 'yes' default: 'yes'
choices: ['yes', 'no'] type: bool
validate_certs: validate_certs:
description: description:
- If C(no), SSL certificates will not be validated. This should only be used - If C(no), SSL certificates will not be validated. This should only be used
on personally controlled sites using self-signed certificates. on personally controlled sites using self-signed certificates.
required: false
default: 'yes' default: 'yes'
choices: ['yes', 'no'] type: bool
timeout: timeout:
description: description:
- Timeout in seconds for URL request - Timeout in seconds for URL request.
required: false
default: 10 default: 10
version_added: '1.8' version_added: '1.8'
headers: headers:
description: description:
- 'Add custom HTTP headers to a request in the format "key:value,key:value"' - Add custom HTTP headers to a request in the format "key:value,key:value".
required: false
default: null
version_added: '2.0' version_added: '2.0'
url_username: url_username:
description: description:
- The username for use in HTTP basic authentication. This parameter can be used - The username for use in HTTP basic authentication.
without C(url_password) for sites that allow empty passwords. - This parameter can be used without C(url_password) for sites that allow empty passwords.
required: false
version_added: '1.6' version_added: '1.6'
url_password: url_password:
description: description:
- The password for use in HTTP basic authentication. If the C(url_username) - The password for use in HTTP basic authentication.
parameter is not specified, the C(url_password) parameter will not be used. - If the C(url_username) parameter is not specified, the C(url_password) parameter will not be used.
required: false
version_added: '1.6' version_added: '1.6'
force_basic_auth: force_basic_auth:
version_added: '2.0' version_added: '2.0'
@ -155,88 +142,87 @@ options:
responds to an initial request with a 401 status. Since some basic auth services do not properly responds to an initial request with a 401 status. Since some basic auth services do not properly
send a 401, logins will fail. This option forces the sending of the Basic authentication header send a 401, logins will fail. This option forces the sending of the Basic authentication header
upon initial request. upon initial request.
required: false default: 'no'
choices: [ "yes", "no" ] type: bool
default: "no"
client_cert: client_cert:
required: false
default: null
description: description:
- PEM formatted certificate chain file to be used for SSL client - PEM formatted certificate chain file to be used for SSL client
authentication. This file can also include the key as well, and if authentication. This file can also include the key as well, and if
the key is included, I(client_key) is not required the key is included, C(client_key) is not required.
version_added: 2.4 version_added: '2.4'
client_key: client_key:
required: false
default: null
description: description:
- PEM formatted file that contains your private key to be used for SSL - PEM formatted file that contains your private key to be used for SSL
client authentication. If I(client_cert) contains both the certificate client authentication. If C(client_cert) contains both the certificate
and key, this option is not required. and key, this option is not required.
version_added: 2.4 version_added: '2.4'
others: others:
description: description:
- all arguments accepted by the M(file) module also work here - all arguments accepted by the M(file) module also work here
required: false
# informational: requirements for nodes # informational: requirements for nodes
requirements: [ ]
extends_documentation_fragment: extends_documentation_fragment:
- files - files
author: "Jan-Piet Mens (@jpmens)" author: Jan-Piet Mens (@jpmens)
''' '''
EXAMPLES=''' EXAMPLES = r'''
- name: download foo.conf - name: Download foo.conf
get_url: get_url:
url: http://example.com/path/file.conf url: http://example.com/path/file.conf
dest: /etc/foo.conf dest: /etc/foo.conf
mode: 0440 mode: 0440
- name: download file and force basic auth - name: Download file and force basic auth
get_url: get_url:
url: http://example.com/path/file.conf url: http://example.com/path/file.conf
dest: /etc/foo.conf dest: /etc/foo.conf
force_basic_auth: yes force_basic_auth: yes
- name: download file with custom HTTP headers - name: Download file with custom HTTP headers
get_url: get_url:
url: http://example.com/path/file.conf url: http://example.com/path/file.conf
dest: /etc/foo.conf dest: /etc/foo.conf
headers: 'key:value,key:value' headers: 'key:value,key:value'
- name: download file with check (sha256) - name: Download file with check (sha256)
get_url: get_url:
url: http://example.com/path/file.conf url: http://example.com/path/file.conf
dest: /etc/foo.conf dest: /etc/foo.conf
checksum: sha256:b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c checksum: sha256:b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c
- name: download file with check (md5) - name: Download file with check (md5)
get_url: get_url:
url: http://example.com/path/file.conf url: http://example.com/path/file.conf
dest: /etc/foo.conf dest: /etc/foo.conf
checksum: md5:66dffb5228a211e61d6d7ef4a86f5758 checksum: md5:66dffb5228a211e61d6d7ef4a86f5758
- name: download file from a file path - name: Download file from a file path
get_url: get_url:
url: "file:///tmp/afile.txt" url: file:///tmp/afile.txt
dest: /tmp/afilecopy.txt dest: /tmp/afilecopy.txt
''' '''
import shutil
import datetime import datetime
import os
import re import re
import shutil
import tempfile import tempfile
from ansible.module_utils.basic import AnsibleModule, get_exception
from ansible.module_utils.six.moves.urllib.parse import urlsplit from ansible.module_utils.six.moves.urllib.parse import urlsplit
from ansible.module_utils.urls import fetch_url, url_argument_spec
# ============================================================== # ==============================================================
# url handling # url handling
def url_filename(url): def url_filename(url):
fn = os.path.basename(urlsplit(url)[2]) fn = os.path.basename(urlsplit(url)[2])
if fn == '': if fn == '':
return 'index.html' return 'index.html'
return fn return fn
def url_get(module, url, dest, use_proxy, last_mod_time, force, timeout=10, headers=None, tmp_dest=''): def url_get(module, url, dest, use_proxy, last_mod_time, force, timeout=10, headers=None, tmp_dest=''):
""" """
Download data from the url and store in a temporary file. Download data from the url and store in a temporary file.
@ -244,9 +230,9 @@ def url_get(module, url, dest, use_proxy, last_mod_time, force, timeout=10, head
Return (tempfile, info about the request) Return (tempfile, info about the request)
""" """
if module.check_mode: if module.check_mode:
method='HEAD' method = 'HEAD'
else: else:
method='GET' method = 'GET'
rsp, info = fetch_url(module, url, use_proxy=use_proxy, force=force, last_mod_time=last_mod_time, timeout=timeout, headers=headers, method=method) rsp, info = fetch_url(module, url, use_proxy=use_proxy, force=force, last_mod_time=last_mod_time, timeout=timeout, headers=headers, method=method)
@ -261,7 +247,7 @@ def url_get(module, url, dest, use_proxy, last_mod_time, force, timeout=10, head
module.fail_json(msg="Request failed", status_code=info['status'], response=info['msg'], url=url, dest=dest) module.fail_json(msg="Request failed", status_code=info['status'], response=info['msg'], url=url, dest=dest)
# create a temporary file and copy content to do checksum-based replacement # create a temporary file and copy content to do checksum-based replacement
if tmp_dest != '': if tmp_dest:
# tmp_dest should be an existing dir # tmp_dest should be an existing dir
tmp_dest_is_dir = os.path.isdir(tmp_dest) tmp_dest_is_dir = os.path.isdir(tmp_dest)
if not tmp_dest_is_dir: if not tmp_dest_is_dir:
@ -278,13 +264,14 @@ def url_get(module, url, dest, use_proxy, last_mod_time, force, timeout=10, head
try: try:
shutil.copyfileobj(rsp, f) shutil.copyfileobj(rsp, f)
except Exception: except Exception:
err = get_exception() e = get_exception()
os.remove(tempname) os.remove(tempname)
module.fail_json(msg="failed to create temporary content file: %s" % str(err)) module.fail_json(msg="failed to create temporary content file: %s" % e)
f.close() f.close()
rsp.close() rsp.close()
return tempname, info return tempname, info
def extract_filename_from_headers(headers): def extract_filename_from_headers(headers):
""" """
Extracts a filename from the given dict of HTTP headers. Extracts a filename from the given dict of HTTP headers.
@ -311,24 +298,25 @@ def extract_filename_from_headers(headers):
def main(): def main():
argument_spec = url_argument_spec() argument_spec = url_argument_spec()
argument_spec.update( argument_spec.update(
url = dict(required=True), url=dict(type='str', required=True),
dest = dict(required=True, type='path'), dest=dict(type='path', required=True),
backup = dict(default=False, type='bool'), backup=dict(type='bool'),
sha256sum = dict(default=''), sha256sum=dict(type='str', default=''),
checksum = dict(default=''), checksum=dict(type='str', default=''),
timeout = dict(required=False, type='int', default=10), timeout=dict(type='int', default=10),
headers = dict(required=False, default=None), headers=dict(type='str'),
tmp_dest = dict(required=False, default='', type='path'), tmp_dest=dict(type='path'),
) )
module = AnsibleModule( module = AnsibleModule(
# not checking because of daisy chain to file module # not checking because of daisy chain to file module
argument_spec = argument_spec, argument_spec=argument_spec,
add_file_common_args=True, add_file_common_args=True,
supports_check_mode=True, supports_check_mode=True,
mutually_exclusive=(['checksum', 'sha256sum']),
) )
url = module.params['url'] url = module.params['url']
dest = module.params['dest'] dest = module.params['dest']
backup = module.params['backup'] backup = module.params['backup']
force = module.params['force'] force = module.params['force']
@ -351,11 +339,11 @@ def main():
last_mod_time = None last_mod_time = None
# workaround for usage of deprecated sha256sum parameter # workaround for usage of deprecated sha256sum parameter
if sha256sum != '': if sha256sum:
checksum = 'sha256:%s' % (sha256sum) checksum = 'sha256:%s' % (sha256sum)
# checksum specified, parse for algorithm and checksum # checksum specified, parse for algorithm and checksum
if checksum != '': if checksum:
try: try:
algorithm, checksum = checksum.rsplit(':', 1) algorithm, checksum = checksum.rsplit(':', 1)
# Remove any non-alphanumeric characters, including the infamous # Remove any non-alphanumeric characters, including the infamous
@ -416,13 +404,13 @@ def main():
filename = url_filename(info['url']) filename = url_filename(info['url'])
dest = os.path.join(dest, filename) dest = os.path.join(dest, filename)
checksum_src = None checksum_src = None
checksum_dest = None checksum_dest = None
# If the remote URL exists, we're done with check mode # If the remote URL exists, we're done with check mode
if module.check_mode: if module.check_mode:
os.remove(tmpsrc) os.remove(tmpsrc)
res_args = dict( url = url, dest = dest, src = tmpsrc, changed = True, msg = info.get('msg', '')) res_args = dict(url=url, dest=dest, src=tmpsrc, changed=True, msg=info.get('msg', ''))
module.exit_json(**res_args) module.exit_json(**res_args)
# raise an error if there is no tmpsrc file # raise an error if there is no tmpsrc file
@ -431,7 +419,7 @@ def main():
module.fail_json(msg="Request failed", status_code=info['status'], response=info['msg']) module.fail_json(msg="Request failed", status_code=info['status'], response=info['msg'])
if not os.access(tmpsrc, os.R_OK): if not os.access(tmpsrc, os.R_OK):
os.remove(tmpsrc) os.remove(tmpsrc)
module.fail_json( msg="Source %s not readable" % (tmpsrc)) module.fail_json(msg="Source %s not readable" % (tmpsrc))
checksum_src = module.sha1(tmpsrc) checksum_src = module.sha1(tmpsrc)
# check if there is no dest file # check if there is no dest file
@ -439,15 +427,15 @@ def main():
# raise an error if copy has no permission on dest # raise an error if copy has no permission on dest
if not os.access(dest, os.W_OK): if not os.access(dest, os.W_OK):
os.remove(tmpsrc) os.remove(tmpsrc)
module.fail_json( msg="Destination %s not writable" % (dest)) module.fail_json(msg="Destination %s not writable" % (dest))
if not os.access(dest, os.R_OK): if not os.access(dest, os.R_OK):
os.remove(tmpsrc) os.remove(tmpsrc)
module.fail_json( msg="Destination %s not readable" % (dest)) module.fail_json(msg="Destination %s not readable" % (dest))
checksum_dest = module.sha1(dest) checksum_dest = module.sha1(dest)
else: else:
if not os.access(os.path.dirname(dest), os.W_OK): if not os.access(os.path.dirname(dest), os.W_OK):
os.remove(tmpsrc) os.remove(tmpsrc)
module.fail_json( msg="Destination %s not writable" % (os.path.dirname(dest))) module.fail_json(msg="Destination %s not writable" % (os.path.dirname(dest)))
backup_file = None backup_file = None
if checksum_src != checksum_dest: if checksum_src != checksum_dest:
@ -457,9 +445,9 @@ def main():
backup_file = module.backup_local(dest) backup_file = module.backup_local(dest)
shutil.copyfile(tmpsrc, dest) shutil.copyfile(tmpsrc, dest)
except Exception: except Exception:
err = get_exception() e = get_exception()
os.remove(tmpsrc) os.remove(tmpsrc)
module.fail_json(msg="failed to copy %s to %s: %s" % (tmpsrc, dest, str(err))) module.fail_json(msg="failed to copy %s to %s: %s" % (tmpsrc, dest, e))
changed = True changed = True
else: else:
changed = False changed = False
@ -486,8 +474,8 @@ def main():
md5sum = None md5sum = None
res_args = dict( res_args = dict(
url = url, dest = dest, src = tmpsrc, md5sum = md5sum, checksum_src = checksum_src, url=url, dest=dest, src=tmpsrc, md5sum=md5sum, checksum_src=checksum_src,
checksum_dest = checksum_dest, changed = changed, msg = info.get('msg', ''), status_code=info.get('status','') checksum_dest=checksum_dest, changed=changed, msg=info.get('msg', ''), status_code=info.get('status', '')
) )
if backup_file: if backup_file:
res_args['backup_file'] = backup_file res_args['backup_file'] = backup_file
@ -495,8 +483,6 @@ def main():
# Mission complete # Mission complete
module.exit_json(**res_args) module.exit_json(**res_args)
# import module snippets
from ansible.module_utils.basic import *
from ansible.module_utils.urls import *
if __name__ == '__main__': if __name__ == '__main__':
main() main()

1
test/sanity/pep8/legacy-files.txt
View file

@ -374,7 +374,6 @@ lib/ansible/modules/network/aos/aos_template.py
lib/ansible/modules/network/asa/asa_acl.py lib/ansible/modules/network/asa/asa_acl.py
lib/ansible/modules/network/asa/asa_command.py lib/ansible/modules/network/asa/asa_command.py
lib/ansible/modules/network/asa/asa_config.py lib/ansible/modules/network/asa/asa_config.py
lib/ansible/modules/net_tools/basics/get_url.py
lib/ansible/modules/net_tools/basics/uri.py lib/ansible/modules/net_tools/basics/uri.py
lib/ansible/modules/network/bigswitch/bigmon_chain.py lib/ansible/modules/network/bigswitch/bigmon_chain.py
lib/ansible/modules/network/bigswitch/bigmon_policy.py lib/ansible/modules/network/bigswitch/bigmon_policy.py