Ansible.Basic added generic fragment merger for module options (#69719)

This commit is contained in:
Jordan Borean 2020-05-29 16:11:38 +10:00 committed by GitHub
parent 40f21dfd3c
commit f81f5da20e
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
8 changed files with 485 additions and 69 deletions

View file

@ -0,0 +1,2 @@
minor_changes:
- Ansible.Basic - Added the ability to specify multiple fragments to load in a generic way for modules that use a module_util with fragment options

View file

@ -209,10 +209,11 @@ options set:
- ``aliases``: A list of aliases for the module option
- ``choices``: A list of valid values for the module option, if ``type=list`` then each list value is validated against the choices and not the list itself
- ``default``: The default value for the module option if not set
- ``deprecated_aliases``: A list of hashtables that define aliases that are deprecated and the versions they will be removed in. Each entry must contain the keys ``name`` and ``version``
- ``deprecated_aliases``: A list of hashtables that define aliases that are deprecated and the versions they will be removed in. Each entry must contain the key ``name`` with either ``version`` or ``date``
- ``elements``: When ``type=list``, this sets the type of each list value, the values are the same as ``type``
- ``no_log``: Will sanitise the input value before being returned in the ``module_invocation`` return value
- ``removed_in_version``: States when a deprecated module option is to be removed, a warning is displayed to the end user if set
- ``removed_at_date``: States the date when a deprecated module option will be removed, a warning is displayed to the end user if set
- ``required``: Will fail when the module option is not set
- ``type``: The type of the module option, if not set then it defaults to ``str``. The valid types are;
* ``bool``: A boolean value
@ -388,6 +389,126 @@ at the end of the file. For example
Export-ModuleMember -Function Invoke-CustomUtil, Get-CustomInfo
Exposing shared module options
++++++++++++++++++++++++++++++
PowerShell module utils can easily expose common module options that a module can use when building its argument spec.
This allows common features to be stored and maintained in one location and have those features used by multiple
modules with minimal effort. Any new features or bugifxes added to one of these utils are then automatically used by
the various modules that call that util.
An example of this would be to have a module util that handles authentication and communication against an API This
util can be used by multiple modules to expose a common set of module options like the API endpoint, username,
password, timeout, cert validation, etc, without having to add those options to each module spec.
The standard convention for a module util that has a shared argument spec would have
- A ``Get-<namespace.name.util name>Spec`` function that outputs the common spec for a module
* It is highly recommended to make this function name be unique to the module to avoid any conflicts with other utils that can be loaded
* The format of the output spec is a Hashtable in the same format as the ``$spec`` used for normal modules
- A function that takes in an ``AnsibleModule`` object called under the ``-Module`` parameter which it can use to get the shared options
Because these options can be shared across various module it is highly recommended to keep the module option names and
aliases in the shared spec as specific as they can be. For example do not have a util option called ``password``,
rather you should prefix it with a unique name like ``acme_password``.
.. warning::
Failure to have a unique option name or alias can prevent the util being used by module that also use those names or
aliases for its own options.
The following is an example module util called ``ServiceAuth.psm1`` in a collection that implements a common way for
modules to authentication with a service.
.. code-block:: powershell
Invoke-MyServiceResource {
[CmdletBinding()]
param (
[Parameter(Mandatory=$true)]
[ValidateScript({ $_.GetType().FullName -eq 'Ansible.Basic.AnsibleModule' })]
$Module,
[Parameter(Mandatory=$true)]
[String]
$ResourceId
[String]
$State = 'present'
)
# Process the common module options known to the util
$params = @{
ServerUri = $Module.Params.my_service_url
}
if ($Module.Params.my_service_username) {
$params.Credential = Get-MyServiceCredential
}
if ($State -eq 'absent') {
Remove-MyService @params -ResourceId $ResourceId
} else {
New-MyService @params -ResourceId $ResourceId
}
}
Get-MyNamespaceMyCollectionServiceAuthSpec {
# Output the util spec
@{
options = @{
my_service_url = @{ type = 'str'; required = $true }
my_service_username = @{ type = 'str' }
my_service_password = @{ type = 'str'; no_log = $true }
}
required_together = @(
,@('my_service_username', 'my_service_password')
)
}
}
$exportMembers = @{
Function = 'Get-MyNamespaceMyCollectionServiceAuthSpec', 'Invoke-MyServiceResource'
}
Export-ModuleMember @exportMembers
For a module to take advantage of this common argument spec it can be set out like
.. code-block:: powershell
#!powershell
# Include the module util ServiceAuth.psm1 from the my_namespace.my_collection collection
#AnsibleRequires -PowerShell ansible_collections.my_namespace.my_collection.plugins.module_utils.ServiceAuth
# Create the module spec like normal
$spec = @{
options = @{
resource_id = @{ type = 'str'; required = $true }
state = @{ type = 'str'; choices = 'absent', 'present' }
}
}
# Create the module from the module spec but also include the util spec to merge into our own.
$module = [Ansible.Basic.AnsibleModule]::Create($args, $spec, @(Get-MyNamespaceMyCollectionServiceAuthSpec))
# Call the ServiceAuth module util and pass in the module object so it can access the module options.
Invoke-MyServiceResource -Module $module -ResourceId $module.Params.resource_id -State $module.params.state
$module.ExitJson()
.. note::
Options defined in the module spec will always have precedence over a util spec. Any list values under the same key
in a util spec will be appended to the module spec for that same key. Dictionary values will add any keys that are
missing from the module spec and merge any values that are lists or dictionaries. This is similar to how the doc
fragment plugins work when extending module documentation.
To document these shared util options for a module, create a doc fragment plugin that documents the options implemented
by the module util and extend the module docs for every module that implements the util to include that fragment in
its docs.
Windows playbook module testing
===============================

View file

@ -81,16 +81,16 @@ namespace Ansible.Basic
{ "default", new List<object>() { null, null } },
{ "deprecated_aliases", new List<object>() { typeof(List<Hashtable>), typeof(List<Hashtable>) } },
{ "elements", new List<object>() { null, null } },
{ "mutually_exclusive", new List<object>() { typeof(List<List<string>>), null } },
{ "mutually_exclusive", new List<object>() { typeof(List<List<string>>), typeof(List<object>) } },
{ "no_log", new List<object>() { false, typeof(bool) } },
{ "options", new List<object>() { typeof(Hashtable), typeof(Hashtable) } },
{ "removed_in_version", new List<object>() { null, typeof(string) } },
{ "removed_at_date", new List<object>() { null, typeof(DateTime) } },
{ "required", new List<object>() { false, typeof(bool) } },
{ "required_by", new List<object>() { typeof(Hashtable), typeof(Hashtable) } },
{ "required_if", new List<object>() { typeof(List<List<object>>), null } },
{ "required_one_of", new List<object>() { typeof(List<List<string>>), null } },
{ "required_together", new List<object>() { typeof(List<List<string>>), null } },
{ "required_if", new List<object>() { typeof(List<List<object>>), typeof(List<object>) } },
{ "required_one_of", new List<object>() { typeof(List<List<string>>), typeof(List<object>) } },
{ "required_together", new List<object>() { typeof(List<List<string>>), typeof(List<object>) } },
{ "supports_check_mode", new List<object>() { false, typeof(bool) } },
{ "type", new List<object>() { "str", null } },
};
@ -187,7 +187,7 @@ namespace Ansible.Basic
}
}
public AnsibleModule(string[] args, IDictionary argumentSpec)
public AnsibleModule(string[] args, IDictionary argumentSpec, IDictionary[] fragments = null)
{
// NoLog is not set yet, we cannot rely on FailJson to sanitize the output
// Do the minimum amount to get this running before we actually parse the params
@ -196,6 +196,16 @@ namespace Ansible.Basic
{
ValidateArgumentSpec(argumentSpec);
// Merge the fragments if present into the main arg spec.
if (fragments != null)
{
foreach (IDictionary fragment in fragments)
{
ValidateArgumentSpec(fragment);
MergeFragmentSpec(argumentSpec, fragment);
}
}
// Used by ansible-test to retrieve the module argument spec, not designed for public use.
if (_DebugArgSpec)
{
@ -252,9 +262,9 @@ namespace Ansible.Basic
LogEvent(String.Format("Invoked with:\r\n {0}", FormatLogData(Params, 2)), sanitise: false);
}
public static AnsibleModule Create(string[] args, IDictionary argumentSpec)
public static AnsibleModule Create(string[] args, IDictionary argumentSpec, IDictionary[] fragments = null)
{
return new AnsibleModule(args, argumentSpec);
return new AnsibleModule(args, argumentSpec, fragments);
}
public void Debug(string message)
@ -608,13 +618,19 @@ namespace Ansible.Basic
{
// verify the actual type is not just a single value of the list type
Type entryType = optionType.GetGenericArguments()[0];
object[] arrayElementTypes = new object[]
{
null, // ArrayList does not have an ElementType
entryType,
typeof(object), // Hope the object is actually entryType or it can at least be casted.
};
bool isArray = actualType.IsArray && (actualType.GetElementType() == entryType || actualType.GetElementType() == typeof(object));
bool isArray = entry.Value is IList && arrayElementTypes.Contains(actualType.GetElementType());
if (actualType == entryType || isArray)
{
object[] rawArray;
object rawArray;
if (isArray)
rawArray = (object[])entry.Value;
rawArray = entry.Value;
else
rawArray = new object[1] { entry.Value };
@ -679,6 +695,32 @@ namespace Ansible.Basic
argumentSpec[changedValue.Key] = changedValue.Value;
}
private void MergeFragmentSpec(IDictionary argumentSpec, IDictionary fragment)
{
foreach (DictionaryEntry fragmentEntry in fragment)
{
string fragmentKey = fragmentEntry.Key.ToString();
if (argumentSpec.Contains(fragmentKey))
{
// We only want to add new list entries and merge dictionary new keys and values. Leave the other
// values as is in the argument spec as that takes priority over the fragment.
if (fragmentEntry.Value is IDictionary)
{
MergeFragmentSpec((IDictionary)argumentSpec[fragmentKey], (IDictionary)fragmentEntry.Value);
}
else if (fragmentEntry.Value is IList)
{
IList specValue = (IList)argumentSpec[fragmentKey];
foreach (object fragmentValue in (IList)fragmentEntry.Value)
specValue.Add(fragmentValue);
}
}
else
argumentSpec[fragmentKey] = fragmentEntry.Value;
}
}
private void SetArgumentSpecDefaults(IDictionary argumentSpec)
{
foreach (KeyValuePair<string, List<object>> metadataEntry in specDefaults)

View file

@ -84,8 +84,7 @@ Function Get-AnsibleWebRequest {
$spec = @{
options = @{}
}
$spec.options += $ansible_web_request_options
$module = Ansible.Basic.AnsibleModule]::Create($args, $spec)
$module = Ansible.Basic.AnsibleModule]::Create($args, $spec, @(Get-AnsibleWebRequestSpec))
$web_request = Get-AnsibleWebRequest -Module $module
#>
@ -371,8 +370,7 @@ Function Invoke-WithWebRequest {
path = @{ type = "path"; required = $true }
}
}
$spec.options += $ansible_web_request_options
$module = Ansible.Basic.AnsibleModule]::Create($args, $spec)
$module = Ansible.Basic.AnsibleModule]::Create($args, $spec, @(Get-AnsibleWebRequestSpec))
$web_request = Get-AnsibleWebRequest -Module $module
@ -467,58 +465,23 @@ Function Invoke-WithWebRequest {
}
}
Function Merge-WebRequestSpec {
Function Get-AnsibleWebRequestSpec {
<#
.SYNOPSIS
Merges a modules spec definition with extra options supplied by this module_util. Options from the module take
priority over the module util spec.
Used by modules to get the argument spec fragment for AnsibleModule.
.PARAMETER ModuleSpec
The root $spec of a module option definition to merge with.
.EXAMPLE
.EXAMPLES
$spec = @{
options = @{
name = @{ type = "str" }
}
supports_check_mode = $true
options = @{}
}
$spec = Merge-WebRequestSpec -ModuleSpec $spec
$module = [Ansible.Basic.AnsibleModule]::Create($args, $spec, @(Get-AnsibleWebRequestSpec))
#>
[CmdletBinding()]
param (
[Parameter(Mandatory=$true)]
[System.Collections.IDictionary]
$ModuleSpec,
[System.Collections.IDictionary]
$SpecToMerge = @{ options = $ansible_web_request_options }
)
foreach ($option_kvp in $SpecToMerge.GetEnumerator()) {
$k = $option_kvp.Key
$v = $option_kvp.Value
if ($ModuleSpec.Contains($k)) {
if ($v -is [System.Collections.IDictionary]) {
$ModuleSpec[$k] = Merge-WebRequestSpec -ModuleSpec $ModuleSpec[$k] -SpecToMerge $v
} elseif ($v -is [Array] -or $v -is [System.Collections.IList]) {
$sourceList = [System.Collections.Generic.List[Object]]$ModuleSpec[$k]
foreach ($entry in $v) {
$sourceList.Add($entry)
}
$ModuleSpec[$k] = $sourceList
}
} else {
$ModuleSpec[$k] = $v
}
}
$ModuleSpec
@{ options = $ansible_web_request_options }
}
# See lib/ansible/plugins/doc_fragments/url_windows.py
# Kept here for backwards compat as this variable was added in Ansible 2.9. Ultimately this util should be removed
# once the deprecation period has been added.
$ansible_web_request_options = @{
method = @{ type="str" }
follow_redirects = @{ type="str"; choices=@("all","none","safe"); default="safe" }
@ -545,7 +508,7 @@ $ansible_web_request_options = @{
}
$export_members = @{
Function = "Get-AnsibleWebRequest", "Invoke-WithWebRequest", "Merge-WebRequestSpec"
Function = "Get-AnsibleWebRequest", "Get-AnsibleWebRequestSpec", "Invoke-WithWebRequest"
Variable = "ansible_web_request_options"
}
Export-ModuleMember @export_members

View file

@ -11,8 +11,6 @@ $spec = @{
my_opt = @{ type = "str"; required = $true }
}
}
$util_spec = Get-PSUtilSpec
$spec.options += $util_spec.options
$module = [Ansible.Basic.AnsibleModule]::Create($args, $spec)
$module = [Ansible.Basic.AnsibleModule]::Create($args, $spec, @(Get-PSUtilSpec))
$module.ExitJson()

View file

@ -2701,6 +2701,301 @@ test_no_log - Invoked with:
$actual.Length | Assert-Equals -Expected 1
$actual[0] | Assert-DictionaryEquals -Expected @{"abc" = "def"}
}
"Spec with fragments" = {
$spec = @{
options = @{
option1 = @{ type = "str" }
}
}
$fragment1 = @{
options = @{
option2 = @{ type = "str" }
}
}
Set-Variable -Name complex_args -Scope Global -Value @{
option1 = "option1"
option2 = "option2"
}
$m = [Ansible.Basic.AnsibleModule]::Create(@(), $spec, @($fragment1))
$failed = $false
try {
$m.ExitJson()
} catch [System.Management.Automation.RuntimeException] {
$failed = $true
$_.Exception.Message | Assert-Equals -Expected "exit: 0"
$actual = [Ansible.Basic.AnsibleModule]::FromJson($_.Exception.InnerException.Output)
}
$failed | Assert-Equals -Expected $true
$actual.changed | Assert-Equals -Expected $false
$actual.invocation | Assert-DictionaryEquals -Expected @{module_args = $complex_args}
}
"Fragment spec that with a deprecated alias" = {
$spec = @{
options = @{
option1 = @{
aliases = @("alias1_spec")
type = "str"
deprecated_aliases = @(
@{name = "alias1_spec"; version = "2.0"}
)
}
option2 = @{
aliases = @("alias2_spec")
deprecated_aliases = @(
@{name = "alias2_spec"; version = "2.0"}
)
}
}
}
$fragment1 = @{
options = @{
option1 = @{
aliases = @("alias1")
deprecated_aliases = @() # Makes sure it doesn't overwrite the spec, just adds to it.
}
option2 = @{
aliases = @("alias2")
deprecated_aliases = @(
@{name = "alias2"; version = "2.0"}
)
type = "str"
}
}
}
Set-Variable -Name complex_args -Scope Global -Value @{
alias1_spec = "option1"
alias2 = "option2"
}
$m = [Ansible.Basic.AnsibleModule]::Create(@(), $spec, @($fragment1))
$failed = $false
try {
$m.ExitJson()
} catch [System.Management.Automation.RuntimeException] {
$failed = $true
$_.Exception.Message | Assert-Equals -Expected "exit: 0"
$actual = [Ansible.Basic.AnsibleModule]::FromJson($_.Exception.InnerException.Output)
}
$failed | Assert-Equals -Expected $true
$actual.deprecations.Count | Assert-Equals -Expected 2
$actual.deprecations[0] | Assert-DictionaryEquals -Expected @{
msg = "Alias 'alias1_spec' is deprecated. See the module docs for more information"; version = "2.0"
}
$actual.deprecations[1] | Assert-DictionaryEquals -Expected @{
msg = "Alias 'alias2' is deprecated. See the module docs for more information"; version = "2.0"
}
$actual.changed | Assert-Equals -Expected $false
$actual.invocation | Assert-DictionaryEquals -Expected @{
module_args = @{
option1 = "option1"
alias1_spec = "option1"
option2 = "option2"
alias2 = "option2"
}
}
}
"Fragment spec with mutual args" = {
$spec = @{
options = @{
option1 = @{ type = "str" }
option2 = @{ type = "str" }
}
mutually_exclusive = @(
,@('option1', 'option2')
)
}
$fragment1 = @{
options = @{
fragment1_1 = @{ type = "str" }
fragment1_2 = @{ type = "str" }
}
mutually_exclusive = @(
,@('fragment1_1', 'fragment1_2')
)
}
$fragment2 = @{
options = @{
fragment2 = @{ type = "str" }
}
}
Set-Variable -Name complex_args -Scope Global -Value @{
option1 = "option1"
fragment1_1 = "fragment1_1"
fragment1_2 = "fragment1_2"
fragment2 = "fragment2"
}
$failed = $false
try {
[Ansible.Basic.AnsibleModule]::Create(@(), $spec, @($fragment1, $fragment2))
} catch [System.Management.Automation.RuntimeException] {
$failed = $true
$_.Exception.Message | Assert-Equals -Expected "exit: 1"
$actual = [Ansible.Basic.AnsibleModule]::FromJson($_.Exception.InnerException.Output)
}
$failed | Assert-Equals -Expected $true
$actual.changed | Assert-Equals -Expected $false
$actual.failed | Assert-Equals -Expected $true
$actual.msg | Assert-Equals -Expected "parameters are mutually exclusive: fragment1_1, fragment1_2"
$actual.invocation | Assert-DictionaryEquals -Expected @{ module_args = $complex_args }
}
"Fragment spec with no_log" = {
$spec = @{
options = @{
option1 = @{
aliases = @("alias")
}
}
}
$fragment1 = @{
options = @{
option1 = @{
no_log = $true # Makes sure that a value set in the fragment but not in the spec is respected.
type = "str"
}
}
}
Set-Variable -Name complex_args -Scope Global -Value @{
alias = "option1"
}
$m = [Ansible.Basic.AnsibleModule]::Create(@(), $spec, @($fragment1))
$failed = $false
try {
$m.ExitJson()
} catch [System.Management.Automation.RuntimeException] {
$failed = $true
$_.Exception.Message | Assert-Equals -Expected "exit: 0"
$actual = [Ansible.Basic.AnsibleModule]::FromJson($_.Exception.InnerException.Output)
}
$failed | Assert-Equals -Expected $true
$actual.changed | Assert-Equals -Expected $false
$actual.invocation | Assert-DictionaryEquals -Expected @{
module_args = @{
option1 = "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER"
alias = "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER"
}
}
}
"Catch invalid fragment spec format" = {
$spec = @{
options = @{
option1 = @{ type = "str" }
}
}
$fragment = @{
options = @{}
invalid = "will fail"
}
Set-Variable -Name complex_args -Scope Global -Value @{
option1 = "option1"
}
$failed = $false
try {
[Ansible.Basic.AnsibleModule]::Create(@(), $spec, @($fragment))
} catch [System.Management.Automation.RuntimeException] {
$failed = $true
$_.Exception.Message | Assert-Equals -Expected "exit: 1"
$actual = [Ansible.Basic.AnsibleModule]::FromJson($_.Exception.InnerException.Output)
}
$failed | Assert-Equals -Expected $true
$actual.failed | Assert-Equals -Expected $true
$actual.msg.StartsWith("internal error: argument spec entry contains an invalid key 'invalid', valid keys: ") | Assert-Equals -Expected $true
}
"Spec with different list types" = {
$spec = @{
options = @{
# Single element of the same list type not in a list
option1 = @{
aliases = "alias1"
deprecated_aliases = @{name="alias1";version="2.0"}
}
# Arrays
option2 = @{
aliases = ,"alias2"
deprecated_aliases = ,@{name="alias2";version="2.0"}
}
# ArrayList
option3 = @{
aliases = [System.Collections.ArrayList]@("alias3")
deprecated_aliases = [System.Collections.ArrayList]@(@{name="alias3";version="2.0"})
}
# Generic.List[Object]
option4 = @{
aliases = [System.Collections.Generic.List[Object]]@("alias4")
deprecated_aliases = [System.Collections.Generic.List[Object]]@(@{name="alias4";version="2.0"})
}
# Generic.List[T]
option5 = @{
aliases = [System.Collections.Generic.List[String]]@("alias5")
deprecated_aliases = [System.Collections.Generic.List[Hashtable]]@()
}
}
}
$spec.options.option5.deprecated_aliases.Add(@{name="alias5";version="2.0"})
Set-Variable -Name complex_args -Scope Global -Value @{
alias1 = "option1"
alias2 = "option2"
alias3 = "option3"
alias4 = "option4"
alias5 = "option5"
}
$m = [Ansible.Basic.AnsibleModule]::Create(@(), $spec)
$failed = $false
try {
$m.ExitJson()
} catch [System.Management.Automation.RuntimeException] {
$failed = $true
$_.Exception.Message | Assert-Equals -Expected "exit: 0"
$actual = [Ansible.Basic.AnsibleModule]::FromJson($_.Exception.InnerException.Output)
}
$failed | Assert-Equals -Expected $true
$actual.changed | Assert-Equals -Expected $false
$actual.deprecations.Count | Assert-Equals -Expected 5
foreach ($dep in $actual.deprecations) {
$dep.msg -like "Alias 'alias?' is deprecated. See the module docs for more information" | Assert-Equals -Expected $true
$dep.version | Assert-Equals -Expected '2.0'
}
$actual.invocation | Assert-DictionaryEquals -Expected @{
module_args = @{
alias1 = "option1"
option1 = "option1"
alias2 = "option2"
option2 = "option2"
alias3 = "option3"
option3 = "option3"
alias4 = "option4"
option4 = "option4"
alias5 = "option5"
option5 = "option5"
}
}
}
}
try {
@ -2730,4 +3025,3 @@ try {
}
Exit-Module

View file

@ -423,9 +423,8 @@ $tests = [Ordered]@{
}
mutually_exclusive = @(,@('url', 'test'))
}
$spec = Merge-WebRequestSpec -ModuleSpec $spec
$testModule = [Ansible.Basic.AnsibleModule]::Create(@(), $spec)
$testModule = [Ansible.Basic.AnsibleModule]::Create(@(), $spec, @(Get-AnsibleWebRequestSpec))
$r = Get-AnsibleWebRequest -Url $testModule.Params.url -Module $testModule
$actual = Invoke-WithWebRequest -Module $testModule -Request $r -Script {

View file

@ -40,9 +40,7 @@ $spec = @{
)
supports_check_mode = $true
}
$spec = Merge-WebRequestSpec -ModuleSpec $spec
$module = [Ansible.Basic.AnsibleModule]::Create($args, $spec)
$module = [Ansible.Basic.AnsibleModule]::Create($args, $spec, @(Get-AnsibleWebRequestSpec))
$url = $module.Params.url
$dest = $module.Params.dest
@ -274,4 +272,3 @@ if ((-not $module.Result.ContainsKey("checksum_dest")) -and (Test-Path -LiteralP
}
$module.ExitJson()