These changes fix a bug in the Python runtime that would cause any
awaitable input properties passed to a resource that are missing
from the resource's output properties to be awaited twice. The fix is
straightforward: rather than roundtripping an input property through
serialize/deserialize, just deserialized the already-serialized input
property.
Fixes#2940.
These changes make the `pulumi.runtime.invoke` function invokable in a
synchronous manner. Because this function still needs to perform
asynchronous work under the covers--namely awaiting a provider URN and
ID if a provider instance is present in the `InvokeOptions`--this
requires some creativity. This creativity comes in the form of a helper
function, `_sync_await`, that performs a logical yield from the
currently running event, manually runs the event loop until the given
future completes, performs a logical resume back to the
currently executing event, and returns the result of the future.
The code in `_sync_await` is a bit scary, as it relies upon knowledge of
(and functions in) the internals of the `asyncio` package. The necessary
work performed in this function was derived from the implementations of
`task_step` (which pointed out the need to call `_{enter,leave}_task`)
and `BaseEventLoop.run_forever` (which illustrated how the event loop is
pumped). In addition to potential breaking changes to these internals,
the code may not work if a user has provided an alternative implementation
for `EventLoop`. That said, the code is a close enough copy of
`BaseEventLoop.run_forever` that it should be a reasonable solution.
Provides an additional helper function to read outputs from a stack reference in case it is known that the stack output must be present. This is similar to the design for config.get and config.require.
Fixes#2343.
These changes add support for passing `ignoreChanges` paths to resource
providers. This is intended to accommodate providers that perform diffs
between resource inputs and resource state (e.g. all Terraform-based
providers, the k8s provider when using API server dry-runs). These paths
are specified using the same syntax as the paths used in detailed diffs.
In addition to passing these paths to providers, the existing support
for `ignoreChanges` in inputs has been extended to accept paths rather
than top-level keys. It is an error to specify a path that is missing
one or more component in the old or new inputs.
Fixes#2936, #2663.
* Remove pulumi.io reference in tests
* Remove pulumi.io references in Dockerfiles
* Remove pulumi.io references in CONTRIBUTING.md
* Update README.md's
* Use correct logo
Dynamic providers in Python.
This PR uses [dill](https://pypi.org/project/dill/) for code serialization, along with a customization to help ensure deterministic serialization results.
One notable limitation - which I believe is a general requirement of Python - is that any serialization of Python functions must serialize byte code, and byte code is not safely versioned across Python versions. So any resource created with Python `3.x.y` can only be updated by exactly the same version of Python. This is very constraining, but it's not clear there is any other option within the realm of what "dynamic providers" are as a feature. It is plausible that we could ensure that updates which only update the serialized provider can avoid calling the dynamic provider operations, so that version updates could still be accomplished. We can explore this separately.
```py
from pulumi import ComponentResource, export, Input, Output
from pulumi.dynamic import Resource, ResourceProvider, CreateResult, UpdateResult
from typing import Optional
from github import Github, GithubObject
auth = "<auth token>"
g = Github(auth)
class GithubLabelArgs(object):
owner: Input[str]
repo: Input[str]
name: Input[str]
color: Input[str]
description: Optional[Input[str]]
def __init__(self, owner, repo, name, color, description=None):
self.owner = owner
self.repo = repo
self.name = name
self.color = color
self.description = description
class GithubLabelProvider(ResourceProvider):
def create(self, props):
l = g.get_user(props["owner"]).get_repo(props["repo"]).create_label(
name=props["name"],
color=props["color"],
description=props.get("description", GithubObject.NotSet))
return CreateResult(l.name, {**props, **l.raw_data})
def update(self, id, _olds, props):
l = g.get_user(props["owner"]).get_repo(props["repo"]).get_label(id)
l.edit(name=props["name"],
color=props["color"],
description=props.get("description", GithubObject.NotSet))
return UpdateResult({**props, **l.raw_data})
def delete(self, id, props):
l = g.get_user(props["owner"]).get_repo(props["repo"]).get_label(id)
l.delete()
class GithubLabel(Resource):
name: Output[str]
color: Output[str]
url: Output[str]
description: Output[str]
def __init__(self, name, args: GithubLabelArgs, opts = None):
full_args = {'url':None, 'description':None, 'name':None, 'color':None, **vars(args)}
super().__init__(GithubLabelProvider(), name, full_args, opts)
label = GithubLabel("foo", GithubLabelArgs("lukehoban", "todo", "mylabel", "d94f0b"))
export("label_color", label.color)
export("label_url", label.url)
```
Fixes https://github.com/pulumi/pulumi/issues/2902.
`GetAnalyzerInfo` is an RPC command that allows the Pulumi CLI to ask an
`Analyzer` for metadata about the resource policies it defines.
This is useful because the Pulumi service needs to be able to render
metadata about a policy pack after the user runs a `pulumi policy
publish`. Since we don't execute code on the server, the first step of
`policy publish` is to interrogate the policy pack program about what
policies it has, so that this metadata can be sent to the server, along
with a packed version of those policies.
* Plumbing the custom timeouts from the engine to the providers
* Plumbing the CustomTimeouts through to the engine and adding test to show this
* Change the provider proto to include individual timeouts
* Plumbing the CustomTimeouts from the engine through to the Provider RPC interface
* Change how the CustomTimeouts are sent across RPC
These errors were spotted in testing. We can now see that the timeout
information is arriving in the RegisterResourceRequest
```
req=&pulumirpc.RegisterResourceRequest{
Type: "aws:s3/bucket:Bucket",
Name: "my-bucket",
Parent: "urn:pulumi:dev::aws-vpc::pulumi:pulumi:Stack::aws-vpc-dev",
Custom: true,
Object: &structpb.Struct{},
Protect: false,
Dependencies: nil,
Provider: "",
PropertyDependencies: {},
DeleteBeforeReplace: false,
Version: "",
IgnoreChanges: nil,
AcceptSecrets: true,
AdditionalSecretOutputs: nil,
Aliases: nil,
CustomTimeouts: &pulumirpc.RegisterResourceRequest_CustomTimeouts{
Create: 300,
Update: 400,
Delete: 500,
XXX_NoUnkeyedLiteral: struct {}{},
XXX_unrecognized: nil,
XXX_sizecache: 0,
},
XXX_NoUnkeyedLiteral: struct {}{},
XXX_unrecognized: nil,
XXX_sizecache: 0,
}
```
* Changing the design to use strings
* CHANGELOG entry to include the CustomTimeouts work
* Changing custom timeouts to be passed around the engine as converted value
We don't want to pass around strings - the user can provide it but we want
to make the engine aware of the timeout in seconds as a float64
A resource can be imported by setting the `import` property in the
resource options bag when instantiating a resource. In order to
successfully import a resource, its desired configuration (i.e. its
inputs) must not differ from its actual configuration (i.e. its state)
as calculated by the resource's provider.
There are a few interesting state transitions hiding here when importing
a resource:
1. No prior resource exists in the checkpoint file. In this case, the
resource is simply imported.
2. An external resource exists in the checkpoint file. In this case, the
resource is imported and the old external state is discarded.
3. A non-external resource exists in the checkpoint file and its ID is
different from the ID to import. In this case, the new resource is
imported and the old resource is deleted.
4. A non-external resource exists in the checkpoint file, but the ID is
the same as the ID to import. In this case, the import ID is ignored
and the resource is treated as it would be in all cases except for
changes that would replace the resource. In that case, the step
generator issues an error that indicates that the import ID should be
removed: were we to move forward with the replace, the new state of
the stack would fall under case (3), which is almost certainly not
what the user intends.
Fixes#1662.
There current RPC model for Pulumi allows secret values to be deeply
embedded in lists or maps, however at the language level, since we
track secrets via `Output<T>` we need to ensure that during
deserialization, if a list or a map contains a secret, we need to
instead treat it as if the entire list or map was a secret.
We have logic in the language runtimes to do this as part of
serialization. There were a few issues this commit addresses:
- We were not promoting secretness across arrays in either Node or
Python
- For Python, our promotion logic was buggy and caused it to behave in
a manner where if any value was secret, the output values of the
object would be corrupted, because we'd incorrectly treat the
outputs as a secret who's value was a map, instead of a map of
values (some of which may be secret).
This caused very confusing behavior, because it would appear that a
resource creation call just did not set various output properties when
one or more of them ended up containing a secret.
Thse changes make a subtle but critical adjustment to the process the
Pulumi engine uses to determine whether or not a difference exists
between a resource's actual and desired states, and adjusts the way this
difference is calculated and displayed accordingly.
Today, the Pulumi engine get the first chance to decide whether or not
there is a difference between a resource's actual and desired states. It
does this by comparing the current set of inputs for a resource (i.e.
the inputs from the running Pulumi program) with the last set of inputs
used to update the resource. If there is no difference between the old
and new inputs, the engine decides that no change is necessary without
consulting the resource's provider. Only if there are changes does the
engine consult the resource's provider for more information about the
difference. This can be problematic for a number of reasons:
- Not all providers do input-input comparison; some do input-state
comparison
- Not all providers are able to update the last deployed set of inputs
when performing a refresh
- Some providers--either intentionally or due to bugs--may see changes
in resources whose inputs have not changed
All of these situations are confusing at the very least, and the first
is problematic with respect to correctness. Furthermore, the display
code only renders diffs it observes rather than rendering the diffs
observed by the provider, which can obscure the actual changes detected
at runtime.
These changes address both of these issues:
- Rather than comparing the current inputs against the last inputs
before calling a resource provider's Diff function, the engine calls
the Diff function in all cases.
- Providers may now return a list of properties that differ between the
requested and actual state and the way in which they differ. This
information will then be used by the CLI to render the diff
appropriately. A provider may also indicate that a particular diff is
between old and new inputs rather than old state and new inputs.
Fixes#2453.
Currently, if a secret was present, the value of variable "value" is used as the
key for the dictionary object containing the output. This leads to KeyError
exceptions in various places, as reported in #2782. This PR changes that to use
the literal string "value".
Fixes#2782.
Running `pulumi query` over state resources will require the ability to
filter resources by type.
This commit begins the process of making this possible simply, using the
TypeScript user-defined type guards feature. This commit changes the
`CustomResource` constructor to record the `t` argument in the
`__pulumiType` field, which acts as a "sentinel" value in the resource
base, which deriving classes need only check to ensure they are of that
type. For example:
aws.s3.Bucket.isInstance(someOb)
would check the `aws.s3.Bucket.__pulumiType` field, and make sure that
the type field lines up.
Adds a new resource option `aliases` which can be used to rename a resource. When making a breaking change to the name or type of a resource or component, the old name can be added to the list of `aliases` for a resource to ensure that existing resources will be migrated to the new name instead of being deleted and replaced with the new named resource.
There are two key places this change is implemented.
The first is the step generator in the engine. When computing whether there is an old version of a registered resource, we now take into account the aliases specified on the registered resource. That is, we first look up the resource by its new URN in the old state, and then by any aliases provided (in order). This can allow the resource to be matched as a (potential) update to an existing resource with a different URN.
The second is the core `Resource` constructor in the JavaScript (and soon Python) SDKs. This change ensures that when a parent resource is aliased, that all children implicitly inherit corresponding aliases. It is similar to how many other resource options are "inherited" implicitly from the parent.
Four specific scenarios are explicitly tested as part of this PR:
1. Renaming a resource
2. Adopting a resource into a component (as the owner of both component and consumption codebases)
3. Renaming a component instance (as the owner of the consumption codebase without changes to the component)
4. Changing the type of a component (as the owner of the component codebase without changes to the consumption codebase)
4. Combining (1) and (3) to make both changes to a resource at the same time
This commit implements read_resource functionality for Python in a
manner identical to the NodeJS implementation. If an "id" option is
passed to a resource via ResourceOptions on construction, that resource
will be read and not created.
Since we don't support nesting secrets (as they are modeled as
Outputs), as we deserialize, we push the secretness up to top level,
where we will correctly use it to mark the output as secret.
This fixes an issue where if you created a StackReference resource,
with a mix of secret and non secret properties, you would see the
"wire form" of the secrets as values on the `outputs` map of the
StackReference resource.
In our system, we model secrets as outputs with an additional bit of
metadata that says they are secret. For Read and Register resource
calls, our RPC interface says if the client side of the interface can
handle secrets being returned (i.e. the language SDK knows how to
sniff for the special signiture and resolve the output with the
special bit set).
For Invoke, we have no such model. Instead, we return a `Promise<T>`
where T's shape has just regular property fields. There's no place
for us to tack the secretness onto, since there are no Outputs.
So, for now, don't even return secret values back across the invoke
channel. We can still take them as arguments (which is good) but we
can't even return secrets as part of invoke calls. This is not ideal,
but given the way we model these sources, there's no way around
this. Fortunately, the result of these invoke calls are not stored in
the checkpoint and since the type is not Output<T> it will be clear
that the underlying value is just present in plaintext. A user that
wants to pass the result of an invoke into a resource can turn an
existing property into a secret via `pulumi.secret`.
We changed the `pulumi update` command to be `pulumi up` a while back
(`update` is an alias of `up`). This change just makes it so we refer to
the actual command, `pulumi up`, instead of the older `pulumi update`.
`pulumi query` requires that language plugins know about "query mode" so
that they don't do things like try to register the default stack
resource.
To communicate that a language host should boot into query mode, we
augment the language plugin protocol to include this information.
* NodeJS: allow callers to override provider version
* Python: allow callers to override provider version
* NodeJS: add version for invoke
* Python: add version to invoke
* NodeJS: add tests for ReadResource
* Post-merge cleanup
* update doc comments
Fixes#2277.
Adds a new ignoreChanges resource option that allows specifying a list of property names whose values will be ignored during updates. The property values will be used for Create, but will be ignored for purposes of updates, and as a result also cannot trigger replacements.
This is a feature of the Pulumi engine, not of the resource providers, so no new logic is needed in providers to support this feature. Instead, the engine simply replaces the values of input properties in the goal state with old inputs for properties marked as ignoreChanges.
Currently, only top level properties may be specified in ignoreChanges. In the future, this could be extended to support paths to nested properties (including into array elements) with a JSONPath/JMESPath syntax.
* Enable unit testing for Pulumi programs
This change enables rudimentary unit testing of your Pulumi programs, by introducing a `PULUMI_TEST_MODE` envvar that, when set, allows programs to run without a CLI. That includes
* Just being able to import your Pulumi modules, and test ordinary functions -- which otherwise would have often accidentally triggered the "Not Running in a CLI" error message
* Being able to verify a subset of resource properties and shapes, with the caveat that outputs are not included, due to the fact that this is a perpetual "dry run" without any engine operations occurring
In principle, this also means you can attach a debugger and step through your code.
* Finish the unit testing features
This change
1) Incorporates CR feedback, namely requiring that test mode be
explicitly enabled for any of this to work.
2) Implements Python support for the same capabilities.
3) Includes tests for both JavaScript and Python SDKs.
* Add a note on unit testing to the CHANGELOG
* Use Node 8 friendly assert API
* Embellish the CHANGELOG entry a bit
In pursuit of pulumi/pulumi#2389, this commit adds the necessary changes
to the resource monitor protocol so that language hosts can communicate
exactly what version of a provider should be used when servicing an
Invoke, ReadResource, or RegisterResource. The expectation here is that,
if a language host provides a version, the engine MUST use EXACTLY that
version of a provider plugin in order to service the request.
This makes the Python SDK's Config constructor name argument
optional, similar to what we do in the Node.js SDK. If not supplied,
the current project name is used as the default.
