These changes restore a more-correct version of the behavior that was
disabled with #3014. The original implementation of this behavior was
done in the SDKs, which do not have access to the complete inputs for a
resource (in particular, default values filled in by the provider during
`Check` are not exposed to the SDK). This lack of information meant that
the resolved output values could disagree with the typings present in
a provider SDK. Exacerbating this problem was the fact that unknown
values were dropped entirely, causing `undefined` values to appear in
unexpected places.
By doing this in the engine and allowing unknown values to be
represented in a first-class manner in the SDK, we can attack both of
these issues.
Although this behavior is not _strictly_ consistent with respect to the
resource model--in an update, a resource's output properties will come
from its provider and may differ from its input properties--this
behavior was present in the product for a fairly long time without
significant issues. In the future, we may be able to improve the
accuracy of resource outputs during a preview by allowing the provider
to dry-run CRUD operations and return partially-known values where
possible.
These changes also introduce new APIs in the Node and Python SDKs
that work with unknown values in a first-class fashion:
- A new parameter to the `apply` function that indicates that the
callback should be run even if the result of the apply contains
unknown values
- `containsUnknowns` and `isUnknown`, which return true if a value
either contains nested unknown values or is exactly an unknown value
- The `Unknown` type, which represents unknown values
The primary use case for these APIs is to allow nested, properties with
known values to be accessed via the lifted property accessor even when
the containing property is not fully know. A common example of this
pattern is the `metadata.name` property of a Kubernetes `Namespace`
object: while other properties of the `metadata` bag may be unknown,
`name` is often known. These APIs allow `ns.metadata.name` to return a
known value in this case.
In order to avoid exposing downlevel SDKs to unknown values--a change
which could break user code by exposing it to unexpected values--a
language SDK must indicate whether or not it supports first-class
unknown values as part of each `RegisterResourceRequest`.
These changes also allow us to avoid breaking user code with the new
behavior introduced by the prior commit.
Fixes#3190.
If a stack output includes a `Resource`, we will as of a recent change
always show the output diff, but this diff will potentially include
unknowns, leading to spurious output like:
```
+ namePrefix : output<string>
```
These changes supress these diffs by adding a special key to the POJO
we generate for resources *during preview only* that indicates that the
POJO represents a Pulumi resource, then stripping all adds of unknown
values from diffs for objects marked with that key.
Fixes#3314.
Adds Python support for resource transformations aligned with the existing NodeJS support in #3174.
This PR also moves processing of transformations to earlier in the resource construction process (for both NodeJS and Python) to ensure that invariants established in the constructor cannot be violated by transformations. This change can technically be a breaking change, but given that (a) the transformations features was just released in 1.3.0 and (b) the cases where this is a breaking change are uncommon and unlikely to have been reliable anyway - it feels like a change we should make now.
Fixes#3283.
These changes restore a more-correct version of the behavior that was
disabled with #3014. The original implementation of this behavior was
done in the SDKs, which do not have access to the complete inputs for a
resource (in particular, default values filled in by the provider during
`Check` are not exposed to the SDK). This lack of information meant that
the resolved output values could disagree with the typings present in
a provider SDK. Exacerbating this problem was the fact that unknown
values were dropped entirely, causing `undefined` values to appear in
unexpected places.
By doing this in the engine and allowing unknown values to be
represented in a first-class manner in the SDK, we can attack both of
these issues.
Although this behavior is not _strictly_ consistent with respect to the
resource model--in an update, a resource's output properties will come
from its provider and may differ from its input properties--this
behavior was present in the product for a fairly long time without
significant issues. In the future, we may be able to improve the
accuracy of resource outputs during a preview by allowing the provider
to dry-run CRUD operations and return partially-known values where
possible.
These changes also introduce new APIs in the Node and Python SDKs
that work with unknown values in a first-class fashion:
- A new parameter to the `apply` function that indicates that the
callback should be run even if the result of the apply contains
unknown values
- `containsUnknowns` and `isUnknown`, which return true if a value
either contains nested unknown values or is exactly an unknown value
- The `Unknown` type, which represents unknown values
The primary use case for these APIs is to allow nested, properties with
known values to be accessed via the lifted property accessor even when
the containing property is not fully know. A common example of this
pattern is the `metadata.name` property of a Kubernetes `Namespace`
object: while other properties of the `metadata` bag may be unknown,
`name` is often known. These APIs allow `ns.metadata.name` to return a
known value in this case.
In order to avoid exposing downlevel SDKs to unknown values--a change
which could break user code by exposing it to unexpected values--a
language SDK must indicate whether or not it supports first-class
unknown values as part of each `RegisterResourceRequest`.
These changes also allow us to avoid breaking user code with the new
behavior introduced by the prior commit.
Fixes#3190.
We had been using `pandoc` to convert our README.md into a README.rst
for use with `setup.py` and the python package ecosystem. It turns out
that we can use markdown if we set a content type. So let's do that
and make things a little simpler.
While I was in the area, I made the encoding of UTF-8 explicit when
opening README.md.
- Do not use a non-zero-to-zero transition in the number of outstanding
RPCs to determine the completion of a Python program until after the
synchronous piece of the program has finished running is complete
- Instead of using a future to indicate that either a) a zero-to-one
transition in the number of outstanding RPCs has occurred, or b) an
unhandled exception has occurred, a) observe the transition itself,
and b) use an optional exception field to track the presence or
absence of an exception.
Fixes#3162.
We intend to replace PULUMI_TEST_MODE with better testing support
that doesn't suffer from all the pitfalls of our current approach.
Unfortunately, we don't yet have complete guidance or validation
that the new approaches will work for all existing end users. So,
until we do, we'll take a lighter touch approach here, and simply
not encourage new usage of PULUMI_TEST_MODE.
Issue #3045 will remain open to track a mroe permanent fix.
With these changes, a user may explicitly set `deleteBeforeReplace` to
`false` in order to disable DBR behavior for a particular resource. This
is the SDK + CLI escape hatch for cases where the changes in
https://github.com/pulumi/pulumi-terraform/pull/465 cause undesirable
behavior.
When using StackReference, if the stack you reference contains any
secret outputs, we have to mark the entire `outputs` member as a
secret output. This is because we only track secretness on a per
`Output<T>` basis.
For `getSecret` and friends, however, we know the name of the output
you are looking up and we can be smarter about if the returned
`Output<T>` should be treated as a secret or not.
This change augments the provider for StackReference such that it also
returns a list of top level stack output names who's values contain
secrets. In the language SDKs, we use this information, when present,
to decide if we should return an `Output<T>` that is marked as a
secret or not. Since the SDK and CLI are independent components, care
is taken to ensure that when the CLI does not return this information,
we behave as we did before (i.e. if any output is a secret, we treat
every output as a secret).
Fixes#2744
_sync_await was not reentrant with respect to _run_once: the latter
captures the length of the ready list before it iterates it, and the
former drains the ready list by reentering _run_once. Fix this by
tracking the length of the list before pumping the event loop and then
pushing cancelled handles on to the list as necessary after pumping the
loop.
These changes also fix an issue with `export`ing awaitables.
Fixes#3038.
These changes fix a bug in the Python runtime that would cause any
awaitable input properties passed to a resource that are missing
from the resource's output properties to be awaited twice. The fix is
straightforward: rather than roundtripping an input property through
serialize/deserialize, just deserialized the already-serialized input
property.
Fixes#2940.
These changes make the `pulumi.runtime.invoke` function invokable in a
synchronous manner. Because this function still needs to perform
asynchronous work under the covers--namely awaiting a provider URN and
ID if a provider instance is present in the `InvokeOptions`--this
requires some creativity. This creativity comes in the form of a helper
function, `_sync_await`, that performs a logical yield from the
currently running event, manually runs the event loop until the given
future completes, performs a logical resume back to the
currently executing event, and returns the result of the future.
The code in `_sync_await` is a bit scary, as it relies upon knowledge of
(and functions in) the internals of the `asyncio` package. The necessary
work performed in this function was derived from the implementations of
`task_step` (which pointed out the need to call `_{enter,leave}_task`)
and `BaseEventLoop.run_forever` (which illustrated how the event loop is
pumped). In addition to potential breaking changes to these internals,
the code may not work if a user has provided an alternative implementation
for `EventLoop`. That said, the code is a close enough copy of
`BaseEventLoop.run_forever` that it should be a reasonable solution.
Provides an additional helper function to read outputs from a stack reference in case it is known that the stack output must be present. This is similar to the design for config.get and config.require.
Fixes#2343.
These changes add support for passing `ignoreChanges` paths to resource
providers. This is intended to accommodate providers that perform diffs
between resource inputs and resource state (e.g. all Terraform-based
providers, the k8s provider when using API server dry-runs). These paths
are specified using the same syntax as the paths used in detailed diffs.
In addition to passing these paths to providers, the existing support
for `ignoreChanges` in inputs has been extended to accept paths rather
than top-level keys. It is an error to specify a path that is missing
one or more component in the old or new inputs.
Fixes#2936, #2663.
* Remove pulumi.io reference in tests
* Remove pulumi.io references in Dockerfiles
* Remove pulumi.io references in CONTRIBUTING.md
* Update README.md's
* Use correct logo
Dynamic providers in Python.
This PR uses [dill](https://pypi.org/project/dill/) for code serialization, along with a customization to help ensure deterministic serialization results.
One notable limitation - which I believe is a general requirement of Python - is that any serialization of Python functions must serialize byte code, and byte code is not safely versioned across Python versions. So any resource created with Python `3.x.y` can only be updated by exactly the same version of Python. This is very constraining, but it's not clear there is any other option within the realm of what "dynamic providers" are as a feature. It is plausible that we could ensure that updates which only update the serialized provider can avoid calling the dynamic provider operations, so that version updates could still be accomplished. We can explore this separately.
```py
from pulumi import ComponentResource, export, Input, Output
from pulumi.dynamic import Resource, ResourceProvider, CreateResult, UpdateResult
from typing import Optional
from github import Github, GithubObject
auth = "<auth token>"
g = Github(auth)
class GithubLabelArgs(object):
owner: Input[str]
repo: Input[str]
name: Input[str]
color: Input[str]
description: Optional[Input[str]]
def __init__(self, owner, repo, name, color, description=None):
self.owner = owner
self.repo = repo
self.name = name
self.color = color
self.description = description
class GithubLabelProvider(ResourceProvider):
def create(self, props):
l = g.get_user(props["owner"]).get_repo(props["repo"]).create_label(
name=props["name"],
color=props["color"],
description=props.get("description", GithubObject.NotSet))
return CreateResult(l.name, {**props, **l.raw_data})
def update(self, id, _olds, props):
l = g.get_user(props["owner"]).get_repo(props["repo"]).get_label(id)
l.edit(name=props["name"],
color=props["color"],
description=props.get("description", GithubObject.NotSet))
return UpdateResult({**props, **l.raw_data})
def delete(self, id, props):
l = g.get_user(props["owner"]).get_repo(props["repo"]).get_label(id)
l.delete()
class GithubLabel(Resource):
name: Output[str]
color: Output[str]
url: Output[str]
description: Output[str]
def __init__(self, name, args: GithubLabelArgs, opts = None):
full_args = {'url':None, 'description':None, 'name':None, 'color':None, **vars(args)}
super().__init__(GithubLabelProvider(), name, full_args, opts)
label = GithubLabel("foo", GithubLabelArgs("lukehoban", "todo", "mylabel", "d94f0b"))
export("label_color", label.color)
export("label_url", label.url)
```
Fixes https://github.com/pulumi/pulumi/issues/2902.
`GetAnalyzerInfo` is an RPC command that allows the Pulumi CLI to ask an
`Analyzer` for metadata about the resource policies it defines.
This is useful because the Pulumi service needs to be able to render
metadata about a policy pack after the user runs a `pulumi policy
publish`. Since we don't execute code on the server, the first step of
`policy publish` is to interrogate the policy pack program about what
policies it has, so that this metadata can be sent to the server, along
with a packed version of those policies.
* Plumbing the custom timeouts from the engine to the providers
* Plumbing the CustomTimeouts through to the engine and adding test to show this
* Change the provider proto to include individual timeouts
* Plumbing the CustomTimeouts from the engine through to the Provider RPC interface
* Change how the CustomTimeouts are sent across RPC
These errors were spotted in testing. We can now see that the timeout
information is arriving in the RegisterResourceRequest
```
req=&pulumirpc.RegisterResourceRequest{
Type: "aws:s3/bucket:Bucket",
Name: "my-bucket",
Parent: "urn:pulumi:dev::aws-vpc::pulumi:pulumi:Stack::aws-vpc-dev",
Custom: true,
Object: &structpb.Struct{},
Protect: false,
Dependencies: nil,
Provider: "",
PropertyDependencies: {},
DeleteBeforeReplace: false,
Version: "",
IgnoreChanges: nil,
AcceptSecrets: true,
AdditionalSecretOutputs: nil,
Aliases: nil,
CustomTimeouts: &pulumirpc.RegisterResourceRequest_CustomTimeouts{
Create: 300,
Update: 400,
Delete: 500,
XXX_NoUnkeyedLiteral: struct {}{},
XXX_unrecognized: nil,
XXX_sizecache: 0,
},
XXX_NoUnkeyedLiteral: struct {}{},
XXX_unrecognized: nil,
XXX_sizecache: 0,
}
```
* Changing the design to use strings
* CHANGELOG entry to include the CustomTimeouts work
* Changing custom timeouts to be passed around the engine as converted value
We don't want to pass around strings - the user can provide it but we want
to make the engine aware of the timeout in seconds as a float64
A resource can be imported by setting the `import` property in the
resource options bag when instantiating a resource. In order to
successfully import a resource, its desired configuration (i.e. its
inputs) must not differ from its actual configuration (i.e. its state)
as calculated by the resource's provider.
There are a few interesting state transitions hiding here when importing
a resource:
1. No prior resource exists in the checkpoint file. In this case, the
resource is simply imported.
2. An external resource exists in the checkpoint file. In this case, the
resource is imported and the old external state is discarded.
3. A non-external resource exists in the checkpoint file and its ID is
different from the ID to import. In this case, the new resource is
imported and the old resource is deleted.
4. A non-external resource exists in the checkpoint file, but the ID is
the same as the ID to import. In this case, the import ID is ignored
and the resource is treated as it would be in all cases except for
changes that would replace the resource. In that case, the step
generator issues an error that indicates that the import ID should be
removed: were we to move forward with the replace, the new state of
the stack would fall under case (3), which is almost certainly not
what the user intends.
Fixes#1662.
There current RPC model for Pulumi allows secret values to be deeply
embedded in lists or maps, however at the language level, since we
track secrets via `Output<T>` we need to ensure that during
deserialization, if a list or a map contains a secret, we need to
instead treat it as if the entire list or map was a secret.
We have logic in the language runtimes to do this as part of
serialization. There were a few issues this commit addresses:
- We were not promoting secretness across arrays in either Node or
Python
- For Python, our promotion logic was buggy and caused it to behave in
a manner where if any value was secret, the output values of the
object would be corrupted, because we'd incorrectly treat the
outputs as a secret who's value was a map, instead of a map of
values (some of which may be secret).
This caused very confusing behavior, because it would appear that a
resource creation call just did not set various output properties when
one or more of them ended up containing a secret.
Thse changes make a subtle but critical adjustment to the process the
Pulumi engine uses to determine whether or not a difference exists
between a resource's actual and desired states, and adjusts the way this
difference is calculated and displayed accordingly.
Today, the Pulumi engine get the first chance to decide whether or not
there is a difference between a resource's actual and desired states. It
does this by comparing the current set of inputs for a resource (i.e.
the inputs from the running Pulumi program) with the last set of inputs
used to update the resource. If there is no difference between the old
and new inputs, the engine decides that no change is necessary without
consulting the resource's provider. Only if there are changes does the
engine consult the resource's provider for more information about the
difference. This can be problematic for a number of reasons:
- Not all providers do input-input comparison; some do input-state
comparison
- Not all providers are able to update the last deployed set of inputs
when performing a refresh
- Some providers--either intentionally or due to bugs--may see changes
in resources whose inputs have not changed
All of these situations are confusing at the very least, and the first
is problematic with respect to correctness. Furthermore, the display
code only renders diffs it observes rather than rendering the diffs
observed by the provider, which can obscure the actual changes detected
at runtime.
These changes address both of these issues:
- Rather than comparing the current inputs against the last inputs
before calling a resource provider's Diff function, the engine calls
the Diff function in all cases.
- Providers may now return a list of properties that differ between the
requested and actual state and the way in which they differ. This
information will then be used by the CLI to render the diff
appropriately. A provider may also indicate that a particular diff is
between old and new inputs rather than old state and new inputs.
Fixes#2453.
Currently, if a secret was present, the value of variable "value" is used as the
key for the dictionary object containing the output. This leads to KeyError
exceptions in various places, as reported in #2782. This PR changes that to use
the literal string "value".
Fixes#2782.
Running `pulumi query` over state resources will require the ability to
filter resources by type.
This commit begins the process of making this possible simply, using the
TypeScript user-defined type guards feature. This commit changes the
`CustomResource` constructor to record the `t` argument in the
`__pulumiType` field, which acts as a "sentinel" value in the resource
base, which deriving classes need only check to ensure they are of that
type. For example:
aws.s3.Bucket.isInstance(someOb)
would check the `aws.s3.Bucket.__pulumiType` field, and make sure that
the type field lines up.
Adds a new resource option `aliases` which can be used to rename a resource. When making a breaking change to the name or type of a resource or component, the old name can be added to the list of `aliases` for a resource to ensure that existing resources will be migrated to the new name instead of being deleted and replaced with the new named resource.
There are two key places this change is implemented.
The first is the step generator in the engine. When computing whether there is an old version of a registered resource, we now take into account the aliases specified on the registered resource. That is, we first look up the resource by its new URN in the old state, and then by any aliases provided (in order). This can allow the resource to be matched as a (potential) update to an existing resource with a different URN.
The second is the core `Resource` constructor in the JavaScript (and soon Python) SDKs. This change ensures that when a parent resource is aliased, that all children implicitly inherit corresponding aliases. It is similar to how many other resource options are "inherited" implicitly from the parent.
Four specific scenarios are explicitly tested as part of this PR:
1. Renaming a resource
2. Adopting a resource into a component (as the owner of both component and consumption codebases)
3. Renaming a component instance (as the owner of the consumption codebase without changes to the component)
4. Changing the type of a component (as the owner of the component codebase without changes to the consumption codebase)
4. Combining (1) and (3) to make both changes to a resource at the same time
This commit implements read_resource functionality for Python in a
manner identical to the NodeJS implementation. If an "id" option is
passed to a resource via ResourceOptions on construction, that resource
will be read and not created.
Since we don't support nesting secrets (as they are modeled as
Outputs), as we deserialize, we push the secretness up to top level,
where we will correctly use it to mark the output as secret.
This fixes an issue where if you created a StackReference resource,
with a mix of secret and non secret properties, you would see the
"wire form" of the secrets as values on the `outputs` map of the
StackReference resource.
In our system, we model secrets as outputs with an additional bit of
metadata that says they are secret. For Read and Register resource
calls, our RPC interface says if the client side of the interface can
handle secrets being returned (i.e. the language SDK knows how to
sniff for the special signiture and resolve the output with the
special bit set).
For Invoke, we have no such model. Instead, we return a `Promise<T>`
where T's shape has just regular property fields. There's no place
for us to tack the secretness onto, since there are no Outputs.
So, for now, don't even return secret values back across the invoke
channel. We can still take them as arguments (which is good) but we
can't even return secrets as part of invoke calls. This is not ideal,
but given the way we model these sources, there's no way around
this. Fortunately, the result of these invoke calls are not stored in
the checkpoint and since the type is not Output<T> it will be clear
that the underlying value is just present in plaintext. A user that
wants to pass the result of an invoke into a resource can turn an
existing property into a secret via `pulumi.secret`.
We changed the `pulumi update` command to be `pulumi up` a while back
(`update` is an alias of `up`). This change just makes it so we refer to
the actual command, `pulumi up`, instead of the older `pulumi update`.
`pulumi query` requires that language plugins know about "query mode" so
that they don't do things like try to register the default stack
resource.
To communicate that a language host should boot into query mode, we
augment the language plugin protocol to include this information.
* NodeJS: allow callers to override provider version
* Python: allow callers to override provider version
* NodeJS: add version for invoke
* Python: add version to invoke
* NodeJS: add tests for ReadResource
* Post-merge cleanup
* update doc comments
Fixes#2277.
Adds a new ignoreChanges resource option that allows specifying a list of property names whose values will be ignored during updates. The property values will be used for Create, but will be ignored for purposes of updates, and as a result also cannot trigger replacements.
This is a feature of the Pulumi engine, not of the resource providers, so no new logic is needed in providers to support this feature. Instead, the engine simply replaces the values of input properties in the goal state with old inputs for properties marked as ignoreChanges.
Currently, only top level properties may be specified in ignoreChanges. In the future, this could be extended to support paths to nested properties (including into array elements) with a JSONPath/JMESPath syntax.
* Enable unit testing for Pulumi programs
This change enables rudimentary unit testing of your Pulumi programs, by introducing a `PULUMI_TEST_MODE` envvar that, when set, allows programs to run without a CLI. That includes
* Just being able to import your Pulumi modules, and test ordinary functions -- which otherwise would have often accidentally triggered the "Not Running in a CLI" error message
* Being able to verify a subset of resource properties and shapes, with the caveat that outputs are not included, due to the fact that this is a perpetual "dry run" without any engine operations occurring
In principle, this also means you can attach a debugger and step through your code.
* Finish the unit testing features
This change
1) Incorporates CR feedback, namely requiring that test mode be
explicitly enabled for any of this to work.
2) Implements Python support for the same capabilities.
3) Includes tests for both JavaScript and Python SDKs.
* Add a note on unit testing to the CHANGELOG
* Use Node 8 friendly assert API
* Embellish the CHANGELOG entry a bit
In pursuit of pulumi/pulumi#2389, this commit adds the necessary changes
to the resource monitor protocol so that language hosts can communicate
exactly what version of a provider should be used when servicing an
Invoke, ReadResource, or RegisterResource. The expectation here is that,
if a language host provides a version, the engine MUST use EXACTLY that
version of a provider plugin in order to service the request.
This makes the Python SDK's Config constructor name argument
optional, similar to what we do in the Node.js SDK. If not supplied,
the current project name is used as the default.
These changes take advantage of the newly-added support for returning
inputs from Read to update a resource's inputs as part of a refresh.
As a consequence, the Pulumi engine will now properly detect drift
between the actual state of a resource and the desired state described
by the program and generate appropriate update or replace steps.
As part of these changes, a resource's old inputs are now passed to the
provider when performing a refresh. The provider can take advantage of
this to maintain the accuracy of any additional data or metadata in the
resource's inputs that may need to be updated during the refresh.
This is required for the complete implementation of
https://github.com/pulumi/pulumi-terraform/pull/349. Without access to
the old inputs for a resource, TF-based providers would lose all
information about default population during a refresh.
These changes add two new methods to the provider interface and extend
the results of three others.
The new methods are `CheckConfig` and `DiffConfig`, which fill out the
set of methods required for a complete implementation of the
first-class provider design. Though these methods are optional for
backwards compatibility, they should be implemented by all future
providers for the best possible user experience.
The adjusted result types are `DiffResponse`, `ReadResponse`, and
`ErrorResourceInitFailed`. The first has been updated to include a list
of the properties that changed (if any). The latter two now include
an estimated set of inputs for the resource as well as the resource's
state. Together, these three changes enable the engine to determine the
set of inputs that should be specified by a user in order to match those
that describe the resource's current state.
This contributes to #2453, #1662, #1635, and #1718.
* Fix an issue with empty ID for CustomResource
The Python runtime was checking the ID field it receives from the engine
against None, assuming that the engine would not set the ID field if one
was not present. However, it does set the ID field; it is set to the
empty string when an ID is not known.
This commit fixes an issue that can cause certain IDs to be erroneously
considered to be known during previews, which can cause problems during
the Check phase of resources that directly reference IDs of other
resources.
* Add CHANGELOG
These changes add a new flag to the various `ResourceOptions` types that
indicates that a resource should be deleted before it is replaced, even
if the provider does not require this behavior. The usual
delete-before-replace cascade semantics apply.
Fixes#1620.
* Clean up documentation for Python SDK
The Python SDK currently does a couple of bad things that make it
difficult to generate documentation:
1. It "wildcard-imports" submodules without each module specifying an
__all__ member
2. Documentation strings don't have a consistent format
3. Documentation strings are in Markdown and not reStructuredText
To remedy this, this commit addresses 1 by explicitly specifying which
members are being exported from submodules, so that we can see in one
place exactly what the public surface area of the pulumi package is. For
2 and 3, this commit fixes a large number of documentation strings to
contain metadata tags that Sphinx is capable of reading. This allows us
to generate high-quality documentation directly from the source without
having to manually parse docstrings.
* Please pylint
- Add support for per-property dependencies to the Go SDK
- Add tests for first-class secret rejection in the checkpoint and RPC
layers and language SDKs
This implements the new algorithm for deciding which resources must be
deleted due to a delete-before-replace operation.
We need to compute the set of resources that may be replaced by a
change to the resource under consideration. We do this by taking the
complete set of transitive dependents on the resource under
consideration and removing any resources that would not be replaced by
changes to their dependencies. We determine whether or not a resource
may be replaced by substituting unknowns for input properties that may
change due to deletion of the resources their value depends on and
calling the resource provider's Diff method.
This is perhaps clearer when described by example. Consider the
following dependency graph:
A
__|__
B C
| _|_
D E F
In this graph, all of B, C, D, E, and F transitively depend on A. It may
be the case, however, that changes to the specific properties of any of
those resources R that would occur if a resource on the path to A were
deleted and recreated may not cause R to be replaced. For example, the
edge from B to A may be a simple dependsOn edge such that a change to
B does not actually influence any of B's input properties. In that case,
neither B nor D would need to be deleted before A could be deleted.
In order to make the above algorithm a reality, the resource monitor
interface has been updated to include a map that associates an input
property key with the list of resources that input property depends on.
Older clients of the resource monitor will leave this map empty, in
which case all input properties will be treated as depending on all
dependencies of the resource. This is probably overly conservative, but
it is less conservative than what we currently implement, and is
certainly correct.
On startup, when we were populating the Settings object, we failed to
coerce the dry_run parameter from a string to a boolean, which resulted
in is_dry_run always believing that it was a preview. This PR fixes this
oversight by explicitly coercing to a boolean prior to sending the value
to Settings.
When is resource is waiting for its dependencies to resolve, it first
informs the RPC_MANAGER that it intends to do an RPC - this is to
prevent premature termination of the program while RPCs are still in
flight or queued to execute.
However, this is a problem whenever a resource fails to create for
whatever reason. The most common ways for this to happen are for invokes
to fail, resource creation itself to fail, or throwing in an apply.
Today, this causes a deadlock, since all consumers of the failed
resources block forever while never decrementing the RPC count.
This commit addresses the issue by deliberately (abnormally) resolving
all futures that are created in the process of preparing a resource.
This solves the problem by immediately terminating all resources that
are waiting for the failed resource's outputs to resolve - they resolve
immediately, and exceptionally.
The end result is now that, instead of deadlocking, a doomed program now
terminates as expected with a single exception message.
When previewing a real first-class provider, it is often the case that
the provider's ID is unknown during a preview. This commit fixes a bug
where we did not translate an unknown ID into the rpc layer's sentinel
UNKNOWN value where we should have, which caused the engine to fail to
resolve the provider reference.
Invoke in Node.js allows users to optionally pass a parent or a provider
to the invoke, which dictates either explicitly or implicitly which
provider to use when performing an invoke. If a provider is specified
explicitly, that provider is used to perform the invoke. If a parent is
specified, that parent's provider is used to perform the invoke.
* Implement first-class providers for Python
First-class providers are an explicit projection of providers themselves
into Pulumi programs. For the most post, providers are just regular
resources, but the addition of providers to the fray (and the ability of
resources to be constructed by providers in the same program) requires
some changes to the Python resource model.
A summary of the changes:
1. Added ProviderResource, a custom resource that is the base class of
all providers.
2. ResourceOptions now has 'provider' and 'providers' fields.
'provider', when passed to a custom resource, allows users to override
the provider that is used to construct a resource to an instance of a
ProviderResource. 'providers', when passed to a component resource,
allows users to override providers used to construct children of the
component resource.
3. 'protect', 'providers', and 'provider' are all now inherited from
a resource's parent if they aren't specified in the child.
This commit adds the requisite code for the above changes and, in
addition, adds a number of new tests that exercise them and related code
paths.
* Rebase against master
* Add 'Output.all' combinator for Python
Output.all is a useful combinator that we already have in Node that
allows the composition of a list of outputs into an output of a list.
This is very useful when authoring components and its lack of presence
in Python was an oversight.
This commit adds 'Output.all' and 'Output.from_input', adding tests and
documentation for each.
* start unwrap
* Add functionality and test for nested inputs
* Fix, formalize and add tests for property rewrites
The Python SDK provides two hooks for resources to override how their
properties are communicated to and from the engine. The code that
performs this transformation is subtle and, before this commit, subtly
incorrect.
This commit adds a test that verifies that the SDK correctly transforms
properties recursively according to the two transformation hooks, while
also fixing a smattering of test issues encountered when adding the new
test.
* CR feedback
* Implement RegisterResourceOutputs for Python 3
RegisterResourceOutputs allows Python 3 programs to export stack outputs
and export outputs off of component resources (which, under the hood,
are the same thing).
Adds a new integration test for stack outputs for Python programs, as
well as add a langhost test for register resource outputs.
Fixespulumi/pulumi#2163
* CR: Rename stack_output -> export
Fix integration tests that hardcoded paths to stack_outputs
* Fix one more reference to stack_outputs
future_input tests that it's possible to use coroutines as inputs to
Pulumi resources. resource_thens tests that it's possible to use outputs
to chain resource inputs and outputs together and that the SDK reports
correct dependencies to the engine.
This PR also fixes two bugs exposed by the new tests: first, coroutines
needed to be scheduled before awaiting (otherwise we'd deadlock) and
Nones in maps needed to be ignored when serializing and deserializing.