This change allows importing modules with calls to `pulumi.export` in unit tests. Previously, you'd have to structure the Python program in a way that avoids the `pulumi.export` from being called from unit tests.
* [sdk/python] Improve `ResoruceOptions.merge` type
The implementation correctly handles `None` inputs, so the type should allow these as well.
* Add CHANGELOG
* Make `async:true` the default for `invoke` calls (#3750)
* Switch away from native grpc impl. (#3728)
* Remove usage of the 'deasync' library from @pulumi/pulumi. (#3752)
* Only retry as long as we get unavailable back. Anything else continues. (#3769)
* Handle all errors for now. (#3781)
* Do not assume --yes was present when using pulumi in non-interactive mode (#3793)
* Upgrade all paths for sdk and pkg to v2
* Backport C# invoke classes and other recent gen changes (#4288)
Adjust C# generation
* Replace IDeployment with a sealed class (#4318)
Replace IDeployment with a sealed class
* .NET: default to args subtype rather than Args.Empty (#4320)
* Adding system namespace for Dotnet code gen
This is required for using Obsolute attributes for deprecations
```
Iam/InstanceProfile.cs(142,10): error CS0246: The type or namespace name 'ObsoleteAttribute' could not be found (are you missing a using directive or an assembly reference?) [/Users/stack72/code/go/src/github.com/pulumi/pulumi-aws/sdk/dotnet/Pulumi.Aws.csproj]
Iam/InstanceProfile.cs(142,10): error CS0246: The type or namespace name 'Obsolete' could not be found (are you missing a using directive or an assembly reference?) [/Users/stack72/code/go/src/github.com/pulumi/pulumi-aws/sdk/dotnet/Pulumi.Aws.csproj]
```
* Fix the nullability of config type properties in C# codegen (#4379)
The initial config represents any config that was specified programmatically to the Policy Pack, for Policy Packs that support programmatic configuration like AWSGuard.
It's not entirely clear why gRPC doesn't already report these cleanly as unimplemented, but for now we'll explicitly implement them to avoid any spurious warnings.
Fixes#4028.
The original version of this code caused inconsistencies in the event
loop associated with a given thread. These changes elimintate the event
loop shenanigans the mocks were trying to play by updating _sync_await
to create an event loop if none exists in the current thread.
It's possible that this will cause problems if the tests run on a
different thread than the original program, as the tests are likely to
end up waiting on outputs created by the program, which is not supported
in Python.
Also adds test coverage of the mocking/testing support in Python.
These changes add support for mocking the resource monitor to the NodeJS
and Python SDKs. The proposed mock interface is a simplified version of
the standard resource monitor that allows an end-user to replace the
usual implementations of ReadResource/RegisterResource and Invoke with
their own. This can be used in unit tests to allow for precise control
of resource outputs and invoke results.
These changes add a new method to the resource provider gRPC interface,
`GetSchema`, that allows consumers of these providers to extract
JSON-serialized schema information for the provider's types, resources,
and functions.
Set an option to increase the memory limit on protobuf
parsing so that we can handle larger gRPC payloads.
Co-authored-by: Evan Boyle <EvanBoyle@users.noreply.github.com>
The provider plugin protocol is to write a port number followed by `\n`. We must guarantee we do that even on Windows, so must avoid Python `print` statements which implicitly rewrite newlines to platform specific character sequences.
Fixes#3807.
PEP 561 specifies that packages which contain either inline type hints
or type stubs should indicate their support for type hints via
including a file named `py.typed` in the root of the package. Since
Pulumi already includes inline type hints, adding `py.typed` to the
Python SDK simply allows these hints to be used by mypy.
Earlier changes to the get-version script were not adopted by the Python
SDK Makefile. This caused package publishing to skip uploads due to
missing versions.
These changes restore a more-correct version of the behavior that was
disabled with #3014. The original implementation of this behavior was
done in the SDKs, which do not have access to the complete inputs for a
resource (in particular, default values filled in by the provider during
`Check` are not exposed to the SDK). This lack of information meant that
the resolved output values could disagree with the typings present in
a provider SDK. Exacerbating this problem was the fact that unknown
values were dropped entirely, causing `undefined` values to appear in
unexpected places.
By doing this in the engine and allowing unknown values to be
represented in a first-class manner in the SDK, we can attack both of
these issues.
Although this behavior is not _strictly_ consistent with respect to the
resource model--in an update, a resource's output properties will come
from its provider and may differ from its input properties--this
behavior was present in the product for a fairly long time without
significant issues. In the future, we may be able to improve the
accuracy of resource outputs during a preview by allowing the provider
to dry-run CRUD operations and return partially-known values where
possible.
These changes also introduce new APIs in the Node and Python SDKs
that work with unknown values in a first-class fashion:
- A new parameter to the `apply` function that indicates that the
callback should be run even if the result of the apply contains
unknown values
- `containsUnknowns` and `isUnknown`, which return true if a value
either contains nested unknown values or is exactly an unknown value
- The `Unknown` type, which represents unknown values
The primary use case for these APIs is to allow nested, properties with
known values to be accessed via the lifted property accessor even when
the containing property is not fully know. A common example of this
pattern is the `metadata.name` property of a Kubernetes `Namespace`
object: while other properties of the `metadata` bag may be unknown,
`name` is often known. These APIs allow `ns.metadata.name` to return a
known value in this case.
In order to avoid exposing downlevel SDKs to unknown values--a change
which could break user code by exposing it to unexpected values--a
language SDK must indicate whether or not it supports first-class
unknown values as part of each `RegisterResourceRequest`.
These changes also allow us to avoid breaking user code with the new
behavior introduced by the prior commit.
Fixes#3190.
If a stack output includes a `Resource`, we will as of a recent change
always show the output diff, but this diff will potentially include
unknowns, leading to spurious output like:
```
+ namePrefix : output<string>
```
These changes supress these diffs by adding a special key to the POJO
we generate for resources *during preview only* that indicates that the
POJO represents a Pulumi resource, then stripping all adds of unknown
values from diffs for objects marked with that key.
Fixes#3314.
Adds Python support for resource transformations aligned with the existing NodeJS support in #3174.
This PR also moves processing of transformations to earlier in the resource construction process (for both NodeJS and Python) to ensure that invariants established in the constructor cannot be violated by transformations. This change can technically be a breaking change, but given that (a) the transformations features was just released in 1.3.0 and (b) the cases where this is a breaking change are uncommon and unlikely to have been reliable anyway - it feels like a change we should make now.
Fixes#3283.
These changes restore a more-correct version of the behavior that was
disabled with #3014. The original implementation of this behavior was
done in the SDKs, which do not have access to the complete inputs for a
resource (in particular, default values filled in by the provider during
`Check` are not exposed to the SDK). This lack of information meant that
the resolved output values could disagree with the typings present in
a provider SDK. Exacerbating this problem was the fact that unknown
values were dropped entirely, causing `undefined` values to appear in
unexpected places.
By doing this in the engine and allowing unknown values to be
represented in a first-class manner in the SDK, we can attack both of
these issues.
Although this behavior is not _strictly_ consistent with respect to the
resource model--in an update, a resource's output properties will come
from its provider and may differ from its input properties--this
behavior was present in the product for a fairly long time without
significant issues. In the future, we may be able to improve the
accuracy of resource outputs during a preview by allowing the provider
to dry-run CRUD operations and return partially-known values where
possible.
These changes also introduce new APIs in the Node and Python SDKs
that work with unknown values in a first-class fashion:
- A new parameter to the `apply` function that indicates that the
callback should be run even if the result of the apply contains
unknown values
- `containsUnknowns` and `isUnknown`, which return true if a value
either contains nested unknown values or is exactly an unknown value
- The `Unknown` type, which represents unknown values
The primary use case for these APIs is to allow nested, properties with
known values to be accessed via the lifted property accessor even when
the containing property is not fully know. A common example of this
pattern is the `metadata.name` property of a Kubernetes `Namespace`
object: while other properties of the `metadata` bag may be unknown,
`name` is often known. These APIs allow `ns.metadata.name` to return a
known value in this case.
In order to avoid exposing downlevel SDKs to unknown values--a change
which could break user code by exposing it to unexpected values--a
language SDK must indicate whether or not it supports first-class
unknown values as part of each `RegisterResourceRequest`.
These changes also allow us to avoid breaking user code with the new
behavior introduced by the prior commit.
Fixes#3190.
* Fix some tracing issues.
- Add endpoints for `startUpdate` and `postEngineEventsBatch` so that
spans for these invocations have proper names
- Inject a tracing span when walking a plan so that resource operations
are properly parented
- When handling gRPC calls, inject a tracing span into the call's
metadata if no span is already present so that resource monitor and
engine spans are properly parented
- Do not trace client gRPC invocations of the empty method so that these
calls (which are used to determine server availability) do not muddy
the trace. Note that I tried parenting these spans appropriately, but
doing so broke the trace entirely.
With these changes, the only unparented span in a typical Pulumi
invocation is a single call to `getUser`. This span is unparented
because that call does not have a context available. Plumbing a context
into that particular call is surprisingly tricky, as it is often called
by other context-less functions.
* Make tracing support more flexible.
- Add support for writing trace data to a local file using Appdash
- Add support for viewing Appdash traces via the CLI
We had been using `pandoc` to convert our README.md into a README.rst
for use with `setup.py` and the python package ecosystem. It turns out
that we can use markdown if we set a content type. So let's do that
and make things a little simpler.
While I was in the area, I made the encoding of UTF-8 explicit when
opening README.md.
- Do not use a non-zero-to-zero transition in the number of outstanding
RPCs to determine the completion of a Python program until after the
synchronous piece of the program has finished running is complete
- Instead of using a future to indicate that either a) a zero-to-one
transition in the number of outstanding RPCs has occurred, or b) an
unhandled exception has occurred, a) observe the transition itself,
and b) use an optional exception field to track the presence or
absence of an exception.
Fixes#3162.
We intend to replace PULUMI_TEST_MODE with better testing support
that doesn't suffer from all the pitfalls of our current approach.
Unfortunately, we don't yet have complete guidance or validation
that the new approaches will work for all existing end users. So,
until we do, we'll take a lighter touch approach here, and simply
not encourage new usage of PULUMI_TEST_MODE.
Issue #3045 will remain open to track a mroe permanent fix.
With these changes, a user may explicitly set `deleteBeforeReplace` to
`false` in order to disable DBR behavior for a particular resource. This
is the SDK + CLI escape hatch for cases where the changes in
https://github.com/pulumi/pulumi-terraform/pull/465 cause undesirable
behavior.
When using StackReference, if the stack you reference contains any
secret outputs, we have to mark the entire `outputs` member as a
secret output. This is because we only track secretness on a per
`Output<T>` basis.
For `getSecret` and friends, however, we know the name of the output
you are looking up and we can be smarter about if the returned
`Output<T>` should be treated as a secret or not.
This change augments the provider for StackReference such that it also
returns a list of top level stack output names who's values contain
secrets. In the language SDKs, we use this information, when present,
to decide if we should return an `Output<T>` that is marked as a
secret or not. Since the SDK and CLI are independent components, care
is taken to ensure that when the CLI does not return this information,
we behave as we did before (i.e. if any output is a secret, we treat
every output as a secret).
Fixes#2744
_sync_await was not reentrant with respect to _run_once: the latter
captures the length of the ready list before it iterates it, and the
former drains the ready list by reentering _run_once. Fix this by
tracking the length of the list before pumping the event loop and then
pushing cancelled handles on to the list as necessary after pumping the
loop.
These changes also fix an issue with `export`ing awaitables.
Fixes#3038.
These changes fix a bug in the Python runtime that would cause any
awaitable input properties passed to a resource that are missing
from the resource's output properties to be awaited twice. The fix is
straightforward: rather than roundtripping an input property through
serialize/deserialize, just deserialized the already-serialized input
property.
Fixes#2940.
These changes make the `pulumi.runtime.invoke` function invokable in a
synchronous manner. Because this function still needs to perform
asynchronous work under the covers--namely awaiting a provider URN and
ID if a provider instance is present in the `InvokeOptions`--this
requires some creativity. This creativity comes in the form of a helper
function, `_sync_await`, that performs a logical yield from the
currently running event, manually runs the event loop until the given
future completes, performs a logical resume back to the
currently executing event, and returns the result of the future.
The code in `_sync_await` is a bit scary, as it relies upon knowledge of
(and functions in) the internals of the `asyncio` package. The necessary
work performed in this function was derived from the implementations of
`task_step` (which pointed out the need to call `_{enter,leave}_task`)
and `BaseEventLoop.run_forever` (which illustrated how the event loop is
pumped). In addition to potential breaking changes to these internals,
the code may not work if a user has provided an alternative implementation
for `EventLoop`. That said, the code is a close enough copy of
`BaseEventLoop.run_forever` that it should be a reasonable solution.
Provides an additional helper function to read outputs from a stack reference in case it is known that the stack output must be present. This is similar to the design for config.get and config.require.
Fixes#2343.
These changes add support for passing `ignoreChanges` paths to resource
providers. This is intended to accommodate providers that perform diffs
between resource inputs and resource state (e.g. all Terraform-based
providers, the k8s provider when using API server dry-runs). These paths
are specified using the same syntax as the paths used in detailed diffs.
In addition to passing these paths to providers, the existing support
for `ignoreChanges` in inputs has been extended to accept paths rather
than top-level keys. It is an error to specify a path that is missing
one or more component in the old or new inputs.
Fixes#2936, #2663.
* Remove pulumi.io reference in tests
* Remove pulumi.io references in Dockerfiles
* Remove pulumi.io references in CONTRIBUTING.md
* Update README.md's
* Use correct logo
Dynamic providers in Python.
This PR uses [dill](https://pypi.org/project/dill/) for code serialization, along with a customization to help ensure deterministic serialization results.
One notable limitation - which I believe is a general requirement of Python - is that any serialization of Python functions must serialize byte code, and byte code is not safely versioned across Python versions. So any resource created with Python `3.x.y` can only be updated by exactly the same version of Python. This is very constraining, but it's not clear there is any other option within the realm of what "dynamic providers" are as a feature. It is plausible that we could ensure that updates which only update the serialized provider can avoid calling the dynamic provider operations, so that version updates could still be accomplished. We can explore this separately.
```py
from pulumi import ComponentResource, export, Input, Output
from pulumi.dynamic import Resource, ResourceProvider, CreateResult, UpdateResult
from typing import Optional
from github import Github, GithubObject
auth = "<auth token>"
g = Github(auth)
class GithubLabelArgs(object):
owner: Input[str]
repo: Input[str]
name: Input[str]
color: Input[str]
description: Optional[Input[str]]
def __init__(self, owner, repo, name, color, description=None):
self.owner = owner
self.repo = repo
self.name = name
self.color = color
self.description = description
class GithubLabelProvider(ResourceProvider):
def create(self, props):
l = g.get_user(props["owner"]).get_repo(props["repo"]).create_label(
name=props["name"],
color=props["color"],
description=props.get("description", GithubObject.NotSet))
return CreateResult(l.name, {**props, **l.raw_data})
def update(self, id, _olds, props):
l = g.get_user(props["owner"]).get_repo(props["repo"]).get_label(id)
l.edit(name=props["name"],
color=props["color"],
description=props.get("description", GithubObject.NotSet))
return UpdateResult({**props, **l.raw_data})
def delete(self, id, props):
l = g.get_user(props["owner"]).get_repo(props["repo"]).get_label(id)
l.delete()
class GithubLabel(Resource):
name: Output[str]
color: Output[str]
url: Output[str]
description: Output[str]
def __init__(self, name, args: GithubLabelArgs, opts = None):
full_args = {'url':None, 'description':None, 'name':None, 'color':None, **vars(args)}
super().__init__(GithubLabelProvider(), name, full_args, opts)
label = GithubLabel("foo", GithubLabelArgs("lukehoban", "todo", "mylabel", "d94f0b"))
export("label_color", label.color)
export("label_url", label.url)
```
Fixes https://github.com/pulumi/pulumi/issues/2902.
`GetAnalyzerInfo` is an RPC command that allows the Pulumi CLI to ask an
`Analyzer` for metadata about the resource policies it defines.
This is useful because the Pulumi service needs to be able to render
metadata about a policy pack after the user runs a `pulumi policy
publish`. Since we don't execute code on the server, the first step of
`policy publish` is to interrogate the policy pack program about what
policies it has, so that this metadata can be sent to the server, along
with a packed version of those policies.
* Plumbing the custom timeouts from the engine to the providers
* Plumbing the CustomTimeouts through to the engine and adding test to show this
* Change the provider proto to include individual timeouts
* Plumbing the CustomTimeouts from the engine through to the Provider RPC interface
* Change how the CustomTimeouts are sent across RPC
These errors were spotted in testing. We can now see that the timeout
information is arriving in the RegisterResourceRequest
```
req=&pulumirpc.RegisterResourceRequest{
Type: "aws:s3/bucket:Bucket",
Name: "my-bucket",
Parent: "urn:pulumi:dev::aws-vpc::pulumi:pulumi:Stack::aws-vpc-dev",
Custom: true,
Object: &structpb.Struct{},
Protect: false,
Dependencies: nil,
Provider: "",
PropertyDependencies: {},
DeleteBeforeReplace: false,
Version: "",
IgnoreChanges: nil,
AcceptSecrets: true,
AdditionalSecretOutputs: nil,
Aliases: nil,
CustomTimeouts: &pulumirpc.RegisterResourceRequest_CustomTimeouts{
Create: 300,
Update: 400,
Delete: 500,
XXX_NoUnkeyedLiteral: struct {}{},
XXX_unrecognized: nil,
XXX_sizecache: 0,
},
XXX_NoUnkeyedLiteral: struct {}{},
XXX_unrecognized: nil,
XXX_sizecache: 0,
}
```
* Changing the design to use strings
* CHANGELOG entry to include the CustomTimeouts work
* Changing custom timeouts to be passed around the engine as converted value
We don't want to pass around strings - the user can provide it but we want
to make the engine aware of the timeout in seconds as a float64
A resource can be imported by setting the `import` property in the
resource options bag when instantiating a resource. In order to
successfully import a resource, its desired configuration (i.e. its
inputs) must not differ from its actual configuration (i.e. its state)
as calculated by the resource's provider.
There are a few interesting state transitions hiding here when importing
a resource:
1. No prior resource exists in the checkpoint file. In this case, the
resource is simply imported.
2. An external resource exists in the checkpoint file. In this case, the
resource is imported and the old external state is discarded.
3. A non-external resource exists in the checkpoint file and its ID is
different from the ID to import. In this case, the new resource is
imported and the old resource is deleted.
4. A non-external resource exists in the checkpoint file, but the ID is
the same as the ID to import. In this case, the import ID is ignored
and the resource is treated as it would be in all cases except for
changes that would replace the resource. In that case, the step
generator issues an error that indicates that the import ID should be
removed: were we to move forward with the replace, the new state of
the stack would fall under case (3), which is almost certainly not
what the user intends.
Fixes#1662.
There current RPC model for Pulumi allows secret values to be deeply
embedded in lists or maps, however at the language level, since we
track secrets via `Output<T>` we need to ensure that during
deserialization, if a list or a map contains a secret, we need to
instead treat it as if the entire list or map was a secret.
We have logic in the language runtimes to do this as part of
serialization. There were a few issues this commit addresses:
- We were not promoting secretness across arrays in either Node or
Python
- For Python, our promotion logic was buggy and caused it to behave in
a manner where if any value was secret, the output values of the
object would be corrupted, because we'd incorrectly treat the
outputs as a secret who's value was a map, instead of a map of
values (some of which may be secret).
This caused very confusing behavior, because it would appear that a
resource creation call just did not set various output properties when
one or more of them ended up containing a secret.
Thse changes make a subtle but critical adjustment to the process the
Pulumi engine uses to determine whether or not a difference exists
between a resource's actual and desired states, and adjusts the way this
difference is calculated and displayed accordingly.
Today, the Pulumi engine get the first chance to decide whether or not
there is a difference between a resource's actual and desired states. It
does this by comparing the current set of inputs for a resource (i.e.
the inputs from the running Pulumi program) with the last set of inputs
used to update the resource. If there is no difference between the old
and new inputs, the engine decides that no change is necessary without
consulting the resource's provider. Only if there are changes does the
engine consult the resource's provider for more information about the
difference. This can be problematic for a number of reasons:
- Not all providers do input-input comparison; some do input-state
comparison
- Not all providers are able to update the last deployed set of inputs
when performing a refresh
- Some providers--either intentionally or due to bugs--may see changes
in resources whose inputs have not changed
All of these situations are confusing at the very least, and the first
is problematic with respect to correctness. Furthermore, the display
code only renders diffs it observes rather than rendering the diffs
observed by the provider, which can obscure the actual changes detected
at runtime.
These changes address both of these issues:
- Rather than comparing the current inputs against the last inputs
before calling a resource provider's Diff function, the engine calls
the Diff function in all cases.
- Providers may now return a list of properties that differ between the
requested and actual state and the way in which they differ. This
information will then be used by the CLI to render the diff
appropriately. A provider may also indicate that a particular diff is
between old and new inputs rather than old state and new inputs.
Fixes#2453.
Currently, if a secret was present, the value of variable "value" is used as the
key for the dictionary object containing the output. This leads to KeyError
exceptions in various places, as reported in #2782. This PR changes that to use
the literal string "value".
Fixes#2782.
