We previously looked for `python3` and fallback to `python` on all systems. However, our Windows CI images include a `python3.exe` symlink to `python.exe` which does not work with `venv`. So on Windows, just look for `python` first, falling back to `python3`. (The default python.org Windows installation only includes `python.exe`).
Adds a `--limit` flag to `pulumi stack history. This allows limiting to the last few entries rather than fetching the entirety of a stack's update history (which can be quite slow for stacks with lots of updates). Example: `pulumi stack history --limit 1` fetches the last history entry only.
`stack.up` and related operations in the Automation API have been updated to consume this change, drastically reducing overhead.
`Output.from_input` deeply unwraps nested output values in dicts and lists, but doesn't currently do that for the more recently added "input types" (i.e. args classes). This leads to errors when using args classes with output values with `Provider` resources, which uses `Output.from_input` on each input property and then serializes the value to JSON in an `apply`. This changes fixes `Output.from_input` to recurse into values within args classes to properly unwrap any nested outputs.
- Improve the existing coverage to use real resources and mocks
- Add tests for deserialization as well as serialization
- Add tests that serialize custom resources during preview
Contributes to #5943.
- Reset the runtime prior to each test
- Use the SDK's `test` decorator instead of `async_test`
- Rename a couple classes to avoid warnings from pytest
* Enable resource reference feature by default
Unless the PULUMI_DISABLE_RESOURCE_REFERENCES flag
is explicitly set to a truthy value, the resource reference feature is now
enabled by default.
* Set AcceptResources in the language SDKs
This can be disabled by setting the `PULUMI_DISABLE_RESOURCE_REFERENCES` environment variable to a truthy value.
Co-authored-by: Justin Van Patten <jvp@justinvp.com>
Implement GetRequiredPlugins for Python, which determines the plugins
required by the program.
Also, if the `virtualenv` runtime option is set, and the specified
virtual directory is missing or empty, automatically create it and
install dependencies into it.
There are two significant changes in this commit: one to the way
resource packages/modules are stored and retrieved, and one to resource
ref deserialization in the face of missing resource packages/modules.
Resource packages and modules no longer require an exact version match
during deserialization. Instead, the newest compatible version of the
package or module is selected. If no version was specified, the newest
version of the package or module will be chosen. As a special case, a
package or module that has no version will always be treated as the best
version for that package or module.
If a resource package or module is not found when attempting to
deserialize a resource reference, the SDK no longer emits an error, and
instead deserializes the reference as its URN or ID (if present). This
accommodates providers that have not yet been updated to include the
appropriate factory registrations.
- Differentiate between resource references that have no ID (i.e. because
the referenced resource is not a CustomResource) and resource references
that have IDs that are not known. This is necessary for proper
backwards-compatible serialization of resource references.
- Fix the key that stores a resource reference's package version in the
.NET, NodeJS, and Python SDKs.
- Ensure that the resource monitor's marshalling/unmarshalling of inputs
and outputs to/from calls to `Construct` retain resource references as
appropriate.
- Fix serialization behavior for resources -> resource references in the
Go SDK: if a resource's ID is unknown, it should still be serialized
as a resource reference, albeit a reference with an unknown ID.
Just what it says on the tin.
The SDK code generator will be updated to use the new `urn`
resource option inside of each module's implementation of
`ResourceModule.construct`.
Part of #2430.
Co-authored-by: Justin Van Patten <jvp@justinvp.com>
Just what it says on the tin. This is implemented by changing the
`GetPackageConfig` method of `ConfigSource` to return a `PropertyMap`
and ensuring that any secret config is represented by a `Secret`.
This is necessary due to the way we've factored the libraries imported
by users into modules. The primary alternative is to ensure that each
child module imports the root module for a package and registers itself
with that package where necessary to prevent circular dependencies. This
simplifies the core SDKs slightly at the cost of greater complications
in the generated SDKs; the approach taken by these changes seems like a
more maintainable option.
Contributes to #2430.
Co-authored-by: Justin Van Patten <jvp@justinvp.com>
Resources are serialized as their URN, ID, and package version. Each
Pulumi package is expected to register itself with the SDK. The package
will be invoked to construct appropriate instances of rehydrated
resources. Packages are distinguished by their name and their version.
This is the foundation of cross-process resources.
Related to #2430.
Co-authored-by: Mikhail Shilkov <github@mikhail.io>
Co-authored-by: Luke Hoban <luke@pulumi.com>
Co-authored-by: Levi Blackstone <levi@pulumi.com>
These changes add support for provider-side previews of create and
update operations, which allows resource providers to supply output
property values for resources that are being created or updated during a
preview.
If a plugin supports provider-side preview, its create/update methods
will be invoked during previews with the `preview` property set to true.
It is the responsibility of the provider to fill in any output
properties that are known before returning. It is a best practice for
providers to only fill in property values that are guaranteed to be
identical if the preview were instead an update (i.e. only those output
properties whose values can be conclusively determined without
actually performing the create/update operation should be populated).
Providers that support previews must accept unknown values in their
create and update methods.
If a plugin does not support provider-side preview, the inputs to a
create or update operation will be propagated to the outputs as they are
today.
Fixes#4992.
A recent change to output deserialization resulted in secrets being returned unwrapped. This change addresses the regression, ensuring any unwrapped secret values are rewrapped before being returned.
We recently made a change to the Python codegen to emit `int` type annotations, instead of `float`, for properties that are typed as `schema.IntType`.
But the number values that come back from protobuf structs are always floats (like JSON), so we need to cast the values intended to be integers to `int`.
When installing a plugin, if it contains a `PulumiPlugin.yaml` file with a `runtime` value of `nodejs` or `python`, install dependencies for the plugin.
For Node.js, `npm install` is run (or `yarn install` if `PULUMI_PREFER_YARN` is set).
For Python, a virtual environment is created and deps installed into it.
Non-string provider inputs must be projected as JSON formatted strings. The current codegen simply calls `json.dumps` for such properties, but this does not work for the new input types, which aren't JSON serializable.
This commit adds a new `pulumi.runtime.to_json` utility function to the core SDK, which is capable of serializing both raw dicts and input types as JSON. The codegen will be updated to make use of this new function rather than `json.dumps`.
We currently emit array types as `List[T]` for Python, but `List[T]` is invariant, which causes type checkers like mypy to produce errors when values like `["foo", "bar"]` are passed as args typed as `List[pulumi.Input[str]]` (since `Input[str]` is an alias for `Union[T, Awaitable[T], Output[T]]`. To address this, we should move to using [`Sequence[T]`](https://docs.python.org/3/library/typing.html#typing.Sequence) which is covariant, and does not have this problem.
We actually already do this for `Dict` vs. `Mapping`, emitting map types as `Mapping[str, T]` rather than `Dict[str, T]` because `Mapping[str, T]` is covariant for the value. This change makes us consistent for array types.
These are the SDK changes necessary to support `Sequence[T]`.
These changes add initial support for the construction of remote
components. For now, this support is limited to the NodeJS SDK;
follow-up changes will implement support for the other SDKs.
Remote components are component resources that are constructed and
managed by plugins rather than by Pulumi programs. In this sense, they
are a bit like cloud resources, and are supported by the same
distribution and plugin loading mechanisms and described by the same
schema system.
The construction of a remote component is initiated by a
`RegisterResourceRequest` with the new `remote` field set to `true`.
When the resource monitor receives such a request, it loads the plugin
that implements the component resource and calls the `Construct`
method added to the resource provider interface as part of these
changes. This method accepts the information necessary to construct the
component and its children: the component's name, type, resource
options, inputs, and input dependencies. It is responsible for
dispatching to the appropriate component factory to create the
component, then returning its URN, resolved output properties, and
output property dependencies. The dependency information is necessary to
support features such as delete-before-replace, which rely on precise
dependency information for custom resources.
These changes also add initial support for more conveniently
implementing resource providers in NodeJS. The interface used to
implement such a provider is similar to the dynamic provider interface
(and may be unified with that interface in the future).
An example of a NodeJS program constructing a remote component resource
also implemented in NodeJS can be found in
`tests/construct_component/nodejs`.
This is the core of #2430.
Update pip, setuptools, and wheel in the virtual environment before installing dependencies as recommended by the Python documentation. This should help avoid failures when only source distributions are available for a package and pip attempts to build a wheel locally.
Pylint currently reports `E1101: Instance of 'Bucket' has no 'id' member (no-member)` on lines in Pulumi Python programs like:
```python
pulumi.export('bucket_name', bucket.id)
```
Here's a description of this message from http://pylint-messages.wikidot.com/messages:e1101:
> Used when an object (variable, function, …) is accessed for a non-existent member.
>
> False positives: This message may report object members that are created dynamically, but exist at the time they are accessed.
This appears to be a false positive case: `id` isn't set in the constructor (it's set later in `register_resource`) and Pylint isn't able to figure this out statically. `urn` has the same problem. (Oddly, Pylint doesn't complain when accessing other resource output properties).
This change refactors `register_resource` so that `id` and `urn` can be assigned in the resource's constructor, so that Pylint can see it being assigned. The change also does the same with `read_resource`.
Automatically create a virtual environment and install dependencies in it with `pulumi new` and `pulumi policy new` for Python templates.
This will save a new `virtualenv` runtime option in `Pulumi.yaml` (`PulumiPolicy.yaml` for policy packs):
```yaml
runtime:
name: python
options:
virtualenv: venv
```
`virtualenv` is the path to a virtual environment that Pulumi will use when running `python` commands.
Existing projects are unaffected and can opt-in to using this by setting `virtualenv`, otherwise, they'll continue to work as-is.
This class was available in the pulumi.resource module, but was not exported from the core `pulumi` module as intended for all public APIs at this level.
The previous attempt to allow this didn't actually allow it, so this is
take two. As part of the previous attempt, I thought after tweaking the
test I had observed the test failing, and then succeeding after making
the product changes, but I must have been mistaken.
It turns out that our existing mocks tests weren't running at all
because of a missing `__init__.py` file. Once the missing `__init__.py`
is added, the tests run, but other tests ("test mode" tests) fail
because the code that creates the mocks and resources will run during
test discovery, and setting the mocks modifies global state.
To address the test issue, I've moved the mocks tests into their own
`test_with_mocks` package that can be run separately from other tests.
And addressed the original issue, by creating a root Stack resource if
one isn't already present when the mocks are set.
This change allows importing modules with calls to `pulumi.export` in unit tests. Previously, you'd have to structure the Python program in a way that avoids the `pulumi.export` from being called from unit tests.
* [sdk/python] Improve `ResoruceOptions.merge` type
The implementation correctly handles `None` inputs, so the type should allow these as well.
* Add CHANGELOG
* Make `async:true` the default for `invoke` calls (#3750)
* Switch away from native grpc impl. (#3728)
* Remove usage of the 'deasync' library from @pulumi/pulumi. (#3752)
* Only retry as long as we get unavailable back. Anything else continues. (#3769)
* Handle all errors for now. (#3781)
* Do not assume --yes was present when using pulumi in non-interactive mode (#3793)
* Upgrade all paths for sdk and pkg to v2
* Backport C# invoke classes and other recent gen changes (#4288)
Adjust C# generation
* Replace IDeployment with a sealed class (#4318)
Replace IDeployment with a sealed class
* .NET: default to args subtype rather than Args.Empty (#4320)
* Adding system namespace for Dotnet code gen
This is required for using Obsolute attributes for deprecations
```
Iam/InstanceProfile.cs(142,10): error CS0246: The type or namespace name 'ObsoleteAttribute' could not be found (are you missing a using directive or an assembly reference?) [/Users/stack72/code/go/src/github.com/pulumi/pulumi-aws/sdk/dotnet/Pulumi.Aws.csproj]
Iam/InstanceProfile.cs(142,10): error CS0246: The type or namespace name 'Obsolete' could not be found (are you missing a using directive or an assembly reference?) [/Users/stack72/code/go/src/github.com/pulumi/pulumi-aws/sdk/dotnet/Pulumi.Aws.csproj]
```
* Fix the nullability of config type properties in C# codegen (#4379)
The initial config represents any config that was specified programmatically to the Policy Pack, for Policy Packs that support programmatic configuration like AWSGuard.
It's not entirely clear why gRPC doesn't already report these cleanly as unimplemented, but for now we'll explicitly implement them to avoid any spurious warnings.
Fixes#4028.
The original version of this code caused inconsistencies in the event
loop associated with a given thread. These changes elimintate the event
loop shenanigans the mocks were trying to play by updating _sync_await
to create an event loop if none exists in the current thread.
It's possible that this will cause problems if the tests run on a
different thread than the original program, as the tests are likely to
end up waiting on outputs created by the program, which is not supported
in Python.
Also adds test coverage of the mocking/testing support in Python.
These changes add support for mocking the resource monitor to the NodeJS
and Python SDKs. The proposed mock interface is a simplified version of
the standard resource monitor that allows an end-user to replace the
usual implementations of ReadResource/RegisterResource and Invoke with
their own. This can be used in unit tests to allow for precise control
of resource outputs and invoke results.
These changes add a new method to the resource provider gRPC interface,
`GetSchema`, that allows consumers of these providers to extract
JSON-serialized schema information for the provider's types, resources,
and functions.
Set an option to increase the memory limit on protobuf
parsing so that we can handle larger gRPC payloads.
Co-authored-by: Evan Boyle <EvanBoyle@users.noreply.github.com>
The provider plugin protocol is to write a port number followed by `\n`. We must guarantee we do that even on Windows, so must avoid Python `print` statements which implicitly rewrite newlines to platform specific character sequences.
Fixes#3807.
PEP 561 specifies that packages which contain either inline type hints
or type stubs should indicate their support for type hints via
including a file named `py.typed` in the root of the package. Since
Pulumi already includes inline type hints, adding `py.typed` to the
Python SDK simply allows these hints to be used by mypy.
Earlier changes to the get-version script were not adopted by the Python
SDK Makefile. This caused package publishing to skip uploads due to
missing versions.
These changes restore a more-correct version of the behavior that was
disabled with #3014. The original implementation of this behavior was
done in the SDKs, which do not have access to the complete inputs for a
resource (in particular, default values filled in by the provider during
`Check` are not exposed to the SDK). This lack of information meant that
the resolved output values could disagree with the typings present in
a provider SDK. Exacerbating this problem was the fact that unknown
values were dropped entirely, causing `undefined` values to appear in
unexpected places.
By doing this in the engine and allowing unknown values to be
represented in a first-class manner in the SDK, we can attack both of
these issues.
Although this behavior is not _strictly_ consistent with respect to the
resource model--in an update, a resource's output properties will come
from its provider and may differ from its input properties--this
behavior was present in the product for a fairly long time without
significant issues. In the future, we may be able to improve the
accuracy of resource outputs during a preview by allowing the provider
to dry-run CRUD operations and return partially-known values where
possible.
These changes also introduce new APIs in the Node and Python SDKs
that work with unknown values in a first-class fashion:
- A new parameter to the `apply` function that indicates that the
callback should be run even if the result of the apply contains
unknown values
- `containsUnknowns` and `isUnknown`, which return true if a value
either contains nested unknown values or is exactly an unknown value
- The `Unknown` type, which represents unknown values
The primary use case for these APIs is to allow nested, properties with
known values to be accessed via the lifted property accessor even when
the containing property is not fully know. A common example of this
pattern is the `metadata.name` property of a Kubernetes `Namespace`
object: while other properties of the `metadata` bag may be unknown,
`name` is often known. These APIs allow `ns.metadata.name` to return a
known value in this case.
In order to avoid exposing downlevel SDKs to unknown values--a change
which could break user code by exposing it to unexpected values--a
language SDK must indicate whether or not it supports first-class
unknown values as part of each `RegisterResourceRequest`.
These changes also allow us to avoid breaking user code with the new
behavior introduced by the prior commit.
Fixes#3190.
If a stack output includes a `Resource`, we will as of a recent change
always show the output diff, but this diff will potentially include
unknowns, leading to spurious output like:
```
+ namePrefix : output<string>
```
These changes supress these diffs by adding a special key to the POJO
we generate for resources *during preview only* that indicates that the
POJO represents a Pulumi resource, then stripping all adds of unknown
values from diffs for objects marked with that key.
Fixes#3314.
Adds Python support for resource transformations aligned with the existing NodeJS support in #3174.
This PR also moves processing of transformations to earlier in the resource construction process (for both NodeJS and Python) to ensure that invariants established in the constructor cannot be violated by transformations. This change can technically be a breaking change, but given that (a) the transformations features was just released in 1.3.0 and (b) the cases where this is a breaking change are uncommon and unlikely to have been reliable anyway - it feels like a change we should make now.
Fixes#3283.
These changes restore a more-correct version of the behavior that was
disabled with #3014. The original implementation of this behavior was
done in the SDKs, which do not have access to the complete inputs for a
resource (in particular, default values filled in by the provider during
`Check` are not exposed to the SDK). This lack of information meant that
the resolved output values could disagree with the typings present in
a provider SDK. Exacerbating this problem was the fact that unknown
values were dropped entirely, causing `undefined` values to appear in
unexpected places.
By doing this in the engine and allowing unknown values to be
represented in a first-class manner in the SDK, we can attack both of
these issues.
Although this behavior is not _strictly_ consistent with respect to the
resource model--in an update, a resource's output properties will come
from its provider and may differ from its input properties--this
behavior was present in the product for a fairly long time without
significant issues. In the future, we may be able to improve the
accuracy of resource outputs during a preview by allowing the provider
to dry-run CRUD operations and return partially-known values where
possible.
These changes also introduce new APIs in the Node and Python SDKs
that work with unknown values in a first-class fashion:
- A new parameter to the `apply` function that indicates that the
callback should be run even if the result of the apply contains
unknown values
- `containsUnknowns` and `isUnknown`, which return true if a value
either contains nested unknown values or is exactly an unknown value
- The `Unknown` type, which represents unknown values
The primary use case for these APIs is to allow nested, properties with
known values to be accessed via the lifted property accessor even when
the containing property is not fully know. A common example of this
pattern is the `metadata.name` property of a Kubernetes `Namespace`
object: while other properties of the `metadata` bag may be unknown,
`name` is often known. These APIs allow `ns.metadata.name` to return a
known value in this case.
In order to avoid exposing downlevel SDKs to unknown values--a change
which could break user code by exposing it to unexpected values--a
language SDK must indicate whether or not it supports first-class
unknown values as part of each `RegisterResourceRequest`.
These changes also allow us to avoid breaking user code with the new
behavior introduced by the prior commit.
Fixes#3190.
* Fix some tracing issues.
- Add endpoints for `startUpdate` and `postEngineEventsBatch` so that
spans for these invocations have proper names
- Inject a tracing span when walking a plan so that resource operations
are properly parented
- When handling gRPC calls, inject a tracing span into the call's
metadata if no span is already present so that resource monitor and
engine spans are properly parented
- Do not trace client gRPC invocations of the empty method so that these
calls (which are used to determine server availability) do not muddy
the trace. Note that I tried parenting these spans appropriately, but
doing so broke the trace entirely.
With these changes, the only unparented span in a typical Pulumi
invocation is a single call to `getUser`. This span is unparented
because that call does not have a context available. Plumbing a context
into that particular call is surprisingly tricky, as it is often called
by other context-less functions.
* Make tracing support more flexible.
- Add support for writing trace data to a local file using Appdash
- Add support for viewing Appdash traces via the CLI
We had been using `pandoc` to convert our README.md into a README.rst
for use with `setup.py` and the python package ecosystem. It turns out
that we can use markdown if we set a content type. So let's do that
and make things a little simpler.
While I was in the area, I made the encoding of UTF-8 explicit when
opening README.md.
- Do not use a non-zero-to-zero transition in the number of outstanding
RPCs to determine the completion of a Python program until after the
synchronous piece of the program has finished running is complete
- Instead of using a future to indicate that either a) a zero-to-one
transition in the number of outstanding RPCs has occurred, or b) an
unhandled exception has occurred, a) observe the transition itself,
and b) use an optional exception field to track the presence or
absence of an exception.
Fixes#3162.
We intend to replace PULUMI_TEST_MODE with better testing support
that doesn't suffer from all the pitfalls of our current approach.
Unfortunately, we don't yet have complete guidance or validation
that the new approaches will work for all existing end users. So,
until we do, we'll take a lighter touch approach here, and simply
not encourage new usage of PULUMI_TEST_MODE.
Issue #3045 will remain open to track a mroe permanent fix.
With these changes, a user may explicitly set `deleteBeforeReplace` to
`false` in order to disable DBR behavior for a particular resource. This
is the SDK + CLI escape hatch for cases where the changes in
https://github.com/pulumi/pulumi-terraform/pull/465 cause undesirable
behavior.
When using StackReference, if the stack you reference contains any
secret outputs, we have to mark the entire `outputs` member as a
secret output. This is because we only track secretness on a per
`Output<T>` basis.
For `getSecret` and friends, however, we know the name of the output
you are looking up and we can be smarter about if the returned
`Output<T>` should be treated as a secret or not.
This change augments the provider for StackReference such that it also
returns a list of top level stack output names who's values contain
secrets. In the language SDKs, we use this information, when present,
to decide if we should return an `Output<T>` that is marked as a
secret or not. Since the SDK and CLI are independent components, care
is taken to ensure that when the CLI does not return this information,
we behave as we did before (i.e. if any output is a secret, we treat
every output as a secret).
Fixes#2744
_sync_await was not reentrant with respect to _run_once: the latter
captures the length of the ready list before it iterates it, and the
former drains the ready list by reentering _run_once. Fix this by
tracking the length of the list before pumping the event loop and then
pushing cancelled handles on to the list as necessary after pumping the
loop.
These changes also fix an issue with `export`ing awaitables.
Fixes#3038.
