`Output.from_input` deeply unwraps nested output values in dicts and lists, but doesn't currently do that for the more recently added "input types" (i.e. args classes). This leads to errors when using args classes with output values with `Provider` resources, which uses `Output.from_input` on each input property and then serializes the value to JSON in an `apply`. This changes fixes `Output.from_input` to recurse into values within args classes to properly unwrap any nested outputs.
- Improve the existing coverage to use real resources and mocks
- Add tests for deserialization as well as serialization
- Add tests that serialize custom resources during preview
Contributes to #5943.
- Reset the runtime prior to each test
- Use the SDK's `test` decorator instead of `async_test`
- Rename a couple classes to avoid warnings from pytest
Resources are serialized as their URN, ID, and package version. Each
Pulumi package is expected to register itself with the SDK. The package
will be invoked to construct appropriate instances of rehydrated
resources. Packages are distinguished by their name and their version.
This is the foundation of cross-process resources.
Related to #2430.
Co-authored-by: Mikhail Shilkov <github@mikhail.io>
Co-authored-by: Luke Hoban <luke@pulumi.com>
Co-authored-by: Levi Blackstone <levi@pulumi.com>
A recent change to output deserialization resulted in secrets being returned unwrapped. This change addresses the regression, ensuring any unwrapped secret values are rewrapped before being returned.
We recently made a change to the Python codegen to emit `int` type annotations, instead of `float`, for properties that are typed as `schema.IntType`.
But the number values that come back from protobuf structs are always floats (like JSON), so we need to cast the values intended to be integers to `int`.
Non-string provider inputs must be projected as JSON formatted strings. The current codegen simply calls `json.dumps` for such properties, but this does not work for the new input types, which aren't JSON serializable.
This commit adds a new `pulumi.runtime.to_json` utility function to the core SDK, which is capable of serializing both raw dicts and input types as JSON. The codegen will be updated to make use of this new function rather than `json.dumps`.
We currently emit array types as `List[T]` for Python, but `List[T]` is invariant, which causes type checkers like mypy to produce errors when values like `["foo", "bar"]` are passed as args typed as `List[pulumi.Input[str]]` (since `Input[str]` is an alias for `Union[T, Awaitable[T], Output[T]]`. To address this, we should move to using [`Sequence[T]`](https://docs.python.org/3/library/typing.html#typing.Sequence) which is covariant, and does not have this problem.
We actually already do this for `Dict` vs. `Mapping`, emitting map types as `Mapping[str, T]` rather than `Dict[str, T]` because `Mapping[str, T]` is covariant for the value. This change makes us consistent for array types.
These are the SDK changes necessary to support `Sequence[T]`.
The previous attempt to allow this didn't actually allow it, so this is
take two. As part of the previous attempt, I thought after tweaking the
test I had observed the test failing, and then succeeding after making
the product changes, but I must have been mistaken.
It turns out that our existing mocks tests weren't running at all
because of a missing `__init__.py` file. Once the missing `__init__.py`
is added, the tests run, but other tests ("test mode" tests) fail
because the code that creates the mocks and resources will run during
test discovery, and setting the mocks modifies global state.
To address the test issue, I've moved the mocks tests into their own
`test_with_mocks` package that can be run separately from other tests.
And addressed the original issue, by creating a root Stack resource if
one isn't already present when the mocks are set.
This change allows importing modules with calls to `pulumi.export` in unit tests. Previously, you'd have to structure the Python program in a way that avoids the `pulumi.export` from being called from unit tests.
The original version of this code caused inconsistencies in the event
loop associated with a given thread. These changes elimintate the event
loop shenanigans the mocks were trying to play by updating _sync_await
to create an event loop if none exists in the current thread.
It's possible that this will cause problems if the tests run on a
different thread than the original program, as the tests are likely to
end up waiting on outputs created by the program, which is not supported
in Python.
Also adds test coverage of the mocking/testing support in Python.
These changes restore a more-correct version of the behavior that was
disabled with #3014. The original implementation of this behavior was
done in the SDKs, which do not have access to the complete inputs for a
resource (in particular, default values filled in by the provider during
`Check` are not exposed to the SDK). This lack of information meant that
the resolved output values could disagree with the typings present in
a provider SDK. Exacerbating this problem was the fact that unknown
values were dropped entirely, causing `undefined` values to appear in
unexpected places.
By doing this in the engine and allowing unknown values to be
represented in a first-class manner in the SDK, we can attack both of
these issues.
Although this behavior is not _strictly_ consistent with respect to the
resource model--in an update, a resource's output properties will come
from its provider and may differ from its input properties--this
behavior was present in the product for a fairly long time without
significant issues. In the future, we may be able to improve the
accuracy of resource outputs during a preview by allowing the provider
to dry-run CRUD operations and return partially-known values where
possible.
These changes also introduce new APIs in the Node and Python SDKs
that work with unknown values in a first-class fashion:
- A new parameter to the `apply` function that indicates that the
callback should be run even if the result of the apply contains
unknown values
- `containsUnknowns` and `isUnknown`, which return true if a value
either contains nested unknown values or is exactly an unknown value
- The `Unknown` type, which represents unknown values
The primary use case for these APIs is to allow nested, properties with
known values to be accessed via the lifted property accessor even when
the containing property is not fully know. A common example of this
pattern is the `metadata.name` property of a Kubernetes `Namespace`
object: while other properties of the `metadata` bag may be unknown,
`name` is often known. These APIs allow `ns.metadata.name` to return a
known value in this case.
In order to avoid exposing downlevel SDKs to unknown values--a change
which could break user code by exposing it to unexpected values--a
language SDK must indicate whether or not it supports first-class
unknown values as part of each `RegisterResourceRequest`.
These changes also allow us to avoid breaking user code with the new
behavior introduced by the prior commit.
Fixes#3190.
These changes restore a more-correct version of the behavior that was
disabled with #3014. The original implementation of this behavior was
done in the SDKs, which do not have access to the complete inputs for a
resource (in particular, default values filled in by the provider during
`Check` are not exposed to the SDK). This lack of information meant that
the resolved output values could disagree with the typings present in
a provider SDK. Exacerbating this problem was the fact that unknown
values were dropped entirely, causing `undefined` values to appear in
unexpected places.
By doing this in the engine and allowing unknown values to be
represented in a first-class manner in the SDK, we can attack both of
these issues.
Although this behavior is not _strictly_ consistent with respect to the
resource model--in an update, a resource's output properties will come
from its provider and may differ from its input properties--this
behavior was present in the product for a fairly long time without
significant issues. In the future, we may be able to improve the
accuracy of resource outputs during a preview by allowing the provider
to dry-run CRUD operations and return partially-known values where
possible.
These changes also introduce new APIs in the Node and Python SDKs
that work with unknown values in a first-class fashion:
- A new parameter to the `apply` function that indicates that the
callback should be run even if the result of the apply contains
unknown values
- `containsUnknowns` and `isUnknown`, which return true if a value
either contains nested unknown values or is exactly an unknown value
- The `Unknown` type, which represents unknown values
The primary use case for these APIs is to allow nested, properties with
known values to be accessed via the lifted property accessor even when
the containing property is not fully know. A common example of this
pattern is the `metadata.name` property of a Kubernetes `Namespace`
object: while other properties of the `metadata` bag may be unknown,
`name` is often known. These APIs allow `ns.metadata.name` to return a
known value in this case.
In order to avoid exposing downlevel SDKs to unknown values--a change
which could break user code by exposing it to unexpected values--a
language SDK must indicate whether or not it supports first-class
unknown values as part of each `RegisterResourceRequest`.
These changes also allow us to avoid breaking user code with the new
behavior introduced by the prior commit.
Fixes#3190.
- Do not use a non-zero-to-zero transition in the number of outstanding
RPCs to determine the completion of a Python program until after the
synchronous piece of the program has finished running is complete
- Instead of using a future to indicate that either a) a zero-to-one
transition in the number of outstanding RPCs has occurred, or b) an
unhandled exception has occurred, a) observe the transition itself,
and b) use an optional exception field to track the presence or
absence of an exception.
Fixes#3162.
_sync_await was not reentrant with respect to _run_once: the latter
captures the length of the ready list before it iterates it, and the
former drains the ready list by reentering _run_once. Fix this by
tracking the length of the list before pumping the event loop and then
pushing cancelled handles on to the list as necessary after pumping the
loop.
These changes also fix an issue with `export`ing awaitables.
Fixes#3038.
These changes fix a bug in the Python runtime that would cause any
awaitable input properties passed to a resource that are missing
from the resource's output properties to be awaited twice. The fix is
straightforward: rather than roundtripping an input property through
serialize/deserialize, just deserialized the already-serialized input
property.
Fixes#2940.
These changes make the `pulumi.runtime.invoke` function invokable in a
synchronous manner. Because this function still needs to perform
asynchronous work under the covers--namely awaiting a provider URN and
ID if a provider instance is present in the `InvokeOptions`--this
requires some creativity. This creativity comes in the form of a helper
function, `_sync_await`, that performs a logical yield from the
currently running event, manually runs the event loop until the given
future completes, performs a logical resume back to the
currently executing event, and returns the result of the future.
The code in `_sync_await` is a bit scary, as it relies upon knowledge of
(and functions in) the internals of the `asyncio` package. The necessary
work performed in this function was derived from the implementations of
`task_step` (which pointed out the need to call `_{enter,leave}_task`)
and `BaseEventLoop.run_forever` (which illustrated how the event loop is
pumped). In addition to potential breaking changes to these internals,
the code may not work if a user has provided an alternative implementation
for `EventLoop`. That said, the code is a close enough copy of
`BaseEventLoop.run_forever` that it should be a reasonable solution.
