A user who calls `StreamInvoke` probably expects the `AsyncIterable`
that is returned to gracefully terminate. This is currently not the
case.
Where does something like this go wrong? A better question might be
where any of this went right, because several days later, after
wandering into civilization from the great Wilderness of Bugs, I must
confess that I've forgotten if any of it had.
`AsyncIterable` is a pull-based API. `for await (...)` will continuously
call `next` ("pull") on the underlying `AsyncIterator` until the
iterable is exhausted. But, gRPC's streaming-return API is _push_ based.
That is to say, when a streaming RPC is called, data is provided by
callback on the stream object, like:
call.on("data", (thing: any) => {... do thing ...});
Our goal in `StreamInvoke` is to convert the push-based gRPC routines
into the pull-based `AsyncIterable` retrun type. You may remember your
CS theory this is one of those annoying "fundamental mismatches" in
abstraction. So we're off to a good start.
Until this point, we've depended on a library,
`callback-to-async-iterator` to handle the details of being this bridge.
Our trusting nature and innocent charm has mislead us. This library is
not worthy of our trust. Instead of doing what we'd like it to do, it
returns (in our case) an `AsyncIterable` that will never complete.
Yes,, this `AsyncIterable` will patiently wait for eternity, which
honestly is kind of poetic when you sit down in a nice bath and think
about that fun time you considered eating your computer instead of
finishing this idiotic bug.
Indeed, this is the sort of bug that you wonder where it even comes
from. Our query libraries? Why aren't these `finally` blocks executing?
Is our language host terminating early? Is gRPC angry at me, and just
passive-aggrssively not servicing some of my requests? Oh god I've been
up for 48 hours, why is that wallpaper starting to move? And by the way,
a fun interlude to take in an otherwise very productive week is to try
to understand the gRPC streaming node client, which is code-gen'd, but
which also takes the liberty of generating itself at runtime, so that
gRPC is code-gen'ing a code-gen routine, which makes the whole thing
un-introspectable, un-debuggable, and un-knowable. That's fine, I didn't
need to understand any of this anyway, thanks friends.
But we've come out the other side knowing that the weak link in this
very sorry chain of incredibly weak links, is this dependency.
This commit removes this dependency for a better monster: the one we
know.
It is at this time that I'd like to announce that I am quitting my job
at Pulumi. I thank you all for the good times, but mostly, for taking
this code over for me.
These changes restore a more-correct version of the behavior that was
disabled with #3014. The original implementation of this behavior was
done in the SDKs, which do not have access to the complete inputs for a
resource (in particular, default values filled in by the provider during
`Check` are not exposed to the SDK). This lack of information meant that
the resolved output values could disagree with the typings present in
a provider SDK. Exacerbating this problem was the fact that unknown
values were dropped entirely, causing `undefined` values to appear in
unexpected places.
By doing this in the engine and allowing unknown values to be
represented in a first-class manner in the SDK, we can attack both of
these issues.
Although this behavior is not _strictly_ consistent with respect to the
resource model--in an update, a resource's output properties will come
from its provider and may differ from its input properties--this
behavior was present in the product for a fairly long time without
significant issues. In the future, we may be able to improve the
accuracy of resource outputs during a preview by allowing the provider
to dry-run CRUD operations and return partially-known values where
possible.
These changes also introduce new APIs in the Node and Python SDKs
that work with unknown values in a first-class fashion:
- A new parameter to the `apply` function that indicates that the
callback should be run even if the result of the apply contains
unknown values
- `containsUnknowns` and `isUnknown`, which return true if a value
either contains nested unknown values or is exactly an unknown value
- The `Unknown` type, which represents unknown values
The primary use case for these APIs is to allow nested, properties with
known values to be accessed via the lifted property accessor even when
the containing property is not fully know. A common example of this
pattern is the `metadata.name` property of a Kubernetes `Namespace`
object: while other properties of the `metadata` bag may be unknown,
`name` is often known. These APIs allow `ns.metadata.name` to return a
known value in this case.
In order to avoid exposing downlevel SDKs to unknown values--a change
which could break user code by exposing it to unexpected values--a
language SDK must indicate whether or not it supports first-class
unknown values as part of each `RegisterResourceRequest`.
These changes also allow us to avoid breaking user code with the new
behavior introduced by the prior commit.
Fixes#3190.
These changes restore a more-correct version of the behavior that was
disabled with #3014. The original implementation of this behavior was
done in the SDKs, which do not have access to the complete inputs for a
resource (in particular, default values filled in by the provider during
`Check` are not exposed to the SDK). This lack of information meant that
the resolved output values could disagree with the typings present in
a provider SDK. Exacerbating this problem was the fact that unknown
values were dropped entirely, causing `undefined` values to appear in
unexpected places.
By doing this in the engine and allowing unknown values to be
represented in a first-class manner in the SDK, we can attack both of
these issues.
Although this behavior is not _strictly_ consistent with respect to the
resource model--in an update, a resource's output properties will come
from its provider and may differ from its input properties--this
behavior was present in the product for a fairly long time without
significant issues. In the future, we may be able to improve the
accuracy of resource outputs during a preview by allowing the provider
to dry-run CRUD operations and return partially-known values where
possible.
These changes also introduce new APIs in the Node and Python SDKs
that work with unknown values in a first-class fashion:
- A new parameter to the `apply` function that indicates that the
callback should be run even if the result of the apply contains
unknown values
- `containsUnknowns` and `isUnknown`, which return true if a value
either contains nested unknown values or is exactly an unknown value
- The `Unknown` type, which represents unknown values
The primary use case for these APIs is to allow nested, properties with
known values to be accessed via the lifted property accessor even when
the containing property is not fully know. A common example of this
pattern is the `metadata.name` property of a Kubernetes `Namespace`
object: while other properties of the `metadata` bag may be unknown,
`name` is often known. These APIs allow `ns.metadata.name` to return a
known value in this case.
In order to avoid exposing downlevel SDKs to unknown values--a change
which could break user code by exposing it to unexpected values--a
language SDK must indicate whether or not it supports first-class
unknown values as part of each `RegisterResourceRequest`.
These changes also allow us to avoid breaking user code with the new
behavior introduced by the prior commit.
Fixes#3190.
Adds the ability to provide `transformations` to modify the properties and resource options that will be used for any child resource of a component or stack.
This offers an "escape hatch" to modify the behaviour of a component by peeking behind it's abstraction. For example, it can be used to add a resource option (`additionalSecretOutputs`, `aliases`, `protect`, etc.) to a specific known child of a component, or to modify some input property to a child resource if the component does not (yet) expose the ability to control that input directly. It could also be used for more interesting scenarios - such as:
1. Automatically applying tags to all resources that support them in a stack (or component)
2. Injecting real dependencies between stringly-referenced resources in a Helm Chart
3. Injecting explicit names using a preferred naming convention across all resources in a stack
4. Injecting `import` onto all resources by doing a lookup into a name=>id mapping
Because this feature makes it possible to peek behind a component abstraction, it must be used with care in cases where the component is versioned independently of the use of transformations. Also, this can result in "spooky action at a distance", so should be used judiciously. That said - this can be used as an escape hatch to unblock a wide variety of common use cases without waiting on changes to be made in a component implementation.
Each transformation is passed the `resource`, `name`, `type`, `props` and `opts` that are passed into the `Resource` constructor for any resource descended from the resource that has the transformation applied. The transformation callback can optionally return alternate versions of the `props` and `opts` to be used in place of the original values provided to the resource constructor.
Fixes#2068.
Change is 3.6.2 of typescript have caused their code generation to no
longer emit a call to `this` inside an arrow function, so this test is
no longer causing an error to be thrown.
For now, just accept the baseline, but I'll file an issue so we can
actually get a real failing test here.
We intend to replace PULUMI_TEST_MODE with better testing support
that doesn't suffer from all the pitfalls of our current approach.
Unfortunately, we don't yet have complete guidance or validation
that the new approaches will work for all existing end users. So,
until we do, we'll take a lighter touch approach here, and simply
not encourage new usage of PULUMI_TEST_MODE.
Issue #3045 will remain open to track a mroe permanent fix.
- Ensure that type assertions are guarded, and that incorrectly-typed
properties return errors rather than panicking
- Expand the asset/archive tests in the Node SDK to ensure that eventual
archives and assets serialize and deserialize correctly
Fixes#2836.
Fixes#3016.
A resource can be imported by setting the `import` property in the
resource options bag when instantiating a resource. In order to
successfully import a resource, its desired configuration (i.e. its
inputs) must not differ from its actual configuration (i.e. its state)
as calculated by the resource's provider.
There are a few interesting state transitions hiding here when importing
a resource:
1. No prior resource exists in the checkpoint file. In this case, the
resource is simply imported.
2. An external resource exists in the checkpoint file. In this case, the
resource is imported and the old external state is discarded.
3. A non-external resource exists in the checkpoint file and its ID is
different from the ID to import. In this case, the new resource is
imported and the old resource is deleted.
4. A non-external resource exists in the checkpoint file, but the ID is
the same as the ID to import. In this case, the import ID is ignored
and the resource is treated as it would be in all cases except for
changes that would replace the resource. In that case, the step
generator issues an error that indicates that the import ID should be
removed: were we to move forward with the replace, the new state of
the stack would fall under case (3), which is almost certainly not
what the user intends.
Fixes#1662.
There current RPC model for Pulumi allows secret values to be deeply
embedded in lists or maps, however at the language level, since we
track secrets via `Output<T>` we need to ensure that during
deserialization, if a list or a map contains a secret, we need to
instead treat it as if the entire list or map was a secret.
We have logic in the language runtimes to do this as part of
serialization. There were a few issues this commit addresses:
- We were not promoting secretness across arrays in either Node or
Python
- For Python, our promotion logic was buggy and caused it to behave in
a manner where if any value was secret, the output values of the
object would be corrupted, because we'd incorrectly treat the
outputs as a secret who's value was a map, instead of a map of
values (some of which may be secret).
This caused very confusing behavior, because it would appear that a
resource creation call just did not set various output properties when
one or more of them ended up containing a secret.
Adds a new resource option `aliases` which can be used to rename a resource. When making a breaking change to the name or type of a resource or component, the old name can be added to the list of `aliases` for a resource to ensure that existing resources will be migrated to the new name instead of being deleted and replaced with the new named resource.
There are two key places this change is implemented.
The first is the step generator in the engine. When computing whether there is an old version of a registered resource, we now take into account the aliases specified on the registered resource. That is, we first look up the resource by its new URN in the old state, and then by any aliases provided (in order). This can allow the resource to be matched as a (potential) update to an existing resource with a different URN.
The second is the core `Resource` constructor in the JavaScript (and soon Python) SDKs. This change ensures that when a parent resource is aliased, that all children implicitly inherit corresponding aliases. It is similar to how many other resource options are "inherited" implicitly from the parent.
Four specific scenarios are explicitly tested as part of this PR:
1. Renaming a resource
2. Adopting a resource into a component (as the owner of both component and consumption codebases)
3. Renaming a component instance (as the owner of the consumption codebase without changes to the component)
4. Changing the type of a component (as the owner of the component codebase without changes to the consumption codebase)
4. Combining (1) and (3) to make both changes to a resource at the same time
