This commit adds CLI support for resource providers to provide partial
state upon failure. For resource providers that model resource
operations across multiple API calls, the Provider RPC interface can now
accomodate saving bags of state for resource operations that failed.
This is a common pattern for Terraform-backed providers that try to do
post-creation steps on resource as part of Create or Update resource
operations.
A critical part of the partial update protocol is to return a structured
error when a resource is successfully created, but fails to initialize.
This structured error contains the properties of the
partially-initialized resource, and instructs the engine to halt.
Most languages implement this by attaching "details" to the error, i.e.,
an arbitrary proto message attached to the error. The JavaScript
implementation is not mature enough to include all the facilities
required to use this, so here we must add a `Status` message, which
protobuf requires as part of its structure for returning details.
* Test the Python language host end-to-end
This commit introduces an end-to-end language host testing framework for
the Python SDK, similar to what already exists for the Node SDK. The
real language host is used to run Pulumi programs written in Python
while mocking out the resource monitor.
* Add new tests
* Print out better diagnostics when the langhost fails to launch
* Use the in-tree executor for testing
* CR: Place tests and code being tested in the same directory for ease of understanding, add a README
* Turns out I misunderstood the semantics of resource registration - fix two tests so that they pass now and fix a few bugs in the test harness
This change includes the Python and Golang language hosts in the Windows
SDK. As part of this change, I had to adjust how we launched the second
stage of the language host, since we can't depend on the shebang, so now
we invoke `python` passing the executor and then the arguments.
Fixes#1509
1) Use a state block for *Outputs, just to protect against dereferencing
and aliasing. These are mutable due to concurrency.
2) Dig into *Output type aliases, like *URNOutput, et. al, during
RPC marshaling.
This change adds a config package. This is syntactic sugar atop the
underlying config functionality in the pulumi.Context, but mirrors what
we do in our other Node.js and Python SDKs more closely.
This includes three families of functions:
- config.Get*: returns the value or its default if missing.
- config.Require*: returns the value or panics if missing.
- config.Try*: returns the value or an error if missing.
In all cases, there are simple Get/Require/Try functions, that just
deal in terms of strings, in addition to type specific functions,
GetT/RequireT/TryT, for the most common Ts that you might need.
This commit implements unknown outputs in the same style as our Node.js
language provider. That is to say, during previews, it's possible that
certain outputs will not have known values. In those cases, we want to
flow sufficient information through the resolution of values, so that we
may skip applies. We also return this fact from the direct accessors.
This change primarily does two things:
* Adds output marshaling.
* Adds tests for roundtripping inputs to outputs.
It also
* Fixes a bug in the verification of asset archives.
* Change input types to simply `interface{}` and `map[string]interface{}`.
There is no need for wrapper types. This is more idiomatic.
* Reject output properties upon marshaling failure.
* Don't support time.Time as a marshaling concept. This was getting too
cute. It's not clear what its marshaling format ought to be.
This improves the strong typing of output properties, by leveraging the
cast library to support numeric conversions to and from many types,
without hitting interface{}-cast panics. Also adds strongly typed
applies and adds a number of additional tests for these functions.
This change adds some convenience functions and types, to make strongly
typed outputs more pleasant to interact with. It also includes tests
for output generally, in addition to these new functions and types.
This adds rudimentary support for Pulumi programs written in Go. It
is not complete yet but the basic resource registration works.
Note that, stylistically speaking, Go is a bit different from our other
languages. This made it a bit easier to build this initial prototype,
since what we want is actually a rather thin veneer atop our existing
RPC interfaces. The lack of generics, however, adds some friction and
is something I'm continuing to hammer on; this will most likely lead to
little specialized types (e.g. StringOutput) once the dust settles.
There are two primary components:
1) A new language host, `pulumi-language-go`, which is responsible for
communicating with the engine through the usual gRPC interfaces.
Because Go programs are pre-compiled, it very simply loads a binary
with the same name as the project.
2) A client SDK library that Pulumi programs bind against. This exports
the core resource types -- including assets -- properties -- including
output properties -- and configuration.
Most remaining TODOs are marked as such in the code, and this will not
be merged until they have been addressed, and some better tests written.
* Fix a few issues with the Python language host
1. Fix an issue where the UNKNOWN sentinel was leaking into user
programs
2. Fix an issue where Protobuf types were leaking into user programs
In fixing this issues I also added a framework for writing tests against
the Python SDK.
* License headers, and adopt a more idiomatic testing pattern
* Additional idiomatic Python
* That's what I get for trying to be fancy (Travis CI python version is very old and does not respect this form)
* CR feedback: use more comprehensions, typo fix
* Break a circular dependency between resource, runtime.resource, and runtime.rpc
* Don't check in .vscode
* CR: sort inputs, rename global variable, add a test for CustomResource serialization
* Remove accidental code duplication
We retained these modules to support using v0.11.X and earlier
versions of @pulumi/pulumi, which required a native module to do
closure serialization. 0.12.X does not need this, so lets stop
including it.
Closure serialization now keeps track of the `require`d packages it sees in the function bodies that are serialized during a call to `serializeFunction`.
Also, replaces `serializeFunctionAsync` with `serializeFunction` which accepts richer parameters and return type, deprecating the former API (but leaving it available for now to avoid a breaking change).
Before the changes in #1414, all output properties were guaranteed to
have values after deserialization. After #1414, any properties with no
value were no longer resolved, which was treated as an error. These
changes resolve all missing proprties to `undefined`. If a property is
missing during an update, its `undefined` value is marked as known.
These changes add support for distinguishing an output property with
an unknown value from an output property with a known value that is
undefined.
In a broad sense, the Pulumi property type system is just JSON with the
addition of unknown values. Notably absent, however, are undefined
values. As it stands, our marshalers between JavaScript and Pulumi
property values treat all undefined JavaScript values as unknown Pulumi
values. Unfortunately, this conflates two very different concepts:
unknown Pulumi values are intended to represent values of output
properties that are unknown at time of preview, _not_ values that are
known but undefined. This results in difficulty reasoning about when
transforms are run on output properties as well as confusing output in
the `diff` view of Pulumi preview (user-specifed undefined values are
rendered as unknown values).
As it turns out, we already have a way to decide whether or not an
Output value is known or not: Output.performApply. These changes rename
this property to `isKnown`, clarify its meaning, and take advantage of
the result to decide whether or not an Output value should marshal as
an unknown Pulumi value.
This also allowed these changes to improve the serialization of
undefined object keys and array elements s.t. we better match JavaScript
to JSON serialization behavior (undefined object keys are omitted;
undefined array elements are marshaled as `null`).
Fixes https://github.com/pulumi/pulumi-cloud/issues/483.
* Delete Before Create
This commit implements the full semantics of delete before create. If a
resource is replaced and requires deletion before creation, the engine
will use the dependency graph saved in the snapshot to delete all
resources that depend on the resource being replaced prior to the
deletion of the resource to be replaced.
* Rebase against master
* CR: Simplify the control flow in makeRegisterResourceSteps
* Run Check on new inputs when re-creating a resource
* Fix an issue where the planner emitted benign but incorrect deletes of DBR-deleted resources
* CR: produce the list of dependent resources in dependency order and iterate over the list in reverse
* CR: deps->dependents, fix an issue with DependingOn where duplicate nodes could be added to the dependent set
* CR: Fix an issue where we were considering old defaults and new inputs
inappropriately when re-creating a deleted resource
* CR: save 'iter.deletes[urn]' as a local, iterate starting at cursorIndex + 1 for dependency graph
This changes two primary things about dynamic providers:
1) Always echo back the __provider upon read, even if there is a
missing read function on the dynamic provider. In fact, return
the full input state in that case.
2) Store the __provider in the output state of the dynamic resource,
in addition to the input state. My recollection of the "model"
discussion we had weeks ago was that the output properties are
mean to capture the state of a resource in its entirety; not having
this meant that refresh would marshal the outputs only, and find
on the other side of the RPC boundary that __provider was missing.
Note that an alternative to the latter fix would be to use some hybrid
of input and output state, as we used to do, by merging property maps.
* Fix a bug in promise leak detection that leaked promises when errors occur
* Add an opt-in to the super-verbose debug error message on promise leaks
* Fix a bad merge
* was/were grammar improvement in error message
* Fail the deployment if a debuggable promise leaks
* Graceful RPC shutdown: CLI side
* Handle unavailable resource monitor in language hosts
* Fix a comment
* Don't commit package-lock.json
* fix mangled pylint pragma
* Rebase against master and fix Gopkg.lock
* Code review feedback
* Fix a race between closing the callerEventsOpt channel and terminating a goroutine that writes to it
* glog -> logging
These changes enable tracing of Pulumi API calls.
The span with which to associate an API call is passed via a
`context.Context` parameter. This required plumbing a
`context.Context` parameter through a rather large number of APIs,
especially in the backend.
In general, all API calls are associated with a new root span that
exists for essentially the entire lifetime of an invocation of the
Pulumi CLI. There were a few places where the plumbing got a bit hairier
than I was willing to address with these changes; I've used
`context.Background()` in these instances. API calls that receive this
context will create new root spans, but will still be traced.
Instead of using a shell script to jump from the language host into
node, just invoke node directly. This makes our start-up path a little
simpler to understand and indirectly fixespulumi/home#156, where we
would fail on Windows if the `-exec` script was in a folder that had
spaces in it (due to a subtle interaction between how go launches cmd
files and how cmd.exe parses arguments).
Rather than filtering out the `id` and `urn` properties when serializing
the inputs to an invoke, pass these properties along. This enables the
use of invoke endpoints that accepts these as inputs (e.g. the endpoint
that backs `aws.ec2.getSubnet`).
* Improve the error message when npm/yarn install hasn't been run
* Same thing, but for Python
* Use PULUMI_RUN in batch script
* Use -e, -f doesn't work for symlinked paths (e.g. yarn link)
This change adopts `x is T` style of RTTI inquiry, which fits much
more nicely with TypeScript's typechecking flow.
Thanks to @lukehoban for teaching me a new trick today! :-)
This change moves us away from using JavaScript RTTI, by way of
`instanceof`, for built-in Pulumi types. If we use `instanceof`,
then the same logical type loaded from separate copies of the
SDK package -- as will happen in SxS scenarios -- are considered
different. This isn't actually what we want. The solution is
simple: implement our own quasi-RTTI solution, using __pulumi*
properties and manual as* and is* functions. Note that we could
have skipped the as* and is* functions, but I found that they led
to slightly easier to read code.
There is one strange thing in here, which I spoke to
@CyrusNajmabadi about: SerializedOutput<T>, because it implements
Output<T> as an _interface_, did not previously masquerade as an
actual Output<T>. In other words, `instanceof` would have returned
false, and indeed a few important properties (like promise) are
missing. This change preserves that behavior, although I'll admit
that this is slightly odd. I suspect we'll want to revisit this as
part of https://github.com/pulumi/pulumi/issues/1074.
Fixes https://github.com/pulumi/pulumi/issues/1203.
* Introduce a simple repetition operator to match expected error messages against actual ones
* Convert required and optional objects to use a Map (node v9 compat), improve the error formatting for failed tests
* Test node v6, v8, and v9 in CI
* Get rid of PULUMI_API env in .travis.yml, it's set from the Travis console now
While we no longer use the native runtime module, older versions of
@pulumi/pulumi still require it. Let's continue to have the launcher
put the native module location on the `$PATH`. And we'll include them
in the SDK for a while longer.
Fixes#1177
The RPC provider interface needs a way to convey back to the engine
that a resource being read no longer exists. To do this, we'll return
the ID property that was read back. If it is empty, it means the
resource is gone. If it is non-empty, we expect it to match the input.
* Implement closure scope chain analysis in pure TypeScript
This change makes use of four V8 intrinsics to avoid having to use a
native module to inspect the scope chains of live Function objects. This
unfortunately leads to the limitation of not allowing captures of 'this'
in arrow functions, but that is something we are willing to live with
for now.
* Remove native module build and restore from the Makefile
* CR feedback: Be a little more efficient when scanning the scope chain
* Nuke everything related to custom Node versions and the native Node module
* CR feedback: rename native.ts -> v8.ts, document some interfaces in v8.ts
As I began to write code using the ability to perform resource
lookups, especially in the code-generators, I realized the way it
was surfaced as an argument to the Resource base constructor would
lead to overload explosion. Instead of doing that, let's pass it
in the ResourceOptions bag.
Prior to this change, if you ended up with multiple Pulumi SDK
packages loaded side-by-side, we would fail in obscure ways. The
reason for this was that we initialize and store important state
in static variables. In the case that you load the same library
twice, however, you end up with separate copies of said statics,
which means we would be missing engine RPC addresses and so on.
This change adds the ability to recover from this situation by
mirroring the initialized state in process-wide environment
variables. By doing this, we can safely recover simply by reading
them back when we detect that they are missing. I think we can
eventually go even further here, and eliminate the entry point
launcher shim altogether by simply having the engine launch the
Node program with the right environment variables. This would
be a nice simplification to the system (fewer moving pieces).
There is still a risk that the separate copy is incompatible.
Presumably the reason for loading multiple copies is that the
NPM/Yarn version solver couldn't resolve to a shared version.
This may yield obscure failure modes should RPC interfaces change.
Figuring out what to do here is part of pulumi/pulumi#957.
This fixespulumi/pulumi#777 and pulumi/pulumi#1017.
This change skips unknown IDs during read operations. This can happen
when a read is performed using the output property of another resource
during planning. This is intentionally supported via ID being an
Input<ID> and all we need to do for this to work correctly is skip the
actual provider RPC and the runtime will propagate unknown outputs as
usual.
This change wires up the new Read RPC method in such a manner that
Pulumi programs can invoke it. This is technically not required for
refreshing state programmatically (as in pulumi/pulumi#1081), however
it's a feature we had eons ago and have wanted since (see
pulumi/pulumi#83), and will allow us to write code like
let vm = aws.ec2.Instance.get("my-vm", "i-07043cd97bd2c9cfc");
// use any property from here on out ...
The way this works is simply by bridging the Pulumi program via its
existing RPC connection to the engine, much like Invoke and
RegisterResource RPC requests already do, and then invoking the proper
resource provider in order to read the state. Note that some resources
cannot be uniquely identified by their ID alone, and so an extra
resource state bag may be provided with just those properties required.
This came almost for free (okay, not exactly) and will come in handy as
we start gaining experience with reading live state from resources.
This commit changes two things about our resource model:
* Stop performing Pulumi Engine-side diffing of resource state.
Instead, we defer to the resource plugins themselves to determine
whether a change was made and, if so, the extent of it. This
manifests as a simple change to the Diff function; it is done in
a backwards compatible way so that we continue with legacy diffing
for existing resource provider plugins.
* Add a Read RPC method for resource providers. It simply takes a
resource's ID and URN, plus an optional bag of further qualifying
state, and it returns the current property state as read back from
the actual live environment. Note that the optional bag of state
must at least include enough additional properties for resources
wherein the ID is insufficient for the provider to perform a lookup.
It may, however, include the full bag of prior state, for instance
in the case of a refresh operation.
This is part of pulumi/pulumi#1108.
* Improve the error message arising from missing required configs for
resource providers
If the resource provider that we are speaking to is new enough, it will send
across a list of keys and their descriptions alongside an error
indicating that the provider we are configuring is missing required
config. This commit packages up the list of missing keys into an error
that can be presented nicely to the user.
* Code review feedback: renaming simplification and correcting errors in comments
This change actually makes our Python version numbers conformant
to PEP440. Previously we were including the Git commit hash in the
alpha "version number" part, which is incorrect. This simply led to
warnings upon publication and installation, but that warning very
clearly states that support for invalid versions will stop at some
point. This change puts any "informative" parts, like the Git hash,
inside of a local version tag, where such things are permitted.
Also move away from the inline sed silliness so that we can more
easily share this logic across all of our repos.
It appears 15.0.2 generates a bad reference to the pip._install module.
I've tested that 15.2.0 does not so, although I don't really understand
why and when this changed (my current guess is the Travis base image
changed), this should fixpulumi/pulumi#1103.
* Send structured errors across RPC boundaries
This brings us closer to gRPC best practices where we send structured
errors with error codes across RPC endpoints. The new "rpcerrors"
package can wrap errors from RPC endpoints, so RPC servers can attach
some additional context as to why a request failed.
* Code review feedback:
1. Rename rpcerrors -> rpcerror, better package name
2. Rename RPCError -> Error, RPCErrorCause -> ErrorCause, names
suggested by gometalinter to improve their package-qualified names
3. Fix import organization in rpcerror.go
This change includes a bunch of refactorings I made in prep for
doing refresh (first, the command, see pulumi/pulumi#1081):
* The primary change is to change the way the engine's core update
functionality works with respect to deploy.Source. This is the
way we can plug in new sources of resource information during
planning (and, soon, diffing). The way I intend to model refresh
is by having a new kind of source, deploy.RefreshSource, which
will let us do virtually everything about an update/diff the same
way with refreshes, which avoid otherwise duplicative effort.
This includes changing the planOptions (nee deployOptions) to
take a new SourceFunc callback, which is responsible for creating
a source specific to the kind of plan being requested.
Preview, Update, and Destroy now are primarily differentiated by
the kind of deploy.Source that they return, rather than sprinkling
things like `if Destroying` throughout. This tidies up some logic
and, more importantly, gives us precisely the refresh hook we need.
* Originally, we used the deploy.NullSource for Destroy operations.
This simply returns nothing, which is how Destroy works. For some
reason, we were no longer doing this, and instead had some
`if Destroying` cases sprinkled throughout the deploy.EvalSource.
I think this is a vestige of some old way we did configuration, at
least judging by a comment, which is apparently no longer relevant.
* Move diff and diff-printing logic within the engine into its own
pkg/engine/diff.go file, to prepare for upcoming work.
* I keep noticing benign diffs anytime I regenerate protobufs. I
suspect this is because we're also on different versions. I changed
generate.sh to also dump the version into grpc_version.txt. At
least we can understand where the diffs are coming from, decide
whether to take them (i.e., a newer version), and ensure that as
a team we are monotonically increasing, and not going backwards.
* I also tidied up some tiny things I noticed while in there, like
comments, incorrect types, lint suppressions, and so on.
The semantic versions we were using for pre-release PyPI packages wasn't
quite right. We had been using local version identifiers, a la +, rather
than proper pre-release tags, which means that version specifications like
`pulumi==0.11.0` will match `0.11.0+dev.1521506136.g7f043fd.dirty`, in
addition to just `0.11.0`. This is clearly not what we want.
This change moves us over to proper alpha and release candidate versions,
as specified in https://www.python.org/dev/peps/pep-0440/#pre-releases.
Namely, any `x.y.z-dev-123456789-abcdef` version will get translated into
an alpha `x.y.za12345679-abcdef`, and any `x.y.z-rc1` will get translated
into a proper release candidate `x.y.zrc1` version number.