These changes enable tracing of Pulumi API calls.
The span with which to associate an API call is passed via a
`context.Context` parameter. This required plumbing a
`context.Context` parameter through a rather large number of APIs,
especially in the backend.
In general, all API calls are associated with a new root span that
exists for essentially the entire lifetime of an invocation of the
Pulumi CLI. There were a few places where the plumbing got a bit hairier
than I was willing to address with these changes; I've used
`context.Background()` in these instances. API calls that receive this
context will create new root spans, but will still be traced.
Instead of using a shell script to jump from the language host into
node, just invoke node directly. This makes our start-up path a little
simpler to understand and indirectly fixespulumi/home#156, where we
would fail on Windows if the `-exec` script was in a folder that had
spaces in it (due to a subtle interaction between how go launches cmd
files and how cmd.exe parses arguments).
Rather than filtering out the `id` and `urn` properties when serializing
the inputs to an invoke, pass these properties along. This enables the
use of invoke endpoints that accepts these as inputs (e.g. the endpoint
that backs `aws.ec2.getSubnet`).
* Improve the error message when npm/yarn install hasn't been run
* Same thing, but for Python
* Use PULUMI_RUN in batch script
* Use -e, -f doesn't work for symlinked paths (e.g. yarn link)
This change adopts `x is T` style of RTTI inquiry, which fits much
more nicely with TypeScript's typechecking flow.
Thanks to @lukehoban for teaching me a new trick today! :-)
This change moves us away from using JavaScript RTTI, by way of
`instanceof`, for built-in Pulumi types. If we use `instanceof`,
then the same logical type loaded from separate copies of the
SDK package -- as will happen in SxS scenarios -- are considered
different. This isn't actually what we want. The solution is
simple: implement our own quasi-RTTI solution, using __pulumi*
properties and manual as* and is* functions. Note that we could
have skipped the as* and is* functions, but I found that they led
to slightly easier to read code.
There is one strange thing in here, which I spoke to
@CyrusNajmabadi about: SerializedOutput<T>, because it implements
Output<T> as an _interface_, did not previously masquerade as an
actual Output<T>. In other words, `instanceof` would have returned
false, and indeed a few important properties (like promise) are
missing. This change preserves that behavior, although I'll admit
that this is slightly odd. I suspect we'll want to revisit this as
part of https://github.com/pulumi/pulumi/issues/1074.
Fixes https://github.com/pulumi/pulumi/issues/1203.
* Introduce a simple repetition operator to match expected error messages against actual ones
* Convert required and optional objects to use a Map (node v9 compat), improve the error formatting for failed tests
* Test node v6, v8, and v9 in CI
* Get rid of PULUMI_API env in .travis.yml, it's set from the Travis console now
While we no longer use the native runtime module, older versions of
@pulumi/pulumi still require it. Let's continue to have the launcher
put the native module location on the `$PATH`. And we'll include them
in the SDK for a while longer.
Fixes#1177
The RPC provider interface needs a way to convey back to the engine
that a resource being read no longer exists. To do this, we'll return
the ID property that was read back. If it is empty, it means the
resource is gone. If it is non-empty, we expect it to match the input.
* Implement closure scope chain analysis in pure TypeScript
This change makes use of four V8 intrinsics to avoid having to use a
native module to inspect the scope chains of live Function objects. This
unfortunately leads to the limitation of not allowing captures of 'this'
in arrow functions, but that is something we are willing to live with
for now.
* Remove native module build and restore from the Makefile
* CR feedback: Be a little more efficient when scanning the scope chain
* Nuke everything related to custom Node versions and the native Node module
* CR feedback: rename native.ts -> v8.ts, document some interfaces in v8.ts
As I began to write code using the ability to perform resource
lookups, especially in the code-generators, I realized the way it
was surfaced as an argument to the Resource base constructor would
lead to overload explosion. Instead of doing that, let's pass it
in the ResourceOptions bag.
Prior to this change, if you ended up with multiple Pulumi SDK
packages loaded side-by-side, we would fail in obscure ways. The
reason for this was that we initialize and store important state
in static variables. In the case that you load the same library
twice, however, you end up with separate copies of said statics,
which means we would be missing engine RPC addresses and so on.
This change adds the ability to recover from this situation by
mirroring the initialized state in process-wide environment
variables. By doing this, we can safely recover simply by reading
them back when we detect that they are missing. I think we can
eventually go even further here, and eliminate the entry point
launcher shim altogether by simply having the engine launch the
Node program with the right environment variables. This would
be a nice simplification to the system (fewer moving pieces).
There is still a risk that the separate copy is incompatible.
Presumably the reason for loading multiple copies is that the
NPM/Yarn version solver couldn't resolve to a shared version.
This may yield obscure failure modes should RPC interfaces change.
Figuring out what to do here is part of pulumi/pulumi#957.
This fixespulumi/pulumi#777 and pulumi/pulumi#1017.
This change skips unknown IDs during read operations. This can happen
when a read is performed using the output property of another resource
during planning. This is intentionally supported via ID being an
Input<ID> and all we need to do for this to work correctly is skip the
actual provider RPC and the runtime will propagate unknown outputs as
usual.
This change wires up the new Read RPC method in such a manner that
Pulumi programs can invoke it. This is technically not required for
refreshing state programmatically (as in pulumi/pulumi#1081), however
it's a feature we had eons ago and have wanted since (see
pulumi/pulumi#83), and will allow us to write code like
let vm = aws.ec2.Instance.get("my-vm", "i-07043cd97bd2c9cfc");
// use any property from here on out ...
The way this works is simply by bridging the Pulumi program via its
existing RPC connection to the engine, much like Invoke and
RegisterResource RPC requests already do, and then invoking the proper
resource provider in order to read the state. Note that some resources
cannot be uniquely identified by their ID alone, and so an extra
resource state bag may be provided with just those properties required.
This came almost for free (okay, not exactly) and will come in handy as
we start gaining experience with reading live state from resources.
This commit changes two things about our resource model:
* Stop performing Pulumi Engine-side diffing of resource state.
Instead, we defer to the resource plugins themselves to determine
whether a change was made and, if so, the extent of it. This
manifests as a simple change to the Diff function; it is done in
a backwards compatible way so that we continue with legacy diffing
for existing resource provider plugins.
* Add a Read RPC method for resource providers. It simply takes a
resource's ID and URN, plus an optional bag of further qualifying
state, and it returns the current property state as read back from
the actual live environment. Note that the optional bag of state
must at least include enough additional properties for resources
wherein the ID is insufficient for the provider to perform a lookup.
It may, however, include the full bag of prior state, for instance
in the case of a refresh operation.
This is part of pulumi/pulumi#1108.
* Improve the error message arising from missing required configs for
resource providers
If the resource provider that we are speaking to is new enough, it will send
across a list of keys and their descriptions alongside an error
indicating that the provider we are configuring is missing required
config. This commit packages up the list of missing keys into an error
that can be presented nicely to the user.
* Code review feedback: renaming simplification and correcting errors in comments
* Send structured errors across RPC boundaries
This brings us closer to gRPC best practices where we send structured
errors with error codes across RPC endpoints. The new "rpcerrors"
package can wrap errors from RPC endpoints, so RPC servers can attach
some additional context as to why a request failed.
* Code review feedback:
1. Rename rpcerrors -> rpcerror, better package name
2. Rename RPCError -> Error, RPCErrorCause -> ErrorCause, names
suggested by gometalinter to improve their package-qualified names
3. Fix import organization in rpcerror.go
Our previous strategy of just using `git describe --tags --dirty` to
compute a version caused issues. The major one was that since version
sort lexigrapically, git's strategy of having a commit count without
leading zeros lead to cases where 0.11.0-dev-9 was "newer than"
0.11.0-dev-10 which is not what you want at all.
With this change, we compute a version by first seeing if the commit
is tagged, and if so, we use that tag. Otherwise, we take the closest
tag and to it append the unix timestamp of the commit and then append
a git hash.
Because we use the commit timestamp, things will sort correctly again.
Part of pulumi/home#174
We need to support the current version of the nodejs language host
running programs that use older version of @pulumi/pulumi where the
runtime expected config keys to look like
`<package>:config:<name>`. In the language host we actually did the
transformation from the new format to the old one, for compatability
reasons but we then droped the transfomed value on the floor.
The four concerns are:
parsing a v8 function string so we can figure out captured variables.
walkgin the function/object graph producing the graph we will serialize.
rewriting constructors and methods so that 'super' works.
serializing graph to text.
Also, rename/cleanup a bunch of serialization code.
Also, generate better environment names in the serialized closure code. Thsi code should be much easier to make sense of as hte names will better track to the original names in the user code.
Also, dedupe simple non-capturing functions. This helps ensure we don't spit out N copies of __awaiter (one per file it is declared in).
This special error kind should be used by all Pulumi components as the error type for user input validation errors. Although it can already be referenced via `@pulumi/pulumi/errors`, also explicitly export it directly on `@pulumi/pulumi`.
This API was introduced to aid the refactoring, but it isn't something
we want to support long term. Remove it and for a few places, push
passing config.Key around more, instead of converting to the old type
eagerly.
We now unify new Config("package") and new Config("package:config"),
printing a warning when the new Config("package:config") form is
used and pointing consumers towards just new Config("package")
I've updated our examples to use the newer syntax, but I've added a
test ot the langhost to ensure both forms work.
Fixes#923
Make many fixes to closure serialization
Primary things that i've done as part of this change:
Added support for cyclic objects.
Properly serialize objects that are shared across different function. previously you would get multiple copies, now you properly reference the same copy.
Remove the usages of 'hashes' for functions. Because we track identity of objects, we no longer need them.
Serialize properties of functions (if they have any).
Handle Objects/Functions with different __proto__s than normal. i.e. classes/constructors. but also anything the user may have done themselves to the object.
Handle generator functions.
Handle functions with 'computed' names.
Handle functions with 'symbol' names.
Handle serializing Promises as Promises.
Removed the dual Closure/AsyncClosure tree. One existed solely so we could have a tree without promises (for use in testing maybe?). Because this all exists in a part of our codebase that is entirely async, it's fine to have promises in the tree, and to await them when serializing the Closure to a string.
Handle serializing class-constructors and methods. Including properly handling 'super' calls.
* Produce better error messages when the main module is not found
If we fail to load a program's main module, inspect the program's
package.json and attempt to diagnose why the main module load failed.
* Code review feedback: entrypoint -> entry point, call out npm build explicitly, simplify control flow
* Code review feedback: add a little more levity to the unknown exception error message
This change includes a lot more functionality. Enough to actually
run the webserver-py example through previews, updates, and destroys!
* Actually wire up the gRPC connections to the engine/monitor.
* Move the Node.js and Python generated Protobuf/gRPC files underneath
the actual SDK directories to simplify this generally. No more
copying during `make` and, in fact, this was required to give a smoother
experience with good packages/modules for the Python's SDK development.
* Build the Python egg during `make build`.
* Add support for program stacks. Just like with the Node.js runtime,
we will auto-parent any resources without explicit parents to a single
top-level resource component.
* Add support for component resource output properties.
* Add get_project() and get_stack() functions for retrieving the current
project and stack names.
* Properly use UNKNOWN sentinels.
* Add a set_outputs() function on Resource. This is defined by the
code-generator and allows custom logic for output property setting.
This is cleaner than the way we do this in Node.js, and gives us a
way to ensure that output properties are "real" properties, complete
with member documentation. This also gives us a hook to perform
name demangling, which the code-generator typically controls anyway.
* Add package dependencies to setuptools.py and requirements.txt.
1. Various idiomatic Go and TypeScript fixes
2. Add an integration test that end-to-end roundtrips dependency
information for a simple Pulumi program
3. Add an additional test assert that tests that dependency information
comes from the language host as expected
This commit does two things:
1. All dependencies of a resource, both implicit and explicit, are
communicated directly to the engine when registering a resource. The
engine keeps track of these dependencies and ultimately serializes
them out to the checkpoint file upon successful deployment.
2. Once a successful deployment is done, the new `pulumi stack
graph` command reads the checkpoint file and outputs the dependency
information within in the DOT format.
Keeping track of dependency information within the checkpoint file is
desirable for a number of reasons, most notably delete-before-create,
where we want to delete resources before we have created their
replacement when performing an update.
This change refactors the way we do ${VERSION} substitution in both
the Node.js SDK's version.js and package.json, so that it can work on
Windows. This is required now that we are actually parsing semvers.
On Windows, when we launch the language host, it will end up with
a ".exe" file extension at the end of os.Args[0]. This leads us to
produce a garbage filename for the -exec script -- namely,
pulumi-language-nodejs.exe-exec -- which, of course fails. We simply
need to trim off the ".exe" bit before producing the script name.
