This change moves us away from using JavaScript RTTI, by way of
`instanceof`, for built-in Pulumi types. If we use `instanceof`,
then the same logical type loaded from separate copies of the
SDK package -- as will happen in SxS scenarios -- are considered
different. This isn't actually what we want. The solution is
simple: implement our own quasi-RTTI solution, using __pulumi*
properties and manual as* and is* functions. Note that we could
have skipped the as* and is* functions, but I found that they led
to slightly easier to read code.
There is one strange thing in here, which I spoke to
@CyrusNajmabadi about: SerializedOutput<T>, because it implements
Output<T> as an _interface_, did not previously masquerade as an
actual Output<T>. In other words, `instanceof` would have returned
false, and indeed a few important properties (like promise) are
missing. This change preserves that behavior, although I'll admit
that this is slightly odd. I suspect we'll want to revisit this as
part of https://github.com/pulumi/pulumi/issues/1074.
Fixes https://github.com/pulumi/pulumi/issues/1203.
Prior to this change, if you ended up with multiple Pulumi SDK
packages loaded side-by-side, we would fail in obscure ways. The
reason for this was that we initialize and store important state
in static variables. In the case that you load the same library
twice, however, you end up with separate copies of said statics,
which means we would be missing engine RPC addresses and so on.
This change adds the ability to recover from this situation by
mirroring the initialized state in process-wide environment
variables. By doing this, we can safely recover simply by reading
them back when we detect that they are missing. I think we can
eventually go even further here, and eliminate the entry point
launcher shim altogether by simply having the engine launch the
Node program with the right environment variables. This would
be a nice simplification to the system (fewer moving pieces).
There is still a risk that the separate copy is incompatible.
Presumably the reason for loading multiple copies is that the
NPM/Yarn version solver couldn't resolve to a shared version.
This may yield obscure failure modes should RPC interfaces change.
Figuring out what to do here is part of pulumi/pulumi#957.
This fixespulumi/pulumi#777 and pulumi/pulumi#1017.
We now unify new Config("package") and new Config("package:config"),
printing a warning when the new Config("package:config") form is
used and pointing consumers towards just new Config("package")
I've updated our examples to use the newer syntax, but I've added a
test ot the langhost to ensure both forms work.
Fixes#923
* Produce better error messages when the main module is not found
If we fail to load a program's main module, inspect the program's
package.json and attempt to diagnose why the main module load failed.
* Code review feedback: entrypoint -> entry point, call out npm build explicitly, simplify control flow
* Code review feedback: add a little more levity to the unknown exception error message
This change switches from child lists to parent pointers, in the
way resource ancestries are represented. This cleans up a fair bit
of the old parenting logic, including all notion of ambient parent
scopes (and will notably address pulumi/pulumi#435).
This lets us show a more parent/child display in the output when
doing planning and updating. For instance, here is an update of
a lambda's text, which is logically part of a cloud timer:
* cloud:timer:Timer: (same)
[urn=urn:pulumi:malta::lm-cloud:☁️timer:Timer::lm-cts-malta-job-CleanSnapshots]
* cloud:function:Function: (same)
[urn=urn:pulumi:malta::lm-cloud:☁️function:Function::lm-cts-malta-job-CleanSnapshots]
* aws:serverless:Function: (same)
[urn=urn:pulumi:malta::lm-cloud::aws:serverless:Function::lm-cts-malta-job-CleanSnapshots]
~ aws:lambda/function:Function: (modify)
[id=lm-cts-malta-job-CleanSnapshots-fee4f3bf41280741]
[urn=urn:pulumi:malta::lm-cloud::aws:lambda/function:Function::lm-cts-malta-job-CleanSnapshots]
- code : archive(assets:2092f44) {
// etc etc etc
Note that we still get walls of text, but this will be actually
quite nice when combined with pulumi/pulumi#454.
I've also suppressed printing properties that didn't change during
updates when --detailed was not passed, and also suppressed empty
strings and zero-length arrays (since TF uses these as defaults in
many places and it just makes creation and deletion quite verbose).
Note that this is a far cry from everything we can possibly do
here as part of pulumi/pulumi#340 (and even pulumi/pulumi#417).
But it's a good start towards taming some of our output spew.
Adds support for top-level exports in the main script of a Pulumi Program to be captured as stack-level output properties.
This create a new `pulumi:pulumi:Stack` component as the root of the resource tree in all Pulumi programs. That resources has properties for each top-level export in the Node.js script.
Running `pulumi stack` will display the current value of these outputs.
Because the Pulumi.yaml file demarcates the boundary used when
uploading a program to the Pulumi.com service at the moment, we
have trouble when a Pulumi program uses "up and over" references.
For instance, our customer wants to build a Dockerfile located
in some relative path, such as `../../elsewhere/`.
To support this, we will allow the Pulumi.yaml file to live
somewhere other than the main Pulumi entrypoint. For example,
it can live at the root of the repo, while the Pulumi program
lives in, say, `infra/`:
Pulumi.yaml:
name: as-before
main: infra/
This fixespulumi/pulumi#575. Further work can be done here to
provide even more flexibility; see pulumi/pulumi#574.
This change formats log messages the same way that Node.js does
in its console.log/error APIs. This ensures, for example, that
errors have their stack printed if present, and switches over to
just printing the error directly rather than manually toStringing it.
Adds OpenTracing in the Pulumi engine and plugin + langhost subprocesses.
We currently create a single root span for any `Enging.plan` operation - which is a single `preview`, `update`, `destroy`, etc.
The only sub-spans we currently create are at gRPC boundaries, both on the client and server sides and on both the langhost and provider plugin interfaces.
We could extend this to include spans for any other semantically meaningful sections of compute inside the engine, though initial examples show we get pretty good granularity of coverage by focusing on the gRPC boundaries.
In the future, this should be easily extensible to HTTP boundaries and to track other bulky I/O like datastore read/writes once we hook up to the PPC and Pulumi Cloud.
We expose a `--trace <endpoint>` option to enable tracing on the CLI, which we will aim to thread through to subprocesses.
We currently support sending tracing data to a Zipkin-compatible endpoint. This has been validated with both Zipkin and Jaeger UIs.
We do not yet have any tracing inside the TypeScript side of the JS langhost RPC interface. There is not yet automatic gRPC OpenTracing instrumentation (though it looks like it's in progress now) - so we would need to manually create meaningful spans on that side of the interface.
This change remembers that we failed due to an uncaught exception,
and defers the process.exit(1) until we actually reach the process's
exit event. This ensures that we drain the message queue before
exiting, which ensures that outbound messages actually reach their
destination.
As part of fixing the exit bug recently, we accidentally made errors
lead to zero exit codes. As a result, the Pulumi CLI thought the
prgoram exited ordinarily, and proceeded to do its usual planning and
deployment, rather than terminating abruptly.
This is a byproduct of how Node's process.uncaughtException handler
works. It hijacks and replaces all usual error logic, including the
process.exit part. This change simply adds back the non-zero exit.
I also added a test (and fixed one other that began failing
afterwards), so that we can prevent regressions down the road.
The `nodejs` language support is implemented as two programs: one that
manages the initial connection to the engine and provides the language
serivce itself, and another that the language service invokes in order
to run a `nodejs` Pulumi program. The latter is responsible for running
the user's program and communicating its resource requests to the
engine. Currently, `run` effectively assumes that the user's program
will run synchronously from start to finish, and will disconnect from
the engine once the user's program has completed. This assumption breaks
if the user's program requires multiple turns of the event loop to
finish its root resource requests. For example, the following program
would fail to create its second resource because the engine will be
disconnected once it reaches its `await`:
```
(async () => {
let a = new Resource();
await somePromise();
let = new Resource();
})();
```
These changes fix this issue by disconnecting from the engine during
process shutdown rather than after the user's program has finished its
first turn through the event loop.
This change adds functions, `pulumi.getProject()` and `pulumi.getStack()`,
to fetch the names of the project and stack, respectively. These can be
handy in generating names, specializing areas of the code, etc.
This fixespulumi/pulumi#429.
This change adds environment variable fallbacks for configuration
variables, such that you can either set them explicitly, as a specific
variable PULUMI_CONFIG_<K>, or an entire JSON serialized bag via
PULUMI_CONFIG.
This is convenient when simply invoking programs at the command line,
via node, e.g.
PULUMI_CONFIG_AWS_CONFIG_REGION=us-west-2 node bin/index.js
Our language host also now uses this to communicate config when invoking
a Run RPC, rather than at the command line. This fixespulumi/pulumi#336.
This exposes the existing runtime logging functionality in a way meant
for 3rd-parties to consume. This can be useful if we want to introduce
debug logging, warnings, or other things, that fit nicely with the
Pulumi CLI and overall developer workflow.
This change improves our output formatting by generally adding
fewer prefixes. As shown in pulumi/pulumi#359, we were being
excessively verbose in many places, including prefixing every
console.out with "langhost[nodejs].stdout: ", displaying full
stack traces for simple errors like missing configuration, etc.
Overall, this change includes the following:
* Don't prefix stdout and stderr output from the program, other
than the standard "info:" prefix. I experimented with various
schemes here, but they all felt gratuitous. Simply emitting
the output seems fine, especially as it's closer to what would
happen if you just ran the program under node.
* Do NOT make writes to stderr fail the plan/deploy. Previously
we assumed that any console.errors, for instance, meant that
the overall program should fail. This simply isn't how stderr
is treated generally and meant you couldn't use certain
logging techniques and libraries, among other things.
* Do make sure that stderr writes in the program end up going to
stderr in the Pulumi CLI output, however, so that redirection
works as it should. This required a new Infoerr log level.
* Make a small fix to the planning logic so we don't attempt to
print the summary if an error occurs.
* Finally, add a new error type, RunError, that when thrown and
uncaught does not result in a full stack trace being printed.
Anyone can use this, however, we currently use it for config
errors so that we can terminate with a pretty error message,
rather than the monstrosity shown in pulumi/pulumi#359.
This change flips the polarity on parallelism: rather than having a
--serialize flag, we will have a --parallel=P flag, and by default
we will shut off parallelism. We aren't benefiting from it at the
moment (until we implement pulumi/pulumi-fabric#106), and there are
more hidden dependencies in places like AWS Lambdas and Permissions
than I had realized. We may revisit the default, but this allows
us to bite off the messiness of dependsOn only when we benefit from
it. And in any case, the --parallel=P capability will be useful.
This change adds an optiona dependsOn parameter to Resource constructors,
to "force" a fake dependency between resources. We have an extremely strong
desire to resort to using this only in unusual cases -- and instead rely
on the natural dependency DAG based on properties -- but experience in other
resource provisioning frameworks tells us that we're likely to need this in
the general case. Indeed, we've already encountered the need in AWS's
API Gateway resources... and I suspect we'll run into more especially as we
tackle non-serverless resources like EC2 Instances, where "ambient"
dependencies are far more commonplace.
This also makes parallelism the default mode of operation, and we have a
new --serialize flag that can be used to suppress this default behavior.
Full disclosure: I expect this to become more Make-like, i.e. -j 8, where
you can specify the precise width of parallelism, when we tackle
pulumi/pulumi-fabric#106. I also think there's a good chance we will flip
the default, so that serial execution is the default, so that developers
who don't benefit from the parallelism don't need to worry about dependsOn
in awkward ways. This tends to be the way most tools (like Make) operate.
This fixespulumi/pulumi-fabric#335.
There's a fair bit of clean up in here, but the meat is:
* Allocate the language runtime gRPC client connection on the
goroutine that will use it; this eliminates race conditions.
* The biggie: there *appears* to be a bug in gRPC's implementation
on Linux, where it doesn't implement WaitForReady properly. The
behavior I'm observing is that RPC calls will not retry as they
are supposed to, but will instead spuriously fail during the RPC
startup. To work around this, I've added manual retry logic in
the shared plugin creation function so that we won't even try
to use the client connection until it is in a well-known state.
pulumi/pulumi-fabric#337 tracks getting to the bottom of this and,
ideally, removing the work around.
The other minor things are:
* Separate run.js into its own module, so it doesn't include
index.js and do a bunch of random stuff it shouldn't be doing.
* Allow run.js to be invoked without a --monitor. This makes
testing just the run part of invocation easier (including
config, which turned out to be super useful as I was debugging).
* Tidy up some messages.
