This change improves our output formatting by generally adding
fewer prefixes. As shown in pulumi/pulumi#359, we were being
excessively verbose in many places, including prefixing every
console.out with "langhost[nodejs].stdout: ", displaying full
stack traces for simple errors like missing configuration, etc.
Overall, this change includes the following:
* Don't prefix stdout and stderr output from the program, other
than the standard "info:" prefix. I experimented with various
schemes here, but they all felt gratuitous. Simply emitting
the output seems fine, especially as it's closer to what would
happen if you just ran the program under node.
* Do NOT make writes to stderr fail the plan/deploy. Previously
we assumed that any console.errors, for instance, meant that
the overall program should fail. This simply isn't how stderr
is treated generally and meant you couldn't use certain
logging techniques and libraries, among other things.
* Do make sure that stderr writes in the program end up going to
stderr in the Pulumi CLI output, however, so that redirection
works as it should. This required a new Infoerr log level.
* Make a small fix to the planning logic so we don't attempt to
print the summary if an error occurs.
* Finally, add a new error type, RunError, that when thrown and
uncaught does not result in a full stack trace being printed.
Anyone can use this, however, we currently use it for config
errors so that we can terminate with a pretty error message,
rather than the monstrosity shown in pulumi/pulumi#359.
This includes a few changes:
* The repo name -- and hence the Go modules -- changes from pulumi-fabric to pulumi.
* The Node.js SDK package changes from @pulumi/pulumi-fabric to just pulumi.
* The CLI is renamed from lumi to pulumi.
There were two problems:
- node-gyp configure was failing because of different shell syntax
between windows and *nix.
- MSVC 2015 is not smart enough to understand our use of strlen actually
results in a constant value and prevents us from using it to create an
array, move to a macro based solution.
This adds back Computed<T> as a short-hand for Promise<T | undefined>.
Subtly, all resource properties need to permit undefined flowing through
during planning Rather than forcing the long-hand version, which is easy
to forget, we'll keep the convention of preferring Computed<T>. It's
just a typedef and the runtime type is just a Promise.
As part of pulumi/pulumi-fabric#331, we've been exploring just using
undefined to indicate that a property value is absent during planning.
We also considered blocking the message loop to simplify the overall
programming model, so that all asynchrony is hidden.
It turns out ThereBeDragons 🐲 anytime you try to block the
message loop. So, we aren't quite sure about that bit.
But the part we are convicted about is that this Computed/Property
model is far too complex. Furthermore, it's very close to promises, and
yet frustratingly so far away. Indeed, the original thinking in
pulumi/pulumi-fabric#271 was simply to use promises, but we wanted to
encourage dataflow styles, rather than control flow. But we muddied up
our thinking by worrying about awaiting a promise that would never resolve.
It turns out we can achieve a middle ground: resolve planning promises to
undefined, so that they don't lead to hangs, but still use promises so
that asynchrony is explicit in the system. This also avoids blocking the
message loop. Who knows, this may actually be a fine final destination.
This change flips the polarity on parallelism: rather than having a
--serialize flag, we will have a --parallel=P flag, and by default
we will shut off parallelism. We aren't benefiting from it at the
moment (until we implement pulumi/pulumi-fabric#106), and there are
more hidden dependencies in places like AWS Lambdas and Permissions
than I had realized. We may revisit the default, but this allows
us to bite off the messiness of dependsOn only when we benefit from
it. And in any case, the --parallel=P capability will be useful.
This change adds an optiona dependsOn parameter to Resource constructors,
to "force" a fake dependency between resources. We have an extremely strong
desire to resort to using this only in unusual cases -- and instead rely
on the natural dependency DAG based on properties -- but experience in other
resource provisioning frameworks tells us that we're likely to need this in
the general case. Indeed, we've already encountered the need in AWS's
API Gateway resources... and I suspect we'll run into more especially as we
tackle non-serverless resources like EC2 Instances, where "ambient"
dependencies are far more commonplace.
This also makes parallelism the default mode of operation, and we have a
new --serialize flag that can be used to suppress this default behavior.
Full disclosure: I expect this to become more Make-like, i.e. -j 8, where
you can specify the precise width of parallelism, when we tackle
pulumi/pulumi-fabric#106. I also think there's a good chance we will flip
the default, so that serial execution is the default, so that developers
who don't benefit from the parallelism don't need to worry about dependsOn
in awkward ways. This tends to be the way most tools (like Make) operate.
This fixespulumi/pulumi-fabric#335.
This change implements recursive closure captures. This permits
cases like the following
{
function f() { g(); }
function g() { f(); }
}
and the slightly more useful
class C {
this.x = 42;
this.f = () => x;
}
To do this requires caching the environment objects and permitting
cycles in the resulting environment graph. The closure emitter code
already knows how to handle this.
In addition, we must mark captures of `this` as free variables.
This resolvespulumi/pulumi-fabric#333.
This ensures RPC channels stay alive until logs finish. It also
makes provisions for logs that come in *after* shutdown has begun,
but before it has finished, by observing that the keepalive promise
has changed between the time of initiating the callback and running it.
* Initialize the diganostics logger with opts.Debug when doing
a Deploy, like we do Plan.
* Don't spew leaked promises if there were Log.errors.
* Serialize logging RPC calls so that they can't appear out of order.
* Print stack traces in more places and, in particular, remember
the original context for any errors that may occur asynchronously,
like resource registration and calls to mapValue.
* Include origin stack traces generally in more error messages.
* Add some more mapValue test cases.
* Only undefined-propagate mapValue values during dry-runs.
This change serializes all resource operations. Please see
pulumi/pulumi#335 for more details. In a nutshell, there are
resources that have implicit hidden dependencies and now that
the runtime is fully asynchronous, we are tripping over problems
left and right (even worse, they are non-deterministic). All
of the problems have been in the AWS API Gateway resources;
until we come up with a holistic solution here, serializing all
calls should make things more stable in the interim.
This change upgrades gRPC to 1.6.0 to pick up a few bug fixes.
We also use the full address for gRPC endpoints, including the
interface name, as otherwise we pick the wrong interface on Linux.
There's a fair bit of clean up in here, but the meat is:
* Allocate the language runtime gRPC client connection on the
goroutine that will use it; this eliminates race conditions.
* The biggie: there *appears* to be a bug in gRPC's implementation
on Linux, where it doesn't implement WaitForReady properly. The
behavior I'm observing is that RPC calls will not retry as they
are supposed to, but will instead spuriously fail during the RPC
startup. To work around this, I've added manual retry logic in
the shared plugin creation function so that we won't even try
to use the client connection until it is in a well-known state.
pulumi/pulumi-fabric#337 tracks getting to the bottom of this and,
ideally, removing the work around.
The other minor things are:
* Separate run.js into its own module, so it doesn't include
index.js and do a bunch of random stuff it shouldn't be doing.
* Allow run.js to be invoked without a --monitor. This makes
testing just the run part of invocation easier (including
config, which turned out to be super useful as I was debugging).
* Tidy up some messages.
The change to tear down RPC connections after the program exits --
to fix problems on Linux presumably due to the way libuv is implemented --
unfortunately introduces nondeterminism and overzealous termination that
can happen at inopportune times. Instead, we need to wait for the current
RPC queue to drain. To fix this, we'll maintain a list of currently active
RPC calls and, only once they have completed, will we close the clients.
This change closes the gRPC client connections, as they keep the
Node.js message loop alive on Linux (but, strangely, not Mac;
regardless, a good thing to do anyway...)
We have an issue in the runtime right now where we serialize closures
asynchronously, meaning we make it possible to form cycles between
resource graphs (something that ought to be impossible in our model,
where resources are "immutable" after creation and cannot form cycles).
Let me tell you a tale of debugging this ...
Well, no, let's not do that. But thankfully I've left behind some
little utilities that might make debugging such a thing easier down
the road. Namely:
* By default, most of our core runtime promises leverage a leak handler
that will log an error message should the process exit with certain
critical unresolved promises. This error message will include some
handy context (like whether it was an input promise) as well as a
stack trace for its point of creation.
* Optionally, with a flag in runtime/debuggable.ts, you may wire up
a hang detector, for situations where we may want to detect this
situation sooner than process exit, using the regular message loop.
This uses a defined timeout, prints the same diagnostics as the
leak detector when a hang is detected, and is disabled by default.
This fixes a few problems with dependent resolutions and hardens
even more promises-related error paths, so we swallow precisely zero
errors (or at least we hope so). This also digs through multi-level
chains of promises and computed properties as needed for nested mapValues.
This change adds support for awaiting any Computed<T> and Promise<T>s
that were captured inside of a function's closure. This preserves our
ability to capture, for example, resource state that ends up getting
serialized as the final resource state, rather than a snapshot of the
(mostly unresolved) resource state at the time of serialization.
This change moves the environment entry serialization logic into
JavaScript, where it's a bit easier to author and maintain. We
also switch to using Object.keys, so that we only walk the enumerable
properties of objects (to avoid internal member functions and to
generally leverage our current style of writing code). This is
just a temporary stopgap until we figure out more rigorous semantics
for what it means to serialize entire objects ...
* Use `global.hasOwnProperty(ident)`, rather than `global[ident] !== undefined`,
to avoid classifying references to globals as free variables. Surprise(!!),
the prior logic wouldn't work for `undefined` itself... 😒
* Expand this check to include the built-in Node.js module variables, namely
`__dirname`, `__filename`, `exports`, `module`, and `require`, so that
references to them don't get classified as serializable free variables either.
* Place catch variables in scope, so that `catch (err) { ... }` won't yield
free variables for references to `err` within `...`.
* Place recursive function definitions into the top-level `var`-like scope of
variables so that we don't consider references to them free.
* Harden all error pathways in the native C++ add-on so that we terminate
anytime an exception is in-flight, rather than limping along and making
things worse...
As I started rolling this out, I realized that end user code actually
has to use this type sometimes. And that the current names are inconsistent,
after eschewing Property<T> in favor of Computed<T>. The new names read better.
This change makes a few simplifications to how properties are exposed in
the system, mostly in the name of usability, but also to feel a bit more
like "idiomatic JavaScript". Namely:
* Rename `then` to `mapValue`. This hopefully helps to suggest that this
is meant for a dataflow style of programming.
* Move Property<T> into the runtime module, and remove PropertyState<T>,
collapsing back down to a single type. This also eliminates some of the
messy internal runtime casting, accessing of internal members, etc.
* Export a Computed<T> interface from the root of the module. This is
the entirety of the public-facing surface area for properties, and
exposes that single `mapValue` member function. The internal runtime
logic understands how to handle Property<T>s specifically in addition
to Computed<T>s more generally (in case someone writes their own).
If a resource's planning operation is to do nothing, we can safely
assume that all of its properties are stable. This can be used during
planning to avoid cascading updates that we know will never happen.
This changes a few aspects of resource properties:
* Move all runtime-related goo into the runtime module, in an
internal PromiseState class. This encapsulates the internal
state transitions and protects against misuse. It also allows
us to clean up the public API for the Property<T> type so that
it's entirely suitable for external usage.
* Track input and output property values distinctly. It turns
out we want to key off events differently. For example, to marshal
property values to a resource provider, we only care about the
inputs. For final property values that are used in, say, thens
or as inputs to other properties, we want the output property value.
* Be more precise about when an output is truly final, and known, or
unknown due to planning/dry-runs. Note that this does mean that
we'll encounter unknown values more frequently because, aside from
IDs and URNs, we can't say for sure that arbitrary properties will never
change post-creation. We have ideas on how to denote this; see
pulumi/pulumi-fabric#330 for more details.
This change renames String, File, and Remote to StringAsset, FileAsset,
and RemoteAsset, largely to avoid conflicting with the built-in JavaScript
String type, but also because it mirrors our Archive naming strategy.
This fixes a few things in the SDK preventing deployments (versus plans):
* Don't fully resolve when a link resolves. This will be handled during
the final completion of the resource state.
* Skip "id" and "urn" for property resolution, since they are handled
explicitly based on the RPC messages. The "id" is often in the response
payload because Terraform stores it as a property. We don't need it.
* Lazily allocate Property<T> objects if necessary when the response
from the resulting resource operation comes back.
* Improve a few error messages.
This change adds the ability to perform runtime logging, including
debug logging, that wires up to the Pulumi Fabric engine in the usual
ways. Most stdout/stderr will automatically go to the right place,
but this lets us add some debug tracing in the implementation of the
runtime itself (and should come in handy in other places, like perhaps
the Pulumi Framework and even low-level end-user code).
We don't actually need to copy the headers, becasue the include
path order for the GYP-generated project files will include them
in the correct order. This simplifies the script and ordering.
This change adds a `make configure` target, which handles preparing
the environment for building the project. This includes existing
steps, like dep ensure and yarn installing the Node.js SDK NPM
dependencies, and also includes downloading the right Node.js/V8
includes, putting them in the right place, and then generating the
appropriate node-gyp project files that reference those includes.
This change implements free variable calculations and wires it up
to closure serialization. This is recursive, in the sense that
the serializer may need to call back to fetch free variables for
nested functions encountered during serialization.
The free variable calculation works by parsing the serialized
function text and walking the AST, applying the usual scoping rules
to determine what is free. In particular, it respects nested
function boundaries, and rules around var, let, and const scoping.
We are using Acorn to perform the parsing. I'd originally gone
down the path of using V8, so that we have one consistent parser
in the game, however unfortunately neither V8's parser nor its AST
is a stable API meant for 3rd parties. Unlike the exising internal
V8 dependencies, this one got very deep very quickly, and I became
nervous about maintaining all those dependencies. Furthermore,
by doing it this way, we can write the free variable logic in
JavaScript, which means one fewer C++ component to maintain.
This also includes a fairly significant amount of testing, all
of which passes! 🎉
This change contains an initial implementation of closure serialization
built atop V8, rather than our own custom runtime. This requires that
we use a Node.js dynamic C++ module, so that we can access the V8
APIs directly. No build magic is required beyond node-gyp.
For the most part, this was straight forward, except for one part: we
have to use internal V8 APIs. This is required for two reasons:
1) We need access to the function's lexical closure environment, so
that we may look up closure variables. Although there is a
tantalizingly-close v8::Object::CreationContext, its implementation
intentionally pokes through closure contexts in order to recover
the Function constructor context instead. That's not what we
want. We want the raw, unadulterated Function::context.
2) We need to control the lexical lookups of free variables so that
they can look past chained contexts, lexical contexts, withs, and
eval-style context extensions. Simply runing a v8::Script, or
simulating an eval, doesn't do the trick. Hence, we need to access
the unexported v8::internal::Context::Lookup function.
There is a third reason which is not yet implemented: free variable
calculation. I could use Esprima, or do my own scanner for free
variables, but I'd prefer to simply use the V8 parser so that we're
using the same JavaScript parser across all components. That too
is not part of the v8.h API, so we'll need to crack it open more.
To be clear, these are still exported public APIs, in proper headers
that are distributed with both Node and V8. They simply aren't part
of the "stable" v8.h surface area. As a result, I do expect that
maintaining this will be tricky, and I'd like to keep exploring how
to do this without needing the internal dependency. For instance,
although this works with node-gyp just fine, we will probably be
brittle across versions of Node/V8, when the internal APIs might be
changing. This will introduce unfortunate versioning headaches (all,
hopefully and thankfully, caught at compile-time).
The organization of packages underneath lib/ breaks the easy consumption
of submodules, a la
import {FileAsset} from "@pulumi/pulumi-fabric/asset";
We will go back to having everything hanging off the module root directory.
This change stops throwing an error if the resource monitor hasn't been
configured, and instead emits a warning. This will only go out a single
time, and can be suppressed by setting a config flag, but enables running
Pulumi programs directly via `node`, which can be useful for testing.
Of course, when this is done, allocating resource objects has no effect.
This change rearanges serialization of properties in a few ways:
* Mirror the asset/archive serialization that we use in the fabric
itself, so that we can recover the nature of these objects on
both side of the RPC boundary.
* Wait for promises to settle before marshaling resource properties.
This allows for I/O in creating a resource's state. Note that
we of course still do not block awaiting resolution of resource
output properties during dry runs (planning), because they will
never resolve. This is distinctly different from promises.
* Add tests for the above.
The definition of PropertyValue<T> should not imply undefined as a legal
value, since this depends entirely on whether it is a required or optional
property. The inner guts of the runtime logic that populates properties,
of course, needs to permit undefined, but this shouldn't leak into the
user model. This change thus eliminates undefined from PropertyValue<T>'s
definition, and pushes it into the few places where undefined is actually legal.
This change adds getX and requireX helper functions for configuration,
making it easy for packages to convert from Lumi's current weakly typed
config system, where everything is a string, into the internal JavaScript
representation, which is often a boolean, number, or complex array/object.
As explained in pulumi/pulumi-fabric#293, we were a little ad-hoc in
how configuration was "applied" to resource providers.
In fact, config wasn't ever communicated directly to providers; instead,
the resource providers would simply ask the engine to read random heap
locations (via tokens). Now that we're on a plan where configuration gets
handed to the program at startup, and that's that, and where generally
speaking resource providers never communicate directly with the language
runtime, we need to take a different approach.
As such, the resource provider interface now offers a Configure RPC
method that the resource planning engine will invoke at the right
times with the right subset of configuration variables filtered to
just that provider's package. This fixespulumi/pulumi#293.
This makes a few tweaks to get the integration tests passing:
* Add `runtime: nodejs` to the minimal example's `Lumi.yaml` file.
* Remove usage of `@lumi/lumirt { printf }` and just use `console.log`.
* Remove calls to `lumijs` in the integration test framework and
the minimal example's package.json. Instead, we just run
`yarn run build`, which itself internally just invokes `tsc`.
* Add package validation logic and eliminate the pkg/compiler/metadata
library, in favor of the simpler code in pkg/engine.
* Simplify the Node.js langhost plugin CLI, and simply take an
argument rather than requiring required and optional --flags.
* Use a default path of "." if the program path isn't provided. This
is a legal scenario if you've passed a pwd and just want to load
the package's default module ("./index.js" or whatever main says).
* Add an executable script, lumi-langhost-nodejs, that fires up the
`bin/cmd/langhost/index.js` file to serve the Node.js language plugin.
This changes Resource's constructor slightly, to take a map of
PropertyValues, rather than Properties. This simplifies the interface,
lets us hide the creation of Properties (meaning we can also hide the
resolution capabilities entirely), and also avoids mistakes like
accidentally passing values and/or other resource properties directly.
This change introduces the notion of a "dry run" into the property
serialization logic, since this controls whether we wait for dependent
linked property values to arrive or not. It also changes the test
harness to run all tests both ways: once in planning mode (when properties
will show up as "unknown" and the second time in deployment mode (when
properties will have settled to their final values).
This adds a test case for the simple input/output property cases.
In particular, it neither covers "linked" properties resulting from
dataflow nor promise properties resulting from I/O operations. But
it does test many basic JSON input and output cases.
Also fixes a few things:
* Property's `resolver` property must be set to undefined to prevent
multiple resolutions. (This is still in flux and I'm sure will
change shape before being settled.)
* Use `this.link`, not `this.linked`, to tell if a property is linked.
* Push all property initialization down into the
`runtime.registerResource` routine. In practice, the old pattern
didn't really work, since `this` is inaccessible prior to `super(..)`.
* Eliminate our custom marshaling and unmarshaling routines in favor
of the nifty built-in gRPC ones.
This is the initial step towards redefining Lumi as a library that runs
atop vanilla Node.js/V8, rather than as its own runtime.
This change is woefully incomplete but this includes some of the more
stable pieces of my current work-in-progress.
The new structure is that within the sdk/ directory we will have a client
library per language. This client library contains the object model for
Lumi (resources, properties, assets, config, etc), in addition to the
"language runtime host" components required to interoperate with the
Lumi resource monitor. This resource monitor is effectively what we call
"Lumi" today, in that it's the thing orchestrating plans and deployments.
Inside the sdk/ directory, you will find nodejs/, the Node.js client
library, alongside proto/, the definitions for RPC interop between the
different pieces of the system. This includes existing RPC definitions
for resource providers, etc., in addition to the new ones for hosting
different language runtimes from within Lumi.
These new interfaces are surprisingly simple. There is effectively a
bidirectional RPC channel between the Lumi resource monitor, represented
by the lumirpc.ResourceMonitor interface, and each language runtime,
represented by the lumirpc.LanguageRuntime interface.
The overall orchestration goes as follows:
1) Lumi decides it needs to run a program written in language X, so
it dynamically loads the language runtime plugin for language X.
2) Lumi passes that runtime a loopback address to its ResourceMonitor
service, while language X will publish a connection back to its
LanguageRuntime service, which Lumi will talk to.
3) Lumi then invokes LanguageRuntime.Run, passing information like
the desired working directory, program name, arguments, and optional
configuration variables to make available to the program.
4) The language X runtime receives this, unpacks it and sets up the
necessary context, and then invokes the program. The program then
calls into Lumi object model abstractions that internally communicate
back to Lumi using the ResourceMonitor interface.
5) The key here is ResourceMonitor.NewResource, which Lumi uses to
serialize state about newly allocated resources. Lumi receives these
and registers them as part of the plan, doing the usual diffing, etc.,
to decide how to proceed. This interface is perhaps one of the
most subtle parts of the new design, as it necessitates the use of
promises internally to allow parallel evaluation of the resource plan,
letting dataflow determine the available concurrency.
6) The program exits, and Lumi continues on its merry way. If the program
fails, the RunResponse will include information about the failure.
Due to (5), all properties on resources are now instances of a new
Property<T> type. A Property<T> is just a thin wrapper over a T, but it
encodes the special properties of Lumi resource properties. Namely, it
is possible to create one out of a T, other Property<T>, Promise<T>, or
to freshly allocate one. In all cases, the Property<T> does not "settle"
until its final state is known. This cannot occur before the deployment
actually completes, and so in general it's not safe to depend on concrete
resolutions of values (unlike ordinary Promise<T>s which are usually
expected to resolve). As a result, all derived computations are meant to
use the `then` function (as in `someValue.then(v => v+x)`).
Although this change includes tests that may be run in isolation to test
the various RPC interactions, we are nowhere near finished. The remaining
work primarily boils down to three things:
1) Wiring all of this up to the Lumi code.
2) Fixing the handful of known loose ends required to make this work,
primarily around the serialization of properties (waiting on
unresolved ones, serializing assets properly, etc).
3) Implementing lambda closure serialization as a native extension.
This ongoing work is part of pulumi/pulumi-fabric#311.