As I started rolling this out, I realized that end user code actually
has to use this type sometimes. And that the current names are inconsistent,
after eschewing Property<T> in favor of Computed<T>. The new names read better.
This change makes a few simplifications to how properties are exposed in
the system, mostly in the name of usability, but also to feel a bit more
like "idiomatic JavaScript". Namely:
* Rename `then` to `mapValue`. This hopefully helps to suggest that this
is meant for a dataflow style of programming.
* Move Property<T> into the runtime module, and remove PropertyState<T>,
collapsing back down to a single type. This also eliminates some of the
messy internal runtime casting, accessing of internal members, etc.
* Export a Computed<T> interface from the root of the module. This is
the entirety of the public-facing surface area for properties, and
exposes that single `mapValue` member function. The internal runtime
logic understands how to handle Property<T>s specifically in addition
to Computed<T>s more generally (in case someone writes their own).
If a resource's planning operation is to do nothing, we can safely
assume that all of its properties are stable. This can be used during
planning to avoid cascading updates that we know will never happen.
This changes a few aspects of resource properties:
* Move all runtime-related goo into the runtime module, in an
internal PromiseState class. This encapsulates the internal
state transitions and protects against misuse. It also allows
us to clean up the public API for the Property<T> type so that
it's entirely suitable for external usage.
* Track input and output property values distinctly. It turns
out we want to key off events differently. For example, to marshal
property values to a resource provider, we only care about the
inputs. For final property values that are used in, say, thens
or as inputs to other properties, we want the output property value.
* Be more precise about when an output is truly final, and known, or
unknown due to planning/dry-runs. Note that this does mean that
we'll encounter unknown values more frequently because, aside from
IDs and URNs, we can't say for sure that arbitrary properties will never
change post-creation. We have ideas on how to denote this; see
pulumi/pulumi-fabric#330 for more details.
This change renames String, File, and Remote to StringAsset, FileAsset,
and RemoteAsset, largely to avoid conflicting with the built-in JavaScript
String type, but also because it mirrors our Archive naming strategy.
This fixes a few things in the SDK preventing deployments (versus plans):
* Don't fully resolve when a link resolves. This will be handled during
the final completion of the resource state.
* Skip "id" and "urn" for property resolution, since they are handled
explicitly based on the RPC messages. The "id" is often in the response
payload because Terraform stores it as a property. We don't need it.
* Lazily allocate Property<T> objects if necessary when the response
from the resulting resource operation comes back.
* Improve a few error messages.
This change adds the ability to perform runtime logging, including
debug logging, that wires up to the Pulumi Fabric engine in the usual
ways. Most stdout/stderr will automatically go to the right place,
but this lets us add some debug tracing in the implementation of the
runtime itself (and should come in handy in other places, like perhaps
the Pulumi Framework and even low-level end-user code).
We don't actually need to copy the headers, becasue the include
path order for the GYP-generated project files will include them
in the correct order. This simplifies the script and ordering.
This change adds a `make configure` target, which handles preparing
the environment for building the project. This includes existing
steps, like dep ensure and yarn installing the Node.js SDK NPM
dependencies, and also includes downloading the right Node.js/V8
includes, putting them in the right place, and then generating the
appropriate node-gyp project files that reference those includes.
This change implements free variable calculations and wires it up
to closure serialization. This is recursive, in the sense that
the serializer may need to call back to fetch free variables for
nested functions encountered during serialization.
The free variable calculation works by parsing the serialized
function text and walking the AST, applying the usual scoping rules
to determine what is free. In particular, it respects nested
function boundaries, and rules around var, let, and const scoping.
We are using Acorn to perform the parsing. I'd originally gone
down the path of using V8, so that we have one consistent parser
in the game, however unfortunately neither V8's parser nor its AST
is a stable API meant for 3rd parties. Unlike the exising internal
V8 dependencies, this one got very deep very quickly, and I became
nervous about maintaining all those dependencies. Furthermore,
by doing it this way, we can write the free variable logic in
JavaScript, which means one fewer C++ component to maintain.
This also includes a fairly significant amount of testing, all
of which passes! 🎉
This change contains an initial implementation of closure serialization
built atop V8, rather than our own custom runtime. This requires that
we use a Node.js dynamic C++ module, so that we can access the V8
APIs directly. No build magic is required beyond node-gyp.
For the most part, this was straight forward, except for one part: we
have to use internal V8 APIs. This is required for two reasons:
1) We need access to the function's lexical closure environment, so
that we may look up closure variables. Although there is a
tantalizingly-close v8::Object::CreationContext, its implementation
intentionally pokes through closure contexts in order to recover
the Function constructor context instead. That's not what we
want. We want the raw, unadulterated Function::context.
2) We need to control the lexical lookups of free variables so that
they can look past chained contexts, lexical contexts, withs, and
eval-style context extensions. Simply runing a v8::Script, or
simulating an eval, doesn't do the trick. Hence, we need to access
the unexported v8::internal::Context::Lookup function.
There is a third reason which is not yet implemented: free variable
calculation. I could use Esprima, or do my own scanner for free
variables, but I'd prefer to simply use the V8 parser so that we're
using the same JavaScript parser across all components. That too
is not part of the v8.h API, so we'll need to crack it open more.
To be clear, these are still exported public APIs, in proper headers
that are distributed with both Node and V8. They simply aren't part
of the "stable" v8.h surface area. As a result, I do expect that
maintaining this will be tricky, and I'd like to keep exploring how
to do this without needing the internal dependency. For instance,
although this works with node-gyp just fine, we will probably be
brittle across versions of Node/V8, when the internal APIs might be
changing. This will introduce unfortunate versioning headaches (all,
hopefully and thankfully, caught at compile-time).
The organization of packages underneath lib/ breaks the easy consumption
of submodules, a la
import {FileAsset} from "@pulumi/pulumi-fabric/asset";
We will go back to having everything hanging off the module root directory.
This change stops throwing an error if the resource monitor hasn't been
configured, and instead emits a warning. This will only go out a single
time, and can be suppressed by setting a config flag, but enables running
Pulumi programs directly via `node`, which can be useful for testing.
Of course, when this is done, allocating resource objects has no effect.
This change rearanges serialization of properties in a few ways:
* Mirror the asset/archive serialization that we use in the fabric
itself, so that we can recover the nature of these objects on
both side of the RPC boundary.
* Wait for promises to settle before marshaling resource properties.
This allows for I/O in creating a resource's state. Note that
we of course still do not block awaiting resolution of resource
output properties during dry runs (planning), because they will
never resolve. This is distinctly different from promises.
* Add tests for the above.
The definition of PropertyValue<T> should not imply undefined as a legal
value, since this depends entirely on whether it is a required or optional
property. The inner guts of the runtime logic that populates properties,
of course, needs to permit undefined, but this shouldn't leak into the
user model. This change thus eliminates undefined from PropertyValue<T>'s
definition, and pushes it into the few places where undefined is actually legal.
This change adds getX and requireX helper functions for configuration,
making it easy for packages to convert from Lumi's current weakly typed
config system, where everything is a string, into the internal JavaScript
representation, which is often a boolean, number, or complex array/object.
As explained in pulumi/pulumi-fabric#293, we were a little ad-hoc in
how configuration was "applied" to resource providers.
In fact, config wasn't ever communicated directly to providers; instead,
the resource providers would simply ask the engine to read random heap
locations (via tokens). Now that we're on a plan where configuration gets
handed to the program at startup, and that's that, and where generally
speaking resource providers never communicate directly with the language
runtime, we need to take a different approach.
As such, the resource provider interface now offers a Configure RPC
method that the resource planning engine will invoke at the right
times with the right subset of configuration variables filtered to
just that provider's package. This fixespulumi/pulumi#293.
This makes a few tweaks to get the integration tests passing:
* Add `runtime: nodejs` to the minimal example's `Lumi.yaml` file.
* Remove usage of `@lumi/lumirt { printf }` and just use `console.log`.
* Remove calls to `lumijs` in the integration test framework and
the minimal example's package.json. Instead, we just run
`yarn run build`, which itself internally just invokes `tsc`.
* Add package validation logic and eliminate the pkg/compiler/metadata
library, in favor of the simpler code in pkg/engine.
* Simplify the Node.js langhost plugin CLI, and simply take an
argument rather than requiring required and optional --flags.
* Use a default path of "." if the program path isn't provided. This
is a legal scenario if you've passed a pwd and just want to load
the package's default module ("./index.js" or whatever main says).
* Add an executable script, lumi-langhost-nodejs, that fires up the
`bin/cmd/langhost/index.js` file to serve the Node.js language plugin.
This changes Resource's constructor slightly, to take a map of
PropertyValues, rather than Properties. This simplifies the interface,
lets us hide the creation of Properties (meaning we can also hide the
resolution capabilities entirely), and also avoids mistakes like
accidentally passing values and/or other resource properties directly.
This change introduces the notion of a "dry run" into the property
serialization logic, since this controls whether we wait for dependent
linked property values to arrive or not. It also changes the test
harness to run all tests both ways: once in planning mode (when properties
will show up as "unknown" and the second time in deployment mode (when
properties will have settled to their final values).
This adds a test case for the simple input/output property cases.
In particular, it neither covers "linked" properties resulting from
dataflow nor promise properties resulting from I/O operations. But
it does test many basic JSON input and output cases.
Also fixes a few things:
* Property's `resolver` property must be set to undefined to prevent
multiple resolutions. (This is still in flux and I'm sure will
change shape before being settled.)
* Use `this.link`, not `this.linked`, to tell if a property is linked.
* Push all property initialization down into the
`runtime.registerResource` routine. In practice, the old pattern
didn't really work, since `this` is inaccessible prior to `super(..)`.
* Eliminate our custom marshaling and unmarshaling routines in favor
of the nifty built-in gRPC ones.
This is the initial step towards redefining Lumi as a library that runs
atop vanilla Node.js/V8, rather than as its own runtime.
This change is woefully incomplete but this includes some of the more
stable pieces of my current work-in-progress.
The new structure is that within the sdk/ directory we will have a client
library per language. This client library contains the object model for
Lumi (resources, properties, assets, config, etc), in addition to the
"language runtime host" components required to interoperate with the
Lumi resource monitor. This resource monitor is effectively what we call
"Lumi" today, in that it's the thing orchestrating plans and deployments.
Inside the sdk/ directory, you will find nodejs/, the Node.js client
library, alongside proto/, the definitions for RPC interop between the
different pieces of the system. This includes existing RPC definitions
for resource providers, etc., in addition to the new ones for hosting
different language runtimes from within Lumi.
These new interfaces are surprisingly simple. There is effectively a
bidirectional RPC channel between the Lumi resource monitor, represented
by the lumirpc.ResourceMonitor interface, and each language runtime,
represented by the lumirpc.LanguageRuntime interface.
The overall orchestration goes as follows:
1) Lumi decides it needs to run a program written in language X, so
it dynamically loads the language runtime plugin for language X.
2) Lumi passes that runtime a loopback address to its ResourceMonitor
service, while language X will publish a connection back to its
LanguageRuntime service, which Lumi will talk to.
3) Lumi then invokes LanguageRuntime.Run, passing information like
the desired working directory, program name, arguments, and optional
configuration variables to make available to the program.
4) The language X runtime receives this, unpacks it and sets up the
necessary context, and then invokes the program. The program then
calls into Lumi object model abstractions that internally communicate
back to Lumi using the ResourceMonitor interface.
5) The key here is ResourceMonitor.NewResource, which Lumi uses to
serialize state about newly allocated resources. Lumi receives these
and registers them as part of the plan, doing the usual diffing, etc.,
to decide how to proceed. This interface is perhaps one of the
most subtle parts of the new design, as it necessitates the use of
promises internally to allow parallel evaluation of the resource plan,
letting dataflow determine the available concurrency.
6) The program exits, and Lumi continues on its merry way. If the program
fails, the RunResponse will include information about the failure.
Due to (5), all properties on resources are now instances of a new
Property<T> type. A Property<T> is just a thin wrapper over a T, but it
encodes the special properties of Lumi resource properties. Namely, it
is possible to create one out of a T, other Property<T>, Promise<T>, or
to freshly allocate one. In all cases, the Property<T> does not "settle"
until its final state is known. This cannot occur before the deployment
actually completes, and so in general it's not safe to depend on concrete
resolutions of values (unlike ordinary Promise<T>s which are usually
expected to resolve). As a result, all derived computations are meant to
use the `then` function (as in `someValue.then(v => v+x)`).
Although this change includes tests that may be run in isolation to test
the various RPC interactions, we are nowhere near finished. The remaining
work primarily boils down to three things:
1) Wiring all of this up to the Lumi code.
2) Fixing the handful of known loose ends required to make this work,
primarily around the serialization of properties (waiting on
unresolved ones, serializing assets properly, etc).
3) Implementing lambda closure serialization as a native extension.
This ongoing work is part of pulumi/pulumi-fabric#311.
