This includes a few changes:
* The repo name -- and hence the Go modules -- changes from pulumi-fabric to pulumi.
* The Node.js SDK package changes from @pulumi/pulumi-fabric to just pulumi.
* The CLI is renamed from lumi to pulumi.
There were two problems:
- node-gyp configure was failing because of different shell syntax
between windows and *nix.
- MSVC 2015 is not smart enough to understand our use of strlen actually
results in a constant value and prevents us from using it to create an
array, move to a macro based solution.
This adds back Computed<T> as a short-hand for Promise<T | undefined>.
Subtly, all resource properties need to permit undefined flowing through
during planning Rather than forcing the long-hand version, which is easy
to forget, we'll keep the convention of preferring Computed<T>. It's
just a typedef and the runtime type is just a Promise.
As part of pulumi/pulumi-fabric#331, we've been exploring just using
undefined to indicate that a property value is absent during planning.
We also considered blocking the message loop to simplify the overall
programming model, so that all asynchrony is hidden.
It turns out ThereBeDragons 🐲 anytime you try to block the
message loop. So, we aren't quite sure about that bit.
But the part we are convicted about is that this Computed/Property
model is far too complex. Furthermore, it's very close to promises, and
yet frustratingly so far away. Indeed, the original thinking in
pulumi/pulumi-fabric#271 was simply to use promises, but we wanted to
encourage dataflow styles, rather than control flow. But we muddied up
our thinking by worrying about awaiting a promise that would never resolve.
It turns out we can achieve a middle ground: resolve planning promises to
undefined, so that they don't lead to hangs, but still use promises so
that asynchrony is explicit in the system. This also avoids blocking the
message loop. Who knows, this may actually be a fine final destination.
This change flips the polarity on parallelism: rather than having a
--serialize flag, we will have a --parallel=P flag, and by default
we will shut off parallelism. We aren't benefiting from it at the
moment (until we implement pulumi/pulumi-fabric#106), and there are
more hidden dependencies in places like AWS Lambdas and Permissions
than I had realized. We may revisit the default, but this allows
us to bite off the messiness of dependsOn only when we benefit from
it. And in any case, the --parallel=P capability will be useful.
This change adds an optiona dependsOn parameter to Resource constructors,
to "force" a fake dependency between resources. We have an extremely strong
desire to resort to using this only in unusual cases -- and instead rely
on the natural dependency DAG based on properties -- but experience in other
resource provisioning frameworks tells us that we're likely to need this in
the general case. Indeed, we've already encountered the need in AWS's
API Gateway resources... and I suspect we'll run into more especially as we
tackle non-serverless resources like EC2 Instances, where "ambient"
dependencies are far more commonplace.
This also makes parallelism the default mode of operation, and we have a
new --serialize flag that can be used to suppress this default behavior.
Full disclosure: I expect this to become more Make-like, i.e. -j 8, where
you can specify the precise width of parallelism, when we tackle
pulumi/pulumi-fabric#106. I also think there's a good chance we will flip
the default, so that serial execution is the default, so that developers
who don't benefit from the parallelism don't need to worry about dependsOn
in awkward ways. This tends to be the way most tools (like Make) operate.
This fixespulumi/pulumi-fabric#335.
This change implements recursive closure captures. This permits
cases like the following
{
function f() { g(); }
function g() { f(); }
}
and the slightly more useful
class C {
this.x = 42;
this.f = () => x;
}
To do this requires caching the environment objects and permitting
cycles in the resulting environment graph. The closure emitter code
already knows how to handle this.
In addition, we must mark captures of `this` as free variables.
This resolvespulumi/pulumi-fabric#333.
This ensures RPC channels stay alive until logs finish. It also
makes provisions for logs that come in *after* shutdown has begun,
but before it has finished, by observing that the keepalive promise
has changed between the time of initiating the callback and running it.
* Initialize the diganostics logger with opts.Debug when doing
a Deploy, like we do Plan.
* Don't spew leaked promises if there were Log.errors.
* Serialize logging RPC calls so that they can't appear out of order.
* Print stack traces in more places and, in particular, remember
the original context for any errors that may occur asynchronously,
like resource registration and calls to mapValue.
* Include origin stack traces generally in more error messages.
* Add some more mapValue test cases.
* Only undefined-propagate mapValue values during dry-runs.
This change serializes all resource operations. Please see
pulumi/pulumi#335 for more details. In a nutshell, there are
resources that have implicit hidden dependencies and now that
the runtime is fully asynchronous, we are tripping over problems
left and right (even worse, they are non-deterministic). All
of the problems have been in the AWS API Gateway resources;
until we come up with a holistic solution here, serializing all
calls should make things more stable in the interim.
This change upgrades gRPC to 1.6.0 to pick up a few bug fixes.
We also use the full address for gRPC endpoints, including the
interface name, as otherwise we pick the wrong interface on Linux.
There's a fair bit of clean up in here, but the meat is:
* Allocate the language runtime gRPC client connection on the
goroutine that will use it; this eliminates race conditions.
* The biggie: there *appears* to be a bug in gRPC's implementation
on Linux, where it doesn't implement WaitForReady properly. The
behavior I'm observing is that RPC calls will not retry as they
are supposed to, but will instead spuriously fail during the RPC
startup. To work around this, I've added manual retry logic in
the shared plugin creation function so that we won't even try
to use the client connection until it is in a well-known state.
pulumi/pulumi-fabric#337 tracks getting to the bottom of this and,
ideally, removing the work around.
The other minor things are:
* Separate run.js into its own module, so it doesn't include
index.js and do a bunch of random stuff it shouldn't be doing.
* Allow run.js to be invoked without a --monitor. This makes
testing just the run part of invocation easier (including
config, which turned out to be super useful as I was debugging).
* Tidy up some messages.
The change to tear down RPC connections after the program exits --
to fix problems on Linux presumably due to the way libuv is implemented --
unfortunately introduces nondeterminism and overzealous termination that
can happen at inopportune times. Instead, we need to wait for the current
RPC queue to drain. To fix this, we'll maintain a list of currently active
RPC calls and, only once they have completed, will we close the clients.
This change closes the gRPC client connections, as they keep the
Node.js message loop alive on Linux (but, strangely, not Mac;
regardless, a good thing to do anyway...)
We have an issue in the runtime right now where we serialize closures
asynchronously, meaning we make it possible to form cycles between
resource graphs (something that ought to be impossible in our model,
where resources are "immutable" after creation and cannot form cycles).
Let me tell you a tale of debugging this ...
Well, no, let's not do that. But thankfully I've left behind some
little utilities that might make debugging such a thing easier down
the road. Namely:
* By default, most of our core runtime promises leverage a leak handler
that will log an error message should the process exit with certain
critical unresolved promises. This error message will include some
handy context (like whether it was an input promise) as well as a
stack trace for its point of creation.
* Optionally, with a flag in runtime/debuggable.ts, you may wire up
a hang detector, for situations where we may want to detect this
situation sooner than process exit, using the regular message loop.
This uses a defined timeout, prints the same diagnostics as the
leak detector when a hang is detected, and is disabled by default.
This fixes a few problems with dependent resolutions and hardens
even more promises-related error paths, so we swallow precisely zero
errors (or at least we hope so). This also digs through multi-level
chains of promises and computed properties as needed for nested mapValues.
This change adds support for awaiting any Computed<T> and Promise<T>s
that were captured inside of a function's closure. This preserves our
ability to capture, for example, resource state that ends up getting
serialized as the final resource state, rather than a snapshot of the
(mostly unresolved) resource state at the time of serialization.
This change moves the environment entry serialization logic into
JavaScript, where it's a bit easier to author and maintain. We
also switch to using Object.keys, so that we only walk the enumerable
properties of objects (to avoid internal member functions and to
generally leverage our current style of writing code). This is
just a temporary stopgap until we figure out more rigorous semantics
for what it means to serialize entire objects ...
* Use `global.hasOwnProperty(ident)`, rather than `global[ident] !== undefined`,
to avoid classifying references to globals as free variables. Surprise(!!),
the prior logic wouldn't work for `undefined` itself... 😒
* Expand this check to include the built-in Node.js module variables, namely
`__dirname`, `__filename`, `exports`, `module`, and `require`, so that
references to them don't get classified as serializable free variables either.
* Place catch variables in scope, so that `catch (err) { ... }` won't yield
free variables for references to `err` within `...`.
* Place recursive function definitions into the top-level `var`-like scope of
variables so that we don't consider references to them free.
* Harden all error pathways in the native C++ add-on so that we terminate
anytime an exception is in-flight, rather than limping along and making
things worse...
