2018-05-22 21:43:36 +02:00
|
|
|
// Copyright 2016-2018, Pulumi Corporation.
|
|
|
|
//
|
|
|
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
|
|
|
// you may not use this file except in compliance with the License.
|
|
|
|
// You may obtain a copy of the License at
|
|
|
|
//
|
|
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
//
|
|
|
|
// Unless required by applicable law or agreed to in writing, software
|
|
|
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
|
|
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
|
|
// See the License for the specific language governing permissions and
|
|
|
|
// limitations under the License.
|
Implement initial Lumi-as-a-library
This is the initial step towards redefining Lumi as a library that runs
atop vanilla Node.js/V8, rather than as its own runtime.
This change is woefully incomplete but this includes some of the more
stable pieces of my current work-in-progress.
The new structure is that within the sdk/ directory we will have a client
library per language. This client library contains the object model for
Lumi (resources, properties, assets, config, etc), in addition to the
"language runtime host" components required to interoperate with the
Lumi resource monitor. This resource monitor is effectively what we call
"Lumi" today, in that it's the thing orchestrating plans and deployments.
Inside the sdk/ directory, you will find nodejs/, the Node.js client
library, alongside proto/, the definitions for RPC interop between the
different pieces of the system. This includes existing RPC definitions
for resource providers, etc., in addition to the new ones for hosting
different language runtimes from within Lumi.
These new interfaces are surprisingly simple. There is effectively a
bidirectional RPC channel between the Lumi resource monitor, represented
by the lumirpc.ResourceMonitor interface, and each language runtime,
represented by the lumirpc.LanguageRuntime interface.
The overall orchestration goes as follows:
1) Lumi decides it needs to run a program written in language X, so
it dynamically loads the language runtime plugin for language X.
2) Lumi passes that runtime a loopback address to its ResourceMonitor
service, while language X will publish a connection back to its
LanguageRuntime service, which Lumi will talk to.
3) Lumi then invokes LanguageRuntime.Run, passing information like
the desired working directory, program name, arguments, and optional
configuration variables to make available to the program.
4) The language X runtime receives this, unpacks it and sets up the
necessary context, and then invokes the program. The program then
calls into Lumi object model abstractions that internally communicate
back to Lumi using the ResourceMonitor interface.
5) The key here is ResourceMonitor.NewResource, which Lumi uses to
serialize state about newly allocated resources. Lumi receives these
and registers them as part of the plan, doing the usual diffing, etc.,
to decide how to proceed. This interface is perhaps one of the
most subtle parts of the new design, as it necessitates the use of
promises internally to allow parallel evaluation of the resource plan,
letting dataflow determine the available concurrency.
6) The program exits, and Lumi continues on its merry way. If the program
fails, the RunResponse will include information about the failure.
Due to (5), all properties on resources are now instances of a new
Property<T> type. A Property<T> is just a thin wrapper over a T, but it
encodes the special properties of Lumi resource properties. Namely, it
is possible to create one out of a T, other Property<T>, Promise<T>, or
to freshly allocate one. In all cases, the Property<T> does not "settle"
until its final state is known. This cannot occur before the deployment
actually completes, and so in general it's not safe to depend on concrete
resolutions of values (unlike ordinary Promise<T>s which are usually
expected to resolve). As a result, all derived computations are meant to
use the `then` function (as in `someValue.then(v => v+x)`).
Although this change includes tests that may be run in isolation to test
the various RPC interactions, we are nowhere near finished. The remaining
work primarily boils down to three things:
1) Wiring all of this up to the Lumi code.
2) Fixing the handful of known loose ends required to make this work,
primarily around the serialization of properties (waiting on
unresolved ones, serializing assets properly, etc).
3) Implementing lambda closure serialization as a native extension.
This ongoing work is part of pulumi/pulumi-fabric#311.
2017-08-26 21:07:54 +02:00
|
|
|
|
2019-03-27 22:40:29 +01:00
|
|
|
// The very first thing we do is set up unhandled exception and rejection hooks to ensure that these
|
|
|
|
// events cause us to exit with a non-zero code. It is critically important that we do this early:
|
|
|
|
// if we do not, unhandled rejections in particular may cause us to exit with a 0 exit code, which
|
|
|
|
// will trick the engine into thinking that the program ran successfully. This can cause the engine
|
|
|
|
// to decide to delete all of a stack's resources.
|
|
|
|
//
|
|
|
|
// We track all uncaught errors here. If we have any, we will make sure we always have a non-0 exit
|
|
|
|
// code.
|
|
|
|
const uncaughtErrors = new Set<Error>();
|
|
|
|
|
|
|
|
// We also track errors we know were logged to the user using our standard `log.error` call from
|
|
|
|
// inside our uncaught-error-handler in run.ts. If all uncaught-errors above were also known to all
|
|
|
|
// be logged properly to the user, then we know the user has the information they need to proceed.
|
|
|
|
// We can then report the langhost that it should just stop running immediately and not print any
|
|
|
|
// additional superfluous information.
|
|
|
|
const loggedErrors = new Set<Error>();
|
|
|
|
|
2019-01-07 18:59:29 +01:00
|
|
|
let programRunning = false;
|
|
|
|
const uncaughtHandler = (err: Error) => {
|
2019-03-27 22:40:29 +01:00
|
|
|
uncaughtErrors.add(err);
|
2019-01-07 18:59:29 +01:00
|
|
|
if (!programRunning) {
|
|
|
|
console.error(err.stack || err.message);
|
|
|
|
}
|
|
|
|
};
|
2019-03-27 22:40:29 +01:00
|
|
|
|
|
|
|
// Keep track if we already logged the information about an unhandled error to the user.. If
|
|
|
|
// so, we end with a different exit code. The language host recognizes this and will not print
|
|
|
|
// any further messages to the user since we already took care of it.
|
|
|
|
//
|
|
|
|
// 32 was picked so as to be very unlikely to collide with any of the error codes documented by
|
|
|
|
// nodejs here:
|
|
|
|
// https://github.com/nodejs/node-v0.x-archive/blob/master/doc/api/process.markdown#exit-codes
|
|
|
|
export const nodeJSProcessExitedAfterLoggingUserActionableMessage = 32;
|
|
|
|
|
2019-01-07 18:59:29 +01:00
|
|
|
process.on("uncaughtException", uncaughtHandler);
|
|
|
|
process.on("unhandledRejection", uncaughtHandler);
|
|
|
|
process.on("exit", (code: number) => {
|
2019-03-27 22:40:29 +01:00
|
|
|
|
|
|
|
// If there were any uncaught errors at all, we always want to exit with an error code. If we
|
|
|
|
// did not, it could be disastrous for the user. i.e. not all resources may have been created,
|
|
|
|
// but the 0 code would indicate we could proceed. That could lead to many (or all) of the
|
|
|
|
// user resources being deleted.
|
|
|
|
if (code === 0 && uncaughtErrors.size > 0) {
|
|
|
|
// Now Check if this error was already logged to the user in a visible fashion. If not
|
|
|
|
// we will exit with '1', indicating that the host should give a generic message about
|
|
|
|
// things not working.
|
|
|
|
for (const err of uncaughtErrors) {
|
|
|
|
if (!loggedErrors.has(err)) {
|
|
|
|
process.exitCode = 1;
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
process.exitCode = nodeJSProcessExitedAfterLoggingUserActionableMessage;
|
2019-01-07 18:59:29 +01:00
|
|
|
}
|
|
|
|
});
|
|
|
|
|
|
|
|
// As the second thing we do, ensure that we're connected to v8's inspector API. We need to do
|
2018-11-01 23:46:21 +01:00
|
|
|
// this as some information is only sent out as events, without any way to query for it after the
|
|
|
|
// fact. For example, we want to keep track of ScriptId->FileNames so that we can appropriately
|
|
|
|
// report errors for Functions we cannot serialize. This can only be done (up to Node11 at least)
|
|
|
|
// by register to hear about scripts being parsed.
|
|
|
|
import * as v8Hooks from "../../runtime/closure/v8Hooks";
|
|
|
|
|
Implement initial Lumi-as-a-library
This is the initial step towards redefining Lumi as a library that runs
atop vanilla Node.js/V8, rather than as its own runtime.
This change is woefully incomplete but this includes some of the more
stable pieces of my current work-in-progress.
The new structure is that within the sdk/ directory we will have a client
library per language. This client library contains the object model for
Lumi (resources, properties, assets, config, etc), in addition to the
"language runtime host" components required to interoperate with the
Lumi resource monitor. This resource monitor is effectively what we call
"Lumi" today, in that it's the thing orchestrating plans and deployments.
Inside the sdk/ directory, you will find nodejs/, the Node.js client
library, alongside proto/, the definitions for RPC interop between the
different pieces of the system. This includes existing RPC definitions
for resource providers, etc., in addition to the new ones for hosting
different language runtimes from within Lumi.
These new interfaces are surprisingly simple. There is effectively a
bidirectional RPC channel between the Lumi resource monitor, represented
by the lumirpc.ResourceMonitor interface, and each language runtime,
represented by the lumirpc.LanguageRuntime interface.
The overall orchestration goes as follows:
1) Lumi decides it needs to run a program written in language X, so
it dynamically loads the language runtime plugin for language X.
2) Lumi passes that runtime a loopback address to its ResourceMonitor
service, while language X will publish a connection back to its
LanguageRuntime service, which Lumi will talk to.
3) Lumi then invokes LanguageRuntime.Run, passing information like
the desired working directory, program name, arguments, and optional
configuration variables to make available to the program.
4) The language X runtime receives this, unpacks it and sets up the
necessary context, and then invokes the program. The program then
calls into Lumi object model abstractions that internally communicate
back to Lumi using the ResourceMonitor interface.
5) The key here is ResourceMonitor.NewResource, which Lumi uses to
serialize state about newly allocated resources. Lumi receives these
and registers them as part of the plan, doing the usual diffing, etc.,
to decide how to proceed. This interface is perhaps one of the
most subtle parts of the new design, as it necessitates the use of
promises internally to allow parallel evaluation of the resource plan,
letting dataflow determine the available concurrency.
6) The program exits, and Lumi continues on its merry way. If the program
fails, the RunResponse will include information about the failure.
Due to (5), all properties on resources are now instances of a new
Property<T> type. A Property<T> is just a thin wrapper over a T, but it
encodes the special properties of Lumi resource properties. Namely, it
is possible to create one out of a T, other Property<T>, Promise<T>, or
to freshly allocate one. In all cases, the Property<T> does not "settle"
until its final state is known. This cannot occur before the deployment
actually completes, and so in general it's not safe to depend on concrete
resolutions of values (unlike ordinary Promise<T>s which are usually
expected to resolve). As a result, all derived computations are meant to
use the `then` function (as in `someValue.then(v => v+x)`).
Although this change includes tests that may be run in isolation to test
the various RPC interactions, we are nowhere near finished. The remaining
work primarily boils down to three things:
1) Wiring all of this up to the Lumi code.
2) Fixing the handful of known loose ends required to make this work,
primarily around the serialization of properties (waiting on
unresolved ones, serializing assets properly, etc).
3) Implementing lambda closure serialization as a native extension.
This ongoing work is part of pulumi/pulumi-fabric#311.
2017-08-26 21:07:54 +02:00
|
|
|
// This is the entrypoint for running a Node.js program with minimal scaffolding.
|
2017-09-04 17:30:39 +02:00
|
|
|
import * as minimist from "minimist";
|
Implement initial Lumi-as-a-library
This is the initial step towards redefining Lumi as a library that runs
atop vanilla Node.js/V8, rather than as its own runtime.
This change is woefully incomplete but this includes some of the more
stable pieces of my current work-in-progress.
The new structure is that within the sdk/ directory we will have a client
library per language. This client library contains the object model for
Lumi (resources, properties, assets, config, etc), in addition to the
"language runtime host" components required to interoperate with the
Lumi resource monitor. This resource monitor is effectively what we call
"Lumi" today, in that it's the thing orchestrating plans and deployments.
Inside the sdk/ directory, you will find nodejs/, the Node.js client
library, alongside proto/, the definitions for RPC interop between the
different pieces of the system. This includes existing RPC definitions
for resource providers, etc., in addition to the new ones for hosting
different language runtimes from within Lumi.
These new interfaces are surprisingly simple. There is effectively a
bidirectional RPC channel between the Lumi resource monitor, represented
by the lumirpc.ResourceMonitor interface, and each language runtime,
represented by the lumirpc.LanguageRuntime interface.
The overall orchestration goes as follows:
1) Lumi decides it needs to run a program written in language X, so
it dynamically loads the language runtime plugin for language X.
2) Lumi passes that runtime a loopback address to its ResourceMonitor
service, while language X will publish a connection back to its
LanguageRuntime service, which Lumi will talk to.
3) Lumi then invokes LanguageRuntime.Run, passing information like
the desired working directory, program name, arguments, and optional
configuration variables to make available to the program.
4) The language X runtime receives this, unpacks it and sets up the
necessary context, and then invokes the program. The program then
calls into Lumi object model abstractions that internally communicate
back to Lumi using the ResourceMonitor interface.
5) The key here is ResourceMonitor.NewResource, which Lumi uses to
serialize state about newly allocated resources. Lumi receives these
and registers them as part of the plan, doing the usual diffing, etc.,
to decide how to proceed. This interface is perhaps one of the
most subtle parts of the new design, as it necessitates the use of
promises internally to allow parallel evaluation of the resource plan,
letting dataflow determine the available concurrency.
6) The program exits, and Lumi continues on its merry way. If the program
fails, the RunResponse will include information about the failure.
Due to (5), all properties on resources are now instances of a new
Property<T> type. A Property<T> is just a thin wrapper over a T, but it
encodes the special properties of Lumi resource properties. Namely, it
is possible to create one out of a T, other Property<T>, Promise<T>, or
to freshly allocate one. In all cases, the Property<T> does not "settle"
until its final state is known. This cannot occur before the deployment
actually completes, and so in general it's not safe to depend on concrete
resolutions of values (unlike ordinary Promise<T>s which are usually
expected to resolve). As a result, all derived computations are meant to
use the `then` function (as in `someValue.then(v => v+x)`).
Although this change includes tests that may be run in isolation to test
the various RPC interactions, we are nowhere near finished. The remaining
work primarily boils down to three things:
1) Wiring all of this up to the Lumi code.
2) Fixing the handful of known loose ends required to make this work,
primarily around the serialization of properties (waiting on
unresolved ones, serializing assets properly, etc).
3) Implementing lambda closure serialization as a native extension.
This ongoing work is part of pulumi/pulumi-fabric#311.
2017-08-26 21:07:54 +02:00
|
|
|
|
2017-09-04 17:30:39 +02:00
|
|
|
function usage(): void {
|
|
|
|
console.error(`usage: RUN <flags> [program] <[arg]...>`);
|
|
|
|
console.error(``);
|
|
|
|
console.error(` where [flags] may include`);
|
2017-10-19 17:26:57 +02:00
|
|
|
console.error(` --project=p set the project name to p`);
|
|
|
|
console.error(` --stack=s set the stack name to s`);
|
2017-09-04 17:30:39 +02:00
|
|
|
console.error(` --config.k=v... set runtime config key k to value v`);
|
2017-09-17 17:10:46 +02:00
|
|
|
console.error(` --parallel=p run up to p resource operations in parallel (default is serial)`);
|
2017-09-04 17:30:39 +02:00
|
|
|
console.error(` --dry-run true to simulate resource changes, but without making them`);
|
|
|
|
console.error(` --pwd=pwd change the working directory before running the program`);
|
2018-01-26 00:26:39 +01:00
|
|
|
console.error(` --monitor=addr [required] the RPC address for a resource monitor to connect to`);
|
2017-09-04 17:30:39 +02:00
|
|
|
console.error(` --engine=addr the RPC address for a resource engine to connect to`);
|
2017-11-09 02:08:51 +01:00
|
|
|
console.error(` --tracing=url a Zipkin-compatible endpoint to send tracing data to`);
|
2017-09-04 17:30:39 +02:00
|
|
|
console.error(``);
|
|
|
|
console.error(` and [program] is a JavaScript program to run in Node.js, and [arg]... optional args to it.`);
|
|
|
|
}
|
|
|
|
|
2018-01-26 00:26:39 +01:00
|
|
|
function printErrorUsageAndExit(message: string): never {
|
|
|
|
console.error(message);
|
|
|
|
usage();
|
|
|
|
return process.exit(-1);
|
|
|
|
}
|
|
|
|
|
Do not lazy initialize config or settings
The pulumi runtime used to lazily load and parse both config and
settings data set by the language host. The initial reason for this
design was that we wanted the runtime to be usable in a normal node
environment, but we have moved away from supporting that.
In addition, while we claimed we loaded these value "lazily", we
actually forced their loading quite eagerly when we started
up. However, when capturing config (or settings, as we now do), we
would capture all the logic about loading these values from the
environment.
Even worse, in the case where you had two copies of @pulumi/pulumi
loaded, it would be possible to capture a config object which was not
initialized and then at runtime the initialization logic would try to
read PULUMI_CONFIG from the process environment and fail.
So we adopt a new model where configuration and settings are parsed as
we load their containing modules. In addition, to support SxS
scinerios, we continue to use `process.env` as a way to control both
configuration and settings. This means that `run.ts` must now ensure
that these values are present in the environment before either the
config or runtime modules have been loaded.
2018-08-04 00:33:15 +02:00
|
|
|
function main(args: string[]): void {
|
2017-09-04 17:30:39 +02:00
|
|
|
// See usage above for the intended usage of this program, including flags and required args.
|
2017-10-10 23:50:55 +02:00
|
|
|
const argv: minimist.ParsedArgs = minimist(args, {
|
2017-09-17 17:10:46 +02:00
|
|
|
boolean: [ "dry-run" ],
|
2017-11-09 02:08:51 +01:00
|
|
|
string: [ "project", "stack", "parallel", "pwd", "monitor", "engine", "tracing" ],
|
2017-09-04 17:30:39 +02:00
|
|
|
unknown: (arg: string) => {
|
|
|
|
return true;
|
|
|
|
},
|
|
|
|
stopEarly: true,
|
|
|
|
});
|
Implement initial Lumi-as-a-library
This is the initial step towards redefining Lumi as a library that runs
atop vanilla Node.js/V8, rather than as its own runtime.
This change is woefully incomplete but this includes some of the more
stable pieces of my current work-in-progress.
The new structure is that within the sdk/ directory we will have a client
library per language. This client library contains the object model for
Lumi (resources, properties, assets, config, etc), in addition to the
"language runtime host" components required to interoperate with the
Lumi resource monitor. This resource monitor is effectively what we call
"Lumi" today, in that it's the thing orchestrating plans and deployments.
Inside the sdk/ directory, you will find nodejs/, the Node.js client
library, alongside proto/, the definitions for RPC interop between the
different pieces of the system. This includes existing RPC definitions
for resource providers, etc., in addition to the new ones for hosting
different language runtimes from within Lumi.
These new interfaces are surprisingly simple. There is effectively a
bidirectional RPC channel between the Lumi resource monitor, represented
by the lumirpc.ResourceMonitor interface, and each language runtime,
represented by the lumirpc.LanguageRuntime interface.
The overall orchestration goes as follows:
1) Lumi decides it needs to run a program written in language X, so
it dynamically loads the language runtime plugin for language X.
2) Lumi passes that runtime a loopback address to its ResourceMonitor
service, while language X will publish a connection back to its
LanguageRuntime service, which Lumi will talk to.
3) Lumi then invokes LanguageRuntime.Run, passing information like
the desired working directory, program name, arguments, and optional
configuration variables to make available to the program.
4) The language X runtime receives this, unpacks it and sets up the
necessary context, and then invokes the program. The program then
calls into Lumi object model abstractions that internally communicate
back to Lumi using the ResourceMonitor interface.
5) The key here is ResourceMonitor.NewResource, which Lumi uses to
serialize state about newly allocated resources. Lumi receives these
and registers them as part of the plan, doing the usual diffing, etc.,
to decide how to proceed. This interface is perhaps one of the
most subtle parts of the new design, as it necessitates the use of
promises internally to allow parallel evaluation of the resource plan,
letting dataflow determine the available concurrency.
6) The program exits, and Lumi continues on its merry way. If the program
fails, the RunResponse will include information about the failure.
Due to (5), all properties on resources are now instances of a new
Property<T> type. A Property<T> is just a thin wrapper over a T, but it
encodes the special properties of Lumi resource properties. Namely, it
is possible to create one out of a T, other Property<T>, Promise<T>, or
to freshly allocate one. In all cases, the Property<T> does not "settle"
until its final state is known. This cannot occur before the deployment
actually completes, and so in general it's not safe to depend on concrete
resolutions of values (unlike ordinary Promise<T>s which are usually
expected to resolve). As a result, all derived computations are meant to
use the `then` function (as in `someValue.then(v => v+x)`).
Although this change includes tests that may be run in isolation to test
the various RPC interactions, we are nowhere near finished. The remaining
work primarily boils down to three things:
1) Wiring all of this up to the Lumi code.
2) Fixing the handful of known loose ends required to make this work,
primarily around the serialization of properties (waiting on
unresolved ones, serializing assets properly, etc).
3) Implementing lambda closure serialization as a native extension.
This ongoing work is part of pulumi/pulumi-fabric#311.
2017-08-26 21:07:54 +02:00
|
|
|
|
Do not lazy initialize config or settings
The pulumi runtime used to lazily load and parse both config and
settings data set by the language host. The initial reason for this
design was that we wanted the runtime to be usable in a normal node
environment, but we have moved away from supporting that.
In addition, while we claimed we loaded these value "lazily", we
actually forced their loading quite eagerly when we started
up. However, when capturing config (or settings, as we now do), we
would capture all the logic about loading these values from the
environment.
Even worse, in the case where you had two copies of @pulumi/pulumi
loaded, it would be possible to capture a config object which was not
initialized and then at runtime the initialization logic would try to
read PULUMI_CONFIG from the process environment and fail.
So we adopt a new model where configuration and settings are parsed as
we load their containing modules. In addition, to support SxS
scinerios, we continue to use `process.env` as a way to control both
configuration and settings. This means that `run.ts` must now ensure
that these values are present in the environment before either the
config or runtime modules have been loaded.
2018-08-04 00:33:15 +02:00
|
|
|
// If parallel was passed, validate it is an number
|
2017-09-17 17:10:46 +02:00
|
|
|
if (argv["parallel"]) {
|
Do not lazy initialize config or settings
The pulumi runtime used to lazily load and parse both config and
settings data set by the language host. The initial reason for this
design was that we wanted the runtime to be usable in a normal node
environment, but we have moved away from supporting that.
In addition, while we claimed we loaded these value "lazily", we
actually forced their loading quite eagerly when we started
up. However, when capturing config (or settings, as we now do), we
would capture all the logic about loading these values from the
environment.
Even worse, in the case where you had two copies of @pulumi/pulumi
loaded, it would be possible to capture a config object which was not
initialized and then at runtime the initialization logic would try to
read PULUMI_CONFIG from the process environment and fail.
So we adopt a new model where configuration and settings are parsed as
we load their containing modules. In addition, to support SxS
scinerios, we continue to use `process.env` as a way to control both
configuration and settings. This means that `run.ts` must now ensure
that these values are present in the environment before either the
config or runtime modules have been loaded.
2018-08-04 00:33:15 +02:00
|
|
|
if (isNaN(parseInt(argv["parallel"], 10))) {
|
2018-01-26 00:26:39 +01:00
|
|
|
return printErrorUsageAndExit(
|
|
|
|
`error: --parallel flag must specify a number: ${argv["parallel"]} is not a number`);
|
2017-09-17 17:10:46 +02:00
|
|
|
}
|
|
|
|
}
|
2017-09-16 01:38:52 +02:00
|
|
|
|
Do not lazy initialize config or settings
The pulumi runtime used to lazily load and parse both config and
settings data set by the language host. The initial reason for this
design was that we wanted the runtime to be usable in a normal node
environment, but we have moved away from supporting that.
In addition, while we claimed we loaded these value "lazily", we
actually forced their loading quite eagerly when we started
up. However, when capturing config (or settings, as we now do), we
would capture all the logic about loading these values from the
environment.
Even worse, in the case where you had two copies of @pulumi/pulumi
loaded, it would be possible to capture a config object which was not
initialized and then at runtime the initialization logic would try to
read PULUMI_CONFIG from the process environment and fail.
So we adopt a new model where configuration and settings are parsed as
we load their containing modules. In addition, to support SxS
scinerios, we continue to use `process.env` as a way to control both
configuration and settings. This means that `run.ts` must now ensure
that these values are present in the environment before either the
config or runtime modules have been loaded.
2018-08-04 00:33:15 +02:00
|
|
|
// Ensure a monitor address was passed
|
2018-01-26 00:26:39 +01:00
|
|
|
const monitorAddr = argv["monitor"];
|
|
|
|
if (!monitorAddr) {
|
|
|
|
return printErrorUsageAndExit(`error: --monitor=addr must be provided.`);
|
2017-09-04 17:30:39 +02:00
|
|
|
}
|
|
|
|
|
Do not lazy initialize config or settings
The pulumi runtime used to lazily load and parse both config and
settings data set by the language host. The initial reason for this
design was that we wanted the runtime to be usable in a normal node
environment, but we have moved away from supporting that.
In addition, while we claimed we loaded these value "lazily", we
actually forced their loading quite eagerly when we started
up. However, when capturing config (or settings, as we now do), we
would capture all the logic about loading these values from the
environment.
Even worse, in the case where you had two copies of @pulumi/pulumi
loaded, it would be possible to capture a config object which was not
initialized and then at runtime the initialization logic would try to
read PULUMI_CONFIG from the process environment and fail.
So we adopt a new model where configuration and settings are parsed as
we load their containing modules. In addition, to support SxS
scinerios, we continue to use `process.env` as a way to control both
configuration and settings. This means that `run.ts` must now ensure
that these values are present in the environment before either the
config or runtime modules have been loaded.
2018-08-04 00:33:15 +02:00
|
|
|
// Finally, ensure we have a program to run.
|
2017-09-04 17:30:39 +02:00
|
|
|
if (argv._.length === 0) {
|
2018-01-26 00:26:39 +01:00
|
|
|
return printErrorUsageAndExit("error: Missing program to execute");
|
Implement initial Lumi-as-a-library
This is the initial step towards redefining Lumi as a library that runs
atop vanilla Node.js/V8, rather than as its own runtime.
This change is woefully incomplete but this includes some of the more
stable pieces of my current work-in-progress.
The new structure is that within the sdk/ directory we will have a client
library per language. This client library contains the object model for
Lumi (resources, properties, assets, config, etc), in addition to the
"language runtime host" components required to interoperate with the
Lumi resource monitor. This resource monitor is effectively what we call
"Lumi" today, in that it's the thing orchestrating plans and deployments.
Inside the sdk/ directory, you will find nodejs/, the Node.js client
library, alongside proto/, the definitions for RPC interop between the
different pieces of the system. This includes existing RPC definitions
for resource providers, etc., in addition to the new ones for hosting
different language runtimes from within Lumi.
These new interfaces are surprisingly simple. There is effectively a
bidirectional RPC channel between the Lumi resource monitor, represented
by the lumirpc.ResourceMonitor interface, and each language runtime,
represented by the lumirpc.LanguageRuntime interface.
The overall orchestration goes as follows:
1) Lumi decides it needs to run a program written in language X, so
it dynamically loads the language runtime plugin for language X.
2) Lumi passes that runtime a loopback address to its ResourceMonitor
service, while language X will publish a connection back to its
LanguageRuntime service, which Lumi will talk to.
3) Lumi then invokes LanguageRuntime.Run, passing information like
the desired working directory, program name, arguments, and optional
configuration variables to make available to the program.
4) The language X runtime receives this, unpacks it and sets up the
necessary context, and then invokes the program. The program then
calls into Lumi object model abstractions that internally communicate
back to Lumi using the ResourceMonitor interface.
5) The key here is ResourceMonitor.NewResource, which Lumi uses to
serialize state about newly allocated resources. Lumi receives these
and registers them as part of the plan, doing the usual diffing, etc.,
to decide how to proceed. This interface is perhaps one of the
most subtle parts of the new design, as it necessitates the use of
promises internally to allow parallel evaluation of the resource plan,
letting dataflow determine the available concurrency.
6) The program exits, and Lumi continues on its merry way. If the program
fails, the RunResponse will include information about the failure.
Due to (5), all properties on resources are now instances of a new
Property<T> type. A Property<T> is just a thin wrapper over a T, but it
encodes the special properties of Lumi resource properties. Namely, it
is possible to create one out of a T, other Property<T>, Promise<T>, or
to freshly allocate one. In all cases, the Property<T> does not "settle"
until its final state is known. This cannot occur before the deployment
actually completes, and so in general it's not safe to depend on concrete
resolutions of values (unlike ordinary Promise<T>s which are usually
expected to resolve). As a result, all derived computations are meant to
use the `then` function (as in `someValue.then(v => v+x)`).
Although this change includes tests that may be run in isolation to test
the various RPC interactions, we are nowhere near finished. The remaining
work primarily boils down to three things:
1) Wiring all of this up to the Lumi code.
2) Fixing the handful of known loose ends required to make this work,
primarily around the serialization of properties (waiting on
unresolved ones, serializing assets properly, etc).
3) Implementing lambda closure serialization as a native extension.
This ongoing work is part of pulumi/pulumi-fabric#311.
2017-08-26 21:07:54 +02:00
|
|
|
}
|
|
|
|
|
Do not lazy initialize config or settings
The pulumi runtime used to lazily load and parse both config and
settings data set by the language host. The initial reason for this
design was that we wanted the runtime to be usable in a normal node
environment, but we have moved away from supporting that.
In addition, while we claimed we loaded these value "lazily", we
actually forced their loading quite eagerly when we started
up. However, when capturing config (or settings, as we now do), we
would capture all the logic about loading these values from the
environment.
Even worse, in the case where you had two copies of @pulumi/pulumi
loaded, it would be possible to capture a config object which was not
initialized and then at runtime the initialization logic would try to
read PULUMI_CONFIG from the process environment and fail.
So we adopt a new model where configuration and settings are parsed as
we load their containing modules. In addition, to support SxS
scinerios, we continue to use `process.env` as a way to control both
configuration and settings. This means that `run.ts` must now ensure
that these values are present in the environment before either the
config or runtime modules have been loaded.
2018-08-04 00:33:15 +02:00
|
|
|
// Due to node module loading semantics, multiple copies of @pulumi/pulumi could be loaded at runtime. So we need
|
|
|
|
// to squirel these settings in the environment such that other copies which may be loaded later can recover them.
|
|
|
|
//
|
|
|
|
// Config is already an environment variaible set by the language plugin.
|
|
|
|
addToEnvIfDefined("PULUMI_NODEJS_PROJECT", argv["project"]);
|
|
|
|
addToEnvIfDefined("PULUMI_NODEJS_STACK", argv["stack"]);
|
|
|
|
addToEnvIfDefined("PULUMI_NODEJS_DRY_RUN", argv["dry-run"]);
|
|
|
|
addToEnvIfDefined("PULUMI_NODEJS_PARALLEL", argv["parallel"]);
|
|
|
|
addToEnvIfDefined("PULUMI_NODEJS_MONITOR", argv["monitor"]);
|
2018-08-25 01:50:09 +02:00
|
|
|
addToEnvIfDefined("PULUMI_NODEJS_ENGINE", argv["engine"]);
|
Do not lazy initialize config or settings
The pulumi runtime used to lazily load and parse both config and
settings data set by the language host. The initial reason for this
design was that we wanted the runtime to be usable in a normal node
environment, but we have moved away from supporting that.
In addition, while we claimed we loaded these value "lazily", we
actually forced their loading quite eagerly when we started
up. However, when capturing config (or settings, as we now do), we
would capture all the logic about loading these values from the
environment.
Even worse, in the case where you had two copies of @pulumi/pulumi
loaded, it would be possible to capture a config object which was not
initialized and then at runtime the initialization logic would try to
read PULUMI_CONFIG from the process environment and fail.
So we adopt a new model where configuration and settings are parsed as
we load their containing modules. In addition, to support SxS
scinerios, we continue to use `process.env` as a way to control both
configuration and settings. This means that `run.ts` must now ensure
that these values are present in the environment before either the
config or runtime modules have been loaded.
2018-08-04 00:33:15 +02:00
|
|
|
|
2018-11-01 23:46:21 +01:00
|
|
|
// Ensure that our v8 hooks have been initialized. Then actually load and run the user program.
|
|
|
|
v8Hooks.isInitializedAsync().then(() => {
|
2019-03-27 22:40:29 +01:00
|
|
|
const promise: Promise<void> = require("./run").run(
|
|
|
|
argv,
|
|
|
|
/*programStarted: */ () => programRunning = true,
|
|
|
|
/*reportLoggedError:*/ (err: Error) => loggedErrors.add(err));
|
2019-03-12 23:56:43 +01:00
|
|
|
|
|
|
|
// when the user's program completes successfully, set programRunning back to false. That way, if the Pulumi
|
|
|
|
// scaffolding code ends up throwing an exception during teardown, it will get printed directly to the console.
|
|
|
|
//
|
|
|
|
// Note: we only do this in the 'resolved' arg of '.then' (not the 'rejected' arg). If the users code throws
|
|
|
|
// an exception, this promise will get rejected, and we don't want touch or otherwise intercept the exception
|
|
|
|
// or change the programRunning state here at all.
|
|
|
|
promise.then(() => { programRunning = false; });
|
2018-11-01 23:46:21 +01:00
|
|
|
});
|
Do not lazy initialize config or settings
The pulumi runtime used to lazily load and parse both config and
settings data set by the language host. The initial reason for this
design was that we wanted the runtime to be usable in a normal node
environment, but we have moved away from supporting that.
In addition, while we claimed we loaded these value "lazily", we
actually forced their loading quite eagerly when we started
up. However, when capturing config (or settings, as we now do), we
would capture all the logic about loading these values from the
environment.
Even worse, in the case where you had two copies of @pulumi/pulumi
loaded, it would be possible to capture a config object which was not
initialized and then at runtime the initialization logic would try to
read PULUMI_CONFIG from the process environment and fail.
So we adopt a new model where configuration and settings are parsed as
we load their containing modules. In addition, to support SxS
scinerios, we continue to use `process.env` as a way to control both
configuration and settings. This means that `run.ts` must now ensure
that these values are present in the environment before either the
config or runtime modules have been loaded.
2018-08-04 00:33:15 +02:00
|
|
|
}
|
2018-02-26 19:54:56 +01:00
|
|
|
|
Do not lazy initialize config or settings
The pulumi runtime used to lazily load and parse both config and
settings data set by the language host. The initial reason for this
design was that we wanted the runtime to be usable in a normal node
environment, but we have moved away from supporting that.
In addition, while we claimed we loaded these value "lazily", we
actually forced their loading quite eagerly when we started
up. However, when capturing config (or settings, as we now do), we
would capture all the logic about loading these values from the
environment.
Even worse, in the case where you had two copies of @pulumi/pulumi
loaded, it would be possible to capture a config object which was not
initialized and then at runtime the initialization logic would try to
read PULUMI_CONFIG from the process environment and fail.
So we adopt a new model where configuration and settings are parsed as
we load their containing modules. In addition, to support SxS
scinerios, we continue to use `process.env` as a way to control both
configuration and settings. This means that `run.ts` must now ensure
that these values are present in the environment before either the
config or runtime modules have been loaded.
2018-08-04 00:33:15 +02:00
|
|
|
function addToEnvIfDefined(key: string, value: string | undefined) {
|
|
|
|
if (value) {
|
|
|
|
process.env[key] = value;
|
|
|
|
}
|
Implement initial Lumi-as-a-library
This is the initial step towards redefining Lumi as a library that runs
atop vanilla Node.js/V8, rather than as its own runtime.
This change is woefully incomplete but this includes some of the more
stable pieces of my current work-in-progress.
The new structure is that within the sdk/ directory we will have a client
library per language. This client library contains the object model for
Lumi (resources, properties, assets, config, etc), in addition to the
"language runtime host" components required to interoperate with the
Lumi resource monitor. This resource monitor is effectively what we call
"Lumi" today, in that it's the thing orchestrating plans and deployments.
Inside the sdk/ directory, you will find nodejs/, the Node.js client
library, alongside proto/, the definitions for RPC interop between the
different pieces of the system. This includes existing RPC definitions
for resource providers, etc., in addition to the new ones for hosting
different language runtimes from within Lumi.
These new interfaces are surprisingly simple. There is effectively a
bidirectional RPC channel between the Lumi resource monitor, represented
by the lumirpc.ResourceMonitor interface, and each language runtime,
represented by the lumirpc.LanguageRuntime interface.
The overall orchestration goes as follows:
1) Lumi decides it needs to run a program written in language X, so
it dynamically loads the language runtime plugin for language X.
2) Lumi passes that runtime a loopback address to its ResourceMonitor
service, while language X will publish a connection back to its
LanguageRuntime service, which Lumi will talk to.
3) Lumi then invokes LanguageRuntime.Run, passing information like
the desired working directory, program name, arguments, and optional
configuration variables to make available to the program.
4) The language X runtime receives this, unpacks it and sets up the
necessary context, and then invokes the program. The program then
calls into Lumi object model abstractions that internally communicate
back to Lumi using the ResourceMonitor interface.
5) The key here is ResourceMonitor.NewResource, which Lumi uses to
serialize state about newly allocated resources. Lumi receives these
and registers them as part of the plan, doing the usual diffing, etc.,
to decide how to proceed. This interface is perhaps one of the
most subtle parts of the new design, as it necessitates the use of
promises internally to allow parallel evaluation of the resource plan,
letting dataflow determine the available concurrency.
6) The program exits, and Lumi continues on its merry way. If the program
fails, the RunResponse will include information about the failure.
Due to (5), all properties on resources are now instances of a new
Property<T> type. A Property<T> is just a thin wrapper over a T, but it
encodes the special properties of Lumi resource properties. Namely, it
is possible to create one out of a T, other Property<T>, Promise<T>, or
to freshly allocate one. In all cases, the Property<T> does not "settle"
until its final state is known. This cannot occur before the deployment
actually completes, and so in general it's not safe to depend on concrete
resolutions of values (unlike ordinary Promise<T>s which are usually
expected to resolve). As a result, all derived computations are meant to
use the `then` function (as in `someValue.then(v => v+x)`).
Although this change includes tests that may be run in isolation to test
the various RPC interactions, we are nowhere near finished. The remaining
work primarily boils down to three things:
1) Wiring all of this up to the Lumi code.
2) Fixing the handful of known loose ends required to make this work,
primarily around the serialization of properties (waiting on
unresolved ones, serializing assets properly, etc).
3) Implementing lambda closure serialization as a native extension.
This ongoing work is part of pulumi/pulumi-fabric#311.
2017-08-26 21:07:54 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
main(process.argv.slice(2));
|