2018-05-22 21:43:36 +02:00
|
|
|
// Copyright 2016-2018, Pulumi Corporation.
|
|
|
|
//
|
|
|
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
|
|
|
// you may not use this file except in compliance with the License.
|
|
|
|
// You may obtain a copy of the License at
|
|
|
|
//
|
|
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
//
|
|
|
|
// Unless required by applicable law or agreed to in writing, software
|
|
|
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
|
|
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
|
|
// See the License for the specific language governing permissions and
|
|
|
|
// limitations under the License.
|
Implement initial Lumi-as-a-library
This is the initial step towards redefining Lumi as a library that runs
atop vanilla Node.js/V8, rather than as its own runtime.
This change is woefully incomplete but this includes some of the more
stable pieces of my current work-in-progress.
The new structure is that within the sdk/ directory we will have a client
library per language. This client library contains the object model for
Lumi (resources, properties, assets, config, etc), in addition to the
"language runtime host" components required to interoperate with the
Lumi resource monitor. This resource monitor is effectively what we call
"Lumi" today, in that it's the thing orchestrating plans and deployments.
Inside the sdk/ directory, you will find nodejs/, the Node.js client
library, alongside proto/, the definitions for RPC interop between the
different pieces of the system. This includes existing RPC definitions
for resource providers, etc., in addition to the new ones for hosting
different language runtimes from within Lumi.
These new interfaces are surprisingly simple. There is effectively a
bidirectional RPC channel between the Lumi resource monitor, represented
by the lumirpc.ResourceMonitor interface, and each language runtime,
represented by the lumirpc.LanguageRuntime interface.
The overall orchestration goes as follows:
1) Lumi decides it needs to run a program written in language X, so
it dynamically loads the language runtime plugin for language X.
2) Lumi passes that runtime a loopback address to its ResourceMonitor
service, while language X will publish a connection back to its
LanguageRuntime service, which Lumi will talk to.
3) Lumi then invokes LanguageRuntime.Run, passing information like
the desired working directory, program name, arguments, and optional
configuration variables to make available to the program.
4) The language X runtime receives this, unpacks it and sets up the
necessary context, and then invokes the program. The program then
calls into Lumi object model abstractions that internally communicate
back to Lumi using the ResourceMonitor interface.
5) The key here is ResourceMonitor.NewResource, which Lumi uses to
serialize state about newly allocated resources. Lumi receives these
and registers them as part of the plan, doing the usual diffing, etc.,
to decide how to proceed. This interface is perhaps one of the
most subtle parts of the new design, as it necessitates the use of
promises internally to allow parallel evaluation of the resource plan,
letting dataflow determine the available concurrency.
6) The program exits, and Lumi continues on its merry way. If the program
fails, the RunResponse will include information about the failure.
Due to (5), all properties on resources are now instances of a new
Property<T> type. A Property<T> is just a thin wrapper over a T, but it
encodes the special properties of Lumi resource properties. Namely, it
is possible to create one out of a T, other Property<T>, Promise<T>, or
to freshly allocate one. In all cases, the Property<T> does not "settle"
until its final state is known. This cannot occur before the deployment
actually completes, and so in general it's not safe to depend on concrete
resolutions of values (unlike ordinary Promise<T>s which are usually
expected to resolve). As a result, all derived computations are meant to
use the `then` function (as in `someValue.then(v => v+x)`).
Although this change includes tests that may be run in isolation to test
the various RPC interactions, we are nowhere near finished. The remaining
work primarily boils down to three things:
1) Wiring all of this up to the Lumi code.
2) Fixing the handful of known loose ends required to make this work,
primarily around the serialization of properties (waiting on
unresolved ones, serializing assets properly, etc).
3) Implementing lambda closure serialization as a native extension.
This ongoing work is part of pulumi/pulumi-fabric#311.
2017-08-26 21:07:54 +02:00
|
|
|
|
2020-04-14 10:30:25 +02:00
|
|
|
import * as grpc from "@grpc/grpc-js";
|
2019-04-19 00:25:15 +02:00
|
|
|
import * as query from "@pulumi/query";
|
2017-10-08 21:10:46 +02:00
|
|
|
import * as log from "../log";
|
2019-08-08 21:11:46 +02:00
|
|
|
import * as utils from "../utils";
|
|
|
|
|
2019-12-18 04:04:09 +01:00
|
|
|
import { getAllResources, Input, Inputs, Output, output } from "../output";
|
2019-04-19 00:25:15 +02:00
|
|
|
import { ResolvedResource } from "../queryable";
|
2019-04-17 07:20:01 +02:00
|
|
|
import {
|
|
|
|
ComponentResource,
|
2019-06-01 08:01:01 +02:00
|
|
|
createUrn,
|
2019-04-17 07:20:01 +02:00
|
|
|
CustomResource,
|
|
|
|
CustomResourceOptions,
|
|
|
|
ID,
|
2019-10-15 07:08:06 +02:00
|
|
|
ProviderResource,
|
2019-04-17 07:20:01 +02:00
|
|
|
Resource,
|
|
|
|
ResourceOptions,
|
|
|
|
URN,
|
|
|
|
} from "../resource";
|
2019-02-01 03:08:17 +01:00
|
|
|
import { debuggablePromise } from "./debuggable";
|
2019-04-19 00:25:15 +02:00
|
|
|
import { invoke } from "./invoke";
|
2019-02-22 05:18:29 +01:00
|
|
|
|
2018-04-05 18:48:09 +02:00
|
|
|
import {
|
|
|
|
deserializeProperties,
|
2018-05-23 23:47:40 +02:00
|
|
|
deserializeProperty,
|
2018-04-05 18:48:09 +02:00
|
|
|
OutputResolvers,
|
|
|
|
resolveProperties,
|
|
|
|
serializeProperties,
|
|
|
|
serializeProperty,
|
2018-05-02 00:05:42 +02:00
|
|
|
serializeResourceProperties,
|
2020-09-24 04:06:26 +02:00
|
|
|
suppressUnhandledGrpcRejections,
|
2018-04-05 18:48:09 +02:00
|
|
|
transferProperties,
|
|
|
|
} from "./rpc";
|
2019-04-17 07:20:01 +02:00
|
|
|
import {
|
|
|
|
excessiveDebugOutput,
|
|
|
|
getMonitor,
|
|
|
|
getProject,
|
|
|
|
getRootResource,
|
|
|
|
getStack,
|
2019-08-05 21:44:04 +02:00
|
|
|
isDryRun,
|
|
|
|
isLegacyApplyEnabled,
|
2019-04-17 07:20:01 +02:00
|
|
|
rpcKeepAlive,
|
|
|
|
serialize,
|
2020-09-24 04:06:26 +02:00
|
|
|
terminateRpcs,
|
2019-04-17 07:20:01 +02:00
|
|
|
} from "./settings";
|
Implement initial Lumi-as-a-library
This is the initial step towards redefining Lumi as a library that runs
atop vanilla Node.js/V8, rather than as its own runtime.
This change is woefully incomplete but this includes some of the more
stable pieces of my current work-in-progress.
The new structure is that within the sdk/ directory we will have a client
library per language. This client library contains the object model for
Lumi (resources, properties, assets, config, etc), in addition to the
"language runtime host" components required to interoperate with the
Lumi resource monitor. This resource monitor is effectively what we call
"Lumi" today, in that it's the thing orchestrating plans and deployments.
Inside the sdk/ directory, you will find nodejs/, the Node.js client
library, alongside proto/, the definitions for RPC interop between the
different pieces of the system. This includes existing RPC definitions
for resource providers, etc., in addition to the new ones for hosting
different language runtimes from within Lumi.
These new interfaces are surprisingly simple. There is effectively a
bidirectional RPC channel between the Lumi resource monitor, represented
by the lumirpc.ResourceMonitor interface, and each language runtime,
represented by the lumirpc.LanguageRuntime interface.
The overall orchestration goes as follows:
1) Lumi decides it needs to run a program written in language X, so
it dynamically loads the language runtime plugin for language X.
2) Lumi passes that runtime a loopback address to its ResourceMonitor
service, while language X will publish a connection back to its
LanguageRuntime service, which Lumi will talk to.
3) Lumi then invokes LanguageRuntime.Run, passing information like
the desired working directory, program name, arguments, and optional
configuration variables to make available to the program.
4) The language X runtime receives this, unpacks it and sets up the
necessary context, and then invokes the program. The program then
calls into Lumi object model abstractions that internally communicate
back to Lumi using the ResourceMonitor interface.
5) The key here is ResourceMonitor.NewResource, which Lumi uses to
serialize state about newly allocated resources. Lumi receives these
and registers them as part of the plan, doing the usual diffing, etc.,
to decide how to proceed. This interface is perhaps one of the
most subtle parts of the new design, as it necessitates the use of
promises internally to allow parallel evaluation of the resource plan,
letting dataflow determine the available concurrency.
6) The program exits, and Lumi continues on its merry way. If the program
fails, the RunResponse will include information about the failure.
Due to (5), all properties on resources are now instances of a new
Property<T> type. A Property<T> is just a thin wrapper over a T, but it
encodes the special properties of Lumi resource properties. Namely, it
is possible to create one out of a T, other Property<T>, Promise<T>, or
to freshly allocate one. In all cases, the Property<T> does not "settle"
until its final state is known. This cannot occur before the deployment
actually completes, and so in general it's not safe to depend on concrete
resolutions of values (unlike ordinary Promise<T>s which are usually
expected to resolve). As a result, all derived computations are meant to
use the `then` function (as in `someValue.then(v => v+x)`).
Although this change includes tests that may be run in isolation to test
the various RPC interactions, we are nowhere near finished. The remaining
work primarily boils down to three things:
1) Wiring all of this up to the Lumi code.
2) Fixing the handful of known loose ends required to make this work,
primarily around the serialization of properties (waiting on
unresolved ones, serializing assets properly, etc).
3) Implementing lambda closure serialization as a native extension.
This ongoing work is part of pulumi/pulumi-fabric#311.
2017-08-26 21:07:54 +02:00
|
|
|
|
2018-01-25 22:34:21 +01:00
|
|
|
const gstruct = require("google-protobuf/google/protobuf/struct_pb.js");
|
2020-12-01 19:58:15 +01:00
|
|
|
const providerproto = require("../proto/provider_pb.js");
|
2017-11-17 03:21:41 +01:00
|
|
|
const resproto = require("../proto/resource_pb.js");
|
2017-09-09 22:49:50 +02:00
|
|
|
|
2018-04-05 18:48:09 +02:00
|
|
|
interface ResourceResolverOperation {
|
|
|
|
// A resolver for a resource's URN.
|
2020-09-24 04:06:26 +02:00
|
|
|
resolveURN: (urn: URN, err?: Error) => void;
|
2018-04-05 18:48:09 +02:00
|
|
|
// A resolver for a resource's ID (for custom resources only).
|
2020-09-24 04:06:26 +02:00
|
|
|
resolveID: ((v: ID, performApply: boolean, err?: Error) => void) | undefined;
|
2018-04-05 18:48:09 +02:00
|
|
|
// A collection of resolvers for a resource's properties.
|
|
|
|
resolvers: OutputResolvers;
|
|
|
|
// A parent URN, fully resolved, if any.
|
|
|
|
parentURN: URN | undefined;
|
Implement first-class providers. (#1695)
### First-Class Providers
These changes implement support for first-class providers. First-class
providers are provider plugins that are exposed as resources via the
Pulumi programming model so that they may be explicitly and multiply
instantiated. Each instance of a provider resource may be configured
differently, and configuration parameters may be source from the
outputs of other resources.
### Provider Plugin Changes
In order to accommodate the need to verify and diff provider
configuration and configure providers without complete configuration
information, these changes adjust the high-level provider plugin
interface. Two new methods for validating a provider's configuration
and diffing changes to the same have been added (`CheckConfig` and
`DiffConfig`, respectively), and the type of the configuration bag
accepted by `Configure` has been changed to a `PropertyMap`.
These changes have not yet been reflected in the provider plugin gRPC
interface. We will do this in a set of follow-up changes. Until then,
these methods are implemented by adapters:
- `CheckConfig` validates that all configuration parameters are string
or unknown properties. This is necessary because existing plugins
only accept string-typed configuration values.
- `DiffConfig` either returns "never replace" if all configuration
values are known or "must replace" if any configuration value is
unknown. The justification for this behavior is given
[here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106)
- `Configure` converts the config bag to a legacy config map and
configures the provider plugin if all config values are known. If any
config value is unknown, the underlying plugin is not configured and
the provider may only perform `Check`, `Read`, and `Invoke`, all of
which return empty results. We justify this behavior becuase it is
only possible during a preview and provides the best experience we
can manage with the existing gRPC interface.
### Resource Model Changes
Providers are now exposed as resources that participate in a stack's
dependency graph. Like other resources, they are explicitly created,
may have multiple instances, and may have dependencies on other
resources. Providers are referred to using provider references, which
are a combination of the provider's URN and its ID. This design
addresses the need during a preview to refer to providers that have not
yet been physically created and therefore have no ID.
All custom resources that are not themselves providers must specify a
single provider via a provider reference. The named provider will be
used to manage that resource's CRUD operations. If a resource's
provider reference changes, the resource must be replaced. Though its
URN is not present in the resource's dependency list, the provider
should be treated as a dependency of the resource when topologically
sorting the dependency graph.
Finally, `Invoke` operations must now specify a provider to use for the
invocation via a provider reference.
### Engine Changes
First-class providers support requires a few changes to the engine:
- The engine must have some way to map from provider references to
provider plugins. It must be possible to add providers from a stack's
checkpoint to this map and to register new/updated providers during
the execution of a plan in response to CRUD operations on provider
resources.
- In order to support updating existing stacks using existing Pulumi
programs that may not explicitly instantiate providers, the engine
must be able to manage the "default" providers for each package
referenced by a checkpoint or Pulumi program. The configuration for
a "default" provider is taken from the stack's configuration data.
The former need is addressed by adding a provider registry type that is
responsible for managing all of the plugins required by a plan. In
addition to loading plugins froma checkpoint and providing the ability
to map from a provider reference to a provider plugin, this type serves
as the provider plugin for providers themselves (i.e. it is the
"provider provider").
The latter need is solved via two relatively self-contained changes to
plan setup and the eval source.
During plan setup, the old checkpoint is scanned for custom resources
that do not have a provider reference in order to compute the set of
packages that require a default provider. Once this set has been
computed, the required default provider definitions are conjured and
prepended to the checkpoint's resource list. Each resource that
requires a default provider is then updated to refer to the default
provider for its package.
While an eval source is running, each custom resource registration,
resource read, and invoke that does not name a provider is trapped
before being returned by the source iterator. If no default provider
for the appropriate package has been registered, the eval source
synthesizes an appropriate registration, waits for it to complete, and
records the registered provider's reference. This reference is injected
into the original request, which is then processed as usual. If a
default provider was already registered, the recorded reference is
used and no new registration occurs.
### SDK Changes
These changes only expose first-class providers from the Node.JS SDK.
- A new abstract class, `ProviderResource`, can be subclassed and used
to instantiate first-class providers.
- A new field in `ResourceOptions`, `provider`, can be used to supply
a particular provider instance to manage a `CustomResource`'s CRUD
operations.
- A new type, `InvokeOptions`, can be used to specify options that
control the behavior of a call to `pulumi.runtime.invoke`. This type
includes a `provider` field that is analogous to
`ResourceOptions.provider`.
2018-08-07 02:50:29 +02:00
|
|
|
// A provider reference, fully resolved, if any.
|
|
|
|
providerRef: string | undefined;
|
2018-04-05 18:48:09 +02:00
|
|
|
// All serialized properties, fully awaited, serialized, and ready to go.
|
|
|
|
serializedProps: Record<string, any>;
|
2019-03-06 02:06:57 +01:00
|
|
|
// A set of URNs that this resource is directly dependent upon. These will all be URNs of
|
|
|
|
// custom resources, not component resources.
|
2019-02-22 05:18:29 +01:00
|
|
|
allDirectDependencyURNs: Set<URN>;
|
2019-03-06 02:06:57 +01:00
|
|
|
// Set of URNs that this resource is directly dependent upon, keyed by the property that causes
|
|
|
|
// the dependency. All urns in this map must exist in [allDirectDependencyURNs]. These will
|
|
|
|
// all be URNs of custom resources, not component resources.
|
2019-02-22 05:18:29 +01:00
|
|
|
propertyToDirectDependencyURNs: Map<string, Set<URN>>;
|
2019-06-01 08:01:01 +02:00
|
|
|
// A list of aliases applied to this resource.
|
|
|
|
aliases: URN[];
|
2019-07-12 20:12:01 +02:00
|
|
|
// An ID to import, if any.
|
|
|
|
import: ID | undefined;
|
2019-04-17 07:20:01 +02:00
|
|
|
}
|
|
|
|
|
2020-12-01 19:58:15 +01:00
|
|
|
/**
|
|
|
|
* Get an existing resource's state from the engine.
|
|
|
|
*/
|
|
|
|
export function getResource(res: Resource, props: Inputs, custom: boolean, urn: string): void {
|
|
|
|
// Extract the resource type from the URN.
|
|
|
|
const urnParts = urn.split("::");
|
|
|
|
const qualifiedType = urnParts[2];
|
|
|
|
const urnName = urnParts[3];
|
|
|
|
const type = qualifiedType.split("$").pop()!;
|
|
|
|
|
|
|
|
const label = `resource:urn=${urn}`;
|
|
|
|
log.debug(`Getting resource: urn=${urn}`);
|
|
|
|
|
|
|
|
const monitor: any = getMonitor();
|
|
|
|
const resopAsync = prepareResource(label, res, custom, props, {});
|
|
|
|
|
|
|
|
const preallocError = new Error();
|
|
|
|
debuggablePromise(resopAsync.then(async (resop) => {
|
|
|
|
const inputs = await serializeProperties(label, { urn });
|
|
|
|
|
|
|
|
const req = new providerproto.InvokeRequest();
|
|
|
|
req.setTok("pulumi:pulumi:getResource");
|
|
|
|
req.setArgs(gstruct.Struct.fromJavaScript(inputs));
|
|
|
|
req.setProvider("");
|
|
|
|
req.setVersion("");
|
|
|
|
|
|
|
|
// Now run the operation, serializing the invocation if necessary.
|
|
|
|
const opLabel = `monitor.getResource(${label})`;
|
|
|
|
runAsyncResourceOp(opLabel, async () => {
|
|
|
|
let resp: any;
|
|
|
|
let err: Error | undefined;
|
|
|
|
try {
|
|
|
|
if (monitor) {
|
|
|
|
resp = await debuggablePromise(new Promise((resolve, reject) =>
|
|
|
|
monitor.invoke(req, (rpcError: grpc.ServiceError, innerResponse: any) => {
|
|
|
|
log.debug(`getResource Invoke RPC finished: err: ${rpcError}, resp: ${innerResponse}`);
|
|
|
|
if (rpcError) {
|
|
|
|
if (rpcError.code === grpc.status.UNAVAILABLE || rpcError.code === grpc.status.CANCELLED) {
|
|
|
|
err = rpcError;
|
|
|
|
terminateRpcs();
|
|
|
|
rpcError.message = "Resource monitor is terminating";
|
|
|
|
(<any>preallocError).code = rpcError.code;
|
|
|
|
}
|
|
|
|
|
|
|
|
preallocError.message = `failed to get resource:urn=${urn}: ${rpcError.message}`;
|
|
|
|
reject(new Error(rpcError.details));
|
|
|
|
}
|
|
|
|
else {
|
|
|
|
resolve(innerResponse);
|
|
|
|
}
|
|
|
|
})), opLabel);
|
|
|
|
|
|
|
|
// If the invoke failed, raise an error
|
|
|
|
const failures: any = resp.getFailuresList();
|
|
|
|
if (failures && failures.length) {
|
|
|
|
let reasons = "";
|
|
|
|
for (let i = 0; i < failures.length; i++) {
|
|
|
|
if (reasons !== "") {
|
|
|
|
reasons += "; ";
|
|
|
|
}
|
|
|
|
reasons += `${failures[i].getReason()} (${failures[i].getProperty()})`;
|
|
|
|
}
|
|
|
|
throw new Error(`getResource Invoke failed: ${reasons}`);
|
|
|
|
}
|
|
|
|
|
|
|
|
// Otherwise, return the response.
|
|
|
|
const m = resp.getReturn().getFieldsMap();
|
|
|
|
resp = {
|
|
|
|
urn: m.get("urn").toJavaScript(),
|
|
|
|
id: m.get("id").toJavaScript() || undefined,
|
|
|
|
state: m.get("state").getStructValue(),
|
|
|
|
};
|
|
|
|
}
|
|
|
|
} catch (e) {
|
|
|
|
err = e;
|
|
|
|
resp = {
|
|
|
|
urn: "",
|
|
|
|
id: undefined,
|
|
|
|
state: undefined,
|
|
|
|
};
|
|
|
|
}
|
|
|
|
|
|
|
|
resop.resolveURN(resp.urn, err);
|
|
|
|
|
|
|
|
// Note: 'id || undefined' is intentional. We intentionally collapse falsy values to
|
|
|
|
// undefined so that later parts of our system don't have to deal with values like 'null'.
|
|
|
|
if (resop.resolveID) {
|
|
|
|
const id = resp.id || undefined;
|
|
|
|
resop.resolveID(id, id !== undefined, err);
|
|
|
|
}
|
|
|
|
|
|
|
|
await resolveOutputs(res, type, urnName, props, resp.state, {}, resop.resolvers, err);
|
|
|
|
});
|
|
|
|
}), label);
|
|
|
|
}
|
|
|
|
|
2018-04-05 18:48:09 +02:00
|
|
|
/**
|
|
|
|
* Reads an existing custom resource's state from the resource monitor. Note that resources read in this way
|
|
|
|
* will not be part of the resulting stack's state, as they are presumed to belong to another.
|
|
|
|
*/
|
2018-04-07 19:15:58 +02:00
|
|
|
export function readResource(res: Resource, t: string, name: string, props: Inputs, opts: ResourceOptions): void {
|
|
|
|
const id: Input<ID> | undefined = opts.id;
|
|
|
|
if (!id) {
|
|
|
|
throw new Error("Cannot read resource whose options are lacking an ID value");
|
|
|
|
}
|
|
|
|
|
2018-04-05 18:48:09 +02:00
|
|
|
const label = `resource:${name}[${t}]#...`;
|
2019-02-28 01:00:54 +01:00
|
|
|
log.debug(`Reading resource: id=${Output.isInstance(id) ? "Output<T>" : id}, t=${t}, name=${name}`);
|
2018-04-05 18:48:09 +02:00
|
|
|
|
2019-04-17 07:20:01 +02:00
|
|
|
const monitor = getMonitor();
|
2018-04-05 18:48:09 +02:00
|
|
|
const resopAsync = prepareResource(label, res, true, props, opts);
|
2019-02-22 05:18:29 +01:00
|
|
|
|
2018-09-08 00:19:18 +02:00
|
|
|
const preallocError = new Error();
|
2018-04-05 18:48:09 +02:00
|
|
|
debuggablePromise(resopAsync.then(async (resop) => {
|
2019-02-22 05:18:29 +01:00
|
|
|
const resolvedID = await serializeProperty(label, id, new Set());
|
2018-04-05 18:48:09 +02:00
|
|
|
log.debug(`ReadResource RPC prepared: id=${resolvedID}, t=${t}, name=${name}` +
|
|
|
|
(excessiveDebugOutput ? `, obj=${JSON.stringify(resop.serializedProps)}` : ``));
|
|
|
|
|
2018-04-07 16:52:10 +02:00
|
|
|
// Create a resource request and do the RPC.
|
2018-04-05 18:48:09 +02:00
|
|
|
const req = new resproto.ReadResourceRequest();
|
|
|
|
req.setType(t);
|
|
|
|
req.setName(name);
|
|
|
|
req.setId(resolvedID);
|
|
|
|
req.setParent(resop.parentURN);
|
Implement first-class providers. (#1695)
### First-Class Providers
These changes implement support for first-class providers. First-class
providers are provider plugins that are exposed as resources via the
Pulumi programming model so that they may be explicitly and multiply
instantiated. Each instance of a provider resource may be configured
differently, and configuration parameters may be source from the
outputs of other resources.
### Provider Plugin Changes
In order to accommodate the need to verify and diff provider
configuration and configure providers without complete configuration
information, these changes adjust the high-level provider plugin
interface. Two new methods for validating a provider's configuration
and diffing changes to the same have been added (`CheckConfig` and
`DiffConfig`, respectively), and the type of the configuration bag
accepted by `Configure` has been changed to a `PropertyMap`.
These changes have not yet been reflected in the provider plugin gRPC
interface. We will do this in a set of follow-up changes. Until then,
these methods are implemented by adapters:
- `CheckConfig` validates that all configuration parameters are string
or unknown properties. This is necessary because existing plugins
only accept string-typed configuration values.
- `DiffConfig` either returns "never replace" if all configuration
values are known or "must replace" if any configuration value is
unknown. The justification for this behavior is given
[here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106)
- `Configure` converts the config bag to a legacy config map and
configures the provider plugin if all config values are known. If any
config value is unknown, the underlying plugin is not configured and
the provider may only perform `Check`, `Read`, and `Invoke`, all of
which return empty results. We justify this behavior becuase it is
only possible during a preview and provides the best experience we
can manage with the existing gRPC interface.
### Resource Model Changes
Providers are now exposed as resources that participate in a stack's
dependency graph. Like other resources, they are explicitly created,
may have multiple instances, and may have dependencies on other
resources. Providers are referred to using provider references, which
are a combination of the provider's URN and its ID. This design
addresses the need during a preview to refer to providers that have not
yet been physically created and therefore have no ID.
All custom resources that are not themselves providers must specify a
single provider via a provider reference. The named provider will be
used to manage that resource's CRUD operations. If a resource's
provider reference changes, the resource must be replaced. Though its
URN is not present in the resource's dependency list, the provider
should be treated as a dependency of the resource when topologically
sorting the dependency graph.
Finally, `Invoke` operations must now specify a provider to use for the
invocation via a provider reference.
### Engine Changes
First-class providers support requires a few changes to the engine:
- The engine must have some way to map from provider references to
provider plugins. It must be possible to add providers from a stack's
checkpoint to this map and to register new/updated providers during
the execution of a plan in response to CRUD operations on provider
resources.
- In order to support updating existing stacks using existing Pulumi
programs that may not explicitly instantiate providers, the engine
must be able to manage the "default" providers for each package
referenced by a checkpoint or Pulumi program. The configuration for
a "default" provider is taken from the stack's configuration data.
The former need is addressed by adding a provider registry type that is
responsible for managing all of the plugins required by a plan. In
addition to loading plugins froma checkpoint and providing the ability
to map from a provider reference to a provider plugin, this type serves
as the provider plugin for providers themselves (i.e. it is the
"provider provider").
The latter need is solved via two relatively self-contained changes to
plan setup and the eval source.
During plan setup, the old checkpoint is scanned for custom resources
that do not have a provider reference in order to compute the set of
packages that require a default provider. Once this set has been
computed, the required default provider definitions are conjured and
prepended to the checkpoint's resource list. Each resource that
requires a default provider is then updated to refer to the default
provider for its package.
While an eval source is running, each custom resource registration,
resource read, and invoke that does not name a provider is trapped
before being returned by the source iterator. If no default provider
for the appropriate package has been registered, the eval source
synthesizes an appropriate registration, waits for it to complete, and
records the registered provider's reference. This reference is injected
into the original request, which is then processed as usual. If a
default provider was already registered, the recorded reference is
used and no new registration occurs.
### SDK Changes
These changes only expose first-class providers from the Node.JS SDK.
- A new abstract class, `ProviderResource`, can be subclassed and used
to instantiate first-class providers.
- A new field in `ResourceOptions`, `provider`, can be used to supply
a particular provider instance to manage a `CustomResource`'s CRUD
operations.
- A new type, `InvokeOptions`, can be used to specify options that
control the behavior of a call to `pulumi.runtime.invoke`. This type
includes a `provider` field that is analogous to
`ResourceOptions.provider`.
2018-08-07 02:50:29 +02:00
|
|
|
req.setProvider(resop.providerRef);
|
2018-04-05 18:48:09 +02:00
|
|
|
req.setProperties(gstruct.Struct.fromJavaScript(resop.serializedProps));
|
2019-02-22 05:18:29 +01:00
|
|
|
req.setDependenciesList(Array.from(resop.allDirectDependencyURNs));
|
2019-04-23 20:02:51 +02:00
|
|
|
req.setVersion(opts.version || "");
|
2019-04-13 01:59:18 +02:00
|
|
|
req.setAcceptsecrets(true);
|
2020-12-10 19:21:05 +01:00
|
|
|
req.setAcceptresources(!utils.disableResourceReferences);
|
2020-04-06 23:50:52 +02:00
|
|
|
req.setAdditionalsecretoutputsList((<any>opts).additionalSecretOutputs || []);
|
2018-04-05 18:48:09 +02:00
|
|
|
|
|
|
|
// Now run the operation, serializing the invocation if necessary.
|
|
|
|
const opLabel = `monitor.readResource(${label})`;
|
|
|
|
runAsyncResourceOp(opLabel, async () => {
|
2019-04-17 07:20:01 +02:00
|
|
|
let resp: any;
|
2020-09-24 04:06:26 +02:00
|
|
|
let err: Error | undefined;
|
|
|
|
try {
|
|
|
|
if (monitor) {
|
|
|
|
// If we're attached to the engine, make an RPC call and wait for it to resolve.
|
|
|
|
resp = await debuggablePromise(new Promise((resolve, reject) =>
|
|
|
|
(monitor as any).readResource(req, (rpcError: grpc.ServiceError, innerResponse: any) => {
|
|
|
|
log.debug(`ReadResource RPC finished: ${label}; err: ${rpcError}, resp: ${innerResponse}`);
|
|
|
|
if (rpcError) {
|
|
|
|
if (rpcError.code === grpc.status.UNAVAILABLE || rpcError.code === grpc.status.CANCELLED) {
|
2020-10-01 20:04:06 +02:00
|
|
|
err = rpcError;
|
2020-09-24 04:06:26 +02:00
|
|
|
terminateRpcs();
|
|
|
|
rpcError.message = "Resource monitor is terminating";
|
|
|
|
(<any>preallocError).code = rpcError.code;
|
|
|
|
}
|
|
|
|
|
|
|
|
preallocError.message =
|
|
|
|
`failed to read resource #${resolvedID} '${name}' [${t}]: ${rpcError.message}`;
|
|
|
|
reject(preallocError);
|
|
|
|
}
|
|
|
|
else {
|
|
|
|
resolve(innerResponse);
|
|
|
|
}
|
|
|
|
})), opLabel);
|
|
|
|
} else {
|
|
|
|
// If we aren't attached to the engine, in test mode, mock up a fake response for testing purposes.
|
|
|
|
const mockurn = await createUrn(req.getName(), req.getType(), req.getParent()).promise();
|
|
|
|
resp = {
|
|
|
|
getUrn: () => mockurn,
|
|
|
|
getProperties: () => req.getProperties(),
|
|
|
|
};
|
|
|
|
}
|
|
|
|
} catch (e) {
|
|
|
|
err = e;
|
2019-04-17 07:20:01 +02:00
|
|
|
resp = {
|
2020-09-24 04:06:26 +02:00
|
|
|
getUrn: () => "",
|
|
|
|
getProperties: () => undefined,
|
2019-04-17 07:20:01 +02:00
|
|
|
};
|
|
|
|
}
|
2018-04-05 18:48:09 +02:00
|
|
|
|
|
|
|
// Now resolve everything: the URN, the ID (supplied as input), and the output properties.
|
2020-09-24 04:06:26 +02:00
|
|
|
resop.resolveURN(resp.getUrn(), err);
|
|
|
|
resop.resolveID!(resolvedID, resolvedID !== undefined, err);
|
|
|
|
await resolveOutputs(res, t, name, props, resp.getProperties(), {}, resop.resolvers, err);
|
2018-04-05 18:48:09 +02:00
|
|
|
});
|
2018-11-25 03:57:17 +01:00
|
|
|
}), label);
|
2018-04-05 18:48:09 +02:00
|
|
|
}
|
|
|
|
|
2017-09-22 03:15:29 +02:00
|
|
|
/**
|
2017-11-29 20:27:32 +01:00
|
|
|
* registerResource registers a new resource object with a given type t and name. It returns the auto-generated
|
Bring back component outputs
This change brings back component outputs to the overall system again.
In doing so, it generally overhauls the way we do resource RPCs a bit:
* Instead of RegisterResource and CompleteResource, we call these
BeginRegisterResource and EndRegisterResource, which begins to model
these as effectively "asynchronous" resource requests. This should also
help with parallelism (https://github.com/pulumi/pulumi/issues/106).
* Flip the CLI/engine a little on its head. Rather than it driving the
planning and deployment process, we move more to a model where it
simply observes it. This is done by implementing an event handler
interface with three events: OnResourceStepPre, OnResourceStepPost,
and OnResourceComplete. The first two are invoked immediately before
and after any step operation, and the latter is invoked whenever a
EndRegisterResource comes in. The reason for the asymmetry here is
that the checkpointing logic in the deployment engine is largely
untouched (intentionally, as this is a sensitive part of the system),
and so the "begin"/"end" nature doesn't flow through faithfully.
* Also make the engine more event-oriented in its terminology and the
way it handles the incoming BeginRegisterResource and
EndRegisterResource events from the language host. This is the first
step down a long road of incrementally refactoring the engine to work
this way, a necessary prerequisite for parallelism.
2017-11-29 16:42:14 +01:00
|
|
|
* URN and the ID that will resolve after the deployment has completed. All properties will be initialized to property
|
2017-09-22 03:15:29 +02:00
|
|
|
* objects that the registration operation will resolve at the right time (or remain unresolved for deployments).
|
|
|
|
*/
|
Initial support for remote component construction. (#5280)
These changes add initial support for the construction of remote
components. For now, this support is limited to the NodeJS SDK;
follow-up changes will implement support for the other SDKs.
Remote components are component resources that are constructed and
managed by plugins rather than by Pulumi programs. In this sense, they
are a bit like cloud resources, and are supported by the same
distribution and plugin loading mechanisms and described by the same
schema system.
The construction of a remote component is initiated by a
`RegisterResourceRequest` with the new `remote` field set to `true`.
When the resource monitor receives such a request, it loads the plugin
that implements the component resource and calls the `Construct`
method added to the resource provider interface as part of these
changes. This method accepts the information necessary to construct the
component and its children: the component's name, type, resource
options, inputs, and input dependencies. It is responsible for
dispatching to the appropriate component factory to create the
component, then returning its URN, resolved output properties, and
output property dependencies. The dependency information is necessary to
support features such as delete-before-replace, which rely on precise
dependency information for custom resources.
These changes also add initial support for more conveniently
implementing resource providers in NodeJS. The interface used to
implement such a provider is similar to the dynamic provider interface
(and may be unified with that interface in the future).
An example of a NodeJS program constructing a remote component resource
also implemented in NodeJS can be found in
`tests/construct_component/nodejs`.
This is the core of #2430.
2020-09-08 04:33:55 +02:00
|
|
|
export function registerResource(res: Resource, t: string, name: string, custom: boolean, remote: boolean,
|
|
|
|
newDependency: (urn: URN) => Resource, props: Inputs, opts: ResourceOptions): void {
|
2017-10-31 15:48:59 +01:00
|
|
|
const label = `resource:${name}[${t}]`;
|
Initial support for remote component construction. (#5280)
These changes add initial support for the construction of remote
components. For now, this support is limited to the NodeJS SDK;
follow-up changes will implement support for the other SDKs.
Remote components are component resources that are constructed and
managed by plugins rather than by Pulumi programs. In this sense, they
are a bit like cloud resources, and are supported by the same
distribution and plugin loading mechanisms and described by the same
schema system.
The construction of a remote component is initiated by a
`RegisterResourceRequest` with the new `remote` field set to `true`.
When the resource monitor receives such a request, it loads the plugin
that implements the component resource and calls the `Construct`
method added to the resource provider interface as part of these
changes. This method accepts the information necessary to construct the
component and its children: the component's name, type, resource
options, inputs, and input dependencies. It is responsible for
dispatching to the appropriate component factory to create the
component, then returning its URN, resolved output properties, and
output property dependencies. The dependency information is necessary to
support features such as delete-before-replace, which rely on precise
dependency information for custom resources.
These changes also add initial support for more conveniently
implementing resource providers in NodeJS. The interface used to
implement such a provider is similar to the dynamic provider interface
(and may be unified with that interface in the future).
An example of a NodeJS program constructing a remote component resource
also implemented in NodeJS can be found in
`tests/construct_component/nodejs`.
This is the core of #2430.
2020-09-08 04:33:55 +02:00
|
|
|
log.debug(`Registering resource: t=${t}, name=${name}, custom=${custom}, remote=${remote}`);
|
2018-04-05 18:48:09 +02:00
|
|
|
|
2019-04-17 07:20:01 +02:00
|
|
|
const monitor = getMonitor();
|
2018-04-05 18:48:09 +02:00
|
|
|
const resopAsync = prepareResource(label, res, custom, props, opts);
|
2018-09-08 00:19:18 +02:00
|
|
|
|
|
|
|
// In order to present a useful stack trace if an error does occur, we preallocate potential
|
|
|
|
// errors here. V8 captures a stack trace at the moment an Error is created and this stack
|
|
|
|
// trace will lead directly to user code. Throwing in `runAsyncResourceOp` results in an Error
|
|
|
|
// with a non-useful stack trace.
|
|
|
|
const preallocError = new Error();
|
2018-04-05 18:48:09 +02:00
|
|
|
debuggablePromise(resopAsync.then(async (resop) => {
|
|
|
|
log.debug(`RegisterResource RPC prepared: t=${t}, name=${name}` +
|
|
|
|
(excessiveDebugOutput ? `, obj=${JSON.stringify(resop.serializedProps)}` : ``));
|
|
|
|
|
|
|
|
const req = new resproto.RegisterResourceRequest();
|
|
|
|
req.setType(t);
|
|
|
|
req.setName(name);
|
|
|
|
req.setParent(resop.parentURN);
|
|
|
|
req.setCustom(custom);
|
|
|
|
req.setObject(gstruct.Struct.fromJavaScript(resop.serializedProps));
|
|
|
|
req.setProtect(opts.protect);
|
Implement first-class providers. (#1695)
### First-Class Providers
These changes implement support for first-class providers. First-class
providers are provider plugins that are exposed as resources via the
Pulumi programming model so that they may be explicitly and multiply
instantiated. Each instance of a provider resource may be configured
differently, and configuration parameters may be source from the
outputs of other resources.
### Provider Plugin Changes
In order to accommodate the need to verify and diff provider
configuration and configure providers without complete configuration
information, these changes adjust the high-level provider plugin
interface. Two new methods for validating a provider's configuration
and diffing changes to the same have been added (`CheckConfig` and
`DiffConfig`, respectively), and the type of the configuration bag
accepted by `Configure` has been changed to a `PropertyMap`.
These changes have not yet been reflected in the provider plugin gRPC
interface. We will do this in a set of follow-up changes. Until then,
these methods are implemented by adapters:
- `CheckConfig` validates that all configuration parameters are string
or unknown properties. This is necessary because existing plugins
only accept string-typed configuration values.
- `DiffConfig` either returns "never replace" if all configuration
values are known or "must replace" if any configuration value is
unknown. The justification for this behavior is given
[here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106)
- `Configure` converts the config bag to a legacy config map and
configures the provider plugin if all config values are known. If any
config value is unknown, the underlying plugin is not configured and
the provider may only perform `Check`, `Read`, and `Invoke`, all of
which return empty results. We justify this behavior becuase it is
only possible during a preview and provides the best experience we
can manage with the existing gRPC interface.
### Resource Model Changes
Providers are now exposed as resources that participate in a stack's
dependency graph. Like other resources, they are explicitly created,
may have multiple instances, and may have dependencies on other
resources. Providers are referred to using provider references, which
are a combination of the provider's URN and its ID. This design
addresses the need during a preview to refer to providers that have not
yet been physically created and therefore have no ID.
All custom resources that are not themselves providers must specify a
single provider via a provider reference. The named provider will be
used to manage that resource's CRUD operations. If a resource's
provider reference changes, the resource must be replaced. Though its
URN is not present in the resource's dependency list, the provider
should be treated as a dependency of the resource when topologically
sorting the dependency graph.
Finally, `Invoke` operations must now specify a provider to use for the
invocation via a provider reference.
### Engine Changes
First-class providers support requires a few changes to the engine:
- The engine must have some way to map from provider references to
provider plugins. It must be possible to add providers from a stack's
checkpoint to this map and to register new/updated providers during
the execution of a plan in response to CRUD operations on provider
resources.
- In order to support updating existing stacks using existing Pulumi
programs that may not explicitly instantiate providers, the engine
must be able to manage the "default" providers for each package
referenced by a checkpoint or Pulumi program. The configuration for
a "default" provider is taken from the stack's configuration data.
The former need is addressed by adding a provider registry type that is
responsible for managing all of the plugins required by a plan. In
addition to loading plugins froma checkpoint and providing the ability
to map from a provider reference to a provider plugin, this type serves
as the provider plugin for providers themselves (i.e. it is the
"provider provider").
The latter need is solved via two relatively self-contained changes to
plan setup and the eval source.
During plan setup, the old checkpoint is scanned for custom resources
that do not have a provider reference in order to compute the set of
packages that require a default provider. Once this set has been
computed, the required default provider definitions are conjured and
prepended to the checkpoint's resource list. Each resource that
requires a default provider is then updated to refer to the default
provider for its package.
While an eval source is running, each custom resource registration,
resource read, and invoke that does not name a provider is trapped
before being returned by the source iterator. If no default provider
for the appropriate package has been registered, the eval source
synthesizes an appropriate registration, waits for it to complete, and
records the registered provider's reference. This reference is injected
into the original request, which is then processed as usual. If a
default provider was already registered, the recorded reference is
used and no new registration occurs.
### SDK Changes
These changes only expose first-class providers from the Node.JS SDK.
- A new abstract class, `ProviderResource`, can be subclassed and used
to instantiate first-class providers.
- A new field in `ResourceOptions`, `provider`, can be used to supply
a particular provider instance to manage a `CustomResource`'s CRUD
operations.
- A new type, `InvokeOptions`, can be used to specify options that
control the behavior of a call to `pulumi.runtime.invoke`. This type
includes a `provider` field that is analogous to
`ResourceOptions.provider`.
2018-08-07 02:50:29 +02:00
|
|
|
req.setProvider(resop.providerRef);
|
2019-02-22 05:18:29 +01:00
|
|
|
req.setDependenciesList(Array.from(resop.allDirectDependencyURNs));
|
2019-01-31 23:27:53 +01:00
|
|
|
req.setDeletebeforereplace((<any>opts).deleteBeforeReplace || false);
|
2019-08-21 00:51:02 +02:00
|
|
|
req.setDeletebeforereplacedefined((<any>opts).deleteBeforeReplace !== undefined);
|
2019-04-22 22:54:48 +02:00
|
|
|
req.setIgnorechangesList(opts.ignoreChanges || []);
|
2019-04-23 20:02:51 +02:00
|
|
|
req.setVersion(opts.version || "");
|
2019-04-13 01:59:18 +02:00
|
|
|
req.setAcceptsecrets(true);
|
2020-12-10 19:21:05 +01:00
|
|
|
req.setAcceptresources(!utils.disableResourceReferences);
|
2019-05-09 23:27:34 +02:00
|
|
|
req.setAdditionalsecretoutputsList((<any>opts).additionalSecretOutputs || []);
|
2019-06-01 08:01:01 +02:00
|
|
|
req.setAliasesList(resop.aliases);
|
2019-07-12 20:12:01 +02:00
|
|
|
req.setImportid(resop.import || "");
|
Propagate inputs to outputs during preview. (#3327)
These changes restore a more-correct version of the behavior that was
disabled with #3014. The original implementation of this behavior was
done in the SDKs, which do not have access to the complete inputs for a
resource (in particular, default values filled in by the provider during
`Check` are not exposed to the SDK). This lack of information meant that
the resolved output values could disagree with the typings present in
a provider SDK. Exacerbating this problem was the fact that unknown
values were dropped entirely, causing `undefined` values to appear in
unexpected places.
By doing this in the engine and allowing unknown values to be
represented in a first-class manner in the SDK, we can attack both of
these issues.
Although this behavior is not _strictly_ consistent with respect to the
resource model--in an update, a resource's output properties will come
from its provider and may differ from its input properties--this
behavior was present in the product for a fairly long time without
significant issues. In the future, we may be able to improve the
accuracy of resource outputs during a preview by allowing the provider
to dry-run CRUD operations and return partially-known values where
possible.
These changes also introduce new APIs in the Node and Python SDKs
that work with unknown values in a first-class fashion:
- A new parameter to the `apply` function that indicates that the
callback should be run even if the result of the apply contains
unknown values
- `containsUnknowns` and `isUnknown`, which return true if a value
either contains nested unknown values or is exactly an unknown value
- The `Unknown` type, which represents unknown values
The primary use case for these APIs is to allow nested, properties with
known values to be accessed via the lifted property accessor even when
the containing property is not fully know. A common example of this
pattern is the `metadata.name` property of a Kubernetes `Namespace`
object: while other properties of the `metadata` bag may be unknown,
`name` is often known. These APIs allow `ns.metadata.name` to return a
known value in this case.
In order to avoid exposing downlevel SDKs to unknown values--a change
which could break user code by exposing it to unexpected values--a
language SDK must indicate whether or not it supports first-class
unknown values as part of each `RegisterResourceRequest`.
These changes also allow us to avoid breaking user code with the new
behavior introduced by the prior commit.
Fixes #3190.
2019-11-11 21:09:34 +01:00
|
|
|
req.setSupportspartialvalues(true);
|
Initial support for remote component construction. (#5280)
These changes add initial support for the construction of remote
components. For now, this support is limited to the NodeJS SDK;
follow-up changes will implement support for the other SDKs.
Remote components are component resources that are constructed and
managed by plugins rather than by Pulumi programs. In this sense, they
are a bit like cloud resources, and are supported by the same
distribution and plugin loading mechanisms and described by the same
schema system.
The construction of a remote component is initiated by a
`RegisterResourceRequest` with the new `remote` field set to `true`.
When the resource monitor receives such a request, it loads the plugin
that implements the component resource and calls the `Construct`
method added to the resource provider interface as part of these
changes. This method accepts the information necessary to construct the
component and its children: the component's name, type, resource
options, inputs, and input dependencies. It is responsible for
dispatching to the appropriate component factory to create the
component, then returning its URN, resolved output properties, and
output property dependencies. The dependency information is necessary to
support features such as delete-before-replace, which rely on precise
dependency information for custom resources.
These changes also add initial support for more conveniently
implementing resource providers in NodeJS. The interface used to
implement such a provider is similar to the dynamic provider interface
(and may be unified with that interface in the future).
An example of a NodeJS program constructing a remote component resource
also implemented in NodeJS can be found in
`tests/construct_component/nodejs`.
This is the core of #2430.
2020-09-08 04:33:55 +02:00
|
|
|
req.setRemote(remote);
|
2018-04-05 18:48:09 +02:00
|
|
|
|
Addition of Custom Timeouts (#2885)
* Plumbing the custom timeouts from the engine to the providers
* Plumbing the CustomTimeouts through to the engine and adding test to show this
* Change the provider proto to include individual timeouts
* Plumbing the CustomTimeouts from the engine through to the Provider RPC interface
* Change how the CustomTimeouts are sent across RPC
These errors were spotted in testing. We can now see that the timeout
information is arriving in the RegisterResourceRequest
```
req=&pulumirpc.RegisterResourceRequest{
Type: "aws:s3/bucket:Bucket",
Name: "my-bucket",
Parent: "urn:pulumi:dev::aws-vpc::pulumi:pulumi:Stack::aws-vpc-dev",
Custom: true,
Object: &structpb.Struct{},
Protect: false,
Dependencies: nil,
Provider: "",
PropertyDependencies: {},
DeleteBeforeReplace: false,
Version: "",
IgnoreChanges: nil,
AcceptSecrets: true,
AdditionalSecretOutputs: nil,
Aliases: nil,
CustomTimeouts: &pulumirpc.RegisterResourceRequest_CustomTimeouts{
Create: 300,
Update: 400,
Delete: 500,
XXX_NoUnkeyedLiteral: struct {}{},
XXX_unrecognized: nil,
XXX_sizecache: 0,
},
XXX_NoUnkeyedLiteral: struct {}{},
XXX_unrecognized: nil,
XXX_sizecache: 0,
}
```
* Changing the design to use strings
* CHANGELOG entry to include the CustomTimeouts work
* Changing custom timeouts to be passed around the engine as converted value
We don't want to pass around strings - the user can provide it but we want
to make the engine aware of the timeout in seconds as a float64
2019-07-15 23:26:28 +02:00
|
|
|
const customTimeouts = new resproto.RegisterResourceRequest.CustomTimeouts();
|
|
|
|
if (opts.customTimeouts != null) {
|
|
|
|
customTimeouts.setCreate(opts.customTimeouts.create);
|
|
|
|
customTimeouts.setUpdate(opts.customTimeouts.update);
|
|
|
|
customTimeouts.setDelete(opts.customTimeouts.delete);
|
|
|
|
}
|
|
|
|
req.setCustomtimeouts(customTimeouts);
|
|
|
|
|
Implement more precise delete-before-replace semantics. (#2369)
This implements the new algorithm for deciding which resources must be
deleted due to a delete-before-replace operation.
We need to compute the set of resources that may be replaced by a
change to the resource under consideration. We do this by taking the
complete set of transitive dependents on the resource under
consideration and removing any resources that would not be replaced by
changes to their dependencies. We determine whether or not a resource
may be replaced by substituting unknowns for input properties that may
change due to deletion of the resources their value depends on and
calling the resource provider's Diff method.
This is perhaps clearer when described by example. Consider the
following dependency graph:
A
__|__
B C
| _|_
D E F
In this graph, all of B, C, D, E, and F transitively depend on A. It may
be the case, however, that changes to the specific properties of any of
those resources R that would occur if a resource on the path to A were
deleted and recreated may not cause R to be replaced. For example, the
edge from B to A may be a simple dependsOn edge such that a change to
B does not actually influence any of B's input properties. In that case,
neither B nor D would need to be deleted before A could be deleted.
In order to make the above algorithm a reality, the resource monitor
interface has been updated to include a map that associates an input
property key with the list of resources that input property depends on.
Older clients of the resource monitor will leave this map empty, in
which case all input properties will be treated as depending on all
dependencies of the resource. This is probably overly conservative, but
it is less conservative than what we currently implement, and is
certainly correct.
2019-01-28 18:46:30 +01:00
|
|
|
const propertyDependencies = req.getPropertydependenciesMap();
|
2019-02-22 05:18:29 +01:00
|
|
|
for (const [key, resourceURNs] of resop.propertyToDirectDependencyURNs) {
|
Implement more precise delete-before-replace semantics. (#2369)
This implements the new algorithm for deciding which resources must be
deleted due to a delete-before-replace operation.
We need to compute the set of resources that may be replaced by a
change to the resource under consideration. We do this by taking the
complete set of transitive dependents on the resource under
consideration and removing any resources that would not be replaced by
changes to their dependencies. We determine whether or not a resource
may be replaced by substituting unknowns for input properties that may
change due to deletion of the resources their value depends on and
calling the resource provider's Diff method.
This is perhaps clearer when described by example. Consider the
following dependency graph:
A
__|__
B C
| _|_
D E F
In this graph, all of B, C, D, E, and F transitively depend on A. It may
be the case, however, that changes to the specific properties of any of
those resources R that would occur if a resource on the path to A were
deleted and recreated may not cause R to be replaced. For example, the
edge from B to A may be a simple dependsOn edge such that a change to
B does not actually influence any of B's input properties. In that case,
neither B nor D would need to be deleted before A could be deleted.
In order to make the above algorithm a reality, the resource monitor
interface has been updated to include a map that associates an input
property key with the list of resources that input property depends on.
Older clients of the resource monitor will leave this map empty, in
which case all input properties will be treated as depending on all
dependencies of the resource. This is probably overly conservative, but
it is less conservative than what we currently implement, and is
certainly correct.
2019-01-28 18:46:30 +01:00
|
|
|
const deps = new resproto.RegisterResourceRequest.PropertyDependencies();
|
2019-02-22 05:18:29 +01:00
|
|
|
deps.setUrnsList(Array.from(resourceURNs));
|
Implement more precise delete-before-replace semantics. (#2369)
This implements the new algorithm for deciding which resources must be
deleted due to a delete-before-replace operation.
We need to compute the set of resources that may be replaced by a
change to the resource under consideration. We do this by taking the
complete set of transitive dependents on the resource under
consideration and removing any resources that would not be replaced by
changes to their dependencies. We determine whether or not a resource
may be replaced by substituting unknowns for input properties that may
change due to deletion of the resources their value depends on and
calling the resource provider's Diff method.
This is perhaps clearer when described by example. Consider the
following dependency graph:
A
__|__
B C
| _|_
D E F
In this graph, all of B, C, D, E, and F transitively depend on A. It may
be the case, however, that changes to the specific properties of any of
those resources R that would occur if a resource on the path to A were
deleted and recreated may not cause R to be replaced. For example, the
edge from B to A may be a simple dependsOn edge such that a change to
B does not actually influence any of B's input properties. In that case,
neither B nor D would need to be deleted before A could be deleted.
In order to make the above algorithm a reality, the resource monitor
interface has been updated to include a map that associates an input
property key with the list of resources that input property depends on.
Older clients of the resource monitor will leave this map empty, in
which case all input properties will be treated as depending on all
dependencies of the resource. This is probably overly conservative, but
it is less conservative than what we currently implement, and is
certainly correct.
2019-01-28 18:46:30 +01:00
|
|
|
propertyDependencies.set(key, deps);
|
|
|
|
}
|
|
|
|
|
2018-04-05 18:48:09 +02:00
|
|
|
// Now run the operation, serializing the invocation if necessary.
|
|
|
|
const opLabel = `monitor.registerResource(${label})`;
|
|
|
|
runAsyncResourceOp(opLabel, async () => {
|
2019-04-17 07:20:01 +02:00
|
|
|
let resp: any;
|
2020-09-24 04:06:26 +02:00
|
|
|
let err: Error | undefined;
|
|
|
|
try {
|
|
|
|
if (monitor) {
|
|
|
|
// If we're running with an attachment to the engine, perform the operation.
|
|
|
|
resp = await debuggablePromise(new Promise((resolve, reject) =>
|
|
|
|
(monitor as any).registerResource(req, (rpcErr: grpc.ServiceError, innerResponse: any) => {
|
|
|
|
if (rpcErr) {
|
2020-10-01 20:04:06 +02:00
|
|
|
err = rpcErr;
|
2020-09-24 04:06:26 +02:00
|
|
|
// If the monitor is unavailable, it is in the process of shutting down or has already
|
|
|
|
// shut down. Don't emit an error and don't do any more RPCs, just exit.
|
|
|
|
if (rpcErr.code === grpc.status.UNAVAILABLE || rpcErr.code === grpc.status.CANCELLED) {
|
|
|
|
// Re-emit the message
|
|
|
|
terminateRpcs();
|
|
|
|
rpcErr.message = "Resource monitor is terminating";
|
|
|
|
(<any>preallocError).code = rpcErr.code;
|
|
|
|
}
|
|
|
|
|
|
|
|
// Node lets us hack the message as long as we do it before accessing the `stack` property.
|
|
|
|
log.debug(`RegisterResource RPC finished: ${label}; err: ${rpcErr}, resp: ${innerResponse}`);
|
|
|
|
preallocError.message = `failed to register new resource ${name} [${t}]: ${rpcErr.message}`;
|
|
|
|
reject(preallocError);
|
2019-04-17 07:20:01 +02:00
|
|
|
}
|
2020-09-24 04:06:26 +02:00
|
|
|
else {
|
|
|
|
log.debug(`RegisterResource RPC finished: ${label}; err: ${rpcErr}, resp: ${innerResponse}`);
|
|
|
|
resolve(innerResponse);
|
|
|
|
}
|
|
|
|
})), opLabel);
|
|
|
|
} else {
|
|
|
|
// If we aren't attached to the engine, in test mode, mock up a fake response for testing purposes.
|
|
|
|
const mockurn = await createUrn(req.getName(), req.getType(), req.getParent()).promise();
|
|
|
|
resp = {
|
|
|
|
getUrn: () => mockurn,
|
|
|
|
getId: () => undefined,
|
|
|
|
getObject: () => req.getObject(),
|
2020-10-01 21:31:52 +02:00
|
|
|
getPropertydependenciesMap: () => undefined,
|
2020-09-24 04:06:26 +02:00
|
|
|
};
|
|
|
|
}
|
|
|
|
} catch (e) {
|
|
|
|
err = e;
|
2019-04-17 07:20:01 +02:00
|
|
|
resp = {
|
2020-09-24 04:06:26 +02:00
|
|
|
getUrn: () => "",
|
2019-04-17 07:20:01 +02:00
|
|
|
getId: () => undefined,
|
|
|
|
getObject: () => req.getObject(),
|
Initial support for remote component construction. (#5280)
These changes add initial support for the construction of remote
components. For now, this support is limited to the NodeJS SDK;
follow-up changes will implement support for the other SDKs.
Remote components are component resources that are constructed and
managed by plugins rather than by Pulumi programs. In this sense, they
are a bit like cloud resources, and are supported by the same
distribution and plugin loading mechanisms and described by the same
schema system.
The construction of a remote component is initiated by a
`RegisterResourceRequest` with the new `remote` field set to `true`.
When the resource monitor receives such a request, it loads the plugin
that implements the component resource and calls the `Construct`
method added to the resource provider interface as part of these
changes. This method accepts the information necessary to construct the
component and its children: the component's name, type, resource
options, inputs, and input dependencies. It is responsible for
dispatching to the appropriate component factory to create the
component, then returning its URN, resolved output properties, and
output property dependencies. The dependency information is necessary to
support features such as delete-before-replace, which rely on precise
dependency information for custom resources.
These changes also add initial support for more conveniently
implementing resource providers in NodeJS. The interface used to
implement such a provider is similar to the dynamic provider interface
(and may be unified with that interface in the future).
An example of a NodeJS program constructing a remote component resource
also implemented in NodeJS can be found in
`tests/construct_component/nodejs`.
This is the core of #2430.
2020-09-08 04:33:55 +02:00
|
|
|
getPropertydependenciesMap: () => undefined,
|
2019-04-17 07:20:01 +02:00
|
|
|
};
|
|
|
|
}
|
2018-04-05 18:48:09 +02:00
|
|
|
|
2020-09-24 04:06:26 +02:00
|
|
|
resop.resolveURN(resp.getUrn(), err);
|
2018-04-05 18:48:09 +02:00
|
|
|
|
|
|
|
// Note: 'id || undefined' is intentional. We intentionally collapse falsy values to
|
|
|
|
// undefined so that later parts of our system don't have to deal with values like 'null'.
|
|
|
|
if (resop.resolveID) {
|
|
|
|
const id = resp.getId() || undefined;
|
2020-09-24 04:06:26 +02:00
|
|
|
resop.resolveID(id, id !== undefined, err);
|
2018-04-05 18:48:09 +02:00
|
|
|
}
|
2017-10-31 15:48:59 +01:00
|
|
|
|
Initial support for remote component construction. (#5280)
These changes add initial support for the construction of remote
components. For now, this support is limited to the NodeJS SDK;
follow-up changes will implement support for the other SDKs.
Remote components are component resources that are constructed and
managed by plugins rather than by Pulumi programs. In this sense, they
are a bit like cloud resources, and are supported by the same
distribution and plugin loading mechanisms and described by the same
schema system.
The construction of a remote component is initiated by a
`RegisterResourceRequest` with the new `remote` field set to `true`.
When the resource monitor receives such a request, it loads the plugin
that implements the component resource and calls the `Construct`
method added to the resource provider interface as part of these
changes. This method accepts the information necessary to construct the
component and its children: the component's name, type, resource
options, inputs, and input dependencies. It is responsible for
dispatching to the appropriate component factory to create the
component, then returning its URN, resolved output properties, and
output property dependencies. The dependency information is necessary to
support features such as delete-before-replace, which rely on precise
dependency information for custom resources.
These changes also add initial support for more conveniently
implementing resource providers in NodeJS. The interface used to
implement such a provider is similar to the dynamic provider interface
(and may be unified with that interface in the future).
An example of a NodeJS program constructing a remote component resource
also implemented in NodeJS can be found in
`tests/construct_component/nodejs`.
This is the core of #2430.
2020-09-08 04:33:55 +02:00
|
|
|
const deps: Record<string, Resource[]> = {};
|
|
|
|
const rpcDeps = resp.getPropertydependenciesMap();
|
|
|
|
if (rpcDeps) {
|
|
|
|
for (const [k, propertyDeps] of resp.getPropertydependenciesMap().entries()) {
|
|
|
|
const urns = <URN[]>propertyDeps.getUrnsList();
|
|
|
|
deps[k] = urns.map(urn => newDependency(urn));
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2018-04-05 18:48:09 +02:00
|
|
|
// Now resolve the output properties.
|
2020-09-24 04:06:26 +02:00
|
|
|
await resolveOutputs(res, t, name, props, resp.getObject(), deps, resop.resolvers, err);
|
2018-04-05 18:48:09 +02:00
|
|
|
});
|
2018-11-25 03:57:17 +01:00
|
|
|
}), label);
|
2018-04-05 18:48:09 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
2019-02-22 05:18:29 +01:00
|
|
|
* Prepares for an RPC that will manufacture a resource, and hence deals with input and output
|
|
|
|
* properties.
|
2018-04-05 18:48:09 +02:00
|
|
|
*/
|
|
|
|
async function prepareResource(label: string, res: Resource, custom: boolean,
|
|
|
|
props: Inputs, opts: ResourceOptions): Promise<ResourceResolverOperation> {
|
2019-02-22 05:18:29 +01:00
|
|
|
|
2017-11-17 03:21:41 +01:00
|
|
|
// Simply initialize the URN property and get prepared to resolve it later on.
|
2018-03-18 08:15:22 +01:00
|
|
|
// Note: a resource urn will always get a value, and thus the output property
|
|
|
|
// for it can always run .apply calls.
|
2020-09-24 04:06:26 +02:00
|
|
|
let resolveURN: (urn: URN, err?: Error) => void;
|
|
|
|
{
|
|
|
|
let resolveValue: (urn: URN) => void;
|
|
|
|
let resolveIsKnown: (isKnown: boolean) => void;
|
|
|
|
(res as any).urn = new Output(
|
|
|
|
res,
|
|
|
|
debuggablePromise(
|
|
|
|
new Promise<URN>(resolve => resolveValue = resolve),
|
|
|
|
`resolveURN(${label})`),
|
|
|
|
debuggablePromise(
|
|
|
|
new Promise<boolean>(resolve => resolveIsKnown = resolve),
|
|
|
|
`resolveURNIsKnown(${label})`),
|
|
|
|
/*isSecret:*/ Promise.resolve(false),
|
|
|
|
Promise.resolve(res));
|
|
|
|
|
|
|
|
resolveURN = (v, err) => {
|
|
|
|
resolveValue(v);
|
|
|
|
resolveIsKnown(err === undefined);
|
|
|
|
};
|
|
|
|
}
|
2017-11-17 03:21:41 +01:00
|
|
|
|
2017-10-15 12:52:04 +02:00
|
|
|
// If a custom resource, make room for the ID property.
|
2020-09-24 04:06:26 +02:00
|
|
|
let resolveID: ((v: any, performApply: boolean, err?: Error) => void) | undefined;
|
2017-10-15 12:52:04 +02:00
|
|
|
if (custom) {
|
2018-03-18 08:15:22 +01:00
|
|
|
let resolveValue: (v: ID) => void;
|
2019-04-13 01:59:18 +02:00
|
|
|
let resolveIsKnown: (v: boolean) => void;
|
2018-10-27 05:49:16 +02:00
|
|
|
(res as any).id = new Output(
|
2018-02-05 23:44:23 +01:00
|
|
|
res,
|
2020-09-24 04:06:26 +02:00
|
|
|
debuggablePromise(new Promise<ID>(resolve => resolveValue = resolve),
|
|
|
|
`resolveID(${label})`),
|
2018-03-18 08:15:22 +01:00
|
|
|
debuggablePromise(new Promise<boolean>(
|
2019-04-13 01:59:18 +02:00
|
|
|
resolve => resolveIsKnown = resolve), `resolveIDIsKnown(${label})`),
|
2019-12-18 04:04:09 +01:00
|
|
|
Promise.resolve(false),
|
|
|
|
Promise.resolve(res));
|
2018-04-05 18:48:09 +02:00
|
|
|
|
2020-09-24 04:06:26 +02:00
|
|
|
resolveID = (v, isKnown, err) => {
|
2018-04-05 18:48:09 +02:00
|
|
|
resolveValue(v);
|
2020-09-24 04:06:26 +02:00
|
|
|
resolveIsKnown(err ? false : isKnown);
|
2018-04-05 18:48:09 +02:00
|
|
|
};
|
Implement components
This change implements core support for "components" in the Pulumi
Fabric. This work is described further in pulumi/pulumi#340, where
we are still discussing some of the finer points.
In a nutshell, resources no longer imply external providers. It's
entirely possible to have a resource that logically represents
something but without having a physical manifestation that needs to
be tracked and managed by our typical CRUD operations.
For example, the aws/serverless/Function helper is one such type.
It aggregates Lambda-related resources and exposes a nice interface.
All of the Pulumi Cloud Framework resources are also examples.
To indicate that a resource does participate in the usual CRUD resource
provider, it simply derives from ExternalResource instead of Resource.
All resources now have the ability to adopt children. This is purely
a metadata/tagging thing, and will help us roll up displays, provide
attribution to the developer, and even hide aspects of the resource
graph as appropriate (e.g., when they are implementation details).
Our use of this capability is ultra limited right now; in fact, the
only place we display children is in the CLI output. For instance:
+ aws:serverless:Function: (create)
[urn=urn:pulumi:demo::serverless::aws:serverless:Function::mylambda]
=> urn:pulumi:demo::serverless::aws:iam/role:Role::mylambda-iamrole
=> urn:pulumi:demo::serverless::aws:iam/rolePolicyAttachment:RolePolicyAttachment::mylambda-iampolicy-0
=> urn:pulumi:demo::serverless::aws:lambda/function:Function::mylambda
The bit indicating whether a resource is external or not is tracked
in the resulting checkpoint file, along with any of its children.
2017-10-14 23:18:43 +02:00
|
|
|
}
|
|
|
|
|
2018-01-25 22:34:21 +01:00
|
|
|
// Now "transfer" all input properties into unresolved Promises on res. This way,
|
|
|
|
// this resource will look like it has all its output properties to anyone it is
|
|
|
|
// passed to. However, those promises won't actually resolve until the registerResource
|
|
|
|
// RPC returns
|
2018-04-05 18:48:09 +02:00
|
|
|
const resolvers = transferProperties(res, label, props);
|
|
|
|
|
|
|
|
/** IMPORTANT! We should never await prior to this line, otherwise the Resource will be partly uninitialized. */
|
2017-11-21 02:38:09 +01:00
|
|
|
|
2018-07-12 03:33:53 +02:00
|
|
|
// Before we can proceed, all our dependencies must be finished.
|
2019-02-22 05:18:29 +01:00
|
|
|
const explicitDirectDependencies = new Set(await gatherExplicitDependencies(opts.dependsOn));
|
2018-02-05 23:44:23 +01:00
|
|
|
|
2018-07-12 03:33:53 +02:00
|
|
|
// Serialize out all our props to their final values. In doing so, we'll also collect all
|
|
|
|
// the Resources pointed to by any Dependency objects we encounter, adding them to 'propertyDependencies'.
|
2019-02-22 05:18:29 +01:00
|
|
|
const [serializedProps, propertyToDirectDependencies] = await serializeResourceProperties(label, props);
|
|
|
|
|
|
|
|
// Wait for the parent to complete.
|
|
|
|
// If no parent was provided, parent to the root resource.
|
|
|
|
const parentURN = opts.parent
|
|
|
|
? await opts.parent.urn.promise()
|
|
|
|
: await getRootResource();
|
2018-07-11 01:41:56 +02:00
|
|
|
|
Implement first-class providers. (#1695)
### First-Class Providers
These changes implement support for first-class providers. First-class
providers are provider plugins that are exposed as resources via the
Pulumi programming model so that they may be explicitly and multiply
instantiated. Each instance of a provider resource may be configured
differently, and configuration parameters may be source from the
outputs of other resources.
### Provider Plugin Changes
In order to accommodate the need to verify and diff provider
configuration and configure providers without complete configuration
information, these changes adjust the high-level provider plugin
interface. Two new methods for validating a provider's configuration
and diffing changes to the same have been added (`CheckConfig` and
`DiffConfig`, respectively), and the type of the configuration bag
accepted by `Configure` has been changed to a `PropertyMap`.
These changes have not yet been reflected in the provider plugin gRPC
interface. We will do this in a set of follow-up changes. Until then,
these methods are implemented by adapters:
- `CheckConfig` validates that all configuration parameters are string
or unknown properties. This is necessary because existing plugins
only accept string-typed configuration values.
- `DiffConfig` either returns "never replace" if all configuration
values are known or "must replace" if any configuration value is
unknown. The justification for this behavior is given
[here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106)
- `Configure` converts the config bag to a legacy config map and
configures the provider plugin if all config values are known. If any
config value is unknown, the underlying plugin is not configured and
the provider may only perform `Check`, `Read`, and `Invoke`, all of
which return empty results. We justify this behavior becuase it is
only possible during a preview and provides the best experience we
can manage with the existing gRPC interface.
### Resource Model Changes
Providers are now exposed as resources that participate in a stack's
dependency graph. Like other resources, they are explicitly created,
may have multiple instances, and may have dependencies on other
resources. Providers are referred to using provider references, which
are a combination of the provider's URN and its ID. This design
addresses the need during a preview to refer to providers that have not
yet been physically created and therefore have no ID.
All custom resources that are not themselves providers must specify a
single provider via a provider reference. The named provider will be
used to manage that resource's CRUD operations. If a resource's
provider reference changes, the resource must be replaced. Though its
URN is not present in the resource's dependency list, the provider
should be treated as a dependency of the resource when topologically
sorting the dependency graph.
Finally, `Invoke` operations must now specify a provider to use for the
invocation via a provider reference.
### Engine Changes
First-class providers support requires a few changes to the engine:
- The engine must have some way to map from provider references to
provider plugins. It must be possible to add providers from a stack's
checkpoint to this map and to register new/updated providers during
the execution of a plan in response to CRUD operations on provider
resources.
- In order to support updating existing stacks using existing Pulumi
programs that may not explicitly instantiate providers, the engine
must be able to manage the "default" providers for each package
referenced by a checkpoint or Pulumi program. The configuration for
a "default" provider is taken from the stack's configuration data.
The former need is addressed by adding a provider registry type that is
responsible for managing all of the plugins required by a plan. In
addition to loading plugins froma checkpoint and providing the ability
to map from a provider reference to a provider plugin, this type serves
as the provider plugin for providers themselves (i.e. it is the
"provider provider").
The latter need is solved via two relatively self-contained changes to
plan setup and the eval source.
During plan setup, the old checkpoint is scanned for custom resources
that do not have a provider reference in order to compute the set of
packages that require a default provider. Once this set has been
computed, the required default provider definitions are conjured and
prepended to the checkpoint's resource list. Each resource that
requires a default provider is then updated to refer to the default
provider for its package.
While an eval source is running, each custom resource registration,
resource read, and invoke that does not name a provider is trapped
before being returned by the source iterator. If no default provider
for the appropriate package has been registered, the eval source
synthesizes an appropriate registration, waits for it to complete, and
records the registered provider's reference. This reference is injected
into the original request, which is then processed as usual. If a
default provider was already registered, the recorded reference is
used and no new registration occurs.
### SDK Changes
These changes only expose first-class providers from the Node.JS SDK.
- A new abstract class, `ProviderResource`, can be subclassed and used
to instantiate first-class providers.
- A new field in `ResourceOptions`, `provider`, can be used to supply
a particular provider instance to manage a `CustomResource`'s CRUD
operations.
- A new type, `InvokeOptions`, can be used to specify options that
control the behavior of a call to `pulumi.runtime.invoke`. This type
includes a `provider` field that is analogous to
`ResourceOptions.provider`.
2018-08-07 02:50:29 +02:00
|
|
|
let providerRef: string | undefined;
|
2019-07-12 20:12:01 +02:00
|
|
|
let importID: ID | undefined;
|
|
|
|
if (custom) {
|
2019-10-15 07:08:06 +02:00
|
|
|
const customOpts = <CustomResourceOptions>opts;
|
|
|
|
importID = customOpts.import;
|
2019-10-18 02:12:45 +02:00
|
|
|
providerRef = await ProviderResource.register(opts.provider);
|
Implement first-class providers. (#1695)
### First-Class Providers
These changes implement support for first-class providers. First-class
providers are provider plugins that are exposed as resources via the
Pulumi programming model so that they may be explicitly and multiply
instantiated. Each instance of a provider resource may be configured
differently, and configuration parameters may be source from the
outputs of other resources.
### Provider Plugin Changes
In order to accommodate the need to verify and diff provider
configuration and configure providers without complete configuration
information, these changes adjust the high-level provider plugin
interface. Two new methods for validating a provider's configuration
and diffing changes to the same have been added (`CheckConfig` and
`DiffConfig`, respectively), and the type of the configuration bag
accepted by `Configure` has been changed to a `PropertyMap`.
These changes have not yet been reflected in the provider plugin gRPC
interface. We will do this in a set of follow-up changes. Until then,
these methods are implemented by adapters:
- `CheckConfig` validates that all configuration parameters are string
or unknown properties. This is necessary because existing plugins
only accept string-typed configuration values.
- `DiffConfig` either returns "never replace" if all configuration
values are known or "must replace" if any configuration value is
unknown. The justification for this behavior is given
[here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106)
- `Configure` converts the config bag to a legacy config map and
configures the provider plugin if all config values are known. If any
config value is unknown, the underlying plugin is not configured and
the provider may only perform `Check`, `Read`, and `Invoke`, all of
which return empty results. We justify this behavior becuase it is
only possible during a preview and provides the best experience we
can manage with the existing gRPC interface.
### Resource Model Changes
Providers are now exposed as resources that participate in a stack's
dependency graph. Like other resources, they are explicitly created,
may have multiple instances, and may have dependencies on other
resources. Providers are referred to using provider references, which
are a combination of the provider's URN and its ID. This design
addresses the need during a preview to refer to providers that have not
yet been physically created and therefore have no ID.
All custom resources that are not themselves providers must specify a
single provider via a provider reference. The named provider will be
used to manage that resource's CRUD operations. If a resource's
provider reference changes, the resource must be replaced. Though its
URN is not present in the resource's dependency list, the provider
should be treated as a dependency of the resource when topologically
sorting the dependency graph.
Finally, `Invoke` operations must now specify a provider to use for the
invocation via a provider reference.
### Engine Changes
First-class providers support requires a few changes to the engine:
- The engine must have some way to map from provider references to
provider plugins. It must be possible to add providers from a stack's
checkpoint to this map and to register new/updated providers during
the execution of a plan in response to CRUD operations on provider
resources.
- In order to support updating existing stacks using existing Pulumi
programs that may not explicitly instantiate providers, the engine
must be able to manage the "default" providers for each package
referenced by a checkpoint or Pulumi program. The configuration for
a "default" provider is taken from the stack's configuration data.
The former need is addressed by adding a provider registry type that is
responsible for managing all of the plugins required by a plan. In
addition to loading plugins froma checkpoint and providing the ability
to map from a provider reference to a provider plugin, this type serves
as the provider plugin for providers themselves (i.e. it is the
"provider provider").
The latter need is solved via two relatively self-contained changes to
plan setup and the eval source.
During plan setup, the old checkpoint is scanned for custom resources
that do not have a provider reference in order to compute the set of
packages that require a default provider. Once this set has been
computed, the required default provider definitions are conjured and
prepended to the checkpoint's resource list. Each resource that
requires a default provider is then updated to refer to the default
provider for its package.
While an eval source is running, each custom resource registration,
resource read, and invoke that does not name a provider is trapped
before being returned by the source iterator. If no default provider
for the appropriate package has been registered, the eval source
synthesizes an appropriate registration, waits for it to complete, and
records the registered provider's reference. This reference is injected
into the original request, which is then processed as usual. If a
default provider was already registered, the recorded reference is
used and no new registration occurs.
### SDK Changes
These changes only expose first-class providers from the Node.JS SDK.
- A new abstract class, `ProviderResource`, can be subclassed and used
to instantiate first-class providers.
- A new field in `ResourceOptions`, `provider`, can be used to supply
a particular provider instance to manage a `CustomResource`'s CRUD
operations.
- A new type, `InvokeOptions`, can be used to specify options that
control the behavior of a call to `pulumi.runtime.invoke`. This type
includes a `provider` field that is analogous to
`ResourceOptions.provider`.
2018-08-07 02:50:29 +02:00
|
|
|
}
|
|
|
|
|
2019-02-22 05:18:29 +01:00
|
|
|
// Collect the URNs for explicit/implicit dependencies for the engine so that it can understand
|
|
|
|
// the dependency graph and optimize operations accordingly.
|
|
|
|
|
|
|
|
// The list of all dependencies (implicit or explicit).
|
|
|
|
const allDirectDependencies = new Set<Resource>(explicitDirectDependencies);
|
|
|
|
|
2019-03-06 02:06:57 +01:00
|
|
|
const allDirectDependencyURNs = await getAllTransitivelyReferencedCustomResourceURNs(explicitDirectDependencies);
|
2019-02-22 05:18:29 +01:00
|
|
|
const propertyToDirectDependencyURNs = new Map<string, Set<URN>>();
|
|
|
|
|
|
|
|
for (const [propertyName, directDependencies] of propertyToDirectDependencies) {
|
|
|
|
addAll(allDirectDependencies, directDependencies);
|
|
|
|
|
2019-03-06 02:06:57 +01:00
|
|
|
const urns = await getAllTransitivelyReferencedCustomResourceURNs(directDependencies);
|
2019-02-22 05:18:29 +01:00
|
|
|
addAll(allDirectDependencyURNs, urns);
|
|
|
|
propertyToDirectDependencyURNs.set(propertyName, urns);
|
2018-07-12 03:33:53 +02:00
|
|
|
}
|
2018-02-21 18:43:17 +01:00
|
|
|
|
2019-06-01 08:01:01 +02:00
|
|
|
// Wait for all aliases. Note that we use `res.__aliases` instead of `opts.aliases` as the former has been processed
|
|
|
|
// in the Resource constructor prior to calling `registerResource` - both adding new inherited aliases and
|
|
|
|
// simplifying aliases down to URNs.
|
|
|
|
const aliases = [];
|
2019-06-23 18:49:04 +02:00
|
|
|
const uniqueAliases = new Set<string>();
|
2019-07-16 20:15:26 +02:00
|
|
|
for (const alias of (res.__aliases || [])) {
|
2019-06-23 18:49:04 +02:00
|
|
|
const aliasVal = await output(alias).promise();
|
|
|
|
if (!uniqueAliases.has(aliasVal)) {
|
|
|
|
uniqueAliases.add(aliasVal);
|
|
|
|
aliases.push(aliasVal);
|
|
|
|
}
|
2019-06-01 08:01:01 +02:00
|
|
|
}
|
|
|
|
|
2018-04-05 18:48:09 +02:00
|
|
|
return {
|
2020-09-24 04:06:26 +02:00
|
|
|
resolveURN: resolveURN,
|
2018-04-05 18:48:09 +02:00
|
|
|
resolveID: resolveID,
|
|
|
|
resolvers: resolvers,
|
|
|
|
serializedProps: serializedProps,
|
|
|
|
parentURN: parentURN,
|
Implement first-class providers. (#1695)
### First-Class Providers
These changes implement support for first-class providers. First-class
providers are provider plugins that are exposed as resources via the
Pulumi programming model so that they may be explicitly and multiply
instantiated. Each instance of a provider resource may be configured
differently, and configuration parameters may be source from the
outputs of other resources.
### Provider Plugin Changes
In order to accommodate the need to verify and diff provider
configuration and configure providers without complete configuration
information, these changes adjust the high-level provider plugin
interface. Two new methods for validating a provider's configuration
and diffing changes to the same have been added (`CheckConfig` and
`DiffConfig`, respectively), and the type of the configuration bag
accepted by `Configure` has been changed to a `PropertyMap`.
These changes have not yet been reflected in the provider plugin gRPC
interface. We will do this in a set of follow-up changes. Until then,
these methods are implemented by adapters:
- `CheckConfig` validates that all configuration parameters are string
or unknown properties. This is necessary because existing plugins
only accept string-typed configuration values.
- `DiffConfig` either returns "never replace" if all configuration
values are known or "must replace" if any configuration value is
unknown. The justification for this behavior is given
[here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106)
- `Configure` converts the config bag to a legacy config map and
configures the provider plugin if all config values are known. If any
config value is unknown, the underlying plugin is not configured and
the provider may only perform `Check`, `Read`, and `Invoke`, all of
which return empty results. We justify this behavior becuase it is
only possible during a preview and provides the best experience we
can manage with the existing gRPC interface.
### Resource Model Changes
Providers are now exposed as resources that participate in a stack's
dependency graph. Like other resources, they are explicitly created,
may have multiple instances, and may have dependencies on other
resources. Providers are referred to using provider references, which
are a combination of the provider's URN and its ID. This design
addresses the need during a preview to refer to providers that have not
yet been physically created and therefore have no ID.
All custom resources that are not themselves providers must specify a
single provider via a provider reference. The named provider will be
used to manage that resource's CRUD operations. If a resource's
provider reference changes, the resource must be replaced. Though its
URN is not present in the resource's dependency list, the provider
should be treated as a dependency of the resource when topologically
sorting the dependency graph.
Finally, `Invoke` operations must now specify a provider to use for the
invocation via a provider reference.
### Engine Changes
First-class providers support requires a few changes to the engine:
- The engine must have some way to map from provider references to
provider plugins. It must be possible to add providers from a stack's
checkpoint to this map and to register new/updated providers during
the execution of a plan in response to CRUD operations on provider
resources.
- In order to support updating existing stacks using existing Pulumi
programs that may not explicitly instantiate providers, the engine
must be able to manage the "default" providers for each package
referenced by a checkpoint or Pulumi program. The configuration for
a "default" provider is taken from the stack's configuration data.
The former need is addressed by adding a provider registry type that is
responsible for managing all of the plugins required by a plan. In
addition to loading plugins froma checkpoint and providing the ability
to map from a provider reference to a provider plugin, this type serves
as the provider plugin for providers themselves (i.e. it is the
"provider provider").
The latter need is solved via two relatively self-contained changes to
plan setup and the eval source.
During plan setup, the old checkpoint is scanned for custom resources
that do not have a provider reference in order to compute the set of
packages that require a default provider. Once this set has been
computed, the required default provider definitions are conjured and
prepended to the checkpoint's resource list. Each resource that
requires a default provider is then updated to refer to the default
provider for its package.
While an eval source is running, each custom resource registration,
resource read, and invoke that does not name a provider is trapped
before being returned by the source iterator. If no default provider
for the appropriate package has been registered, the eval source
synthesizes an appropriate registration, waits for it to complete, and
records the registered provider's reference. This reference is injected
into the original request, which is then processed as usual. If a
default provider was already registered, the recorded reference is
used and no new registration occurs.
### SDK Changes
These changes only expose first-class providers from the Node.JS SDK.
- A new abstract class, `ProviderResource`, can be subclassed and used
to instantiate first-class providers.
- A new field in `ResourceOptions`, `provider`, can be used to supply
a particular provider instance to manage a `CustomResource`'s CRUD
operations.
- A new type, `InvokeOptions`, can be used to specify options that
control the behavior of a call to `pulumi.runtime.invoke`. This type
includes a `provider` field that is analogous to
`ResourceOptions.provider`.
2018-08-07 02:50:29 +02:00
|
|
|
providerRef: providerRef,
|
2019-02-22 05:18:29 +01:00
|
|
|
allDirectDependencyURNs: allDirectDependencyURNs,
|
|
|
|
propertyToDirectDependencyURNs: propertyToDirectDependencyURNs,
|
2019-06-01 08:01:01 +02:00
|
|
|
aliases: aliases,
|
2019-07-12 20:12:01 +02:00
|
|
|
import: importID,
|
2018-04-05 18:48:09 +02:00
|
|
|
};
|
|
|
|
}
|
2018-01-26 00:26:39 +01:00
|
|
|
|
2019-02-22 05:18:29 +01:00
|
|
|
function addAll<T>(to: Set<T>, from: Set<T>) {
|
|
|
|
for (const val of from) {
|
|
|
|
to.add(val);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2019-03-06 02:06:57 +01:00
|
|
|
async function getAllTransitivelyReferencedCustomResourceURNs(resources: Set<Resource>) {
|
|
|
|
// Go through 'resources', but transitively walk through **Component** resources, collecting any
|
|
|
|
// of their child resources. This way, a Component acts as an aggregation really of all the
|
|
|
|
// reachable custom resources it parents. This walking will transitively walk through other
|
|
|
|
// child ComponentResources, but will stop when it hits custom resources. in other words, if we
|
|
|
|
// had:
|
|
|
|
//
|
|
|
|
// Comp1
|
|
|
|
// / \
|
|
|
|
// Cust1 Comp2
|
|
|
|
// / \
|
|
|
|
// Cust2 Cust3
|
|
|
|
// /
|
|
|
|
// Cust4
|
|
|
|
//
|
|
|
|
// Then the transitively reachable custom resources of Comp1 will be [Cust1, Cust2, Cust3]. It
|
|
|
|
// will *not* include `Cust4`.
|
|
|
|
|
|
|
|
// To do this, first we just get the transitively reachable set of resources (not diving
|
|
|
|
// into custom resources). In the above picture, if we start with 'Comp1', this will be
|
|
|
|
// [Comp1, Cust1, Comp2, Cust2, Cust3]
|
2019-12-18 00:34:30 +01:00
|
|
|
const transitivelyReachableResources = await getTransitivelyReferencedChildResourcesOfComponentResources(resources);
|
2019-03-06 02:06:57 +01:00
|
|
|
|
2019-10-15 07:08:06 +02:00
|
|
|
const transitivelyReachableCustomResources = [...transitivelyReachableResources].filter(r => CustomResource.isInstance(r));
|
2019-03-29 01:27:51 +01:00
|
|
|
const promises = transitivelyReachableCustomResources.map(r => r.urn.promise());
|
2019-03-06 02:06:57 +01:00
|
|
|
const urns = await Promise.all(promises);
|
|
|
|
return new Set<string>(urns);
|
|
|
|
}
|
2019-02-22 05:18:29 +01:00
|
|
|
|
2019-03-06 02:06:57 +01:00
|
|
|
/**
|
|
|
|
* Recursively walk the resources passed in, returning them and all resources reachable from
|
|
|
|
* [Resource.__childResources] through any **Component** resources we encounter.
|
|
|
|
*/
|
2019-12-18 00:34:30 +01:00
|
|
|
async function getTransitivelyReferencedChildResourcesOfComponentResources(resources: Set<Resource>) {
|
2019-03-06 02:06:57 +01:00
|
|
|
// Recursively walk the dependent resources through their children, adding them to the result set.
|
|
|
|
const result = new Set<Resource>();
|
2019-12-18 00:34:30 +01:00
|
|
|
await addTransitivelyReferencedChildResourcesOfComponentResources(resources, result);
|
2019-02-22 05:18:29 +01:00
|
|
|
return result;
|
|
|
|
}
|
|
|
|
|
2019-12-18 00:34:30 +01:00
|
|
|
async function addTransitivelyReferencedChildResourcesOfComponentResources(resources: Set<Resource> | undefined, result: Set<Resource>) {
|
2019-03-06 02:06:57 +01:00
|
|
|
if (resources) {
|
|
|
|
for (const resource of resources) {
|
|
|
|
if (!result.has(resource)) {
|
|
|
|
result.add(resource);
|
|
|
|
|
|
|
|
if (ComponentResource.isInstance(resource)) {
|
2019-12-18 00:34:30 +01:00
|
|
|
// This await is safe even if __isConstructed is undefined. Ensure that the
|
|
|
|
// resource has completely finished construction. That way all parent/child
|
|
|
|
// relationships will have been setup.
|
|
|
|
await resource.__data;
|
2019-03-06 02:06:57 +01:00
|
|
|
addTransitivelyReferencedChildResourcesOfComponentResources(resource.__childResources, result);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2018-11-19 17:22:55 +01:00
|
|
|
/**
|
2019-02-22 05:18:29 +01:00
|
|
|
* Gathers explicit dependent Resources from a list of Resources (possibly Promises and/or Outputs).
|
2018-11-19 17:22:55 +01:00
|
|
|
*/
|
|
|
|
async function gatherExplicitDependencies(
|
2019-02-22 05:18:29 +01:00
|
|
|
dependsOn: Input<Input<Resource>[]> | Input<Resource> | undefined): Promise<Resource[]> {
|
2018-11-19 17:22:55 +01:00
|
|
|
|
|
|
|
if (dependsOn) {
|
|
|
|
if (Array.isArray(dependsOn)) {
|
2019-02-22 05:18:29 +01:00
|
|
|
const dos: Resource[] = [];
|
2018-11-19 17:22:55 +01:00
|
|
|
for (const d of dependsOn) {
|
|
|
|
dos.push(...(await gatherExplicitDependencies(d)));
|
|
|
|
}
|
|
|
|
return dos;
|
|
|
|
} else if (dependsOn instanceof Promise) {
|
|
|
|
return gatherExplicitDependencies(await dependsOn);
|
|
|
|
} else if (Output.isInstance(dependsOn)) {
|
|
|
|
// Recursively gather dependencies, await the promise, and append the output's dependencies.
|
2019-05-10 05:59:15 +02:00
|
|
|
const dos = (dependsOn as Output<Input<Resource>[] | Input<Resource>>).apply(v => gatherExplicitDependencies(v));
|
2018-11-19 17:22:55 +01:00
|
|
|
const urns = await dos.promise();
|
2019-12-18 04:04:09 +01:00
|
|
|
const dosResources = await getAllResources(dos);
|
|
|
|
const implicits = await gatherExplicitDependencies([...dosResources]);
|
2020-05-09 02:57:08 +02:00
|
|
|
return (urns ?? []).concat(implicits);
|
2018-11-19 17:22:55 +01:00
|
|
|
} else {
|
2019-03-29 01:27:51 +01:00
|
|
|
if (!Resource.isInstance(dependsOn)) {
|
|
|
|
throw new Error("'dependsOn' was passed a value that was not a Resource.");
|
|
|
|
}
|
|
|
|
|
|
|
|
return [dependsOn];
|
2018-11-19 17:22:55 +01:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return [];
|
|
|
|
}
|
|
|
|
|
2018-04-05 18:48:09 +02:00
|
|
|
/**
|
|
|
|
* Finishes a resource creation RPC operation by resolving its outputs to the resulting RPC payload.
|
|
|
|
*/
|
|
|
|
async function resolveOutputs(res: Resource, t: string, name: string,
|
Initial support for remote component construction. (#5280)
These changes add initial support for the construction of remote
components. For now, this support is limited to the NodeJS SDK;
follow-up changes will implement support for the other SDKs.
Remote components are component resources that are constructed and
managed by plugins rather than by Pulumi programs. In this sense, they
are a bit like cloud resources, and are supported by the same
distribution and plugin loading mechanisms and described by the same
schema system.
The construction of a remote component is initiated by a
`RegisterResourceRequest` with the new `remote` field set to `true`.
When the resource monitor receives such a request, it loads the plugin
that implements the component resource and calls the `Construct`
method added to the resource provider interface as part of these
changes. This method accepts the information necessary to construct the
component and its children: the component's name, type, resource
options, inputs, and input dependencies. It is responsible for
dispatching to the appropriate component factory to create the
component, then returning its URN, resolved output properties, and
output property dependencies. The dependency information is necessary to
support features such as delete-before-replace, which rely on precise
dependency information for custom resources.
These changes also add initial support for more conveniently
implementing resource providers in NodeJS. The interface used to
implement such a provider is similar to the dynamic provider interface
(and may be unified with that interface in the future).
An example of a NodeJS program constructing a remote component resource
also implemented in NodeJS can be found in
`tests/construct_component/nodejs`.
This is the core of #2430.
2020-09-08 04:33:55 +02:00
|
|
|
props: Inputs, outputs: any, deps: Record<string, Resource[]>,
|
2020-09-24 04:06:26 +02:00
|
|
|
resolvers: OutputResolvers, err?: Error): Promise<void> {
|
Initial support for remote component construction. (#5280)
These changes add initial support for the construction of remote
components. For now, this support is limited to the NodeJS SDK;
follow-up changes will implement support for the other SDKs.
Remote components are component resources that are constructed and
managed by plugins rather than by Pulumi programs. In this sense, they
are a bit like cloud resources, and are supported by the same
distribution and plugin loading mechanisms and described by the same
schema system.
The construction of a remote component is initiated by a
`RegisterResourceRequest` with the new `remote` field set to `true`.
When the resource monitor receives such a request, it loads the plugin
that implements the component resource and calls the `Construct`
method added to the resource provider interface as part of these
changes. This method accepts the information necessary to construct the
component and its children: the component's name, type, resource
options, inputs, and input dependencies. It is responsible for
dispatching to the appropriate component factory to create the
component, then returning its URN, resolved output properties, and
output property dependencies. The dependency information is necessary to
support features such as delete-before-replace, which rely on precise
dependency information for custom resources.
These changes also add initial support for more conveniently
implementing resource providers in NodeJS. The interface used to
implement such a provider is similar to the dynamic provider interface
(and may be unified with that interface in the future).
An example of a NodeJS program constructing a remote component resource
also implemented in NodeJS can be found in
`tests/construct_component/nodejs`.
This is the core of #2430.
2020-09-08 04:33:55 +02:00
|
|
|
|
2018-04-05 18:48:09 +02:00
|
|
|
// Produce a combined set of property states, starting with inputs and then applying
|
|
|
|
// outputs. If the same property exists in the inputs and outputs states, the output wins.
|
|
|
|
const allProps: Record<string, any> = {};
|
|
|
|
if (outputs) {
|
|
|
|
Object.assign(allProps, deserializeProperties(outputs));
|
|
|
|
}
|
2018-02-05 23:44:23 +01:00
|
|
|
|
2018-05-23 23:47:40 +02:00
|
|
|
const label = `resource:${name}[${t}]#...`;
|
2019-08-05 21:44:04 +02:00
|
|
|
if (!isDryRun() || isLegacyApplyEnabled()) {
|
|
|
|
for (const key of Object.keys(props)) {
|
|
|
|
if (!allProps.hasOwnProperty(key)) {
|
|
|
|
// input prop the engine didn't give us a final value for. Just use the value passed into the resource
|
|
|
|
// after round-tripping it through serialization. We do the round-tripping primarily s.t. we ensure that
|
|
|
|
// Output values are handled properly w.r.t. unknowns.
|
|
|
|
const inputProp = await serializeProperty(label, props[key], new Set());
|
|
|
|
if (inputProp === undefined) {
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
allProps[key] = deserializeProperty(inputProp);
|
2018-02-05 23:44:23 +01:00
|
|
|
}
|
2018-04-05 18:48:09 +02:00
|
|
|
}
|
|
|
|
}
|
2018-02-05 23:44:23 +01:00
|
|
|
|
2020-09-24 04:06:26 +02:00
|
|
|
resolveProperties(res, resolvers, t, name, allProps, deps, err);
|
2017-11-29 20:27:32 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* registerResourceOutputs completes the resource registration, attaching an optional set of computed outputs.
|
|
|
|
*/
|
2018-09-02 21:15:58 +02:00
|
|
|
export function registerResourceOutputs(res: Resource, outputs: Inputs | Promise<Inputs> | Output<Inputs>) {
|
2018-01-25 22:34:21 +01:00
|
|
|
// Now run the operation. Note that we explicitly do not serialize output registration with
|
|
|
|
// respect to other resource operations, as outputs may depend on properties of other resources
|
|
|
|
// that will not resolve until later turns. This would create a circular promise chain that can
|
|
|
|
// never resolve.
|
2017-11-29 20:27:32 +01:00
|
|
|
const opLabel = `monitor.registerResourceOutputs(...)`;
|
2017-12-15 00:22:01 +01:00
|
|
|
runAsyncResourceOp(opLabel, async () => {
|
2018-09-02 19:38:35 +02:00
|
|
|
// The registration could very well still be taking place, so we will need to wait for its URN.
|
|
|
|
// Additionally, the output properties might have come from other resources, so we must await those too.
|
2018-02-05 23:44:23 +01:00
|
|
|
const urn = await res.urn.promise();
|
2019-02-22 05:18:29 +01:00
|
|
|
const resolved = await serializeProperties(opLabel, { outputs });
|
2018-09-02 19:38:35 +02:00
|
|
|
const outputsObj = gstruct.Struct.fromJavaScript(resolved.outputs);
|
2017-11-29 20:27:32 +01:00
|
|
|
log.debug(`RegisterResourceOutputs RPC prepared: urn=${urn}` +
|
|
|
|
(excessiveDebugOutput ? `, outputs=${JSON.stringify(outputsObj)}` : ``));
|
|
|
|
|
|
|
|
// Fetch the monitor and make an RPC request.
|
2019-04-17 07:20:01 +02:00
|
|
|
const monitor = getMonitor();
|
|
|
|
if (monitor) {
|
|
|
|
const req = new resproto.RegisterResourceOutputsRequest();
|
|
|
|
req.setUrn(urn);
|
|
|
|
req.setOutputs(outputsObj);
|
|
|
|
|
|
|
|
const label = `monitor.registerResourceOutputs(${urn}, ...)`;
|
|
|
|
await debuggablePromise(new Promise((resolve, reject) =>
|
|
|
|
(monitor as any).registerResourceOutputs(req, (err: grpc.ServiceError, innerResponse: any) => {
|
2019-10-15 07:08:06 +02:00
|
|
|
log.debug(`RegisterResourceOutputs RPC finished: urn=${urn}; ` +
|
2019-04-17 07:20:01 +02:00
|
|
|
`err: ${err}, resp: ${innerResponse}`);
|
|
|
|
if (err) {
|
|
|
|
// If the monitor is unavailable, it is in the process of shutting down or has already
|
|
|
|
// shut down. Don't emit an error and don't do any more RPCs, just exit.
|
2020-09-24 04:06:26 +02:00
|
|
|
if (err.code === grpc.status.UNAVAILABLE || err.code === grpc.status.CANCELLED) {
|
|
|
|
terminateRpcs();
|
|
|
|
err.message = "Resource monitor is terminating";
|
2019-04-17 07:20:01 +02:00
|
|
|
}
|
2018-05-17 00:37:34 +02:00
|
|
|
|
2019-04-17 07:20:01 +02:00
|
|
|
reject(err);
|
|
|
|
}
|
|
|
|
else {
|
2020-09-24 04:06:26 +02:00
|
|
|
log.debug(`RegisterResourceOutputs RPC finished: urn=${urn}; ` +
|
|
|
|
`err: ${err}, resp: ${innerResponse}`);
|
2019-04-17 07:20:01 +02:00
|
|
|
resolve();
|
|
|
|
}
|
|
|
|
})), label);
|
2019-10-15 07:08:06 +02:00
|
|
|
}
|
2017-12-15 01:24:08 +01:00
|
|
|
}, false);
|
Bring back component outputs
This change brings back component outputs to the overall system again.
In doing so, it generally overhauls the way we do resource RPCs a bit:
* Instead of RegisterResource and CompleteResource, we call these
BeginRegisterResource and EndRegisterResource, which begins to model
these as effectively "asynchronous" resource requests. This should also
help with parallelism (https://github.com/pulumi/pulumi/issues/106).
* Flip the CLI/engine a little on its head. Rather than it driving the
planning and deployment process, we move more to a model where it
simply observes it. This is done by implementing an event handler
interface with three events: OnResourceStepPre, OnResourceStepPost,
and OnResourceComplete. The first two are invoked immediately before
and after any step operation, and the latter is invoked whenever a
EndRegisterResource comes in. The reason for the asymmetry here is
that the checkpointing logic in the deployment engine is largely
untouched (intentionally, as this is a sensitive part of the system),
and so the "begin"/"end" nature doesn't flow through faithfully.
* Also make the engine more event-oriented in its terminology and the
way it handles the incoming BeginRegisterResource and
EndRegisterResource events from the language host. This is the first
step down a long road of incrementally refactoring the engine to work
this way, a necessary prerequisite for parallelism.
2017-11-29 16:42:14 +01:00
|
|
|
}
|
|
|
|
|
2019-04-19 00:25:15 +02:00
|
|
|
function isAny(o: any): o is any {
|
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
|
|
|
/**
|
|
|
|
* listResourceOutputs returns the resource outputs (if any) for a stack, or an error if the stack
|
|
|
|
* cannot be found. Resources are retrieved from the latest stack snapshot, which may include
|
|
|
|
* ongoing updates.
|
|
|
|
*
|
|
|
|
* @param stackName Name of stack to retrieve resource outputs for. Defaults to the current stack.
|
|
|
|
* @param typeFilter A [type
|
|
|
|
* guard](https://www.typescriptlang.org/docs/handbook/advanced-types.html#user-defined-type-guards)
|
|
|
|
* that specifies which resource types to list outputs of.
|
|
|
|
*
|
|
|
|
* @example
|
|
|
|
* const buckets = pulumi.runtime.listResourceOutput(aws.s3.Bucket.isInstance);
|
|
|
|
*/
|
|
|
|
export function listResourceOutputs<U extends Resource>(
|
|
|
|
typeFilter?: (o: any) => o is U,
|
|
|
|
stackName?: string,
|
|
|
|
): query.AsyncQueryable<ResolvedResource<U>> {
|
|
|
|
if (typeFilter === undefined) {
|
|
|
|
typeFilter = isAny;
|
|
|
|
}
|
|
|
|
|
|
|
|
return query
|
|
|
|
.from(
|
|
|
|
invoke("pulumi:pulumi:readStackResourceOutputs", {
|
|
|
|
stackName: stackName || getStack(),
|
2019-08-08 21:11:46 +02:00
|
|
|
}).then<any[]>(({ outputs }) => utils.values(outputs)),
|
2019-04-19 00:25:15 +02:00
|
|
|
)
|
|
|
|
.map<ResolvedResource<U>>(({ type: typ, outputs }) => {
|
|
|
|
return { ...outputs, __pulumiType: typ };
|
|
|
|
})
|
|
|
|
.filter(typeFilter);
|
|
|
|
}
|
|
|
|
|
|
|
|
|
Bring back component outputs
This change brings back component outputs to the overall system again.
In doing so, it generally overhauls the way we do resource RPCs a bit:
* Instead of RegisterResource and CompleteResource, we call these
BeginRegisterResource and EndRegisterResource, which begins to model
these as effectively "asynchronous" resource requests. This should also
help with parallelism (https://github.com/pulumi/pulumi/issues/106).
* Flip the CLI/engine a little on its head. Rather than it driving the
planning and deployment process, we move more to a model where it
simply observes it. This is done by implementing an event handler
interface with three events: OnResourceStepPre, OnResourceStepPost,
and OnResourceComplete. The first two are invoked immediately before
and after any step operation, and the latter is invoked whenever a
EndRegisterResource comes in. The reason for the asymmetry here is
that the checkpointing logic in the deployment engine is largely
untouched (intentionally, as this is a sensitive part of the system),
and so the "begin"/"end" nature doesn't flow through faithfully.
* Also make the engine more event-oriented in its terminology and the
way it handles the incoming BeginRegisterResource and
EndRegisterResource events from the language host. This is the first
step down a long road of incrementally refactoring the engine to work
this way, a necessary prerequisite for parallelism.
2017-11-29 16:42:14 +01:00
|
|
|
/**
|
|
|
|
* resourceChain is used to serialize all resource requests. If we don't do this, all resource operations will be
|
|
|
|
* entirely asynchronous, meaning the dataflow graph that results will determine ordering of operations. This
|
|
|
|
* causes problems with some resource providers, so for now we will serialize all of them. The issue
|
|
|
|
* pulumi/pulumi#335 tracks coming up with a long-term solution here.
|
|
|
|
*/
|
|
|
|
let resourceChain: Promise<void> = Promise.resolve();
|
|
|
|
let resourceChainLabel: string | undefined = undefined;
|
|
|
|
|
|
|
|
// runAsyncResourceOp runs an asynchronous resource operation, possibly serializing it as necessary.
|
2017-12-15 01:24:08 +01:00
|
|
|
function runAsyncResourceOp(label: string, callback: () => Promise<void>, serial?: boolean): void {
|
Bring back component outputs
This change brings back component outputs to the overall system again.
In doing so, it generally overhauls the way we do resource RPCs a bit:
* Instead of RegisterResource and CompleteResource, we call these
BeginRegisterResource and EndRegisterResource, which begins to model
these as effectively "asynchronous" resource requests. This should also
help with parallelism (https://github.com/pulumi/pulumi/issues/106).
* Flip the CLI/engine a little on its head. Rather than it driving the
planning and deployment process, we move more to a model where it
simply observes it. This is done by implementing an event handler
interface with three events: OnResourceStepPre, OnResourceStepPost,
and OnResourceComplete. The first two are invoked immediately before
and after any step operation, and the latter is invoked whenever a
EndRegisterResource comes in. The reason for the asymmetry here is
that the checkpointing logic in the deployment engine is largely
untouched (intentionally, as this is a sensitive part of the system),
and so the "begin"/"end" nature doesn't flow through faithfully.
* Also make the engine more event-oriented in its terminology and the
way it handles the incoming BeginRegisterResource and
EndRegisterResource events from the language host. This is the first
step down a long road of incrementally refactoring the engine to work
this way, a necessary prerequisite for parallelism.
2017-11-29 16:42:14 +01:00
|
|
|
// Serialize the invocation if necessary.
|
2017-12-15 01:24:08 +01:00
|
|
|
if (serial === undefined) {
|
|
|
|
serial = serialize();
|
|
|
|
}
|
2020-09-24 04:06:26 +02:00
|
|
|
const resourceOp: Promise<void> = suppressUnhandledGrpcRejections(debuggablePromise(resourceChain.then(async () => {
|
Bring back component outputs
This change brings back component outputs to the overall system again.
In doing so, it generally overhauls the way we do resource RPCs a bit:
* Instead of RegisterResource and CompleteResource, we call these
BeginRegisterResource and EndRegisterResource, which begins to model
these as effectively "asynchronous" resource requests. This should also
help with parallelism (https://github.com/pulumi/pulumi/issues/106).
* Flip the CLI/engine a little on its head. Rather than it driving the
planning and deployment process, we move more to a model where it
simply observes it. This is done by implementing an event handler
interface with three events: OnResourceStepPre, OnResourceStepPost,
and OnResourceComplete. The first two are invoked immediately before
and after any step operation, and the latter is invoked whenever a
EndRegisterResource comes in. The reason for the asymmetry here is
that the checkpointing logic in the deployment engine is largely
untouched (intentionally, as this is a sensitive part of the system),
and so the "begin"/"end" nature doesn't flow through faithfully.
* Also make the engine more event-oriented in its terminology and the
way it handles the incoming BeginRegisterResource and
EndRegisterResource events from the language host. This is the first
step down a long road of incrementally refactoring the engine to work
this way, a necessary prerequisite for parallelism.
2017-11-29 16:42:14 +01:00
|
|
|
if (serial) {
|
|
|
|
resourceChainLabel = label;
|
|
|
|
log.debug(`Resource RPC serialization requested: ${label} is current`);
|
|
|
|
}
|
|
|
|
return callback();
|
2020-09-24 04:06:26 +02:00
|
|
|
}), label + "-initial"));
|
2017-09-16 01:38:52 +02:00
|
|
|
|
2017-12-15 00:22:01 +01:00
|
|
|
// Ensure the process won't exit until this RPC call finishes and resolve it when appropriate.
|
2017-10-19 00:03:56 +02:00
|
|
|
const done: () => void = rpcKeepAlive();
|
2018-11-25 03:57:17 +01:00
|
|
|
const finalOp: Promise<void> = debuggablePromise(
|
|
|
|
resourceOp.then(() => { done(); }, () => { done(); }),
|
|
|
|
label + "-final");
|
2017-12-15 00:22:01 +01:00
|
|
|
|
|
|
|
// Set up another promise that propagates the error, if any, so that it triggers unhandled rejection logic.
|
|
|
|
resourceOp.catch((err) => Promise.reject(err));
|
Eliminate Computed/Property in favor of Promises
As part of pulumi/pulumi-fabric#331, we've been exploring just using
undefined to indicate that a property value is absent during planning.
We also considered blocking the message loop to simplify the overall
programming model, so that all asynchrony is hidden.
It turns out ThereBeDragons :dragon_face: anytime you try to block the
message loop. So, we aren't quite sure about that bit.
But the part we are convicted about is that this Computed/Property
model is far too complex. Furthermore, it's very close to promises, and
yet frustratingly so far away. Indeed, the original thinking in
pulumi/pulumi-fabric#271 was simply to use promises, but we wanted to
encourage dataflow styles, rather than control flow. But we muddied up
our thinking by worrying about awaiting a promise that would never resolve.
It turns out we can achieve a middle ground: resolve planning promises to
undefined, so that they don't lead to hangs, but still use promises so
that asynchrony is explicit in the system. This also avoids blocking the
message loop. Who knows, this may actually be a fine final destination.
2017-09-20 16:40:09 +02:00
|
|
|
|
2017-09-16 01:38:52 +02:00
|
|
|
// If serialization is requested, wait for the prior resource operation to finish before we proceed, serializing
|
|
|
|
// them, and make this the current resource operation so that everybody piles up on it.
|
Bring back component outputs
This change brings back component outputs to the overall system again.
In doing so, it generally overhauls the way we do resource RPCs a bit:
* Instead of RegisterResource and CompleteResource, we call these
BeginRegisterResource and EndRegisterResource, which begins to model
these as effectively "asynchronous" resource requests. This should also
help with parallelism (https://github.com/pulumi/pulumi/issues/106).
* Flip the CLI/engine a little on its head. Rather than it driving the
planning and deployment process, we move more to a model where it
simply observes it. This is done by implementing an event handler
interface with three events: OnResourceStepPre, OnResourceStepPost,
and OnResourceComplete. The first two are invoked immediately before
and after any step operation, and the latter is invoked whenever a
EndRegisterResource comes in. The reason for the asymmetry here is
that the checkpointing logic in the deployment engine is largely
untouched (intentionally, as this is a sensitive part of the system),
and so the "begin"/"end" nature doesn't flow through faithfully.
* Also make the engine more event-oriented in its terminology and the
way it handles the incoming BeginRegisterResource and
EndRegisterResource events from the language host. This is the first
step down a long road of incrementally refactoring the engine to work
this way, a necessary prerequisite for parallelism.
2017-11-29 16:42:14 +01:00
|
|
|
if (serial) {
|
Eliminate Computed/Property in favor of Promises
As part of pulumi/pulumi-fabric#331, we've been exploring just using
undefined to indicate that a property value is absent during planning.
We also considered blocking the message loop to simplify the overall
programming model, so that all asynchrony is hidden.
It turns out ThereBeDragons :dragon_face: anytime you try to block the
message loop. So, we aren't quite sure about that bit.
But the part we are convicted about is that this Computed/Property
model is far too complex. Furthermore, it's very close to promises, and
yet frustratingly so far away. Indeed, the original thinking in
pulumi/pulumi-fabric#271 was simply to use promises, but we wanted to
encourage dataflow styles, rather than control flow. But we muddied up
our thinking by worrying about awaiting a promise that would never resolve.
It turns out we can achieve a middle ground: resolve planning promises to
undefined, so that they don't lead to hangs, but still use promises so
that asynchrony is explicit in the system. This also avoids blocking the
message loop. Who knows, this may actually be a fine final destination.
2017-09-20 16:40:09 +02:00
|
|
|
resourceChain = finalOp;
|
2017-10-31 14:52:42 +01:00
|
|
|
if (resourceChainLabel) {
|
Bring back component outputs
This change brings back component outputs to the overall system again.
In doing so, it generally overhauls the way we do resource RPCs a bit:
* Instead of RegisterResource and CompleteResource, we call these
BeginRegisterResource and EndRegisterResource, which begins to model
these as effectively "asynchronous" resource requests. This should also
help with parallelism (https://github.com/pulumi/pulumi/issues/106).
* Flip the CLI/engine a little on its head. Rather than it driving the
planning and deployment process, we move more to a model where it
simply observes it. This is done by implementing an event handler
interface with three events: OnResourceStepPre, OnResourceStepPost,
and OnResourceComplete. The first two are invoked immediately before
and after any step operation, and the latter is invoked whenever a
EndRegisterResource comes in. The reason for the asymmetry here is
that the checkpointing logic in the deployment engine is largely
untouched (intentionally, as this is a sensitive part of the system),
and so the "begin"/"end" nature doesn't flow through faithfully.
* Also make the engine more event-oriented in its terminology and the
way it handles the incoming BeginRegisterResource and
EndRegisterResource events from the language host. This is the first
step down a long road of incrementally refactoring the engine to work
this way, a necessary prerequisite for parallelism.
2017-11-29 16:42:14 +01:00
|
|
|
log.debug(`Resource RPC serialization requested: ${label} is behind ${resourceChainLabel}`);
|
2017-10-31 14:52:42 +01:00
|
|
|
}
|
2017-09-16 01:38:52 +02:00
|
|
|
}
|
Implement initial Lumi-as-a-library
This is the initial step towards redefining Lumi as a library that runs
atop vanilla Node.js/V8, rather than as its own runtime.
This change is woefully incomplete but this includes some of the more
stable pieces of my current work-in-progress.
The new structure is that within the sdk/ directory we will have a client
library per language. This client library contains the object model for
Lumi (resources, properties, assets, config, etc), in addition to the
"language runtime host" components required to interoperate with the
Lumi resource monitor. This resource monitor is effectively what we call
"Lumi" today, in that it's the thing orchestrating plans and deployments.
Inside the sdk/ directory, you will find nodejs/, the Node.js client
library, alongside proto/, the definitions for RPC interop between the
different pieces of the system. This includes existing RPC definitions
for resource providers, etc., in addition to the new ones for hosting
different language runtimes from within Lumi.
These new interfaces are surprisingly simple. There is effectively a
bidirectional RPC channel between the Lumi resource monitor, represented
by the lumirpc.ResourceMonitor interface, and each language runtime,
represented by the lumirpc.LanguageRuntime interface.
The overall orchestration goes as follows:
1) Lumi decides it needs to run a program written in language X, so
it dynamically loads the language runtime plugin for language X.
2) Lumi passes that runtime a loopback address to its ResourceMonitor
service, while language X will publish a connection back to its
LanguageRuntime service, which Lumi will talk to.
3) Lumi then invokes LanguageRuntime.Run, passing information like
the desired working directory, program name, arguments, and optional
configuration variables to make available to the program.
4) The language X runtime receives this, unpacks it and sets up the
necessary context, and then invokes the program. The program then
calls into Lumi object model abstractions that internally communicate
back to Lumi using the ResourceMonitor interface.
5) The key here is ResourceMonitor.NewResource, which Lumi uses to
serialize state about newly allocated resources. Lumi receives these
and registers them as part of the plan, doing the usual diffing, etc.,
to decide how to proceed. This interface is perhaps one of the
most subtle parts of the new design, as it necessitates the use of
promises internally to allow parallel evaluation of the resource plan,
letting dataflow determine the available concurrency.
6) The program exits, and Lumi continues on its merry way. If the program
fails, the RunResponse will include information about the failure.
Due to (5), all properties on resources are now instances of a new
Property<T> type. A Property<T> is just a thin wrapper over a T, but it
encodes the special properties of Lumi resource properties. Namely, it
is possible to create one out of a T, other Property<T>, Promise<T>, or
to freshly allocate one. In all cases, the Property<T> does not "settle"
until its final state is known. This cannot occur before the deployment
actually completes, and so in general it's not safe to depend on concrete
resolutions of values (unlike ordinary Promise<T>s which are usually
expected to resolve). As a result, all derived computations are meant to
use the `then` function (as in `someValue.then(v => v+x)`).
Although this change includes tests that may be run in isolation to test
the various RPC interactions, we are nowhere near finished. The remaining
work primarily boils down to three things:
1) Wiring all of this up to the Lumi code.
2) Fixing the handful of known loose ends required to make this work,
primarily around the serialization of properties (waiting on
unresolved ones, serializing assets properly, etc).
3) Implementing lambda closure serialization as a native extension.
This ongoing work is part of pulumi/pulumi-fabric#311.
2017-08-26 21:07:54 +02:00
|
|
|
}
|