pulumi/sdk/nodejs/runtime/resource.ts

790 lines
35 KiB
TypeScript
Raw Normal View History

2018-05-22 21:43:36 +02:00
// Copyright 2016-2018, Pulumi Corporation.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
Implement initial Lumi-as-a-library This is the initial step towards redefining Lumi as a library that runs atop vanilla Node.js/V8, rather than as its own runtime. This change is woefully incomplete but this includes some of the more stable pieces of my current work-in-progress. The new structure is that within the sdk/ directory we will have a client library per language. This client library contains the object model for Lumi (resources, properties, assets, config, etc), in addition to the "language runtime host" components required to interoperate with the Lumi resource monitor. This resource monitor is effectively what we call "Lumi" today, in that it's the thing orchestrating plans and deployments. Inside the sdk/ directory, you will find nodejs/, the Node.js client library, alongside proto/, the definitions for RPC interop between the different pieces of the system. This includes existing RPC definitions for resource providers, etc., in addition to the new ones for hosting different language runtimes from within Lumi. These new interfaces are surprisingly simple. There is effectively a bidirectional RPC channel between the Lumi resource monitor, represented by the lumirpc.ResourceMonitor interface, and each language runtime, represented by the lumirpc.LanguageRuntime interface. The overall orchestration goes as follows: 1) Lumi decides it needs to run a program written in language X, so it dynamically loads the language runtime plugin for language X. 2) Lumi passes that runtime a loopback address to its ResourceMonitor service, while language X will publish a connection back to its LanguageRuntime service, which Lumi will talk to. 3) Lumi then invokes LanguageRuntime.Run, passing information like the desired working directory, program name, arguments, and optional configuration variables to make available to the program. 4) The language X runtime receives this, unpacks it and sets up the necessary context, and then invokes the program. The program then calls into Lumi object model abstractions that internally communicate back to Lumi using the ResourceMonitor interface. 5) The key here is ResourceMonitor.NewResource, which Lumi uses to serialize state about newly allocated resources. Lumi receives these and registers them as part of the plan, doing the usual diffing, etc., to decide how to proceed. This interface is perhaps one of the most subtle parts of the new design, as it necessitates the use of promises internally to allow parallel evaluation of the resource plan, letting dataflow determine the available concurrency. 6) The program exits, and Lumi continues on its merry way. If the program fails, the RunResponse will include information about the failure. Due to (5), all properties on resources are now instances of a new Property<T> type. A Property<T> is just a thin wrapper over a T, but it encodes the special properties of Lumi resource properties. Namely, it is possible to create one out of a T, other Property<T>, Promise<T>, or to freshly allocate one. In all cases, the Property<T> does not "settle" until its final state is known. This cannot occur before the deployment actually completes, and so in general it's not safe to depend on concrete resolutions of values (unlike ordinary Promise<T>s which are usually expected to resolve). As a result, all derived computations are meant to use the `then` function (as in `someValue.then(v => v+x)`). Although this change includes tests that may be run in isolation to test the various RPC interactions, we are nowhere near finished. The remaining work primarily boils down to three things: 1) Wiring all of this up to the Lumi code. 2) Fixing the handful of known loose ends required to make this work, primarily around the serialization of properties (waiting on unresolved ones, serializing assets properly, etc). 3) Implementing lambda closure serialization as a native extension. This ongoing work is part of pulumi/pulumi-fabric#311.
2017-08-26 21:07:54 +02:00
import * as grpc from "@grpc/grpc-js";
import * as query from "@pulumi/query";
import * as log from "../log";
import * as utils from "../utils";
import { getAllResources, Input, Inputs, Output, output } from "../output";
import { ResolvedResource } from "../queryable";
import {
ComponentResource,
Support aliases for renaming, re-typing, or re-parenting resources (#2774) Adds a new resource option `aliases` which can be used to rename a resource. When making a breaking change to the name or type of a resource or component, the old name can be added to the list of `aliases` for a resource to ensure that existing resources will be migrated to the new name instead of being deleted and replaced with the new named resource. There are two key places this change is implemented. The first is the step generator in the engine. When computing whether there is an old version of a registered resource, we now take into account the aliases specified on the registered resource. That is, we first look up the resource by its new URN in the old state, and then by any aliases provided (in order). This can allow the resource to be matched as a (potential) update to an existing resource with a different URN. The second is the core `Resource` constructor in the JavaScript (and soon Python) SDKs. This change ensures that when a parent resource is aliased, that all children implicitly inherit corresponding aliases. It is similar to how many other resource options are "inherited" implicitly from the parent. Four specific scenarios are explicitly tested as part of this PR: 1. Renaming a resource 2. Adopting a resource into a component (as the owner of both component and consumption codebases) 3. Renaming a component instance (as the owner of the consumption codebase without changes to the component) 4. Changing the type of a component (as the owner of the component codebase without changes to the consumption codebase) 4. Combining (1) and (3) to make both changes to a resource at the same time
2019-06-01 08:01:01 +02:00
createUrn,
CustomResource,
CustomResourceOptions,
ID,
ProviderResource,
Resource,
ResourceOptions,
URN,
} from "../resource";
import { debuggablePromise } from "./debuggable";
import { invoke } from "./invoke";
import {
deserializeProperties,
Discriminate unknown values in the JS runtime. (#1414) These changes add support for distinguishing an output property with an unknown value from an output property with a known value that is undefined. In a broad sense, the Pulumi property type system is just JSON with the addition of unknown values. Notably absent, however, are undefined values. As it stands, our marshalers between JavaScript and Pulumi property values treat all undefined JavaScript values as unknown Pulumi values. Unfortunately, this conflates two very different concepts: unknown Pulumi values are intended to represent values of output properties that are unknown at time of preview, _not_ values that are known but undefined. This results in difficulty reasoning about when transforms are run on output properties as well as confusing output in the `diff` view of Pulumi preview (user-specifed undefined values are rendered as unknown values). As it turns out, we already have a way to decide whether or not an Output value is known or not: Output.performApply. These changes rename this property to `isKnown`, clarify its meaning, and take advantage of the result to decide whether or not an Output value should marshal as an unknown Pulumi value. This also allowed these changes to improve the serialization of undefined object keys and array elements s.t. we better match JavaScript to JSON serialization behavior (undefined object keys are omitted; undefined array elements are marshaled as `null`). Fixes https://github.com/pulumi/pulumi-cloud/issues/483.
2018-05-23 23:47:40 +02:00
deserializeProperty,
OutputResolvers,
resolveProperties,
serializeProperties,
serializeProperty,
serializeResourceProperties,
suppressUnhandledGrpcRejections,
transferProperties,
} from "./rpc";
import {
excessiveDebugOutput,
getMonitor,
getProject,
getRootResource,
getStack,
isDryRun,
isLegacyApplyEnabled,
rpcKeepAlive,
serialize,
terminateRpcs,
} from "./settings";
Implement initial Lumi-as-a-library This is the initial step towards redefining Lumi as a library that runs atop vanilla Node.js/V8, rather than as its own runtime. This change is woefully incomplete but this includes some of the more stable pieces of my current work-in-progress. The new structure is that within the sdk/ directory we will have a client library per language. This client library contains the object model for Lumi (resources, properties, assets, config, etc), in addition to the "language runtime host" components required to interoperate with the Lumi resource monitor. This resource monitor is effectively what we call "Lumi" today, in that it's the thing orchestrating plans and deployments. Inside the sdk/ directory, you will find nodejs/, the Node.js client library, alongside proto/, the definitions for RPC interop between the different pieces of the system. This includes existing RPC definitions for resource providers, etc., in addition to the new ones for hosting different language runtimes from within Lumi. These new interfaces are surprisingly simple. There is effectively a bidirectional RPC channel between the Lumi resource monitor, represented by the lumirpc.ResourceMonitor interface, and each language runtime, represented by the lumirpc.LanguageRuntime interface. The overall orchestration goes as follows: 1) Lumi decides it needs to run a program written in language X, so it dynamically loads the language runtime plugin for language X. 2) Lumi passes that runtime a loopback address to its ResourceMonitor service, while language X will publish a connection back to its LanguageRuntime service, which Lumi will talk to. 3) Lumi then invokes LanguageRuntime.Run, passing information like the desired working directory, program name, arguments, and optional configuration variables to make available to the program. 4) The language X runtime receives this, unpacks it and sets up the necessary context, and then invokes the program. The program then calls into Lumi object model abstractions that internally communicate back to Lumi using the ResourceMonitor interface. 5) The key here is ResourceMonitor.NewResource, which Lumi uses to serialize state about newly allocated resources. Lumi receives these and registers them as part of the plan, doing the usual diffing, etc., to decide how to proceed. This interface is perhaps one of the most subtle parts of the new design, as it necessitates the use of promises internally to allow parallel evaluation of the resource plan, letting dataflow determine the available concurrency. 6) The program exits, and Lumi continues on its merry way. If the program fails, the RunResponse will include information about the failure. Due to (5), all properties on resources are now instances of a new Property<T> type. A Property<T> is just a thin wrapper over a T, but it encodes the special properties of Lumi resource properties. Namely, it is possible to create one out of a T, other Property<T>, Promise<T>, or to freshly allocate one. In all cases, the Property<T> does not "settle" until its final state is known. This cannot occur before the deployment actually completes, and so in general it's not safe to depend on concrete resolutions of values (unlike ordinary Promise<T>s which are usually expected to resolve). As a result, all derived computations are meant to use the `then` function (as in `someValue.then(v => v+x)`). Although this change includes tests that may be run in isolation to test the various RPC interactions, we are nowhere near finished. The remaining work primarily boils down to three things: 1) Wiring all of this up to the Lumi code. 2) Fixing the handful of known loose ends required to make this work, primarily around the serialization of properties (waiting on unresolved ones, serializing assets properly, etc). 3) Implementing lambda closure serialization as a native extension. This ongoing work is part of pulumi/pulumi-fabric#311.
2017-08-26 21:07:54 +02:00
const gstruct = require("google-protobuf/google/protobuf/struct_pb.js");
const providerproto = require("../proto/provider_pb.js");
Switch to parent pointers; display components nicely This change switches from child lists to parent pointers, in the way resource ancestries are represented. This cleans up a fair bit of the old parenting logic, including all notion of ambient parent scopes (and will notably address pulumi/pulumi#435). This lets us show a more parent/child display in the output when doing planning and updating. For instance, here is an update of a lambda's text, which is logically part of a cloud timer: * cloud:timer:Timer: (same) [urn=urn:pulumi:malta::lm-cloud::cloud:timer:Timer::lm-cts-malta-job-CleanSnapshots] * cloud:function:Function: (same) [urn=urn:pulumi:malta::lm-cloud::cloud:function:Function::lm-cts-malta-job-CleanSnapshots] * aws:serverless:Function: (same) [urn=urn:pulumi:malta::lm-cloud::aws:serverless:Function::lm-cts-malta-job-CleanSnapshots] ~ aws:lambda/function:Function: (modify) [id=lm-cts-malta-job-CleanSnapshots-fee4f3bf41280741] [urn=urn:pulumi:malta::lm-cloud::aws:lambda/function:Function::lm-cts-malta-job-CleanSnapshots] - code : archive(assets:2092f44) { // etc etc etc Note that we still get walls of text, but this will be actually quite nice when combined with pulumi/pulumi#454. I've also suppressed printing properties that didn't change during updates when --detailed was not passed, and also suppressed empty strings and zero-length arrays (since TF uses these as defaults in many places and it just makes creation and deletion quite verbose). Note that this is a far cry from everything we can possibly do here as part of pulumi/pulumi#340 (and even pulumi/pulumi#417). But it's a good start towards taming some of our output spew.
2017-11-17 03:21:41 +01:00
const resproto = require("../proto/resource_pb.js");
2017-09-09 22:49:50 +02:00
interface ResourceResolverOperation {
// A resolver for a resource's URN.
resolveURN: (urn: URN, err?: Error) => void;
// A resolver for a resource's ID (for custom resources only).
resolveID: ((v: ID, performApply: boolean, err?: Error) => void) | undefined;
// A collection of resolvers for a resource's properties.
resolvers: OutputResolvers;
// A parent URN, fully resolved, if any.
parentURN: URN | undefined;
Implement first-class providers. (#1695) ### First-Class Providers These changes implement support for first-class providers. First-class providers are provider plugins that are exposed as resources via the Pulumi programming model so that they may be explicitly and multiply instantiated. Each instance of a provider resource may be configured differently, and configuration parameters may be source from the outputs of other resources. ### Provider Plugin Changes In order to accommodate the need to verify and diff provider configuration and configure providers without complete configuration information, these changes adjust the high-level provider plugin interface. Two new methods for validating a provider's configuration and diffing changes to the same have been added (`CheckConfig` and `DiffConfig`, respectively), and the type of the configuration bag accepted by `Configure` has been changed to a `PropertyMap`. These changes have not yet been reflected in the provider plugin gRPC interface. We will do this in a set of follow-up changes. Until then, these methods are implemented by adapters: - `CheckConfig` validates that all configuration parameters are string or unknown properties. This is necessary because existing plugins only accept string-typed configuration values. - `DiffConfig` either returns "never replace" if all configuration values are known or "must replace" if any configuration value is unknown. The justification for this behavior is given [here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106) - `Configure` converts the config bag to a legacy config map and configures the provider plugin if all config values are known. If any config value is unknown, the underlying plugin is not configured and the provider may only perform `Check`, `Read`, and `Invoke`, all of which return empty results. We justify this behavior becuase it is only possible during a preview and provides the best experience we can manage with the existing gRPC interface. ### Resource Model Changes Providers are now exposed as resources that participate in a stack's dependency graph. Like other resources, they are explicitly created, may have multiple instances, and may have dependencies on other resources. Providers are referred to using provider references, which are a combination of the provider's URN and its ID. This design addresses the need during a preview to refer to providers that have not yet been physically created and therefore have no ID. All custom resources that are not themselves providers must specify a single provider via a provider reference. The named provider will be used to manage that resource's CRUD operations. If a resource's provider reference changes, the resource must be replaced. Though its URN is not present in the resource's dependency list, the provider should be treated as a dependency of the resource when topologically sorting the dependency graph. Finally, `Invoke` operations must now specify a provider to use for the invocation via a provider reference. ### Engine Changes First-class providers support requires a few changes to the engine: - The engine must have some way to map from provider references to provider plugins. It must be possible to add providers from a stack's checkpoint to this map and to register new/updated providers during the execution of a plan in response to CRUD operations on provider resources. - In order to support updating existing stacks using existing Pulumi programs that may not explicitly instantiate providers, the engine must be able to manage the "default" providers for each package referenced by a checkpoint or Pulumi program. The configuration for a "default" provider is taken from the stack's configuration data. The former need is addressed by adding a provider registry type that is responsible for managing all of the plugins required by a plan. In addition to loading plugins froma checkpoint and providing the ability to map from a provider reference to a provider plugin, this type serves as the provider plugin for providers themselves (i.e. it is the "provider provider"). The latter need is solved via two relatively self-contained changes to plan setup and the eval source. During plan setup, the old checkpoint is scanned for custom resources that do not have a provider reference in order to compute the set of packages that require a default provider. Once this set has been computed, the required default provider definitions are conjured and prepended to the checkpoint's resource list. Each resource that requires a default provider is then updated to refer to the default provider for its package. While an eval source is running, each custom resource registration, resource read, and invoke that does not name a provider is trapped before being returned by the source iterator. If no default provider for the appropriate package has been registered, the eval source synthesizes an appropriate registration, waits for it to complete, and records the registered provider's reference. This reference is injected into the original request, which is then processed as usual. If a default provider was already registered, the recorded reference is used and no new registration occurs. ### SDK Changes These changes only expose first-class providers from the Node.JS SDK. - A new abstract class, `ProviderResource`, can be subclassed and used to instantiate first-class providers. - A new field in `ResourceOptions`, `provider`, can be used to supply a particular provider instance to manage a `CustomResource`'s CRUD operations. - A new type, `InvokeOptions`, can be used to specify options that control the behavior of a call to `pulumi.runtime.invoke`. This type includes a `provider` field that is analogous to `ResourceOptions.provider`.
2018-08-07 02:50:29 +02:00
// A provider reference, fully resolved, if any.
providerRef: string | undefined;
// All serialized properties, fully awaited, serialized, and ready to go.
serializedProps: Record<string, any>;
// A set of URNs that this resource is directly dependent upon. These will all be URNs of
// custom resources, not component resources.
allDirectDependencyURNs: Set<URN>;
// Set of URNs that this resource is directly dependent upon, keyed by the property that causes
// the dependency. All urns in this map must exist in [allDirectDependencyURNs]. These will
// all be URNs of custom resources, not component resources.
propertyToDirectDependencyURNs: Map<string, Set<URN>>;
Support aliases for renaming, re-typing, or re-parenting resources (#2774) Adds a new resource option `aliases` which can be used to rename a resource. When making a breaking change to the name or type of a resource or component, the old name can be added to the list of `aliases` for a resource to ensure that existing resources will be migrated to the new name instead of being deleted and replaced with the new named resource. There are two key places this change is implemented. The first is the step generator in the engine. When computing whether there is an old version of a registered resource, we now take into account the aliases specified on the registered resource. That is, we first look up the resource by its new URN in the old state, and then by any aliases provided (in order). This can allow the resource to be matched as a (potential) update to an existing resource with a different URN. The second is the core `Resource` constructor in the JavaScript (and soon Python) SDKs. This change ensures that when a parent resource is aliased, that all children implicitly inherit corresponding aliases. It is similar to how many other resource options are "inherited" implicitly from the parent. Four specific scenarios are explicitly tested as part of this PR: 1. Renaming a resource 2. Adopting a resource into a component (as the owner of both component and consumption codebases) 3. Renaming a component instance (as the owner of the consumption codebase without changes to the component) 4. Changing the type of a component (as the owner of the component codebase without changes to the consumption codebase) 4. Combining (1) and (3) to make both changes to a resource at the same time
2019-06-01 08:01:01 +02:00
// A list of aliases applied to this resource.
aliases: URN[];
// An ID to import, if any.
import: ID | undefined;
}
/**
* Get an existing resource's state from the engine.
*/
export function getResource(res: Resource, props: Inputs, custom: boolean, urn: string): void {
// Extract the resource type from the URN.
const urnParts = urn.split("::");
const qualifiedType = urnParts[2];
const urnName = urnParts[3];
const type = qualifiedType.split("$").pop()!;
const label = `resource:urn=${urn}`;
log.debug(`Getting resource: urn=${urn}`);
const monitor: any = getMonitor();
const resopAsync = prepareResource(label, res, custom, props, {});
const preallocError = new Error();
debuggablePromise(resopAsync.then(async (resop) => {
const inputs = await serializeProperties(label, { urn });
const req = new providerproto.InvokeRequest();
req.setTok("pulumi:pulumi:getResource");
req.setArgs(gstruct.Struct.fromJavaScript(inputs));
req.setProvider("");
req.setVersion("");
// Now run the operation, serializing the invocation if necessary.
const opLabel = `monitor.getResource(${label})`;
runAsyncResourceOp(opLabel, async () => {
let resp: any;
let err: Error | undefined;
try {
if (monitor) {
resp = await debuggablePromise(new Promise((resolve, reject) =>
monitor.invoke(req, (rpcError: grpc.ServiceError, innerResponse: any) => {
log.debug(`getResource Invoke RPC finished: err: ${rpcError}, resp: ${innerResponse}`);
if (rpcError) {
if (rpcError.code === grpc.status.UNAVAILABLE || rpcError.code === grpc.status.CANCELLED) {
err = rpcError;
terminateRpcs();
rpcError.message = "Resource monitor is terminating";
(<any>preallocError).code = rpcError.code;
}
preallocError.message = `failed to get resource:urn=${urn}: ${rpcError.message}`;
reject(new Error(rpcError.details));
}
else {
resolve(innerResponse);
}
})), opLabel);
// If the invoke failed, raise an error
const failures: any = resp.getFailuresList();
if (failures && failures.length) {
let reasons = "";
for (let i = 0; i < failures.length; i++) {
if (reasons !== "") {
reasons += "; ";
}
reasons += `${failures[i].getReason()} (${failures[i].getProperty()})`;
}
throw new Error(`getResource Invoke failed: ${reasons}`);
}
// Otherwise, return the response.
const m = resp.getReturn().getFieldsMap();
resp = {
urn: m.get("urn").toJavaScript(),
id: m.get("id").toJavaScript() || undefined,
state: m.get("state").getStructValue(),
};
}
} catch (e) {
err = e;
resp = {
urn: "",
id: undefined,
state: undefined,
};
}
resop.resolveURN(resp.urn, err);
// Note: 'id || undefined' is intentional. We intentionally collapse falsy values to
// undefined so that later parts of our system don't have to deal with values like 'null'.
if (resop.resolveID) {
const id = resp.id || undefined;
resop.resolveID(id, id !== undefined, err);
}
await resolveOutputs(res, type, urnName, props, resp.state, {}, resop.resolvers, err);
});
}), label);
}
/**
* Reads an existing custom resource's state from the resource monitor. Note that resources read in this way
* will not be part of the resulting stack's state, as they are presumed to belong to another.
*/
export function readResource(res: Resource, t: string, name: string, props: Inputs, opts: ResourceOptions): void {
const id: Input<ID> | undefined = opts.id;
if (!id) {
throw new Error("Cannot read resource whose options are lacking an ID value");
}
const label = `resource:${name}[${t}]#...`;
log.debug(`Reading resource: id=${Output.isInstance(id) ? "Output<T>" : id}, t=${t}, name=${name}`);
const monitor = getMonitor();
const resopAsync = prepareResource(label, res, true, props, opts);
const preallocError = new Error();
debuggablePromise(resopAsync.then(async (resop) => {
const resolvedID = await serializeProperty(label, id, new Set());
log.debug(`ReadResource RPC prepared: id=${resolvedID}, t=${t}, name=${name}` +
(excessiveDebugOutput ? `, obj=${JSON.stringify(resop.serializedProps)}` : ``));
// Create a resource request and do the RPC.
const req = new resproto.ReadResourceRequest();
req.setType(t);
req.setName(name);
req.setId(resolvedID);
req.setParent(resop.parentURN);
Implement first-class providers. (#1695) ### First-Class Providers These changes implement support for first-class providers. First-class providers are provider plugins that are exposed as resources via the Pulumi programming model so that they may be explicitly and multiply instantiated. Each instance of a provider resource may be configured differently, and configuration parameters may be source from the outputs of other resources. ### Provider Plugin Changes In order to accommodate the need to verify and diff provider configuration and configure providers without complete configuration information, these changes adjust the high-level provider plugin interface. Two new methods for validating a provider's configuration and diffing changes to the same have been added (`CheckConfig` and `DiffConfig`, respectively), and the type of the configuration bag accepted by `Configure` has been changed to a `PropertyMap`. These changes have not yet been reflected in the provider plugin gRPC interface. We will do this in a set of follow-up changes. Until then, these methods are implemented by adapters: - `CheckConfig` validates that all configuration parameters are string or unknown properties. This is necessary because existing plugins only accept string-typed configuration values. - `DiffConfig` either returns "never replace" if all configuration values are known or "must replace" if any configuration value is unknown. The justification for this behavior is given [here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106) - `Configure` converts the config bag to a legacy config map and configures the provider plugin if all config values are known. If any config value is unknown, the underlying plugin is not configured and the provider may only perform `Check`, `Read`, and `Invoke`, all of which return empty results. We justify this behavior becuase it is only possible during a preview and provides the best experience we can manage with the existing gRPC interface. ### Resource Model Changes Providers are now exposed as resources that participate in a stack's dependency graph. Like other resources, they are explicitly created, may have multiple instances, and may have dependencies on other resources. Providers are referred to using provider references, which are a combination of the provider's URN and its ID. This design addresses the need during a preview to refer to providers that have not yet been physically created and therefore have no ID. All custom resources that are not themselves providers must specify a single provider via a provider reference. The named provider will be used to manage that resource's CRUD operations. If a resource's provider reference changes, the resource must be replaced. Though its URN is not present in the resource's dependency list, the provider should be treated as a dependency of the resource when topologically sorting the dependency graph. Finally, `Invoke` operations must now specify a provider to use for the invocation via a provider reference. ### Engine Changes First-class providers support requires a few changes to the engine: - The engine must have some way to map from provider references to provider plugins. It must be possible to add providers from a stack's checkpoint to this map and to register new/updated providers during the execution of a plan in response to CRUD operations on provider resources. - In order to support updating existing stacks using existing Pulumi programs that may not explicitly instantiate providers, the engine must be able to manage the "default" providers for each package referenced by a checkpoint or Pulumi program. The configuration for a "default" provider is taken from the stack's configuration data. The former need is addressed by adding a provider registry type that is responsible for managing all of the plugins required by a plan. In addition to loading plugins froma checkpoint and providing the ability to map from a provider reference to a provider plugin, this type serves as the provider plugin for providers themselves (i.e. it is the "provider provider"). The latter need is solved via two relatively self-contained changes to plan setup and the eval source. During plan setup, the old checkpoint is scanned for custom resources that do not have a provider reference in order to compute the set of packages that require a default provider. Once this set has been computed, the required default provider definitions are conjured and prepended to the checkpoint's resource list. Each resource that requires a default provider is then updated to refer to the default provider for its package. While an eval source is running, each custom resource registration, resource read, and invoke that does not name a provider is trapped before being returned by the source iterator. If no default provider for the appropriate package has been registered, the eval source synthesizes an appropriate registration, waits for it to complete, and records the registered provider's reference. This reference is injected into the original request, which is then processed as usual. If a default provider was already registered, the recorded reference is used and no new registration occurs. ### SDK Changes These changes only expose first-class providers from the Node.JS SDK. - A new abstract class, `ProviderResource`, can be subclassed and used to instantiate first-class providers. - A new field in `ResourceOptions`, `provider`, can be used to supply a particular provider instance to manage a `CustomResource`'s CRUD operations. - A new type, `InvokeOptions`, can be used to specify options that control the behavior of a call to `pulumi.runtime.invoke`. This type includes a `provider` field that is analogous to `ResourceOptions.provider`.
2018-08-07 02:50:29 +02:00
req.setProvider(resop.providerRef);
req.setProperties(gstruct.Struct.fromJavaScript(resop.serializedProps));
req.setDependenciesList(Array.from(resop.allDirectDependencyURNs));
req.setVersion(opts.version || "");
req.setAcceptsecrets(true);
req.setAcceptresources(!utils.disableResourceReferences);
req.setAdditionalsecretoutputsList((<any>opts).additionalSecretOutputs || []);
// Now run the operation, serializing the invocation if necessary.
const opLabel = `monitor.readResource(${label})`;
runAsyncResourceOp(opLabel, async () => {
let resp: any;
let err: Error | undefined;
try {
if (monitor) {
// If we're attached to the engine, make an RPC call and wait for it to resolve.
resp = await debuggablePromise(new Promise((resolve, reject) =>
(monitor as any).readResource(req, (rpcError: grpc.ServiceError, innerResponse: any) => {
log.debug(`ReadResource RPC finished: ${label}; err: ${rpcError}, resp: ${innerResponse}`);
if (rpcError) {
if (rpcError.code === grpc.status.UNAVAILABLE || rpcError.code === grpc.status.CANCELLED) {
err = rpcError;
terminateRpcs();
rpcError.message = "Resource monitor is terminating";
(<any>preallocError).code = rpcError.code;
}
preallocError.message =
`failed to read resource #${resolvedID} '${name}' [${t}]: ${rpcError.message}`;
reject(preallocError);
}
else {
resolve(innerResponse);
}
})), opLabel);
} else {
// If we aren't attached to the engine, in test mode, mock up a fake response for testing purposes.
const mockurn = await createUrn(req.getName(), req.getType(), req.getParent()).promise();
resp = {
getUrn: () => mockurn,
getProperties: () => req.getProperties(),
};
}
} catch (e) {
err = e;
resp = {
getUrn: () => "",
getProperties: () => undefined,
};
}
// Now resolve everything: the URN, the ID (supplied as input), and the output properties.
resop.resolveURN(resp.getUrn(), err);
resop.resolveID!(resolvedID, resolvedID !== undefined, err);
await resolveOutputs(res, t, name, props, resp.getProperties(), {}, resop.resolvers, err);
});
}), label);
}
/**
* registerResource registers a new resource object with a given type t and name. It returns the auto-generated
* URN and the ID that will resolve after the deployment has completed. All properties will be initialized to property
* objects that the registration operation will resolve at the right time (or remain unresolved for deployments).
*/
Initial support for remote component construction. (#5280) These changes add initial support for the construction of remote components. For now, this support is limited to the NodeJS SDK; follow-up changes will implement support for the other SDKs. Remote components are component resources that are constructed and managed by plugins rather than by Pulumi programs. In this sense, they are a bit like cloud resources, and are supported by the same distribution and plugin loading mechanisms and described by the same schema system. The construction of a remote component is initiated by a `RegisterResourceRequest` with the new `remote` field set to `true`. When the resource monitor receives such a request, it loads the plugin that implements the component resource and calls the `Construct` method added to the resource provider interface as part of these changes. This method accepts the information necessary to construct the component and its children: the component's name, type, resource options, inputs, and input dependencies. It is responsible for dispatching to the appropriate component factory to create the component, then returning its URN, resolved output properties, and output property dependencies. The dependency information is necessary to support features such as delete-before-replace, which rely on precise dependency information for custom resources. These changes also add initial support for more conveniently implementing resource providers in NodeJS. The interface used to implement such a provider is similar to the dynamic provider interface (and may be unified with that interface in the future). An example of a NodeJS program constructing a remote component resource also implemented in NodeJS can be found in `tests/construct_component/nodejs`. This is the core of #2430.
2020-09-08 04:33:55 +02:00
export function registerResource(res: Resource, t: string, name: string, custom: boolean, remote: boolean,
newDependency: (urn: URN) => Resource, props: Inputs, opts: ResourceOptions): void {
const label = `resource:${name}[${t}]`;
Initial support for remote component construction. (#5280) These changes add initial support for the construction of remote components. For now, this support is limited to the NodeJS SDK; follow-up changes will implement support for the other SDKs. Remote components are component resources that are constructed and managed by plugins rather than by Pulumi programs. In this sense, they are a bit like cloud resources, and are supported by the same distribution and plugin loading mechanisms and described by the same schema system. The construction of a remote component is initiated by a `RegisterResourceRequest` with the new `remote` field set to `true`. When the resource monitor receives such a request, it loads the plugin that implements the component resource and calls the `Construct` method added to the resource provider interface as part of these changes. This method accepts the information necessary to construct the component and its children: the component's name, type, resource options, inputs, and input dependencies. It is responsible for dispatching to the appropriate component factory to create the component, then returning its URN, resolved output properties, and output property dependencies. The dependency information is necessary to support features such as delete-before-replace, which rely on precise dependency information for custom resources. These changes also add initial support for more conveniently implementing resource providers in NodeJS. The interface used to implement such a provider is similar to the dynamic provider interface (and may be unified with that interface in the future). An example of a NodeJS program constructing a remote component resource also implemented in NodeJS can be found in `tests/construct_component/nodejs`. This is the core of #2430.
2020-09-08 04:33:55 +02:00
log.debug(`Registering resource: t=${t}, name=${name}, custom=${custom}, remote=${remote}`);
const monitor = getMonitor();
const resopAsync = prepareResource(label, res, custom, props, opts);
// In order to present a useful stack trace if an error does occur, we preallocate potential
// errors here. V8 captures a stack trace at the moment an Error is created and this stack
// trace will lead directly to user code. Throwing in `runAsyncResourceOp` results in an Error
// with a non-useful stack trace.
const preallocError = new Error();
debuggablePromise(resopAsync.then(async (resop) => {
log.debug(`RegisterResource RPC prepared: t=${t}, name=${name}` +
(excessiveDebugOutput ? `, obj=${JSON.stringify(resop.serializedProps)}` : ``));
const req = new resproto.RegisterResourceRequest();
req.setType(t);
req.setName(name);
req.setParent(resop.parentURN);
req.setCustom(custom);
req.setObject(gstruct.Struct.fromJavaScript(resop.serializedProps));
req.setProtect(opts.protect);
Implement first-class providers. (#1695) ### First-Class Providers These changes implement support for first-class providers. First-class providers are provider plugins that are exposed as resources via the Pulumi programming model so that they may be explicitly and multiply instantiated. Each instance of a provider resource may be configured differently, and configuration parameters may be source from the outputs of other resources. ### Provider Plugin Changes In order to accommodate the need to verify and diff provider configuration and configure providers without complete configuration information, these changes adjust the high-level provider plugin interface. Two new methods for validating a provider's configuration and diffing changes to the same have been added (`CheckConfig` and `DiffConfig`, respectively), and the type of the configuration bag accepted by `Configure` has been changed to a `PropertyMap`. These changes have not yet been reflected in the provider plugin gRPC interface. We will do this in a set of follow-up changes. Until then, these methods are implemented by adapters: - `CheckConfig` validates that all configuration parameters are string or unknown properties. This is necessary because existing plugins only accept string-typed configuration values. - `DiffConfig` either returns "never replace" if all configuration values are known or "must replace" if any configuration value is unknown. The justification for this behavior is given [here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106) - `Configure` converts the config bag to a legacy config map and configures the provider plugin if all config values are known. If any config value is unknown, the underlying plugin is not configured and the provider may only perform `Check`, `Read`, and `Invoke`, all of which return empty results. We justify this behavior becuase it is only possible during a preview and provides the best experience we can manage with the existing gRPC interface. ### Resource Model Changes Providers are now exposed as resources that participate in a stack's dependency graph. Like other resources, they are explicitly created, may have multiple instances, and may have dependencies on other resources. Providers are referred to using provider references, which are a combination of the provider's URN and its ID. This design addresses the need during a preview to refer to providers that have not yet been physically created and therefore have no ID. All custom resources that are not themselves providers must specify a single provider via a provider reference. The named provider will be used to manage that resource's CRUD operations. If a resource's provider reference changes, the resource must be replaced. Though its URN is not present in the resource's dependency list, the provider should be treated as a dependency of the resource when topologically sorting the dependency graph. Finally, `Invoke` operations must now specify a provider to use for the invocation via a provider reference. ### Engine Changes First-class providers support requires a few changes to the engine: - The engine must have some way to map from provider references to provider plugins. It must be possible to add providers from a stack's checkpoint to this map and to register new/updated providers during the execution of a plan in response to CRUD operations on provider resources. - In order to support updating existing stacks using existing Pulumi programs that may not explicitly instantiate providers, the engine must be able to manage the "default" providers for each package referenced by a checkpoint or Pulumi program. The configuration for a "default" provider is taken from the stack's configuration data. The former need is addressed by adding a provider registry type that is responsible for managing all of the plugins required by a plan. In addition to loading plugins froma checkpoint and providing the ability to map from a provider reference to a provider plugin, this type serves as the provider plugin for providers themselves (i.e. it is the "provider provider"). The latter need is solved via two relatively self-contained changes to plan setup and the eval source. During plan setup, the old checkpoint is scanned for custom resources that do not have a provider reference in order to compute the set of packages that require a default provider. Once this set has been computed, the required default provider definitions are conjured and prepended to the checkpoint's resource list. Each resource that requires a default provider is then updated to refer to the default provider for its package. While an eval source is running, each custom resource registration, resource read, and invoke that does not name a provider is trapped before being returned by the source iterator. If no default provider for the appropriate package has been registered, the eval source synthesizes an appropriate registration, waits for it to complete, and records the registered provider's reference. This reference is injected into the original request, which is then processed as usual. If a default provider was already registered, the recorded reference is used and no new registration occurs. ### SDK Changes These changes only expose first-class providers from the Node.JS SDK. - A new abstract class, `ProviderResource`, can be subclassed and used to instantiate first-class providers. - A new field in `ResourceOptions`, `provider`, can be used to supply a particular provider instance to manage a `CustomResource`'s CRUD operations. - A new type, `InvokeOptions`, can be used to specify options that control the behavior of a call to `pulumi.runtime.invoke`. This type includes a `provider` field that is analogous to `ResourceOptions.provider`.
2018-08-07 02:50:29 +02:00
req.setProvider(resop.providerRef);
req.setDependenciesList(Array.from(resop.allDirectDependencyURNs));
req.setDeletebeforereplace((<any>opts).deleteBeforeReplace || false);
req.setDeletebeforereplacedefined((<any>opts).deleteBeforeReplace !== undefined);
req.setIgnorechangesList(opts.ignoreChanges || []);
req.setVersion(opts.version || "");
req.setAcceptsecrets(true);
req.setAcceptresources(!utils.disableResourceReferences);
req.setAdditionalsecretoutputsList((<any>opts).additionalSecretOutputs || []);
Support aliases for renaming, re-typing, or re-parenting resources (#2774) Adds a new resource option `aliases` which can be used to rename a resource. When making a breaking change to the name or type of a resource or component, the old name can be added to the list of `aliases` for a resource to ensure that existing resources will be migrated to the new name instead of being deleted and replaced with the new named resource. There are two key places this change is implemented. The first is the step generator in the engine. When computing whether there is an old version of a registered resource, we now take into account the aliases specified on the registered resource. That is, we first look up the resource by its new URN in the old state, and then by any aliases provided (in order). This can allow the resource to be matched as a (potential) update to an existing resource with a different URN. The second is the core `Resource` constructor in the JavaScript (and soon Python) SDKs. This change ensures that when a parent resource is aliased, that all children implicitly inherit corresponding aliases. It is similar to how many other resource options are "inherited" implicitly from the parent. Four specific scenarios are explicitly tested as part of this PR: 1. Renaming a resource 2. Adopting a resource into a component (as the owner of both component and consumption codebases) 3. Renaming a component instance (as the owner of the consumption codebase without changes to the component) 4. Changing the type of a component (as the owner of the component codebase without changes to the consumption codebase) 4. Combining (1) and (3) to make both changes to a resource at the same time
2019-06-01 08:01:01 +02:00
req.setAliasesList(resop.aliases);
req.setImportid(resop.import || "");
Propagate inputs to outputs during preview. (#3327) These changes restore a more-correct version of the behavior that was disabled with #3014. The original implementation of this behavior was done in the SDKs, which do not have access to the complete inputs for a resource (in particular, default values filled in by the provider during `Check` are not exposed to the SDK). This lack of information meant that the resolved output values could disagree with the typings present in a provider SDK. Exacerbating this problem was the fact that unknown values were dropped entirely, causing `undefined` values to appear in unexpected places. By doing this in the engine and allowing unknown values to be represented in a first-class manner in the SDK, we can attack both of these issues. Although this behavior is not _strictly_ consistent with respect to the resource model--in an update, a resource's output properties will come from its provider and may differ from its input properties--this behavior was present in the product for a fairly long time without significant issues. In the future, we may be able to improve the accuracy of resource outputs during a preview by allowing the provider to dry-run CRUD operations and return partially-known values where possible. These changes also introduce new APIs in the Node and Python SDKs that work with unknown values in a first-class fashion: - A new parameter to the `apply` function that indicates that the callback should be run even if the result of the apply contains unknown values - `containsUnknowns` and `isUnknown`, which return true if a value either contains nested unknown values or is exactly an unknown value - The `Unknown` type, which represents unknown values The primary use case for these APIs is to allow nested, properties with known values to be accessed via the lifted property accessor even when the containing property is not fully know. A common example of this pattern is the `metadata.name` property of a Kubernetes `Namespace` object: while other properties of the `metadata` bag may be unknown, `name` is often known. These APIs allow `ns.metadata.name` to return a known value in this case. In order to avoid exposing downlevel SDKs to unknown values--a change which could break user code by exposing it to unexpected values--a language SDK must indicate whether or not it supports first-class unknown values as part of each `RegisterResourceRequest`. These changes also allow us to avoid breaking user code with the new behavior introduced by the prior commit. Fixes #3190.
2019-11-11 21:09:34 +01:00
req.setSupportspartialvalues(true);
Initial support for remote component construction. (#5280) These changes add initial support for the construction of remote components. For now, this support is limited to the NodeJS SDK; follow-up changes will implement support for the other SDKs. Remote components are component resources that are constructed and managed by plugins rather than by Pulumi programs. In this sense, they are a bit like cloud resources, and are supported by the same distribution and plugin loading mechanisms and described by the same schema system. The construction of a remote component is initiated by a `RegisterResourceRequest` with the new `remote` field set to `true`. When the resource monitor receives such a request, it loads the plugin that implements the component resource and calls the `Construct` method added to the resource provider interface as part of these changes. This method accepts the information necessary to construct the component and its children: the component's name, type, resource options, inputs, and input dependencies. It is responsible for dispatching to the appropriate component factory to create the component, then returning its URN, resolved output properties, and output property dependencies. The dependency information is necessary to support features such as delete-before-replace, which rely on precise dependency information for custom resources. These changes also add initial support for more conveniently implementing resource providers in NodeJS. The interface used to implement such a provider is similar to the dynamic provider interface (and may be unified with that interface in the future). An example of a NodeJS program constructing a remote component resource also implemented in NodeJS can be found in `tests/construct_component/nodejs`. This is the core of #2430.
2020-09-08 04:33:55 +02:00
req.setRemote(remote);
2019-07-15 23:26:28 +02:00
const customTimeouts = new resproto.RegisterResourceRequest.CustomTimeouts();
if (opts.customTimeouts != null) {
customTimeouts.setCreate(opts.customTimeouts.create);
customTimeouts.setUpdate(opts.customTimeouts.update);
customTimeouts.setDelete(opts.customTimeouts.delete);
}
req.setCustomtimeouts(customTimeouts);
Implement more precise delete-before-replace semantics. (#2369) This implements the new algorithm for deciding which resources must be deleted due to a delete-before-replace operation. We need to compute the set of resources that may be replaced by a change to the resource under consideration. We do this by taking the complete set of transitive dependents on the resource under consideration and removing any resources that would not be replaced by changes to their dependencies. We determine whether or not a resource may be replaced by substituting unknowns for input properties that may change due to deletion of the resources their value depends on and calling the resource provider's Diff method. This is perhaps clearer when described by example. Consider the following dependency graph: A __|__ B C | _|_ D E F In this graph, all of B, C, D, E, and F transitively depend on A. It may be the case, however, that changes to the specific properties of any of those resources R that would occur if a resource on the path to A were deleted and recreated may not cause R to be replaced. For example, the edge from B to A may be a simple dependsOn edge such that a change to B does not actually influence any of B's input properties. In that case, neither B nor D would need to be deleted before A could be deleted. In order to make the above algorithm a reality, the resource monitor interface has been updated to include a map that associates an input property key with the list of resources that input property depends on. Older clients of the resource monitor will leave this map empty, in which case all input properties will be treated as depending on all dependencies of the resource. This is probably overly conservative, but it is less conservative than what we currently implement, and is certainly correct.
2019-01-28 18:46:30 +01:00
const propertyDependencies = req.getPropertydependenciesMap();
for (const [key, resourceURNs] of resop.propertyToDirectDependencyURNs) {
Implement more precise delete-before-replace semantics. (#2369) This implements the new algorithm for deciding which resources must be deleted due to a delete-before-replace operation. We need to compute the set of resources that may be replaced by a change to the resource under consideration. We do this by taking the complete set of transitive dependents on the resource under consideration and removing any resources that would not be replaced by changes to their dependencies. We determine whether or not a resource may be replaced by substituting unknowns for input properties that may change due to deletion of the resources their value depends on and calling the resource provider's Diff method. This is perhaps clearer when described by example. Consider the following dependency graph: A __|__ B C | _|_ D E F In this graph, all of B, C, D, E, and F transitively depend on A. It may be the case, however, that changes to the specific properties of any of those resources R that would occur if a resource on the path to A were deleted and recreated may not cause R to be replaced. For example, the edge from B to A may be a simple dependsOn edge such that a change to B does not actually influence any of B's input properties. In that case, neither B nor D would need to be deleted before A could be deleted. In order to make the above algorithm a reality, the resource monitor interface has been updated to include a map that associates an input property key with the list of resources that input property depends on. Older clients of the resource monitor will leave this map empty, in which case all input properties will be treated as depending on all dependencies of the resource. This is probably overly conservative, but it is less conservative than what we currently implement, and is certainly correct.
2019-01-28 18:46:30 +01:00
const deps = new resproto.RegisterResourceRequest.PropertyDependencies();
deps.setUrnsList(Array.from(resourceURNs));
Implement more precise delete-before-replace semantics. (#2369) This implements the new algorithm for deciding which resources must be deleted due to a delete-before-replace operation. We need to compute the set of resources that may be replaced by a change to the resource under consideration. We do this by taking the complete set of transitive dependents on the resource under consideration and removing any resources that would not be replaced by changes to their dependencies. We determine whether or not a resource may be replaced by substituting unknowns for input properties that may change due to deletion of the resources their value depends on and calling the resource provider's Diff method. This is perhaps clearer when described by example. Consider the following dependency graph: A __|__ B C | _|_ D E F In this graph, all of B, C, D, E, and F transitively depend on A. It may be the case, however, that changes to the specific properties of any of those resources R that would occur if a resource on the path to A were deleted and recreated may not cause R to be replaced. For example, the edge from B to A may be a simple dependsOn edge such that a change to B does not actually influence any of B's input properties. In that case, neither B nor D would need to be deleted before A could be deleted. In order to make the above algorithm a reality, the resource monitor interface has been updated to include a map that associates an input property key with the list of resources that input property depends on. Older clients of the resource monitor will leave this map empty, in which case all input properties will be treated as depending on all dependencies of the resource. This is probably overly conservative, but it is less conservative than what we currently implement, and is certainly correct.
2019-01-28 18:46:30 +01:00
propertyDependencies.set(key, deps);
}
// Now run the operation, serializing the invocation if necessary.
const opLabel = `monitor.registerResource(${label})`;
runAsyncResourceOp(opLabel, async () => {
let resp: any;
let err: Error | undefined;
try {
if (monitor) {
// If we're running with an attachment to the engine, perform the operation.
resp = await debuggablePromise(new Promise((resolve, reject) =>
(monitor as any).registerResource(req, (rpcErr: grpc.ServiceError, innerResponse: any) => {
if (rpcErr) {
err = rpcErr;
// If the monitor is unavailable, it is in the process of shutting down or has already
// shut down. Don't emit an error and don't do any more RPCs, just exit.
if (rpcErr.code === grpc.status.UNAVAILABLE || rpcErr.code === grpc.status.CANCELLED) {
// Re-emit the message
terminateRpcs();
rpcErr.message = "Resource monitor is terminating";
(<any>preallocError).code = rpcErr.code;
}
// Node lets us hack the message as long as we do it before accessing the `stack` property.
log.debug(`RegisterResource RPC finished: ${label}; err: ${rpcErr}, resp: ${innerResponse}`);
preallocError.message = `failed to register new resource ${name} [${t}]: ${rpcErr.message}`;
reject(preallocError);
}
else {
log.debug(`RegisterResource RPC finished: ${label}; err: ${rpcErr}, resp: ${innerResponse}`);
resolve(innerResponse);
}
})), opLabel);
} else {
// If we aren't attached to the engine, in test mode, mock up a fake response for testing purposes.
const mockurn = await createUrn(req.getName(), req.getType(), req.getParent()).promise();
resp = {
getUrn: () => mockurn,
getId: () => undefined,
getObject: () => req.getObject(),
2020-10-01 21:31:52 +02:00
getPropertydependenciesMap: () => undefined,
};
}
} catch (e) {
err = e;
resp = {
getUrn: () => "",
getId: () => undefined,
getObject: () => req.getObject(),
Initial support for remote component construction. (#5280) These changes add initial support for the construction of remote components. For now, this support is limited to the NodeJS SDK; follow-up changes will implement support for the other SDKs. Remote components are component resources that are constructed and managed by plugins rather than by Pulumi programs. In this sense, they are a bit like cloud resources, and are supported by the same distribution and plugin loading mechanisms and described by the same schema system. The construction of a remote component is initiated by a `RegisterResourceRequest` with the new `remote` field set to `true`. When the resource monitor receives such a request, it loads the plugin that implements the component resource and calls the `Construct` method added to the resource provider interface as part of these changes. This method accepts the information necessary to construct the component and its children: the component's name, type, resource options, inputs, and input dependencies. It is responsible for dispatching to the appropriate component factory to create the component, then returning its URN, resolved output properties, and output property dependencies. The dependency information is necessary to support features such as delete-before-replace, which rely on precise dependency information for custom resources. These changes also add initial support for more conveniently implementing resource providers in NodeJS. The interface used to implement such a provider is similar to the dynamic provider interface (and may be unified with that interface in the future). An example of a NodeJS program constructing a remote component resource also implemented in NodeJS can be found in `tests/construct_component/nodejs`. This is the core of #2430.
2020-09-08 04:33:55 +02:00
getPropertydependenciesMap: () => undefined,
};
}
resop.resolveURN(resp.getUrn(), err);
// Note: 'id || undefined' is intentional. We intentionally collapse falsy values to
// undefined so that later parts of our system don't have to deal with values like 'null'.
if (resop.resolveID) {
const id = resp.getId() || undefined;
resop.resolveID(id, id !== undefined, err);
}
Initial support for remote component construction. (#5280) These changes add initial support for the construction of remote components. For now, this support is limited to the NodeJS SDK; follow-up changes will implement support for the other SDKs. Remote components are component resources that are constructed and managed by plugins rather than by Pulumi programs. In this sense, they are a bit like cloud resources, and are supported by the same distribution and plugin loading mechanisms and described by the same schema system. The construction of a remote component is initiated by a `RegisterResourceRequest` with the new `remote` field set to `true`. When the resource monitor receives such a request, it loads the plugin that implements the component resource and calls the `Construct` method added to the resource provider interface as part of these changes. This method accepts the information necessary to construct the component and its children: the component's name, type, resource options, inputs, and input dependencies. It is responsible for dispatching to the appropriate component factory to create the component, then returning its URN, resolved output properties, and output property dependencies. The dependency information is necessary to support features such as delete-before-replace, which rely on precise dependency information for custom resources. These changes also add initial support for more conveniently implementing resource providers in NodeJS. The interface used to implement such a provider is similar to the dynamic provider interface (and may be unified with that interface in the future). An example of a NodeJS program constructing a remote component resource also implemented in NodeJS can be found in `tests/construct_component/nodejs`. This is the core of #2430.
2020-09-08 04:33:55 +02:00
const deps: Record<string, Resource[]> = {};
const rpcDeps = resp.getPropertydependenciesMap();
if (rpcDeps) {
for (const [k, propertyDeps] of resp.getPropertydependenciesMap().entries()) {
const urns = <URN[]>propertyDeps.getUrnsList();
deps[k] = urns.map(urn => newDependency(urn));
}
}
// Now resolve the output properties.
await resolveOutputs(res, t, name, props, resp.getObject(), deps, resop.resolvers, err);
});
}), label);
}
/**
* Prepares for an RPC that will manufacture a resource, and hence deals with input and output
* properties.
*/
async function prepareResource(label: string, res: Resource, custom: boolean,
props: Inputs, opts: ResourceOptions): Promise<ResourceResolverOperation> {
Switch to parent pointers; display components nicely This change switches from child lists to parent pointers, in the way resource ancestries are represented. This cleans up a fair bit of the old parenting logic, including all notion of ambient parent scopes (and will notably address pulumi/pulumi#435). This lets us show a more parent/child display in the output when doing planning and updating. For instance, here is an update of a lambda's text, which is logically part of a cloud timer: * cloud:timer:Timer: (same) [urn=urn:pulumi:malta::lm-cloud::cloud:timer:Timer::lm-cts-malta-job-CleanSnapshots] * cloud:function:Function: (same) [urn=urn:pulumi:malta::lm-cloud::cloud:function:Function::lm-cts-malta-job-CleanSnapshots] * aws:serverless:Function: (same) [urn=urn:pulumi:malta::lm-cloud::aws:serverless:Function::lm-cts-malta-job-CleanSnapshots] ~ aws:lambda/function:Function: (modify) [id=lm-cts-malta-job-CleanSnapshots-fee4f3bf41280741] [urn=urn:pulumi:malta::lm-cloud::aws:lambda/function:Function::lm-cts-malta-job-CleanSnapshots] - code : archive(assets:2092f44) { // etc etc etc Note that we still get walls of text, but this will be actually quite nice when combined with pulumi/pulumi#454. I've also suppressed printing properties that didn't change during updates when --detailed was not passed, and also suppressed empty strings and zero-length arrays (since TF uses these as defaults in many places and it just makes creation and deletion quite verbose). Note that this is a far cry from everything we can possibly do here as part of pulumi/pulumi#340 (and even pulumi/pulumi#417). But it's a good start towards taming some of our output spew.
2017-11-17 03:21:41 +01:00
// Simply initialize the URN property and get prepared to resolve it later on.
// Note: a resource urn will always get a value, and thus the output property
// for it can always run .apply calls.
let resolveURN: (urn: URN, err?: Error) => void;
{
let resolveValue: (urn: URN) => void;
let resolveIsKnown: (isKnown: boolean) => void;
(res as any).urn = new Output(
res,
debuggablePromise(
new Promise<URN>(resolve => resolveValue = resolve),
`resolveURN(${label})`),
debuggablePromise(
new Promise<boolean>(resolve => resolveIsKnown = resolve),
`resolveURNIsKnown(${label})`),
/*isSecret:*/ Promise.resolve(false),
Promise.resolve(res));
resolveURN = (v, err) => {
resolveValue(v);
resolveIsKnown(err === undefined);
};
}
Switch to parent pointers; display components nicely This change switches from child lists to parent pointers, in the way resource ancestries are represented. This cleans up a fair bit of the old parenting logic, including all notion of ambient parent scopes (and will notably address pulumi/pulumi#435). This lets us show a more parent/child display in the output when doing planning and updating. For instance, here is an update of a lambda's text, which is logically part of a cloud timer: * cloud:timer:Timer: (same) [urn=urn:pulumi:malta::lm-cloud::cloud:timer:Timer::lm-cts-malta-job-CleanSnapshots] * cloud:function:Function: (same) [urn=urn:pulumi:malta::lm-cloud::cloud:function:Function::lm-cts-malta-job-CleanSnapshots] * aws:serverless:Function: (same) [urn=urn:pulumi:malta::lm-cloud::aws:serverless:Function::lm-cts-malta-job-CleanSnapshots] ~ aws:lambda/function:Function: (modify) [id=lm-cts-malta-job-CleanSnapshots-fee4f3bf41280741] [urn=urn:pulumi:malta::lm-cloud::aws:lambda/function:Function::lm-cts-malta-job-CleanSnapshots] - code : archive(assets:2092f44) { // etc etc etc Note that we still get walls of text, but this will be actually quite nice when combined with pulumi/pulumi#454. I've also suppressed printing properties that didn't change during updates when --detailed was not passed, and also suppressed empty strings and zero-length arrays (since TF uses these as defaults in many places and it just makes creation and deletion quite verbose). Note that this is a far cry from everything we can possibly do here as part of pulumi/pulumi#340 (and even pulumi/pulumi#417). But it's a good start towards taming some of our output spew.
2017-11-17 03:21:41 +01:00
// If a custom resource, make room for the ID property.
let resolveID: ((v: any, performApply: boolean, err?: Error) => void) | undefined;
if (custom) {
let resolveValue: (v: ID) => void;
let resolveIsKnown: (v: boolean) => void;
(res as any).id = new Output(
res,
debuggablePromise(new Promise<ID>(resolve => resolveValue = resolve),
`resolveID(${label})`),
debuggablePromise(new Promise<boolean>(
resolve => resolveIsKnown = resolve), `resolveIDIsKnown(${label})`),
Promise.resolve(false),
Promise.resolve(res));
resolveID = (v, isKnown, err) => {
resolveValue(v);
resolveIsKnown(err ? false : isKnown);
};
Implement components This change implements core support for "components" in the Pulumi Fabric. This work is described further in pulumi/pulumi#340, where we are still discussing some of the finer points. In a nutshell, resources no longer imply external providers. It's entirely possible to have a resource that logically represents something but without having a physical manifestation that needs to be tracked and managed by our typical CRUD operations. For example, the aws/serverless/Function helper is one such type. It aggregates Lambda-related resources and exposes a nice interface. All of the Pulumi Cloud Framework resources are also examples. To indicate that a resource does participate in the usual CRUD resource provider, it simply derives from ExternalResource instead of Resource. All resources now have the ability to adopt children. This is purely a metadata/tagging thing, and will help us roll up displays, provide attribution to the developer, and even hide aspects of the resource graph as appropriate (e.g., when they are implementation details). Our use of this capability is ultra limited right now; in fact, the only place we display children is in the CLI output. For instance: + aws:serverless:Function: (create) [urn=urn:pulumi:demo::serverless::aws:serverless:Function::mylambda] => urn:pulumi:demo::serverless::aws:iam/role:Role::mylambda-iamrole => urn:pulumi:demo::serverless::aws:iam/rolePolicyAttachment:RolePolicyAttachment::mylambda-iampolicy-0 => urn:pulumi:demo::serverless::aws:lambda/function:Function::mylambda The bit indicating whether a resource is external or not is tracked in the resulting checkpoint file, along with any of its children.
2017-10-14 23:18:43 +02:00
}
// Now "transfer" all input properties into unresolved Promises on res. This way,
// this resource will look like it has all its output properties to anyone it is
// passed to. However, those promises won't actually resolve until the registerResource
// RPC returns
const resolvers = transferProperties(res, label, props);
/** IMPORTANT! We should never await prior to this line, otherwise the Resource will be partly uninitialized. */
// Before we can proceed, all our dependencies must be finished.
const explicitDirectDependencies = new Set(await gatherExplicitDependencies(opts.dependsOn));
// Serialize out all our props to their final values. In doing so, we'll also collect all
// the Resources pointed to by any Dependency objects we encounter, adding them to 'propertyDependencies'.
const [serializedProps, propertyToDirectDependencies] = await serializeResourceProperties(label, props);
// Wait for the parent to complete.
// If no parent was provided, parent to the root resource.
const parentURN = opts.parent
? await opts.parent.urn.promise()
: await getRootResource();
Implement first-class providers. (#1695) ### First-Class Providers These changes implement support for first-class providers. First-class providers are provider plugins that are exposed as resources via the Pulumi programming model so that they may be explicitly and multiply instantiated. Each instance of a provider resource may be configured differently, and configuration parameters may be source from the outputs of other resources. ### Provider Plugin Changes In order to accommodate the need to verify and diff provider configuration and configure providers without complete configuration information, these changes adjust the high-level provider plugin interface. Two new methods for validating a provider's configuration and diffing changes to the same have been added (`CheckConfig` and `DiffConfig`, respectively), and the type of the configuration bag accepted by `Configure` has been changed to a `PropertyMap`. These changes have not yet been reflected in the provider plugin gRPC interface. We will do this in a set of follow-up changes. Until then, these methods are implemented by adapters: - `CheckConfig` validates that all configuration parameters are string or unknown properties. This is necessary because existing plugins only accept string-typed configuration values. - `DiffConfig` either returns "never replace" if all configuration values are known or "must replace" if any configuration value is unknown. The justification for this behavior is given [here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106) - `Configure` converts the config bag to a legacy config map and configures the provider plugin if all config values are known. If any config value is unknown, the underlying plugin is not configured and the provider may only perform `Check`, `Read`, and `Invoke`, all of which return empty results. We justify this behavior becuase it is only possible during a preview and provides the best experience we can manage with the existing gRPC interface. ### Resource Model Changes Providers are now exposed as resources that participate in a stack's dependency graph. Like other resources, they are explicitly created, may have multiple instances, and may have dependencies on other resources. Providers are referred to using provider references, which are a combination of the provider's URN and its ID. This design addresses the need during a preview to refer to providers that have not yet been physically created and therefore have no ID. All custom resources that are not themselves providers must specify a single provider via a provider reference. The named provider will be used to manage that resource's CRUD operations. If a resource's provider reference changes, the resource must be replaced. Though its URN is not present in the resource's dependency list, the provider should be treated as a dependency of the resource when topologically sorting the dependency graph. Finally, `Invoke` operations must now specify a provider to use for the invocation via a provider reference. ### Engine Changes First-class providers support requires a few changes to the engine: - The engine must have some way to map from provider references to provider plugins. It must be possible to add providers from a stack's checkpoint to this map and to register new/updated providers during the execution of a plan in response to CRUD operations on provider resources. - In order to support updating existing stacks using existing Pulumi programs that may not explicitly instantiate providers, the engine must be able to manage the "default" providers for each package referenced by a checkpoint or Pulumi program. The configuration for a "default" provider is taken from the stack's configuration data. The former need is addressed by adding a provider registry type that is responsible for managing all of the plugins required by a plan. In addition to loading plugins froma checkpoint and providing the ability to map from a provider reference to a provider plugin, this type serves as the provider plugin for providers themselves (i.e. it is the "provider provider"). The latter need is solved via two relatively self-contained changes to plan setup and the eval source. During plan setup, the old checkpoint is scanned for custom resources that do not have a provider reference in order to compute the set of packages that require a default provider. Once this set has been computed, the required default provider definitions are conjured and prepended to the checkpoint's resource list. Each resource that requires a default provider is then updated to refer to the default provider for its package. While an eval source is running, each custom resource registration, resource read, and invoke that does not name a provider is trapped before being returned by the source iterator. If no default provider for the appropriate package has been registered, the eval source synthesizes an appropriate registration, waits for it to complete, and records the registered provider's reference. This reference is injected into the original request, which is then processed as usual. If a default provider was already registered, the recorded reference is used and no new registration occurs. ### SDK Changes These changes only expose first-class providers from the Node.JS SDK. - A new abstract class, `ProviderResource`, can be subclassed and used to instantiate first-class providers. - A new field in `ResourceOptions`, `provider`, can be used to supply a particular provider instance to manage a `CustomResource`'s CRUD operations. - A new type, `InvokeOptions`, can be used to specify options that control the behavior of a call to `pulumi.runtime.invoke`. This type includes a `provider` field that is analogous to `ResourceOptions.provider`.
2018-08-07 02:50:29 +02:00
let providerRef: string | undefined;
let importID: ID | undefined;
if (custom) {
const customOpts = <CustomResourceOptions>opts;
importID = customOpts.import;
2019-10-18 02:12:45 +02:00
providerRef = await ProviderResource.register(opts.provider);
Implement first-class providers. (#1695) ### First-Class Providers These changes implement support for first-class providers. First-class providers are provider plugins that are exposed as resources via the Pulumi programming model so that they may be explicitly and multiply instantiated. Each instance of a provider resource may be configured differently, and configuration parameters may be source from the outputs of other resources. ### Provider Plugin Changes In order to accommodate the need to verify and diff provider configuration and configure providers without complete configuration information, these changes adjust the high-level provider plugin interface. Two new methods for validating a provider's configuration and diffing changes to the same have been added (`CheckConfig` and `DiffConfig`, respectively), and the type of the configuration bag accepted by `Configure` has been changed to a `PropertyMap`. These changes have not yet been reflected in the provider plugin gRPC interface. We will do this in a set of follow-up changes. Until then, these methods are implemented by adapters: - `CheckConfig` validates that all configuration parameters are string or unknown properties. This is necessary because existing plugins only accept string-typed configuration values. - `DiffConfig` either returns "never replace" if all configuration values are known or "must replace" if any configuration value is unknown. The justification for this behavior is given [here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106) - `Configure` converts the config bag to a legacy config map and configures the provider plugin if all config values are known. If any config value is unknown, the underlying plugin is not configured and the provider may only perform `Check`, `Read`, and `Invoke`, all of which return empty results. We justify this behavior becuase it is only possible during a preview and provides the best experience we can manage with the existing gRPC interface. ### Resource Model Changes Providers are now exposed as resources that participate in a stack's dependency graph. Like other resources, they are explicitly created, may have multiple instances, and may have dependencies on other resources. Providers are referred to using provider references, which are a combination of the provider's URN and its ID. This design addresses the need during a preview to refer to providers that have not yet been physically created and therefore have no ID. All custom resources that are not themselves providers must specify a single provider via a provider reference. The named provider will be used to manage that resource's CRUD operations. If a resource's provider reference changes, the resource must be replaced. Though its URN is not present in the resource's dependency list, the provider should be treated as a dependency of the resource when topologically sorting the dependency graph. Finally, `Invoke` operations must now specify a provider to use for the invocation via a provider reference. ### Engine Changes First-class providers support requires a few changes to the engine: - The engine must have some way to map from provider references to provider plugins. It must be possible to add providers from a stack's checkpoint to this map and to register new/updated providers during the execution of a plan in response to CRUD operations on provider resources. - In order to support updating existing stacks using existing Pulumi programs that may not explicitly instantiate providers, the engine must be able to manage the "default" providers for each package referenced by a checkpoint or Pulumi program. The configuration for a "default" provider is taken from the stack's configuration data. The former need is addressed by adding a provider registry type that is responsible for managing all of the plugins required by a plan. In addition to loading plugins froma checkpoint and providing the ability to map from a provider reference to a provider plugin, this type serves as the provider plugin for providers themselves (i.e. it is the "provider provider"). The latter need is solved via two relatively self-contained changes to plan setup and the eval source. During plan setup, the old checkpoint is scanned for custom resources that do not have a provider reference in order to compute the set of packages that require a default provider. Once this set has been computed, the required default provider definitions are conjured and prepended to the checkpoint's resource list. Each resource that requires a default provider is then updated to refer to the default provider for its package. While an eval source is running, each custom resource registration, resource read, and invoke that does not name a provider is trapped before being returned by the source iterator. If no default provider for the appropriate package has been registered, the eval source synthesizes an appropriate registration, waits for it to complete, and records the registered provider's reference. This reference is injected into the original request, which is then processed as usual. If a default provider was already registered, the recorded reference is used and no new registration occurs. ### SDK Changes These changes only expose first-class providers from the Node.JS SDK. - A new abstract class, `ProviderResource`, can be subclassed and used to instantiate first-class providers. - A new field in `ResourceOptions`, `provider`, can be used to supply a particular provider instance to manage a `CustomResource`'s CRUD operations. - A new type, `InvokeOptions`, can be used to specify options that control the behavior of a call to `pulumi.runtime.invoke`. This type includes a `provider` field that is analogous to `ResourceOptions.provider`.
2018-08-07 02:50:29 +02:00
}
// Collect the URNs for explicit/implicit dependencies for the engine so that it can understand
// the dependency graph and optimize operations accordingly.
// The list of all dependencies (implicit or explicit).
const allDirectDependencies = new Set<Resource>(explicitDirectDependencies);
const allDirectDependencyURNs = await getAllTransitivelyReferencedCustomResourceURNs(explicitDirectDependencies);
const propertyToDirectDependencyURNs = new Map<string, Set<URN>>();
for (const [propertyName, directDependencies] of propertyToDirectDependencies) {
addAll(allDirectDependencies, directDependencies);
const urns = await getAllTransitivelyReferencedCustomResourceURNs(directDependencies);
addAll(allDirectDependencyURNs, urns);
propertyToDirectDependencyURNs.set(propertyName, urns);
}
Support aliases for renaming, re-typing, or re-parenting resources (#2774) Adds a new resource option `aliases` which can be used to rename a resource. When making a breaking change to the name or type of a resource or component, the old name can be added to the list of `aliases` for a resource to ensure that existing resources will be migrated to the new name instead of being deleted and replaced with the new named resource. There are two key places this change is implemented. The first is the step generator in the engine. When computing whether there is an old version of a registered resource, we now take into account the aliases specified on the registered resource. That is, we first look up the resource by its new URN in the old state, and then by any aliases provided (in order). This can allow the resource to be matched as a (potential) update to an existing resource with a different URN. The second is the core `Resource` constructor in the JavaScript (and soon Python) SDKs. This change ensures that when a parent resource is aliased, that all children implicitly inherit corresponding aliases. It is similar to how many other resource options are "inherited" implicitly from the parent. Four specific scenarios are explicitly tested as part of this PR: 1. Renaming a resource 2. Adopting a resource into a component (as the owner of both component and consumption codebases) 3. Renaming a component instance (as the owner of the consumption codebase without changes to the component) 4. Changing the type of a component (as the owner of the component codebase without changes to the consumption codebase) 4. Combining (1) and (3) to make both changes to a resource at the same time
2019-06-01 08:01:01 +02:00
// Wait for all aliases. Note that we use `res.__aliases` instead of `opts.aliases` as the former has been processed
// in the Resource constructor prior to calling `registerResource` - both adding new inherited aliases and
// simplifying aliases down to URNs.
const aliases = [];
const uniqueAliases = new Set<string>();
for (const alias of (res.__aliases || [])) {
const aliasVal = await output(alias).promise();
if (!uniqueAliases.has(aliasVal)) {
uniqueAliases.add(aliasVal);
aliases.push(aliasVal);
}
Support aliases for renaming, re-typing, or re-parenting resources (#2774) Adds a new resource option `aliases` which can be used to rename a resource. When making a breaking change to the name or type of a resource or component, the old name can be added to the list of `aliases` for a resource to ensure that existing resources will be migrated to the new name instead of being deleted and replaced with the new named resource. There are two key places this change is implemented. The first is the step generator in the engine. When computing whether there is an old version of a registered resource, we now take into account the aliases specified on the registered resource. That is, we first look up the resource by its new URN in the old state, and then by any aliases provided (in order). This can allow the resource to be matched as a (potential) update to an existing resource with a different URN. The second is the core `Resource` constructor in the JavaScript (and soon Python) SDKs. This change ensures that when a parent resource is aliased, that all children implicitly inherit corresponding aliases. It is similar to how many other resource options are "inherited" implicitly from the parent. Four specific scenarios are explicitly tested as part of this PR: 1. Renaming a resource 2. Adopting a resource into a component (as the owner of both component and consumption codebases) 3. Renaming a component instance (as the owner of the consumption codebase without changes to the component) 4. Changing the type of a component (as the owner of the component codebase without changes to the consumption codebase) 4. Combining (1) and (3) to make both changes to a resource at the same time
2019-06-01 08:01:01 +02:00
}
return {
resolveURN: resolveURN,
resolveID: resolveID,
resolvers: resolvers,
serializedProps: serializedProps,
parentURN: parentURN,
Implement first-class providers. (#1695) ### First-Class Providers These changes implement support for first-class providers. First-class providers are provider plugins that are exposed as resources via the Pulumi programming model so that they may be explicitly and multiply instantiated. Each instance of a provider resource may be configured differently, and configuration parameters may be source from the outputs of other resources. ### Provider Plugin Changes In order to accommodate the need to verify and diff provider configuration and configure providers without complete configuration information, these changes adjust the high-level provider plugin interface. Two new methods for validating a provider's configuration and diffing changes to the same have been added (`CheckConfig` and `DiffConfig`, respectively), and the type of the configuration bag accepted by `Configure` has been changed to a `PropertyMap`. These changes have not yet been reflected in the provider plugin gRPC interface. We will do this in a set of follow-up changes. Until then, these methods are implemented by adapters: - `CheckConfig` validates that all configuration parameters are string or unknown properties. This is necessary because existing plugins only accept string-typed configuration values. - `DiffConfig` either returns "never replace" if all configuration values are known or "must replace" if any configuration value is unknown. The justification for this behavior is given [here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106) - `Configure` converts the config bag to a legacy config map and configures the provider plugin if all config values are known. If any config value is unknown, the underlying plugin is not configured and the provider may only perform `Check`, `Read`, and `Invoke`, all of which return empty results. We justify this behavior becuase it is only possible during a preview and provides the best experience we can manage with the existing gRPC interface. ### Resource Model Changes Providers are now exposed as resources that participate in a stack's dependency graph. Like other resources, they are explicitly created, may have multiple instances, and may have dependencies on other resources. Providers are referred to using provider references, which are a combination of the provider's URN and its ID. This design addresses the need during a preview to refer to providers that have not yet been physically created and therefore have no ID. All custom resources that are not themselves providers must specify a single provider via a provider reference. The named provider will be used to manage that resource's CRUD operations. If a resource's provider reference changes, the resource must be replaced. Though its URN is not present in the resource's dependency list, the provider should be treated as a dependency of the resource when topologically sorting the dependency graph. Finally, `Invoke` operations must now specify a provider to use for the invocation via a provider reference. ### Engine Changes First-class providers support requires a few changes to the engine: - The engine must have some way to map from provider references to provider plugins. It must be possible to add providers from a stack's checkpoint to this map and to register new/updated providers during the execution of a plan in response to CRUD operations on provider resources. - In order to support updating existing stacks using existing Pulumi programs that may not explicitly instantiate providers, the engine must be able to manage the "default" providers for each package referenced by a checkpoint or Pulumi program. The configuration for a "default" provider is taken from the stack's configuration data. The former need is addressed by adding a provider registry type that is responsible for managing all of the plugins required by a plan. In addition to loading plugins froma checkpoint and providing the ability to map from a provider reference to a provider plugin, this type serves as the provider plugin for providers themselves (i.e. it is the "provider provider"). The latter need is solved via two relatively self-contained changes to plan setup and the eval source. During plan setup, the old checkpoint is scanned for custom resources that do not have a provider reference in order to compute the set of packages that require a default provider. Once this set has been computed, the required default provider definitions are conjured and prepended to the checkpoint's resource list. Each resource that requires a default provider is then updated to refer to the default provider for its package. While an eval source is running, each custom resource registration, resource read, and invoke that does not name a provider is trapped before being returned by the source iterator. If no default provider for the appropriate package has been registered, the eval source synthesizes an appropriate registration, waits for it to complete, and records the registered provider's reference. This reference is injected into the original request, which is then processed as usual. If a default provider was already registered, the recorded reference is used and no new registration occurs. ### SDK Changes These changes only expose first-class providers from the Node.JS SDK. - A new abstract class, `ProviderResource`, can be subclassed and used to instantiate first-class providers. - A new field in `ResourceOptions`, `provider`, can be used to supply a particular provider instance to manage a `CustomResource`'s CRUD operations. - A new type, `InvokeOptions`, can be used to specify options that control the behavior of a call to `pulumi.runtime.invoke`. This type includes a `provider` field that is analogous to `ResourceOptions.provider`.
2018-08-07 02:50:29 +02:00
providerRef: providerRef,
allDirectDependencyURNs: allDirectDependencyURNs,
propertyToDirectDependencyURNs: propertyToDirectDependencyURNs,
Support aliases for renaming, re-typing, or re-parenting resources (#2774) Adds a new resource option `aliases` which can be used to rename a resource. When making a breaking change to the name or type of a resource or component, the old name can be added to the list of `aliases` for a resource to ensure that existing resources will be migrated to the new name instead of being deleted and replaced with the new named resource. There are two key places this change is implemented. The first is the step generator in the engine. When computing whether there is an old version of a registered resource, we now take into account the aliases specified on the registered resource. That is, we first look up the resource by its new URN in the old state, and then by any aliases provided (in order). This can allow the resource to be matched as a (potential) update to an existing resource with a different URN. The second is the core `Resource` constructor in the JavaScript (and soon Python) SDKs. This change ensures that when a parent resource is aliased, that all children implicitly inherit corresponding aliases. It is similar to how many other resource options are "inherited" implicitly from the parent. Four specific scenarios are explicitly tested as part of this PR: 1. Renaming a resource 2. Adopting a resource into a component (as the owner of both component and consumption codebases) 3. Renaming a component instance (as the owner of the consumption codebase without changes to the component) 4. Changing the type of a component (as the owner of the component codebase without changes to the consumption codebase) 4. Combining (1) and (3) to make both changes to a resource at the same time
2019-06-01 08:01:01 +02:00
aliases: aliases,
import: importID,
};
}
function addAll<T>(to: Set<T>, from: Set<T>) {
for (const val of from) {
to.add(val);
}
}
async function getAllTransitivelyReferencedCustomResourceURNs(resources: Set<Resource>) {
// Go through 'resources', but transitively walk through **Component** resources, collecting any
// of their child resources. This way, a Component acts as an aggregation really of all the
// reachable custom resources it parents. This walking will transitively walk through other
// child ComponentResources, but will stop when it hits custom resources. in other words, if we
// had:
//
// Comp1
// / \
// Cust1 Comp2
// / \
// Cust2 Cust3
// /
// Cust4
//
// Then the transitively reachable custom resources of Comp1 will be [Cust1, Cust2, Cust3]. It
// will *not* include `Cust4`.
// To do this, first we just get the transitively reachable set of resources (not diving
// into custom resources). In the above picture, if we start with 'Comp1', this will be
// [Comp1, Cust1, Comp2, Cust2, Cust3]
const transitivelyReachableResources = await getTransitivelyReferencedChildResourcesOfComponentResources(resources);
const transitivelyReachableCustomResources = [...transitivelyReachableResources].filter(r => CustomResource.isInstance(r));
const promises = transitivelyReachableCustomResources.map(r => r.urn.promise());
const urns = await Promise.all(promises);
return new Set<string>(urns);
}
/**
* Recursively walk the resources passed in, returning them and all resources reachable from
* [Resource.__childResources] through any **Component** resources we encounter.
*/
async function getTransitivelyReferencedChildResourcesOfComponentResources(resources: Set<Resource>) {
// Recursively walk the dependent resources through their children, adding them to the result set.
const result = new Set<Resource>();
await addTransitivelyReferencedChildResourcesOfComponentResources(resources, result);
return result;
}
async function addTransitivelyReferencedChildResourcesOfComponentResources(resources: Set<Resource> | undefined, result: Set<Resource>) {
if (resources) {
for (const resource of resources) {
if (!result.has(resource)) {
result.add(resource);
if (ComponentResource.isInstance(resource)) {
// This await is safe even if __isConstructed is undefined. Ensure that the
// resource has completely finished construction. That way all parent/child
// relationships will have been setup.
await resource.__data;
addTransitivelyReferencedChildResourcesOfComponentResources(resource.__childResources, result);
}
}
}
}
}
/**
* Gathers explicit dependent Resources from a list of Resources (possibly Promises and/or Outputs).
*/
async function gatherExplicitDependencies(
dependsOn: Input<Input<Resource>[]> | Input<Resource> | undefined): Promise<Resource[]> {
if (dependsOn) {
if (Array.isArray(dependsOn)) {
const dos: Resource[] = [];
for (const d of dependsOn) {
dos.push(...(await gatherExplicitDependencies(d)));
}
return dos;
} else if (dependsOn instanceof Promise) {
return gatherExplicitDependencies(await dependsOn);
} else if (Output.isInstance(dependsOn)) {
// Recursively gather dependencies, await the promise, and append the output's dependencies.
const dos = (dependsOn as Output<Input<Resource>[] | Input<Resource>>).apply(v => gatherExplicitDependencies(v));
const urns = await dos.promise();
const dosResources = await getAllResources(dos);
const implicits = await gatherExplicitDependencies([...dosResources]);
return (urns ?? []).concat(implicits);
} else {
if (!Resource.isInstance(dependsOn)) {
throw new Error("'dependsOn' was passed a value that was not a Resource.");
}
return [dependsOn];
}
}
return [];
}
/**
* Finishes a resource creation RPC operation by resolving its outputs to the resulting RPC payload.
*/
async function resolveOutputs(res: Resource, t: string, name: string,
Initial support for remote component construction. (#5280) These changes add initial support for the construction of remote components. For now, this support is limited to the NodeJS SDK; follow-up changes will implement support for the other SDKs. Remote components are component resources that are constructed and managed by plugins rather than by Pulumi programs. In this sense, they are a bit like cloud resources, and are supported by the same distribution and plugin loading mechanisms and described by the same schema system. The construction of a remote component is initiated by a `RegisterResourceRequest` with the new `remote` field set to `true`. When the resource monitor receives such a request, it loads the plugin that implements the component resource and calls the `Construct` method added to the resource provider interface as part of these changes. This method accepts the information necessary to construct the component and its children: the component's name, type, resource options, inputs, and input dependencies. It is responsible for dispatching to the appropriate component factory to create the component, then returning its URN, resolved output properties, and output property dependencies. The dependency information is necessary to support features such as delete-before-replace, which rely on precise dependency information for custom resources. These changes also add initial support for more conveniently implementing resource providers in NodeJS. The interface used to implement such a provider is similar to the dynamic provider interface (and may be unified with that interface in the future). An example of a NodeJS program constructing a remote component resource also implemented in NodeJS can be found in `tests/construct_component/nodejs`. This is the core of #2430.
2020-09-08 04:33:55 +02:00
props: Inputs, outputs: any, deps: Record<string, Resource[]>,
resolvers: OutputResolvers, err?: Error): Promise<void> {
Initial support for remote component construction. (#5280) These changes add initial support for the construction of remote components. For now, this support is limited to the NodeJS SDK; follow-up changes will implement support for the other SDKs. Remote components are component resources that are constructed and managed by plugins rather than by Pulumi programs. In this sense, they are a bit like cloud resources, and are supported by the same distribution and plugin loading mechanisms and described by the same schema system. The construction of a remote component is initiated by a `RegisterResourceRequest` with the new `remote` field set to `true`. When the resource monitor receives such a request, it loads the plugin that implements the component resource and calls the `Construct` method added to the resource provider interface as part of these changes. This method accepts the information necessary to construct the component and its children: the component's name, type, resource options, inputs, and input dependencies. It is responsible for dispatching to the appropriate component factory to create the component, then returning its URN, resolved output properties, and output property dependencies. The dependency information is necessary to support features such as delete-before-replace, which rely on precise dependency information for custom resources. These changes also add initial support for more conveniently implementing resource providers in NodeJS. The interface used to implement such a provider is similar to the dynamic provider interface (and may be unified with that interface in the future). An example of a NodeJS program constructing a remote component resource also implemented in NodeJS can be found in `tests/construct_component/nodejs`. This is the core of #2430.
2020-09-08 04:33:55 +02:00
// Produce a combined set of property states, starting with inputs and then applying
// outputs. If the same property exists in the inputs and outputs states, the output wins.
const allProps: Record<string, any> = {};
if (outputs) {
Object.assign(allProps, deserializeProperties(outputs));
}
Discriminate unknown values in the JS runtime. (#1414) These changes add support for distinguishing an output property with an unknown value from an output property with a known value that is undefined. In a broad sense, the Pulumi property type system is just JSON with the addition of unknown values. Notably absent, however, are undefined values. As it stands, our marshalers between JavaScript and Pulumi property values treat all undefined JavaScript values as unknown Pulumi values. Unfortunately, this conflates two very different concepts: unknown Pulumi values are intended to represent values of output properties that are unknown at time of preview, _not_ values that are known but undefined. This results in difficulty reasoning about when transforms are run on output properties as well as confusing output in the `diff` view of Pulumi preview (user-specifed undefined values are rendered as unknown values). As it turns out, we already have a way to decide whether or not an Output value is known or not: Output.performApply. These changes rename this property to `isKnown`, clarify its meaning, and take advantage of the result to decide whether or not an Output value should marshal as an unknown Pulumi value. This also allowed these changes to improve the serialization of undefined object keys and array elements s.t. we better match JavaScript to JSON serialization behavior (undefined object keys are omitted; undefined array elements are marshaled as `null`). Fixes https://github.com/pulumi/pulumi-cloud/issues/483.
2018-05-23 23:47:40 +02:00
const label = `resource:${name}[${t}]#...`;
if (!isDryRun() || isLegacyApplyEnabled()) {
for (const key of Object.keys(props)) {
if (!allProps.hasOwnProperty(key)) {
// input prop the engine didn't give us a final value for. Just use the value passed into the resource
// after round-tripping it through serialization. We do the round-tripping primarily s.t. we ensure that
// Output values are handled properly w.r.t. unknowns.
const inputProp = await serializeProperty(label, props[key], new Set());
if (inputProp === undefined) {
continue;
}
allProps[key] = deserializeProperty(inputProp);
}
}
}
resolveProperties(res, resolvers, t, name, allProps, deps, err);
}
/**
* registerResourceOutputs completes the resource registration, attaching an optional set of computed outputs.
*/
export function registerResourceOutputs(res: Resource, outputs: Inputs | Promise<Inputs> | Output<Inputs>) {
// Now run the operation. Note that we explicitly do not serialize output registration with
// respect to other resource operations, as outputs may depend on properties of other resources
// that will not resolve until later turns. This would create a circular promise chain that can
// never resolve.
const opLabel = `monitor.registerResourceOutputs(...)`;
runAsyncResourceOp(opLabel, async () => {
// The registration could very well still be taking place, so we will need to wait for its URN.
// Additionally, the output properties might have come from other resources, so we must await those too.
const urn = await res.urn.promise();
const resolved = await serializeProperties(opLabel, { outputs });
const outputsObj = gstruct.Struct.fromJavaScript(resolved.outputs);
log.debug(`RegisterResourceOutputs RPC prepared: urn=${urn}` +
(excessiveDebugOutput ? `, outputs=${JSON.stringify(outputsObj)}` : ``));
// Fetch the monitor and make an RPC request.
const monitor = getMonitor();
if (monitor) {
const req = new resproto.RegisterResourceOutputsRequest();
req.setUrn(urn);
req.setOutputs(outputsObj);
const label = `monitor.registerResourceOutputs(${urn}, ...)`;
await debuggablePromise(new Promise((resolve, reject) =>
(monitor as any).registerResourceOutputs(req, (err: grpc.ServiceError, innerResponse: any) => {
log.debug(`RegisterResourceOutputs RPC finished: urn=${urn}; ` +
`err: ${err}, resp: ${innerResponse}`);
if (err) {
// If the monitor is unavailable, it is in the process of shutting down or has already
// shut down. Don't emit an error and don't do any more RPCs, just exit.
if (err.code === grpc.status.UNAVAILABLE || err.code === grpc.status.CANCELLED) {
terminateRpcs();
err.message = "Resource monitor is terminating";
}
reject(err);
}
else {
log.debug(`RegisterResourceOutputs RPC finished: urn=${urn}; ` +
`err: ${err}, resp: ${innerResponse}`);
resolve();
}
})), label);
}
}, false);
}
function isAny(o: any): o is any {
return true;
}
/**
* listResourceOutputs returns the resource outputs (if any) for a stack, or an error if the stack
* cannot be found. Resources are retrieved from the latest stack snapshot, which may include
* ongoing updates.
*
* @param stackName Name of stack to retrieve resource outputs for. Defaults to the current stack.
* @param typeFilter A [type
* guard](https://www.typescriptlang.org/docs/handbook/advanced-types.html#user-defined-type-guards)
* that specifies which resource types to list outputs of.
*
* @example
* const buckets = pulumi.runtime.listResourceOutput(aws.s3.Bucket.isInstance);
*/
export function listResourceOutputs<U extends Resource>(
typeFilter?: (o: any) => o is U,
stackName?: string,
): query.AsyncQueryable<ResolvedResource<U>> {
if (typeFilter === undefined) {
typeFilter = isAny;
}
return query
.from(
invoke("pulumi:pulumi:readStackResourceOutputs", {
stackName: stackName || getStack(),
}).then<any[]>(({ outputs }) => utils.values(outputs)),
)
.map<ResolvedResource<U>>(({ type: typ, outputs }) => {
return { ...outputs, __pulumiType: typ };
})
.filter(typeFilter);
}
/**
* resourceChain is used to serialize all resource requests. If we don't do this, all resource operations will be
* entirely asynchronous, meaning the dataflow graph that results will determine ordering of operations. This
* causes problems with some resource providers, so for now we will serialize all of them. The issue
* pulumi/pulumi#335 tracks coming up with a long-term solution here.
*/
let resourceChain: Promise<void> = Promise.resolve();
let resourceChainLabel: string | undefined = undefined;
// runAsyncResourceOp runs an asynchronous resource operation, possibly serializing it as necessary.
function runAsyncResourceOp(label: string, callback: () => Promise<void>, serial?: boolean): void {
// Serialize the invocation if necessary.
if (serial === undefined) {
serial = serialize();
}
const resourceOp: Promise<void> = suppressUnhandledGrpcRejections(debuggablePromise(resourceChain.then(async () => {
if (serial) {
resourceChainLabel = label;
log.debug(`Resource RPC serialization requested: ${label} is current`);
}
return callback();
}), label + "-initial"));
// Ensure the process won't exit until this RPC call finishes and resolve it when appropriate.
const done: () => void = rpcKeepAlive();
const finalOp: Promise<void> = debuggablePromise(
resourceOp.then(() => { done(); }, () => { done(); }),
label + "-final");
// Set up another promise that propagates the error, if any, so that it triggers unhandled rejection logic.
resourceOp.catch((err) => Promise.reject(err));
// If serialization is requested, wait for the prior resource operation to finish before we proceed, serializing
// them, and make this the current resource operation so that everybody piles up on it.
if (serial) {
resourceChain = finalOp;
if (resourceChainLabel) {
log.debug(`Resource RPC serialization requested: ${label} is behind ${resourceChainLabel}`);
}
}
Implement initial Lumi-as-a-library This is the initial step towards redefining Lumi as a library that runs atop vanilla Node.js/V8, rather than as its own runtime. This change is woefully incomplete but this includes some of the more stable pieces of my current work-in-progress. The new structure is that within the sdk/ directory we will have a client library per language. This client library contains the object model for Lumi (resources, properties, assets, config, etc), in addition to the "language runtime host" components required to interoperate with the Lumi resource monitor. This resource monitor is effectively what we call "Lumi" today, in that it's the thing orchestrating plans and deployments. Inside the sdk/ directory, you will find nodejs/, the Node.js client library, alongside proto/, the definitions for RPC interop between the different pieces of the system. This includes existing RPC definitions for resource providers, etc., in addition to the new ones for hosting different language runtimes from within Lumi. These new interfaces are surprisingly simple. There is effectively a bidirectional RPC channel between the Lumi resource monitor, represented by the lumirpc.ResourceMonitor interface, and each language runtime, represented by the lumirpc.LanguageRuntime interface. The overall orchestration goes as follows: 1) Lumi decides it needs to run a program written in language X, so it dynamically loads the language runtime plugin for language X. 2) Lumi passes that runtime a loopback address to its ResourceMonitor service, while language X will publish a connection back to its LanguageRuntime service, which Lumi will talk to. 3) Lumi then invokes LanguageRuntime.Run, passing information like the desired working directory, program name, arguments, and optional configuration variables to make available to the program. 4) The language X runtime receives this, unpacks it and sets up the necessary context, and then invokes the program. The program then calls into Lumi object model abstractions that internally communicate back to Lumi using the ResourceMonitor interface. 5) The key here is ResourceMonitor.NewResource, which Lumi uses to serialize state about newly allocated resources. Lumi receives these and registers them as part of the plan, doing the usual diffing, etc., to decide how to proceed. This interface is perhaps one of the most subtle parts of the new design, as it necessitates the use of promises internally to allow parallel evaluation of the resource plan, letting dataflow determine the available concurrency. 6) The program exits, and Lumi continues on its merry way. If the program fails, the RunResponse will include information about the failure. Due to (5), all properties on resources are now instances of a new Property<T> type. A Property<T> is just a thin wrapper over a T, but it encodes the special properties of Lumi resource properties. Namely, it is possible to create one out of a T, other Property<T>, Promise<T>, or to freshly allocate one. In all cases, the Property<T> does not "settle" until its final state is known. This cannot occur before the deployment actually completes, and so in general it's not safe to depend on concrete resolutions of values (unlike ordinary Promise<T>s which are usually expected to resolve). As a result, all derived computations are meant to use the `then` function (as in `someValue.then(v => v+x)`). Although this change includes tests that may be run in isolation to test the various RPC interactions, we are nowhere near finished. The remaining work primarily boils down to three things: 1) Wiring all of this up to the Lumi code. 2) Fixing the handful of known loose ends required to make this work, primarily around the serialization of properties (waiting on unresolved ones, serializing assets properly, etc). 3) Implementing lambda closure serialization as a native extension. This ongoing work is part of pulumi/pulumi-fabric#311.
2017-08-26 21:07:54 +02:00
}