From 3a545de34aa6875a8ac99c03f525e0e3908e519e Mon Sep 17 00:00:00 2001 From: Pat Gavlin Date: Thu, 7 Oct 2021 12:14:37 -0700 Subject: [PATCH] [developer-docs] Document resource import. (#8137) These changes add a page to the developer docs that describes the design and implementation of the `import` resource option and the `pulumi import` command. Co-authored-by: Levi Blackstone --- developer-docs/Makefile | 3 +- developer-docs/architecture/import.md | 152 ++++++++++++++++++ developer-docs/architecture/pulumi-import.svg | 38 +++++ developer-docs/architecture/pulumi-import.uml | 28 ++++ developer-docs/index.md | 1 + 5 files changed, 221 insertions(+), 1 deletion(-) create mode 100644 developer-docs/architecture/import.md create mode 100644 developer-docs/architecture/pulumi-import.svg create mode 100644 developer-docs/architecture/pulumi-import.uml diff --git a/developer-docs/Makefile b/developer-docs/Makefile index 271de9a65..9da9b9e72 100644 --- a/developer-docs/Makefile +++ b/developer-docs/Makefile @@ -20,7 +20,8 @@ SVG_FILES = \ architecture/import.svg \ architecture/replace.svg \ architecture/same.svg \ - architecture/update.svg + architecture/update.svg \ + architecture/pulumi-import.svg %.svg: %.uml plantuml -tsvg $< diff --git a/developer-docs/architecture/import.md b/developer-docs/architecture/import.md new file mode 100644 index 000000000..e034a09c0 --- /dev/null +++ b/developer-docs/architecture/import.md @@ -0,0 +1,152 @@ +# Importing Resources + +There are a variety of scenarios that require the ability for users to import existing +resources for management by Pulumi. For example: + +- Migrating from manually-managed resources to IaC +- Migrating from other IaC platforms to Pulumi +- Migrating resources between Pulumi stacks + +At a minimum, importing a resource involves adding the resource's state to the destination +stack's statefile. Once the resource has been added to the stack, the Pulumi CLI is able +to manage the resource like any other. In order to do anything besides delete the resource, +however, the user must also add a definition for the resource to their Pulumi program. + +Both of the import approaches used by Pulumi aim to prevent the accidental modification or +deletion of a resource being imported. Though the user experiences of these approaches are +quite different, they share a common principle: at the point at which a resource is +successfully imported, the stack's Pulumi program must contain a definition for the +resource that accurately describes its current state (i.e. there are no differences +between the state described in the program and the actual state of the imported resource). + +## `import` resource option + +The oldest method supported of importing resources into a stack is the [`import`](https://www.pulumi.com/docs/intro/concepts/resources/#import) +[resource option](https://www.pulumi.com/docs/intro/concepts/resources/#options). When set, +this option specifies the ID of an existing resource to import into the stack. The exact +behavior of this option depends on the current state of the resource within the destination +stack: + +1. If the resource does not exist, it is imported +2. If the resource exists and has the same `ID` or `ImportID`, the resource is treated + like any other resource +3. Otherwise, the current resource is deleted and replaced by importing the resource with + the specified ID + +The trickiest of these three situations is (2). This state transition is intended to allow +users to import a resource and then continue to make changes to their program without +requiring that they remove the resource option. For example, this allows a user to import +a resource in one `pulumi up`, then successfully run another `pulumi up` without removing +the `import` option from their program and without attempting to import the resource a +second time. + +As mentioned in [the introduction](#importing-resources), the `import` resource option +requires that the desired state described by Pulumi program for a resource being imported +matches the actual state of the resource as returned by the provider. More precisely, +given a resource `R` of type `T` with import ID `X` and the resource inputs present in the +Pulumi program `Iₚ`, the engine performs the following sequence of operations: + +1. Fetch the current inputs `Iₐ` and state `Sₐ` for the resource of type `T` with ID `X` + from its provider by calling the provider's [`Read` method](providers/implementers-guide.md#read). + If the provider does not return a value for `Iₐ`, the provider does not support + importing resources and the import fails. +2. Process the [`ignoreChanges` resource option](https://www.pulumi.com/docs/intro/concepts/resources/#import) + by copying the value for any ignored input property from `Iₐ` to `Iₚ`. +3. Validate the resource's inputs and apply any programmatic defaults by passing `Iₚ` and + `Iₐ` to the provider's [`Check` method](providers/implementers-guide.md#check). Let + `Iₖ` be the checked inputs; these inputs form the resource's desired state. +4. Check for differences between `Iₖ` and `Sₐ` by calling the provider's [`Diff` method](providers/implementers-guide.md#diff). + If the provider reports any differences, the import either succeeds with a warning (in + the case of a preview) or fails with an error (in the case of an update). + +If all of these steps succeed, the user is left with a definition for `R` in their Pulumi +program and the statefile of the updated stack that do not differ. + +### Technical Note + +Although the "no diffs" requirement is intended to prevent surprise, it also accommodates +a technical limitation of the Pulumi engine. In order to actually perform the diff--an +operation that is required whether or not the user is permitted to describe a desired +state for the imported resource that differs from its actual state--the engine must fetch +the resource's current imports and state from its provider. In order for this state to +affect the steps the engine issues for the resources, the state would need to be fetched +during or prior to the point at which the resource's registration reaches the [step +generator](resource-registration.md#the-step-generator). In the former case, this would +cause the engine to spend an unacceptable amount of time in the step generator, as it +processes resource registrations serially. In the latter case, the user experience would +likely be negatively affected by a lack of output from the Pulumi CLI, which only displays +the status of [steps](resource-registration.md#the-step-generator). In order to address +these issues, the operations described above happen in a dedicated `ImportStep` that is +run by the [step executor](resource-registration.md#the-step-executor). + +![Import diagram](./import.svg) + +## `pulumi import` + +The second, newer method of importing resources into a stack is the [`pulumi import` +command](https://www.pulumi.com/docs/reference/cli/pulumi_import/). This command accepts a +list of import specs to import, imports the resources into the destination stack, and +generates definitions for the resources in the language used by the stack's Pulumi program. +Each import spec is at least a type token, name, and ID, but may also specify a parent URN, +provider reference, and package version. + +During a `pulumi import`, given a resource `R` of type `T` with import ID X and an empty +set of input properties `Iₚ`, the engine performs the following sequence of operations: + +1. Fetch the current inputs `Iₐ` and state `Sₐ` for the resource of type `T` with ID `X` + from its provider by calling the provider's [`Read` method](providers/implementers-guide.md#read). + If the provider does not return a value for `Iₐ`, the provider does not support + importing resources and the import fails. +2. Fetch the schema for resources of type `T` from the provider. If the provider is not + schematized or if `T` has no schema, the import fails. +3. Copy the value of each required input property defined in the schema for `T` from `Iₐ` + to `Iₚ`. +4. Validate the resource's inputs and apply any programmatic defaults by passing `Iₚ` and + `Iₐ` to the provider's [`Check` method](providers/implementers-guide.md#check). Let + `Iₖ` be the checked inputs; these inputs form the resource's desired state. +5. Check for differences between `Iₖ` and `Sₐ` by calling the provider's [`Diff` method](providers/implementers-guide.md#diff). + If the provider reports any differences, the values of the differing properties are + copied from `Sₐ` to `Iₚ`. This is intended to produce the smallest valid set of inputs + necessary to avoid diffs. This does not use a fixed-point algorithm because there is no + guarantee that the values copied from `Sₐ` are in fact valid (state and inputs with the + same property paths may have different types and validation rules) and there is no + guarantee that such an algorithm would terminate (TF bridge providers have had bugs that + cause persistent diffs, which can only be worked around with `ignoreChanges`). + +If all of these steps succeed, the user is left with a definition for `R` in the statefile +of the updated stack that do not differ. The Pulumi CLI then passes the inputs `Iₚ` stored +in the statefile to the import code generator. The import code generator converts the values +present in `Iₚ` into an equivalent PCL representation of `R`'s desired state, then passes +the PCL to a language-specific code generator to emit a representation of `R`'s desired +state in the language used by the destination stack's Pulumi program. The user can then +copy the generated definition into their Pulumi program. + +Graphically, the import process looks something like this: + +![`pulumi import` diagram](./pulumi-import.svg) + +### Challenges + +The primary challenge in generating appropriate code for `pulumi import` lies in +determining exactly what the input values for a particular resource should be. In many +providers, it is not necessarily possible to accurately recover a resource's inputs from +its state. This observation led to the diff-oriented approach described above, where the +importer begins with an extremely minimal set of inputs and attempts to derive the actual +inputs from the results of a call to the provider's [`Diff` method](providers/implementers-guide.md#diff). +Unfortunately, the results are not always satisfactory, and the relatively small set of +inputs present in the generated code can make it difficult for users to determine what +inputs they _actually_ need to pass to the resource to describe its current state. + +A few other approaches might be: + +- Emit no properties at all; just appropriate constructor calls. This will almost always + emit code that does not compile or run, as nearly every resource has at least one + required property. +- Copy the value for every input property present in a resource's schema from its state. + This risks emitting code that does not compile due to differences in types between + inputs and outputs, and also risks emitting code that does not work at runtime due to + conflicts between mutually-exclusive properties (these are common for TF-based + resources, for example). + +It is likely that some mix of approaches is necessary in order to arrive at a satisfactory +solution, as none of the above solutions seems universally "correct". diff --git a/developer-docs/architecture/pulumi-import.svg b/developer-docs/architecture/pulumi-import.svg new file mode 100644 index 000000000..e0a317206 --- /dev/null +++ b/developer-docs/architecture/pulumi-import.svg @@ -0,0 +1,38 @@ +Engine`pulumi import` Command`pulumi import` CommandPCL ConverterPCL ConverterLanguage Code GeneratorLanguage Code GeneratorImport DriverImport DriverStep ExecutorStep ExecutorResource ProviderResource Providerengine.Import(import specs)[]ImportStep(import spec)All steps run in parallel.ReadRequest(type, id)ReadResponse(current inputs, current state)CheckRequest(type, inputs, current inputs)CheckResponse(inputs', failures)DiffRequest(type, inputs', current state, options)DiffResponse(diffs)done(current states)done(current states)convert(current states)PCL resource definitionsgenerateProgram(PCL resource definitions)Generated resource definitions \ No newline at end of file diff --git a/developer-docs/architecture/pulumi-import.uml b/developer-docs/architecture/pulumi-import.uml new file mode 100644 index 000000000..7f4089117 --- /dev/null +++ b/developer-docs/architecture/pulumi-import.uml @@ -0,0 +1,28 @@ +@startuml +participant "`pulumi import` Command" order 10 +participant "PCL Converter" order 13 +participant "Language Code Generator" order 16 +box "Engine" +participant "Import Driver" order 20 +participant "Step Executor" order 25 +end box +participant "Resource Provider" order 30 + +"`pulumi import` Command" -> "Import Driver" ++ : engine.Import(import specs) +"Import Driver" -> "Step Executor" --++ : []ImportStep(import spec) +note left + All steps run in parallel. +end note +"Step Executor" -> "Resource Provider" ++ : ReadRequest(type, id) +"Step Executor" <- "Resource Provider" -- : ReadResponse(current inputs, current state) +"Step Executor" -> "Resource Provider" ++ : CheckRequest(type, inputs, current inputs) +"Step Executor" <- "Resource Provider" -- : CheckResponse(inputs', failures) +"Step Executor" -> "Resource Provider" ++ : DiffRequest(type, inputs', current state, options) +"Step Executor" <- "Resource Provider" -- : DiffResponse(diffs) +"Import Driver" <- "Step Executor" -- : done(current states) +"`pulumi import` Command" <- "Import Driver" -- : done(current states) +"`pulumi import` Command" -> "PCL Converter" ++ : convert(current states) +"`pulumi import` Command" <- "PCL Converter" -- : PCL resource definitions +"`pulumi import` Command" -> "Language Code Generator" ++ : generateProgram(PCL resource definitions) +"`pulumi import` Command" <- "Language Code Generator" -- : Generated resource definitions +@enduml diff --git a/developer-docs/index.md b/developer-docs/index.md index ca68c118d..36514ef54 100644 --- a/developer-docs/index.md +++ b/developer-docs/index.md @@ -17,6 +17,7 @@ architecture/overview architecture/resource-registration architecture/deployment-schema architecture/type-system +architecture/import ``` ```{toctree}