From a5a818b80252a46c712db2953a9cc704a5d3b54e Mon Sep 17 00:00:00 2001 From: Pat Gavlin Date: Fri, 27 Aug 2021 10:55:06 -0700 Subject: [PATCH] [developer-docs] Add links + a broad arch diagram (#7856) - Add links within the resource registration docs - Add a broad-strokes architectural diagram to the overview - Add a `clean` target to the Makefile - Expand the "Building" docs a bit --- developer-docs/Makefile | 5 +- developer-docs/architecture/overview.md | 21 ++++-- developer-docs/architecture/pulumi-cli.svg | 68 +++++++++++++++++++ developer-docs/architecture/pulumi-cli.uml | 37 ++++++++++ .../architecture/resource-registration.md | 40 ++++++----- developer-docs/build.md | 24 ++++++- developer-docs/conf.py | 6 ++ .../providers/implementers-guide.md | 2 +- 8 files changed, 176 insertions(+), 27 deletions(-) create mode 100644 developer-docs/architecture/pulumi-cli.svg create mode 100644 developer-docs/architecture/pulumi-cli.uml diff --git a/developer-docs/Makefile b/developer-docs/Makefile index 7ce1e0500..1ea5b6649 100644 --- a/developer-docs/Makefile +++ b/developer-docs/Makefile @@ -9,7 +9,7 @@ html: Makefile graphics pdf: Makefile graphics @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) -.PHONY: graphics Makefile +.PHONY: graphics clean Makefile SVG_FILES = \ providers/resource-lifecycle.svg \ @@ -26,3 +26,6 @@ SVG_FILES = \ plantuml -tsvg $< graphics: Makefile $(SVG_FILES) + +clean: + rm -rf $(BUILDDIR)/* diff --git a/developer-docs/architecture/overview.md b/developer-docs/architecture/overview.md index 043903b2e..3097c8a5a 100644 --- a/developer-docs/architecture/overview.md +++ b/developer-docs/architecture/overview.md @@ -2,11 +2,22 @@ Broadly speaking, Pulumi is composed of five components: -1. The deployment engine -2. State storage backends -3. Language SDKs -4. Resource providers -5. Package schemas and code generators +1. [The deployment engine](#the-deployment-engine) +2. [State storage backends](#state-storage-backends) +3. [Language SDKs](#language-sdks) +4. [Resource providers](#resource-providers) +5. [Package schemas and code generators](#package-schemas-and-code-generators) + +These components interact to provide the feature set exposed by the Pulumi CLI and SDKs, including desired-state +deployments using standard programming languages, remote state storage and secret encryption, and the ability to bridge +the gap between existing and Pulumi-managed infrastructure. + +These components are composed like so: + +![Pulumi CLI](./pulumi-cli.svg) + +In most cases, the language plugin, CLI, and resource providers will all live in separate processes, and each instance +of a resource provider will live in its own process. ## The Deployment Engine diff --git a/developer-docs/architecture/pulumi-cli.svg b/developer-docs/architecture/pulumi-cli.svg new file mode 100644 index 000000000..a708785e2 --- /dev/null +++ b/developer-docs/architecture/pulumi-cli.svg @@ -0,0 +1,68 @@ +Language PluginPulumi CLIResource ProvidersgRPCLanguage SDKgRPCDeployment engineState storage backendCode generatorsgRPCPackage schemasMulti-language componentsCloud resources \ No newline at end of file diff --git a/developer-docs/architecture/pulumi-cli.uml b/developer-docs/architecture/pulumi-cli.uml new file mode 100644 index 000000000..23f5a7bbc --- /dev/null +++ b/developer-docs/architecture/pulumi-cli.uml @@ -0,0 +1,37 @@ +@startuml + +node "Language Plugin" as plugin { + interface gRPC as langhost + + component "Language SDK" as sdk +} + +node "Pulumi CLI" as cli { + interface gRPC as resmon + + component "Deployment engine" as engine + component "State storage backend" as backend + component "Code generators" as codegen + + resmon - engine + + sdk -down-> resmon + engine -up-> langhost + engine -right-> backend +} + +node "Resource Providers" as providers { + interface gRPC as provider + + component "Package schemas" as schemas + component "Multi-language components" as mlcs + component "Cloud resources" as resources + + provider - mlcs + provider - resources + + engine -down-> provider + mlcs -up-> resmon +} + +@enduml diff --git a/developer-docs/architecture/resource-registration.md b/developer-docs/architecture/resource-registration.md index 54f445b2b..236f45eed 100644 --- a/developer-docs/architecture/resource-registration.md +++ b/developer-docs/architecture/resource-registration.md @@ -13,12 +13,12 @@ methods to perform it. After the action completes, the engine returns the new st program. Although we typically treat the engine as a single unit, in the case of resource registrations it helps to break it -down into a few of its internal components: the resource monitor, the step generator, and the step executor. Each +down into a few of its internal components: the [resource monitor], the [step generator], and the [step executor]. Each of these components is a participant in the response to a `RegisterResourceRequest`. ## The Resource Monitor -The resource monitor provides serves the `ResourceMonitor` gRPC interface, and provides a shim between language SDKs +The _resource monitor_ provides serves the `ResourceMonitor` gRPC interface, and provides a shim between language SDKs and the rest of the engine. There is a single resource monitor per deployment. As the engine's feature set has grown, the resource monitor has taken on responsibilities beyond its original use as a simple marshaling/unmarshaling layer. It is now responsible for handling default providers (providers for resource registrations that do not reference a @@ -32,7 +32,7 @@ When the resource monitor receives a resource registration, it does the followin provider. 3. If the request is for a multi-language component, dispatches a `Construct` call to the component's provider and waits for the result. -4. If the request is not for a multi-langauge component, sends a `RegisterResourceEvent` to the step generator and +4. If the request is not for a multi-langauge component, sends a `RegisterResourceEvent` to the [step generator] and waits for the result. 5. Marshals the result of the `Construct` call or `RegisterResourceEvent` from the engine's internal representation to the gRPC wire format and returns from the RPC call. @@ -98,7 +98,7 @@ Note that this request _does_ contain a value for the `provider` field. ## The Step Generator -The step generator is responsible for processing `RegisterResourceEvent`s from the resource monitor. The generator +The _step generator_ is responsible for processing `RegisterResourceEvent`s from the [resource monitor]. The generator implements the core logic that determines which actions to take in order to drive the actual state of a resource to its desired state as represented by the input properties in its `RegisterResourceEvent`. In order to simplify reasoning about the actual state of a stack's resources, the step generator processes `RegisterResourceEvent`s @@ -116,32 +116,32 @@ When the step generator receives a `RegisterResourceEvent`, it does the followin states. 3. Pre-process input properties for ignored changes by setting any properties mentioned in the event's ignore changes list to their old value (if any) -4. If the event indicates that the resource should be imported, issue an `ImportStep` to the step executor and return. +4. If the event indicates that the resource should be imported, issue an `ImportStep` to the [step exectuor] and return. 5. Call the resource's provider's `Check` method with the event's input properties and the resource's existing inputs, if any. The existing inputs may be used by the provider to repopulate default values for input properties that are automatically generated when the resource is created but should not be changed with subsequent updates (e.g. automatically generated names). `Check` returns a pre-processed bag of input values to be used with later calls to `Diff`, `Create`, and `Update`. 6. Invoke any analyzers for the stack to perform additional validation of the resource's input properties. -7. If the resource has no existing state, it is being created. Issue a `CreateStep` to the step executor and return. +7. If the resource has no existing state, it is being created. Issue a `CreateStep` to the [step exectuor] and return. 8. Diff the resource in order to determine whether it must be updated, replaced, delete-before-replaced, or has no changes. Diffing is covered in detail later on, but typically consists of calling the reosource's provider's `Diff` method with the checked inputs from step 5. -9. If the resource has no changes, issue a `SameStep` to the step executor and return. -10. If the resource is not being replaced, issue an `UpdateStep` to the step executor and return. +9. If the resource has no changes, issue a `SameStep` to the [step exectuor] and return. +10. If the resource is not being replaced, issue an `UpdateStep` to the [step exectuor] and return. 11. If the resource is being replaced, call the resource's provider's `Check` method again, but with no existing inputs. This call ensures that the input properties used to create the replacement resource do not reuse generated defaults from the existing resource. 12. If the replacement resource is being created before the original is deleted (a normal replacement), issue a - `CreateStep` and a `DeleteStep` to the step executor and return. + `CreateStep` and a `DeleteStep` to the [step exectuor] and return. 13. At this point, the resource must be deleted before its replacement is created (this is the "delete-before-replace" case). Calculate the set of dependent resources that must be deleted prior to deleting the resource being replaced. The details of this calculation are covered in a later section. Once the set of deletions has been calculated, - issue a sequence of `DeleteStep`s followed by a single `CreateStep` to the step executor. + issue a sequence of `DeleteStep`s followed by a single `CreateStep` to the [step exectuor]. Note that all steps that are issued to the step generator are fire-and-forget. Once steps have been issues, the step -generator moves on to process the next `RegisterResourceEvent`. It is the responsibility of the step executor to -communicate the results of each step back to the resource monitor. +generator moves on to process the next `RegisterResourceEvent`. It is the responsibility of the [step exectuor] to +communicate the results of each step back to the [resource monitor]. Once the Pulumi program has exited, the step generator determines which existing resources must be deleted by taking the difference between the set of registered resources and the set of existing resources. These resources are scheduled @@ -149,7 +149,7 @@ for deletion by first sorting the list of resources to delete using the topologi grapth, then decomposing the list into a list of lists where each list can be executed in parallel but a previous list must be executed to completion before advancing to the next list. -In lieu of tracking per-step dependencies and orienting the step executor around these dependencies, this approach +In lieu of tracking per-step dependencies and orienting the [step exectuor] around these dependencies, this approach provides a conservative approximation of what deletions can safely occur in parallel. The insight here is that the resource dependency graph is a partially-ordered set and all partially-ordered sets can be easily decomposed into antichains--subsets of the set that are all not comparable to one another (in this definition, "not comparable" @@ -170,7 +170,7 @@ Translated to a resource dependency graph: The resulting list of antichains is a list of list of delete steps that can be safely executed in parallel. Since deletes must be processed in reverse order (so that resources are not deleted prior to their dependents), the step -generator reverses the list and then issues each sublist to the step executor. +generator reverses the list and then issues each sublist to the [step exectuor]. ### Resource Diffing @@ -214,12 +214,12 @@ In these cases, neither B nor D would need to be deleted before A could be delet ## The Step Executor -The step executor is responsible for executing sequences of steps (called "chains") that perform the resource actions -for a deployment. These chains are issued by the step generator, and most often consist of a single step. While the +The _step executor_ is responsible for executing sequences of steps (called "chains") that perform the resource actions +for a deployment. These chains are issued by the [step generator], and most often consist of a single step. While the steps the make up a chain must be performed serially, chains may be executed in parallel. The step executor uses a (potentially infinite) pool of workers to execute steps. Once a step completes, the step executor communicates its -results to the resource monitor if necessary. If a step fails, the step executor notes the failure and cancels the -deployment. Once the Pulumi program has exited and the step generator has issued all required deletions, the step +results to the [resource monitor] if necessary. If a step fails, the step executor notes the failure and cancels the +deployment. Once the Pulumi program has exited and the [step generator] has issued all required deletions, the step executor waits for all outstanding steps to complete and then returns. ## Example Resource Registration Sequences @@ -253,3 +253,7 @@ The diagram below illustrates the sequence of events that occurs when a multi-la registration of the component's children is elided. ![Multi-language component construction](./construct.svg) + +[resource monitor]: #the-resource-monitor +[step generator]: #the-step-generator +[step executor]: #the-step-executor diff --git a/developer-docs/build.md b/developer-docs/build.md index 042218714..896d63c36 100644 --- a/developer-docs/build.md +++ b/developer-docs/build.md @@ -1,9 +1,13 @@ # Building the Docs +This documentation is generated using [Sphinx] and authored in Markdown. Markdown support for [Sphinx] is provided by +[MyST]. [MyST] provides a number of small syntax extensions to support declaring ReStructuredText directives; see +[the MyST syntax guide](https://myst-parser.readthedocs.io/en/latest/syntax/syntax.html) for details. + In order to build the devloper documentation: -1. Install [PlantUML](https://plantuml.com). On macOS, this can be done via `brew install plantuml`. -2. Install the requirements for Sphinx: +1. Install [PlantUML]. On macOS, this can be done via `brew install plantuml`. +2. Install the requirements for [Sphinx]: ```bash $ pip install requirements.txt @@ -17,3 +21,19 @@ In order to build the devloper documentation: This will regenerate any out-of-date SVGs and build the a local version of the HTML documentation. The documentation can also be built from the repository root by running `make developer_docs`. + +Note that Sphinx doesn't do a great job of rebuilding output files if only the table-of-contents has changed. If you +change the table of contents, you may need to clean the output directory in order to see the effects of your changes: + + ```bash + $ make clean + ``` + +## Notes on Style + +- Do use appropriate links wherever possible. Learn to use [header anchors](https://myst-parser.readthedocs.io/en/latest/syntax/optional.html#auto-generated-header-anchors). +- If a particular link destination is referenced multiple times, prefer [shortcut reference links](https://spec.commonmark.org/0.29/#shortcut-reference-link). + +[Sphinx]: https://www.sphinx-doc.org/en/master/index.html +[MyST]: https://myst-parser.readthedocs.io/en/latest/index.html +[PlantUML]: https://plantuml.com diff --git a/developer-docs/conf.py b/developer-docs/conf.py index 582fcc83e..3118a0c5e 100644 --- a/developer-docs/conf.py +++ b/developer-docs/conf.py @@ -53,3 +53,9 @@ html_theme = 'sphinx_rtd_theme' # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". html_static_path = ['_static'] + + +# -- MyST configuration ------------------------------------------------------ + +# Enable auto-generated header anchors. +myst_heading_anchors = 5 diff --git a/developer-docs/providers/implementers-guide.md b/developer-docs/providers/implementers-guide.md index 878410857..f9e49d003 100644 --- a/developer-docs/providers/implementers-guide.md +++ b/developer-docs/providers/implementers-guide.md @@ -102,7 +102,7 @@ contents). #### Resource References -A `ResourceReference` represents a reference to a [Pulumi resource](#Resources). Although +A `ResourceReference` represents a reference to a [Pulumi resource](#resources). Although all that is necessary to uniquely identify a resource is its URN, a `ResourceReference` also carries the resource's ID (if it is a [custom resource](#custom-resources)) and the version of the provider that manages the resource. If the contents of the referenced