2c479c172d
* Lift snapshot management out of the engine This PR is a prerequisite for parallelism by addressing a major problem that the engine has to deal with when performing parallel resource construction: parallel mutation of the global snapshot. This PR adds a `SnapshotManager` type that is responsible for maintaining and persisting the current resource snapshot. It serializes all reads and writes to the global snapshot and persists the snapshot to persistent storage upon every write. As a side-effect of this, the core engine no longer needs to know about snapshot management at all; all snapshot operations can be handled as callbacks on deployment events. This will greatly simplify the parallelization of the core engine. Worth noting is that the core engine will still need to be able to read the current snapshot, since it is interested in the dependency graphs contained within. The full implications of that are out of scope of this PR. Remove dead code, Steps no longer need a reference to the plan iterator that created them Fixing various issues that arise when bringing up pulumi-aws Line length broke the build Code review: remove dead field, fix yaml name error Rebase against master, provide implementation of StackPersister for cloud backend Code review feedback: comments on MutationStatus, style in snapshot.go Code review feedback: move SnapshotManager to pkg/backend, change engine to use an interface SnapshotManager Code review feedback: use a channel for synchronization Add a comment and a new test * Maintain two checkpoints, an immutable base and a mutable delta, and periodically merge the two to produce snapshots * Add a lot of tests - covers all of the non-error paths of BeginMutation and End * Fix a test resource provider * Add a few tests, fix a few issues * Rebase against master, fixed merge
69 lines
3.7 KiB
Go
69 lines
3.7 KiB
Go
// Copyright 2016-2018, Pulumi Corporation. All rights reserved.
|
|
|
|
package engine
|
|
|
|
import (
|
|
"io"
|
|
|
|
"github.com/pulumi/pulumi/pkg/resource/deploy"
|
|
"github.com/pulumi/pulumi/pkg/workspace"
|
|
)
|
|
|
|
// SnapshotManager is responsible for maintaining the in-memory representation
|
|
// of the current state of the resource world. It wraps a mutable data structure
|
|
// and, as such, is responsible for serializing access to it.
|
|
//
|
|
// Fundamentally, the Pulumi engine operates on two snapshots:
|
|
// 1. The snapshot used to *create the plan* is considered to be the "Old"
|
|
// snapshot. Resources that exist in this snapshot are considered to exist
|
|
// for the purposes of plan generation. Any modifications to input properties
|
|
// on resources in this snapshot will cause the planner to generate Update steps,
|
|
// while the creation of resources not in this snapshot will cause the planner to
|
|
// generate Create steps.
|
|
// 2. The snapshot modified incrementally as the plan progresses. This snapshot is
|
|
// the snapshot contained within a SnapshotManager. As the planner generates and
|
|
// executes steps, it will inform the SnapshotManager of its intention to modify
|
|
// the snapshot. The SnapshotManager will then record this intent to mutate and serialize
|
|
// it to persistent storage so that the snapshot is still valid in the case of catastrophic
|
|
// engine failure.
|
|
//
|
|
// The engine informs a SnapshotManager of its intent to mutate the snapshot using this protocol:
|
|
// 1. The engine calls `BeginMutation` on a SnapshotManager and passes the `Step` that it
|
|
// intends to execute. The SnapshotManager will inspect this step and mark the resources
|
|
// that will be mutated. It will then persist the modified snapshot and return, giving
|
|
// the engine the green light to continue. The SnapshotManager will return a `SnapshotMutation`.
|
|
// 2. The engine applies its Step.
|
|
// 3. The engine calls `End` on the `SnapshotMutation` returned from `BeginMutation`. The SnapshotManager
|
|
// will then record the mutation to be complete and finalize resource states that changed.
|
|
//
|
|
// The engine also informs of a SnapshotManager of its intention to register resource outputs, through
|
|
// the `RegisterResourceOutputs` call.
|
|
//
|
|
// The SnapshotManager ensures that all writes to the in-memory snapshot are serialized.
|
|
type SnapshotManager interface {
|
|
io.Closer
|
|
|
|
// BeginMutation signals to the SnapshotManager that the planner intends to mutate the global
|
|
// snapshot. It provides the step that it intends to execute. Based on that step, BeginMutation
|
|
// will record this intent in the global snapshot and return a `SnapshotMutation` that, when ended,
|
|
// will complete the transaction.
|
|
BeginMutation(step deploy.Step) (SnapshotMutation, error)
|
|
|
|
// RecordPlugins records the given list of plugins in the manifest, so that Destroy operations
|
|
// operating on this snapshot know which plugins need to be loaded without having to inspect
|
|
// the program.
|
|
RecordPlugins(plugins []workspace.PluginInfo) error
|
|
|
|
// RegisterResourceOutputs registers the outputs of a step with the snapshot, once a step has completed. The
|
|
// engine has populated the `New` resource of the given step with the Outputs that it would like to save in
|
|
// the snapshot.
|
|
RegisterResourceOutputs(step deploy.Step) error
|
|
}
|
|
|
|
// SnapshotMutation represents an outstanding mutation that is yet to be completed. When the engine completes
|
|
// a mutation, it must call `End` in order to record the successful completion of the mutation.
|
|
type SnapshotMutation interface {
|
|
// End terminates the transaction and commits the results to the snapshot, returning an error if this
|
|
// failed to complete.
|
|
End(step deploy.Step) error
|
|
}
|