pulumi/pkg/resource/deploy/snapshot.go

119 lines
4.6 KiB
Go
Raw Normal View History

2018-05-22 21:43:36 +02:00
// Copyright 2016-2018, Pulumi Corporation.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
Begin resource modeling and planning This change introduces a new package, pkg/resource, that will form the foundation for actually performing deployment plans and applications. It contains the following key abstractions: * resource.Provider is a wrapper around the CRUD operations exposed by underlying resource plugins. It will eventually defer to resource.Plugin, which itself defers -- over an RPC interface -- to the actual plugin, one per package exposing resources. The provider will also understand how to load, cache, and overall manage the lifetime of each plugin. * resource.Resource is the actual resource object. This is created from the overall evaluation object graph, but is simplified. It contains only serializable properties, for example. Inter-resource references are translated into serializable monikers as part of creating the resource. * resource.Moniker is a serializable string that uniquely identifies a resource in the Mu system. This is in contrast to resource IDs, which are generated by resource providers and generally opaque to the Mu system. See marapongo/mu#69 for more information about monikers and some of their challenges (namely, designing a stable algorithm). * resource.Snapshot is a "snapshot" taken from a graph of resources. This is a transitive closure of state representing one possible configuration of a given environment. This is what plans are created from. Eventually, two snapshots will be diffable, in order to perform incremental updates. One way of thinking about this is that a snapshot of the old world's state is advanced, one step at a time, until it reaches a desired snapshot of the new world's state. * resource.Plan is a plan for carrying out desired CRUD operations on a target environment. Each plan consists of zero-to-many Steps, each of which has a CRUD operation type, a resource target, and a next step. This is an enumerator because it is possible the plan will evolve -- and introduce new steps -- as it is carried out (hence, the Next() method). At the moment, this is linearized; eventually, we want to make this more "graph-like" so that we can exploit available parallelism within the dependencies. There are tons of TODOs remaining. However, the `mu plan` command is functioning with these new changes -- including colorization FTW -- so I'm landing it now. This is part of marapongo/mu#38 and marapongo/mu#41.
2017-02-17 21:31:48 +01:00
package deploy
Begin resource modeling and planning This change introduces a new package, pkg/resource, that will form the foundation for actually performing deployment plans and applications. It contains the following key abstractions: * resource.Provider is a wrapper around the CRUD operations exposed by underlying resource plugins. It will eventually defer to resource.Plugin, which itself defers -- over an RPC interface -- to the actual plugin, one per package exposing resources. The provider will also understand how to load, cache, and overall manage the lifetime of each plugin. * resource.Resource is the actual resource object. This is created from the overall evaluation object graph, but is simplified. It contains only serializable properties, for example. Inter-resource references are translated into serializable monikers as part of creating the resource. * resource.Moniker is a serializable string that uniquely identifies a resource in the Mu system. This is in contrast to resource IDs, which are generated by resource providers and generally opaque to the Mu system. See marapongo/mu#69 for more information about monikers and some of their challenges (namely, designing a stable algorithm). * resource.Snapshot is a "snapshot" taken from a graph of resources. This is a transitive closure of state representing one possible configuration of a given environment. This is what plans are created from. Eventually, two snapshots will be diffable, in order to perform incremental updates. One way of thinking about this is that a snapshot of the old world's state is advanced, one step at a time, until it reaches a desired snapshot of the new world's state. * resource.Plan is a plan for carrying out desired CRUD operations on a target environment. Each plan consists of zero-to-many Steps, each of which has a CRUD operation type, a resource target, and a next step. This is an enumerator because it is possible the plan will evolve -- and introduce new steps -- as it is carried out (hence, the Next() method). At the moment, this is linearized; eventually, we want to make this more "graph-like" so that we can exploit available parallelism within the dependencies. There are tons of TODOs remaining. However, the `mu plan` command is functioning with these new changes -- including colorization FTW -- so I'm landing it now. This is part of marapongo/mu#38 and marapongo/mu#41.
2017-02-17 21:31:48 +01:00
import (
"crypto/sha256"
"fmt"
"time"
"github.com/pkg/errors"
"github.com/pulumi/pulumi/pkg/resource"
"github.com/pulumi/pulumi/pkg/workspace"
Begin resource modeling and planning This change introduces a new package, pkg/resource, that will form the foundation for actually performing deployment plans and applications. It contains the following key abstractions: * resource.Provider is a wrapper around the CRUD operations exposed by underlying resource plugins. It will eventually defer to resource.Plugin, which itself defers -- over an RPC interface -- to the actual plugin, one per package exposing resources. The provider will also understand how to load, cache, and overall manage the lifetime of each plugin. * resource.Resource is the actual resource object. This is created from the overall evaluation object graph, but is simplified. It contains only serializable properties, for example. Inter-resource references are translated into serializable monikers as part of creating the resource. * resource.Moniker is a serializable string that uniquely identifies a resource in the Mu system. This is in contrast to resource IDs, which are generated by resource providers and generally opaque to the Mu system. See marapongo/mu#69 for more information about monikers and some of their challenges (namely, designing a stable algorithm). * resource.Snapshot is a "snapshot" taken from a graph of resources. This is a transitive closure of state representing one possible configuration of a given environment. This is what plans are created from. Eventually, two snapshots will be diffable, in order to perform incremental updates. One way of thinking about this is that a snapshot of the old world's state is advanced, one step at a time, until it reaches a desired snapshot of the new world's state. * resource.Plan is a plan for carrying out desired CRUD operations on a target environment. Each plan consists of zero-to-many Steps, each of which has a CRUD operation type, a resource target, and a next step. This is an enumerator because it is possible the plan will evolve -- and introduce new steps -- as it is carried out (hence, the Next() method). At the moment, this is linearized; eventually, we want to make this more "graph-like" so that we can exploit available parallelism within the dependencies. There are tons of TODOs remaining. However, the `mu plan` command is functioning with these new changes -- including colorization FTW -- so I'm landing it now. This is part of marapongo/mu#38 and marapongo/mu#41.
2017-02-17 21:31:48 +01:00
)
// Snapshot is a view of a collection of resources in an stack at a point in time. It describes resources; their
Begin resource modeling and planning This change introduces a new package, pkg/resource, that will form the foundation for actually performing deployment plans and applications. It contains the following key abstractions: * resource.Provider is a wrapper around the CRUD operations exposed by underlying resource plugins. It will eventually defer to resource.Plugin, which itself defers -- over an RPC interface -- to the actual plugin, one per package exposing resources. The provider will also understand how to load, cache, and overall manage the lifetime of each plugin. * resource.Resource is the actual resource object. This is created from the overall evaluation object graph, but is simplified. It contains only serializable properties, for example. Inter-resource references are translated into serializable monikers as part of creating the resource. * resource.Moniker is a serializable string that uniquely identifies a resource in the Mu system. This is in contrast to resource IDs, which are generated by resource providers and generally opaque to the Mu system. See marapongo/mu#69 for more information about monikers and some of their challenges (namely, designing a stable algorithm). * resource.Snapshot is a "snapshot" taken from a graph of resources. This is a transitive closure of state representing one possible configuration of a given environment. This is what plans are created from. Eventually, two snapshots will be diffable, in order to perform incremental updates. One way of thinking about this is that a snapshot of the old world's state is advanced, one step at a time, until it reaches a desired snapshot of the new world's state. * resource.Plan is a plan for carrying out desired CRUD operations on a target environment. Each plan consists of zero-to-many Steps, each of which has a CRUD operation type, a resource target, and a next step. This is an enumerator because it is possible the plan will evolve -- and introduce new steps -- as it is carried out (hence, the Next() method). At the moment, this is linearized; eventually, we want to make this more "graph-like" so that we can exploit available parallelism within the dependencies. There are tons of TODOs remaining. However, the `mu plan` command is functioning with these new changes -- including colorization FTW -- so I'm landing it now. This is part of marapongo/mu#38 and marapongo/mu#41.
2017-02-17 21:31:48 +01:00
// IDs, names, and properties; their dependencies; and more. A snapshot is a diffable entity and can be used to create
// or apply an infrastructure deployment plan in order to make reality match the snapshot state.
type Snapshot struct {
Manifest Manifest // a deployment manifest of versions, checksums, and so on.
Resources []*resource.State // fetches all resources and their associated states.
Begin resource modeling and planning This change introduces a new package, pkg/resource, that will form the foundation for actually performing deployment plans and applications. It contains the following key abstractions: * resource.Provider is a wrapper around the CRUD operations exposed by underlying resource plugins. It will eventually defer to resource.Plugin, which itself defers -- over an RPC interface -- to the actual plugin, one per package exposing resources. The provider will also understand how to load, cache, and overall manage the lifetime of each plugin. * resource.Resource is the actual resource object. This is created from the overall evaluation object graph, but is simplified. It contains only serializable properties, for example. Inter-resource references are translated into serializable monikers as part of creating the resource. * resource.Moniker is a serializable string that uniquely identifies a resource in the Mu system. This is in contrast to resource IDs, which are generated by resource providers and generally opaque to the Mu system. See marapongo/mu#69 for more information about monikers and some of their challenges (namely, designing a stable algorithm). * resource.Snapshot is a "snapshot" taken from a graph of resources. This is a transitive closure of state representing one possible configuration of a given environment. This is what plans are created from. Eventually, two snapshots will be diffable, in order to perform incremental updates. One way of thinking about this is that a snapshot of the old world's state is advanced, one step at a time, until it reaches a desired snapshot of the new world's state. * resource.Plan is a plan for carrying out desired CRUD operations on a target environment. Each plan consists of zero-to-many Steps, each of which has a CRUD operation type, a resource target, and a next step. This is an enumerator because it is possible the plan will evolve -- and introduce new steps -- as it is carried out (hence, the Next() method). At the moment, this is linearized; eventually, we want to make this more "graph-like" so that we can exploit available parallelism within the dependencies. There are tons of TODOs remaining. However, the `mu plan` command is functioning with these new changes -- including colorization FTW -- so I'm landing it now. This is part of marapongo/mu#38 and marapongo/mu#41.
2017-02-17 21:31:48 +01:00
}
// Manifest captures versions for all binaries used to construct this snapshot.
type Manifest struct {
Time time.Time // the time this snapshot was taken.
Magic string // a magic cookie.
Version string // the pulumi command version.
Plugins []workspace.PluginInfo // the plugin versions also loaded.
}
// NewMagic creates a magic cookie out of a manifest; this can be used to check for tampering. This ignores
// any existing magic value already stored on the manifest.
func (m Manifest) NewMagic() string {
if m.Version == "" {
return ""
}
return fmt.Sprintf("%x", sha256.Sum256([]byte(m.Version)))
}
// NewSnapshot creates a snapshot from the given arguments. The resources must be in topologically sorted order.
// This property is not checked; for verification, please refer to the VerifyIntegrity function below.
func NewSnapshot(manifest Manifest, resources []*resource.State) *Snapshot {
return &Snapshot{
Manifest: manifest,
Resources: resources,
Begin resource modeling and planning This change introduces a new package, pkg/resource, that will form the foundation for actually performing deployment plans and applications. It contains the following key abstractions: * resource.Provider is a wrapper around the CRUD operations exposed by underlying resource plugins. It will eventually defer to resource.Plugin, which itself defers -- over an RPC interface -- to the actual plugin, one per package exposing resources. The provider will also understand how to load, cache, and overall manage the lifetime of each plugin. * resource.Resource is the actual resource object. This is created from the overall evaluation object graph, but is simplified. It contains only serializable properties, for example. Inter-resource references are translated into serializable monikers as part of creating the resource. * resource.Moniker is a serializable string that uniquely identifies a resource in the Mu system. This is in contrast to resource IDs, which are generated by resource providers and generally opaque to the Mu system. See marapongo/mu#69 for more information about monikers and some of their challenges (namely, designing a stable algorithm). * resource.Snapshot is a "snapshot" taken from a graph of resources. This is a transitive closure of state representing one possible configuration of a given environment. This is what plans are created from. Eventually, two snapshots will be diffable, in order to perform incremental updates. One way of thinking about this is that a snapshot of the old world's state is advanced, one step at a time, until it reaches a desired snapshot of the new world's state. * resource.Plan is a plan for carrying out desired CRUD operations on a target environment. Each plan consists of zero-to-many Steps, each of which has a CRUD operation type, a resource target, and a next step. This is an enumerator because it is possible the plan will evolve -- and introduce new steps -- as it is carried out (hence, the Next() method). At the moment, this is linearized; eventually, we want to make this more "graph-like" so that we can exploit available parallelism within the dependencies. There are tons of TODOs remaining. However, the `mu plan` command is functioning with these new changes -- including colorization FTW -- so I'm landing it now. This is part of marapongo/mu#38 and marapongo/mu#41.
2017-02-17 21:31:48 +01:00
}
}
// VerifyIntegrity checks a snapshot to ensure it is well-formed. Because of the cost of this operation,
// integrity verification is only performed on demand, and not automatically during snapshot construction.
//
// This function enforces a couple of invariants:
// 1. Parents should always come before children in the resource list
// 2. Dependents should always come before their dependencies in the resource list
// 3. For every URN in the snapshot, there must be at most one resource with that URN that is not pending deletion
// 4. The magic manifest number should change every time the snapshot is mutated
func (snap *Snapshot) VerifyIntegrity() error {
if snap != nil {
// Ensure the magic cookie checks out.
if snap.Manifest.Magic != snap.Manifest.NewMagic() {
return errors.Errorf("magic cookie mismatch; possible tampering/corruption detected")
Lift snapshot management out of the engine and serialize writes to snapshot (#1069) * Lift snapshot management out of the engine This PR is a prerequisite for parallelism by addressing a major problem that the engine has to deal with when performing parallel resource construction: parallel mutation of the global snapshot. This PR adds a `SnapshotManager` type that is responsible for maintaining and persisting the current resource snapshot. It serializes all reads and writes to the global snapshot and persists the snapshot to persistent storage upon every write. As a side-effect of this, the core engine no longer needs to know about snapshot management at all; all snapshot operations can be handled as callbacks on deployment events. This will greatly simplify the parallelization of the core engine. Worth noting is that the core engine will still need to be able to read the current snapshot, since it is interested in the dependency graphs contained within. The full implications of that are out of scope of this PR. Remove dead code, Steps no longer need a reference to the plan iterator that created them Fixing various issues that arise when bringing up pulumi-aws Line length broke the build Code review: remove dead field, fix yaml name error Rebase against master, provide implementation of StackPersister for cloud backend Code review feedback: comments on MutationStatus, style in snapshot.go Code review feedback: move SnapshotManager to pkg/backend, change engine to use an interface SnapshotManager Code review feedback: use a channel for synchronization Add a comment and a new test * Maintain two checkpoints, an immutable base and a mutable delta, and periodically merge the two to produce snapshots * Add a lot of tests - covers all of the non-error paths of BeginMutation and End * Fix a test resource provider * Add a few tests, fix a few issues * Rebase against master, fixed merge
2018-04-12 18:55:34 +02:00
}
// Now check the resources. For now, we just verify that parents come before children, and that there aren't
// any duplicate URNs.
urns := make(map[resource.URN]*resource.State)
for i, state := range snap.Resources {
urn := state.URN
if par := state.Parent; par != "" {
if _, has := urns[par]; !has {
// The parent isn't there; to give a good error message, see whether it's missing entirely, or
// whether it comes later in the snapshot (neither of which should ever happen).
for _, other := range snap.Resources[i+1:] {
if other.URN == par {
return errors.Errorf("child resource %s's parent %s comes after it", urn, par)
}
Lift snapshot management out of the engine and serialize writes to snapshot (#1069) * Lift snapshot management out of the engine This PR is a prerequisite for parallelism by addressing a major problem that the engine has to deal with when performing parallel resource construction: parallel mutation of the global snapshot. This PR adds a `SnapshotManager` type that is responsible for maintaining and persisting the current resource snapshot. It serializes all reads and writes to the global snapshot and persists the snapshot to persistent storage upon every write. As a side-effect of this, the core engine no longer needs to know about snapshot management at all; all snapshot operations can be handled as callbacks on deployment events. This will greatly simplify the parallelization of the core engine. Worth noting is that the core engine will still need to be able to read the current snapshot, since it is interested in the dependency graphs contained within. The full implications of that are out of scope of this PR. Remove dead code, Steps no longer need a reference to the plan iterator that created them Fixing various issues that arise when bringing up pulumi-aws Line length broke the build Code review: remove dead field, fix yaml name error Rebase against master, provide implementation of StackPersister for cloud backend Code review feedback: comments on MutationStatus, style in snapshot.go Code review feedback: move SnapshotManager to pkg/backend, change engine to use an interface SnapshotManager Code review feedback: use a channel for synchronization Add a comment and a new test * Maintain two checkpoints, an immutable base and a mutable delta, and periodically merge the two to produce snapshots * Add a lot of tests - covers all of the non-error paths of BeginMutation and End * Fix a test resource provider * Add a few tests, fix a few issues * Rebase against master, fixed merge
2018-04-12 18:55:34 +02:00
}
return errors.Errorf("child resource %s refers to missing parent %s", urn, par)
Lift snapshot management out of the engine and serialize writes to snapshot (#1069) * Lift snapshot management out of the engine This PR is a prerequisite for parallelism by addressing a major problem that the engine has to deal with when performing parallel resource construction: parallel mutation of the global snapshot. This PR adds a `SnapshotManager` type that is responsible for maintaining and persisting the current resource snapshot. It serializes all reads and writes to the global snapshot and persists the snapshot to persistent storage upon every write. As a side-effect of this, the core engine no longer needs to know about snapshot management at all; all snapshot operations can be handled as callbacks on deployment events. This will greatly simplify the parallelization of the core engine. Worth noting is that the core engine will still need to be able to read the current snapshot, since it is interested in the dependency graphs contained within. The full implications of that are out of scope of this PR. Remove dead code, Steps no longer need a reference to the plan iterator that created them Fixing various issues that arise when bringing up pulumi-aws Line length broke the build Code review: remove dead field, fix yaml name error Rebase against master, provide implementation of StackPersister for cloud backend Code review feedback: comments on MutationStatus, style in snapshot.go Code review feedback: move SnapshotManager to pkg/backend, change engine to use an interface SnapshotManager Code review feedback: use a channel for synchronization Add a comment and a new test * Maintain two checkpoints, an immutable base and a mutable delta, and periodically merge the two to produce snapshots * Add a lot of tests - covers all of the non-error paths of BeginMutation and End * Fix a test resource provider * Add a few tests, fix a few issues * Rebase against master, fixed merge
2018-04-12 18:55:34 +02:00
}
}
Lift snapshot management out of the engine and serialize writes to snapshot (#1069) * Lift snapshot management out of the engine This PR is a prerequisite for parallelism by addressing a major problem that the engine has to deal with when performing parallel resource construction: parallel mutation of the global snapshot. This PR adds a `SnapshotManager` type that is responsible for maintaining and persisting the current resource snapshot. It serializes all reads and writes to the global snapshot and persists the snapshot to persistent storage upon every write. As a side-effect of this, the core engine no longer needs to know about snapshot management at all; all snapshot operations can be handled as callbacks on deployment events. This will greatly simplify the parallelization of the core engine. Worth noting is that the core engine will still need to be able to read the current snapshot, since it is interested in the dependency graphs contained within. The full implications of that are out of scope of this PR. Remove dead code, Steps no longer need a reference to the plan iterator that created them Fixing various issues that arise when bringing up pulumi-aws Line length broke the build Code review: remove dead field, fix yaml name error Rebase against master, provide implementation of StackPersister for cloud backend Code review feedback: comments on MutationStatus, style in snapshot.go Code review feedback: move SnapshotManager to pkg/backend, change engine to use an interface SnapshotManager Code review feedback: use a channel for synchronization Add a comment and a new test * Maintain two checkpoints, an immutable base and a mutable delta, and periodically merge the two to produce snapshots * Add a lot of tests - covers all of the non-error paths of BeginMutation and End * Fix a test resource provider * Add a few tests, fix a few issues * Rebase against master, fixed merge
2018-04-12 18:55:34 +02:00
for _, dep := range state.Dependencies {
if _, has := urns[dep]; !has {
// same as above - doing this for better error messages
for _, other := range snap.Resources[i+1:] {
if other.URN == dep {
return errors.Errorf("resource %s's dependency %s comes after it", urn, other.URN)
}
}
return errors.Errorf("resource %s dependency %s refers to missing resource", urn, dep)
}
}
if _, has := urns[urn]; has && !state.Delete {
// The only time we should have duplicate URNs is when all but one of them are marked for deletion.
return errors.Errorf("duplicate resource %s (not marked for deletion)", urn)
}
urns[urn] = state
}
}
return nil
}