pulumi/pkg/resource/resource.go

161 lines
6.2 KiB
Go
Raw Normal View History

Begin resource modeling and planning This change introduces a new package, pkg/resource, that will form the foundation for actually performing deployment plans and applications. It contains the following key abstractions: * resource.Provider is a wrapper around the CRUD operations exposed by underlying resource plugins. It will eventually defer to resource.Plugin, which itself defers -- over an RPC interface -- to the actual plugin, one per package exposing resources. The provider will also understand how to load, cache, and overall manage the lifetime of each plugin. * resource.Resource is the actual resource object. This is created from the overall evaluation object graph, but is simplified. It contains only serializable properties, for example. Inter-resource references are translated into serializable monikers as part of creating the resource. * resource.Moniker is a serializable string that uniquely identifies a resource in the Mu system. This is in contrast to resource IDs, which are generated by resource providers and generally opaque to the Mu system. See marapongo/mu#69 for more information about monikers and some of their challenges (namely, designing a stable algorithm). * resource.Snapshot is a "snapshot" taken from a graph of resources. This is a transitive closure of state representing one possible configuration of a given environment. This is what plans are created from. Eventually, two snapshots will be diffable, in order to perform incremental updates. One way of thinking about this is that a snapshot of the old world's state is advanced, one step at a time, until it reaches a desired snapshot of the new world's state. * resource.Plan is a plan for carrying out desired CRUD operations on a target environment. Each plan consists of zero-to-many Steps, each of which has a CRUD operation type, a resource target, and a next step. This is an enumerator because it is possible the plan will evolve -- and introduce new steps -- as it is carried out (hence, the Next() method). At the moment, this is linearized; eventually, we want to make this more "graph-like" so that we can exploit available parallelism within the dependencies. There are tons of TODOs remaining. However, the `mu plan` command is functioning with these new changes -- including colorization FTW -- so I'm landing it now. This is part of marapongo/mu#38 and marapongo/mu#41.
2017-02-17 21:31:48 +01:00
// Copyright 2016 Marapongo, Inc. All rights reserved.
package resource
import (
"reflect"
"github.com/golang/glog"
"github.com/marapongo/mu/pkg/compiler/symbols"
"github.com/marapongo/mu/pkg/compiler/types"
"github.com/marapongo/mu/pkg/compiler/types/predef"
Implement updates This change is a first whack at implementing updates. Creation and deletion plans are pretty straightforward; we just take a single graph, topologically sort it, and perform the operations in the right order. For creation, this is in dependency order (things that are depended upon must be created before dependents); for deletion, this is in reverse-dependency order (things that depend on others must be deleted before dependencies). These are just special cases of the more general idea of performing DAG operations in dependency order. Updates must work in terms of this more general notion. For example: * It is an error to delete a resource while another refers to it; thus, resources are deleted after deleting dependents, or after updating dependent properties that reference the resource to new values. * It is an error to depend on a create a resource before it is created; thus, resources must be created before dependents are created, and/or before updates to existing resource properties that would cause them to refer to the new resource. Of course, all of this is tangled up in a graph of dependencies. As a result, we must create a DAG of the dependencies between creates, updates, and deletes, and then topologically sort this DAG, in order to determine the proper order of update operations. To do this, we slightly generalize the existing graph infrastructure, while also specializing two kinds of graphs; the existing one becomes a heapstate.ObjectGraph, while this new one is resource.planGraph (internal).
2017-02-23 23:56:23 +01:00
"github.com/marapongo/mu/pkg/eval/heapstate"
Begin resource modeling and planning This change introduces a new package, pkg/resource, that will form the foundation for actually performing deployment plans and applications. It contains the following key abstractions: * resource.Provider is a wrapper around the CRUD operations exposed by underlying resource plugins. It will eventually defer to resource.Plugin, which itself defers -- over an RPC interface -- to the actual plugin, one per package exposing resources. The provider will also understand how to load, cache, and overall manage the lifetime of each plugin. * resource.Resource is the actual resource object. This is created from the overall evaluation object graph, but is simplified. It contains only serializable properties, for example. Inter-resource references are translated into serializable monikers as part of creating the resource. * resource.Moniker is a serializable string that uniquely identifies a resource in the Mu system. This is in contrast to resource IDs, which are generated by resource providers and generally opaque to the Mu system. See marapongo/mu#69 for more information about monikers and some of their challenges (namely, designing a stable algorithm). * resource.Snapshot is a "snapshot" taken from a graph of resources. This is a transitive closure of state representing one possible configuration of a given environment. This is what plans are created from. Eventually, two snapshots will be diffable, in order to perform incremental updates. One way of thinking about this is that a snapshot of the old world's state is advanced, one step at a time, until it reaches a desired snapshot of the new world's state. * resource.Plan is a plan for carrying out desired CRUD operations on a target environment. Each plan consists of zero-to-many Steps, each of which has a CRUD operation type, a resource target, and a next step. This is an enumerator because it is possible the plan will evolve -- and introduce new steps -- as it is carried out (hence, the Next() method). At the moment, this is linearized; eventually, we want to make this more "graph-like" so that we can exploit available parallelism within the dependencies. There are tons of TODOs remaining. However, the `mu plan` command is functioning with these new changes -- including colorization FTW -- so I'm landing it now. This is part of marapongo/mu#38 and marapongo/mu#41.
2017-02-17 21:31:48 +01:00
"github.com/marapongo/mu/pkg/eval/rt"
"github.com/marapongo/mu/pkg/tokens"
"github.com/marapongo/mu/pkg/util/contract"
)
// ID is a unique resource identifier; it is managed by the provider and is mostly opaque to Mu.
type ID string
// Resource is an instance of a resource with an ID, type, and bag of state.
type Resource interface {
ID() ID // the resource's unique ID, assigned by the resource provider (or blank if uncreated).
Moniker() Moniker // the resource's object moniker, a human-friendly, unique name for the resource.
Type() tokens.Type // the resource's type.
Begin resource modeling and planning This change introduces a new package, pkg/resource, that will form the foundation for actually performing deployment plans and applications. It contains the following key abstractions: * resource.Provider is a wrapper around the CRUD operations exposed by underlying resource plugins. It will eventually defer to resource.Plugin, which itself defers -- over an RPC interface -- to the actual plugin, one per package exposing resources. The provider will also understand how to load, cache, and overall manage the lifetime of each plugin. * resource.Resource is the actual resource object. This is created from the overall evaluation object graph, but is simplified. It contains only serializable properties, for example. Inter-resource references are translated into serializable monikers as part of creating the resource. * resource.Moniker is a serializable string that uniquely identifies a resource in the Mu system. This is in contrast to resource IDs, which are generated by resource providers and generally opaque to the Mu system. See marapongo/mu#69 for more information about monikers and some of their challenges (namely, designing a stable algorithm). * resource.Snapshot is a "snapshot" taken from a graph of resources. This is a transitive closure of state representing one possible configuration of a given environment. This is what plans are created from. Eventually, two snapshots will be diffable, in order to perform incremental updates. One way of thinking about this is that a snapshot of the old world's state is advanced, one step at a time, until it reaches a desired snapshot of the new world's state. * resource.Plan is a plan for carrying out desired CRUD operations on a target environment. Each plan consists of zero-to-many Steps, each of which has a CRUD operation type, a resource target, and a next step. This is an enumerator because it is possible the plan will evolve -- and introduce new steps -- as it is carried out (hence, the Next() method). At the moment, this is linearized; eventually, we want to make this more "graph-like" so that we can exploit available parallelism within the dependencies. There are tons of TODOs remaining. However, the `mu plan` command is functioning with these new changes -- including colorization FTW -- so I'm landing it now. This is part of marapongo/mu#38 and marapongo/mu#41.
2017-02-17 21:31:48 +01:00
Properties() PropertyMap // the resource's property map.
2017-02-22 23:43:08 +01:00
HasID() bool // returns true if the resource has been assigned an ID.
SetID(id ID) // assignes an ID to this resource, for those under creation.
Redo object monikers This change overhauls the way we do object monikers. The old mechanism, generating monikers using graph paths, was far too brittle and prone to collisions. The new approach mixes some amount of "automatic scoping" plus some "explicit naming." Although there is some explicitness, this is arguably a good thing, as the monikers will be relatable back to the source more readily by developers inspecting the graph and resource state. Each moniker has four parts: <Namespace>::<AllocModule>::<Type>::<Name> wherein each element is the following: <Namespace> The namespace being deployed into <AllocModule> The module in which the object was allocated <Type> The type of the resource <Name> The assigned name of the resource The <Namespace> is essentially the deployment target -- so "prod", "stage", etc -- although it is more general purpose to allow for future namespacing within a target (e.g., "prod/customer1", etc); for now this is rudimentary, however, see marapongo/mu#94. The <AllocModule> is the token for the code that contained the 'new' that led to this object being created. In the future, we may wish to extend this to also track the module under evaluation. (This is a nice aspect of monikers; they can become arbitrarily complex, so long as they are precise, and not prone to false positives/negatives.) The <Name> warrants more discussion. The resource provider is consulted via a new gRPC method, Name, that fetches the name. How the provider does this is entirely up to it. For some resource types, the resource may have properties that developers must set (e.g., `new Bucket("foo")`); for other providers, perhaps the resource intrinsically has a property that explicitly and uniquely qualifies the object (e.g., AWS SecurityGroups, via `new SecurityGroup({groupName: "my-sg"}`); and finally, it's conceivable that a provider might auto-generate the name (e.g., such as an AWS Lambda whose name could simply be a hash of the source code contents). This should overall produce better results with respect to moniker collisions, ability to match resources, and the usability of the system.
2017-02-24 23:50:02 +01:00
HasMoniker() bool // returns true if the resource has been assigned moniker.
SetMoniker(m Moniker) // assignes a moniker to this resource, for those under creation.
Begin resource modeling and planning This change introduces a new package, pkg/resource, that will form the foundation for actually performing deployment plans and applications. It contains the following key abstractions: * resource.Provider is a wrapper around the CRUD operations exposed by underlying resource plugins. It will eventually defer to resource.Plugin, which itself defers -- over an RPC interface -- to the actual plugin, one per package exposing resources. The provider will also understand how to load, cache, and overall manage the lifetime of each plugin. * resource.Resource is the actual resource object. This is created from the overall evaluation object graph, but is simplified. It contains only serializable properties, for example. Inter-resource references are translated into serializable monikers as part of creating the resource. * resource.Moniker is a serializable string that uniquely identifies a resource in the Mu system. This is in contrast to resource IDs, which are generated by resource providers and generally opaque to the Mu system. See marapongo/mu#69 for more information about monikers and some of their challenges (namely, designing a stable algorithm). * resource.Snapshot is a "snapshot" taken from a graph of resources. This is a transitive closure of state representing one possible configuration of a given environment. This is what plans are created from. Eventually, two snapshots will be diffable, in order to perform incremental updates. One way of thinking about this is that a snapshot of the old world's state is advanced, one step at a time, until it reaches a desired snapshot of the new world's state. * resource.Plan is a plan for carrying out desired CRUD operations on a target environment. Each plan consists of zero-to-many Steps, each of which has a CRUD operation type, a resource target, and a next step. This is an enumerator because it is possible the plan will evolve -- and introduce new steps -- as it is carried out (hence, the Next() method). At the moment, this is linearized; eventually, we want to make this more "graph-like" so that we can exploit available parallelism within the dependencies. There are tons of TODOs remaining. However, the `mu plan` command is functioning with these new changes -- including colorization FTW -- so I'm landing it now. This is part of marapongo/mu#38 and marapongo/mu#41.
2017-02-17 21:31:48 +01:00
}
// ResourceState is returned when an error has occurred during a resource provider operation. It indicates whether the
// operation could be rolled back cleanly (OK). If not, it means the resource was left in an indeterminate state.
type ResourceState int
const (
StateOK ResourceState = iota
StateUnknown
)
Implement updates This change is a first whack at implementing updates. Creation and deletion plans are pretty straightforward; we just take a single graph, topologically sort it, and perform the operations in the right order. For creation, this is in dependency order (things that are depended upon must be created before dependents); for deletion, this is in reverse-dependency order (things that depend on others must be deleted before dependencies). These are just special cases of the more general idea of performing DAG operations in dependency order. Updates must work in terms of this more general notion. For example: * It is an error to delete a resource while another refers to it; thus, resources are deleted after deleting dependents, or after updating dependent properties that reference the resource to new values. * It is an error to depend on a create a resource before it is created; thus, resources must be created before dependents are created, and/or before updates to existing resource properties that would cause them to refer to the new resource. Of course, all of this is tangled up in a graph of dependencies. As a result, we must create a DAG of the dependencies between creates, updates, and deletes, and then topologically sort this DAG, in order to determine the proper order of update operations. To do this, we slightly generalize the existing graph infrastructure, while also specializing two kinds of graphs; the existing one becomes a heapstate.ObjectGraph, while this new one is resource.planGraph (internal).
2017-02-23 23:56:23 +01:00
func IsResourceType(t symbols.Type) bool { return types.HasBaseName(t, predef.MuResourceClass) }
func IsResourceVertex(v *heapstate.ObjectVertex) bool { return IsResourceType(v.Obj().Type()) }
Begin resource modeling and planning This change introduces a new package, pkg/resource, that will form the foundation for actually performing deployment plans and applications. It contains the following key abstractions: * resource.Provider is a wrapper around the CRUD operations exposed by underlying resource plugins. It will eventually defer to resource.Plugin, which itself defers -- over an RPC interface -- to the actual plugin, one per package exposing resources. The provider will also understand how to load, cache, and overall manage the lifetime of each plugin. * resource.Resource is the actual resource object. This is created from the overall evaluation object graph, but is simplified. It contains only serializable properties, for example. Inter-resource references are translated into serializable monikers as part of creating the resource. * resource.Moniker is a serializable string that uniquely identifies a resource in the Mu system. This is in contrast to resource IDs, which are generated by resource providers and generally opaque to the Mu system. See marapongo/mu#69 for more information about monikers and some of their challenges (namely, designing a stable algorithm). * resource.Snapshot is a "snapshot" taken from a graph of resources. This is a transitive closure of state representing one possible configuration of a given environment. This is what plans are created from. Eventually, two snapshots will be diffable, in order to perform incremental updates. One way of thinking about this is that a snapshot of the old world's state is advanced, one step at a time, until it reaches a desired snapshot of the new world's state. * resource.Plan is a plan for carrying out desired CRUD operations on a target environment. Each plan consists of zero-to-many Steps, each of which has a CRUD operation type, a resource target, and a next step. This is an enumerator because it is possible the plan will evolve -- and introduce new steps -- as it is carried out (hence, the Next() method). At the moment, this is linearized; eventually, we want to make this more "graph-like" so that we can exploit available parallelism within the dependencies. There are tons of TODOs remaining. However, the `mu plan` command is functioning with these new changes -- including colorization FTW -- so I'm landing it now. This is part of marapongo/mu#38 and marapongo/mu#41.
2017-02-17 21:31:48 +01:00
type resource struct {
id ID // the resource's unique ID, assigned by the resource provider (or blank if uncreated).
moniker Moniker // the resource's object moniker, a human-friendly, unique name for the resource.
t tokens.Type // the resource's type.
Begin resource modeling and planning This change introduces a new package, pkg/resource, that will form the foundation for actually performing deployment plans and applications. It contains the following key abstractions: * resource.Provider is a wrapper around the CRUD operations exposed by underlying resource plugins. It will eventually defer to resource.Plugin, which itself defers -- over an RPC interface -- to the actual plugin, one per package exposing resources. The provider will also understand how to load, cache, and overall manage the lifetime of each plugin. * resource.Resource is the actual resource object. This is created from the overall evaluation object graph, but is simplified. It contains only serializable properties, for example. Inter-resource references are translated into serializable monikers as part of creating the resource. * resource.Moniker is a serializable string that uniquely identifies a resource in the Mu system. This is in contrast to resource IDs, which are generated by resource providers and generally opaque to the Mu system. See marapongo/mu#69 for more information about monikers and some of their challenges (namely, designing a stable algorithm). * resource.Snapshot is a "snapshot" taken from a graph of resources. This is a transitive closure of state representing one possible configuration of a given environment. This is what plans are created from. Eventually, two snapshots will be diffable, in order to perform incremental updates. One way of thinking about this is that a snapshot of the old world's state is advanced, one step at a time, until it reaches a desired snapshot of the new world's state. * resource.Plan is a plan for carrying out desired CRUD operations on a target environment. Each plan consists of zero-to-many Steps, each of which has a CRUD operation type, a resource target, and a next step. This is an enumerator because it is possible the plan will evolve -- and introduce new steps -- as it is carried out (hence, the Next() method). At the moment, this is linearized; eventually, we want to make this more "graph-like" so that we can exploit available parallelism within the dependencies. There are tons of TODOs remaining. However, the `mu plan` command is functioning with these new changes -- including colorization FTW -- so I'm landing it now. This is part of marapongo/mu#38 and marapongo/mu#41.
2017-02-17 21:31:48 +01:00
properties PropertyMap // the resource's property map.
}
func (r *resource) ID() ID { return r.id }
func (r *resource) Moniker() Moniker { return r.moniker }
func (r *resource) Type() tokens.Type { return r.t }
Begin resource modeling and planning This change introduces a new package, pkg/resource, that will form the foundation for actually performing deployment plans and applications. It contains the following key abstractions: * resource.Provider is a wrapper around the CRUD operations exposed by underlying resource plugins. It will eventually defer to resource.Plugin, which itself defers -- over an RPC interface -- to the actual plugin, one per package exposing resources. The provider will also understand how to load, cache, and overall manage the lifetime of each plugin. * resource.Resource is the actual resource object. This is created from the overall evaluation object graph, but is simplified. It contains only serializable properties, for example. Inter-resource references are translated into serializable monikers as part of creating the resource. * resource.Moniker is a serializable string that uniquely identifies a resource in the Mu system. This is in contrast to resource IDs, which are generated by resource providers and generally opaque to the Mu system. See marapongo/mu#69 for more information about monikers and some of their challenges (namely, designing a stable algorithm). * resource.Snapshot is a "snapshot" taken from a graph of resources. This is a transitive closure of state representing one possible configuration of a given environment. This is what plans are created from. Eventually, two snapshots will be diffable, in order to perform incremental updates. One way of thinking about this is that a snapshot of the old world's state is advanced, one step at a time, until it reaches a desired snapshot of the new world's state. * resource.Plan is a plan for carrying out desired CRUD operations on a target environment. Each plan consists of zero-to-many Steps, each of which has a CRUD operation type, a resource target, and a next step. This is an enumerator because it is possible the plan will evolve -- and introduce new steps -- as it is carried out (hence, the Next() method). At the moment, this is linearized; eventually, we want to make this more "graph-like" so that we can exploit available parallelism within the dependencies. There are tons of TODOs remaining. However, the `mu plan` command is functioning with these new changes -- including colorization FTW -- so I'm landing it now. This is part of marapongo/mu#38 and marapongo/mu#41.
2017-02-17 21:31:48 +01:00
func (r *resource) Properties() PropertyMap { return r.properties }
2017-02-22 23:43:08 +01:00
func (r *resource) HasID() bool { return (string(r.id) != "") }
func (r *resource) SetID(id ID) {
2017-02-22 23:43:08 +01:00
contract.Requiref(!r.HasID(), "id", "empty")
r.id = id
}
Redo object monikers This change overhauls the way we do object monikers. The old mechanism, generating monikers using graph paths, was far too brittle and prone to collisions. The new approach mixes some amount of "automatic scoping" plus some "explicit naming." Although there is some explicitness, this is arguably a good thing, as the monikers will be relatable back to the source more readily by developers inspecting the graph and resource state. Each moniker has four parts: <Namespace>::<AllocModule>::<Type>::<Name> wherein each element is the following: <Namespace> The namespace being deployed into <AllocModule> The module in which the object was allocated <Type> The type of the resource <Name> The assigned name of the resource The <Namespace> is essentially the deployment target -- so "prod", "stage", etc -- although it is more general purpose to allow for future namespacing within a target (e.g., "prod/customer1", etc); for now this is rudimentary, however, see marapongo/mu#94. The <AllocModule> is the token for the code that contained the 'new' that led to this object being created. In the future, we may wish to extend this to also track the module under evaluation. (This is a nice aspect of monikers; they can become arbitrarily complex, so long as they are precise, and not prone to false positives/negatives.) The <Name> warrants more discussion. The resource provider is consulted via a new gRPC method, Name, that fetches the name. How the provider does this is entirely up to it. For some resource types, the resource may have properties that developers must set (e.g., `new Bucket("foo")`); for other providers, perhaps the resource intrinsically has a property that explicitly and uniquely qualifies the object (e.g., AWS SecurityGroups, via `new SecurityGroup({groupName: "my-sg"}`); and finally, it's conceivable that a provider might auto-generate the name (e.g., such as an AWS Lambda whose name could simply be a hash of the source code contents). This should overall produce better results with respect to moniker collisions, ability to match resources, and the usability of the system.
2017-02-24 23:50:02 +01:00
func (r *resource) HasMoniker() bool { return (string(r.moniker) != "") }
func (r *resource) SetMoniker(m Moniker) {
contract.Requiref(!r.HasMoniker(), "moniker", "empty")
r.moniker = m
}
Implement resource provider plugins This change adds basic support for discovering, loading, binding to, and invoking RPC methods on, resource provider plugins. In a nutshell, we add a new context object that will share cached state such as loaded plugins and connections to them. It will be a policy decision in server scenarios how much state to share and between whom. This context also controls per-resource context allocation, which in the future will allow us to perform structured cancellation and teardown amongst entire groups of requests. Plugins are loaded based on their name, and can be found in one of two ways: either simply by having them on your path (with a name of "mu-ressrv-<pkg>", where "<pkg>" is the resource package name with any "/"s replaced with "_"s); or by placing them in the standard library installation location, which need not be on the path for this to work (since we know precisely where to look). If we find a protocol, we will load it as a child process. The protocol for plugins is that they will choose a port on their own -- to eliminate races that'd be involved should Mu attempt to pre-pick one for them -- and then write that out as the first line to STDOUT (terminated by a "\n"). This is the only STDERR/STDOUT that Mu cares about; from there, the plugin is free to write all it pleases (e.g., for logging, debugging purposes, etc). Afterwards, we then bind our gRPC connection to that port, and create a typed resource provider client. The CRUD operations that get driven by plan application are then simple wrappers atop the underlying gRPC calls. For now, we interpret all errors as catastrophic; in the near future, we will probably want to introduce a "structured error" mechanism in the gRPC interface for "transactional errors"; that is, errors for which the server was able to recover to a safe checkpoint, which can be interpreted as ResourceOK rather than ResourceUnknown.
2017-02-19 20:08:06 +01:00
// NewResource creates a new resource from the information provided.
func NewResource(id ID, moniker Moniker, t tokens.Type, properties PropertyMap) Resource {
return &resource{
id: id,
moniker: moniker,
t: t,
properties: properties,
}
}
Redo object monikers This change overhauls the way we do object monikers. The old mechanism, generating monikers using graph paths, was far too brittle and prone to collisions. The new approach mixes some amount of "automatic scoping" plus some "explicit naming." Although there is some explicitness, this is arguably a good thing, as the monikers will be relatable back to the source more readily by developers inspecting the graph and resource state. Each moniker has four parts: <Namespace>::<AllocModule>::<Type>::<Name> wherein each element is the following: <Namespace> The namespace being deployed into <AllocModule> The module in which the object was allocated <Type> The type of the resource <Name> The assigned name of the resource The <Namespace> is essentially the deployment target -- so "prod", "stage", etc -- although it is more general purpose to allow for future namespacing within a target (e.g., "prod/customer1", etc); for now this is rudimentary, however, see marapongo/mu#94. The <AllocModule> is the token for the code that contained the 'new' that led to this object being created. In the future, we may wish to extend this to also track the module under evaluation. (This is a nice aspect of monikers; they can become arbitrarily complex, so long as they are precise, and not prone to false positives/negatives.) The <Name> warrants more discussion. The resource provider is consulted via a new gRPC method, Name, that fetches the name. How the provider does this is entirely up to it. For some resource types, the resource may have properties that developers must set (e.g., `new Bucket("foo")`); for other providers, perhaps the resource intrinsically has a property that explicitly and uniquely qualifies the object (e.g., AWS SecurityGroups, via `new SecurityGroup({groupName: "my-sg"}`); and finally, it's conceivable that a provider might auto-generate the name (e.g., such as an AWS Lambda whose name could simply be a hash of the source code contents). This should overall produce better results with respect to moniker collisions, ability to match resources, and the usability of the system.
2017-02-24 23:50:02 +01:00
// NewObjectResource creates a new resource object out of the runtime object provided. The context is used to resolve
// dependencies between resources and must contain all references that could be encountered.
Redo object monikers This change overhauls the way we do object monikers. The old mechanism, generating monikers using graph paths, was far too brittle and prone to collisions. The new approach mixes some amount of "automatic scoping" plus some "explicit naming." Although there is some explicitness, this is arguably a good thing, as the monikers will be relatable back to the source more readily by developers inspecting the graph and resource state. Each moniker has four parts: <Namespace>::<AllocModule>::<Type>::<Name> wherein each element is the following: <Namespace> The namespace being deployed into <AllocModule> The module in which the object was allocated <Type> The type of the resource <Name> The assigned name of the resource The <Namespace> is essentially the deployment target -- so "prod", "stage", etc -- although it is more general purpose to allow for future namespacing within a target (e.g., "prod/customer1", etc); for now this is rudimentary, however, see marapongo/mu#94. The <AllocModule> is the token for the code that contained the 'new' that led to this object being created. In the future, we may wish to extend this to also track the module under evaluation. (This is a nice aspect of monikers; they can become arbitrarily complex, so long as they are precise, and not prone to false positives/negatives.) The <Name> warrants more discussion. The resource provider is consulted via a new gRPC method, Name, that fetches the name. How the provider does this is entirely up to it. For some resource types, the resource may have properties that developers must set (e.g., `new Bucket("foo")`); for other providers, perhaps the resource intrinsically has a property that explicitly and uniquely qualifies the object (e.g., AWS SecurityGroups, via `new SecurityGroup({groupName: "my-sg"}`); and finally, it's conceivable that a provider might auto-generate the name (e.g., such as an AWS Lambda whose name could simply be a hash of the source code contents). This should overall produce better results with respect to moniker collisions, ability to match resources, and the usability of the system.
2017-02-24 23:50:02 +01:00
func NewObjectResource(ctx *Context, obj *rt.Object) Resource {
t := obj.Type()
contract.Assert(IsResourceType(t))
Begin resource modeling and planning This change introduces a new package, pkg/resource, that will form the foundation for actually performing deployment plans and applications. It contains the following key abstractions: * resource.Provider is a wrapper around the CRUD operations exposed by underlying resource plugins. It will eventually defer to resource.Plugin, which itself defers -- over an RPC interface -- to the actual plugin, one per package exposing resources. The provider will also understand how to load, cache, and overall manage the lifetime of each plugin. * resource.Resource is the actual resource object. This is created from the overall evaluation object graph, but is simplified. It contains only serializable properties, for example. Inter-resource references are translated into serializable monikers as part of creating the resource. * resource.Moniker is a serializable string that uniquely identifies a resource in the Mu system. This is in contrast to resource IDs, which are generated by resource providers and generally opaque to the Mu system. See marapongo/mu#69 for more information about monikers and some of their challenges (namely, designing a stable algorithm). * resource.Snapshot is a "snapshot" taken from a graph of resources. This is a transitive closure of state representing one possible configuration of a given environment. This is what plans are created from. Eventually, two snapshots will be diffable, in order to perform incremental updates. One way of thinking about this is that a snapshot of the old world's state is advanced, one step at a time, until it reaches a desired snapshot of the new world's state. * resource.Plan is a plan for carrying out desired CRUD operations on a target environment. Each plan consists of zero-to-many Steps, each of which has a CRUD operation type, a resource target, and a next step. This is an enumerator because it is possible the plan will evolve -- and introduce new steps -- as it is carried out (hence, the Next() method). At the moment, this is linearized; eventually, we want to make this more "graph-like" so that we can exploit available parallelism within the dependencies. There are tons of TODOs remaining. However, the `mu plan` command is functioning with these new changes -- including colorization FTW -- so I'm landing it now. This is part of marapongo/mu#38 and marapongo/mu#41.
2017-02-17 21:31:48 +01:00
// Extract the moniker. This must already exist.
Redo object monikers This change overhauls the way we do object monikers. The old mechanism, generating monikers using graph paths, was far too brittle and prone to collisions. The new approach mixes some amount of "automatic scoping" plus some "explicit naming." Although there is some explicitness, this is arguably a good thing, as the monikers will be relatable back to the source more readily by developers inspecting the graph and resource state. Each moniker has four parts: <Namespace>::<AllocModule>::<Type>::<Name> wherein each element is the following: <Namespace> The namespace being deployed into <AllocModule> The module in which the object was allocated <Type> The type of the resource <Name> The assigned name of the resource The <Namespace> is essentially the deployment target -- so "prod", "stage", etc -- although it is more general purpose to allow for future namespacing within a target (e.g., "prod/customer1", etc); for now this is rudimentary, however, see marapongo/mu#94. The <AllocModule> is the token for the code that contained the 'new' that led to this object being created. In the future, we may wish to extend this to also track the module under evaluation. (This is a nice aspect of monikers; they can become arbitrarily complex, so long as they are precise, and not prone to false positives/negatives.) The <Name> warrants more discussion. The resource provider is consulted via a new gRPC method, Name, that fetches the name. How the provider does this is entirely up to it. For some resource types, the resource may have properties that developers must set (e.g., `new Bucket("foo")`); for other providers, perhaps the resource intrinsically has a property that explicitly and uniquely qualifies the object (e.g., AWS SecurityGroups, via `new SecurityGroup({groupName: "my-sg"}`); and finally, it's conceivable that a provider might auto-generate the name (e.g., such as an AWS Lambda whose name could simply be a hash of the source code contents). This should overall produce better results with respect to moniker collisions, ability to match resources, and the usability of the system.
2017-02-24 23:50:02 +01:00
m, hasm := ctx.ObjMks[obj]
contract.Assertf(!hasm, "Object already assigned a moniker '%v'; double allocation detected", m)
Begin resource modeling and planning This change introduces a new package, pkg/resource, that will form the foundation for actually performing deployment plans and applications. It contains the following key abstractions: * resource.Provider is a wrapper around the CRUD operations exposed by underlying resource plugins. It will eventually defer to resource.Plugin, which itself defers -- over an RPC interface -- to the actual plugin, one per package exposing resources. The provider will also understand how to load, cache, and overall manage the lifetime of each plugin. * resource.Resource is the actual resource object. This is created from the overall evaluation object graph, but is simplified. It contains only serializable properties, for example. Inter-resource references are translated into serializable monikers as part of creating the resource. * resource.Moniker is a serializable string that uniquely identifies a resource in the Mu system. This is in contrast to resource IDs, which are generated by resource providers and generally opaque to the Mu system. See marapongo/mu#69 for more information about monikers and some of their challenges (namely, designing a stable algorithm). * resource.Snapshot is a "snapshot" taken from a graph of resources. This is a transitive closure of state representing one possible configuration of a given environment. This is what plans are created from. Eventually, two snapshots will be diffable, in order to perform incremental updates. One way of thinking about this is that a snapshot of the old world's state is advanced, one step at a time, until it reaches a desired snapshot of the new world's state. * resource.Plan is a plan for carrying out desired CRUD operations on a target environment. Each plan consists of zero-to-many Steps, each of which has a CRUD operation type, a resource target, and a next step. This is an enumerator because it is possible the plan will evolve -- and introduce new steps -- as it is carried out (hence, the Next() method). At the moment, this is linearized; eventually, we want to make this more "graph-like" so that we can exploit available parallelism within the dependencies. There are tons of TODOs remaining. However, the `mu plan` command is functioning with these new changes -- including colorization FTW -- so I'm landing it now. This is part of marapongo/mu#38 and marapongo/mu#41.
2017-02-17 21:31:48 +01:00
// Do a deep copy of the resource properties. This ensures property serializability.
Redo object monikers This change overhauls the way we do object monikers. The old mechanism, generating monikers using graph paths, was far too brittle and prone to collisions. The new approach mixes some amount of "automatic scoping" plus some "explicit naming." Although there is some explicitness, this is arguably a good thing, as the monikers will be relatable back to the source more readily by developers inspecting the graph and resource state. Each moniker has four parts: <Namespace>::<AllocModule>::<Type>::<Name> wherein each element is the following: <Namespace> The namespace being deployed into <AllocModule> The module in which the object was allocated <Type> The type of the resource <Name> The assigned name of the resource The <Namespace> is essentially the deployment target -- so "prod", "stage", etc -- although it is more general purpose to allow for future namespacing within a target (e.g., "prod/customer1", etc); for now this is rudimentary, however, see marapongo/mu#94. The <AllocModule> is the token for the code that contained the 'new' that led to this object being created. In the future, we may wish to extend this to also track the module under evaluation. (This is a nice aspect of monikers; they can become arbitrarily complex, so long as they are precise, and not prone to false positives/negatives.) The <Name> warrants more discussion. The resource provider is consulted via a new gRPC method, Name, that fetches the name. How the provider does this is entirely up to it. For some resource types, the resource may have properties that developers must set (e.g., `new Bucket("foo")`); for other providers, perhaps the resource intrinsically has a property that explicitly and uniquely qualifies the object (e.g., AWS SecurityGroups, via `new SecurityGroup({groupName: "my-sg"}`); and finally, it's conceivable that a provider might auto-generate the name (e.g., such as an AWS Lambda whose name could simply be a hash of the source code contents). This should overall produce better results with respect to moniker collisions, ability to match resources, and the usability of the system.
2017-02-24 23:50:02 +01:00
props := cloneObject(ctx, obj)
Begin resource modeling and planning This change introduces a new package, pkg/resource, that will form the foundation for actually performing deployment plans and applications. It contains the following key abstractions: * resource.Provider is a wrapper around the CRUD operations exposed by underlying resource plugins. It will eventually defer to resource.Plugin, which itself defers -- over an RPC interface -- to the actual plugin, one per package exposing resources. The provider will also understand how to load, cache, and overall manage the lifetime of each plugin. * resource.Resource is the actual resource object. This is created from the overall evaluation object graph, but is simplified. It contains only serializable properties, for example. Inter-resource references are translated into serializable monikers as part of creating the resource. * resource.Moniker is a serializable string that uniquely identifies a resource in the Mu system. This is in contrast to resource IDs, which are generated by resource providers and generally opaque to the Mu system. See marapongo/mu#69 for more information about monikers and some of their challenges (namely, designing a stable algorithm). * resource.Snapshot is a "snapshot" taken from a graph of resources. This is a transitive closure of state representing one possible configuration of a given environment. This is what plans are created from. Eventually, two snapshots will be diffable, in order to perform incremental updates. One way of thinking about this is that a snapshot of the old world's state is advanced, one step at a time, until it reaches a desired snapshot of the new world's state. * resource.Plan is a plan for carrying out desired CRUD operations on a target environment. Each plan consists of zero-to-many Steps, each of which has a CRUD operation type, a resource target, and a next step. This is an enumerator because it is possible the plan will evolve -- and introduce new steps -- as it is carried out (hence, the Next() method). At the moment, this is linearized; eventually, we want to make this more "graph-like" so that we can exploit available parallelism within the dependencies. There are tons of TODOs remaining. However, the `mu plan` command is functioning with these new changes -- including colorization FTW -- so I'm landing it now. This is part of marapongo/mu#38 and marapongo/mu#41.
2017-02-17 21:31:48 +01:00
// Finally allocate and return the resource object; note that ID is left blank until the provider assignes one.
return &resource{
t: t.TypeToken(),
Begin resource modeling and planning This change introduces a new package, pkg/resource, that will form the foundation for actually performing deployment plans and applications. It contains the following key abstractions: * resource.Provider is a wrapper around the CRUD operations exposed by underlying resource plugins. It will eventually defer to resource.Plugin, which itself defers -- over an RPC interface -- to the actual plugin, one per package exposing resources. The provider will also understand how to load, cache, and overall manage the lifetime of each plugin. * resource.Resource is the actual resource object. This is created from the overall evaluation object graph, but is simplified. It contains only serializable properties, for example. Inter-resource references are translated into serializable monikers as part of creating the resource. * resource.Moniker is a serializable string that uniquely identifies a resource in the Mu system. This is in contrast to resource IDs, which are generated by resource providers and generally opaque to the Mu system. See marapongo/mu#69 for more information about monikers and some of their challenges (namely, designing a stable algorithm). * resource.Snapshot is a "snapshot" taken from a graph of resources. This is a transitive closure of state representing one possible configuration of a given environment. This is what plans are created from. Eventually, two snapshots will be diffable, in order to perform incremental updates. One way of thinking about this is that a snapshot of the old world's state is advanced, one step at a time, until it reaches a desired snapshot of the new world's state. * resource.Plan is a plan for carrying out desired CRUD operations on a target environment. Each plan consists of zero-to-many Steps, each of which has a CRUD operation type, a resource target, and a next step. This is an enumerator because it is possible the plan will evolve -- and introduce new steps -- as it is carried out (hence, the Next() method). At the moment, this is linearized; eventually, we want to make this more "graph-like" so that we can exploit available parallelism within the dependencies. There are tons of TODOs remaining. However, the `mu plan` command is functioning with these new changes -- including colorization FTW -- so I'm landing it now. This is part of marapongo/mu#38 and marapongo/mu#41.
2017-02-17 21:31:48 +01:00
properties: props,
}
}
// cloneObject creates a property map out of a runtime object. The result is fully serializable in the sense that it
// can be stored in a JSON or YAML file, serialized over an RPC interface, etc. In particular, any references to other
// resources are replaced with their moniker equivalents, which the runtime understands.
Redo object monikers This change overhauls the way we do object monikers. The old mechanism, generating monikers using graph paths, was far too brittle and prone to collisions. The new approach mixes some amount of "automatic scoping" plus some "explicit naming." Although there is some explicitness, this is arguably a good thing, as the monikers will be relatable back to the source more readily by developers inspecting the graph and resource state. Each moniker has four parts: <Namespace>::<AllocModule>::<Type>::<Name> wherein each element is the following: <Namespace> The namespace being deployed into <AllocModule> The module in which the object was allocated <Type> The type of the resource <Name> The assigned name of the resource The <Namespace> is essentially the deployment target -- so "prod", "stage", etc -- although it is more general purpose to allow for future namespacing within a target (e.g., "prod/customer1", etc); for now this is rudimentary, however, see marapongo/mu#94. The <AllocModule> is the token for the code that contained the 'new' that led to this object being created. In the future, we may wish to extend this to also track the module under evaluation. (This is a nice aspect of monikers; they can become arbitrarily complex, so long as they are precise, and not prone to false positives/negatives.) The <Name> warrants more discussion. The resource provider is consulted via a new gRPC method, Name, that fetches the name. How the provider does this is entirely up to it. For some resource types, the resource may have properties that developers must set (e.g., `new Bucket("foo")`); for other providers, perhaps the resource intrinsically has a property that explicitly and uniquely qualifies the object (e.g., AWS SecurityGroups, via `new SecurityGroup({groupName: "my-sg"}`); and finally, it's conceivable that a provider might auto-generate the name (e.g., such as an AWS Lambda whose name could simply be a hash of the source code contents). This should overall produce better results with respect to moniker collisions, ability to match resources, and the usability of the system.
2017-02-24 23:50:02 +01:00
func cloneObject(ctx *Context, obj *rt.Object) PropertyMap {
Begin resource modeling and planning This change introduces a new package, pkg/resource, that will form the foundation for actually performing deployment plans and applications. It contains the following key abstractions: * resource.Provider is a wrapper around the CRUD operations exposed by underlying resource plugins. It will eventually defer to resource.Plugin, which itself defers -- over an RPC interface -- to the actual plugin, one per package exposing resources. The provider will also understand how to load, cache, and overall manage the lifetime of each plugin. * resource.Resource is the actual resource object. This is created from the overall evaluation object graph, but is simplified. It contains only serializable properties, for example. Inter-resource references are translated into serializable monikers as part of creating the resource. * resource.Moniker is a serializable string that uniquely identifies a resource in the Mu system. This is in contrast to resource IDs, which are generated by resource providers and generally opaque to the Mu system. See marapongo/mu#69 for more information about monikers and some of their challenges (namely, designing a stable algorithm). * resource.Snapshot is a "snapshot" taken from a graph of resources. This is a transitive closure of state representing one possible configuration of a given environment. This is what plans are created from. Eventually, two snapshots will be diffable, in order to perform incremental updates. One way of thinking about this is that a snapshot of the old world's state is advanced, one step at a time, until it reaches a desired snapshot of the new world's state. * resource.Plan is a plan for carrying out desired CRUD operations on a target environment. Each plan consists of zero-to-many Steps, each of which has a CRUD operation type, a resource target, and a next step. This is an enumerator because it is possible the plan will evolve -- and introduce new steps -- as it is carried out (hence, the Next() method). At the moment, this is linearized; eventually, we want to make this more "graph-like" so that we can exploit available parallelism within the dependencies. There are tons of TODOs remaining. However, the `mu plan` command is functioning with these new changes -- including colorization FTW -- so I'm landing it now. This is part of marapongo/mu#38 and marapongo/mu#41.
2017-02-17 21:31:48 +01:00
contract.Assert(obj != nil)
src := obj.PropertyValues()
dest := make(PropertyMap)
for _, k := range rt.StablePropertyKeys(src) {
// TODO: detect cycles.
Redo object monikers This change overhauls the way we do object monikers. The old mechanism, generating monikers using graph paths, was far too brittle and prone to collisions. The new approach mixes some amount of "automatic scoping" plus some "explicit naming." Although there is some explicitness, this is arguably a good thing, as the monikers will be relatable back to the source more readily by developers inspecting the graph and resource state. Each moniker has four parts: <Namespace>::<AllocModule>::<Type>::<Name> wherein each element is the following: <Namespace> The namespace being deployed into <AllocModule> The module in which the object was allocated <Type> The type of the resource <Name> The assigned name of the resource The <Namespace> is essentially the deployment target -- so "prod", "stage", etc -- although it is more general purpose to allow for future namespacing within a target (e.g., "prod/customer1", etc); for now this is rudimentary, however, see marapongo/mu#94. The <AllocModule> is the token for the code that contained the 'new' that led to this object being created. In the future, we may wish to extend this to also track the module under evaluation. (This is a nice aspect of monikers; they can become arbitrarily complex, so long as they are precise, and not prone to false positives/negatives.) The <Name> warrants more discussion. The resource provider is consulted via a new gRPC method, Name, that fetches the name. How the provider does this is entirely up to it. For some resource types, the resource may have properties that developers must set (e.g., `new Bucket("foo")`); for other providers, perhaps the resource intrinsically has a property that explicitly and uniquely qualifies the object (e.g., AWS SecurityGroups, via `new SecurityGroup({groupName: "my-sg"}`); and finally, it's conceivable that a provider might auto-generate the name (e.g., such as an AWS Lambda whose name could simply be a hash of the source code contents). This should overall produce better results with respect to moniker collisions, ability to match resources, and the usability of the system.
2017-02-24 23:50:02 +01:00
if v, ok := cloneObjectValue(ctx, src[k].Obj()); ok {
Begin resource modeling and planning This change introduces a new package, pkg/resource, that will form the foundation for actually performing deployment plans and applications. It contains the following key abstractions: * resource.Provider is a wrapper around the CRUD operations exposed by underlying resource plugins. It will eventually defer to resource.Plugin, which itself defers -- over an RPC interface -- to the actual plugin, one per package exposing resources. The provider will also understand how to load, cache, and overall manage the lifetime of each plugin. * resource.Resource is the actual resource object. This is created from the overall evaluation object graph, but is simplified. It contains only serializable properties, for example. Inter-resource references are translated into serializable monikers as part of creating the resource. * resource.Moniker is a serializable string that uniquely identifies a resource in the Mu system. This is in contrast to resource IDs, which are generated by resource providers and generally opaque to the Mu system. See marapongo/mu#69 for more information about monikers and some of their challenges (namely, designing a stable algorithm). * resource.Snapshot is a "snapshot" taken from a graph of resources. This is a transitive closure of state representing one possible configuration of a given environment. This is what plans are created from. Eventually, two snapshots will be diffable, in order to perform incremental updates. One way of thinking about this is that a snapshot of the old world's state is advanced, one step at a time, until it reaches a desired snapshot of the new world's state. * resource.Plan is a plan for carrying out desired CRUD operations on a target environment. Each plan consists of zero-to-many Steps, each of which has a CRUD operation type, a resource target, and a next step. This is an enumerator because it is possible the plan will evolve -- and introduce new steps -- as it is carried out (hence, the Next() method). At the moment, this is linearized; eventually, we want to make this more "graph-like" so that we can exploit available parallelism within the dependencies. There are tons of TODOs remaining. However, the `mu plan` command is functioning with these new changes -- including colorization FTW -- so I'm landing it now. This is part of marapongo/mu#38 and marapongo/mu#41.
2017-02-17 21:31:48 +01:00
dest[PropertyKey(k)] = v
}
}
return dest
}
// cloneObjectValue creates a single property value out of a runtime object. It returns false if the property could not
// be stored in a property (e.g., it is a function or other unrecognized or unserializable runtime object).
Redo object monikers This change overhauls the way we do object monikers. The old mechanism, generating monikers using graph paths, was far too brittle and prone to collisions. The new approach mixes some amount of "automatic scoping" plus some "explicit naming." Although there is some explicitness, this is arguably a good thing, as the monikers will be relatable back to the source more readily by developers inspecting the graph and resource state. Each moniker has four parts: <Namespace>::<AllocModule>::<Type>::<Name> wherein each element is the following: <Namespace> The namespace being deployed into <AllocModule> The module in which the object was allocated <Type> The type of the resource <Name> The assigned name of the resource The <Namespace> is essentially the deployment target -- so "prod", "stage", etc -- although it is more general purpose to allow for future namespacing within a target (e.g., "prod/customer1", etc); for now this is rudimentary, however, see marapongo/mu#94. The <AllocModule> is the token for the code that contained the 'new' that led to this object being created. In the future, we may wish to extend this to also track the module under evaluation. (This is a nice aspect of monikers; they can become arbitrarily complex, so long as they are precise, and not prone to false positives/negatives.) The <Name> warrants more discussion. The resource provider is consulted via a new gRPC method, Name, that fetches the name. How the provider does this is entirely up to it. For some resource types, the resource may have properties that developers must set (e.g., `new Bucket("foo")`); for other providers, perhaps the resource intrinsically has a property that explicitly and uniquely qualifies the object (e.g., AWS SecurityGroups, via `new SecurityGroup({groupName: "my-sg"}`); and finally, it's conceivable that a provider might auto-generate the name (e.g., such as an AWS Lambda whose name could simply be a hash of the source code contents). This should overall produce better results with respect to moniker collisions, ability to match resources, and the usability of the system.
2017-02-24 23:50:02 +01:00
func cloneObjectValue(ctx *Context, obj *rt.Object) (PropertyValue, bool) {
Begin resource modeling and planning This change introduces a new package, pkg/resource, that will form the foundation for actually performing deployment plans and applications. It contains the following key abstractions: * resource.Provider is a wrapper around the CRUD operations exposed by underlying resource plugins. It will eventually defer to resource.Plugin, which itself defers -- over an RPC interface -- to the actual plugin, one per package exposing resources. The provider will also understand how to load, cache, and overall manage the lifetime of each plugin. * resource.Resource is the actual resource object. This is created from the overall evaluation object graph, but is simplified. It contains only serializable properties, for example. Inter-resource references are translated into serializable monikers as part of creating the resource. * resource.Moniker is a serializable string that uniquely identifies a resource in the Mu system. This is in contrast to resource IDs, which are generated by resource providers and generally opaque to the Mu system. See marapongo/mu#69 for more information about monikers and some of their challenges (namely, designing a stable algorithm). * resource.Snapshot is a "snapshot" taken from a graph of resources. This is a transitive closure of state representing one possible configuration of a given environment. This is what plans are created from. Eventually, two snapshots will be diffable, in order to perform incremental updates. One way of thinking about this is that a snapshot of the old world's state is advanced, one step at a time, until it reaches a desired snapshot of the new world's state. * resource.Plan is a plan for carrying out desired CRUD operations on a target environment. Each plan consists of zero-to-many Steps, each of which has a CRUD operation type, a resource target, and a next step. This is an enumerator because it is possible the plan will evolve -- and introduce new steps -- as it is carried out (hence, the Next() method). At the moment, this is linearized; eventually, we want to make this more "graph-like" so that we can exploit available parallelism within the dependencies. There are tons of TODOs remaining. However, the `mu plan` command is functioning with these new changes -- including colorization FTW -- so I'm landing it now. This is part of marapongo/mu#38 and marapongo/mu#41.
2017-02-17 21:31:48 +01:00
t := obj.Type()
if IsResourceType(t) {
// For resources, simply look up the moniker from the resource map.
Redo object monikers This change overhauls the way we do object monikers. The old mechanism, generating monikers using graph paths, was far too brittle and prone to collisions. The new approach mixes some amount of "automatic scoping" plus some "explicit naming." Although there is some explicitness, this is arguably a good thing, as the monikers will be relatable back to the source more readily by developers inspecting the graph and resource state. Each moniker has four parts: <Namespace>::<AllocModule>::<Type>::<Name> wherein each element is the following: <Namespace> The namespace being deployed into <AllocModule> The module in which the object was allocated <Type> The type of the resource <Name> The assigned name of the resource The <Namespace> is essentially the deployment target -- so "prod", "stage", etc -- although it is more general purpose to allow for future namespacing within a target (e.g., "prod/customer1", etc); for now this is rudimentary, however, see marapongo/mu#94. The <AllocModule> is the token for the code that contained the 'new' that led to this object being created. In the future, we may wish to extend this to also track the module under evaluation. (This is a nice aspect of monikers; they can become arbitrarily complex, so long as they are precise, and not prone to false positives/negatives.) The <Name> warrants more discussion. The resource provider is consulted via a new gRPC method, Name, that fetches the name. How the provider does this is entirely up to it. For some resource types, the resource may have properties that developers must set (e.g., `new Bucket("foo")`); for other providers, perhaps the resource intrinsically has a property that explicitly and uniquely qualifies the object (e.g., AWS SecurityGroups, via `new SecurityGroup({groupName: "my-sg"}`); and finally, it's conceivable that a provider might auto-generate the name (e.g., such as an AWS Lambda whose name could simply be a hash of the source code contents). This should overall produce better results with respect to moniker collisions, ability to match resources, and the usability of the system.
2017-02-24 23:50:02 +01:00
m, hasm := ctx.ObjMks[obj]
contract.Assertf(hasm, "Missing object reference; possible out of order dependency walk")
return NewPropertyResource(m), true
Begin resource modeling and planning This change introduces a new package, pkg/resource, that will form the foundation for actually performing deployment plans and applications. It contains the following key abstractions: * resource.Provider is a wrapper around the CRUD operations exposed by underlying resource plugins. It will eventually defer to resource.Plugin, which itself defers -- over an RPC interface -- to the actual plugin, one per package exposing resources. The provider will also understand how to load, cache, and overall manage the lifetime of each plugin. * resource.Resource is the actual resource object. This is created from the overall evaluation object graph, but is simplified. It contains only serializable properties, for example. Inter-resource references are translated into serializable monikers as part of creating the resource. * resource.Moniker is a serializable string that uniquely identifies a resource in the Mu system. This is in contrast to resource IDs, which are generated by resource providers and generally opaque to the Mu system. See marapongo/mu#69 for more information about monikers and some of their challenges (namely, designing a stable algorithm). * resource.Snapshot is a "snapshot" taken from a graph of resources. This is a transitive closure of state representing one possible configuration of a given environment. This is what plans are created from. Eventually, two snapshots will be diffable, in order to perform incremental updates. One way of thinking about this is that a snapshot of the old world's state is advanced, one step at a time, until it reaches a desired snapshot of the new world's state. * resource.Plan is a plan for carrying out desired CRUD operations on a target environment. Each plan consists of zero-to-many Steps, each of which has a CRUD operation type, a resource target, and a next step. This is an enumerator because it is possible the plan will evolve -- and introduce new steps -- as it is carried out (hence, the Next() method). At the moment, this is linearized; eventually, we want to make this more "graph-like" so that we can exploit available parallelism within the dependencies. There are tons of TODOs remaining. However, the `mu plan` command is functioning with these new changes -- including colorization FTW -- so I'm landing it now. This is part of marapongo/mu#38 and marapongo/mu#41.
2017-02-17 21:31:48 +01:00
}
switch t {
case types.Null:
return NewPropertyNull(), true
Begin resource modeling and planning This change introduces a new package, pkg/resource, that will form the foundation for actually performing deployment plans and applications. It contains the following key abstractions: * resource.Provider is a wrapper around the CRUD operations exposed by underlying resource plugins. It will eventually defer to resource.Plugin, which itself defers -- over an RPC interface -- to the actual plugin, one per package exposing resources. The provider will also understand how to load, cache, and overall manage the lifetime of each plugin. * resource.Resource is the actual resource object. This is created from the overall evaluation object graph, but is simplified. It contains only serializable properties, for example. Inter-resource references are translated into serializable monikers as part of creating the resource. * resource.Moniker is a serializable string that uniquely identifies a resource in the Mu system. This is in contrast to resource IDs, which are generated by resource providers and generally opaque to the Mu system. See marapongo/mu#69 for more information about monikers and some of their challenges (namely, designing a stable algorithm). * resource.Snapshot is a "snapshot" taken from a graph of resources. This is a transitive closure of state representing one possible configuration of a given environment. This is what plans are created from. Eventually, two snapshots will be diffable, in order to perform incremental updates. One way of thinking about this is that a snapshot of the old world's state is advanced, one step at a time, until it reaches a desired snapshot of the new world's state. * resource.Plan is a plan for carrying out desired CRUD operations on a target environment. Each plan consists of zero-to-many Steps, each of which has a CRUD operation type, a resource target, and a next step. This is an enumerator because it is possible the plan will evolve -- and introduce new steps -- as it is carried out (hence, the Next() method). At the moment, this is linearized; eventually, we want to make this more "graph-like" so that we can exploit available parallelism within the dependencies. There are tons of TODOs remaining. However, the `mu plan` command is functioning with these new changes -- including colorization FTW -- so I'm landing it now. This is part of marapongo/mu#38 and marapongo/mu#41.
2017-02-17 21:31:48 +01:00
case types.Bool:
return NewPropertyBool(obj.BoolValue()), true
case types.Number:
return NewPropertyNumber(obj.NumberValue()), true
case types.String:
return NewPropertyString(obj.StringValue()), true
case types.Object, types.Dynamic:
Redo object monikers This change overhauls the way we do object monikers. The old mechanism, generating monikers using graph paths, was far too brittle and prone to collisions. The new approach mixes some amount of "automatic scoping" plus some "explicit naming." Although there is some explicitness, this is arguably a good thing, as the monikers will be relatable back to the source more readily by developers inspecting the graph and resource state. Each moniker has four parts: <Namespace>::<AllocModule>::<Type>::<Name> wherein each element is the following: <Namespace> The namespace being deployed into <AllocModule> The module in which the object was allocated <Type> The type of the resource <Name> The assigned name of the resource The <Namespace> is essentially the deployment target -- so "prod", "stage", etc -- although it is more general purpose to allow for future namespacing within a target (e.g., "prod/customer1", etc); for now this is rudimentary, however, see marapongo/mu#94. The <AllocModule> is the token for the code that contained the 'new' that led to this object being created. In the future, we may wish to extend this to also track the module under evaluation. (This is a nice aspect of monikers; they can become arbitrarily complex, so long as they are precise, and not prone to false positives/negatives.) The <Name> warrants more discussion. The resource provider is consulted via a new gRPC method, Name, that fetches the name. How the provider does this is entirely up to it. For some resource types, the resource may have properties that developers must set (e.g., `new Bucket("foo")`); for other providers, perhaps the resource intrinsically has a property that explicitly and uniquely qualifies the object (e.g., AWS SecurityGroups, via `new SecurityGroup({groupName: "my-sg"}`); and finally, it's conceivable that a provider might auto-generate the name (e.g., such as an AWS Lambda whose name could simply be a hash of the source code contents). This should overall produce better results with respect to moniker collisions, ability to match resources, and the usability of the system.
2017-02-24 23:50:02 +01:00
obj := cloneObject(ctx, obj) // an object literal, clone it
Begin resource modeling and planning This change introduces a new package, pkg/resource, that will form the foundation for actually performing deployment plans and applications. It contains the following key abstractions: * resource.Provider is a wrapper around the CRUD operations exposed by underlying resource plugins. It will eventually defer to resource.Plugin, which itself defers -- over an RPC interface -- to the actual plugin, one per package exposing resources. The provider will also understand how to load, cache, and overall manage the lifetime of each plugin. * resource.Resource is the actual resource object. This is created from the overall evaluation object graph, but is simplified. It contains only serializable properties, for example. Inter-resource references are translated into serializable monikers as part of creating the resource. * resource.Moniker is a serializable string that uniquely identifies a resource in the Mu system. This is in contrast to resource IDs, which are generated by resource providers and generally opaque to the Mu system. See marapongo/mu#69 for more information about monikers and some of their challenges (namely, designing a stable algorithm). * resource.Snapshot is a "snapshot" taken from a graph of resources. This is a transitive closure of state representing one possible configuration of a given environment. This is what plans are created from. Eventually, two snapshots will be diffable, in order to perform incremental updates. One way of thinking about this is that a snapshot of the old world's state is advanced, one step at a time, until it reaches a desired snapshot of the new world's state. * resource.Plan is a plan for carrying out desired CRUD operations on a target environment. Each plan consists of zero-to-many Steps, each of which has a CRUD operation type, a resource target, and a next step. This is an enumerator because it is possible the plan will evolve -- and introduce new steps -- as it is carried out (hence, the Next() method). At the moment, this is linearized; eventually, we want to make this more "graph-like" so that we can exploit available parallelism within the dependencies. There are tons of TODOs remaining. However, the `mu plan` command is functioning with these new changes -- including colorization FTW -- so I'm landing it now. This is part of marapongo/mu#38 and marapongo/mu#41.
2017-02-17 21:31:48 +01:00
return NewPropertyObject(obj), true
}
switch t.(type) {
case *symbols.ArrayType:
var result []PropertyValue
for _, e := range *obj.ArrayValue() {
Redo object monikers This change overhauls the way we do object monikers. The old mechanism, generating monikers using graph paths, was far too brittle and prone to collisions. The new approach mixes some amount of "automatic scoping" plus some "explicit naming." Although there is some explicitness, this is arguably a good thing, as the monikers will be relatable back to the source more readily by developers inspecting the graph and resource state. Each moniker has four parts: <Namespace>::<AllocModule>::<Type>::<Name> wherein each element is the following: <Namespace> The namespace being deployed into <AllocModule> The module in which the object was allocated <Type> The type of the resource <Name> The assigned name of the resource The <Namespace> is essentially the deployment target -- so "prod", "stage", etc -- although it is more general purpose to allow for future namespacing within a target (e.g., "prod/customer1", etc); for now this is rudimentary, however, see marapongo/mu#94. The <AllocModule> is the token for the code that contained the 'new' that led to this object being created. In the future, we may wish to extend this to also track the module under evaluation. (This is a nice aspect of monikers; they can become arbitrarily complex, so long as they are precise, and not prone to false positives/negatives.) The <Name> warrants more discussion. The resource provider is consulted via a new gRPC method, Name, that fetches the name. How the provider does this is entirely up to it. For some resource types, the resource may have properties that developers must set (e.g., `new Bucket("foo")`); for other providers, perhaps the resource intrinsically has a property that explicitly and uniquely qualifies the object (e.g., AWS SecurityGroups, via `new SecurityGroup({groupName: "my-sg"}`); and finally, it's conceivable that a provider might auto-generate the name (e.g., such as an AWS Lambda whose name could simply be a hash of the source code contents). This should overall produce better results with respect to moniker collisions, ability to match resources, and the usability of the system.
2017-02-24 23:50:02 +01:00
if v, ok := cloneObjectValue(ctx, e.Obj()); ok {
Begin resource modeling and planning This change introduces a new package, pkg/resource, that will form the foundation for actually performing deployment plans and applications. It contains the following key abstractions: * resource.Provider is a wrapper around the CRUD operations exposed by underlying resource plugins. It will eventually defer to resource.Plugin, which itself defers -- over an RPC interface -- to the actual plugin, one per package exposing resources. The provider will also understand how to load, cache, and overall manage the lifetime of each plugin. * resource.Resource is the actual resource object. This is created from the overall evaluation object graph, but is simplified. It contains only serializable properties, for example. Inter-resource references are translated into serializable monikers as part of creating the resource. * resource.Moniker is a serializable string that uniquely identifies a resource in the Mu system. This is in contrast to resource IDs, which are generated by resource providers and generally opaque to the Mu system. See marapongo/mu#69 for more information about monikers and some of their challenges (namely, designing a stable algorithm). * resource.Snapshot is a "snapshot" taken from a graph of resources. This is a transitive closure of state representing one possible configuration of a given environment. This is what plans are created from. Eventually, two snapshots will be diffable, in order to perform incremental updates. One way of thinking about this is that a snapshot of the old world's state is advanced, one step at a time, until it reaches a desired snapshot of the new world's state. * resource.Plan is a plan for carrying out desired CRUD operations on a target environment. Each plan consists of zero-to-many Steps, each of which has a CRUD operation type, a resource target, and a next step. This is an enumerator because it is possible the plan will evolve -- and introduce new steps -- as it is carried out (hence, the Next() method). At the moment, this is linearized; eventually, we want to make this more "graph-like" so that we can exploit available parallelism within the dependencies. There are tons of TODOs remaining. However, the `mu plan` command is functioning with these new changes -- including colorization FTW -- so I'm landing it now. This is part of marapongo/mu#38 and marapongo/mu#41.
2017-02-17 21:31:48 +01:00
result = append(result, v)
}
}
return NewPropertyArray(result), true
case *symbols.Class:
Redo object monikers This change overhauls the way we do object monikers. The old mechanism, generating monikers using graph paths, was far too brittle and prone to collisions. The new approach mixes some amount of "automatic scoping" plus some "explicit naming." Although there is some explicitness, this is arguably a good thing, as the monikers will be relatable back to the source more readily by developers inspecting the graph and resource state. Each moniker has four parts: <Namespace>::<AllocModule>::<Type>::<Name> wherein each element is the following: <Namespace> The namespace being deployed into <AllocModule> The module in which the object was allocated <Type> The type of the resource <Name> The assigned name of the resource The <Namespace> is essentially the deployment target -- so "prod", "stage", etc -- although it is more general purpose to allow for future namespacing within a target (e.g., "prod/customer1", etc); for now this is rudimentary, however, see marapongo/mu#94. The <AllocModule> is the token for the code that contained the 'new' that led to this object being created. In the future, we may wish to extend this to also track the module under evaluation. (This is a nice aspect of monikers; they can become arbitrarily complex, so long as they are precise, and not prone to false positives/negatives.) The <Name> warrants more discussion. The resource provider is consulted via a new gRPC method, Name, that fetches the name. How the provider does this is entirely up to it. For some resource types, the resource may have properties that developers must set (e.g., `new Bucket("foo")`); for other providers, perhaps the resource intrinsically has a property that explicitly and uniquely qualifies the object (e.g., AWS SecurityGroups, via `new SecurityGroup({groupName: "my-sg"}`); and finally, it's conceivable that a provider might auto-generate the name (e.g., such as an AWS Lambda whose name could simply be a hash of the source code contents). This should overall produce better results with respect to moniker collisions, ability to match resources, and the usability of the system.
2017-02-24 23:50:02 +01:00
obj := cloneObject(ctx, obj) // a class, just deep clone it
Begin resource modeling and planning This change introduces a new package, pkg/resource, that will form the foundation for actually performing deployment plans and applications. It contains the following key abstractions: * resource.Provider is a wrapper around the CRUD operations exposed by underlying resource plugins. It will eventually defer to resource.Plugin, which itself defers -- over an RPC interface -- to the actual plugin, one per package exposing resources. The provider will also understand how to load, cache, and overall manage the lifetime of each plugin. * resource.Resource is the actual resource object. This is created from the overall evaluation object graph, but is simplified. It contains only serializable properties, for example. Inter-resource references are translated into serializable monikers as part of creating the resource. * resource.Moniker is a serializable string that uniquely identifies a resource in the Mu system. This is in contrast to resource IDs, which are generated by resource providers and generally opaque to the Mu system. See marapongo/mu#69 for more information about monikers and some of their challenges (namely, designing a stable algorithm). * resource.Snapshot is a "snapshot" taken from a graph of resources. This is a transitive closure of state representing one possible configuration of a given environment. This is what plans are created from. Eventually, two snapshots will be diffable, in order to perform incremental updates. One way of thinking about this is that a snapshot of the old world's state is advanced, one step at a time, until it reaches a desired snapshot of the new world's state. * resource.Plan is a plan for carrying out desired CRUD operations on a target environment. Each plan consists of zero-to-many Steps, each of which has a CRUD operation type, a resource target, and a next step. This is an enumerator because it is possible the plan will evolve -- and introduce new steps -- as it is carried out (hence, the Next() method). At the moment, this is linearized; eventually, we want to make this more "graph-like" so that we can exploit available parallelism within the dependencies. There are tons of TODOs remaining. However, the `mu plan` command is functioning with these new changes -- including colorization FTW -- so I'm landing it now. This is part of marapongo/mu#38 and marapongo/mu#41.
2017-02-17 21:31:48 +01:00
return NewPropertyObject(obj), true
}
// TODO: handle symbols.MapType.
// TODO: it's unclear if we should do something more drastic here. There will always be unrecognized property
// kinds because objects contain things like constructors, methods, etc. But we may want to ratchet this a bit.
glog.V(5).Infof("Ignoring object value of type '%v': unrecognized kind %v", t, reflect.TypeOf(t))
return PropertyValue{}, false
}