Begin resource modeling and planning
This change introduces a new package, pkg/resource, that will form
the foundation for actually performing deployment plans and applications.
It contains the following key abstractions:
* resource.Provider is a wrapper around the CRUD operations exposed by
underlying resource plugins. It will eventually defer to resource.Plugin,
which itself defers -- over an RPC interface -- to the actual plugin, one
per package exposing resources. The provider will also understand how to
load, cache, and overall manage the lifetime of each plugin.
* resource.Resource is the actual resource object. This is created from
the overall evaluation object graph, but is simplified. It contains only
serializable properties, for example. Inter-resource references are
translated into serializable monikers as part of creating the resource.
* resource.Moniker is a serializable string that uniquely identifies
a resource in the Mu system. This is in contrast to resource IDs, which
are generated by resource providers and generally opaque to the Mu
system. See marapongo/mu#69 for more information about monikers and some
of their challenges (namely, designing a stable algorithm).
* resource.Snapshot is a "snapshot" taken from a graph of resources. This
is a transitive closure of state representing one possible configuration
of a given environment. This is what plans are created from. Eventually,
two snapshots will be diffable, in order to perform incremental updates.
One way of thinking about this is that a snapshot of the old world's state
is advanced, one step at a time, until it reaches a desired snapshot of
the new world's state.
* resource.Plan is a plan for carrying out desired CRUD operations on a target
environment. Each plan consists of zero-to-many Steps, each of which has
a CRUD operation type, a resource target, and a next step. This is an
enumerator because it is possible the plan will evolve -- and introduce new
steps -- as it is carried out (hence, the Next() method). At the moment, this
is linearized; eventually, we want to make this more "graph-like" so that we
can exploit available parallelism within the dependencies.
There are tons of TODOs remaining. However, the `mu plan` command is functioning
with these new changes -- including colorization FTW -- so I'm landing it now.
This is part of marapongo/mu#38 and marapongo/mu#41.
2017-02-17 21:31:48 +01:00
|
|
|
// Copyright 2016 Marapongo, Inc. All rights reserved.
|
|
|
|
|
|
|
|
package resource
|
|
|
|
|
|
|
|
import (
|
|
|
|
"reflect"
|
|
|
|
|
|
|
|
"github.com/golang/glog"
|
|
|
|
|
|
|
|
"github.com/marapongo/mu/pkg/compiler/symbols"
|
|
|
|
"github.com/marapongo/mu/pkg/compiler/types"
|
|
|
|
"github.com/marapongo/mu/pkg/compiler/types/predef"
|
Implement updates
This change is a first whack at implementing updates.
Creation and deletion plans are pretty straightforward; we just take
a single graph, topologically sort it, and perform the operations in
the right order. For creation, this is in dependency order (things
that are depended upon must be created before dependents); for deletion,
this is in reverse-dependency order (things that depend on others must
be deleted before dependencies). These are just special cases of the more
general idea of performing DAG operations in dependency order.
Updates must work in terms of this more general notion. For example:
* It is an error to delete a resource while another refers to it; thus,
resources are deleted after deleting dependents, or after updating
dependent properties that reference the resource to new values.
* It is an error to depend on a create a resource before it is created;
thus, resources must be created before dependents are created, and/or
before updates to existing resource properties that would cause them
to refer to the new resource.
Of course, all of this is tangled up in a graph of dependencies. As a
result, we must create a DAG of the dependencies between creates, updates,
and deletes, and then topologically sort this DAG, in order to determine
the proper order of update operations.
To do this, we slightly generalize the existing graph infrastructure,
while also specializing two kinds of graphs; the existing one becomes a
heapstate.ObjectGraph, while this new one is resource.planGraph (internal).
2017-02-23 23:56:23 +01:00
|
|
|
"github.com/marapongo/mu/pkg/eval/heapstate"
|
Begin resource modeling and planning
This change introduces a new package, pkg/resource, that will form
the foundation for actually performing deployment plans and applications.
It contains the following key abstractions:
* resource.Provider is a wrapper around the CRUD operations exposed by
underlying resource plugins. It will eventually defer to resource.Plugin,
which itself defers -- over an RPC interface -- to the actual plugin, one
per package exposing resources. The provider will also understand how to
load, cache, and overall manage the lifetime of each plugin.
* resource.Resource is the actual resource object. This is created from
the overall evaluation object graph, but is simplified. It contains only
serializable properties, for example. Inter-resource references are
translated into serializable monikers as part of creating the resource.
* resource.Moniker is a serializable string that uniquely identifies
a resource in the Mu system. This is in contrast to resource IDs, which
are generated by resource providers and generally opaque to the Mu
system. See marapongo/mu#69 for more information about monikers and some
of their challenges (namely, designing a stable algorithm).
* resource.Snapshot is a "snapshot" taken from a graph of resources. This
is a transitive closure of state representing one possible configuration
of a given environment. This is what plans are created from. Eventually,
two snapshots will be diffable, in order to perform incremental updates.
One way of thinking about this is that a snapshot of the old world's state
is advanced, one step at a time, until it reaches a desired snapshot of
the new world's state.
* resource.Plan is a plan for carrying out desired CRUD operations on a target
environment. Each plan consists of zero-to-many Steps, each of which has
a CRUD operation type, a resource target, and a next step. This is an
enumerator because it is possible the plan will evolve -- and introduce new
steps -- as it is carried out (hence, the Next() method). At the moment, this
is linearized; eventually, we want to make this more "graph-like" so that we
can exploit available parallelism within the dependencies.
There are tons of TODOs remaining. However, the `mu plan` command is functioning
with these new changes -- including colorization FTW -- so I'm landing it now.
This is part of marapongo/mu#38 and marapongo/mu#41.
2017-02-17 21:31:48 +01:00
|
|
|
"github.com/marapongo/mu/pkg/eval/rt"
|
|
|
|
"github.com/marapongo/mu/pkg/tokens"
|
|
|
|
"github.com/marapongo/mu/pkg/util/contract"
|
|
|
|
)
|
|
|
|
|
|
|
|
// ID is a unique resource identifier; it is managed by the provider and is mostly opaque to Mu.
|
|
|
|
type ID string
|
|
|
|
|
|
|
|
// Resource is an instance of a resource with an ID, type, and bag of state.
|
|
|
|
type Resource interface {
|
|
|
|
ID() ID // the resource's unique ID, assigned by the resource provider (or blank if uncreated).
|
|
|
|
Moniker() Moniker // the resource's object moniker, a human-friendly, unique name for the resource.
|
2017-02-18 20:54:24 +01:00
|
|
|
Type() tokens.Type // the resource's type.
|
Begin resource modeling and planning
This change introduces a new package, pkg/resource, that will form
the foundation for actually performing deployment plans and applications.
It contains the following key abstractions:
* resource.Provider is a wrapper around the CRUD operations exposed by
underlying resource plugins. It will eventually defer to resource.Plugin,
which itself defers -- over an RPC interface -- to the actual plugin, one
per package exposing resources. The provider will also understand how to
load, cache, and overall manage the lifetime of each plugin.
* resource.Resource is the actual resource object. This is created from
the overall evaluation object graph, but is simplified. It contains only
serializable properties, for example. Inter-resource references are
translated into serializable monikers as part of creating the resource.
* resource.Moniker is a serializable string that uniquely identifies
a resource in the Mu system. This is in contrast to resource IDs, which
are generated by resource providers and generally opaque to the Mu
system. See marapongo/mu#69 for more information about monikers and some
of their challenges (namely, designing a stable algorithm).
* resource.Snapshot is a "snapshot" taken from a graph of resources. This
is a transitive closure of state representing one possible configuration
of a given environment. This is what plans are created from. Eventually,
two snapshots will be diffable, in order to perform incremental updates.
One way of thinking about this is that a snapshot of the old world's state
is advanced, one step at a time, until it reaches a desired snapshot of
the new world's state.
* resource.Plan is a plan for carrying out desired CRUD operations on a target
environment. Each plan consists of zero-to-many Steps, each of which has
a CRUD operation type, a resource target, and a next step. This is an
enumerator because it is possible the plan will evolve -- and introduce new
steps -- as it is carried out (hence, the Next() method). At the moment, this
is linearized; eventually, we want to make this more "graph-like" so that we
can exploit available parallelism within the dependencies.
There are tons of TODOs remaining. However, the `mu plan` command is functioning
with these new changes -- including colorization FTW -- so I'm landing it now.
This is part of marapongo/mu#38 and marapongo/mu#41.
2017-02-17 21:31:48 +01:00
|
|
|
Properties() PropertyMap // the resource's property map.
|
2017-02-22 23:43:08 +01:00
|
|
|
HasID() bool // returns true if the resource has been assigned an ID.
|
2017-02-18 20:54:24 +01:00
|
|
|
SetID(id ID) // assignes an ID to this resource, for those under creation.
|
Redo object monikers
This change overhauls the way we do object monikers. The old mechanism,
generating monikers using graph paths, was far too brittle and prone to
collisions. The new approach mixes some amount of "automatic scoping"
plus some "explicit naming." Although there is some explicitness, this
is arguably a good thing, as the monikers will be relatable back to the
source more readily by developers inspecting the graph and resource state.
Each moniker has four parts:
<Namespace>::<AllocModule>::<Type>::<Name>
wherein each element is the following:
<Namespace> The namespace being deployed into
<AllocModule> The module in which the object was allocated
<Type> The type of the resource
<Name> The assigned name of the resource
The <Namespace> is essentially the deployment target -- so "prod",
"stage", etc -- although it is more general purpose to allow for future
namespacing within a target (e.g., "prod/customer1", etc); for now
this is rudimentary, however, see marapongo/mu#94.
The <AllocModule> is the token for the code that contained the 'new'
that led to this object being created. In the future, we may wish to
extend this to also track the module under evaluation. (This is a nice
aspect of monikers; they can become arbitrarily complex, so long as
they are precise, and not prone to false positives/negatives.)
The <Name> warrants more discussion. The resource provider is consulted
via a new gRPC method, Name, that fetches the name. How the provider
does this is entirely up to it. For some resource types, the resource
may have properties that developers must set (e.g., `new Bucket("foo")`);
for other providers, perhaps the resource intrinsically has a property
that explicitly and uniquely qualifies the object (e.g., AWS SecurityGroups,
via `new SecurityGroup({groupName: "my-sg"}`); and finally, it's conceivable
that a provider might auto-generate the name (e.g., such as an AWS Lambda
whose name could simply be a hash of the source code contents).
This should overall produce better results with respect to moniker
collisions, ability to match resources, and the usability of the system.
2017-02-24 23:50:02 +01:00
|
|
|
HasMoniker() bool // returns true if the resource has been assigned moniker.
|
|
|
|
SetMoniker(m Moniker) // assignes a moniker to this resource, for those under creation.
|
Begin resource modeling and planning
This change introduces a new package, pkg/resource, that will form
the foundation for actually performing deployment plans and applications.
It contains the following key abstractions:
* resource.Provider is a wrapper around the CRUD operations exposed by
underlying resource plugins. It will eventually defer to resource.Plugin,
which itself defers -- over an RPC interface -- to the actual plugin, one
per package exposing resources. The provider will also understand how to
load, cache, and overall manage the lifetime of each plugin.
* resource.Resource is the actual resource object. This is created from
the overall evaluation object graph, but is simplified. It contains only
serializable properties, for example. Inter-resource references are
translated into serializable monikers as part of creating the resource.
* resource.Moniker is a serializable string that uniquely identifies
a resource in the Mu system. This is in contrast to resource IDs, which
are generated by resource providers and generally opaque to the Mu
system. See marapongo/mu#69 for more information about monikers and some
of their challenges (namely, designing a stable algorithm).
* resource.Snapshot is a "snapshot" taken from a graph of resources. This
is a transitive closure of state representing one possible configuration
of a given environment. This is what plans are created from. Eventually,
two snapshots will be diffable, in order to perform incremental updates.
One way of thinking about this is that a snapshot of the old world's state
is advanced, one step at a time, until it reaches a desired snapshot of
the new world's state.
* resource.Plan is a plan for carrying out desired CRUD operations on a target
environment. Each plan consists of zero-to-many Steps, each of which has
a CRUD operation type, a resource target, and a next step. This is an
enumerator because it is possible the plan will evolve -- and introduce new
steps -- as it is carried out (hence, the Next() method). At the moment, this
is linearized; eventually, we want to make this more "graph-like" so that we
can exploit available parallelism within the dependencies.
There are tons of TODOs remaining. However, the `mu plan` command is functioning
with these new changes -- including colorization FTW -- so I'm landing it now.
This is part of marapongo/mu#38 and marapongo/mu#41.
2017-02-17 21:31:48 +01:00
|
|
|
}
|
|
|
|
|
2017-02-18 20:54:24 +01:00
|
|
|
// ResourceState is returned when an error has occurred during a resource provider operation. It indicates whether the
|
|
|
|
// operation could be rolled back cleanly (OK). If not, it means the resource was left in an indeterminate state.
|
|
|
|
type ResourceState int
|
|
|
|
|
|
|
|
const (
|
|
|
|
StateOK ResourceState = iota
|
|
|
|
StateUnknown
|
|
|
|
)
|
|
|
|
|
Implement updates
This change is a first whack at implementing updates.
Creation and deletion plans are pretty straightforward; we just take
a single graph, topologically sort it, and perform the operations in
the right order. For creation, this is in dependency order (things
that are depended upon must be created before dependents); for deletion,
this is in reverse-dependency order (things that depend on others must
be deleted before dependencies). These are just special cases of the more
general idea of performing DAG operations in dependency order.
Updates must work in terms of this more general notion. For example:
* It is an error to delete a resource while another refers to it; thus,
resources are deleted after deleting dependents, or after updating
dependent properties that reference the resource to new values.
* It is an error to depend on a create a resource before it is created;
thus, resources must be created before dependents are created, and/or
before updates to existing resource properties that would cause them
to refer to the new resource.
Of course, all of this is tangled up in a graph of dependencies. As a
result, we must create a DAG of the dependencies between creates, updates,
and deletes, and then topologically sort this DAG, in order to determine
the proper order of update operations.
To do this, we slightly generalize the existing graph infrastructure,
while also specializing two kinds of graphs; the existing one becomes a
heapstate.ObjectGraph, while this new one is resource.planGraph (internal).
2017-02-23 23:56:23 +01:00
|
|
|
func IsResourceType(t symbols.Type) bool { return types.HasBaseName(t, predef.MuResourceClass) }
|
|
|
|
func IsResourceVertex(v *heapstate.ObjectVertex) bool { return IsResourceType(v.Obj().Type()) }
|
Begin resource modeling and planning
This change introduces a new package, pkg/resource, that will form
the foundation for actually performing deployment plans and applications.
It contains the following key abstractions:
* resource.Provider is a wrapper around the CRUD operations exposed by
underlying resource plugins. It will eventually defer to resource.Plugin,
which itself defers -- over an RPC interface -- to the actual plugin, one
per package exposing resources. The provider will also understand how to
load, cache, and overall manage the lifetime of each plugin.
* resource.Resource is the actual resource object. This is created from
the overall evaluation object graph, but is simplified. It contains only
serializable properties, for example. Inter-resource references are
translated into serializable monikers as part of creating the resource.
* resource.Moniker is a serializable string that uniquely identifies
a resource in the Mu system. This is in contrast to resource IDs, which
are generated by resource providers and generally opaque to the Mu
system. See marapongo/mu#69 for more information about monikers and some
of their challenges (namely, designing a stable algorithm).
* resource.Snapshot is a "snapshot" taken from a graph of resources. This
is a transitive closure of state representing one possible configuration
of a given environment. This is what plans are created from. Eventually,
two snapshots will be diffable, in order to perform incremental updates.
One way of thinking about this is that a snapshot of the old world's state
is advanced, one step at a time, until it reaches a desired snapshot of
the new world's state.
* resource.Plan is a plan for carrying out desired CRUD operations on a target
environment. Each plan consists of zero-to-many Steps, each of which has
a CRUD operation type, a resource target, and a next step. This is an
enumerator because it is possible the plan will evolve -- and introduce new
steps -- as it is carried out (hence, the Next() method). At the moment, this
is linearized; eventually, we want to make this more "graph-like" so that we
can exploit available parallelism within the dependencies.
There are tons of TODOs remaining. However, the `mu plan` command is functioning
with these new changes -- including colorization FTW -- so I'm landing it now.
This is part of marapongo/mu#38 and marapongo/mu#41.
2017-02-17 21:31:48 +01:00
|
|
|
|
|
|
|
type resource struct {
|
|
|
|
id ID // the resource's unique ID, assigned by the resource provider (or blank if uncreated).
|
|
|
|
moniker Moniker // the resource's object moniker, a human-friendly, unique name for the resource.
|
2017-02-18 20:54:24 +01:00
|
|
|
t tokens.Type // the resource's type.
|
Begin resource modeling and planning
This change introduces a new package, pkg/resource, that will form
the foundation for actually performing deployment plans and applications.
It contains the following key abstractions:
* resource.Provider is a wrapper around the CRUD operations exposed by
underlying resource plugins. It will eventually defer to resource.Plugin,
which itself defers -- over an RPC interface -- to the actual plugin, one
per package exposing resources. The provider will also understand how to
load, cache, and overall manage the lifetime of each plugin.
* resource.Resource is the actual resource object. This is created from
the overall evaluation object graph, but is simplified. It contains only
serializable properties, for example. Inter-resource references are
translated into serializable monikers as part of creating the resource.
* resource.Moniker is a serializable string that uniquely identifies
a resource in the Mu system. This is in contrast to resource IDs, which
are generated by resource providers and generally opaque to the Mu
system. See marapongo/mu#69 for more information about monikers and some
of their challenges (namely, designing a stable algorithm).
* resource.Snapshot is a "snapshot" taken from a graph of resources. This
is a transitive closure of state representing one possible configuration
of a given environment. This is what plans are created from. Eventually,
two snapshots will be diffable, in order to perform incremental updates.
One way of thinking about this is that a snapshot of the old world's state
is advanced, one step at a time, until it reaches a desired snapshot of
the new world's state.
* resource.Plan is a plan for carrying out desired CRUD operations on a target
environment. Each plan consists of zero-to-many Steps, each of which has
a CRUD operation type, a resource target, and a next step. This is an
enumerator because it is possible the plan will evolve -- and introduce new
steps -- as it is carried out (hence, the Next() method). At the moment, this
is linearized; eventually, we want to make this more "graph-like" so that we
can exploit available parallelism within the dependencies.
There are tons of TODOs remaining. However, the `mu plan` command is functioning
with these new changes -- including colorization FTW -- so I'm landing it now.
This is part of marapongo/mu#38 and marapongo/mu#41.
2017-02-17 21:31:48 +01:00
|
|
|
properties PropertyMap // the resource's property map.
|
|
|
|
}
|
|
|
|
|
|
|
|
func (r *resource) ID() ID { return r.id }
|
|
|
|
func (r *resource) Moniker() Moniker { return r.moniker }
|
2017-02-18 20:54:24 +01:00
|
|
|
func (r *resource) Type() tokens.Type { return r.t }
|
Begin resource modeling and planning
This change introduces a new package, pkg/resource, that will form
the foundation for actually performing deployment plans and applications.
It contains the following key abstractions:
* resource.Provider is a wrapper around the CRUD operations exposed by
underlying resource plugins. It will eventually defer to resource.Plugin,
which itself defers -- over an RPC interface -- to the actual plugin, one
per package exposing resources. The provider will also understand how to
load, cache, and overall manage the lifetime of each plugin.
* resource.Resource is the actual resource object. This is created from
the overall evaluation object graph, but is simplified. It contains only
serializable properties, for example. Inter-resource references are
translated into serializable monikers as part of creating the resource.
* resource.Moniker is a serializable string that uniquely identifies
a resource in the Mu system. This is in contrast to resource IDs, which
are generated by resource providers and generally opaque to the Mu
system. See marapongo/mu#69 for more information about monikers and some
of their challenges (namely, designing a stable algorithm).
* resource.Snapshot is a "snapshot" taken from a graph of resources. This
is a transitive closure of state representing one possible configuration
of a given environment. This is what plans are created from. Eventually,
two snapshots will be diffable, in order to perform incremental updates.
One way of thinking about this is that a snapshot of the old world's state
is advanced, one step at a time, until it reaches a desired snapshot of
the new world's state.
* resource.Plan is a plan for carrying out desired CRUD operations on a target
environment. Each plan consists of zero-to-many Steps, each of which has
a CRUD operation type, a resource target, and a next step. This is an
enumerator because it is possible the plan will evolve -- and introduce new
steps -- as it is carried out (hence, the Next() method). At the moment, this
is linearized; eventually, we want to make this more "graph-like" so that we
can exploit available parallelism within the dependencies.
There are tons of TODOs remaining. However, the `mu plan` command is functioning
with these new changes -- including colorization FTW -- so I'm landing it now.
This is part of marapongo/mu#38 and marapongo/mu#41.
2017-02-17 21:31:48 +01:00
|
|
|
func (r *resource) Properties() PropertyMap { return r.properties }
|
|
|
|
|
2017-02-22 23:43:08 +01:00
|
|
|
func (r *resource) HasID() bool { return (string(r.id) != "") }
|
2017-02-18 20:54:24 +01:00
|
|
|
func (r *resource) SetID(id ID) {
|
2017-02-22 23:43:08 +01:00
|
|
|
contract.Requiref(!r.HasID(), "id", "empty")
|
2017-02-18 20:54:24 +01:00
|
|
|
r.id = id
|
|
|
|
}
|
|
|
|
|
Redo object monikers
This change overhauls the way we do object monikers. The old mechanism,
generating monikers using graph paths, was far too brittle and prone to
collisions. The new approach mixes some amount of "automatic scoping"
plus some "explicit naming." Although there is some explicitness, this
is arguably a good thing, as the monikers will be relatable back to the
source more readily by developers inspecting the graph and resource state.
Each moniker has four parts:
<Namespace>::<AllocModule>::<Type>::<Name>
wherein each element is the following:
<Namespace> The namespace being deployed into
<AllocModule> The module in which the object was allocated
<Type> The type of the resource
<Name> The assigned name of the resource
The <Namespace> is essentially the deployment target -- so "prod",
"stage", etc -- although it is more general purpose to allow for future
namespacing within a target (e.g., "prod/customer1", etc); for now
this is rudimentary, however, see marapongo/mu#94.
The <AllocModule> is the token for the code that contained the 'new'
that led to this object being created. In the future, we may wish to
extend this to also track the module under evaluation. (This is a nice
aspect of monikers; they can become arbitrarily complex, so long as
they are precise, and not prone to false positives/negatives.)
The <Name> warrants more discussion. The resource provider is consulted
via a new gRPC method, Name, that fetches the name. How the provider
does this is entirely up to it. For some resource types, the resource
may have properties that developers must set (e.g., `new Bucket("foo")`);
for other providers, perhaps the resource intrinsically has a property
that explicitly and uniquely qualifies the object (e.g., AWS SecurityGroups,
via `new SecurityGroup({groupName: "my-sg"}`); and finally, it's conceivable
that a provider might auto-generate the name (e.g., such as an AWS Lambda
whose name could simply be a hash of the source code contents).
This should overall produce better results with respect to moniker
collisions, ability to match resources, and the usability of the system.
2017-02-24 23:50:02 +01:00
|
|
|
func (r *resource) HasMoniker() bool { return (string(r.moniker) != "") }
|
|
|
|
func (r *resource) SetMoniker(m Moniker) {
|
|
|
|
contract.Requiref(!r.HasMoniker(), "moniker", "empty")
|
|
|
|
r.moniker = m
|
|
|
|
}
|
|
|
|
|
Implement resource provider plugins
This change adds basic support for discovering, loading, binding to,
and invoking RPC methods on, resource provider plugins.
In a nutshell, we add a new context object that will share cached
state such as loaded plugins and connections to them. It will be
a policy decision in server scenarios how much state to share and
between whom. This context also controls per-resource context
allocation, which in the future will allow us to perform structured
cancellation and teardown amongst entire groups of requests.
Plugins are loaded based on their name, and can be found in one of
two ways: either simply by having them on your path (with a name of
"mu-ressrv-<pkg>", where "<pkg>" is the resource package name with
any "/"s replaced with "_"s); or by placing them in the standard
library installation location, which need not be on the path for this
to work (since we know precisely where to look).
If we find a protocol, we will load it as a child process.
The protocol for plugins is that they will choose a port on their
own -- to eliminate races that'd be involved should Mu attempt to
pre-pick one for them -- and then write that out as the first line
to STDOUT (terminated by a "\n"). This is the only STDERR/STDOUT
that Mu cares about; from there, the plugin is free to write all it
pleases (e.g., for logging, debugging purposes, etc).
Afterwards, we then bind our gRPC connection to that port, and create
a typed resource provider client. The CRUD operations that get driven
by plan application are then simple wrappers atop the underlying gRPC
calls. For now, we interpret all errors as catastrophic; in the near
future, we will probably want to introduce a "structured error"
mechanism in the gRPC interface for "transactional errors"; that is,
errors for which the server was able to recover to a safe checkpoint,
which can be interpreted as ResourceOK rather than ResourceUnknown.
2017-02-19 20:08:06 +01:00
|
|
|
// NewResource creates a new resource from the information provided.
|
|
|
|
func NewResource(id ID, moniker Moniker, t tokens.Type, properties PropertyMap) Resource {
|
|
|
|
return &resource{
|
|
|
|
id: id,
|
|
|
|
moniker: moniker,
|
|
|
|
t: t,
|
|
|
|
properties: properties,
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
Redo object monikers
This change overhauls the way we do object monikers. The old mechanism,
generating monikers using graph paths, was far too brittle and prone to
collisions. The new approach mixes some amount of "automatic scoping"
plus some "explicit naming." Although there is some explicitness, this
is arguably a good thing, as the monikers will be relatable back to the
source more readily by developers inspecting the graph and resource state.
Each moniker has four parts:
<Namespace>::<AllocModule>::<Type>::<Name>
wherein each element is the following:
<Namespace> The namespace being deployed into
<AllocModule> The module in which the object was allocated
<Type> The type of the resource
<Name> The assigned name of the resource
The <Namespace> is essentially the deployment target -- so "prod",
"stage", etc -- although it is more general purpose to allow for future
namespacing within a target (e.g., "prod/customer1", etc); for now
this is rudimentary, however, see marapongo/mu#94.
The <AllocModule> is the token for the code that contained the 'new'
that led to this object being created. In the future, we may wish to
extend this to also track the module under evaluation. (This is a nice
aspect of monikers; they can become arbitrarily complex, so long as
they are precise, and not prone to false positives/negatives.)
The <Name> warrants more discussion. The resource provider is consulted
via a new gRPC method, Name, that fetches the name. How the provider
does this is entirely up to it. For some resource types, the resource
may have properties that developers must set (e.g., `new Bucket("foo")`);
for other providers, perhaps the resource intrinsically has a property
that explicitly and uniquely qualifies the object (e.g., AWS SecurityGroups,
via `new SecurityGroup({groupName: "my-sg"}`); and finally, it's conceivable
that a provider might auto-generate the name (e.g., such as an AWS Lambda
whose name could simply be a hash of the source code contents).
This should overall produce better results with respect to moniker
collisions, ability to match resources, and the usability of the system.
2017-02-24 23:50:02 +01:00
|
|
|
// NewObjectResource creates a new resource object out of the runtime object provided. The context is used to resolve
|
2017-02-18 19:22:04 +01:00
|
|
|
// dependencies between resources and must contain all references that could be encountered.
|
Redo object monikers
This change overhauls the way we do object monikers. The old mechanism,
generating monikers using graph paths, was far too brittle and prone to
collisions. The new approach mixes some amount of "automatic scoping"
plus some "explicit naming." Although there is some explicitness, this
is arguably a good thing, as the monikers will be relatable back to the
source more readily by developers inspecting the graph and resource state.
Each moniker has four parts:
<Namespace>::<AllocModule>::<Type>::<Name>
wherein each element is the following:
<Namespace> The namespace being deployed into
<AllocModule> The module in which the object was allocated
<Type> The type of the resource
<Name> The assigned name of the resource
The <Namespace> is essentially the deployment target -- so "prod",
"stage", etc -- although it is more general purpose to allow for future
namespacing within a target (e.g., "prod/customer1", etc); for now
this is rudimentary, however, see marapongo/mu#94.
The <AllocModule> is the token for the code that contained the 'new'
that led to this object being created. In the future, we may wish to
extend this to also track the module under evaluation. (This is a nice
aspect of monikers; they can become arbitrarily complex, so long as
they are precise, and not prone to false positives/negatives.)
The <Name> warrants more discussion. The resource provider is consulted
via a new gRPC method, Name, that fetches the name. How the provider
does this is entirely up to it. For some resource types, the resource
may have properties that developers must set (e.g., `new Bucket("foo")`);
for other providers, perhaps the resource intrinsically has a property
that explicitly and uniquely qualifies the object (e.g., AWS SecurityGroups,
via `new SecurityGroup({groupName: "my-sg"}`); and finally, it's conceivable
that a provider might auto-generate the name (e.g., such as an AWS Lambda
whose name could simply be a hash of the source code contents).
This should overall produce better results with respect to moniker
collisions, ability to match resources, and the usability of the system.
2017-02-24 23:50:02 +01:00
|
|
|
func NewObjectResource(ctx *Context, obj *rt.Object) Resource {
|
2017-02-18 19:22:04 +01:00
|
|
|
t := obj.Type()
|
|
|
|
contract.Assert(IsResourceType(t))
|
Begin resource modeling and planning
This change introduces a new package, pkg/resource, that will form
the foundation for actually performing deployment plans and applications.
It contains the following key abstractions:
* resource.Provider is a wrapper around the CRUD operations exposed by
underlying resource plugins. It will eventually defer to resource.Plugin,
which itself defers -- over an RPC interface -- to the actual plugin, one
per package exposing resources. The provider will also understand how to
load, cache, and overall manage the lifetime of each plugin.
* resource.Resource is the actual resource object. This is created from
the overall evaluation object graph, but is simplified. It contains only
serializable properties, for example. Inter-resource references are
translated into serializable monikers as part of creating the resource.
* resource.Moniker is a serializable string that uniquely identifies
a resource in the Mu system. This is in contrast to resource IDs, which
are generated by resource providers and generally opaque to the Mu
system. See marapongo/mu#69 for more information about monikers and some
of their challenges (namely, designing a stable algorithm).
* resource.Snapshot is a "snapshot" taken from a graph of resources. This
is a transitive closure of state representing one possible configuration
of a given environment. This is what plans are created from. Eventually,
two snapshots will be diffable, in order to perform incremental updates.
One way of thinking about this is that a snapshot of the old world's state
is advanced, one step at a time, until it reaches a desired snapshot of
the new world's state.
* resource.Plan is a plan for carrying out desired CRUD operations on a target
environment. Each plan consists of zero-to-many Steps, each of which has
a CRUD operation type, a resource target, and a next step. This is an
enumerator because it is possible the plan will evolve -- and introduce new
steps -- as it is carried out (hence, the Next() method). At the moment, this
is linearized; eventually, we want to make this more "graph-like" so that we
can exploit available parallelism within the dependencies.
There are tons of TODOs remaining. However, the `mu plan` command is functioning
with these new changes -- including colorization FTW -- so I'm landing it now.
This is part of marapongo/mu#38 and marapongo/mu#41.
2017-02-17 21:31:48 +01:00
|
|
|
|
2017-02-18 19:22:04 +01:00
|
|
|
// Extract the moniker. This must already exist.
|
Redo object monikers
This change overhauls the way we do object monikers. The old mechanism,
generating monikers using graph paths, was far too brittle and prone to
collisions. The new approach mixes some amount of "automatic scoping"
plus some "explicit naming." Although there is some explicitness, this
is arguably a good thing, as the monikers will be relatable back to the
source more readily by developers inspecting the graph and resource state.
Each moniker has four parts:
<Namespace>::<AllocModule>::<Type>::<Name>
wherein each element is the following:
<Namespace> The namespace being deployed into
<AllocModule> The module in which the object was allocated
<Type> The type of the resource
<Name> The assigned name of the resource
The <Namespace> is essentially the deployment target -- so "prod",
"stage", etc -- although it is more general purpose to allow for future
namespacing within a target (e.g., "prod/customer1", etc); for now
this is rudimentary, however, see marapongo/mu#94.
The <AllocModule> is the token for the code that contained the 'new'
that led to this object being created. In the future, we may wish to
extend this to also track the module under evaluation. (This is a nice
aspect of monikers; they can become arbitrarily complex, so long as
they are precise, and not prone to false positives/negatives.)
The <Name> warrants more discussion. The resource provider is consulted
via a new gRPC method, Name, that fetches the name. How the provider
does this is entirely up to it. For some resource types, the resource
may have properties that developers must set (e.g., `new Bucket("foo")`);
for other providers, perhaps the resource intrinsically has a property
that explicitly and uniquely qualifies the object (e.g., AWS SecurityGroups,
via `new SecurityGroup({groupName: "my-sg"}`); and finally, it's conceivable
that a provider might auto-generate the name (e.g., such as an AWS Lambda
whose name could simply be a hash of the source code contents).
This should overall produce better results with respect to moniker
collisions, ability to match resources, and the usability of the system.
2017-02-24 23:50:02 +01:00
|
|
|
m, hasm := ctx.ObjMks[obj]
|
|
|
|
contract.Assertf(!hasm, "Object already assigned a moniker '%v'; double allocation detected", m)
|
Begin resource modeling and planning
This change introduces a new package, pkg/resource, that will form
the foundation for actually performing deployment plans and applications.
It contains the following key abstractions:
* resource.Provider is a wrapper around the CRUD operations exposed by
underlying resource plugins. It will eventually defer to resource.Plugin,
which itself defers -- over an RPC interface -- to the actual plugin, one
per package exposing resources. The provider will also understand how to
load, cache, and overall manage the lifetime of each plugin.
* resource.Resource is the actual resource object. This is created from
the overall evaluation object graph, but is simplified. It contains only
serializable properties, for example. Inter-resource references are
translated into serializable monikers as part of creating the resource.
* resource.Moniker is a serializable string that uniquely identifies
a resource in the Mu system. This is in contrast to resource IDs, which
are generated by resource providers and generally opaque to the Mu
system. See marapongo/mu#69 for more information about monikers and some
of their challenges (namely, designing a stable algorithm).
* resource.Snapshot is a "snapshot" taken from a graph of resources. This
is a transitive closure of state representing one possible configuration
of a given environment. This is what plans are created from. Eventually,
two snapshots will be diffable, in order to perform incremental updates.
One way of thinking about this is that a snapshot of the old world's state
is advanced, one step at a time, until it reaches a desired snapshot of
the new world's state.
* resource.Plan is a plan for carrying out desired CRUD operations on a target
environment. Each plan consists of zero-to-many Steps, each of which has
a CRUD operation type, a resource target, and a next step. This is an
enumerator because it is possible the plan will evolve -- and introduce new
steps -- as it is carried out (hence, the Next() method). At the moment, this
is linearized; eventually, we want to make this more "graph-like" so that we
can exploit available parallelism within the dependencies.
There are tons of TODOs remaining. However, the `mu plan` command is functioning
with these new changes -- including colorization FTW -- so I'm landing it now.
This is part of marapongo/mu#38 and marapongo/mu#41.
2017-02-17 21:31:48 +01:00
|
|
|
|
2017-02-18 19:22:04 +01:00
|
|
|
// Do a deep copy of the resource properties. This ensures property serializability.
|
Redo object monikers
This change overhauls the way we do object monikers. The old mechanism,
generating monikers using graph paths, was far too brittle and prone to
collisions. The new approach mixes some amount of "automatic scoping"
plus some "explicit naming." Although there is some explicitness, this
is arguably a good thing, as the monikers will be relatable back to the
source more readily by developers inspecting the graph and resource state.
Each moniker has four parts:
<Namespace>::<AllocModule>::<Type>::<Name>
wherein each element is the following:
<Namespace> The namespace being deployed into
<AllocModule> The module in which the object was allocated
<Type> The type of the resource
<Name> The assigned name of the resource
The <Namespace> is essentially the deployment target -- so "prod",
"stage", etc -- although it is more general purpose to allow for future
namespacing within a target (e.g., "prod/customer1", etc); for now
this is rudimentary, however, see marapongo/mu#94.
The <AllocModule> is the token for the code that contained the 'new'
that led to this object being created. In the future, we may wish to
extend this to also track the module under evaluation. (This is a nice
aspect of monikers; they can become arbitrarily complex, so long as
they are precise, and not prone to false positives/negatives.)
The <Name> warrants more discussion. The resource provider is consulted
via a new gRPC method, Name, that fetches the name. How the provider
does this is entirely up to it. For some resource types, the resource
may have properties that developers must set (e.g., `new Bucket("foo")`);
for other providers, perhaps the resource intrinsically has a property
that explicitly and uniquely qualifies the object (e.g., AWS SecurityGroups,
via `new SecurityGroup({groupName: "my-sg"}`); and finally, it's conceivable
that a provider might auto-generate the name (e.g., such as an AWS Lambda
whose name could simply be a hash of the source code contents).
This should overall produce better results with respect to moniker
collisions, ability to match resources, and the usability of the system.
2017-02-24 23:50:02 +01:00
|
|
|
props := cloneObject(ctx, obj)
|
Begin resource modeling and planning
This change introduces a new package, pkg/resource, that will form
the foundation for actually performing deployment plans and applications.
It contains the following key abstractions:
* resource.Provider is a wrapper around the CRUD operations exposed by
underlying resource plugins. It will eventually defer to resource.Plugin,
which itself defers -- over an RPC interface -- to the actual plugin, one
per package exposing resources. The provider will also understand how to
load, cache, and overall manage the lifetime of each plugin.
* resource.Resource is the actual resource object. This is created from
the overall evaluation object graph, but is simplified. It contains only
serializable properties, for example. Inter-resource references are
translated into serializable monikers as part of creating the resource.
* resource.Moniker is a serializable string that uniquely identifies
a resource in the Mu system. This is in contrast to resource IDs, which
are generated by resource providers and generally opaque to the Mu
system. See marapongo/mu#69 for more information about monikers and some
of their challenges (namely, designing a stable algorithm).
* resource.Snapshot is a "snapshot" taken from a graph of resources. This
is a transitive closure of state representing one possible configuration
of a given environment. This is what plans are created from. Eventually,
two snapshots will be diffable, in order to perform incremental updates.
One way of thinking about this is that a snapshot of the old world's state
is advanced, one step at a time, until it reaches a desired snapshot of
the new world's state.
* resource.Plan is a plan for carrying out desired CRUD operations on a target
environment. Each plan consists of zero-to-many Steps, each of which has
a CRUD operation type, a resource target, and a next step. This is an
enumerator because it is possible the plan will evolve -- and introduce new
steps -- as it is carried out (hence, the Next() method). At the moment, this
is linearized; eventually, we want to make this more "graph-like" so that we
can exploit available parallelism within the dependencies.
There are tons of TODOs remaining. However, the `mu plan` command is functioning
with these new changes -- including colorization FTW -- so I'm landing it now.
This is part of marapongo/mu#38 and marapongo/mu#41.
2017-02-17 21:31:48 +01:00
|
|
|
|
|
|
|
// Finally allocate and return the resource object; note that ID is left blank until the provider assignes one.
|
|
|
|
return &resource{
|
2017-02-18 20:54:24 +01:00
|
|
|
t: t.TypeToken(),
|
Begin resource modeling and planning
This change introduces a new package, pkg/resource, that will form
the foundation for actually performing deployment plans and applications.
It contains the following key abstractions:
* resource.Provider is a wrapper around the CRUD operations exposed by
underlying resource plugins. It will eventually defer to resource.Plugin,
which itself defers -- over an RPC interface -- to the actual plugin, one
per package exposing resources. The provider will also understand how to
load, cache, and overall manage the lifetime of each plugin.
* resource.Resource is the actual resource object. This is created from
the overall evaluation object graph, but is simplified. It contains only
serializable properties, for example. Inter-resource references are
translated into serializable monikers as part of creating the resource.
* resource.Moniker is a serializable string that uniquely identifies
a resource in the Mu system. This is in contrast to resource IDs, which
are generated by resource providers and generally opaque to the Mu
system. See marapongo/mu#69 for more information about monikers and some
of their challenges (namely, designing a stable algorithm).
* resource.Snapshot is a "snapshot" taken from a graph of resources. This
is a transitive closure of state representing one possible configuration
of a given environment. This is what plans are created from. Eventually,
two snapshots will be diffable, in order to perform incremental updates.
One way of thinking about this is that a snapshot of the old world's state
is advanced, one step at a time, until it reaches a desired snapshot of
the new world's state.
* resource.Plan is a plan for carrying out desired CRUD operations on a target
environment. Each plan consists of zero-to-many Steps, each of which has
a CRUD operation type, a resource target, and a next step. This is an
enumerator because it is possible the plan will evolve -- and introduce new
steps -- as it is carried out (hence, the Next() method). At the moment, this
is linearized; eventually, we want to make this more "graph-like" so that we
can exploit available parallelism within the dependencies.
There are tons of TODOs remaining. However, the `mu plan` command is functioning
with these new changes -- including colorization FTW -- so I'm landing it now.
This is part of marapongo/mu#38 and marapongo/mu#41.
2017-02-17 21:31:48 +01:00
|
|
|
properties: props,
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
// cloneObject creates a property map out of a runtime object. The result is fully serializable in the sense that it
|
|
|
|
// can be stored in a JSON or YAML file, serialized over an RPC interface, etc. In particular, any references to other
|
|
|
|
// resources are replaced with their moniker equivalents, which the runtime understands.
|
Redo object monikers
This change overhauls the way we do object monikers. The old mechanism,
generating monikers using graph paths, was far too brittle and prone to
collisions. The new approach mixes some amount of "automatic scoping"
plus some "explicit naming." Although there is some explicitness, this
is arguably a good thing, as the monikers will be relatable back to the
source more readily by developers inspecting the graph and resource state.
Each moniker has four parts:
<Namespace>::<AllocModule>::<Type>::<Name>
wherein each element is the following:
<Namespace> The namespace being deployed into
<AllocModule> The module in which the object was allocated
<Type> The type of the resource
<Name> The assigned name of the resource
The <Namespace> is essentially the deployment target -- so "prod",
"stage", etc -- although it is more general purpose to allow for future
namespacing within a target (e.g., "prod/customer1", etc); for now
this is rudimentary, however, see marapongo/mu#94.
The <AllocModule> is the token for the code that contained the 'new'
that led to this object being created. In the future, we may wish to
extend this to also track the module under evaluation. (This is a nice
aspect of monikers; they can become arbitrarily complex, so long as
they are precise, and not prone to false positives/negatives.)
The <Name> warrants more discussion. The resource provider is consulted
via a new gRPC method, Name, that fetches the name. How the provider
does this is entirely up to it. For some resource types, the resource
may have properties that developers must set (e.g., `new Bucket("foo")`);
for other providers, perhaps the resource intrinsically has a property
that explicitly and uniquely qualifies the object (e.g., AWS SecurityGroups,
via `new SecurityGroup({groupName: "my-sg"}`); and finally, it's conceivable
that a provider might auto-generate the name (e.g., such as an AWS Lambda
whose name could simply be a hash of the source code contents).
This should overall produce better results with respect to moniker
collisions, ability to match resources, and the usability of the system.
2017-02-24 23:50:02 +01:00
|
|
|
func cloneObject(ctx *Context, obj *rt.Object) PropertyMap {
|
Begin resource modeling and planning
This change introduces a new package, pkg/resource, that will form
the foundation for actually performing deployment plans and applications.
It contains the following key abstractions:
* resource.Provider is a wrapper around the CRUD operations exposed by
underlying resource plugins. It will eventually defer to resource.Plugin,
which itself defers -- over an RPC interface -- to the actual plugin, one
per package exposing resources. The provider will also understand how to
load, cache, and overall manage the lifetime of each plugin.
* resource.Resource is the actual resource object. This is created from
the overall evaluation object graph, but is simplified. It contains only
serializable properties, for example. Inter-resource references are
translated into serializable monikers as part of creating the resource.
* resource.Moniker is a serializable string that uniquely identifies
a resource in the Mu system. This is in contrast to resource IDs, which
are generated by resource providers and generally opaque to the Mu
system. See marapongo/mu#69 for more information about monikers and some
of their challenges (namely, designing a stable algorithm).
* resource.Snapshot is a "snapshot" taken from a graph of resources. This
is a transitive closure of state representing one possible configuration
of a given environment. This is what plans are created from. Eventually,
two snapshots will be diffable, in order to perform incremental updates.
One way of thinking about this is that a snapshot of the old world's state
is advanced, one step at a time, until it reaches a desired snapshot of
the new world's state.
* resource.Plan is a plan for carrying out desired CRUD operations on a target
environment. Each plan consists of zero-to-many Steps, each of which has
a CRUD operation type, a resource target, and a next step. This is an
enumerator because it is possible the plan will evolve -- and introduce new
steps -- as it is carried out (hence, the Next() method). At the moment, this
is linearized; eventually, we want to make this more "graph-like" so that we
can exploit available parallelism within the dependencies.
There are tons of TODOs remaining. However, the `mu plan` command is functioning
with these new changes -- including colorization FTW -- so I'm landing it now.
This is part of marapongo/mu#38 and marapongo/mu#41.
2017-02-17 21:31:48 +01:00
|
|
|
contract.Assert(obj != nil)
|
|
|
|
src := obj.PropertyValues()
|
|
|
|
dest := make(PropertyMap)
|
|
|
|
for _, k := range rt.StablePropertyKeys(src) {
|
|
|
|
// TODO: detect cycles.
|
Redo object monikers
This change overhauls the way we do object monikers. The old mechanism,
generating monikers using graph paths, was far too brittle and prone to
collisions. The new approach mixes some amount of "automatic scoping"
plus some "explicit naming." Although there is some explicitness, this
is arguably a good thing, as the monikers will be relatable back to the
source more readily by developers inspecting the graph and resource state.
Each moniker has four parts:
<Namespace>::<AllocModule>::<Type>::<Name>
wherein each element is the following:
<Namespace> The namespace being deployed into
<AllocModule> The module in which the object was allocated
<Type> The type of the resource
<Name> The assigned name of the resource
The <Namespace> is essentially the deployment target -- so "prod",
"stage", etc -- although it is more general purpose to allow for future
namespacing within a target (e.g., "prod/customer1", etc); for now
this is rudimentary, however, see marapongo/mu#94.
The <AllocModule> is the token for the code that contained the 'new'
that led to this object being created. In the future, we may wish to
extend this to also track the module under evaluation. (This is a nice
aspect of monikers; they can become arbitrarily complex, so long as
they are precise, and not prone to false positives/negatives.)
The <Name> warrants more discussion. The resource provider is consulted
via a new gRPC method, Name, that fetches the name. How the provider
does this is entirely up to it. For some resource types, the resource
may have properties that developers must set (e.g., `new Bucket("foo")`);
for other providers, perhaps the resource intrinsically has a property
that explicitly and uniquely qualifies the object (e.g., AWS SecurityGroups,
via `new SecurityGroup({groupName: "my-sg"}`); and finally, it's conceivable
that a provider might auto-generate the name (e.g., such as an AWS Lambda
whose name could simply be a hash of the source code contents).
This should overall produce better results with respect to moniker
collisions, ability to match resources, and the usability of the system.
2017-02-24 23:50:02 +01:00
|
|
|
if v, ok := cloneObjectValue(ctx, src[k].Obj()); ok {
|
Begin resource modeling and planning
This change introduces a new package, pkg/resource, that will form
the foundation for actually performing deployment plans and applications.
It contains the following key abstractions:
* resource.Provider is a wrapper around the CRUD operations exposed by
underlying resource plugins. It will eventually defer to resource.Plugin,
which itself defers -- over an RPC interface -- to the actual plugin, one
per package exposing resources. The provider will also understand how to
load, cache, and overall manage the lifetime of each plugin.
* resource.Resource is the actual resource object. This is created from
the overall evaluation object graph, but is simplified. It contains only
serializable properties, for example. Inter-resource references are
translated into serializable monikers as part of creating the resource.
* resource.Moniker is a serializable string that uniquely identifies
a resource in the Mu system. This is in contrast to resource IDs, which
are generated by resource providers and generally opaque to the Mu
system. See marapongo/mu#69 for more information about monikers and some
of their challenges (namely, designing a stable algorithm).
* resource.Snapshot is a "snapshot" taken from a graph of resources. This
is a transitive closure of state representing one possible configuration
of a given environment. This is what plans are created from. Eventually,
two snapshots will be diffable, in order to perform incremental updates.
One way of thinking about this is that a snapshot of the old world's state
is advanced, one step at a time, until it reaches a desired snapshot of
the new world's state.
* resource.Plan is a plan for carrying out desired CRUD operations on a target
environment. Each plan consists of zero-to-many Steps, each of which has
a CRUD operation type, a resource target, and a next step. This is an
enumerator because it is possible the plan will evolve -- and introduce new
steps -- as it is carried out (hence, the Next() method). At the moment, this
is linearized; eventually, we want to make this more "graph-like" so that we
can exploit available parallelism within the dependencies.
There are tons of TODOs remaining. However, the `mu plan` command is functioning
with these new changes -- including colorization FTW -- so I'm landing it now.
This is part of marapongo/mu#38 and marapongo/mu#41.
2017-02-17 21:31:48 +01:00
|
|
|
dest[PropertyKey(k)] = v
|
|
|
|
}
|
|
|
|
}
|
|
|
|
return dest
|
|
|
|
}
|
|
|
|
|
|
|
|
// cloneObjectValue creates a single property value out of a runtime object. It returns false if the property could not
|
|
|
|
// be stored in a property (e.g., it is a function or other unrecognized or unserializable runtime object).
|
Redo object monikers
This change overhauls the way we do object monikers. The old mechanism,
generating monikers using graph paths, was far too brittle and prone to
collisions. The new approach mixes some amount of "automatic scoping"
plus some "explicit naming." Although there is some explicitness, this
is arguably a good thing, as the monikers will be relatable back to the
source more readily by developers inspecting the graph and resource state.
Each moniker has four parts:
<Namespace>::<AllocModule>::<Type>::<Name>
wherein each element is the following:
<Namespace> The namespace being deployed into
<AllocModule> The module in which the object was allocated
<Type> The type of the resource
<Name> The assigned name of the resource
The <Namespace> is essentially the deployment target -- so "prod",
"stage", etc -- although it is more general purpose to allow for future
namespacing within a target (e.g., "prod/customer1", etc); for now
this is rudimentary, however, see marapongo/mu#94.
The <AllocModule> is the token for the code that contained the 'new'
that led to this object being created. In the future, we may wish to
extend this to also track the module under evaluation. (This is a nice
aspect of monikers; they can become arbitrarily complex, so long as
they are precise, and not prone to false positives/negatives.)
The <Name> warrants more discussion. The resource provider is consulted
via a new gRPC method, Name, that fetches the name. How the provider
does this is entirely up to it. For some resource types, the resource
may have properties that developers must set (e.g., `new Bucket("foo")`);
for other providers, perhaps the resource intrinsically has a property
that explicitly and uniquely qualifies the object (e.g., AWS SecurityGroups,
via `new SecurityGroup({groupName: "my-sg"}`); and finally, it's conceivable
that a provider might auto-generate the name (e.g., such as an AWS Lambda
whose name could simply be a hash of the source code contents).
This should overall produce better results with respect to moniker
collisions, ability to match resources, and the usability of the system.
2017-02-24 23:50:02 +01:00
|
|
|
func cloneObjectValue(ctx *Context, obj *rt.Object) (PropertyValue, bool) {
|
Begin resource modeling and planning
This change introduces a new package, pkg/resource, that will form
the foundation for actually performing deployment plans and applications.
It contains the following key abstractions:
* resource.Provider is a wrapper around the CRUD operations exposed by
underlying resource plugins. It will eventually defer to resource.Plugin,
which itself defers -- over an RPC interface -- to the actual plugin, one
per package exposing resources. The provider will also understand how to
load, cache, and overall manage the lifetime of each plugin.
* resource.Resource is the actual resource object. This is created from
the overall evaluation object graph, but is simplified. It contains only
serializable properties, for example. Inter-resource references are
translated into serializable monikers as part of creating the resource.
* resource.Moniker is a serializable string that uniquely identifies
a resource in the Mu system. This is in contrast to resource IDs, which
are generated by resource providers and generally opaque to the Mu
system. See marapongo/mu#69 for more information about monikers and some
of their challenges (namely, designing a stable algorithm).
* resource.Snapshot is a "snapshot" taken from a graph of resources. This
is a transitive closure of state representing one possible configuration
of a given environment. This is what plans are created from. Eventually,
two snapshots will be diffable, in order to perform incremental updates.
One way of thinking about this is that a snapshot of the old world's state
is advanced, one step at a time, until it reaches a desired snapshot of
the new world's state.
* resource.Plan is a plan for carrying out desired CRUD operations on a target
environment. Each plan consists of zero-to-many Steps, each of which has
a CRUD operation type, a resource target, and a next step. This is an
enumerator because it is possible the plan will evolve -- and introduce new
steps -- as it is carried out (hence, the Next() method). At the moment, this
is linearized; eventually, we want to make this more "graph-like" so that we
can exploit available parallelism within the dependencies.
There are tons of TODOs remaining. However, the `mu plan` command is functioning
with these new changes -- including colorization FTW -- so I'm landing it now.
This is part of marapongo/mu#38 and marapongo/mu#41.
2017-02-17 21:31:48 +01:00
|
|
|
t := obj.Type()
|
|
|
|
if IsResourceType(t) {
|
2017-02-18 19:22:04 +01:00
|
|
|
// For resources, simply look up the moniker from the resource map.
|
Redo object monikers
This change overhauls the way we do object monikers. The old mechanism,
generating monikers using graph paths, was far too brittle and prone to
collisions. The new approach mixes some amount of "automatic scoping"
plus some "explicit naming." Although there is some explicitness, this
is arguably a good thing, as the monikers will be relatable back to the
source more readily by developers inspecting the graph and resource state.
Each moniker has four parts:
<Namespace>::<AllocModule>::<Type>::<Name>
wherein each element is the following:
<Namespace> The namespace being deployed into
<AllocModule> The module in which the object was allocated
<Type> The type of the resource
<Name> The assigned name of the resource
The <Namespace> is essentially the deployment target -- so "prod",
"stage", etc -- although it is more general purpose to allow for future
namespacing within a target (e.g., "prod/customer1", etc); for now
this is rudimentary, however, see marapongo/mu#94.
The <AllocModule> is the token for the code that contained the 'new'
that led to this object being created. In the future, we may wish to
extend this to also track the module under evaluation. (This is a nice
aspect of monikers; they can become arbitrarily complex, so long as
they are precise, and not prone to false positives/negatives.)
The <Name> warrants more discussion. The resource provider is consulted
via a new gRPC method, Name, that fetches the name. How the provider
does this is entirely up to it. For some resource types, the resource
may have properties that developers must set (e.g., `new Bucket("foo")`);
for other providers, perhaps the resource intrinsically has a property
that explicitly and uniquely qualifies the object (e.g., AWS SecurityGroups,
via `new SecurityGroup({groupName: "my-sg"}`); and finally, it's conceivable
that a provider might auto-generate the name (e.g., such as an AWS Lambda
whose name could simply be a hash of the source code contents).
This should overall produce better results with respect to moniker
collisions, ability to match resources, and the usability of the system.
2017-02-24 23:50:02 +01:00
|
|
|
m, hasm := ctx.ObjMks[obj]
|
|
|
|
contract.Assertf(hasm, "Missing object reference; possible out of order dependency walk")
|
2017-02-18 19:22:04 +01:00
|
|
|
return NewPropertyResource(m), true
|
Begin resource modeling and planning
This change introduces a new package, pkg/resource, that will form
the foundation for actually performing deployment plans and applications.
It contains the following key abstractions:
* resource.Provider is a wrapper around the CRUD operations exposed by
underlying resource plugins. It will eventually defer to resource.Plugin,
which itself defers -- over an RPC interface -- to the actual plugin, one
per package exposing resources. The provider will also understand how to
load, cache, and overall manage the lifetime of each plugin.
* resource.Resource is the actual resource object. This is created from
the overall evaluation object graph, but is simplified. It contains only
serializable properties, for example. Inter-resource references are
translated into serializable monikers as part of creating the resource.
* resource.Moniker is a serializable string that uniquely identifies
a resource in the Mu system. This is in contrast to resource IDs, which
are generated by resource providers and generally opaque to the Mu
system. See marapongo/mu#69 for more information about monikers and some
of their challenges (namely, designing a stable algorithm).
* resource.Snapshot is a "snapshot" taken from a graph of resources. This
is a transitive closure of state representing one possible configuration
of a given environment. This is what plans are created from. Eventually,
two snapshots will be diffable, in order to perform incremental updates.
One way of thinking about this is that a snapshot of the old world's state
is advanced, one step at a time, until it reaches a desired snapshot of
the new world's state.
* resource.Plan is a plan for carrying out desired CRUD operations on a target
environment. Each plan consists of zero-to-many Steps, each of which has
a CRUD operation type, a resource target, and a next step. This is an
enumerator because it is possible the plan will evolve -- and introduce new
steps -- as it is carried out (hence, the Next() method). At the moment, this
is linearized; eventually, we want to make this more "graph-like" so that we
can exploit available parallelism within the dependencies.
There are tons of TODOs remaining. However, the `mu plan` command is functioning
with these new changes -- including colorization FTW -- so I'm landing it now.
This is part of marapongo/mu#38 and marapongo/mu#41.
2017-02-17 21:31:48 +01:00
|
|
|
}
|
|
|
|
|
|
|
|
switch t {
|
2017-02-20 22:27:31 +01:00
|
|
|
case types.Null:
|
|
|
|
return NewPropertyNull(), true
|
Begin resource modeling and planning
This change introduces a new package, pkg/resource, that will form
the foundation for actually performing deployment plans and applications.
It contains the following key abstractions:
* resource.Provider is a wrapper around the CRUD operations exposed by
underlying resource plugins. It will eventually defer to resource.Plugin,
which itself defers -- over an RPC interface -- to the actual plugin, one
per package exposing resources. The provider will also understand how to
load, cache, and overall manage the lifetime of each plugin.
* resource.Resource is the actual resource object. This is created from
the overall evaluation object graph, but is simplified. It contains only
serializable properties, for example. Inter-resource references are
translated into serializable monikers as part of creating the resource.
* resource.Moniker is a serializable string that uniquely identifies
a resource in the Mu system. This is in contrast to resource IDs, which
are generated by resource providers and generally opaque to the Mu
system. See marapongo/mu#69 for more information about monikers and some
of their challenges (namely, designing a stable algorithm).
* resource.Snapshot is a "snapshot" taken from a graph of resources. This
is a transitive closure of state representing one possible configuration
of a given environment. This is what plans are created from. Eventually,
two snapshots will be diffable, in order to perform incremental updates.
One way of thinking about this is that a snapshot of the old world's state
is advanced, one step at a time, until it reaches a desired snapshot of
the new world's state.
* resource.Plan is a plan for carrying out desired CRUD operations on a target
environment. Each plan consists of zero-to-many Steps, each of which has
a CRUD operation type, a resource target, and a next step. This is an
enumerator because it is possible the plan will evolve -- and introduce new
steps -- as it is carried out (hence, the Next() method). At the moment, this
is linearized; eventually, we want to make this more "graph-like" so that we
can exploit available parallelism within the dependencies.
There are tons of TODOs remaining. However, the `mu plan` command is functioning
with these new changes -- including colorization FTW -- so I'm landing it now.
This is part of marapongo/mu#38 and marapongo/mu#41.
2017-02-17 21:31:48 +01:00
|
|
|
case types.Bool:
|
|
|
|
return NewPropertyBool(obj.BoolValue()), true
|
|
|
|
case types.Number:
|
|
|
|
return NewPropertyNumber(obj.NumberValue()), true
|
|
|
|
case types.String:
|
|
|
|
return NewPropertyString(obj.StringValue()), true
|
|
|
|
case types.Object, types.Dynamic:
|
Redo object monikers
This change overhauls the way we do object monikers. The old mechanism,
generating monikers using graph paths, was far too brittle and prone to
collisions. The new approach mixes some amount of "automatic scoping"
plus some "explicit naming." Although there is some explicitness, this
is arguably a good thing, as the monikers will be relatable back to the
source more readily by developers inspecting the graph and resource state.
Each moniker has four parts:
<Namespace>::<AllocModule>::<Type>::<Name>
wherein each element is the following:
<Namespace> The namespace being deployed into
<AllocModule> The module in which the object was allocated
<Type> The type of the resource
<Name> The assigned name of the resource
The <Namespace> is essentially the deployment target -- so "prod",
"stage", etc -- although it is more general purpose to allow for future
namespacing within a target (e.g., "prod/customer1", etc); for now
this is rudimentary, however, see marapongo/mu#94.
The <AllocModule> is the token for the code that contained the 'new'
that led to this object being created. In the future, we may wish to
extend this to also track the module under evaluation. (This is a nice
aspect of monikers; they can become arbitrarily complex, so long as
they are precise, and not prone to false positives/negatives.)
The <Name> warrants more discussion. The resource provider is consulted
via a new gRPC method, Name, that fetches the name. How the provider
does this is entirely up to it. For some resource types, the resource
may have properties that developers must set (e.g., `new Bucket("foo")`);
for other providers, perhaps the resource intrinsically has a property
that explicitly and uniquely qualifies the object (e.g., AWS SecurityGroups,
via `new SecurityGroup({groupName: "my-sg"}`); and finally, it's conceivable
that a provider might auto-generate the name (e.g., such as an AWS Lambda
whose name could simply be a hash of the source code contents).
This should overall produce better results with respect to moniker
collisions, ability to match resources, and the usability of the system.
2017-02-24 23:50:02 +01:00
|
|
|
obj := cloneObject(ctx, obj) // an object literal, clone it
|
Begin resource modeling and planning
This change introduces a new package, pkg/resource, that will form
the foundation for actually performing deployment plans and applications.
It contains the following key abstractions:
* resource.Provider is a wrapper around the CRUD operations exposed by
underlying resource plugins. It will eventually defer to resource.Plugin,
which itself defers -- over an RPC interface -- to the actual plugin, one
per package exposing resources. The provider will also understand how to
load, cache, and overall manage the lifetime of each plugin.
* resource.Resource is the actual resource object. This is created from
the overall evaluation object graph, but is simplified. It contains only
serializable properties, for example. Inter-resource references are
translated into serializable monikers as part of creating the resource.
* resource.Moniker is a serializable string that uniquely identifies
a resource in the Mu system. This is in contrast to resource IDs, which
are generated by resource providers and generally opaque to the Mu
system. See marapongo/mu#69 for more information about monikers and some
of their challenges (namely, designing a stable algorithm).
* resource.Snapshot is a "snapshot" taken from a graph of resources. This
is a transitive closure of state representing one possible configuration
of a given environment. This is what plans are created from. Eventually,
two snapshots will be diffable, in order to perform incremental updates.
One way of thinking about this is that a snapshot of the old world's state
is advanced, one step at a time, until it reaches a desired snapshot of
the new world's state.
* resource.Plan is a plan for carrying out desired CRUD operations on a target
environment. Each plan consists of zero-to-many Steps, each of which has
a CRUD operation type, a resource target, and a next step. This is an
enumerator because it is possible the plan will evolve -- and introduce new
steps -- as it is carried out (hence, the Next() method). At the moment, this
is linearized; eventually, we want to make this more "graph-like" so that we
can exploit available parallelism within the dependencies.
There are tons of TODOs remaining. However, the `mu plan` command is functioning
with these new changes -- including colorization FTW -- so I'm landing it now.
This is part of marapongo/mu#38 and marapongo/mu#41.
2017-02-17 21:31:48 +01:00
|
|
|
return NewPropertyObject(obj), true
|
|
|
|
}
|
|
|
|
|
|
|
|
switch t.(type) {
|
|
|
|
case *symbols.ArrayType:
|
|
|
|
var result []PropertyValue
|
|
|
|
for _, e := range *obj.ArrayValue() {
|
Redo object monikers
This change overhauls the way we do object monikers. The old mechanism,
generating monikers using graph paths, was far too brittle and prone to
collisions. The new approach mixes some amount of "automatic scoping"
plus some "explicit naming." Although there is some explicitness, this
is arguably a good thing, as the monikers will be relatable back to the
source more readily by developers inspecting the graph and resource state.
Each moniker has four parts:
<Namespace>::<AllocModule>::<Type>::<Name>
wherein each element is the following:
<Namespace> The namespace being deployed into
<AllocModule> The module in which the object was allocated
<Type> The type of the resource
<Name> The assigned name of the resource
The <Namespace> is essentially the deployment target -- so "prod",
"stage", etc -- although it is more general purpose to allow for future
namespacing within a target (e.g., "prod/customer1", etc); for now
this is rudimentary, however, see marapongo/mu#94.
The <AllocModule> is the token for the code that contained the 'new'
that led to this object being created. In the future, we may wish to
extend this to also track the module under evaluation. (This is a nice
aspect of monikers; they can become arbitrarily complex, so long as
they are precise, and not prone to false positives/negatives.)
The <Name> warrants more discussion. The resource provider is consulted
via a new gRPC method, Name, that fetches the name. How the provider
does this is entirely up to it. For some resource types, the resource
may have properties that developers must set (e.g., `new Bucket("foo")`);
for other providers, perhaps the resource intrinsically has a property
that explicitly and uniquely qualifies the object (e.g., AWS SecurityGroups,
via `new SecurityGroup({groupName: "my-sg"}`); and finally, it's conceivable
that a provider might auto-generate the name (e.g., such as an AWS Lambda
whose name could simply be a hash of the source code contents).
This should overall produce better results with respect to moniker
collisions, ability to match resources, and the usability of the system.
2017-02-24 23:50:02 +01:00
|
|
|
if v, ok := cloneObjectValue(ctx, e.Obj()); ok {
|
Begin resource modeling and planning
This change introduces a new package, pkg/resource, that will form
the foundation for actually performing deployment plans and applications.
It contains the following key abstractions:
* resource.Provider is a wrapper around the CRUD operations exposed by
underlying resource plugins. It will eventually defer to resource.Plugin,
which itself defers -- over an RPC interface -- to the actual plugin, one
per package exposing resources. The provider will also understand how to
load, cache, and overall manage the lifetime of each plugin.
* resource.Resource is the actual resource object. This is created from
the overall evaluation object graph, but is simplified. It contains only
serializable properties, for example. Inter-resource references are
translated into serializable monikers as part of creating the resource.
* resource.Moniker is a serializable string that uniquely identifies
a resource in the Mu system. This is in contrast to resource IDs, which
are generated by resource providers and generally opaque to the Mu
system. See marapongo/mu#69 for more information about monikers and some
of their challenges (namely, designing a stable algorithm).
* resource.Snapshot is a "snapshot" taken from a graph of resources. This
is a transitive closure of state representing one possible configuration
of a given environment. This is what plans are created from. Eventually,
two snapshots will be diffable, in order to perform incremental updates.
One way of thinking about this is that a snapshot of the old world's state
is advanced, one step at a time, until it reaches a desired snapshot of
the new world's state.
* resource.Plan is a plan for carrying out desired CRUD operations on a target
environment. Each plan consists of zero-to-many Steps, each of which has
a CRUD operation type, a resource target, and a next step. This is an
enumerator because it is possible the plan will evolve -- and introduce new
steps -- as it is carried out (hence, the Next() method). At the moment, this
is linearized; eventually, we want to make this more "graph-like" so that we
can exploit available parallelism within the dependencies.
There are tons of TODOs remaining. However, the `mu plan` command is functioning
with these new changes -- including colorization FTW -- so I'm landing it now.
This is part of marapongo/mu#38 and marapongo/mu#41.
2017-02-17 21:31:48 +01:00
|
|
|
result = append(result, v)
|
|
|
|
}
|
|
|
|
}
|
|
|
|
return NewPropertyArray(result), true
|
|
|
|
case *symbols.Class:
|
Redo object monikers
This change overhauls the way we do object monikers. The old mechanism,
generating monikers using graph paths, was far too brittle and prone to
collisions. The new approach mixes some amount of "automatic scoping"
plus some "explicit naming." Although there is some explicitness, this
is arguably a good thing, as the monikers will be relatable back to the
source more readily by developers inspecting the graph and resource state.
Each moniker has four parts:
<Namespace>::<AllocModule>::<Type>::<Name>
wherein each element is the following:
<Namespace> The namespace being deployed into
<AllocModule> The module in which the object was allocated
<Type> The type of the resource
<Name> The assigned name of the resource
The <Namespace> is essentially the deployment target -- so "prod",
"stage", etc -- although it is more general purpose to allow for future
namespacing within a target (e.g., "prod/customer1", etc); for now
this is rudimentary, however, see marapongo/mu#94.
The <AllocModule> is the token for the code that contained the 'new'
that led to this object being created. In the future, we may wish to
extend this to also track the module under evaluation. (This is a nice
aspect of monikers; they can become arbitrarily complex, so long as
they are precise, and not prone to false positives/negatives.)
The <Name> warrants more discussion. The resource provider is consulted
via a new gRPC method, Name, that fetches the name. How the provider
does this is entirely up to it. For some resource types, the resource
may have properties that developers must set (e.g., `new Bucket("foo")`);
for other providers, perhaps the resource intrinsically has a property
that explicitly and uniquely qualifies the object (e.g., AWS SecurityGroups,
via `new SecurityGroup({groupName: "my-sg"}`); and finally, it's conceivable
that a provider might auto-generate the name (e.g., such as an AWS Lambda
whose name could simply be a hash of the source code contents).
This should overall produce better results with respect to moniker
collisions, ability to match resources, and the usability of the system.
2017-02-24 23:50:02 +01:00
|
|
|
obj := cloneObject(ctx, obj) // a class, just deep clone it
|
Begin resource modeling and planning
This change introduces a new package, pkg/resource, that will form
the foundation for actually performing deployment plans and applications.
It contains the following key abstractions:
* resource.Provider is a wrapper around the CRUD operations exposed by
underlying resource plugins. It will eventually defer to resource.Plugin,
which itself defers -- over an RPC interface -- to the actual plugin, one
per package exposing resources. The provider will also understand how to
load, cache, and overall manage the lifetime of each plugin.
* resource.Resource is the actual resource object. This is created from
the overall evaluation object graph, but is simplified. It contains only
serializable properties, for example. Inter-resource references are
translated into serializable monikers as part of creating the resource.
* resource.Moniker is a serializable string that uniquely identifies
a resource in the Mu system. This is in contrast to resource IDs, which
are generated by resource providers and generally opaque to the Mu
system. See marapongo/mu#69 for more information about monikers and some
of their challenges (namely, designing a stable algorithm).
* resource.Snapshot is a "snapshot" taken from a graph of resources. This
is a transitive closure of state representing one possible configuration
of a given environment. This is what plans are created from. Eventually,
two snapshots will be diffable, in order to perform incremental updates.
One way of thinking about this is that a snapshot of the old world's state
is advanced, one step at a time, until it reaches a desired snapshot of
the new world's state.
* resource.Plan is a plan for carrying out desired CRUD operations on a target
environment. Each plan consists of zero-to-many Steps, each of which has
a CRUD operation type, a resource target, and a next step. This is an
enumerator because it is possible the plan will evolve -- and introduce new
steps -- as it is carried out (hence, the Next() method). At the moment, this
is linearized; eventually, we want to make this more "graph-like" so that we
can exploit available parallelism within the dependencies.
There are tons of TODOs remaining. However, the `mu plan` command is functioning
with these new changes -- including colorization FTW -- so I'm landing it now.
This is part of marapongo/mu#38 and marapongo/mu#41.
2017-02-17 21:31:48 +01:00
|
|
|
return NewPropertyObject(obj), true
|
|
|
|
}
|
|
|
|
|
|
|
|
// TODO: handle symbols.MapType.
|
|
|
|
// TODO: it's unclear if we should do something more drastic here. There will always be unrecognized property
|
|
|
|
// kinds because objects contain things like constructors, methods, etc. But we may want to ratchet this a bit.
|
|
|
|
glog.V(5).Infof("Ignoring object value of type '%v': unrecognized kind %v", t, reflect.TypeOf(t))
|
|
|
|
return PropertyValue{}, false
|
|
|
|
}
|