This change adds support for serializing snapshots in MuGL, per the
design document in docs/design/mugl.md. At the moment, it is only
exposed from the `mu plan` command, which allows you to specify an
output location using `mu plan --output=file.json` (or `-o=file.json`
for short). This serializes the snapshot with monikers, resources,
and so on. Deserialization is not yet supported; that comes next.
* Specify MinCount/MaxCount when creating an EC2 instance. These
are required properties on the RunInstances API.
* Only attempt to unmarshal egressArray/ingressArray when non-nil.
* Remember the context object on the instanceProvider.
* Move the moniker and object maps into the shared context object.
* Marshal object monikers as the resource IDs to which they refer,
since monikers are useless on "the other side" of the RPC boundary.
This ensures that, for example, the AWS provider gets IDs it can use.
* Add some paranoia assertions.
This commit includes a basic AWS resource provider. Mostly it is just
scaffolding, however, it also includes prototype implementations for EC2
instance and security group resource creation operations.
This change adds basic support for discovering, loading, binding to,
and invoking RPC methods on, resource provider plugins.
In a nutshell, we add a new context object that will share cached
state such as loaded plugins and connections to them. It will be
a policy decision in server scenarios how much state to share and
between whom. This context also controls per-resource context
allocation, which in the future will allow us to perform structured
cancellation and teardown amongst entire groups of requests.
Plugins are loaded based on their name, and can be found in one of
two ways: either simply by having them on your path (with a name of
"mu-ressrv-<pkg>", where "<pkg>" is the resource package name with
any "/"s replaced with "_"s); or by placing them in the standard
library installation location, which need not be on the path for this
to work (since we know precisely where to look).
If we find a protocol, we will load it as a child process.
The protocol for plugins is that they will choose a port on their
own -- to eliminate races that'd be involved should Mu attempt to
pre-pick one for them -- and then write that out as the first line
to STDOUT (terminated by a "\n"). This is the only STDERR/STDOUT
that Mu cares about; from there, the plugin is free to write all it
pleases (e.g., for logging, debugging purposes, etc).
Afterwards, we then bind our gRPC connection to that port, and create
a typed resource provider client. The CRUD operations that get driven
by plan application are then simple wrappers atop the underlying gRPC
calls. For now, we interpret all errors as catastrophic; in the near
future, we will probably want to introduce a "structured error"
mechanism in the gRPC interface for "transactional errors"; that is,
errors for which the server was able to recover to a safe checkpoint,
which can be interpreted as ResourceOK rather than ResourceUnknown.
This moves us one step closer from planning to application (marapongo/mu#21).
Namely, we now drive the right resource provider operations in response to
the plan's steps. Those providers, however, are still empty shells.
This change adds a flag to `plan` so that we can create deletion plans:
$ mu plan --delete
This will have an equivalent in the `apply` command, achieving the ability
to delete entire sets of resources altogether (see marapongo/mu#58).
This change introduces object monikers. These are unique, serializable
names that refer to resources created during the execution of a MuIL
program. They are pretty darned ugly at the moment, but at least they
serve their desired purpose. I suspect we will eventually want to use
more information (like edge "labels" (variable names and what not)),
but this should suffice for the time being. The names right now are
particularly sensitive to simple refactorings.
This is enough for marapongo/mu#69 during the current sprint, although
I will keep the work item (in a later sprint) to think more about how
to make these more stable. I'd prefer to do that with a bit of
experience under our belts first.
This change introduces a new package, pkg/resource, that will form
the foundation for actually performing deployment plans and applications.
It contains the following key abstractions:
* resource.Provider is a wrapper around the CRUD operations exposed by
underlying resource plugins. It will eventually defer to resource.Plugin,
which itself defers -- over an RPC interface -- to the actual plugin, one
per package exposing resources. The provider will also understand how to
load, cache, and overall manage the lifetime of each plugin.
* resource.Resource is the actual resource object. This is created from
the overall evaluation object graph, but is simplified. It contains only
serializable properties, for example. Inter-resource references are
translated into serializable monikers as part of creating the resource.
* resource.Moniker is a serializable string that uniquely identifies
a resource in the Mu system. This is in contrast to resource IDs, which
are generated by resource providers and generally opaque to the Mu
system. See marapongo/mu#69 for more information about monikers and some
of their challenges (namely, designing a stable algorithm).
* resource.Snapshot is a "snapshot" taken from a graph of resources. This
is a transitive closure of state representing one possible configuration
of a given environment. This is what plans are created from. Eventually,
two snapshots will be diffable, in order to perform incremental updates.
One way of thinking about this is that a snapshot of the old world's state
is advanced, one step at a time, until it reaches a desired snapshot of
the new world's state.
* resource.Plan is a plan for carrying out desired CRUD operations on a target
environment. Each plan consists of zero-to-many Steps, each of which has
a CRUD operation type, a resource target, and a next step. This is an
enumerator because it is possible the plan will evolve -- and introduce new
steps -- as it is carried out (hence, the Next() method). At the moment, this
is linearized; eventually, we want to make this more "graph-like" so that we
can exploit available parallelism within the dependencies.
There are tons of TODOs remaining. However, the `mu plan` command is functioning
with these new changes -- including colorization FTW -- so I'm landing it now.
This is part of marapongo/mu#38 and marapongo/mu#41.
This change moves the RPC definitions to the pkg/murpc package,
underneath the proto folder. This is to ensure that the resulting
Protobufs get a good package name "murpc" that is unique within the
overall toolchain. It also includes a `generate.sh` script that
can be used to manually regenerate client/server code. Finally,
we are actually checking in the generated files underneath pkg/murpc.
This change deletes a bunch of old legacy code. I've tagged the
prior changelist as "0.1" in case we need to go back and recover
some of the code (as I expect some of the constraint type logic
and AWS-specific code to come in handy down the road).
🔥🔥🔥
In the old system, the core runtime/toolset understood that we are targeting
specific cloud providers at a very deep level. In fact, the whole code-generation
phase of the compiler was based on it.
In the new system, this difference is less of a "special" concern, and more of
a general one of mapping MuIL objects to resource providers, and letting *them*
gather up any configuration they need in a more general purpose way.
Therefore, most of this stuff can go. I've merged in a small amount of it to
the mu/x MuPackage, since that has to switch on cloud IaaS and CaaS providers in
order to decide what kind of resources to provision. For example, it has a
mu.x.Cluster stack type that itself provisions a lot of the barebone essential
resources, like a virtual private cloud and its associated networking components.
I suspect *some* knowledge of this will surface again as we implement more
runtime presence (discovery, etc). But for the time being, it's a distraction
getting the core model running. I've retained some of the old AWS code in the
new pkg/resource/providers/aws package, in case I want to reuse some of it when
implementing our first AWS resource providers. (Although we won't be using
CloudFormation, some of the name generation code might be useful.) So, the
ships aren't completely burned to the ground, but they are certainly on 🔥.
