This change articulates Mu's packaging format, MuPack, along with its
corresponding intermediate language and type system, MuIL. This is very
much a work in progress.
This documents our latest thinking on Mu languages. At a high level,
there are three classes of language at play:
1. Mu Metadata Languages (MuML): these are the high-level language
subsets that a programmer uses to specify Mu modules, etc. Examples
include MuJS, MuPy, MuRu, and MyGo, each representing a deterministic
subset of JavaScript, Python, Ruby, and Go, respectively.
2. Mu Intermediate Language (MuIL): this is the intermediate form that
all of the above compile down to. It is capable of representing
computations like functions, conditionals, and basic expressions like
string concatenation, etc. This is fully statically analyzable and
can be used to create deterministic plans and topology graphs.
3. Mu Graph Language (MuGL): this is the "final" form in which any Mu
service topology is represented. It never contains computations and
is merely a metadata description of services-as-nodes, dependencies-as-
edges, and all known properties. In the planning form, it may contain
"holes" because output properties aren't known until execution has
occurred, while in the actual applied form, those holes have been
plugged. MuGLs can be diffed, and a MuGL can be generated from an
existing live environment (for bootstrapping and/or drift analysis).
There are several TODOs in here, but this is braindump of where we're at.
This checkin includes a new design doc for the Mull metadata language.
This is very much a work-in-progress.
This will eventually supersede the language.md document.
In an older version of this doc, `if` was called `when`, and `for`
was called `each`, to encourage a more declarative feel. I renamed
them back to their familiar forms. I missed a few spots. But I also
thought at least documenting this heritage would be useful, hence
this descriptive checkin comment.
This change includes a miniature spec for what we'd want out of a little
markup language that extends YAML/JSON with typing and minimal templating.
We've begun to reach the limits of what Go's templates give us; the usability
is quite poor: the order of template expansion is "confused" (as it must
happen before verification of stack properties); it is dumb textual copy-and-
paste, and thus knows nothing about the lexical and semantic rules; evaluation
of expressions that should produce actual objects inserted into the metadata
stream as-is must actually be serialized into text (problematic for the above
reasons); and, finally, as a result of all of this, failure modes are terrible.
But worse than this, we simply can't do what we need in many places. For
instance, mapping a stack's properties onto the services that it creates works
in simple cases -- like strings, booleans, and ints -- but quickly breaks down
when referencing complex objects (for the same above reasons). This is why
we've needed to special case property mapping in the aws/x/cf provider, but
clearly this won't generalize to all the compositional situations that arise.
It's worth nothing Hashicorp's HCL/HIL is closest to what we want. (The
language used for Terraform.) It isn't exactly what we want, however, for two
reasons. First, it lacks conditionals and iteration. This is likely to appear
at some point (see https://github.com/hashicorp/terraform/issues/1604), and
indeed in this past week alone, a new C-like conditional operator (which I
actually don't love) got added to HIL:
5fe4b10b43.
Second, and perhaps more importantly, its approach is to create a new language.
The design I list here is a natural extension that adds typechecking and
minimal templating to the existing YAML/JSON formats. As a stand-alone
project, this whould have a much broader appeal. And whether or not we use it
for Mu depends on whether we really want an entirely new markup language or not.
To cut to the chase, I'm shelving this for now. I'm going to keep hacking my
way through the current Go templates plus special-casing for now. My eye is
on the initial end-to-end prototype. But, no doubt, we'll need to revisit this
immediately afterwards, make a decision, and make it happen.
As part of marapongo/mu#9, we want to enable extensible schema types
for stronger typechecking at compile-time. JSON Schema seems like a
decent starting place (http://json-schema.org/), although it's not yet
clear whether we want to use its module/naming schema or our own. I
suspect we want to use our own so that stack schemas can be managed
using the same discipline as stack management generally.
This change eliminates the special type mu/extension in favor of extensible
intrinsic types. This subsumes the previous functionality while also fixing
a number of warts with the old model.
In particular, the old mu/extension approach deferred property binding until
very late in the compiler. In fact, too late. The backend provider for an
extension simply received an untyped bag of stuff, which it then had to
deal with. Unfortunately, some operations in the binder are inaccessible
at this point because doing so would cause a cycle. Furthermore, some
pertinent information is gone at this point, like the scopes and symtables.
The canonical example where we need this is binding services names to the
services themselves; e.g., the AWS CloudFormation "DependsOn" property should
resolve to the actual service names, not the string values. In the limit,
this requires full binding information.
There were a few solutions I considered, including ones that've required
less code motion, however this one feels the most elegant.
Now we permit types to be marked as "intrinsic." Binding to these names
is done exactly as ordinary name binding, unlike the special mu/extension
provider name. In fact, just about everything except code-generation for
these types is the same as ordinary types. This is perfect for the use case
at hand, which is binding properties.
After this change, for example, "DependsOn" is expanded to real service
names precisely as we need.
As part of this change, I added support for three new basic schema types:
* ast.StringList ("string[]"): a list of strings.
* ast.StringMap ("map[string]any"): a map of strings to anys.
* ast.ServiceList ("service[]"): a list of service references.
Obviously we need to revisit this and add a more complete set. This work
is already tracked by marapongo/mu#9.
At the end of the day, it's likely I will replace all hard-coded predefined
types with intrinsic types, for similar reasons to the above.
This change makes workspace file naming a little more consistent with respect
to Mufile naming. Instead of having a .mu/ directory, under which a workspace.yaml
and/or a stacks directory might exist, we now have a Muspace.yaml (or .json) file,
and a .Mudeps/ directory. This has nicer symmetric with respect to Mu.yaml files.
This change introduces the notion of "perturbing" properties. Changing
one of these impacts the live service, possibly leading to downtime. As
such, we will likely encourage blue/green deployments of them just to be
safe. Note that this is really just a placeholder so I can keep track of
metadata as we go, since AWS CF has a similar notion to this.
I'm not in love with the name. I considered `interrupts`, however,
I must admit I liked that `readonly` and `perturbs` are symmetric in
the number of characters (meaning stuff lines up nicely...)
This change adds the notion of readonly properties to stacks. Although these
*can* be "changed", doing so implies recreation of the resources all over again.
As a result, all dependents must be recreated, in a cascading manner.
For now, we can simply auto-map the Mu properties to CF properties,
eliminating the need to manually map them in the templates. Eventually
we'll want more sophistication here to control various aspects of the CF
templates, but this eliminates a lot of tedious manual work in the meantime.
I've gone backwards and forwards on the design for dependency version
management. However, I think what's written in this commit represents
a pretty sane "sweet spot" between all available options.
In a nutshell, anytime reference to a stack type in your Mufile is a
full-blown StackRef; in other words, it has a protocol (e.g., "https://"),
a base URL (e.g., "hub.mu.com/"), a name (e.g., "aws/s3/bucket"), and a
version ("@^1.0.6"). Each reference carries all of these components.
For convenience, you may omit the components. In that case, Mu chooses
reasonable defaults:
* "https://" as the default protocol (or "git://"; this is TBD).
* "hub.mu.com/" as the default base URL.
* "@latest" as the default version number.
Note that a version can be "latest" to mean "tip", a specific SHA hash
to pin to a precise version, or a semantic version number/range.
I had originally shied away from specifying versions inline as you reference
stacks in your Mufile, and was leaning towards an explicit dependencies
section, however I was swayed for two reasons:
1. It's very common to only consume a given stack once in a file. Needing
to specify it in two places each time is verbose and violates DRY.
2. We have decided that each Mufile stands on its own and forms the unit
of distribution. I had previously thought we might move dependencies
out of Mufiles into a "package manager" specification. Lacking that,
there is even less reason to call them out in a dedicated section.
Now, managing all these embedded version numbers across multiple stacks in
a single workspace would have been annoying. (Many edits for a single
version bump.) Instead, I've added provisions for storing this in your
workspace.yaml file. The way it works is if any StackRef lacks a version
number, before defaulting to "@latest" we check the workspace.yaml file and,
if a default is found in there, we will use it. For example:
dependencies:
aws/s3/bucket: ^1.0.6
The provision for pinning an entire namespace is also preserved. E.g.:
dependencies:
aws/...: ^1.0.6
This changes the probing logic for dependency resolution. The old logic was
inconsistent between the various roots. The new approach simply prefers locations
with a base URL component -- since they are more specific -- but will allow for
locations missing a base URL component. This is convenient for developers managing
a workspace where needing to specify the base URL in the path is annoying and
slightly too "opinionated" for my taste (especially for migrating existing services).
We will use the $MUROOT envvar to determine where Mu has been installed,
which will by default be /usr/local/mu. From there, we can access the
predefined library of stacks (underneath $MUROOT/bin/stacks).
This change implements the aws/cf extension provider, so that AWS resources
may be described and encapsulated inside of other stacks. Each aws/cf instantiation
requires just two fields -- type and properties -- corresponding to the equivalent
AWS resource object. The result is simply plugged in as an AWS resource, after
Mu templates have been expanded, permitting stack properties, etc. to be used.
The more I live with the current system, the more I prefer "properties" to
"parameters" for stacks and services. Although it is true that these things
are essentially construction-time arguments, they manifest more like properties
in the way they are used; in fact, if you think of the world in terms of primary
constructors, the distinction is pretty subtle anyway.
For example, when creating a new service, we say the following:
services:
private:
some/service:
a: 0
b: true
c: foo
This looks like a, b, and c are properties of the type some/service. If, on
the other hand, we kept calling these parameters, then you'd arguably prefer to
see the following:
services:
private:
some/service:
arguments:
a: 0
b: true
c: foo
This is a more imperative than declarative view of the world, which I dislike
(especially because it is more verbose).
Time will tell whether this is the right decision or not ...
This change introduces the notion of "Stack subclassing" in two ways:
1. A Stack may declare that it subclasses another one using the base property:
name: mystack
base: other/stack
.. as before ..
2. A Stack may declare that it is abstract; in other words, that it is meant
solely for subclassing, and cannot be compiled and deployed independently:
name: mystack
abstract: true
.. as before ..
Note that non-abstract Stacks are required to declare at least one Service,
whether that is public, private, or both.
This change separates the metadata specification from the targets doc,
since they are both meant to be consumable independent of one another
and will evolve at their own paces.
This sketches out some more details in the Metadata format, in addition to
starting to articulate what the AWS IaaS provider looks like. Sadly, more
questions than answers, but it's good to have the placeholders...
Simple developer scenarios can be done solely in terms of Stacks. However, to
support more sophisticated cases -- where multiplexing many Stacks onto a shared
set of resources is preferred (for cost savings, provisioning time savings,
control, etc) -- we need to introduce the concept of a Cluster. Each Cluster has
a fixed combination of IaaS/CaaS provider, decided at provisioning time.
This just contains primarily scaffolding, however lays out a general direction
and table of contents for the metadata specification document (marapongo/mu#1).
This doc will describe the overall architecture for the Mu tools, platform,
and related artifacts. It will be a companion to the more detailed drill-down
into the Mufile format and its specific target transformations (marapongo/mu#1).
There are tons of todos, etc., however this is at least a start, with some
amount of conceptual overview plus scaffolding and structure to fill out.
This snapshots thinking as of maybe two weeks ago. I'm preparing to
do a big overhaul of all of the write-ups, and specifically start on the
Mu.yaml file format spec, and so I want to archive everything that's in flight.
Instead of leading in with the Service-based voting application, and then
simplifying it to a serverless Function-based one, do it in the reverse order.
This allows us to "grow" the sample, motivating each change as we go.
