The stacks.md document used to describe the metadata format. Now that we've
moved away from the old model of YAML + Go templates, and created the MuPack/MuIL
format, this document needed to be overhauled.
It's pretty bare bones now, however it will eventually evolve into a document
describing how the Mu abstractions map down onto MuPack/MuIL concepts. For example,
it describes how subclassing mu.Stack creates a "stack", even though at the MuPack/MuIL
level it's just any old subclass.
Some say the world will end in fire,
Some say in ice.
From what I've tasted of desire
I hold with those who favor fire.
But if it had to perish twice,
I think I know enough of hate
To say that for destruction ice
Is also great
And would suffice.
This change adopts some standard aspects of classes, like statics, abstract,
and virtuals (sealing). It also recognizes records and interfaces as explicit
annotations on classes (both of which can be called "conjurable"); they were
already being treated specially with respect to structural/duck conversions,
and I always hate inferring defining properties of types like this, since simple
edits can completely change the type's semantics and break consumers of it.
This change articulates Mu's packaging format, MuPack, along with its
corresponding intermediate language and type system, MuIL. This is very
much a work in progress.
This documents our latest thinking on Mu languages. At a high level,
there are three classes of language at play:
1. Mu Metadata Languages (MuML): these are the high-level language
subsets that a programmer uses to specify Mu modules, etc. Examples
include MuJS, MuPy, MuRu, and MyGo, each representing a deterministic
subset of JavaScript, Python, Ruby, and Go, respectively.
2. Mu Intermediate Language (MuIL): this is the intermediate form that
all of the above compile down to. It is capable of representing
computations like functions, conditionals, and basic expressions like
string concatenation, etc. This is fully statically analyzable and
can be used to create deterministic plans and topology graphs.
3. Mu Graph Language (MuGL): this is the "final" form in which any Mu
service topology is represented. It never contains computations and
is merely a metadata description of services-as-nodes, dependencies-as-
edges, and all known properties. In the planning form, it may contain
"holes" because output properties aren't known until execution has
occurred, while in the actual applied form, those holes have been
plugged. MuGLs can be diffed, and a MuGL can be generated from an
existing live environment (for bootstrapping and/or drift analysis).
There are several TODOs in here, but this is braindump of where we're at.
This checkin includes a new design doc for the Mull metadata language.
This is very much a work-in-progress.
This will eventually supersede the language.md document.
In an older version of this doc, `if` was called `when`, and `for`
was called `each`, to encourage a more declarative feel. I renamed
them back to their familiar forms. I missed a few spots. But I also
thought at least documenting this heritage would be useful, hence
this descriptive checkin comment.
This change includes a miniature spec for what we'd want out of a little
markup language that extends YAML/JSON with typing and minimal templating.
We've begun to reach the limits of what Go's templates give us; the usability
is quite poor: the order of template expansion is "confused" (as it must
happen before verification of stack properties); it is dumb textual copy-and-
paste, and thus knows nothing about the lexical and semantic rules; evaluation
of expressions that should produce actual objects inserted into the metadata
stream as-is must actually be serialized into text (problematic for the above
reasons); and, finally, as a result of all of this, failure modes are terrible.
But worse than this, we simply can't do what we need in many places. For
instance, mapping a stack's properties onto the services that it creates works
in simple cases -- like strings, booleans, and ints -- but quickly breaks down
when referencing complex objects (for the same above reasons). This is why
we've needed to special case property mapping in the aws/x/cf provider, but
clearly this won't generalize to all the compositional situations that arise.
It's worth nothing Hashicorp's HCL/HIL is closest to what we want. (The
language used for Terraform.) It isn't exactly what we want, however, for two
reasons. First, it lacks conditionals and iteration. This is likely to appear
at some point (see https://github.com/hashicorp/terraform/issues/1604), and
indeed in this past week alone, a new C-like conditional operator (which I
actually don't love) got added to HIL:
5fe4b10b43.
Second, and perhaps more importantly, its approach is to create a new language.
The design I list here is a natural extension that adds typechecking and
minimal templating to the existing YAML/JSON formats. As a stand-alone
project, this whould have a much broader appeal. And whether or not we use it
for Mu depends on whether we really want an entirely new markup language or not.
To cut to the chase, I'm shelving this for now. I'm going to keep hacking my
way through the current Go templates plus special-casing for now. My eye is
on the initial end-to-end prototype. But, no doubt, we'll need to revisit this
immediately afterwards, make a decision, and make it happen.
As part of marapongo/mu#9, we want to enable extensible schema types
for stronger typechecking at compile-time. JSON Schema seems like a
decent starting place (http://json-schema.org/), although it's not yet
clear whether we want to use its module/naming schema or our own. I
suspect we want to use our own so that stack schemas can be managed
using the same discipline as stack management generally.
This change eliminates the special type mu/extension in favor of extensible
intrinsic types. This subsumes the previous functionality while also fixing
a number of warts with the old model.
In particular, the old mu/extension approach deferred property binding until
very late in the compiler. In fact, too late. The backend provider for an
extension simply received an untyped bag of stuff, which it then had to
deal with. Unfortunately, some operations in the binder are inaccessible
at this point because doing so would cause a cycle. Furthermore, some
pertinent information is gone at this point, like the scopes and symtables.
The canonical example where we need this is binding services names to the
services themselves; e.g., the AWS CloudFormation "DependsOn" property should
resolve to the actual service names, not the string values. In the limit,
this requires full binding information.
There were a few solutions I considered, including ones that've required
less code motion, however this one feels the most elegant.
Now we permit types to be marked as "intrinsic." Binding to these names
is done exactly as ordinary name binding, unlike the special mu/extension
provider name. In fact, just about everything except code-generation for
these types is the same as ordinary types. This is perfect for the use case
at hand, which is binding properties.
After this change, for example, "DependsOn" is expanded to real service
names precisely as we need.
As part of this change, I added support for three new basic schema types:
* ast.StringList ("string[]"): a list of strings.
* ast.StringMap ("map[string]any"): a map of strings to anys.
* ast.ServiceList ("service[]"): a list of service references.
Obviously we need to revisit this and add a more complete set. This work
is already tracked by marapongo/mu#9.
At the end of the day, it's likely I will replace all hard-coded predefined
types with intrinsic types, for similar reasons to the above.
This change makes workspace file naming a little more consistent with respect
to Mufile naming. Instead of having a .mu/ directory, under which a workspace.yaml
and/or a stacks directory might exist, we now have a Muspace.yaml (or .json) file,
and a .Mudeps/ directory. This has nicer symmetric with respect to Mu.yaml files.
This change introduces the notion of "perturbing" properties. Changing
one of these impacts the live service, possibly leading to downtime. As
such, we will likely encourage blue/green deployments of them just to be
safe. Note that this is really just a placeholder so I can keep track of
metadata as we go, since AWS CF has a similar notion to this.
I'm not in love with the name. I considered `interrupts`, however,
I must admit I liked that `readonly` and `perturbs` are symmetric in
the number of characters (meaning stuff lines up nicely...)
