This change articulates Mu's packaging format, MuPack, along with its
corresponding intermediate language and type system, MuIL. This is very
much a work in progress.
This documents our latest thinking on Mu languages. At a high level,
there are three classes of language at play:
1. Mu Metadata Languages (MuML): these are the high-level language
subsets that a programmer uses to specify Mu modules, etc. Examples
include MuJS, MuPy, MuRu, and MyGo, each representing a deterministic
subset of JavaScript, Python, Ruby, and Go, respectively.
2. Mu Intermediate Language (MuIL): this is the intermediate form that
all of the above compile down to. It is capable of representing
computations like functions, conditionals, and basic expressions like
string concatenation, etc. This is fully statically analyzable and
can be used to create deterministic plans and topology graphs.
3. Mu Graph Language (MuGL): this is the "final" form in which any Mu
service topology is represented. It never contains computations and
is merely a metadata description of services-as-nodes, dependencies-as-
edges, and all known properties. In the planning form, it may contain
"holes" because output properties aren't known until execution has
occurred, while in the actual applied form, those holes have been
plugged. MuGLs can be diffed, and a MuGL can be generated from an
existing live environment (for bootstrapping and/or drift analysis).
There are several TODOs in here, but this is braindump of where we're at.
This splits the overall example rack service into many sub-services.
This leads to a much cleaner factoring of the code. Note that there are
some missing properties -- it's hard to eyeball this without a real compiler.
But the essence of the example is pretty spot on.
I personally prefer this syntax. It is more "declarative" and, particularly
because we support true assignments, I feel like it's more representative of
what's going on. It's easier to scan for real imperative assignments. This
approach also eliminates a single horizontal character per assignment.
In this change, ":= new" declares a new resource.
In addition, we change arrays to be Go-like, in that zero-initialized
ones can be appended to safely, such that declaring a new instance is
done merely by stating "var arr: T[]".
This change encapsulates what's required for an AWS cluster in new service,
mu/clouds/aws/Cluster, so that mu/Cluster can remain fairly simplistic (it
just switches on the names and dispatches to the appropriate sub-modules).
This checkin includes a new design doc for the Mull metadata language.
This is very much a work-in-progress.
This will eventually supersede the language.md document.
There's no need for quotes here and having them makes it feel somehow
"dynamically typed" (I was subconsciously influenced by Go, I think).
Removing them...
This change renames a few things; more to come, but this is at least
a self-consistent checkpoint:
* Use the "new" keyword to create service objects and not "resource".
* Use the "func" keyword to indicate functions and not "macro".
* Use "bool" instead of "boolean" for boolean types (more Go-like).
First, I meant to rename "func" to "macro". This is perhaps contentious,
however my thinking was to differentiate between functions which have runtime
representations -- like lambdas, API gateway handlers, and the like -- and
macros which are purely a metadata/compile-time construct.
Second, there was a resource naming typo.
This sketches out a minimal configuration language, inspired by a combination
of the Mull document, Hashicorp's various language efforts, Protobufs, and a
mixture of TypeScript. I'm attempting to whittle down the concepts to the bare
minimum necessary for a universal "intermediate language" for our runtime that
is (a) complete enough to serve our needs, (b) advanced enough to deliver some
of the improvements in componentization and reuse that we desire, (c) appealing
enough that humans are able to write code in the language while possibly even
enjoying it, and, yet, (d) sufficient for machine interoperability needs.
Of course, I will capture all of this in a proper specification (overwriting
the existing Mull one), however I wanted to land these as a proof of concept and
for feedback purposes. Obviously, none of this code actually compiles or works!
This is a work-in-progress. In fact, it's going to radically change
given the new approach to representing resource construction, however
I wanted the snapshot in source control as we evolve things.
This change introduces a basic JavaScript SDK (actually in TypeScript,
but consumable either way). This is just scaffolding but provides the
minimal set of abstractions necessary to start writing real stacks.
This demonstrates what infra-as-code might look like. Given the difficulties
in expressing everything purely as metadata, I've decided to embark upon a little
spike to see what this would look like. At first glance, I like it.
To see the difference, compare these to the Mu.yaml files alongside them.
In an older version of this doc, `if` was called `when`, and `for`
was called `each`, to encourage a more declarative feel. I renamed
them back to their familiar forms. I missed a few spots. But I also
thought at least documenting this heritage would be useful, hence
this descriptive checkin comment.
This change includes a miniature spec for what we'd want out of a little
markup language that extends YAML/JSON with typing and minimal templating.
We've begun to reach the limits of what Go's templates give us; the usability
is quite poor: the order of template expansion is "confused" (as it must
happen before verification of stack properties); it is dumb textual copy-and-
paste, and thus knows nothing about the lexical and semantic rules; evaluation
of expressions that should produce actual objects inserted into the metadata
stream as-is must actually be serialized into text (problematic for the above
reasons); and, finally, as a result of all of this, failure modes are terrible.
But worse than this, we simply can't do what we need in many places. For
instance, mapping a stack's properties onto the services that it creates works
in simple cases -- like strings, booleans, and ints -- but quickly breaks down
when referencing complex objects (for the same above reasons). This is why
we've needed to special case property mapping in the aws/x/cf provider, but
clearly this won't generalize to all the compositional situations that arise.
It's worth nothing Hashicorp's HCL/HIL is closest to what we want. (The
language used for Terraform.) It isn't exactly what we want, however, for two
reasons. First, it lacks conditionals and iteration. This is likely to appear
at some point (see https://github.com/hashicorp/terraform/issues/1604), and
indeed in this past week alone, a new C-like conditional operator (which I
actually don't love) got added to HIL:
5fe4b10b43.
Second, and perhaps more importantly, its approach is to create a new language.
The design I list here is a natural extension that adds typechecking and
minimal templating to the existing YAML/JSON formats. As a stand-alone
project, this whould have a much broader appeal. And whether or not we use it
for Mu depends on whether we really want an entirely new markup language or not.
To cut to the chase, I'm shelving this for now. I'm going to keep hacking my
way through the current Go templates plus special-casing for now. My eye is
on the initial end-to-end prototype. But, no doubt, we'll need to revisit this
immediately afterwards, make a decision, and make it happen.