This checkin includes a new design doc for the Mull metadata language.
This is very much a work-in-progress.
This will eventually supersede the language.md document.
There's no need for quotes here and having them makes it feel somehow
"dynamically typed" (I was subconsciously influenced by Go, I think).
Removing them...
This change renames a few things; more to come, but this is at least
a self-consistent checkpoint:
* Use the "new" keyword to create service objects and not "resource".
* Use the "func" keyword to indicate functions and not "macro".
* Use "bool" instead of "boolean" for boolean types (more Go-like).
First, I meant to rename "func" to "macro". This is perhaps contentious,
however my thinking was to differentiate between functions which have runtime
representations -- like lambdas, API gateway handlers, and the like -- and
macros which are purely a metadata/compile-time construct.
Second, there was a resource naming typo.
This sketches out a minimal configuration language, inspired by a combination
of the Mull document, Hashicorp's various language efforts, Protobufs, and a
mixture of TypeScript. I'm attempting to whittle down the concepts to the bare
minimum necessary for a universal "intermediate language" for our runtime that
is (a) complete enough to serve our needs, (b) advanced enough to deliver some
of the improvements in componentization and reuse that we desire, (c) appealing
enough that humans are able to write code in the language while possibly even
enjoying it, and, yet, (d) sufficient for machine interoperability needs.
Of course, I will capture all of this in a proper specification (overwriting
the existing Mull one), however I wanted to land these as a proof of concept and
for feedback purposes. Obviously, none of this code actually compiles or works!
This is a work-in-progress. In fact, it's going to radically change
given the new approach to representing resource construction, however
I wanted the snapshot in source control as we evolve things.
This change introduces a basic JavaScript SDK (actually in TypeScript,
but consumable either way). This is just scaffolding but provides the
minimal set of abstractions necessary to start writing real stacks.
This demonstrates what infra-as-code might look like. Given the difficulties
in expressing everything purely as metadata, I've decided to embark upon a little
spike to see what this would look like. At first glance, I like it.
To see the difference, compare these to the Mu.yaml files alongside them.
In an older version of this doc, `if` was called `when`, and `for`
was called `each`, to encourage a more declarative feel. I renamed
them back to their familiar forms. I missed a few spots. But I also
thought at least documenting this heritage would be useful, hence
this descriptive checkin comment.
This change includes a miniature spec for what we'd want out of a little
markup language that extends YAML/JSON with typing and minimal templating.
We've begun to reach the limits of what Go's templates give us; the usability
is quite poor: the order of template expansion is "confused" (as it must
happen before verification of stack properties); it is dumb textual copy-and-
paste, and thus knows nothing about the lexical and semantic rules; evaluation
of expressions that should produce actual objects inserted into the metadata
stream as-is must actually be serialized into text (problematic for the above
reasons); and, finally, as a result of all of this, failure modes are terrible.
But worse than this, we simply can't do what we need in many places. For
instance, mapping a stack's properties onto the services that it creates works
in simple cases -- like strings, booleans, and ints -- but quickly breaks down
when referencing complex objects (for the same above reasons). This is why
we've needed to special case property mapping in the aws/x/cf provider, but
clearly this won't generalize to all the compositional situations that arise.
It's worth nothing Hashicorp's HCL/HIL is closest to what we want. (The
language used for Terraform.) It isn't exactly what we want, however, for two
reasons. First, it lacks conditionals and iteration. This is likely to appear
at some point (see https://github.com/hashicorp/terraform/issues/1604), and
indeed in this past week alone, a new C-like conditional operator (which I
actually don't love) got added to HIL:
5fe4b10b43.
Second, and perhaps more importantly, its approach is to create a new language.
The design I list here is a natural extension that adds typechecking and
minimal templating to the existing YAML/JSON formats. As a stand-alone
project, this whould have a much broader appeal. And whether or not we use it
for Mu depends on whether we really want an entirely new markup language or not.
To cut to the chase, I'm shelving this for now. I'm going to keep hacking my
way through the current Go templates plus special-casing for now. My eye is
on the initial end-to-end prototype. But, no doubt, we'll need to revisit this
immediately afterwards, make a decision, and make it happen.
* Persue the default/optional checking if a property value == nil.
* Use the Interface() function to convert a reflect.Type to its underlying
interface{} value. This is required for typechecking to check out.
* Also, unrelated to the above, change type assertions to use nil rather than
allocating real objects. Although minimal, this incurs less GC pressure.
In some cases, we want to specialize template generation based on
the options passed to the compiler. This change flows them through
so that they can be accessed as
{{if .Options.SomeSetting}}
...
{{end}}
This change reverts the syntax for arrays back to T[] from []T. The main
reason is that YAML doesn't permit unquoted strings beginning with [], meaning
any array type needs to be quoted as in "[]T", which is annoying compared to all
other primitive types which don't require quotes. And, anyway, this syntax is
more familiar too.
I've also added a number of tests.
Any of the bindXValue routines can fail if there was no way to convert
the interface{} to an ast.Literal. In such a case, we need to issue an
error about the wrong type being passed. Unfortunately, in the most
recent set of changes, we began simply returning nils without issuing
the error. This change fixes that.
This change renames Schemas to Types on Stack. More interestingly, it
renames the JSON/YAML property used to specify them, from "schemas:" to
"types:"; I feel like this reads more naturally, especially as a sister
to the existing "services:" section.
This checkin continues progress on marapongo/mu#9. It's still not
complete, however we're getting there. In particular, this includes:
* Rename of ComplexLiteral to SchemaLiteral, as it is used exclusively
for schematized types. Also includes a set of changes associated
with this, like deep value conversion to `map[string]interface{}`.
* Binding of schema types included within a Stack. This allows names in
type references to be bound to those schema types during typechecking.
This also includes binding schema properties, reusing all the existing
property binding logic for stacks. In this way, properties between
stacks and custom schema types are one and the same, which is nice.
* Enforcement for custom schema constraints; this includes Pattern,
MaxLength, MinLength, Maximum, and Minimum, as per the JSON Schema
specification.
This change overhauls the core of how types are used by the entire
compiler. In particular, we now have an ast.Type, and have begun
using its use where appropriate. An ast.Type is a union representing
precisely one of the possible sources of types in the system:
* Primitive type: any, bool, number, string, or service.
* Stack type: a resolved reference to an actual concrete stack.
* Schema type: a resolved reference to an actual concrete schema.
* Unresolved reference: a textual reference that hasn't yet been
resolved to a concrete artifact.
* Uninstantiated reference: a reference that has been resolved to
an uninstantiated stack, but hasn't been bound to a concrete
result yet. Right now, this can point to a stack, however
eventually we would imagine this supporting inter-stack schema
references also.
* Decorated type: either an array or a map; in the array case, there
is a single inner element type; in the map case, there are two,
the keys and values; in all cases, the type recurses to any of the
possibilities listed here.
All of the relevant AST nodes have been overhauled accordingly.
In addition to this, we now have an ast.Schema type. It is loosely
modeled on JSON Schema in its capabilities (http://json-schema.org/).
Although we parse and perform some visitation and binding of these,
there are mostly placeholders left in the code for the interesting
aspects, such as registering symbols, resolving dependencies, and
typechecking usage of schema types.
This is part of the ongoing work behind marapongo/mu#9.
As part of marapongo/mu#9, we want to enable extensible schema types
for stronger typechecking at compile-time. JSON Schema seems like a
decent starting place (http://json-schema.org/), although it's not yet
clear whether we want to use its module/naming schema or our own. I
suspect we want to use our own so that stack schemas can be managed
using the same discipline as stack management generally.
This projects the basic AWS::ECS::TaskDefinition CloudFormation template
type as a stack. It also maps a bunch of schema types using fictitious
syntax, since we don't yet support this (see marapongo/mu#9). On to that next...
This change leverages intrinsics in place of the predefined types.
It remains to be seen if we can reach 100% on this, however I am hopeful.
It's also nice that the system will be built "out of itself" with this
approach; in other words, each of the types is simply a Mufile that can
use conditional targeting as appropriate for the given cloud providers.
If we find that this isn't enough, we can always bring back the concept.
The route resource is a bit funny, in that it lets you jam in either
an internet gateway ID or a VPC ID as the GatewayId property, whereas
all of the other target kinds get their own property. Our stack type
offers stronger typing than this (using service references), but we
need to map the resulting ref strings correctly.
This change stops using the short-hand "!Ref" YAML syntax. The Golang
marshaler encodes it with quotes and, apparently, has no way to suppress
this behavior; this isn't surprising, since the YAML parser we're using
admits it doesn't support this aspect of the YAML spec fully. But that's
okay, the long-hand syntax works just fine, and has the added benefit
that we don't need to special case the logig for JSON versus YAML.
I think things have gotten a little out of hand with the way mu/x/cf
auto-maps properties. In the beginning, it looked like everything could
be trivially auto-mapped, and I wanted to avoid the verbosity of mapping
each property by hand (since you can easily fat finger a name, mess up
capitalization, forget one, etc). But then we began mapping service
references using proper CloudFormation !Refs, which meant suppressing
some of the auto-mappings, etc., etc. This led to properties, extraProperties,
skipProperties, renamedProperties, and so on... Pretty confusing IMHO.
I just took a step back and decided to eliminate auto-mapping. Instead,
you get two options: properties just lists a set of property name mappings,
and extraProperties lets you do template magic to map thing instead if you
wish to take matters into your own hands. The result isn't too verbose
and has a lot less magic going on so it's easier to understand.
This change eliminates the special type mu/extension in favor of extensible
intrinsic types. This subsumes the previous functionality while also fixing
a number of warts with the old model.
In particular, the old mu/extension approach deferred property binding until
very late in the compiler. In fact, too late. The backend provider for an
extension simply received an untyped bag of stuff, which it then had to
deal with. Unfortunately, some operations in the binder are inaccessible
at this point because doing so would cause a cycle. Furthermore, some
pertinent information is gone at this point, like the scopes and symtables.
The canonical example where we need this is binding services names to the
services themselves; e.g., the AWS CloudFormation "DependsOn" property should
resolve to the actual service names, not the string values. In the limit,
this requires full binding information.
There were a few solutions I considered, including ones that've required
less code motion, however this one feels the most elegant.
Now we permit types to be marked as "intrinsic." Binding to these names
is done exactly as ordinary name binding, unlike the special mu/extension
provider name. In fact, just about everything except code-generation for
these types is the same as ordinary types. This is perfect for the use case
at hand, which is binding properties.
After this change, for example, "DependsOn" is expanded to real service
names precisely as we need.
As part of this change, I added support for three new basic schema types:
* ast.StringList ("string[]"): a list of strings.
* ast.StringMap ("map[string]any"): a map of strings to anys.
* ast.ServiceList ("service[]"): a list of service references.
Obviously we need to revisit this and add a more complete set. This work
is already tracked by marapongo/mu#9.
At the end of the day, it's likely I will replace all hard-coded predefined
types with intrinsic types, for similar reasons to the above.
There are two cases to consider in the case of a stack's bound properties.
First, it was a stack we didn't need to construct. This is the case for
built-in primitives. In that case, we must rebind each service uniquely.
Second, it was an imported stack, which by definition we had to construct.
In this case, we don't need to rebind the properties -- we already did so
-- and can just reuse them in the service's own bound properties.
I am pondering a better way of representing this and will probably do this
soon. The concept of unconstructed vs. constructed types should be unified
between the built-in types and imported ones. (Kind of like generics.)
But, I still need a little bit more time prototyping before I make up my
mind what a better way to represent all of this might look like.
We need to access the bound property values for a given stack, especially
during code-generation. This information was present for services before,
however not for stacks constructed via other means (e.g., the top-most one).
This change adds a PropertyValues bag plus a corresponding BoundPropertyValues
to the ast.Stack type.
