In this change, ":= new" declares a new resource.
In addition, we change arrays to be Go-like, in that zero-initialized
ones can be appended to safely, such that declaring a new instance is
done merely by stating "var arr: T[]".
This change encapsulates what's required for an AWS cluster in new service,
mu/clouds/aws/Cluster, so that mu/Cluster can remain fairly simplistic (it
just switches on the names and dispatches to the appropriate sub-modules).
This checkin includes a new design doc for the Mull metadata language.
This is very much a work-in-progress.
This will eventually supersede the language.md document.
There's no need for quotes here and having them makes it feel somehow
"dynamically typed" (I was subconsciously influenced by Go, I think).
Removing them...
This change renames a few things; more to come, but this is at least
a self-consistent checkpoint:
* Use the "new" keyword to create service objects and not "resource".
* Use the "func" keyword to indicate functions and not "macro".
* Use "bool" instead of "boolean" for boolean types (more Go-like).
First, I meant to rename "func" to "macro". This is perhaps contentious,
however my thinking was to differentiate between functions which have runtime
representations -- like lambdas, API gateway handlers, and the like -- and
macros which are purely a metadata/compile-time construct.
Second, there was a resource naming typo.
This sketches out a minimal configuration language, inspired by a combination
of the Mull document, Hashicorp's various language efforts, Protobufs, and a
mixture of TypeScript. I'm attempting to whittle down the concepts to the bare
minimum necessary for a universal "intermediate language" for our runtime that
is (a) complete enough to serve our needs, (b) advanced enough to deliver some
of the improvements in componentization and reuse that we desire, (c) appealing
enough that humans are able to write code in the language while possibly even
enjoying it, and, yet, (d) sufficient for machine interoperability needs.
Of course, I will capture all of this in a proper specification (overwriting
the existing Mull one), however I wanted to land these as a proof of concept and
for feedback purposes. Obviously, none of this code actually compiles or works!
This is a work-in-progress. In fact, it's going to radically change
given the new approach to representing resource construction, however
I wanted the snapshot in source control as we evolve things.
This change introduces a basic JavaScript SDK (actually in TypeScript,
but consumable either way). This is just scaffolding but provides the
minimal set of abstractions necessary to start writing real stacks.
This demonstrates what infra-as-code might look like. Given the difficulties
in expressing everything purely as metadata, I've decided to embark upon a little
spike to see what this would look like. At first glance, I like it.
To see the difference, compare these to the Mu.yaml files alongside them.
In an older version of this doc, `if` was called `when`, and `for`
was called `each`, to encourage a more declarative feel. I renamed
them back to their familiar forms. I missed a few spots. But I also
thought at least documenting this heritage would be useful, hence
this descriptive checkin comment.
This change includes a miniature spec for what we'd want out of a little
markup language that extends YAML/JSON with typing and minimal templating.
We've begun to reach the limits of what Go's templates give us; the usability
is quite poor: the order of template expansion is "confused" (as it must
happen before verification of stack properties); it is dumb textual copy-and-
paste, and thus knows nothing about the lexical and semantic rules; evaluation
of expressions that should produce actual objects inserted into the metadata
stream as-is must actually be serialized into text (problematic for the above
reasons); and, finally, as a result of all of this, failure modes are terrible.
But worse than this, we simply can't do what we need in many places. For
instance, mapping a stack's properties onto the services that it creates works
in simple cases -- like strings, booleans, and ints -- but quickly breaks down
when referencing complex objects (for the same above reasons). This is why
we've needed to special case property mapping in the aws/x/cf provider, but
clearly this won't generalize to all the compositional situations that arise.
It's worth nothing Hashicorp's HCL/HIL is closest to what we want. (The
language used for Terraform.) It isn't exactly what we want, however, for two
reasons. First, it lacks conditionals and iteration. This is likely to appear
at some point (see https://github.com/hashicorp/terraform/issues/1604), and
indeed in this past week alone, a new C-like conditional operator (which I
actually don't love) got added to HIL:
5fe4b10b43.
Second, and perhaps more importantly, its approach is to create a new language.
The design I list here is a natural extension that adds typechecking and
minimal templating to the existing YAML/JSON formats. As a stand-alone
project, this whould have a much broader appeal. And whether or not we use it
for Mu depends on whether we really want an entirely new markup language or not.
To cut to the chase, I'm shelving this for now. I'm going to keep hacking my
way through the current Go templates plus special-casing for now. My eye is
on the initial end-to-end prototype. But, no doubt, we'll need to revisit this
immediately afterwards, make a decision, and make it happen.
* Persue the default/optional checking if a property value == nil.
* Use the Interface() function to convert a reflect.Type to its underlying
interface{} value. This is required for typechecking to check out.
* Also, unrelated to the above, change type assertions to use nil rather than
allocating real objects. Although minimal, this incurs less GC pressure.
In some cases, we want to specialize template generation based on
the options passed to the compiler. This change flows them through
so that they can be accessed as
{{if .Options.SomeSetting}}
...
{{end}}
This change reverts the syntax for arrays back to T[] from []T. The main
reason is that YAML doesn't permit unquoted strings beginning with [], meaning
any array type needs to be quoted as in "[]T", which is annoying compared to all
other primitive types which don't require quotes. And, anyway, this syntax is
more familiar too.
I've also added a number of tests.
Any of the bindXValue routines can fail if there was no way to convert
the interface{} to an ast.Literal. In such a case, we need to issue an
error about the wrong type being passed. Unfortunately, in the most
recent set of changes, we began simply returning nils without issuing
the error. This change fixes that.
This change renames Schemas to Types on Stack. More interestingly, it
renames the JSON/YAML property used to specify them, from "schemas:" to
"types:"; I feel like this reads more naturally, especially as a sister
to the existing "services:" section.
This checkin continues progress on marapongo/mu#9. It's still not
complete, however we're getting there. In particular, this includes:
* Rename of ComplexLiteral to SchemaLiteral, as it is used exclusively
for schematized types. Also includes a set of changes associated
with this, like deep value conversion to `map[string]interface{}`.
* Binding of schema types included within a Stack. This allows names in
type references to be bound to those schema types during typechecking.
This also includes binding schema properties, reusing all the existing
property binding logic for stacks. In this way, properties between
stacks and custom schema types are one and the same, which is nice.
* Enforcement for custom schema constraints; this includes Pattern,
MaxLength, MinLength, Maximum, and Minimum, as per the JSON Schema
specification.
This change overhauls the core of how types are used by the entire
compiler. In particular, we now have an ast.Type, and have begun
using its use where appropriate. An ast.Type is a union representing
precisely one of the possible sources of types in the system:
* Primitive type: any, bool, number, string, or service.
* Stack type: a resolved reference to an actual concrete stack.
* Schema type: a resolved reference to an actual concrete schema.
* Unresolved reference: a textual reference that hasn't yet been
resolved to a concrete artifact.
* Uninstantiated reference: a reference that has been resolved to
an uninstantiated stack, but hasn't been bound to a concrete
result yet. Right now, this can point to a stack, however
eventually we would imagine this supporting inter-stack schema
references also.
* Decorated type: either an array or a map; in the array case, there
is a single inner element type; in the map case, there are two,
the keys and values; in all cases, the type recurses to any of the
possibilities listed here.
All of the relevant AST nodes have been overhauled accordingly.
In addition to this, we now have an ast.Schema type. It is loosely
modeled on JSON Schema in its capabilities (http://json-schema.org/).
Although we parse and perform some visitation and binding of these,
there are mostly placeholders left in the code for the interesting
aspects, such as registering symbols, resolving dependencies, and
typechecking usage of schema types.
This is part of the ongoing work behind marapongo/mu#9.
As part of marapongo/mu#9, we want to enable extensible schema types
for stronger typechecking at compile-time. JSON Schema seems like a
decent starting place (http://json-schema.org/), although it's not yet
clear whether we want to use its module/naming schema or our own. I
suspect we want to use our own so that stack schemas can be managed
using the same discipline as stack management generally.
This projects the basic AWS::ECS::TaskDefinition CloudFormation template
type as a stack. It also maps a bunch of schema types using fictitious
syntax, since we don't yet support this (see marapongo/mu#9). On to that next...
This change leverages intrinsics in place of the predefined types.
It remains to be seen if we can reach 100% on this, however I am hopeful.
It's also nice that the system will be built "out of itself" with this
approach; in other words, each of the types is simply a Mufile that can
use conditional targeting as appropriate for the given cloud providers.
If we find that this isn't enough, we can always bring back the concept.
The route resource is a bit funny, in that it lets you jam in either
an internet gateway ID or a VPC ID as the GatewayId property, whereas
all of the other target kinds get their own property. Our stack type
offers stronger typing than this (using service references), but we
need to map the resulting ref strings correctly.
This change stops using the short-hand "!Ref" YAML syntax. The Golang
marshaler encodes it with quotes and, apparently, has no way to suppress
this behavior; this isn't surprising, since the YAML parser we're using
admits it doesn't support this aspect of the YAML spec fully. But that's
okay, the long-hand syntax works just fine, and has the added benefit
that we don't need to special case the logig for JSON versus YAML.