There are two cases to consider in the case of a stack's bound properties.
First, it was a stack we didn't need to construct. This is the case for
built-in primitives. In that case, we must rebind each service uniquely.
Second, it was an imported stack, which by definition we had to construct.
In this case, we don't need to rebind the properties -- we already did so
-- and can just reuse them in the service's own bound properties.
I am pondering a better way of representing this and will probably do this
soon. The concept of unconstructed vs. constructed types should be unified
between the built-in types and imported ones. (Kind of like generics.)
But, I still need a little bit more time prototyping before I make up my
mind what a better way to represent all of this might look like.
We need to access the bound property values for a given stack, especially
during code-generation. This information was present for services before,
however not for stacks constructed via other means (e.g., the top-most one).
This change adds a PropertyValues bag plus a corresponding BoundPropertyValues
to the ast.Stack type.
During subtypeOf checking, we need to walk the chain of documents from
which a stack came. This is because, due to template expansion, we'll
end up with a different document for instantiated types than uninstantiated
ones. This change keeps track of the parent and walks it appropriately.
The prior change wasn't quite right vis-a-vis service selection. By default,
we actually want to pick the service itself that was named by a capability reference.
We only want to pick one of its public exported services as the selected one when
a selector is given. For convenience, we still have "<service>:." to pick the sole
public service, however, no longer is a selector absolutely required.
This implements support for arbitrary service types on properties,
not just the weakly typed "service". For example, in the AWS stacks,
the aws/ec2/route type requires a routeTable, among other things:
name: aws/ec2/route
properties:
routeTable:
type: aws/ec2/routeTable
This not only binds the definition of such properties, but also the
callsites of those creating stacks and supplying values for them.
This includes checking for concrete, instantiated, and even base
types, so that, for instance, if a custom stack derived from
aws/ec2/routeTable using the base property, in the above example
it could be supplied as a legal value for the routeTable property.
The previous code stored dependencies in a map. This caused non-determinism
in the order in which the resulting dependencies would be bound. Instead of
doing that, this change tracks them in an array, simply using a map to avoid
binding duplicate dependencies.
A stack property can refer to other stack types. For example:
properties:
gateway:
type: aws/ec2/internetGateway
...
In such cases, we need to validate the property during binding,
in addition to binding it to an actual type so that we can later
validate callers who are constructing instances of this stack
and providing property values that we must typecheck.
Note that this binding is subtly different than existing stack
type binding. All the name validation, resolution, and so forth
are the same. However, notice that in this case we are not actually
supplying any property setters. That is, internetGateway is not
an "expanded" type, in that we have not processed any of its templates.
An analogy might help: this is sort of akin referring to an
uninstantiated generic type in a traditional programming language,
versus its instantiated form. In this case, certain properties aren't
available to us, however we can still use it for type identity, etc.
This change properly transforms literal AST nodes during code-gen.
This includes emitting CloudFormation !Refs where appropriate, for
intra-stack references (capability types).
This change adds a handful of property binding tests.
It also fixes:
* AsName should assert IsName.
* Enumerate properties stably, so that it is deterministic.
* Do not issue errors about unrecognized properties for the special
`mu/extension` type. It's entire purpose in life is to offer an
entirely custom set of properties, which the provider is meant to
validate.
* Default to an empty map if properties are missing.
* Add a "/" to the end of the namespace from the workspace, if present.
And rearranges some code:
* Rename the LiteralX types to XLiteral; e.g., StringLiteral instead of
LiteralString. I kept typing XLiteral erroneously.
* Eliminate the Mu prefix on all of the predefined type and service
functions and types. It's superfluous and reads nicer this way.
* Swap the order of "expected" vs. "got" in the error message about
incorrect property types. It used to say "got %v, expected %v"; I
personally find that it is more helpful if it says "expected %v,
got %v". YMMV.
This is an initial pass at property binding. For all stack instantiations,
we must verify that the set of properties supplied are correct. We also must
remember the bound property information so that code-generation has all of
the information it needs to generate correct code (including capability refs).
This entails:
* Ensuring required properties are provided.
* Expanding missing properties that have Default values.
* Type-checking that supplied properties are of the right type.
* Expanding property values into AST literal nodes.
To do this requires a third AST pass in the semantic analysis part of the
compiler. In the 1st pass, dependencies aren't even known yet; in the 2nd
pass, dependencies have not yet been bound; therefore, we need a 3rd pass,
which can depend on the full binding information for the transitive closure
of AST nodes and dependencies to have been populated with types.
There are a few loose ends in here:
* We don't yet validate top-level stack properties.
* We don't yet validate top-level stack base type properties.
* We don't yet support complex schema property types.
* We don't yet support even "simple" complex property types, like `[ string ]`.
* We don't yet support strongly typed capability property types (just `service`).
That said, I am going to turn to writing a few tests for the basic cases, and then
resume to finishing this afterwards (tracked by marapongo/mu#25).
The prior code could miss arrays of strings during conversion because
the arrays created by the various marshalers are weakly typed. In other
words, even though they contain strings, the array type is []interface{}.
This change introduces the encoding.ArrayOfStrings function to perform
this conversion, first by checking for []string and returning that directly
where possible, and second, if that fails, checking each element and copying.
This changes the way binding dependencies works slightly, to ensure that
the full transitive closure of dependencies is bound appropriately before
hitting code-generation. Namely, now binder.PrepareStack returns a list
of unresolved dependency Refs; the compiler is responsible for turning this
into a map from Ref to the loaded diag.Document, before calling BindStack;
then, BindStack instantiates these as necessary (template expansion, etc),
returning an array of unbound *ast.Stacks that the compiler must then bind.
This change moves parse-tree analysis into the Parse* functions, so that
any callers doing parsing don't need to do this as a multi-step activity.
(We had neglected to do the parse-tree analysis phase during dependency
resolution, for example, meaning services were left untyped.)
This change makes workspace file naming a little more consistent with respect
to Mufile naming. Instead of having a .mu/ directory, under which a workspace.yaml
and/or a stacks directory might exist, we now have a Muspace.yaml (or .json) file,
and a .Mudeps/ directory. This has nicer symmetric with respect to Mu.yaml files.
If compiling a stack that accepts properties directly, we need a way to
pass arguments to that stack at the command line. This change permits this
using the ordinary "--" style delimiter; for example:
$ mu build -- --name=Foo
This is super basic and doesn't handle all the edge cases, but is sufficient
for testing and prototyping purposes.
This change detects the target cloud earlier on in the compilation process.
Prior to this change, we didn't know this information until the backend code-generation.
Clearly we need to know this at least by then, however, templates can specialize on this
information, so we actually need it sooner. This change moves it into the frontend part.
Note that to support this we now eliminate the ability to specify target clusters in
the Mufile alone. That "feels" right to me anyway, since Mufiles are supposed to be
agnostic to their deployment environment, other than template specialization. Instead,
this information can come from the CLI and/or the workspace settings file.
In some cases, a dependency will resolve to a diag.Document, rather than
a fully instantiated ast.Stack (in fact, that is the common case). The
binder needs to detect and tolerate this situation.
The "require" template function simply checks a condition and, if it
is false, issues an error and quits template processing immediately.
This is useful for concisely encoding validation logic.
In some cases, we actually want to suppress auto-mapping for all or
most of the properties. In those cases, it's easier to specify those
that we *do* want rather than the ones we *do not* want. Now with
properties, skipProperties, and extraProperties, we have all the
necessary flexibility to control auto-mapping for CF templates.
The new "has" function lets templates conveniently check the existence of
keys in property bag-like maps. For example:
{{if has .Properties "something"}}
...
{{end}}
Most properties in CF templates are auto-mapped by the aws/cf extension
provider. However, sometimes we want to inject extra properties that are
outside of that auto-mapping (like our convenient shortcut for supplying
name to mean adding a "Name=v" tag). And sometimes we want to skip auto-
mapping for certain properties (like our capability-based approach to
passing service references, versus string IDs).
This change adds a super simple initial whack at a basic cluster topology
comprised of VPC, subnet, internet gateway, attachments, and route tables.
This is actually written in Mu itself, and I am committing this early, since
there are quite a few features required before we can actually make progress
getting this up and running.
This introduces a "panic" template function, so that templates may abandon
evaluation if something unexpected occurs. It accepts a string, indicating
the error, and optional arguments, if the string is to be formatted.
For example:
{{if eq .Target "aws"}}
...
{{else}}
{{panic "Unrecognized cloud target: %v" .Target}}
{{end}}
This lets YAML files include others, often conditionally, based on things
like the cloud target. For example, I am currently using this to define the
overall cluster stack by doing things like:
name: mu/cluster
services:
{{if eq .Target "aws"}}
{{include "Mu-aws.yaml" | indent 4}}
{{else}}
...
{{end}}
This change performs template expansion both for root stack documents in
addition to the transitive closure of dependencies. There are many ongoing
design and implementation questions about how this should actually work;
please see marapongo/mu#7 for a discussion of them.
For now, we can simply auto-map the Mu properties to CF properties,
eliminating the need to manually map them in the templates. Eventually
we'll want more sophistication here to control various aspects of the CF
templates, but this eliminates a lot of tedious manual work in the meantime.
The only two AST nodes that track any semblance of location right now
are ast.Workspace and ast.Stack. This is simply because, using the standard
JSON and YAML parsers, we aren't given any information about the resulting
unmarshaled node locations. To fix that, we'll need to crack open the parsers
and get our hands dirty. In the meantime, we can crudely implement diag.Diagable
on ast.Workspace and ast.Stack, however, to simply return their diag.Documents.
This change adds a new Diagable interface from which you can obtain
a diagnostic's location information (Document and Location). A new
At function replaces WithDocument, et al., and will be used soon to
permit all arbitrary AST nodes to report back their position.
This change completes the implementation of dependency and type binding.
The top-level change here is that, during the first semantic analysis AST walk,
we gather up all unknown dependencies. Then the compiler resolves them, caching
the lookups to ensure that we don't load the same stack twice. Finally, during
the second and final semantic analysis AST walk, we populate the bound nodes
by looking up what the compiler resolved for us.
This change implements dependency versions, including semantic analysis, per the
checkin 83030685c3.
There's quite a bit in here but at a top-level this parses and validates dependency
references of the form
[[proto://]base.url]namespace/.../name[@version]
and verifies that the components are correct, as well as binding them to symbols.
These references can appear in two places at the moment:
* Service types.
* Cluster dependencies.
As part of this change, a number of supporting changes have been made:
* Parse Workspaces using a full-blown parser, parser analysis, and semantic analysis.
This allows us to share logic around the validation of common AST types. This also
moves some of the logic around loading workspace.yaml files back to the parser, where
it can be unified with the way we load Mu.yaml files.
* New ast.Version and ast.VersionSpec types. The former represents a precise version
-- either a specific semantic version or a short or long Git SHA hash -- and the
latter represents a range -- either a Version, "latest", or a semantic range.
* New ast.Ref and ast.RefParts types. The former is an unparsed string that is
thought to contain a Ref, while the latter is a validated Ref that has been parsed
into its components (Proto, Base, Name, and Version).
* Added some type assertions to ensure certain structs implement certain interfaces,
to speed up finding errors. (And remove the coercions that zero-fill vtbl slots.)
* Be consistent about prefixing error types with Error or Warning.
* Organize the core compiler driver's logic into three methods, FE, sema, and BE.
* A bunch of tests for some of the above ... more to come in an upcoming change.
Right now, the AWS ECS scheduler simply passes through to the underlying
AWS cloud provider. However, now we have the necessary hooks to start
incrementally recognizing stack types and emitting specialized code for
them (e.g., starting with mu/container).
This change adds code-generation for Stack references other than the built-in types.
This permits you to bind to a dependency and have it flow all the way through to the
code-generation phases. It still most likely bottoms out on something that fails,
however for pure AWS resources like aws/s3/bucket everything now works.
This change adds support for Workspaces, a convenient way of sharing settings
among many Stacks, like default cluster targets, configuration settings, and the
like, which are not meant to be distributed as part of the Stack itself.
The following things are included in this checkin:
* At workspace initialization time, detect and parse the .mu/workspace.yaml
file. This is pretty rudimentary right now and contains just the default
cluster targets. The results are stored in a new ast.Workspace type.
* Rename "target" to "cluster". This impacts many things, including ast.Target
being changed to ast.Cluster, and all related fields, the command line --target
being changed to --cluster, various internal helper functions, and so on. This
helps to reinforce the desired mental model.
* Eliminate the ast.Metadata type. Instead, the metadata moves directly onto
the Stack. This reflects the decision to make Stacks "the thing" that is
distributed, versioned, and is the granularity of dependency.
* During cluster targeting, add the workspace settings into the probing logic.
We still search in the same order: CLI > Stack > Workspace.
