This change permits a workspace to specify a namespace, which is just a name
part that is trimmed off the front of directories when probing for inter-
workspace dependencies. For example, if our namespace is aws/, normally we'd
need to organize our namespace into directories like:
<root>
| aws/
| | dynamodb/
| | ec2/
| | s3/
... and so on ...
If we instead specify a namespace
namespace: aws
Then we can instead organize our project workspace as follows:
<root>
| dynamodb/
| ec2/
| s3/
... and so on ...
This is an initial pass at property binding. For all stack instantiations,
we must verify that the set of properties supplied are correct. We also must
remember the bound property information so that code-generation has all of
the information it needs to generate correct code (including capability refs).
This entails:
* Ensuring required properties are provided.
* Expanding missing properties that have Default values.
* Type-checking that supplied properties are of the right type.
* Expanding property values into AST literal nodes.
To do this requires a third AST pass in the semantic analysis part of the
compiler. In the 1st pass, dependencies aren't even known yet; in the 2nd
pass, dependencies have not yet been bound; therefore, we need a 3rd pass,
which can depend on the full binding information for the transitive closure
of AST nodes and dependencies to have been populated with types.
There are a few loose ends in here:
* We don't yet validate top-level stack properties.
* We don't yet validate top-level stack base type properties.
* We don't yet support complex schema property types.
* We don't yet support even "simple" complex property types, like `[ string ]`.
* We don't yet support strongly typed capability property types (just `service`).
That said, I am going to turn to writing a few tests for the basic cases, and then
resume to finishing this afterwards (tracked by marapongo/mu#25).
The prior code could miss arrays of strings during conversion because
the arrays created by the various marshalers are weakly typed. In other
words, even though they contain strings, the array type is []interface{}.
This change introduces the encoding.ArrayOfStrings function to perform
this conversion, first by checking for []string and returning that directly
where possible, and second, if that fails, checking each element and copying.
This changes the way binding dependencies works slightly, to ensure that
the full transitive closure of dependencies is bound appropriately before
hitting code-generation. Namely, now binder.PrepareStack returns a list
of unresolved dependency Refs; the compiler is responsible for turning this
into a map from Ref to the loaded diag.Document, before calling BindStack;
then, BindStack instantiates these as necessary (template expansion, etc),
returning an array of unbound *ast.Stacks that the compiler must then bind.
Well, it turns out glog.Fail is slightly better than panic, because it explicitly
dumps the stacks of *all* goroutines. This is especially good in logging scenarios.
It's really annoying that glog suppresses printing the stack trace (see here
https://github.com/golang/glog/blob/master/glog.go#L719), however this is better.
This change switches away from using glog.Fatalf, and instead uses panic,
should a fail-fast arise (due to a call to util.Fail or a failed assertion
by way of util.Assert). This leads to a better debugging experience no
matter what flags have been passed to glog. For example, glog.Fatal* seems
to suppress printing stack traces when --logtostderr is supplied.
This change moves parse-tree analysis into the Parse* functions, so that
any callers doing parsing don't need to do this as a multi-step activity.
(We had neglected to do the parse-tree analysis phase during dependency
resolution, for example, meaning services were left untyped.)
This change makes workspace file naming a little more consistent with respect
to Mufile naming. Instead of having a .mu/ directory, under which a workspace.yaml
and/or a stacks directory might exist, we now have a Muspace.yaml (or .json) file,
and a .Mudeps/ directory. This has nicer symmetric with respect to Mu.yaml files.
If compiling a stack that accepts properties directly, we need a way to
pass arguments to that stack at the command line. This change permits this
using the ordinary "--" style delimiter; for example:
$ mu build -- --name=Foo
This is super basic and doesn't handle all the edge cases, but is sufficient
for testing and prototyping purposes.
I wasn't sure at first how we'd distinguish between strings and cap names.
At first, I thought we'd need special syntax, e.g. `<vpn>`; however, now I
am leaning towards a "type inference" approach, where we will interpret the
string as a cap name rather than string if that's what the target property
expects. Although this isn't quite as explicitly typed, that's probably a
good thing in this case, as it eliminates verbosity and weird characters.
This change introduces the notion of "perturbing" properties. Changing
one of these impacts the live service, possibly leading to downtime. As
such, we will likely encourage blue/green deployments of them just to be
safe. Note that this is really just a placeholder so I can keep track of
metadata as we go, since AWS CF has a similar notion to this.
I'm not in love with the name. I considered `interrupts`, however,
I must admit I liked that `readonly` and `perturbs` are symmetric in
the number of characters (meaning stuff lines up nicely...)
To avoid injecting needless whitespace -- which is unfortunately meaningful
in YAML -- we need to use the new "-" trimming operators, available in Go 1.6.
This change detects the target cloud earlier on in the compilation process.
Prior to this change, we didn't know this information until the backend code-generation.
Clearly we need to know this at least by then, however, templates can specialize on this
information, so we actually need it sooner. This change moves it into the frontend part.
Note that to support this we now eliminate the ability to specify target clusters in
the Mufile alone. That "feels" right to me anyway, since Mufiles are supposed to be
agnostic to their deployment environment, other than template specialization. Instead,
this information can come from the CLI and/or the workspace settings file.
In some cases, a dependency will resolve to a diag.Document, rather than
a fully instantiated ast.Stack (in fact, that is the common case). The
binder needs to detect and tolerate this situation.
This change projects several EC2 Mu stacks that are required for Mu cluster
bootstrapping. This includes:
- aws/ec2/internetGateway
- aws/ec2/route
- aws/ec2/routeTable
- aws/ec2/securityGroup
- aws/ec2/subnet
- aws/ec2/vpc
- aws/ec2/vpcGatewayAttachment
This is still not complete, and in fact some of these reference other EC2
Stacks that do not yet exist. But we're getting a bit closer...
This change adds the notion of readonly properties to stacks. Although these
*can* be "changed", doing so implies recreation of the resources all over again.
As a result, all dependents must be recreated, in a cascading manner.
The "require" template function simply checks a condition and, if it
is false, issues an error and quits template processing immediately.
This is useful for concisely encoding validation logic.
In some cases, we actually want to suppress auto-mapping for all or
most of the properties. In those cases, it's easier to specify those
that we *do* want rather than the ones we *do not* want. Now with
properties, skipProperties, and extraProperties, we have all the
necessary flexibility to control auto-mapping for CF templates.
The new "has" function lets templates conveniently check the existence of
keys in property bag-like maps. For example:
{{if has .Properties "something"}}
...
{{end}}
Most properties in CF templates are auto-mapped by the aws/cf extension
provider. However, sometimes we want to inject extra properties that are
outside of that auto-mapping (like our convenient shortcut for supplying
name to mean adding a "Name=v" tag). And sometimes we want to skip auto-
mapping for certain properties (like our capability-based approach to
passing service references, versus string IDs).
This change adds a super simple initial whack at a basic cluster topology
comprised of VPC, subnet, internet gateway, attachments, and route tables.
This is actually written in Mu itself, and I am committing this early, since
there are quite a few features required before we can actually make progress
getting this up and running.
This introduces a "panic" template function, so that templates may abandon
evaluation if something unexpected occurs. It accepts a string, indicating
the error, and optional arguments, if the string is to be formatted.
For example:
{{if eq .Target "aws"}}
...
{{else}}
{{panic "Unrecognized cloud target: %v" .Target}}
{{end}}
This lets YAML files include others, often conditionally, based on things
like the cloud target. For example, I am currently using this to define the
overall cluster stack by doing things like:
name: mu/cluster
services:
{{if eq .Target "aws"}}
{{include "Mu-aws.yaml" | indent 4}}
{{else}}
...
{{end}}
This change performs template expansion both for root stack documents in
addition to the transitive closure of dependencies. There are many ongoing
design and implementation questions about how this should actually work;
please see marapongo/mu#7 for a discussion of them.
For now, we can simply auto-map the Mu properties to CF properties,
eliminating the need to manually map them in the templates. Eventually
we'll want more sophistication here to control various aspects of the CF
templates, but this eliminates a lot of tedious manual work in the meantime.
The only two AST nodes that track any semblance of location right now
are ast.Workspace and ast.Stack. This is simply because, using the standard
JSON and YAML parsers, we aren't given any information about the resulting
unmarshaled node locations. To fix that, we'll need to crack open the parsers
and get our hands dirty. In the meantime, we can crudely implement diag.Diagable
on ast.Workspace and ast.Stack, however, to simply return their diag.Documents.
This change adds a new Diagable interface from which you can obtain
a diagnostic's location information (Document and Location). A new
At function replaces WithDocument, et al., and will be used soon to
permit all arbitrary AST nodes to report back their position.
This change completes the implementation of dependency and type binding.
The top-level change here is that, during the first semantic analysis AST walk,
we gather up all unknown dependencies. Then the compiler resolves them, caching
the lookups to ensure that we don't load the same stack twice. Finally, during
the second and final semantic analysis AST walk, we populate the bound nodes
by looking up what the compiler resolved for us.
This changes the logic when parsing RefParts so that we can actually
round-trip them faithfully when required. To do this, by default we
preserve blank ""s in the RefPart fields that are legally omitted from
the Ref string. Then, there is a Defaults() method that can populate
the missing fields with the defaults if desired.
