This change adds a new Diagable interface from which you can obtain
a diagnostic's location information (Document and Location). A new
At function replaces WithDocument, et al., and will be used soon to
permit all arbitrary AST nodes to report back their position.
This change completes the implementation of dependency and type binding.
The top-level change here is that, during the first semantic analysis AST walk,
we gather up all unknown dependencies. Then the compiler resolves them, caching
the lookups to ensure that we don't load the same stack twice. Finally, during
the second and final semantic analysis AST walk, we populate the bound nodes
by looking up what the compiler resolved for us.
This changes the logic when parsing RefParts so that we can actually
round-trip them faithfully when required. To do this, by default we
preserve blank ""s in the RefPart fields that are legally omitted from
the Ref string. Then, there is a Defaults() method that can populate
the missing fields with the defaults if desired.
This change implements dependency versions, including semantic analysis, per the
checkin 83030685c3.
There's quite a bit in here but at a top-level this parses and validates dependency
references of the form
[[proto://]base.url]namespace/.../name[@version]
and verifies that the components are correct, as well as binding them to symbols.
These references can appear in two places at the moment:
* Service types.
* Cluster dependencies.
As part of this change, a number of supporting changes have been made:
* Parse Workspaces using a full-blown parser, parser analysis, and semantic analysis.
This allows us to share logic around the validation of common AST types. This also
moves some of the logic around loading workspace.yaml files back to the parser, where
it can be unified with the way we load Mu.yaml files.
* New ast.Version and ast.VersionSpec types. The former represents a precise version
-- either a specific semantic version or a short or long Git SHA hash -- and the
latter represents a range -- either a Version, "latest", or a semantic range.
* New ast.Ref and ast.RefParts types. The former is an unparsed string that is
thought to contain a Ref, while the latter is a validated Ref that has been parsed
into its components (Proto, Base, Name, and Version).
* Added some type assertions to ensure certain structs implement certain interfaces,
to speed up finding errors. (And remove the coercions that zero-fill vtbl slots.)
* Be consistent about prefixing error types with Error or Warning.
* Organize the core compiler driver's logic into three methods, FE, sema, and BE.
* A bunch of tests for some of the above ... more to come in an upcoming change.
I've gone backwards and forwards on the design for dependency version
management. However, I think what's written in this commit represents
a pretty sane "sweet spot" between all available options.
In a nutshell, anytime reference to a stack type in your Mufile is a
full-blown StackRef; in other words, it has a protocol (e.g., "https://"),
a base URL (e.g., "hub.mu.com/"), a name (e.g., "aws/s3/bucket"), and a
version ("@^1.0.6"). Each reference carries all of these components.
For convenience, you may omit the components. In that case, Mu chooses
reasonable defaults:
* "https://" as the default protocol (or "git://"; this is TBD).
* "hub.mu.com/" as the default base URL.
* "@latest" as the default version number.
Note that a version can be "latest" to mean "tip", a specific SHA hash
to pin to a precise version, or a semantic version number/range.
I had originally shied away from specifying versions inline as you reference
stacks in your Mufile, and was leaning towards an explicit dependencies
section, however I was swayed for two reasons:
1. It's very common to only consume a given stack once in a file. Needing
to specify it in two places each time is verbose and violates DRY.
2. We have decided that each Mufile stands on its own and forms the unit
of distribution. I had previously thought we might move dependencies
out of Mufiles into a "package manager" specification. Lacking that,
there is even less reason to call them out in a dedicated section.
Now, managing all these embedded version numbers across multiple stacks in
a single workspace would have been annoying. (Many edits for a single
version bump.) Instead, I've added provisions for storing this in your
workspace.yaml file. The way it works is if any StackRef lacks a version
number, before defaulting to "@latest" we check the workspace.yaml file and,
if a default is found in there, we will use it. For example:
dependencies:
aws/s3/bucket: ^1.0.6
The provision for pinning an entire namespace is also preserved. E.g.:
dependencies:
aws/...: ^1.0.6
Right now, the AWS ECS scheduler simply passes through to the underlying
AWS cloud provider. However, now we have the necessary hooks to start
incrementally recognizing stack types and emitting specialized code for
them (e.g., starting with mu/container).
This change adds code-generation for Stack references other than the built-in types.
This permits you to bind to a dependency and have it flow all the way through to the
code-generation phases. It still most likely bottoms out on something that fails,
however for pure AWS resources like aws/s3/bucket everything now works.
This change adds support for Workspaces, a convenient way of sharing settings
among many Stacks, like default cluster targets, configuration settings, and the
like, which are not meant to be distributed as part of the Stack itself.
The following things are included in this checkin:
* At workspace initialization time, detect and parse the .mu/workspace.yaml
file. This is pretty rudimentary right now and contains just the default
cluster targets. The results are stored in a new ast.Workspace type.
* Rename "target" to "cluster". This impacts many things, including ast.Target
being changed to ast.Cluster, and all related fields, the command line --target
being changed to --cluster, various internal helper functions, and so on. This
helps to reinforce the desired mental model.
* Eliminate the ast.Metadata type. Instead, the metadata moves directly onto
the Stack. This reflects the decision to make Stacks "the thing" that is
distributed, versioned, and is the granularity of dependency.
* During cluster targeting, add the workspace settings into the probing logic.
We still search in the same order: CLI > Stack > Workspace.
This changes the probing logic for dependency resolution. The old logic was
inconsistent between the various roots. The new approach simply prefers locations
with a base URL component -- since they are more specific -- but will allow for
locations missing a base URL component. This is convenient for developers managing
a workspace where needing to specify the base URL in the path is annoying and
slightly too "opinionated" for my taste (especially for migrating existing services).
Also add some convenience helper methods to deal with names, including
IsName and AsName, which validate that names obey the intended regular expressions.
This change includes logic to resolve dependencies declared by stacks. The design
is described in https://github.com/marapongo/mu/blob/master/docs/deps.md.
In summary, each stack may declare dependencies, which are name/semver pairs. A
new structure has been introduced, ast.Ref, to distinguish between ast.Names and
dependency names. An ast.Ref includes a protocol, base part, and a name part (the
latter being an ast.Name); for example, in "https://hub.mu.com/mu/container/",
"https://" is the protocol, "hub.mu.com/" is the base, and "mu/container" is the
name. This is used to resolve URL-like names to package manager-like artifacts.
The dependency resolution phase happens after parsing, but before semantic analysis.
This is because dependencies are "source-like" in that we must load and parse all
dependency metadata files. We stick the full transitive closure of dependencies
into a map attached to the compiler to avoid loading dependencies multiple times.
Note that, although dependencies prohibit cycles, this forms a DAG, meaning multiple
inbound edges to a single stack may come from multiple places.
From there, we rely on ordinary visitation to deal with dependencies further.
This includes inserting symbol entries into the symbol table, mapping names to the
loaded stacks, during the first phase of binding so that they may be found
subsequently when typechecking during the second phase and beyond.
This change introduces a Workspace interface that can be used as a first
class object. We will embellish this as we start binding to dependencies,
which requires us to search multiple paths. This change also introduces a
workspace.InstallRoot() function to fetch the Mu install path.
We will use the $MUROOT envvar to determine where Mu has been installed,
which will by default be /usr/local/mu. From there, we can access the
predefined library of stacks (underneath $MUROOT/bin/stacks).
This change moves the workspace and Mufile detection logic out of the compiler
package and into the workspace one.
This also sketches out the overall workspace structure. A workspace is "delimited"
by the presence of a .mu/ directory anywhere in the parent ancestry. Inside of that
directory we have an optional .mu/clusters.yaml (or .json) file containing cluster
settings shared among the whole workspace. We also have an optional .mu/stacks/
directory that contains dependencies used during package management.
The notion of a "global" workspace will also be present, which is essentially just
a .mu/ directory in your home, ~/.mu/, that has an equivalent structure, but can be
shared among all workspaces on the same machine.
This change moves glogging into the Mu command, so that `mu --help`
actually shows the right thing. (Sadly, glog hijacks the command help.)
It also adds an option --logtostderr that redirects glog output to the
console.
This change mostly replaces explicit if/then/glog.Fatalf calls with
util.Assert calls. In addition, it adds a companion util.Fail family
of methods that does the same thing as a failed assertion, except that
it is unconditional.
If the first rune is unprintable, then we don't want to go ahead and force
capitalization on the next character. (Unlike any other non-first rune,
where of course we do.) In the case of a first rune, we want to let the
current default based on the pascal parameter take charge.
This uses normal AWS resource naming conventions during stack template
creation. Part of this is just a "best practice" thing, however, part of it
is also that we generate illegal names if Mu stacks have illegal characters
like /, -, and so on.
AWS uses capitalized property names for its markup, so we should
be looking for "Type" and "Properties", not "type" and "properties"
when validating that a aws/cf is formatted correctly.
This change eliminates the diag.Sink field, Diag, on the Compiland struct.
Instead, we should provide it at backend provider construction time. This
is consistent with how other phases of the compiler work and also ensures
the backends can properly implement the core.Phase interface.
This change implements the aws/cf extension provider, so that AWS resources
may be described and encapsulated inside of other stacks. Each aws/cf instantiation
requires just two fields -- type and properties -- corresponding to the equivalent
AWS resource object. The result is simply plugged in as an AWS resource, after
Mu templates have been expanded, permitting stack properties, etc. to be used.
The more I live with the current system, the more I prefer "properties" to
"parameters" for stacks and services. Although it is true that these things
are essentially construction-time arguments, they manifest more like properties
in the way they are used; in fact, if you think of the world in terms of primary
constructors, the distinction is pretty subtle anyway.
For example, when creating a new service, we say the following:
services:
private:
some/service:
a: 0
b: true
c: foo
This looks like a, b, and c are properties of the type some/service. If, on
the other hand, we kept calling these parameters, then you'd arguably prefer to
see the following:
services:
private:
some/service:
arguments:
a: 0
b: true
c: foo
This is a more imperative than declarative view of the world, which I dislike
(especially because it is more verbose).
Time will tell whether this is the right decision or not ...
This introduces three assertion functions:
* util.Assert: simply asserts a boolean condition, ripping the process using
glog should it fail, with a stock message.
* util.AssertM: asserts a boolean condition, also using glog if it fails,
but it comes with a message to help with debugging too.
* util.AssertMF: the same, except it formats the message with arguments.
During unmarshaling, the default behavior of the stock Golang JSON marshaler,
and consequently the YAML one we used which mimics its behavior, is to toss away
unrecognized properties. This isn't what we want for two reasons:
First, we want to issue errors/warnings on unrecognized fields to aid in diagnostics;
we will set aside some extensible section for 3rd parties to use. This is not
addressed in this change, however.
Second, and more pertinent, is that we need to retain unrecognized fields for certain
types like services, which are extensible by default.
Until golang/go#6213 is addressed -- imminent, it seems -- we will have to do a
somewhat hacky workaround to this problem. This change contains what I consider to
be the "least bad" in that we won't introduce a lot of performance overhead, and
just have to deal with the slight annoyance of the ast.Services node type containing
both Public/Private *and* PublicUntyped/PrivateUntyped fields alongside one another.
The marshaler dumps property bags into the *Untyped fields, and the parsetree analyzer
expands them out into a structured ast.Service type. Subsequent passes can then
ignore the *Untyped fields altogether.
Note that this would cause some marshaling funkiness if we ever wanted to remarshal
the mutated ASTs back into JSON/YAML. Since we don't do that right now, however, I've
not made any attempt to keep the two pairs in synch. Post-parsetree analyzer, we
literally just forget about the *Untyped guys.
This change rearranges the last checkin a little bit. Rather than storing
shadow BoundPublic/BoundPrivate maps, we will store the *ast.Stack directly on
the ast.Service node itself. This helps with context-free manipulation (e.g.,
you don't need access to the parent map just to interact with the node), and
simplifies the backend code quite a bit (again, less context to pass).
This is another change of mostly placeholders.
In general, there will be three kinds of types handled by code-generation:
* Mu primitives will be expanded into AWS goo in a very specialized way, to
accomplish the desired Mu semantics for those abstractions.
* AWS-specific extension types (mu/extension) will be recognized, so that we
can create special AWS resources like S3 buckets, DynamoDB tables, etc.
* Anything else is interpreted as a reference to another stack that will be
instantiated at deployment time (basically through template expansion).
This change does rearrange two noteworthy things in the core compiler, however:
first, it creates a place for bound nodes in the public and private service
references, so that the backend can access the raw stack types behind them; and
second, it moves the predefined types underneath their own package to avoid cycles.
