Commit graph

76 commits

Author SHA1 Message Date
joeduffy 8d71771391 Repivot plan/apply commands; prepare for updates
This change repivots the plan/apply commands slightly.  This is largely
in preparation for performing deletes and updates of existing environments.

The old way was slightly confusing and made things appear more "magical"
than they actually are.  Namely, different things are needed for different
kinds of deployment operations, and trying to present them each underneath
a single pair of CLI commands just leads to weird modality and options.

The new way is to offer three commands: create, update, and delete.  Each
does what it says on the tin: create provisions a new environment, update
makes resource updates to an existing one, and delete tears down an existing
one entirely.  The arguments are what make this interesting: create demands
a MuPackage to evaluate (producing the new desired state snapshot), update
takes *both* an existing snapshot file plus a MuPackage to evaluate (producing
the new desired state snapshot to diff against the existing one), and delete
merely takes an existing snapshot file and no MuPackage, since all it must
do is tear down an existing known environment.

Replacing the plan functionality is the --dry-run (-n) flag that may be
passed to any of the above commands.  This will print out the plan without
actually performing any opterations.

All commands produce serializable resource files in the MuGL file format,
and attempt to do smart things with respect to backups, etc., to support the
intended "Git-oriented" workflow of the pure CLI dev experience.
2017-02-22 11:21:26 -08:00
joeduffy 330a6c3d25 Probe for Mupacks, not Mufiles, for dependencies 2017-02-09 11:23:27 -08:00
joeduffy e72cf059fb Fix up dependency probing
This changes a few things with dependency probing:

1) Probe for Mupack files, not Mufiles.

2) Substitute defaults in the PackageURL before probing.

3) Trace the full search paths when an import fails to resolve.
   This will help diagnose dependency resolution issues.
2017-02-09 11:01:02 -08:00
joeduffy c8044b66ce Fix up a bunch of golint errors 2017-01-27 15:42:39 -08:00
joeduffy 2c0266c9e4 Clean up package URL logic
This change rearranges the old way we dealt with URLs.  In the old system,
virtually every reference to an element, including types, was fully qualified
with a possible URL-like reference.  (The old pkg/tokens/Ref type.)  In the
new model, only dependency references are URL-like.  All maps and references
within the MuPack/MuIL format are token and name based, using the new
pkg/tokens/Token and pkg/tokens/Name family of related types.

As such, this change renames Ref to PackageURLString, and RefParts to
PackageURL.  (The convenient name is given to the thing with "more" structure,
since we prefer to deal with structured types and not strings.)  It moves
out of the pkg/tokens package and into pkg/pack, since it is exclusively
there to support package resolution.  Similarly, the Version, VersionSpec,
and related types move out of pkg/tokens and into pkg/pack.

This change cleans up the various binder, package, and workspace logic.
Most of these changes are a natural fallout of this overall restructuring,
although in a few places we remained sloppy about the difference between
Token, Name, and URL.  Now the type system supports these distinctions and
forces us to be more methodical about any conversions that take place.
2017-01-20 11:46:36 -08:00
joeduffy 7b0dc8ee8d Overhaul names versus tokens
I was sloppy in my use of names versus tokens in the original AST.
Now that we're actually binding things to concrete symbols, etc., we
need to be more precise.  In particular, names are just identifiers
that must be "interpreted" in a given lexical context for them to
make any sense; whereas, tokens stand alone and can be resolved without
context other than the set of imported packages, modules, and overall
module structure.  As such, names are much simpler than tokens.

As explained in the comments, tokens.Names are simple identifiers:

    Name = [A-Za-z_][A-Za-z0-9_]*

and tokens.QNames are fully qualified identifiers delimited by "/":

    QName = [ <Name> "/" ]* <Name>

The legal grammar for a token depends on the subset of symbols that
token is meant to represent.  However, the most general case, that
accepts all specializations of tokens, is roughly as follows:

    Token       = <Name> |
                  <PackageName>
                    [ ":" <ModuleName>
                        [ "/" <ModuleMemberName>
                            [ "." <Class MemberName> ]
                        ]
                    ]

where:

    PackageName         = <QName>
    ModuleName          = <QName>
    ModuleMemberName    = <Name>
    ClassMemberName     = <Name>

Please refer to the comments in pkg/tokens/tokens.go for more details.
2017-01-19 17:57:20 -08:00
joeduffy 25632886c8 Begin overhauling semantic phases
This change further merges the new AST and MuPack/MuIL formats and
abstractions into the core of the compiler.  A good amount of the old
code is gone now; I decided against ripping it all out in one fell
swoop so that I can methodically check that we are preserving all
relevant decisions and/or functionality we had in the old model.

The changes are too numerous to outline in this commit message,
however, here are the noteworthy ones:

    * Split up the notion of symbols and tokens, resulting in:

        - pkg/symbols for true compiler symbols (bound nodes)
        - pkg/tokens for name-based tokens, identifiers, constants

    * Several packages move underneath pkg/compiler:

        - pkg/ast becomes pkg/compiler/ast
        - pkg/errors becomes pkg/compiler/errors
        - pkg/symbols becomes pkg/compiler/symbols

    * pkg/ast/... becomes pkg/compiler/legacy/ast/...

    * pkg/pack/ast becomes pkg/compiler/ast.

    * pkg/options goes away, merged back into pkg/compiler.

    * All binding functionality moves underneath a dedicated
      package, pkg/compiler/binder.  The legacy.go file contains
      cruft that will eventually go away, while the other files
      represent a halfway point between new and old, but are
      expected to stay roughly in the current shape.

    * All parsing functionality is moved underneath a new
      pkg/compiler/metadata namespace, and we adopt new terminology
      "metadata reading" since real parsing happens in the MetaMu
      compilers.  Hence, Parser has become metadata.Reader.

    * In general phases of the compiler no longer share access to
      the actual compiler.Compiler object.  Instead, shared state is
      moved to the core.Context object underneath pkg/compiler/core.

    * Dependency resolution during binding has been rewritten to
      the new model, including stashing bound package symbols in the
      context object, and detecting import cycles.

    * Compiler construction does not take a workspace object.  Instead,
      creation of a workspace is entirely hidden inside of the compiler's
      constructor logic.

    * There are three Compile* functions on the Compiler interface, to
      support different styles of invoking compilation: Compile() auto-
      detects a Mu package, based on the workspace; CompilePath(string)
      loads the target as a Mu package and compiles it, regardless of
      the workspace settings; and, CompilePackage(*pack.Package) will
      compile a pre-loaded package AST, again regardless of workspace.

    * Delete the _fe, _sema, and parsetree phases.  They are no longer
      relevant and the functionality is largely subsumed by the above.

...and so very much more.  I'm surprised I ever got this to compile again!
2017-01-18 12:18:37 -08:00
joeduffy 0706472a1c Split pkg/ast; merge symbol code into pkg/symbols
This change helps move us one step closer to eliminating the old metadata-
based AST goo, and replacing it with MuPack/MuIL AST and symbol information.
In particular, all name/token "symbol" code -- things like identifiers,
package/member references, and version specs -- move out of the pkg/ast
package and into the top-level pkg/symbols package, alongside the existing
MuPack/MuIL symbol token types.
2017-01-17 17:41:28 -08:00
joeduffy 01658d04bb Begin merging MuPackage/MuIL into the compiler
This is the first change of many to merge the MuPack/MuIL formats
into the heart of the "compiler".

In fact, the entire meaning of the compiler has changed, from
something that took metadata and produced CloudFormation, into
something that takes MuPack/MuIL as input, and produces a MuGL
graph as output.  Although this process is distinctly different,
there are several aspects we can reuse, like workspace management,
dependency resolution, and some amount of name binding and symbol
resolution, just as a few examples.

An overview of the compilation process is available as a comment
inside of the compiler.Compile function, although it is currently
unimplemented.

The relationship between Workspace and Compiler has been semi-
inverted, such that all Compiler instances require a Workspace
object.  This is more natural anyway and moves some of the detection
logic "outside" of the Compiler.  Similarly, Options has moved to
a top-level package, so that Workspace and Compiler may share
access to it without causing package import cycles.

Finally, all that templating crap is gone.  This alone is cause
for mass celebration!
2017-01-17 17:04:15 -08:00
joeduffy 4cf6be0f07 Add some property binding tests
This change adds a handful of property binding tests.

It also fixes:

* AsName should assert IsName.

* Enumerate properties stably, so that it is deterministic.

* Do not issue errors about unrecognized properties for the special
  `mu/extension` type.  It's entire purpose in life is to offer an
  entirely custom set of properties, which the provider is meant to
  validate.

* Default to an empty map if properties are missing.

* Add a "/" to the end of the namespace from the workspace, if present.

And rearranges some code:

* Rename the LiteralX types to XLiteral; e.g., StringLiteral instead of
  LiteralString.  I kept typing XLiteral erroneously.

* Eliminate the Mu prefix on all of the predefined type and service
  functions and types.  It's superfluous and reads nicer this way.

* Swap the order of "expected" vs. "got" in the error message about
  incorrect property types.  It used to say "got %v, expected %v"; I
  personally find that it is more helpful if it says "expected %v,
  got %v".  YMMV.
2016-12-02 14:33:22 -08:00
joeduffy b3e1eab6d8 Allow workspaces to have namespaces
This change permits a workspace to specify a namespace, which is just a name
part that is trimmed off the front of directories when probing for inter-
workspace dependencies.  For example, if our namespace is aws/, normally we'd
need to organize our namespace into directories like:

        <root>
        |       aws/
        |       |       dynamodb/
        |       |       ec2/
        |       |       s3/
        ... and so on ...

If we instead specify a namespace

        namespace: aws

Then we can instead organize our project workspace as follows:

        <root>
        |       dynamodb/
        |       ec2/
        |       s3/
        ... and so on ...
2016-12-02 14:06:39 -08:00
joeduffy 9f2b737715 Clean up workspace file naming
This change makes workspace file naming a little more consistent with respect
to Mufile naming.  Instead of having a .mu/ directory, under which a workspace.yaml
and/or a stacks directory might exist, we now have a Muspace.yaml (or .json) file,
and a .Mudeps/ directory.  This has nicer symmetric with respect to Mu.yaml files.
2016-11-29 20:07:27 -08:00
joeduffy c1b5239667 Detect target cloud earlier on
This change detects the target cloud earlier on in the compilation process.
Prior to this change, we didn't know this information until the backend code-generation.
Clearly we need to know this at least by then, however, templates can specialize on this
information, so we actually need it sooner.  This change moves it into the frontend part.

Note that to support this we now eliminate the ability to specify target clusters in
the Mufile alone.  That "feels" right to me anyway, since Mufiles are supposed to be
agnostic to their deployment environment, other than template specialization.  Instead,
this information can come from the CLI and/or the workspace settings file.
2016-11-29 13:42:39 -08:00
joeduffy c1c43c7907 Use /usr/local/mu as the default install path 2016-11-23 14:18:18 -08:00
joeduffy 2b385b2e20 Prepare for better diagnostics
This change adds a new Diagable interface from which you can obtain
a diagnostic's location information (Document and Location).  A new
At function replaces WithDocument, et al., and will be used soon to
permit all arbitrary AST nodes to report back their position.
2016-11-23 07:44:03 -08:00
joeduffy c84512510a Implement dependency versions
This change implements dependency versions, including semantic analysis, per the
checkin 83030685c3.

There's quite a bit in here but at a top-level this parses and validates dependency
references of the form

        [[proto://]base.url]namespace/.../name[@version]

and verifies that the components are correct, as well as binding them to symbols.

These references can appear in two places at the moment:

* Service types.
* Cluster dependencies.

As part of this change, a number of supporting changes have been made:

* Parse Workspaces using a full-blown parser, parser analysis, and semantic analysis.
  This allows us to share logic around the validation of common AST types.  This also
  moves some of the logic around loading workspace.yaml files back to the parser, where
  it can be unified with the way we load Mu.yaml files.

* New ast.Version and ast.VersionSpec types.  The former represents a precise version
  -- either a specific semantic version or a short or long Git SHA hash -- and the
  latter represents a range -- either a Version, "latest", or a semantic range.

* New ast.Ref and ast.RefParts types.  The former is an unparsed string that is
  thought to contain a Ref, while the latter is a validated Ref that has been parsed
  into its components (Proto, Base, Name, and Version).

* Added some type assertions to ensure certain structs implement certain interfaces,
  to speed up finding errors.  (And remove the coercions that zero-fill vtbl slots.)

* Be consistent about prefixing error types with Error or Warning.

* Organize the core compiler driver's logic into three methods, FE, sema, and BE.

* A bunch of tests for some of the above ...  more to come in an upcoming change.
2016-11-22 16:58:23 -08:00
joeduffy 5f3af891f7 Support Workspaces
This change adds support for Workspaces, a convenient way of sharing settings
among many Stacks, like default cluster targets, configuration settings, and the
like, which are not meant to be distributed as part of the Stack itself.

The following things are included in this checkin:

* At workspace initialization time, detect and parse the .mu/workspace.yaml
  file.  This is pretty rudimentary right now and contains just the default
  cluster targets.  The results are stored in a new ast.Workspace type.

* Rename "target" to "cluster".  This impacts many things, including ast.Target
  being changed to ast.Cluster, and all related fields, the command line --target
  being changed to --cluster, various internal helper functions, and so on.  This
  helps to reinforce the desired mental model.

* Eliminate the ast.Metadata type.  Instead, the metadata moves directly onto
  the Stack.  This reflects the decision to make Stacks "the thing" that is
  distributed, versioned, and is the granularity of dependency.

* During cluster targeting, add the workspace settings into the probing logic.
  We still search in the same order: CLI > Stack > Workspace.
2016-11-22 10:41:07 -08:00
joeduffy 4df1ec8c35 Support baseless and baseful package paths
This changes the probing logic for dependency resolution.  The old logic was
inconsistent between the various roots.  The new approach simply prefers locations
with a base URL component -- since they are more specific -- but will allow for
locations missing a base URL component.  This is convenient for developers managing
a workspace where needing to specify the base URL in the path is annoying and
slightly too "opinionated" for my taste (especially for migrating existing services).
2016-11-22 09:28:25 -08:00
joeduffy d100f77b9c Implement dependency resolution
This change includes logic to resolve dependencies declared by stacks.  The design
is described in https://github.com/marapongo/mu/blob/master/docs/deps.md.

In summary, each stack may declare dependencies, which are name/semver pairs.  A
new structure has been introduced, ast.Ref, to distinguish between ast.Names and
dependency names.  An ast.Ref includes a protocol, base part, and a name part (the
latter being an ast.Name); for example, in "https://hub.mu.com/mu/container/",
"https://" is the protocol, "hub.mu.com/" is the base, and "mu/container" is the
name.  This is used to resolve URL-like names to package manager-like artifacts.

The dependency resolution phase happens after parsing, but before semantic analysis.
This is because dependencies are "source-like" in that we must load and parse all
dependency metadata files.  We stick the full transitive closure of dependencies
into a map attached to the compiler to avoid loading dependencies multiple times.
Note that, although dependencies prohibit cycles, this forms a DAG, meaning multiple
inbound edges to a single stack may come from multiple places.

From there, we rely on ordinary visitation to deal with dependencies further.
This includes inserting symbol entries into the symbol table, mapping names to the
loaded stacks, during the first phase of binding so that they may be found
subsequently when typechecking during the second phase and beyond.
2016-11-21 11:19:25 -08:00
joeduffy 3e766c34c6 Create a Workspace abstraction
This change introduces a Workspace interface that can be used as a first
class object.  We will embellish this as we start binding to dependencies,
which requires us to search multiple paths.  This change also introduces a
workspace.InstallRoot() function to fetch the Mu install path.
2016-11-21 09:23:39 -08:00
joeduffy 9c1b72596c Write up a bit about Workspaces and Dependencies 2016-11-20 09:22:29 -08:00
joeduffy 47f7b0e609 Rearrange workspace logic
This change moves the workspace and Mufile detection logic out of the compiler
package and into the workspace one.

This also sketches out the overall workspace structure.  A workspace is "delimited"
by the presence of a .mu/ directory anywhere in the parent ancestry.  Inside of that
directory we have an optional .mu/clusters.yaml (or .json) file containing cluster
settings shared among the whole workspace.  We also have an optional .mu/stacks/
directory that contains dependencies used during package management.

The notion of a "global" workspace will also be present, which is essentially just
a .mu/ directory in your home, ~/.mu/, that has an equivalent structure, but can be
shared among all workspaces on the same machine.
2016-11-20 08:20:19 -08:00
joeduffy 1614258437 Declare a Mumodules constant
This is a placeholder for future use; .mu_modules will be our moral
equivalent to NPM/Yarn's node_modules directory.  I've chosen a dot
since for the most part developers can ignore its existence.
2016-11-16 10:01:46 -08:00
joeduffy 60a1f02666 Add more compiler tests
This change adds a few more compiler tests and rearranges some bits and pieces
that came up while doing so.  For example, we now issue warnings for incorrect
casing and/or extensions of the Mufile (and test these conditions).  As part of
doing that, it became clear the layering between the mu/compiler and mu/workspace
packages wasn't quite right, so some logic got moved around; additionally, the
separation of concerns between mu/workspace and mu/schema wasn't quite right, so
this has been fixed also (workspace just understands Mufile related things while
schema understands how to unmarshal the specific supported extensions).
2016-11-16 08:19:26 -08:00
joeduffy 79f5f312b8 Support .yml Mufile extensions
This change recognizes .yml in addition to the official .yaml extension,
since .yml is actually very commonly used.  In addition, while in here, I've
centralized more of the extensions logic so that it's more "data-driven"
and easier to manage down the road (one place to change rather than two).
2016-11-15 18:26:21 -08:00
joeduffy e75f06bb2b Sketch a mu build command and its scaffolding
This adds a bunch of general scaffolding and the beginning of a `build` command.

The general engineering scaffolding includes:

* Glide for dependency management.
* A Makefile that runs govet and golint during builds.
* Google's Glog library for logging.
* Cobra for command line functionality.

The Mu-specific scaffolding includes some packages:

* mu/pkg/diag: A package for compiler-like diagnostics.  It's fairly barebones
  at the moment, however we can embellish this over time.
* mu/pkg/errors: A package containing Mu's predefined set of errors.
* mu/pkg/workspace: A package containing workspace-related convenience helpers.

in addition to a main entrypoint that simply wires up and invokes the CLI.  From
there, the mu/cmd package takes over, with the Cobra-defined CLI commands.

Finally, the mu/pkg/compiler package actually implements the compiler behavior.
Or, it will.  For now, it simply parses a JSON or YAML Mufile into the core
mu/pkg/api types, and prints out the result.
2016-11-15 14:30:34 -05:00