This change helps move us one step closer to eliminating the old metadata-
based AST goo, and replacing it with MuPack/MuIL AST and symbol information.
In particular, all name/token "symbol" code -- things like identifiers,
package/member references, and version specs -- move out of the pkg/ast
package and into the top-level pkg/symbols package, alongside the existing
MuPack/MuIL symbol token types.
This is the first change of many to merge the MuPack/MuIL formats
into the heart of the "compiler".
In fact, the entire meaning of the compiler has changed, from
something that took metadata and produced CloudFormation, into
something that takes MuPack/MuIL as input, and produces a MuGL
graph as output. Although this process is distinctly different,
there are several aspects we can reuse, like workspace management,
dependency resolution, and some amount of name binding and symbol
resolution, just as a few examples.
An overview of the compilation process is available as a comment
inside of the compiler.Compile function, although it is currently
unimplemented.
The relationship between Workspace and Compiler has been semi-
inverted, such that all Compiler instances require a Workspace
object. This is more natural anyway and moves some of the detection
logic "outside" of the Compiler. Similarly, Options has moved to
a top-level package, so that Workspace and Compiler may share
access to it without causing package import cycles.
Finally, all that templating crap is gone. This alone is cause
for mass celebration!
This change tracks the set of imported modules in the ast.Module
structure. Although we can in principle gather up all imports simply
by looking through the fully qualified names, that's slightly hokey;
and furthermore, to properly initialize all modules, we need to know
in which order to do it (in case there are dependencies). I briefly
considered leaving it up to MetaMu compilers to inject the module
initialization calls explicitly -- for infinite flexibility and perhaps
greater compatibility with the source languages -- however, I'd much
prefer that all Mu code use a consistent module initialization story.
Therefore, MetaMus declare the module imports, in order, and we will
evaluate the initializers accordingly.
This change makes considerable progress on the `mu describe` command;
the only thing remaining to be implemented now is full IL printing. It
now prints the full package/module structure.
For example, to print the set of exports from our scenarios/point test:
$ mujs tools/mujs/tests/output/scenarios/point/ | mu describe - -e
package "scenarios/point" {
dependencies []
module "index" {
class "Point" [public] {
method "add": (other: any): any
property "x" [public, readonly]: number
property "y" [public, readonly]: number
method ".ctor": (x: number, y: number): any
}
}
}
This is just a pretty-printed, but is coming in handy with debugging.
The NewExpression AST node type was missing a JSON annotation on
its Type field, leading to decoding errors.
Now, with this, the full suite of MuJS test cases can be unmarshaled
into fully populated MuPack and MuIL structures.
This change overhauls the approach to custom decoding. Instead of decoding
the parts of the struct that are "trivial" in one pass, and then patching up
the structure afterwards with custom decoding, the decoder itself understands
the notion of custom decoder functions.
First, the general purpose logic has moved out of pkg/pack/encoding and into
a new package, pkg/util/mapper. Most functions are now members of a new top-
level type, Mapper, which may be initialized with custom decoders. This
is a map from target type to a function that can decode objects into it.
Second, the AST-specific decoding logic is rewritten to use it. All AST nodes
are now supported, including definitions, statements, and expressions. The
overall approach here is to simply define a custom decoder for any interface
type that will occur in a node field position. The mapper, upon encountering
such a type, will consult the custom decoder map; if a decoder is found, it
will be used, otherwise an error results. This decoder then needs to switch
on the type discriminated kind field that is present in the metadata, creating
a concrete struct of the right type, and then converting it to the desired
interface type. Note that, subtly, interface types used only for "marker"
purposes don't require any custom decoding, because they do not appear in
field positions and therefore won't be encountered during the decoding process.
This change splits up the decoding logic into multiple files, to
mirror the AST package structure that the functions correspond to.
Additionally, there is now less "loose" reflection and dynamic lookup
code scattered throughout; it is now consolidated into the decoder,
with a set of "generic" functions like `fieldObject`, `asString`, etc.
This change implements custom class member decoding. As with module methods,
the function body AST nodes remain nil, as custom AST decoding isn't yet done.
This change begins to implement some of the AST custom decoding, beneath
the Package's Module map. In particular, we now unmarshal "one level"
beyond this, populating each Module's ModuleMember map. This includes
Classes, Exports, ModuleProperties, and ModuleMethods. The Class AST's
Members have been marked "custom", in addition to Block's Statements,
because they required kind-directed decoding. But Exports and
ModuleProperties can be decoded entirely using the tag-directed decoding
scheme. Up next, custom decoding of ClassMembers. At that point, all
definition-level decoding will be done, leaving MuIL's ASTs.
This change just moves the assertion/failure functions from the pkg/util
package to pkg/util/contract, so things read a bit nicer (i.e.,
`contract.Assert(x)` versus `util.Assert(x)`).
This fixes a few things so that MuPackages now unmarshal:
* Mark Module.Members as requiring "custom" decoding. This is required
because that's the first point in the tree that leverages polymorphism.
Everything "above" this unmarshals just fine (e.g., package+modules).
* As such, stop marking Package.Modules as "custom".
* Add the Kind field to the Node. Although we won't use this for type
discrimination in the same way, since Go gives us RTTI on the structs,
it is required for unmarshaling (to avoid "unrecognized fields" errors)
and it's probably handy to have around for logging, messages, etc.
* Mark Position.Line and Column with "json" annotations so that they
unmarshal correctly.
This change adjusts pointers correctly when unmarshaling into target
pointer types. This handles arrays and maps of pointer elements, in
addition to consolidating existing logic for marshaling into a
destination top-level pointer as well.
This change eliminates boilerplate decoding logic in all the different
data structures, and instead uses a new tag-directed decoding scheme.
This works a lot like the JSON deserializers, in that it recognizes the
`json:"name"` tags, except that we permit annotation of fields that
require custom deserialization, as `json:"name,custom"`. The existing
`json:"name,omitempty"` tag is recognized for optional fields.
This adds basic custom decoding for the MuPack metadata section of
the incoming JSON/YAML. Because of the type discriminated union nature
of the incoming payload, we cannot rely on the simple built-in JSON/YAML
unmarshaling behavior. Note that for the metadata section -- what is
in this checkin -- we could have, but the IL AST nodes are problematic.
(To know what kind of structure to creat requires inspecting the "kind"
field of the IL.) We will use a reflection-driven walk of the target
structure plus a weakly typed deserialized map[string]interface{}, as
is fairly customary in Go for scenarios like this (though good libaries
seem to be lacking in this area...).
This change carries over all of the metadata shapes in the MuPack
and MuIL file formats to our Go toolset. This includes creating a
proper discriminated AST type tree along with correct annotations
so that the metadata will serialize and deserialize correctly.
* Persue the default/optional checking if a property value == nil.
* Use the Interface() function to convert a reflect.Type to its underlying
interface{} value. This is required for typechecking to check out.
* Also, unrelated to the above, change type assertions to use nil rather than
allocating real objects. Although minimal, this incurs less GC pressure.
In some cases, we want to specialize template generation based on
the options passed to the compiler. This change flows them through
so that they can be accessed as
{{if .Options.SomeSetting}}
...
{{end}}
This change reverts the syntax for arrays back to T[] from []T. The main
reason is that YAML doesn't permit unquoted strings beginning with [], meaning
any array type needs to be quoted as in "[]T", which is annoying compared to all
other primitive types which don't require quotes. And, anyway, this syntax is
more familiar too.
I've also added a number of tests.
Any of the bindXValue routines can fail if there was no way to convert
the interface{} to an ast.Literal. In such a case, we need to issue an
error about the wrong type being passed. Unfortunately, in the most
recent set of changes, we began simply returning nils without issuing
the error. This change fixes that.
This change renames Schemas to Types on Stack. More interestingly, it
renames the JSON/YAML property used to specify them, from "schemas:" to
"types:"; I feel like this reads more naturally, especially as a sister
to the existing "services:" section.
This checkin continues progress on marapongo/mu#9. It's still not
complete, however we're getting there. In particular, this includes:
* Rename of ComplexLiteral to SchemaLiteral, as it is used exclusively
for schematized types. Also includes a set of changes associated
with this, like deep value conversion to `map[string]interface{}`.
* Binding of schema types included within a Stack. This allows names in
type references to be bound to those schema types during typechecking.
This also includes binding schema properties, reusing all the existing
property binding logic for stacks. In this way, properties between
stacks and custom schema types are one and the same, which is nice.
* Enforcement for custom schema constraints; this includes Pattern,
MaxLength, MinLength, Maximum, and Minimum, as per the JSON Schema
specification.
This change overhauls the core of how types are used by the entire
compiler. In particular, we now have an ast.Type, and have begun
using its use where appropriate. An ast.Type is a union representing
precisely one of the possible sources of types in the system:
* Primitive type: any, bool, number, string, or service.
* Stack type: a resolved reference to an actual concrete stack.
* Schema type: a resolved reference to an actual concrete schema.
* Unresolved reference: a textual reference that hasn't yet been
resolved to a concrete artifact.
* Uninstantiated reference: a reference that has been resolved to
an uninstantiated stack, but hasn't been bound to a concrete
result yet. Right now, this can point to a stack, however
eventually we would imagine this supporting inter-stack schema
references also.
* Decorated type: either an array or a map; in the array case, there
is a single inner element type; in the map case, there are two,
the keys and values; in all cases, the type recurses to any of the
possibilities listed here.
All of the relevant AST nodes have been overhauled accordingly.
In addition to this, we now have an ast.Schema type. It is loosely
modeled on JSON Schema in its capabilities (http://json-schema.org/).
Although we parse and perform some visitation and binding of these,
there are mostly placeholders left in the code for the interesting
aspects, such as registering symbols, resolving dependencies, and
typechecking usage of schema types.
This is part of the ongoing work behind marapongo/mu#9.
This change leverages intrinsics in place of the predefined types.
It remains to be seen if we can reach 100% on this, however I am hopeful.
It's also nice that the system will be built "out of itself" with this
approach; in other words, each of the types is simply a Mufile that can
use conditional targeting as appropriate for the given cloud providers.
If we find that this isn't enough, we can always bring back the concept.
This change stops using the short-hand "!Ref" YAML syntax. The Golang
marshaler encodes it with quotes and, apparently, has no way to suppress
this behavior; this isn't surprising, since the YAML parser we're using
admits it doesn't support this aspect of the YAML spec fully. But that's
okay, the long-hand syntax works just fine, and has the added benefit
that we don't need to special case the logig for JSON versus YAML.
I think things have gotten a little out of hand with the way mu/x/cf
auto-maps properties. In the beginning, it looked like everything could
be trivially auto-mapped, and I wanted to avoid the verbosity of mapping
each property by hand (since you can easily fat finger a name, mess up
capitalization, forget one, etc). But then we began mapping service
references using proper CloudFormation !Refs, which meant suppressing
some of the auto-mappings, etc., etc. This led to properties, extraProperties,
skipProperties, renamedProperties, and so on... Pretty confusing IMHO.
I just took a step back and decided to eliminate auto-mapping. Instead,
you get two options: properties just lists a set of property name mappings,
and extraProperties lets you do template magic to map thing instead if you
wish to take matters into your own hands. The result isn't too verbose
and has a lot less magic going on so it's easier to understand.
This change eliminates the special type mu/extension in favor of extensible
intrinsic types. This subsumes the previous functionality while also fixing
a number of warts with the old model.
In particular, the old mu/extension approach deferred property binding until
very late in the compiler. In fact, too late. The backend provider for an
extension simply received an untyped bag of stuff, which it then had to
deal with. Unfortunately, some operations in the binder are inaccessible
at this point because doing so would cause a cycle. Furthermore, some
pertinent information is gone at this point, like the scopes and symtables.
The canonical example where we need this is binding services names to the
services themselves; e.g., the AWS CloudFormation "DependsOn" property should
resolve to the actual service names, not the string values. In the limit,
this requires full binding information.
There were a few solutions I considered, including ones that've required
less code motion, however this one feels the most elegant.
Now we permit types to be marked as "intrinsic." Binding to these names
is done exactly as ordinary name binding, unlike the special mu/extension
provider name. In fact, just about everything except code-generation for
these types is the same as ordinary types. This is perfect for the use case
at hand, which is binding properties.
After this change, for example, "DependsOn" is expanded to real service
names precisely as we need.
As part of this change, I added support for three new basic schema types:
* ast.StringList ("string[]"): a list of strings.
* ast.StringMap ("map[string]any"): a map of strings to anys.
* ast.ServiceList ("service[]"): a list of service references.
Obviously we need to revisit this and add a more complete set. This work
is already tracked by marapongo/mu#9.
At the end of the day, it's likely I will replace all hard-coded predefined
types with intrinsic types, for similar reasons to the above.
