This change splits up the decoding logic into multiple files, to
mirror the AST package structure that the functions correspond to.
Additionally, there is now less "loose" reflection and dynamic lookup
code scattered throughout; it is now consolidated into the decoder,
with a set of "generic" functions like `fieldObject`, `asString`, etc.
This change implements custom class member decoding. As with module methods,
the function body AST nodes remain nil, as custom AST decoding isn't yet done.
This change begins to implement some of the AST custom decoding, beneath
the Package's Module map. In particular, we now unmarshal "one level"
beyond this, populating each Module's ModuleMember map. This includes
Classes, Exports, ModuleProperties, and ModuleMethods. The Class AST's
Members have been marked "custom", in addition to Block's Statements,
because they required kind-directed decoding. But Exports and
ModuleProperties can be decoded entirely using the tag-directed decoding
scheme. Up next, custom decoding of ClassMembers. At that point, all
definition-level decoding will be done, leaving MuIL's ASTs.
This change just moves the assertion/failure functions from the pkg/util
package to pkg/util/contract, so things read a bit nicer (i.e.,
`contract.Assert(x)` versus `util.Assert(x)`).
This fixes a few things so that MuPackages now unmarshal:
* Mark Module.Members as requiring "custom" decoding. This is required
because that's the first point in the tree that leverages polymorphism.
Everything "above" this unmarshals just fine (e.g., package+modules).
* As such, stop marking Package.Modules as "custom".
* Add the Kind field to the Node. Although we won't use this for type
discrimination in the same way, since Go gives us RTTI on the structs,
it is required for unmarshaling (to avoid "unrecognized fields" errors)
and it's probably handy to have around for logging, messages, etc.
* Mark Position.Line and Column with "json" annotations so that they
unmarshal correctly.
This change adjusts pointers correctly when unmarshaling into target
pointer types. This handles arrays and maps of pointer elements, in
addition to consolidating existing logic for marshaling into a
destination top-level pointer as well.
This change eliminates boilerplate decoding logic in all the different
data structures, and instead uses a new tag-directed decoding scheme.
This works a lot like the JSON deserializers, in that it recognizes the
`json:"name"` tags, except that we permit annotation of fields that
require custom deserialization, as `json:"name,custom"`. The existing
`json:"name,omitempty"` tag is recognized for optional fields.
This adds basic custom decoding for the MuPack metadata section of
the incoming JSON/YAML. Because of the type discriminated union nature
of the incoming payload, we cannot rely on the simple built-in JSON/YAML
unmarshaling behavior. Note that for the metadata section -- what is
in this checkin -- we could have, but the IL AST nodes are problematic.
(To know what kind of structure to creat requires inspecting the "kind"
field of the IL.) We will use a reflection-driven walk of the target
structure plus a weakly typed deserialized map[string]interface{}, as
is fairly customary in Go for scenarios like this (though good libaries
seem to be lacking in this area...).
This command will simply pretty-print the contents of a MuPackage.
My plan is to use it for my own development and debugging purposes,
however, I suspect it will be generally useful (since MuIL can be
quite verbose). Again, just scaffolding, but I'll flesh it out
incrementally as more of the goo in here starts working...
This change carries over all of the metadata shapes in the MuPack
and MuIL file formats to our Go toolset. This includes creating a
proper discriminated AST type tree along with correct annotations
so that the metadata will serialize and deserialize correctly.
This change removes some ambiguity that existed in MuIL tokens given the
simplistic naming scheme of only using "/" as a delimiter. We could have
resolved this ambiguity through successive name bindings, but I prefer a
more deterministic approach. A module member token now delimits the
package/module name from the member name by a ":" character, as in
"aws/ec2:VM". Furthermore, when reexporting elements from the current
package/module, we will use the self-referential "." character, as in
".:VM".
This change skips emitting EmptyStatements into module initializers.
This serves no purpose and can in fact lead to us creating module
initializers that are nothing but a set of EmptyStatements, slowing
down initialization of them with no purpose.
This completes support for module imports and exports.
Many forms of exports are supported, and are all now tested, for example:
// Export individual declarations as we go:
export class C {}
export interface I {}
export let x = 42;
// Export individual declarations explicitly:
class D {}
interface J {}
let w = 42;
export {D, J, w};
// Rename exports:
export {D as E, J as K, w as x};
// Re-export members from other modules:
export {A, B, c} from "other";
// Rename re-exports:
export {A as M, B as N, c as o} from "other";
// Re-export entire module as a submodule:
import * as other from "other";
export {other};
This implements renaming exports during reexporting, by recognizing the
propertyName Identifier and using it accordingly in the member map and
corresponding ast.Export object. There is also a new test for this.
This fixes a bug where we didn't consider reexports (ast.Export nodes)
definitions, and thus stuck them into the statements section of a module
initializer rather than the definitions section.
It also adds test cases for reexporting, both `export {a,b,c} from "module"`
and `export * from "module"` kinds.
This change adds true understanding and dependency resolution for
modules. This requires consulting the bound node tree and TypeChecker
object associated with the TypeScript program AST. This introduces a
so-called ModuleReference, which is just a wrapper around a TypeScript
import string, enabling us to continuously look up information about
a module's symbol (from the SourceFile map). It only lowers to a MuIL
ModuleToken for serialization at which point all information required
to load that dependency is expected to have been pre-arranged according
to our package manager resolution process (see deps.md).
As part of this, we can implement the `export * from "module"` export
form.
This is a huge step forward for marapongo/mu#46.
This change first and foremost adds a MuIL export node (ast.Export);
this is for re-exporting members of other modules, potentially under
a different name.
This works by simply by associating a name with a fully qualified token
name that, when binding, can be re-resolved to the real target. This
should be sufficient for expressing everything we need from MuJS.
In addition to that, there is an initial cut at basic re-exports; this
is the form where you would say:
export { some, nifty as cool, stuff } from "module";
In this example, we end up with two Export nodes: one maps "some" to
"some/module", "cool" to "some/nifty", and "stuff" to "some/stuff".
Note that we have not yet implemented two other variants.
First, we will want to support the "export all" variety, e.g.
export * from "module";
Instead of supporting wildcards in MuIL -- which would result in the
odd attribute that you couldn't know the totality of a module's exports
without also examining its dependencies (possibly recursively) -- we
will require the compiler to map the "*" to a concrete list of Export nodes.
Second, we also want to support named exports from the current module.
That is to say, export clauses without `from "module"` parts. E.g.
let some = ...;
let nifty = ...;
let stuff = ...;
export { some, nifty as cool, stuff };
This is less important because you can always export members using the
more familiar `export <decl>` syntax (which is already recognized by MuJS).
But of course, we will want to support it eventually. It is slightly more
complex, because we will need to use static typing to resolve the exported
members to their types -- e.g., variable, class, interface -- and in fact
this could lead us to re-exporting whole modules in their entirety.
Both of these two will require marapongo/mu#46. Since so much is piling up
behind this, I will likely begin biting this off soon. I also believe that
we may want re-export nodes, Export, to carry a specifier that indicates whether
the target is a class, variable, or sub-module.
This change does a rudimentary mapping of the delete operator. I say
"rudimentary" because it's not entirely correct, ECMAScript-wise: delete
actually removes a property entirely from an object, whereas this change
simply transforms `delete <expr>` into `<expr> = null`. Deciding what to
do with `delete` will be handled when we tackle the overall "unsupported
operators" bug, marapongo/mu#50, later on. For `delete` specifically,
it's not clear we will want to allow deletion of properties on strongly
typed instances, however for untyped objects we can probably do this, and
then detect errors when downcasting to types requiring those properties.
This rearranges the various "NYI" spots in the code to defer to a
notYetImplemented helper function. This ensures we have decent
diagnostics associated with failures, like source line numbers.
This change maps the ECMAScript operators ===, !==, >>>, and >>>= to
the MuIL operators ==, !=, >>, and >>=, respectively. This isn't
entirely correct, however, we will most likely emulate these using
runtime helpers down the line; marapongo/mu#50 tracks this issue.
TypeScript type aliases simply turn into MuIL classes with the base
class being the aliased type. In theory, MuIL conversions will support
the kind of conversions offered by aliases (or at least get sufficiently
close). We will need to tackle this in part by marapongo/mu#46.
In addition, while I was in there, I cleaned up the type token emission
to emit "any" (rather than "TODO") and to use a single, central function
for all ts.TypeNodes. This will also get cleaned up by marapongo/mu#46.
At the moment, we do not emit bound node information yet. (See
marapongo/mu#46). When that day comes we will need to decide whether
import declarations are needed or, as I suspect, they can continue to
be discarded safely because all dependency information is available
on the bound nodes themselves. I'm leaving in a TODO in there just as
a reminder to resolve this at that time.
This change maps MuJS interface types to MuIL, including property and
method declarations. The mapping is fairly straightforward, and similar
to the existing class lowering, except for some minor AST differences in
the TypeScript representations. We emit interface types as classes with
the interface bit set, and all method declarations being abstract.
This change improves the line-by-line error reports when a test fails,
both in the diagnostics messages and AST tree baseline comparisons.
Previously, the test would fail at the first occurrence of a mismatch,
often with little information about the context. In this change, the
test will now fail with a full dump of the comparison.
For instance, here is the before for a test failure in a new case I'm adding:
AssertionError: Expected tree line #1 to match
+ expected - actual
- "name": "iface_1",
+ "name": "class_1",
In comparison, here is what the test failure looks like now:
AssertionError: Expected trees to match; 6 did not
+ expected - actual
-1: "name": "iface_1",
-10: "I": {
-14: "ident": "I",
-19: "column": 10
-23: "column": 11
-29: "interface": true,
+1: "name": "class_1",
+10: "C": {
+14: "ident": "C",
+19: "column": 6
+23: "column": 7
+29: "abstract": false,
This change adds a set of true diagnostics constructs, underneath the
new `diag` module.
This includes projecting Mu-specific errors as real diagnostics in a
way that is unified with TypeScript errors. The only difference, of
course, is that Mu errors tend to happen in later passes. But this is
not necessarily always the case.
As part of this, I've rearranged the compiler passes to present a
simpler interface to users of the compiler API (currently just the CLI
and test harness, but it's just a library, so anybody can use it).
Namely, there are three phases:
1. Compilation is the overall process of taking an input path
and driving the entire compilation process, yielding a set of
diagnostics and, ideally, a final MuPackage at the end of it.
2. Script compilation is the process of driving the front-end
compiler -- in this case TypeScript -- yielding a set of Mu
diagnostics and, if that went well, the script's AST.
3. Transformation is the process of taking the script output and
lowering it into the final MuPackage form.
Most people will deal with 1, blissfully unaware of the presence of
independent 2 and 3 phases. We choose to keep them distinct for
white box testing and for future scenarios we have yet to envision.
There were a few places where we didn't propagage source locations
correctly. This fixes that. Part of the issue was we didn't have
good helper methods to propagate "ranges" (e.g., a sequence of
statements or expressions), so this change adds those.
This change also establishes a new naming convention; the addLocation*
methods adds a newly allocated location based on translating a TypeScript
AST node, whereas the copyLocation* methods copy them directly from
other MuIL nodes. Thus, there are now four methods: addLocation,
addLocationRange, copyLocation, and copyLocationRange.
This change adds a new AST node kind to represent "new" expressions.
Originally I had thought we could simply represent these as calls to
a constructor, however, there is a sizeable semantic difference and so
it's better for this node kind to be distinct; for example, we will
most likely permit allocation of objects without explicit constructors.
In addition, I've added a "complex" scenario test, which is just a
Point class. It has a constructor, two properties, and an instance
method that performs arithmetic and allocates a new instance. It works!
This uses explicit type coercions when creating interface types with
object literals. This helps the compiler catch some additional errors,
such as mistyped property names (of which we had two instances, for
load location expressions ("key" vs "name")).
The old way we shared code between VariableLike and FunctionLike cases --
across module, class, and locals -- was slightly confusing, and required a
lot of dynamic object property copying, etc. This change is a little bit
easier on the eyes, correctly handles variable initializers more uniformly,
and prepares us for class properties (the next change to land).
This change maps MuJS access expressions to the MuPack/MuIL forms.
As part of this, I've changed the representation in the AST. It doesn't
make much sense to distinguish between variable, member, and function access.
Instead, all accesses will be uniform, and will evaluate to the proper type.
