This change looks up the main module from a package, and that module's
entrypoint, when performing evaluation. In any case, the arguments are
validated and bound to the resulting function's parameters.
This change eliminates all traces of the old, legacy errors and warnings.
It also renumbers everything so that we don't have any "gaps". Finally,
it introduces a factory method for new errors and warnings, private to
this package, that ensures we don't accidentally register duplicate IDs.
This change deletes a bunch of old legacy code. I've tagged the
prior changelist as "0.1" in case we need to go back and recover
some of the code (as I expect some of the constraint type logic
and AWS-specific code to come in handy down the road).
🔥🔥🔥
This adds support for LoadLocationExpression, which will be key to
getting virtually everything else working, including class and module
member accesses, local variable gets/sets, function calls, and more.
In particular, an extra level of indirection is added to the globals,
locals, and properties maps. This new thing is called a *Reference,
and it is simply a mutable slot that contains an *Object. Instead
of storing and retrieving objects directly, this extra level of
indirection enables us to create *Object instances that themselves
simply refer to other objects (for subsequent loads and stores).
For now, we do not permit overwriting of method properties. This
is a future work item (see marapongo/mu#56).
This change adds support for function calls. We do not yet wire it
into the various function call expressions, however, that should be
relatively trivial after this change. (It is dependent on figuring
out the runtime representation of load location objects and subsequent
uses of them). This change also updates the scoping logic to respect
"activation frame" style of lexical scoping, where an inner function
as a result of a call should not have access to the callee context.
This change also includes logic to detect unhandled exceptions and to
print an error as a result. Eventually, we will want stack traces here.
This change eliminates the scope-based symbol table. Because we now
require that all module, type, function, and variable elements are
encoded as fully qualified tokens, there is no need for the scope-based
lookups. Instead, the languages themselves decide how the names bind
to locations and just encode that information directly.
The scope is still required for local variables, however, since those
don't have a well-defined "fixed" notion of name. This is also how
we will ensure the evaluator stores values correctly -- including
discarding them -- in a lexically scoped manner.
This change checks in an enormously rudimentary interpreter. There is
a lot left to do, as evidenced by the countless TODOs scattered throughout
pkg/compiler/eval/eval.go. Nevertheless, the scaffolding and some important
pieces are included in this change.
In particular, to evaluate a package, we must locate its entrypoint and then,
using the results of binding, systematically walk the full AST. As we do
so, we will assert that aspects of the AST match the expected shape,
including symbols and their types, and produce value objects for expressions.
An *unwind structure is used for returns, throws, breaks, and continues (both
labeled and unlabeled). Each statement or expression visitation may optionally
return one and its presence indicates that control flow transfer is occurring.
The visitation logic then needs to propagate this; usually just by bailing out
of the current context immediately, but sometimes -- as is the case with
TryCatchBlock statements, for instance -- the unwind is manipulated in more
"interesting" ways.
An *Object structure is used for expressions yielding values. This is a
"runtime object" in our system and is comprised of three things: 1) a Type
(essentially its v-table), 2) an optional data pointer (for tagged primitives),
and 3) an optional bag of properties (for complex object property values).
I am still on the fence about whether to unify the data representations.
The hokiest aspect of this change is the scoping and value management. I am
trying to avoid needing to implement any sort of "garbage collection", which
means our bag-of-values approach will not work. Instead, we will need to
arrange for scopes to be erected and discarded in the correct order during
evaluation. I will probably tackle that next, along with fleshing out the
many missing statement and expression cases (...and tests, of course).
This change introduces a binder.Context structure, with a
*core.Context embedded, that carries additional semantic information
forward to future passes in the compiler. In particular, this is how
the evaluation/interpretation phase will gain access to types, scopes,
and symbols.
This change memoizes decorated types to avoid creating boatloads
of redundant symbols at runtime. We want it to be the case that you
can create instances of decorated types as needed throughout the
compiler without worry for excess garbage (e.g., arrays, pointers, etc).
During evaluation, we need to perform very delicate walking of the AST,
and are likely to use the ability to substitute visitors "in situ." As
a result, I'd like to make anonymous visitors easier to produce. This
change permits both single function forms of Visit/After, in addition to
anonymous structures that pair up a Visit and an After, to produce a Visitor.
This change completes my testing of decorator parsing for now. It tests the token
`*[]map[string]map[()*(bool,string,test/package:test/module/Crazy)number][][]test/package:test/module/Crazy`.
This turned up some bugs, most notably in the way we returned the "full" token for
the parsed types. We need to extract the subset of the token consumed by the parsing
routine, rather than the entire thing. To do this, we introduce a tokenBuffer type
that allows for convenient parsing of tokens (eating, advancing, extraction, etc).
This isn't comprehensive yet, however it caught two bugs:
1. parseNextType should operate on "rest" in most cases, not "tok".
2. We must eat the "]" map separator before moving on to the element type.
Part of the token grammar permits so-called "decorated" types. These
are tokens that are pointer, array, map, or function types. For example:
* `*any`: a pointer to anything.
* `[]string`: an array of primitive strings.
* `map[string]number`: a map from strings to numbers.
* `(string,string)bool`: a function with two string parameters and a
boolean return type.
* `[]aws:s3/Bucket`: an array of objects whose class is `Bucket` from
the package `aws` and its module `s3`.
This change introduces this notion into the parsing and handling of
type tokens. In particular, it uses recursive parsing to handle complex
nested structures, and the binder.bindTypeToken routine has been updated
to call out to these as needed, in order to produce the correct symbol.
This changes a few things around binding logic, as part of eliminating
all of the legacy logic and weaving it into the new codebase:
* Give Scopes access to the Context object. Related, add a TryRegister
method to Scope that is like RequireRegister, except that instead of
fail-fast upon encountering a duplicate entry, it will issue an error.
* Move all typecheck visitation functions out of the big honkin' switch
and into their own member functions. As this stuff gets more complex,
having everything in one routine was starting to irk my sensibilities.
* Validate that packages have names.
* Store both the package symbol, plus the canonicalized URL used to
resolve it, in the package map. This will help us verify that versions
match for multiple package references resolving to the same symbol.
* Add nice inquiry methods to the Class symbol (Sealed, Abstract, Record,
Interface) that simplify accessing the modifiers on the underlying node.
The options structure will be shared between multiple passes of
compilation, including evaluation and graph generation. Therefore,
it must not be in the pkg/compile package, else we would create
package cycles. Now that the options structure is barebones --
and, in particular, no more "backend" settings pollute it -- this
refactoring actually works.
This is a bit of paranoia, however, an invariant of the typechecker
is that, after the final pass, all expression nodes are assigned a
type. This assertion checks that this is true.
Instead of serializing simple token strings into the AST -- in place of things
like type references, module references, export references, etc. -- we now use
1st class AST nodes. This ensures that source context flows with the tokens
as we bind them, etc., and also cleans up a few inconsistencies (like using an
ast.Identifier for NewExpression -- clearly wrong since this the resulting
MuIL is meant to contain fully bound semantic references).
This change includes some tests for token parsing and conversions. It
also fixes a bug where we treated Type tokens like ClassMembers, when
we ought to have been treating them like ModuleMembers.
This change performs typechecking during binding. This is less about
typechecking per se -- since higher level languages will have presumably
given us well-typed IL -- and more about preparing the AST so that we
can evaluate the fully bound nodes to produce a MuGL graph. It also
serves as a "verifier" for the incoming MuIL, however.
This is clearly incomplete, as the dozens of TODOs will make obvious.
But it's a clean checkpoint that does enough interesting typechecking
that I am landing it now.
This change begins to bind function bodies. This must be done as a
second pass over the AST, because dependencies between modules, and
even intra-module dependencies, might refer to top-level symbols like
types, variables, and functions, and so must be established first.
At the moment, the only node kind we handle is ast.Block, which
merely pushes and pops lexical scopes; however, the next step is to
implement the AST node-specific visitation logic for all statement
and expression nodes.
I've also rearranged how Scopes work to be a little easier to use.
The Scope type now remembers the **Scope slot in which it is rooted,
so that we can simply call Push and Pop on Scopes and have the right
thing happen.
This introduces symbol factory methods to make creating them
less error prone. In particular, we hadn't been wiring up parents
properly (since they came in after the initial symbol shape).
Now with the factory methods, we'll be reforced to visit creation
sites whenever adding new required elements to symbol types.
This change rearranges the old way we dealt with URLs. In the old system,
virtually every reference to an element, including types, was fully qualified
with a possible URL-like reference. (The old pkg/tokens/Ref type.) In the
new model, only dependency references are URL-like. All maps and references
within the MuPack/MuIL format are token and name based, using the new
pkg/tokens/Token and pkg/tokens/Name family of related types.
As such, this change renames Ref to PackageURLString, and RefParts to
PackageURL. (The convenient name is given to the thing with "more" structure,
since we prefer to deal with structured types and not strings.) It moves
out of the pkg/tokens package and into pkg/pack, since it is exclusively
there to support package resolution. Similarly, the Version, VersionSpec,
and related types move out of pkg/tokens and into pkg/pack.
This change cleans up the various binder, package, and workspace logic.
Most of these changes are a natural fallout of this overall restructuring,
although in a few places we remained sloppy about the difference between
Token, Name, and URL. Now the type system supports these distinctions and
forces us to be more methodical about any conversions that take place.
This rearranges the library code:
* sdk/... goes away.
* What used to be sdk/javascript/ is now lib/mu/, an actual MuPackage
that provides the base abstractions for all other MuPackages to use.
* lib/aws is the @mu/aws MuPackage that exposes all AWS resources.
* lib/mux is the @mu/x MuPackage that provides cross-cloud abstractions.
A lot of what used to be in lib/mu goes here. In particular, autoscaler,
func, ..., all the "general purpose" abstractions, really.
In the old system, the core runtime/toolset understood that we are targeting
specific cloud providers at a very deep level. In fact, the whole code-generation
phase of the compiler was based on it.
In the new system, this difference is less of a "special" concern, and more of
a general one of mapping MuIL objects to resource providers, and letting *them*
gather up any configuration they need in a more general purpose way.
Therefore, most of this stuff can go. I've merged in a small amount of it to
the mu/x MuPackage, since that has to switch on cloud IaaS and CaaS providers in
order to decide what kind of resources to provision. For example, it has a
mu.x.Cluster stack type that itself provisions a lot of the barebone essential
resources, like a virtual private cloud and its associated networking components.
I suspect *some* knowledge of this will surface again as we implement more
runtime presence (discovery, etc). But for the time being, it's a distraction
getting the core model running. I've retained some of the old AWS code in the
new pkg/resource/providers/aws package, in case I want to reuse some of it when
implementing our first AWS resource providers. (Although we won't be using
CloudFormation, some of the name generation code might be useful.) So, the
ships aren't completely burned to the ground, but they are certainly on 🔥.
As I do local development, I noticed errant newlines in the error
messages coming from TypeScript. That's because its formatting appends
newlines, whereas ours does not (and requires the code printing the
diagnostic to add one). To make these uniform, we will strip newlines,
if they exist, from the TypeScript preformatted diagnostics.
