This change implements simple index-based CocoPy subscripts (and
not the more fully featured slicing ones).
Alongside this, we relax a binder-time check that all dynamic
access types must be strings. The eval code already handles
numeric (array) accesses, so we will permit these to flow through.
This change emits all CocoJS interfaces as records. This allows us to
safely construct instances of them from Python using anonymous properties,
essentially emulating object literals.
For example, given a CocoJs interface defined as such:
interface Foo {
x;
y?;
z;
}
we can easily construct fresh instances using normal JS literals:
let f = { x: 42, z: "bar" };
But, Python doesn't have the equivalent literal syntax, wedging us.
It's now possible to initialize an instance from CocoPy as follows:
let f = Foo(x=42, z="bar")
This leverages the notion of records and primary properties, as
described in our CocoPack/CocoIL design documents.
Due to Python's ... interesting ... scoping rules, we want to avoid
forcing block scopes in certain places. Instead, we will let arbitrary
statements take their place. Of course, this could be a block, but it
very well could be a multi-statement (essentially a block that doesn't
imply a new lexical scope), or anything, really.
* Move checking the validity of a derived class with no constructor
from evlauation to binding (verification). Furthermore, make it
a verification error instead of an assert, and make the checking
complete (i.e., don't just check for the existence of a base class,
also check for the existence of a ctor, recursively).
* Refactor the constructor accessor out of the evaluator and make it
a method, Ctor, on the Function abstraction.
* Add a recursive Module population case to property initialization.
* Only treat the top-most frame in a module's initializer as belonging
to the module scope. This avoids interpreting block scopes within
that initializer as belonging to the module scope. Ultimately, it's
up to the language compiler to decide where to place scopes, but this
gives it more precise control over module scoping.
* Assert against unsupported named/star/keyword args in CocoPy.
This change adds support for naming imports. At the moment, this simply
makes the names dynamically accessible for languages that do dynamic loads
against module objects, versus strongly typed tokens. The basic scheme
is to keep two objects per module: one that contains the globals and its
prototype parent that contains just the exports. This ensures we can
share the same slots while attaining the desired information hiding
(e.g., when handing out an object for dynamic access, we give out this
parent objects, while when loading globals from within a module, we use
the childmost one that contains all private and exported variables).
The old model for imports was to use top-level declarations on the
enclosing module itself. This was a laudible attempt to simplify
matters, but just doesn't work.
For one, the order of initialization doesn't precisely correspond
to the imports as they appear in the source code. This could incur
some weird module initialization problems that lead to differing
behavior between a language and its Coconut variant.
But more pressing as we work on CocoPy support, it doesn't give
us an opportunity to dynamically bind names in a correct way. For
example, "import aws" now needs to actually translate into a variable
declaration and assignment of sorts. Furthermore, that variable name
should be visible in the environment block in which it occurs.
This change switches imports to act like statements. For the most
part this doesn't change much compared to the old model. The common
pattern of declaring imports at the top of a file will translate to
the imports happening at the top of the module's initializer. This
has the effect of initializing the transitive closure just as it
happened previously. But it enables alternative models, like imports
inside of functions, and -- per the above -- dynamic name binding.
This rearranges the way dynamic loads work a bit. Previously, they¬
required an object, and did a dynamic lookup in the object's property¬
map. For real dynamic loads -- of the kind Python uses, obviously,¬
but also ECMAScript -- we need to search the "environment".
This change searches the environment by looking first in the lexical¬
scope in the current function. If a variable exists, we will use it.¬
If that misses, we then look in the module scope. If a variable exists¬
there, we will use it. Otherwise, if the variable is used in a non-lval
position, an dynamic error will be raised ("name not declared"). If
an lval, however, we will lazily allocate a slot for it.
Note that Python doesn't use block scoping in the same way that most
languages do. This behavior is simply achieved by Python not emitting
any lexically scoped blocks other than at the function level.
This doesn't perfectly achieve the scoping behavior, because we don't
yet bind every name in a way that they can be dynamically discovered.
The two obvious cases are class names and import names. Those will be
covered in a subsequent commit.
Also note that we are getting lucky here that class static/instance
variables aren't accessible in Python or ECMAScript "ambiently" like
they are in some languages (e.g., C#, Java); as a result, we don't need
to introduce a class scope in the dynamic lookup. Some day, when we
want to support such languages, we'll need to think about how to let
languages control the environment probe order; for instance, perhaps
the LoadDynamicExpression node can have an "environment" property.
This changes the CocoPy default load type from static to dynamic,
since we don't have enough information at compile-time to emit
fully qualified tokens. Previously, Coconut only supported dynamic
loads with object targets, however we will need to support the full
scope search (class, module, global, etc).
The translation from Pythonic underscore_casing to camelCasing is
meant to only apply to serialized AST nodes, not arbitrary map keys
inside of dictionaries within them. This change fixes the previous
behavior, which indiscriminately applied mangling throughout.
As of this change -- just a few minor fixes -- CocoPy produces real Cocopacks!
$ cocopy index.py | coco pack info -a -
// Basic example of an AWS web server accessible over HTTP.
package "webserver" {
dependencies [
aws: "*"
]
module "index.py" {
imports [
"coconut/aws"
]
exports [
".init" -> ".init"
".main" -> ".main"
"_Name" -> "__name__"
"main" -> "main"
]
method ".init": ()
method ".main": ()
property "_Name": dynamic
method "main": (): dynamic
}
}
The contents of various tokens aren't quite right yet (they must be
fully qualified), meaning the pack still won't verify and load.
But we're getting close...
This change adds a lot more aggressive AST type assertions throughout,
ensuring that we catch type mismatches as early as possible.
I realize this is borderline "un-Pythonic", but it will keep me sane.
* Refactor the options on encoding into an options object, to make
it more future proof and a little easier to deal with. (For instance,
we hadn't plumbed the individual parameter options all the way
through in all cases, an easy mistake to make when there are N of them.)
* Produce ast.TypeToken objects in a few requisite places.
* Replace the "." delimiter in Python module names with the CocoPack
equivalent, "/".
This change gets us far enough along to fully parse and lower the
Coconut package metadata for the webserver-py example. It doesn't
serialize everything properly yet (since we haven't implemented the
serializer), and so it isn't runnable, but we're getting close(r).
* Fix a few minor issues in the AST layer:
- is_* functions should accept self parameters.
- All ast.Definition nodes require a name.
- A few assertions were wrong (e.g., is_stmt vs is_statement).
* Refactor the op constants to be a little shorter. I found them
to be gratuitously verbose in calling code.
* Add a compiler.Context object for transformation passes. This
holds state specific to a given module's context, since -- as
soon as we do multi-module passes -- it must remain isolated.
* Introduce support for the special "__name__" Python variable.
This entails creating the variable and initializing it. For now,
since we only support single-module cases, it is always "__main__".
* Properly spill declarations and initialization statements in
the module visitation logic. This includes detecting global
module properties while visiting statements at the top-level,
which gives us an approximation to Python's oddly-whacky scoping
rules (emulating this perfectly is going to be a PITA, I think).
* Properly track and emit imports.
* Add some more defensive assertions throughout, but particularly
in ensuring that the right node types are flowing around
(mostly this means statements vs expressions).
* Support the following Python AST types:
- Blocks.
- Assignments.
- Module-scoped function definitions.
- Imports.
- "Attribute" access (a.k.a., property access, i.e., `x.y`).
- Function calls.
- Comparison operators.
- Names and Name IDs.
This change mostly sketches out placeholders for the various Python
AST nodes and their transformations. A smattering of them have been
implemented -- basically, those that were "trivial and obvious" upon
the first pass over this -- however, the goal of this checkin really
is to make the broad brush strokes that we will now fill in.
A few minor changes are included, like better assertions, a few
fixups to required/optional parameters, stringification and equality
checking functions, and so on.
Additionally, a bit of the logic for transforming modules is included.
For example, we shove statements into the module initializer, wire
up the entrypoint routine, etc. For the most part, this is not
functional, however, and will get a bit more robust in the next commit.
The prior code was trying to be too "clever", skipping initialization
of properties that haven't been given non-None values. This was to
tidy up serialization and ensure the default JSON/YAML behavior skipped
undefined fields. Although that basically works, it causes problems
elsewhere in the code, where certain fields would need to be lazily
initialized; this introduces a level of complexity I'd rather not have,
and furthermore, we will most likely need custom serialization anyway.
This change introduces a new CocoPy compiler. It will implement a
subset of Python and translate it into Coconut packages and IL.
The project is underneath tools/cocopy/, alongside the cocojs/ compiler
directory, and is organized into two top-level directories: cmd/ contains
the command line tools and lib/ contains the library package code.
At the moment, the compiler processes a single file at a time. The
Coconut package metadata is loaded identically to how CocoJS will discover
and load package metadata (including searching "upwards" in the filesystem
for package references).
Most of the Coconut package and AST metadata has been projected as Python
classes in the lib/ast and lib/pack modules. The only missing pieces here
are the token manipulation abstractions which, for Python, will be far less
interesting due to the absence of any strong typing. (Most likely we'll
end up with a bunch of runtime assertions instead of nice types.)
I've opted to use the pythonparser package because (a) it seems to be well-
designed, (b) it is meant for tools just like this, and (c) the Google
Grumpy compiler uses it (which at least tells me it's a safe-ish bet).
The transformer is still mostly an empty shell of placeholders.
This change fixes named import references of the style
import {foo} from "bar";
...foo...;
Previously, we incorrectly assumed all naked variables of the sort
referred to members inside of the current module. Instead, we can
just refer to the existing logic that will use the bound symbol to
resolve the fully resolved module reference.
This changes a few naming things:
* Rename "husk" to "environment" (`coco env` for short).
* Rename NutPack/NutIL to CocoPack/CocoIL.
* Rename the primary Nut.yaml/json project file to Coconut.yaml/json.
* Rename the compiled Nutpack.yaml/json file to Cocopack.yaml/json.
* Rename the package asset directory from nutpack/ to .coconut/.
The TypeScript compiler will often transform Anys into Nevers when it
thinks control flow cannot reach a certain point. This isn't handled
gracefully at the moment. This change just erases it back to Any.
* Improve some logging and assertions.
* Initialize the prototype map (it was always empty before).
* Actually memoize the prototype objects in this map.
* During class initialization, ensure the base class is also initialized.
* Trigger class initialization anytime a property or function is loaded
from that class. Similarly, trigger module initialization on loads too.
* Treat For/ForIn/ForOfStatements as legal parents for local variables.
This ensures for loops at the top-level in a module have their variables
treated as locals, rather than module properties. Add a test also.
* Track source locations for more nodes.
* Mark auto-generated constructors as public so that they may be called.
* Fix the naming of class init methods; it should be ".init", not ".ctor".
* Clone the loop test cases into both function and module top-level variants.
One guiding principle for what makes it into the MuIL AST is that
the gap between source language and AST should not be too great; the
projection of switch statements from MuJS into MuIL clearly violated
that principle, particularly considering that the logic wasn't even
right due to the incorrect emulation of conditional breaks.
Instead of digging deeper into the hole, I've encoded switch logic
in the AST, and implemented support in the evaluator.
These asserts need to use Object.hasOwnProperty, rather than simply
looking up the property; otherwise, if we have a function whose name
happens to match a built-in ECMAScript property name -- like
toString -- then the assert will fire!
This change generalizes the support for default modules (added as
part of marapongo/mu#57), to use an "alias" table. This is just a
table of alternative module names to the real defining module name.
The default module is now just represented as a mapping from the
special default module token, ".default", to the actual defining
module. This generalization was necessary to support Node.js-style
alternative names for modules, like "lib" mapping to "lib/index".
This change adds runtime verification of the super test's expected
behavior. This has proven useful in testing out some of the recent
runtime/interpreter logic around super, etc. Most likely we will
start testing all of the MuJS tests as real MuIL/etc. tests soon.
This change introduces a MuJS standard library. This will allow us
to support an increasing amount of ECMAScript functionality without
needing to do it in the runtime or the compiler.
The basic idea is as follows.
In the TypeScript compiler, all uses of ECMAScript standard types and
functions get bound to the TypeScript library headers. These headers
are in the node_modules/typescript/lib/ directory, and there is one
per ECMAScript "standard" (lib.d.ts for ECMAScript 5, lib.es2015.d.ts
for ECMAScript 6, lib.es5016.d.ts for ECMAScript 7, and so on). Note
that these libraries are "header only", since the functionality for
these are of course supplied by whatever runtime (V8, Chakra, etc)
actually executes the resulting JavaScript code.
In our case, we have no such luxury. So instead, we intercept these
bindings and transform them into bindings to a MuJS standard library.
This library is just a MuIL library written in MuJS code. Only the
set of operations expressable in basic MuIL can be used to implement
these. In select few cases, we will need runtime intrinsics, but those
will not be specific to MuJS and will go into the basic Mu library and
shared among all MetaMu compilers.
At this point, the MuJS standard library only contains the ECMAScript
Error type, so that we can represent throwing errors in MuJS/MuIL.
This changes the way we discover the kind of export in the case of
an export declaration. This might be exporting a module member, as in
let x = 42;
export { x as y };
or it could be exporting an entire module previously imported, as in
import * as other from "other";
export { other };
In each case, we must emit a proper fully qualified token.
Our logic for detecting these cases was a bit busted previously. We
now rely on real symbolic information rather than doing hokey name lookups.
This change redoes the way module exports are represented. The old
mechanism -- although laudible for its attempt at consistency -- was
wrong. For example, consider this case:
let v = 42;
export { v };
The old code would silently add *two* members, both with the name "v",
one of which would be dropped since the entries in the map collided.
It would be easy enough just to detect collisions, and update the
above to mark "v" as public, when the export was encountered. That
doesn't work either, as the following two examples demonstrate:
let v = 42;
export { v as w };
let x = w; // error!
This demonstrates:
* Exporting "v" with a different name, "w" to consumers of the
module. In particular, it should not be possible for module
consumers to access the member through the name "v".
* An inability to access the exported name "w" from within the
module itself. This is solely for external consumption.
Because of this, we will use an export table approach. The exports
live alongside the members, and we are smart about when to consult
the export table, versus the member table, during name binding.
