After 233c5a8 landed, I noticed there are a few things to be fixed up:
* Run gometalinter in all the right places. We need to run both in
lint and lint_quiet targets. I've also cleaned up some of the logic
around what to suppress so there's less repetition.
* We currently @ meaningful commands, which is unfortunate, since it
makes debugging Makefiles tough (especially when looking at CI build
logs). Going forward, we should only use @ for meaningless commands,
like @echo.
* The AWS project wasn't actually running tslint, because it needs to
say `tslint './pack/**/*.ts' --exclude='./pack/node_modules/**'`.
The current script of `tslint lib/aws/pack/...` wasn't actually
running lint, hence we missed a lot of AWS lint issues.
* Fix up the issues that these fixes uncovered. Mostly err shadowing.
We run tests in parallel and recently we began hitting a high enough
degree of parallelism that we've begun seeing "unsynchronized access
to map" errors in our test passes (intermittently). The root cause
is that access to the type symbol caches aren't synchronized. It would
be ideal if we actually rewired these to be cached in the compiler
context -- rather than being global -- but this fix is sufficient for
now. We will simply synchronize access using a Mutex.
We recently changed the Resource base type to have no constructor,
rather than a manual empty constructor. This ought to work just fine.
The LumiJS compiler indeed generates a constructor, however, it is
missing a body and when the interpreter tries to invoke it, we crash
with a nil reference panic. The runtime actually tolerates missing
constructors entirely, although the way LumiJS binds super calls
doesn't tolerate the missing base constructor. This change simply
generates such constructors in LumiJS with empty bodies.
In addition, I've added an error that will catch the empty body
problem during binding, since technically speaking, all functions
must have bodies. (Our runtime happens to support the notion of
"abstract", however, so we only fire the error on concrete functions.)
This change restructures a lot more pertaining to deployments, snapshots,
environments, and the like.
The most notable change is that the notion of a deploy.Source is introduced,
which splits the responsibility between the deploy.Plan -- which simply
understands how to compute and carry out deployment plans -- and the idea
of something that can produce new objects on-demand during deployment.
The primary such implementation is evalSource, which encapsulates an
interpreter and takes a package, args, and config map, and proceeds to run
the interpreter in a distinct goroutine. It synchronizes as needed to
poke and prod the interpreter along its path to create new resource objects.
There are two other sources, however. First, a nullSource, which simply
refuses to create new objects. This can be handy when writing isolated
tests but is also used to simulate the "empty" environment as necessary to
do a complete teardown of the target environment. Second, a fixedSource,
which takes a pre-computed array of objects, and hands those, in order, to
the planning engine; this is mostly useful as a testing technique.
Boatloads of code is now changed and updated in the various CLI commands.
This further chugs along towards pulumi/lumi#90. The end is in sight.
This change guts the deployment planning and execution process, a
necessary component of pulumi/lumi#90.
The major effect of this change is that resources are actually
connected to the live objects, instead of being snapshots taken at
inopportune moments in time.
For lambdas which will execute at runtime,
we want to allow them to reference Node.js
global variables, like `console`.
This change makes Lumijs generated IL
incrementally more dynamic by preferring to
generate `TryLoadDynamic` over `LoadLocation`
for references to global variables (except for
references to imports).
Also introduces `console.log` in LumiJS, though
it is not yet attached to a Lumi global environment.
Fixes#174.
The scope chain currently does not include module-scope
vairables, which are instead stored on a module object. For
now, we are capturing this module object along with the
scope chain as part of a Lambda object so that we can use
it when evaluating variable references within a lambda
expression.
Fixes#175.
Introduces a free variable AST visitor, and uses this to limit
the environment exposed by the `serizlizeClosure` intrinsic
to only those variables that are referenced by the lambda body.
Fixes#177.
This change implements `mapper.Encode` "for real" (that is, in a way
that isn't a complete embarrassment). It uses the obvious reflection
trickery to encode a tagged struct and its values as a JSON-like
in-memory map and collection of keyed values.
During this, I took the opportunity to also clean up a few other things
that had been bugging me. Namely, the presence of `mapper.Object` was
always error prone, since it isn't a true "typedef" in the sence that
it carries extra RTTI. Instead of doing that, let's just use the real
`map[string]interface{}` "JSON-map-like" object type. Even better, we
no longer require resource providers to deal with the mapper
infrastructure. Instead, the `Check` function can simply return an
array of errors. It's still best practice to return field-specific errors
to facilitate better diagnostics, but it's no longer required; and I've
added `resource.NewFieldError` to eliminate the need to import mapper.
As of this change, we can also consistently emit RPC structs with `lumi`
tags, rather than `lumi` tags on the way in and `json` on the way out.
This completes pulumi/lumi#183.
This changes the resource model to persist input and output properties
distinctly, so that when we diff changes, we only do so on the programmer-
specified input properties. This eliminates problems when the outputs
differ slightly; e.g., when the provider normalizes inputs, adds its own
values, or fails to produce new values that match the inputs.
This change simultaneously makes progress on pulumi/lumi#90, by beginning
tracking the resource objects implicated in a computed property's value.
I believe this fixes both #189 and #198.
This change enables parallelism for our tests.
It also introdues a `test_core` Makefile target to just run the
core engine tests, and not the providers, since they take a long time.
This is intended only as part of the inner developer loop.
This change includes approximately 1/3rd of the change necessary
to support output properties, as per pulumi/lumi#90.
In short, the runtime now has a new hidden type, Latent<T>, which
represents a "speculative" value, whose eventual type will be T,
that we can use during evaluation in various ways. Namely,
operations against Latent<T>s generally produce new Latent<U>s.
During planning, any Latent<T>s that end up in resource properties
are transformed into "unknown" property values. An unknown property
value is legal only during planning-time activities, such as Check,
Name, and InspectChange. As a result, those RPC interfaces have
been updated to include lookaside maps indicating which properties
have unknown values. My intent is to add some helper functions to
make dealing with this circumstance more correct-by-construction.
For now, using an unresolved Latent<T> in a conditional will lead
to an error. See pulumi/lumi#67. Speculating beyond these -- by
supporting iterative planning and application -- is something we
want to support eventually, but it makes sense to do that as an
additive change beyond this initial support. That is a missing 1/3.
Finally, the other missing 1/3rd which will happen much sooner
than the rest is restructuing plan application so that it will
correctly observe resolution of Latent<T> values. Right now, the
evaluation happens in one single pass, prior to the application, and
so Latent<T>s never actually get witnessed in a resolved state.
Resolves#137.
This is an initial pass for supporting JavaScript lambda syntax for defining an AWS Lambda Function.
A higher level API for defining AWS Lambda Function objects `aws.lambda.FunctionX` is added which accepts a Lumi lambda as an argument, and uses that lambda to generate the AWS Lambda Function code package.
LumiJS lambdas are serialized as the JavaScript text of the lambda body, along with a serialized version of the environment that is deserialized at runtime and used as the context for the body of the lambda.
Remaining work to further improve support for lambdas is being tracked in #173, #174, #175, and #177.
The eval test in its current form depends on the Lumi standard library
as an external dependency. This means that, in order to run the test,
you must first install the standard library. Not only is this poor
practice, it is also interfering with our ability to get our new CI/CD
system up and running. This change fixes all of that by mocking the one
standard library runtime function that we need in order to hook the intrinsic.
This change adds some machinery to make it easier to write evaluator tests,
and also implements some tests for the lumi:runtime/dynamic:isFunction intrinsic.
This change implements property accessors (getters and setters).
The approach is fairly basic, but is heavily inspired by the ECMAScript5
approach of attaching a getter/setter to any property slot (even if we don't
yet fully exploit this capability). The evaluator then needs to track and
utilize the appropriate accessor functions when loading locations.
This change includes CocoJS support and makes a dent in pulumi/coconut#66.
This addresses a bug where we did not reconstruct the correct lexical
environment when restoring a lambda's captured context. Namely, the local
variables scope "drifted" with respect to the evaluation scope slots.
This is an example program that triggered it:
function mkeighty() {
let eighty = 80;
return () => eighty;
}
let group = new ec2.SecurityGroup(..., {
ingress: [ ..., fromPort: mkeighty()(), ... ],
});
I am going to work on turning this into a regression test with my next
checkin; there's a fair bit of test infrastructure "machinery" I need
to put in place, but the time has come to lay the foundation.
This change completes implementing lambdas in the runtime, closing
out pulumi/coconut#62. The change is mostly straightforward, with
most changes coming down to the fact that functions may now exist
that themselves aren't definitions (like class/module members).
The function stub machinery has also been updated to retain the
environment in which a lambda was created, effectively "capturing"
the lexically available variables. Note that this is *not* dynamic
scoping, which will be a problem down the road when/if we want to
support Ruby. My guess is we'll just have a completely different
DynamicallyScopedLambdaExpression opcode.
The previous shape of SequenceExpression only permitted expressions
in the sequence. This is pretty common in most ILs, however, it usually
leads to complicated manual spilling in the event that a statement is needed.
This is often necessary when, for example, a compiler is deeply nested in some
expression production, and then realizes the code expansion requires a
statement (e.g., maybe a new local variable must be declared, etc).
Instead of requiring complicated code-gen, this change permits SequenceExpression
to contain an arbitrary mixture of expression/statement prelude nodes, terminating
with a single, final Expression which yields the actual expression value. The
runtime bears the burden of implementing this which, frankly, is pretty trivial.
I've tripped over pulumi/coconut#141 a few times now, particularly with
the sort of dynamic payloads required when creating lambdas and API gateways.
This change implements support for computed property initializers.
This change introduces TryLoadDynamicExpression. This is similar to
the existing LoadDynamicExpression opcode, except that it will return
null in response to a missing member (versus the default of raising
an exception). This is to enable languages like JavaScript to encode
operations properly (which always yields undefined/nulls), while still
catering to languages like Python (which throw exceptions).
This change introduces decorator support for CocoJS and the corresponding
IL/AST changes to store them on definition nodes. Nothing consumes these
at the moment, however, I am looking at leveraging this to indicate that
certain program fragments are "code" and should be serialized specially
(in support of Functions-as-lambdas).
This change implements simple index-based CocoPy subscripts (and
not the more fully featured slicing ones).
Alongside this, we relax a binder-time check that all dynamic
access types must be strings. The eval code already handles
numeric (array) accesses, so we will permit these to flow through.
This change emits all CocoJS interfaces as records. This allows us to
safely construct instances of them from Python using anonymous properties,
essentially emulating object literals.
For example, given a CocoJs interface defined as such:
interface Foo {
x;
y?;
z;
}
we can easily construct fresh instances using normal JS literals:
let f = { x: 42, z: "bar" };
But, Python doesn't have the equivalent literal syntax, wedging us.
It's now possible to initialize an instance from CocoPy as follows:
let f = Foo(x=42, z="bar")
This leverages the notion of records and primary properties, as
described in our CocoPack/CocoIL design documents.
Due to Python's ... interesting ... scoping rules, we want to avoid
forcing block scopes in certain places. Instead, we will let arbitrary
statements take their place. Of course, this could be a block, but it
very well could be a multi-statement (essentially a block that doesn't
imply a new lexical scope), or anything, really.
* Move checking the validity of a derived class with no constructor
from evlauation to binding (verification). Furthermore, make it
a verification error instead of an assert, and make the checking
complete (i.e., don't just check for the existence of a base class,
also check for the existence of a ctor, recursively).
* Refactor the constructor accessor out of the evaluator and make it
a method, Ctor, on the Function abstraction.
* Add a recursive Module population case to property initialization.
* Only treat the top-most frame in a module's initializer as belonging
to the module scope. This avoids interpreting block scopes within
that initializer as belonging to the module scope. Ultimately, it's
up to the language compiler to decide where to place scopes, but this
gives it more precise control over module scoping.
* Assert against unsupported named/star/keyword args in CocoPy.
This change adds support for naming imports. At the moment, this simply
makes the names dynamically accessible for languages that do dynamic loads
against module objects, versus strongly typed tokens. The basic scheme
is to keep two objects per module: one that contains the globals and its
prototype parent that contains just the exports. This ensures we can
share the same slots while attaining the desired information hiding
(e.g., when handing out an object for dynamic access, we give out this
parent objects, while when loading globals from within a module, we use
the childmost one that contains all private and exported variables).
The old model for imports was to use top-level declarations on the
enclosing module itself. This was a laudible attempt to simplify
matters, but just doesn't work.
For one, the order of initialization doesn't precisely correspond
to the imports as they appear in the source code. This could incur
some weird module initialization problems that lead to differing
behavior between a language and its Coconut variant.
But more pressing as we work on CocoPy support, it doesn't give
us an opportunity to dynamically bind names in a correct way. For
example, "import aws" now needs to actually translate into a variable
declaration and assignment of sorts. Furthermore, that variable name
should be visible in the environment block in which it occurs.
This change switches imports to act like statements. For the most
part this doesn't change much compared to the old model. The common
pattern of declaring imports at the top of a file will translate to
the imports happening at the top of the module's initializer. This
has the effect of initializing the transitive closure just as it
happened previously. But it enables alternative models, like imports
inside of functions, and -- per the above -- dynamic name binding.
Previously, it was an error to look up a local that didn't exist.
Now it is common, thanks to dynamic lookups. A few code-paths didn't
previously handle this adequately; now they do.
This rearranges the way dynamic loads work a bit. Previously, they¬
required an object, and did a dynamic lookup in the object's property¬
map. For real dynamic loads -- of the kind Python uses, obviously,¬
but also ECMAScript -- we need to search the "environment".
This change searches the environment by looking first in the lexical¬
scope in the current function. If a variable exists, we will use it.¬
If that misses, we then look in the module scope. If a variable exists¬
there, we will use it. Otherwise, if the variable is used in a non-lval
position, an dynamic error will be raised ("name not declared"). If
an lval, however, we will lazily allocate a slot for it.
Note that Python doesn't use block scoping in the same way that most
languages do. This behavior is simply achieved by Python not emitting
any lexically scoped blocks other than at the function level.
This doesn't perfectly achieve the scoping behavior, because we don't
yet bind every name in a way that they can be dynamically discovered.
The two obvious cases are class names and import names. Those will be
covered in a subsequent commit.
Also note that we are getting lucky here that class static/instance
variables aren't accessible in Python or ECMAScript "ambiently" like
they are in some languages (e.g., C#, Java); as a result, we don't need
to introduce a class scope in the dynamic lookup. Some day, when we
want to support such languages, we'll need to think about how to let
languages control the environment probe order; for instance, perhaps
the LoadDynamicExpression node can have an "environment" property.
This changes the CocoPy default load type from static to dynamic,
since we don't have enough information at compile-time to emit
fully qualified tokens. Previously, Coconut only supported dynamic
loads with object targets, however we will need to support the full
scope search (class, module, global, etc).
This change makes node optional in the lookupBasicType function, which is
necessary in cases where diagnostics information isn't available (such as
with configuration application). This eliminates an assert when you fat-
finger a configuration key. The associated message was missing apostrophes.
