This change restructures a lot more pertaining to deployments, snapshots,
environments, and the like.
The most notable change is that the notion of a deploy.Source is introduced,
which splits the responsibility between the deploy.Plan -- which simply
understands how to compute and carry out deployment plans -- and the idea
of something that can produce new objects on-demand during deployment.
The primary such implementation is evalSource, which encapsulates an
interpreter and takes a package, args, and config map, and proceeds to run
the interpreter in a distinct goroutine. It synchronizes as needed to
poke and prod the interpreter along its path to create new resource objects.
There are two other sources, however. First, a nullSource, which simply
refuses to create new objects. This can be handy when writing isolated
tests but is also used to simulate the "empty" environment as necessary to
do a complete teardown of the target environment. Second, a fixedSource,
which takes a pre-computed array of objects, and hands those, in order, to
the planning engine; this is mostly useful as a testing technique.
Boatloads of code is now changed and updated in the various CLI commands.
This further chugs along towards pulumi/lumi#90. The end is in sight.
This change guts the deployment planning and execution process, a
necessary component of pulumi/lumi#90.
The major effect of this change is that resources are actually
connected to the live objects, instead of being snapshots taken at
inopportune moments in time.
This change, part of pulumi/lumi#90, overhauls quite a bit of the
core resource, planning, environments, and related areas.
The biggest amount of movement comes from the splitting of pkg/resource
into multiple sub-packages. This results in:
- pkg/resource: just the core resource data structures.
- pkg/resource/deployment: all planning and deployment logic.
- pkg/resource/environment: all environment, configuration, and
serialized checkpoint structures and logic.
- pkg/resource/plugin: all dynamically loaded analyzer and
provider logic, including the actual loading and RPC mechanisms.
This also splits the resource abstraction up. We now have:
- resource.Resource: a shared interface.
- resource.Object: a resource that is connected to a live object
that will periodically observe mutations due to ongoing
evaluation of computations. Snapshots of its state may be
taken; however, this is purely a "pre-planning" abstraction.
- resource.State: a snapshot of a resource's state that is frozen.
In other words, it is no longer connected to a live object.
This is what will store provider outputs (ID and properties),
and is what may be serialized into a deployment record.
The branch is in a half-baked state as of this change; more changes
are to come...
LumiJS lambdas can now be serialized when they include calls to other LumiJS lambdas. The chain of lambda dependencies is jointly serialized into the target Lambda.
Also, LumiJS lambdas now include `node_modules` automatically in the AWS Lambda, ensuring the the runtime execution environment more closely matches the deployment time environment.
An early version of the gh-cicd example supporting #134 is added which uses these capabilities, currently including a mocked GitHub resource provider.
For lambdas which will execute at runtime,
we want to allow them to reference Node.js
global variables, like `console`.
This change makes Lumijs generated IL
incrementally more dynamic by preferring to
generate `TryLoadDynamic` over `LoadLocation`
for references to global variables (except for
references to imports).
Also introduces `console.log` in LumiJS, though
it is not yet attached to a Lumi global environment.
Fixes#174.
The scope chain currently does not include module-scope
vairables, which are instead stored on a module object. For
now, we are capturing this module object along with the
scope chain as part of a Lambda object so that we can use
it when evaluating variable references within a lambda
expression.
Fixes#175.
Introduces a free variable AST visitor, and uses this to limit
the environment exposed by the `serizlizeClosure` intrinsic
to only those variables that are referenced by the lambda body.
Fixes#177.
This change slightly refactors the way resources are created and
implemented. We now have two implementations of the Resource interface:
* `resource` (in resource_value.go), which is a snapshot of a resource's
state. All values are resolved and there is no live reference to any
heap state or objects. This will be used when serializing and/or
deserializing snapshots of deployments.
* `objectResource` (in resource_object.go), which is an implementation
of the Resource interface that wraps an underlying, live runtime object.
This currently introduces no functional difference, as fetching Inputs()
amounts to taking a snapshot of the full state. But this at least
gives us a leg to stand on in making sure that output properties are
read at the right times during evaluation.
This is a fundamental part of pulumi/lumi#90.
This change begins to track objects that are implicated in the
creation of computed values. This ultimately translates into the
resource URNs which are used during dependency analysis and
serialization. This is part of pulumi/lumi#90.
Adds Check implementation for aws.lambda.Permission
resources using AWS-defined regexps.
Fixes bug in lumidl Check wrapper which was dropping
reported check failures (an regen all rpc files).
Add calls to Get into the AWS provider test framework.
This change fixes the serialization of resource properties during
deployment checkpoints. We erroneously serialized empty arrays and
empty maps as though they were nil; instead, we want to keep them
serialized as non-nil empty collections, since the presence of a
value might be semantically meaningful. (We still skip nils.)
Also added some test cases.
This change implements `mapper.Encode` "for real" (that is, in a way
that isn't a complete embarrassment). It uses the obvious reflection
trickery to encode a tagged struct and its values as a JSON-like
in-memory map and collection of keyed values.
During this, I took the opportunity to also clean up a few other things
that had been bugging me. Namely, the presence of `mapper.Object` was
always error prone, since it isn't a true "typedef" in the sence that
it carries extra RTTI. Instead of doing that, let's just use the real
`map[string]interface{}` "JSON-map-like" object type. Even better, we
no longer require resource providers to deal with the mapper
infrastructure. Instead, the `Check` function can simply return an
array of errors. It's still best practice to return field-specific errors
to facilitate better diagnostics, but it's no longer required; and I've
added `resource.NewFieldError` to eliminate the need to import mapper.
As of this change, we can also consistently emit RPC structs with `lumi`
tags, rather than `lumi` tags on the way in and `json` on the way out.
This completes pulumi/lumi#183.
This changes the resource model to persist input and output properties
distinctly, so that when we diff changes, we only do so on the programmer-
specified input properties. This eliminates problems when the outputs
differ slightly; e.g., when the provider normalizes inputs, adds its own
values, or fails to produce new values that match the inputs.
This change simultaneously makes progress on pulumi/lumi#90, by beginning
tracking the resource objects implicated in a computed property's value.
I believe this fixes both #189 and #198.
This reverts commit 048c35d428.
I have a pending change that fixes this along with a number of
other issues (including pulumi/lumi#198 and pulumi/lumi#187),
however, it's going to take a little longer and I want to unblock.
This fixespulumi/lumi#200.
The @lumi.out decorator ended up not being required after all.
The engine essentially treats all resource properties as potentially
output properties now, given the way we read back the state from
the provider after essential operations.
This is a good thing, because the evaluator currently doesn't
perform dynamic decoration in a way that would be amenable to
a faithful ECMAScript implementation (see pulumi/lumi#128).
This change alters diag.Message to not format strings and, instead,
encourages developers to use the Infof, Errorf, and Warningf varargs
functions. It also tests that arguments are never interepreted as
format strings.
There are a few things that annoyed me about the way our CLI works with
directories when loading packages. For example, `lumi pack info some/pack/dir/`
never worked correctly. This is unfortunate when scripting commands.
This change fixes the workspace detection logic to handle these cases.
The AssumeRolePolicyDocument property returned by the AWS IAM GetRole API returns
a URL-encoded JSON string, so we need to decode this before JSON unmarshalling.
The Code property returned by AWS Lambda GetFunction provides a pre-signed S3 URL,
which changes on each call, and is of a different format to what is provided by the user.
For now, we'll not store this back into the Function object.
Add additional output properties to AWS Lambda Function that are stable values returned
from GetFunction.
Also corrects a gap where some property delete operations were not being correctly reported.
This change enables parallelism for our tests.
It also introdues a `test_core` Makefile target to just run the
core engine tests, and not the providers, since they take a long time.
This is intended only as part of the inner developer loop.
This is a minor refactoring to introduce a ProviderHost interface
that is associated with the context and can be swapped in and out for
custom plugin behavior. This is required to write tests that mock
certain aspects, like loading packages from the filesystem.
In theory, this change incurs zero behavioral changes.
This change fixes up a few things so that updates correctly deal
with output properties. This involves a few things:
1) All outputs stored on the pre snapshot need to get propagated
to the post snapshot during planning at various points. This
ensures that the diffing logic doesn't need to be special cased
everywhere, including both the Lumi and the provider sides.
2) Names are changed to "input" properties (using a new `lumi` tag
option, `in`). These are properties that providers are expected
to know nothing about, which we must treat with care during diffs.
3) We read back properties, via Get, after doing an Update just like
we do after performing a Create. This ensures that if an update
has a cascading impact on other properties, it will be detected.
4) Inspecting a change, prior to updating, must be done using the
computed property set instead of the real one. This is to avoid
mutating the resource objects ahead of actually applying a plan,
which would be wrong and misleading.
This change remembers which properties were computed as outputs,
or even just read back as default values, during a deployment. This
information is required in the before/after comparison in order to
perform an intelligent diff that doesn't flag, for example, the absence
of "default" values in the after image as deletions (among other things).
As I was in here, I also cleaned up the way the provider interface
works, dealing with concrete resource types, making it feel a little
richer and less like we're doing in-memory RPC.
This change makes progress on a few things with respect to properly
receiving properties on the engine side, coming from the provider side,
of the RPC boundary. The issues here are twofold:
1. Properties need to get unmapped using a JSON-tag-sensitive
marshaler, so that they are cased properly, etc. For that, we
have a new mapper.Unmap function (which is ultra lame -- see
pulumi/lumi#138).
2. We have the reverse problem with respect to resource IDs: on
the send side, we must translate from URNs (which the engine
knows about) and provider IDs (which the provider knows about);
similarly, then, on the receive side, we must translate from
provider IDs back into URNs.
As a result of these getting fixed, we can now properly marshal the
resulting properties back into the resource object during the plan
execution, alongside propagating and memoizing its ID.
The change to flow logging to plugins is nice, however, it can be
annoying because all writes to stderr are interepreted on the Lumi
side as errors. After this change, we will only flow if
--logflow is passed, e.g. as in
$ lumi --logtostderr --logflow -v=9 deploy ...
This change modifies the existing resource provider RPC interface slightly.
Instead of the Create API returning the bag of output properties, we will
rely on the Get API to do so. As a result, this change takes an initial
whack at implementing Get on all existing AWS resources. The Get API needs
to return a fully populated structure containing all inputs and outputs.
Believe it or not, this is actually part of pulumi/lumi#90.
This was done because just returning output properties is insufficient.
Any input properties that weren't supplied may have default values, for
example, and it is wholly reasonable to expect Lumi scripts to depend on
those values in addition to output values.
This isn't fully functional in its current form, because doing this
change turned up many other related changes required to enable output
properties. For instance, at the moment resource properties are defined
in terms of `resource.URN`s, and yet unfortunately the provider side
knows nothing of URNs (instead preferring to deal in `resource.ID`s).
I am going to handle that in a subsequent isolated change, since it will
have far-reaching implications beyond just modifying create and get.
This change introduces the notion of a computed versus an output
property on resources. Technically, output is a subset of computed,
however it is a special kind that we want to treat differently during
the evaluation of a deployment plan. Specifically:
* An output property is any property that is populated by the resource
provider, not code running in the Lumi type system. Because these
values aren't available during planning -- since we have not yet
performed the deployment operations -- they will be latent values in
our runtime and generally missing at the time of a plan. This is no
problem and we just want to avoid marshaling them in inopportune places.
* A computed property, on the other hand, is a different beast altogehter.
Although true one of these is missing a value -- by virtue of the fact
that they too are latent values, bottoming out in some manner on an
output property -- they will appear in serializable input positions.
Not only must we treat them differently during the RPC handshake and
in the resource providers, but we also want to guarantee they are gone
by the time we perform any CRUD operations on a resource. They are
purely a planning-time-only construct.
This change adds a @lumi.out decorator and modifies LumIDL to emit it on
output properties of resource types. There still isn't any runtime
awareness, however, this is an isolated change that will facilitate it.
This change includes approximately 1/3rd of the change necessary
to support output properties, as per pulumi/lumi#90.
In short, the runtime now has a new hidden type, Latent<T>, which
represents a "speculative" value, whose eventual type will be T,
that we can use during evaluation in various ways. Namely,
operations against Latent<T>s generally produce new Latent<U>s.
During planning, any Latent<T>s that end up in resource properties
are transformed into "unknown" property values. An unknown property
value is legal only during planning-time activities, such as Check,
Name, and InspectChange. As a result, those RPC interfaces have
been updated to include lookaside maps indicating which properties
have unknown values. My intent is to add some helper functions to
make dealing with this circumstance more correct-by-construction.
For now, using an unresolved Latent<T> in a conditional will lead
to an error. See pulumi/lumi#67. Speculating beyond these -- by
supporting iterative planning and application -- is something we
want to support eventually, but it makes sense to do that as an
additive change beyond this initial support. That is a missing 1/3.
Finally, the other missing 1/3rd which will happen much sooner
than the rest is restructuing plan application so that it will
correctly observe resolution of Latent<T> values. Right now, the
evaluation happens in one single pass, prior to the application, and
so Latent<T>s never actually get witnessed in a resolved state.
This new package is similar to the AWS Serverless Application Model, offering
higher-level interfaces to manage serverless resources. This will be a candidate
for moving into its own package in the future.
The FunctionX class has been moved into this module, and a new API class has
been added
The API class manages a collection of an API Gateway RestAPI, Stage and Deployment,
based on a collection of routes linked to Functions. On changes to the API specification,
it updates the RestAPI, replaces the Deployment and updates the Stage to point to
the updated Deployment.
This change also reorganizes some of the intrinsics.
When serializing a closure, we serialize the lambda environment into the
aws.lambda.Function Environment property. We need to serialize the lambda
environment in a stable order to ensure that we don't cause Lumi to require
updates to the aws.lambda.Function resource.
