The AWS tests take a while to run and it's easy to forget. Further,
they are going to start taking quite a bit longer very soon.
So, this change introduces a `make full` target, which is what our
CI tests run. But the ordinary `make` skips the long tests for faster
inner loop verification. Over time, I'm sure we'll get far more
sophisticated with the split between inner vs. outer loop testing,
especially for performance, stress, and so on.
Golint by default doesn't return a non-zero exit code when lint errors
are present. To address this, we need to pass -set_exit_status.
But it gets more complicated: because we are suppressing some lint
warnings, we cannot use -set_exit_status in the lint_quiet target.
For that, we use test -z to fail if any output passes the filter.
Eventually when we do pulumi/lumi#191, we can remove this hack.
If you try to specify egress rules on an EC2-Classic security group,
you get obscure error messages like "The parameter CidrIp is not recognized."
It turns out, the underlying AuthorizeSecurityGroupEgress function
isn't supported on EC2-Classic security groups (those w/out VPCs); see:
https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_AuthorizeSecurityGroupEgress.html
This change adds a check up front for this condition so developers get
a nicer error message.
Changes the Source property on AWS S3 Objects
to be an `in` property.
Adds an AllOptionSettings output property to
store the settings returned from AWS. This
unblocks keeping Beanstalk workign while
we evaluate options for #189.
Silently skip unsupported AWS ElasticBeanstalk
Environment properties in Get operation.
Fixes ARN resource name pair extraction to skip the
"/" between parts of the name pair.
The AssumeRolePolicyDocument property returned by the AWS IAM GetRole API returns
a URL-encoded JSON string, so we need to decode this before JSON unmarshalling.
The Code property returned by AWS Lambda GetFunction provides a pre-signed S3 URL,
which changes on each call, and is of a different format to what is provided by the user.
For now, we'll not store this back into the Function object.
Add additional output properties to AWS Lambda Function that are stable values returned
from GetFunction.
Also corrects a gap where some property delete operations were not being correctly reported.
This change enables parallelism for our tests.
It also introdues a `test_core` Makefile target to just run the
core engine tests, and not the providers, since they take a long time.
This is intended only as part of the inner developer loop.
This change responds to some great feedback from @lukehoban on
https://github.com/pulumi/lumi/pull/186. Namely:
* Relax many assertions in the providers. A failure here is pretty
bad because it takes down the entire provider due to fail-fast.
That said, I'd rather fail in response to a bug than let it go
silently. Nevertheless, a few of them were candidates for dynamic
failures.
* Use `aws.StringValue` and friends in more places.
* Remove a superfluous waitForTableState in DynamoDB.
* Fix some erroneous references to `Read` in some `Get` comments.
This is a minor refactoring to introduce a ProviderHost interface
that is associated with the context and can be swapped in and out for
custom plugin behavior. This is required to write tests that mock
certain aspects, like loading packages from the filesystem.
In theory, this change incurs zero behavioral changes.
Altering an instance's security group list used to imply a replace, but
it must have gotten broken somewhere along the line during the IDL updates.
(We desperately need those test cases...) For now, we will mark the groups
as `replaces, and when we get around to pulumi/lumi#187, we'll fix this.
This change fixes up a few things so that updates correctly deal
with output properties. This involves a few things:
1) All outputs stored on the pre snapshot need to get propagated
to the post snapshot during planning at various points. This
ensures that the diffing logic doesn't need to be special cased
everywhere, including both the Lumi and the provider sides.
2) Names are changed to "input" properties (using a new `lumi` tag
option, `in`). These are properties that providers are expected
to know nothing about, which we must treat with care during diffs.
3) We read back properties, via Get, after doing an Update just like
we do after performing a Create. This ensures that if an update
has a cascading impact on other properties, it will be detected.
4) Inspecting a change, prior to updating, must be done using the
computed property set instead of the real one. This is to avoid
mutating the resource objects ahead of actually applying a plan,
which would be wrong and misleading.
This change remembers which properties were computed as outputs,
or even just read back as default values, during a deployment. This
information is required in the before/after comparison in order to
perform an intelligent diff that doesn't flag, for example, the absence
of "default" values in the after image as deletions (among other things).
As I was in here, I also cleaned up the way the provider interface
works, dealing with concrete resource types, making it feel a little
richer and less like we're doing in-memory RPC.
* The EC2 instance get function needs to return security group ARNs, not raw IDs.
* The EC2 security group rule CRUD operations printed pointers and not the values.
This change makes progress on a few things with respect to properly
receiving properties on the engine side, coming from the provider side,
of the RPC boundary. The issues here are twofold:
1. Properties need to get unmapped using a JSON-tag-sensitive
marshaler, so that they are cased properly, etc. For that, we
have a new mapper.Unmap function (which is ultra lame -- see
pulumi/lumi#138).
2. We have the reverse problem with respect to resource IDs: on
the send side, we must translate from URNs (which the engine
knows about) and provider IDs (which the provider knows about);
similarly, then, on the receive side, we must translate from
provider IDs back into URNs.
As a result of these getting fixed, we can now properly marshal the
resulting properties back into the resource object during the plan
execution, alongside propagating and memoizing its ID.
This change skips printing output<T> properties as we perform a
deployment, instead showing the real values inline after the resource
has been created. (output<T> is still shown during planning, of course.)
The change to flow logging to plugins is nice, however, it can be
annoying because all writes to stderr are interepreted on the Lumi
side as errors. After this change, we will only flow if
--logflow is passed, e.g. as in
$ lumi --logtostderr --logflow -v=9 deploy ...
This change overhauls our AWS resource provder to use ARNs for all
AWS resource IDs. We have gone backwards and forwards on whether to
use the name, ID, or ARN for this purpose. This confusion was largely
driven by the inconsistencies within the AWS APIs themselves: sometimes
a resource's name is the preferred identifier, sometimes its ID,
sometimes its ARN, and there are even cases where it differs based on
context (e.g., nondefault versus default VPC).
Thankfully, ARNs are perfectly consistent, and well-defined, on this
matter. See http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html.
Although any given API may request an ID or name that isn't part of the
ARN, the ARN is always sufficiently complete and lossless to enable
recovery of the information needed. And furthermore, even if an API
doesn't want the full ARN, the resource name component of the resource's
ARN is easily parseable and useable in this role.
To facilitate this, I've created a new `arn` package that has a number
of factory and parsing helpers.
Overall, some things are more verbose -- e.g., we must always translate
from ARN space to names and vice versa -- however, using the ARN is
clarifying and guarantees that we always have a way to get the information
we need. And in general, thanks to the removal of several workarounds in
the code, I think we come out ahead in general code delta-wise.
This change modifies the existing resource provider RPC interface slightly.
Instead of the Create API returning the bag of output properties, we will
rely on the Get API to do so. As a result, this change takes an initial
whack at implementing Get on all existing AWS resources. The Get API needs
to return a fully populated structure containing all inputs and outputs.
Believe it or not, this is actually part of pulumi/lumi#90.
This was done because just returning output properties is insufficient.
Any input properties that weren't supplied may have default values, for
example, and it is wholly reasonable to expect Lumi scripts to depend on
those values in addition to output values.
This isn't fully functional in its current form, because doing this
change turned up many other related changes required to enable output
properties. For instance, at the moment resource properties are defined
in terms of `resource.URN`s, and yet unfortunately the provider side
knows nothing of URNs (instead preferring to deal in `resource.ID`s).
I am going to handle that in a subsequent isolated change, since it will
have far-reaching implications beyond just modifying create and get.
This change prepares for integrating more planning and deployment logic
closer to the runtime itself. For historical reasons, we ended up with these
in the env.go file which really has nothing to do with deployments anymore.
This change introduces the notion of a computed versus an output
property on resources. Technically, output is a subset of computed,
however it is a special kind that we want to treat differently during
the evaluation of a deployment plan. Specifically:
* An output property is any property that is populated by the resource
provider, not code running in the Lumi type system. Because these
values aren't available during planning -- since we have not yet
performed the deployment operations -- they will be latent values in
our runtime and generally missing at the time of a plan. This is no
problem and we just want to avoid marshaling them in inopportune places.
* A computed property, on the other hand, is a different beast altogehter.
Although true one of these is missing a value -- by virtue of the fact
that they too are latent values, bottoming out in some manner on an
output property -- they will appear in serializable input positions.
Not only must we treat them differently during the RPC handshake and
in the resource providers, but we also want to guarantee they are gone
by the time we perform any CRUD operations on a resource. They are
purely a planning-time-only construct.
This change adds a @lumi.out decorator and modifies LumIDL to emit it on
output properties of resource types. There still isn't any runtime
awareness, however, this is an isolated change that will facilitate it.
We need to smuggle metadata from the resource IDL all the way through
to the runtime, so that it knows which things are output properties. In
order to do this, we'll leverage decorators and the support for serializing
them as attributes. This change adds support for the various kinds
(class, property, method, and parameter), in addition to test cases.
This change includes approximately 1/3rd of the change necessary
to support output properties, as per pulumi/lumi#90.
In short, the runtime now has a new hidden type, Latent<T>, which
represents a "speculative" value, whose eventual type will be T,
that we can use during evaluation in various ways. Namely,
operations against Latent<T>s generally produce new Latent<U>s.
During planning, any Latent<T>s that end up in resource properties
are transformed into "unknown" property values. An unknown property
value is legal only during planning-time activities, such as Check,
Name, and InspectChange. As a result, those RPC interfaces have
been updated to include lookaside maps indicating which properties
have unknown values. My intent is to add some helper functions to
make dealing with this circumstance more correct-by-construction.
For now, using an unresolved Latent<T> in a conditional will lead
to an error. See pulumi/lumi#67. Speculating beyond these -- by
supporting iterative planning and application -- is something we
want to support eventually, but it makes sense to do that as an
additive change beyond this initial support. That is a missing 1/3.
Finally, the other missing 1/3rd which will happen much sooner
than the rest is restructuing plan application so that it will
correctly observe resolution of Latent<T> values. Right now, the
evaluation happens in one single pass, prior to the application, and
so Latent<T>s never actually get witnessed in a resolved state.
This new package is similar to the AWS Serverless Application Model, offering
higher-level interfaces to manage serverless resources. This will be a candidate
for moving into its own package in the future.
The FunctionX class has been moved into this module, and a new API class has
been added
The API class manages a collection of an API Gateway RestAPI, Stage and Deployment,
based on a collection of routes linked to Functions. On changes to the API specification,
it updates the RestAPI, replaces the Deployment and updates the Stage to point to
the updated Deployment.
This change also reorganizes some of the intrinsics.
When serializing a closure, we serialize the lambda environment into the
aws.lambda.Function Environment property. We need to serialize the lambda
environment in a stable order to ensure that we don't cause Lumi to require
updates to the aws.lambda.Function resource.
This change flows --logtostderr and -v=x settings to any dynamically
loaded plugins so that running Lumi's command line with these flags
will also result in the plugins logging at the requested levels. I've
found this handy for debugging purposes.
Adds support for global secondary indexes on DynamoDB Tables.
Also adds a HashSet API to the AWS provider library. This handles part of #178,
providing a standard way for AWS provider implementations to compute set-based
diffs. This new API is used in both aws.dynamodb.Table and aws.elasticbeanstalk.Environment
currently.
Resolves#137.
This is an initial pass for supporting JavaScript lambda syntax for defining an AWS Lambda Function.
A higher level API for defining AWS Lambda Function objects `aws.lambda.FunctionX` is added which accepts a Lumi lambda as an argument, and uses that lambda to generate the AWS Lambda Function code package.
LumiJS lambdas are serialized as the JavaScript text of the lambda body, along with a serialized version of the environment that is deserialized at runtime and used as the context for the body of the lambda.
Remaining work to further improve support for lambdas is being tracked in #173, #174, #175, and #177.
Unifies the notion of BuiltinFunctions with the existing Intrinsic.
Intrinsic is now only a wrapper type, used to indicate the need to lookup the symbol in the
eval pacakges table of registered intrinsics. It does not carry the invoker function used
to eval the intrinsic.