This reverts commit 048c35d428.
I have a pending change that fixes this along with a number of
other issues (including pulumi/lumi#198 and pulumi/lumi#187),
however, it's going to take a little longer and I want to unblock.
This fixespulumi/lumi#200.
The @lumi.out decorator ended up not being required after all.
The engine essentially treats all resource properties as potentially
output properties now, given the way we read back the state from
the provider after essential operations.
This is a good thing, because the evaluator currently doesn't
perform dynamic decoration in a way that would be amenable to
a faithful ECMAScript implementation (see pulumi/lumi#128).
This change eliminates the use of nonstandard tools in our build:
* `go test` automatically uses `GOMAXPROCS` for its parallelism
setting. In modern Go, this is now set to the number of processors.
So, there is no need to set it explicitly using `nproc`.
* We can avoid `realpath` in the `lumijs` executable by making it
a Node.js file and using a relative require import of main.
* We can avoid `realpath` in our Makefiles by just using `pwd`.
* Makeify more; add a "full build" target
This change uses make for more of our tree. Namely, the AWS provider
and LumiJS compilers each now use make to build and/or install them.
Not only does this bring about some consistency to how we build and
test things, but also made it easy to add a full build target:
$ make all
This target will build, test, and install the core Go tools, the LumiJS
compiler, and the AWS provider, in that order.
Each can be made in isolation, however, which ensures that the inner
loop for those is fast and so that, when it comes to finishing
pulumi/lumi#147, we can easily split them out and make from the top.
If you try to specify egress rules on an EC2-Classic security group,
you get obscure error messages like "The parameter CidrIp is not recognized."
It turns out, the underlying AuthorizeSecurityGroupEgress function
isn't supported on EC2-Classic security groups (those w/out VPCs); see:
https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_AuthorizeSecurityGroupEgress.html
This change adds a check up front for this condition so developers get
a nicer error message.
Changes the Source property on AWS S3 Objects
to be an `in` property.
Adds an AllOptionSettings output property to
store the settings returned from AWS. This
unblocks keeping Beanstalk workign while
we evaluate options for #189.
Silently skip unsupported AWS ElasticBeanstalk
Environment properties in Get operation.
Fixes ARN resource name pair extraction to skip the
"/" between parts of the name pair.
The AssumeRolePolicyDocument property returned by the AWS IAM GetRole API returns
a URL-encoded JSON string, so we need to decode this before JSON unmarshalling.
The Code property returned by AWS Lambda GetFunction provides a pre-signed S3 URL,
which changes on each call, and is of a different format to what is provided by the user.
For now, we'll not store this back into the Function object.
Add additional output properties to AWS Lambda Function that are stable values returned
from GetFunction.
Also corrects a gap where some property delete operations were not being correctly reported.
This change enables parallelism for our tests.
It also introdues a `test_core` Makefile target to just run the
core engine tests, and not the providers, since they take a long time.
This is intended only as part of the inner developer loop.
This change responds to some great feedback from @lukehoban on
https://github.com/pulumi/lumi/pull/186. Namely:
* Relax many assertions in the providers. A failure here is pretty
bad because it takes down the entire provider due to fail-fast.
That said, I'd rather fail in response to a bug than let it go
silently. Nevertheless, a few of them were candidates for dynamic
failures.
* Use `aws.StringValue` and friends in more places.
* Remove a superfluous waitForTableState in DynamoDB.
* Fix some erroneous references to `Read` in some `Get` comments.
Altering an instance's security group list used to imply a replace, but
it must have gotten broken somewhere along the line during the IDL updates.
(We desperately need those test cases...) For now, we will mark the groups
as `replaces, and when we get around to pulumi/lumi#187, we'll fix this.
This change fixes up a few things so that updates correctly deal
with output properties. This involves a few things:
1) All outputs stored on the pre snapshot need to get propagated
to the post snapshot during planning at various points. This
ensures that the diffing logic doesn't need to be special cased
everywhere, including both the Lumi and the provider sides.
2) Names are changed to "input" properties (using a new `lumi` tag
option, `in`). These are properties that providers are expected
to know nothing about, which we must treat with care during diffs.
3) We read back properties, via Get, after doing an Update just like
we do after performing a Create. This ensures that if an update
has a cascading impact on other properties, it will be detected.
4) Inspecting a change, prior to updating, must be done using the
computed property set instead of the real one. This is to avoid
mutating the resource objects ahead of actually applying a plan,
which would be wrong and misleading.
* The EC2 instance get function needs to return security group ARNs, not raw IDs.
* The EC2 security group rule CRUD operations printed pointers and not the values.
This change makes progress on a few things with respect to properly
receiving properties on the engine side, coming from the provider side,
of the RPC boundary. The issues here are twofold:
1. Properties need to get unmapped using a JSON-tag-sensitive
marshaler, so that they are cased properly, etc. For that, we
have a new mapper.Unmap function (which is ultra lame -- see
pulumi/lumi#138).
2. We have the reverse problem with respect to resource IDs: on
the send side, we must translate from URNs (which the engine
knows about) and provider IDs (which the provider knows about);
similarly, then, on the receive side, we must translate from
provider IDs back into URNs.
As a result of these getting fixed, we can now properly marshal the
resulting properties back into the resource object during the plan
execution, alongside propagating and memoizing its ID.
This change skips printing output<T> properties as we perform a
deployment, instead showing the real values inline after the resource
has been created. (output<T> is still shown during planning, of course.)
This change overhauls our AWS resource provder to use ARNs for all
AWS resource IDs. We have gone backwards and forwards on whether to
use the name, ID, or ARN for this purpose. This confusion was largely
driven by the inconsistencies within the AWS APIs themselves: sometimes
a resource's name is the preferred identifier, sometimes its ID,
sometimes its ARN, and there are even cases where it differs based on
context (e.g., nondefault versus default VPC).
Thankfully, ARNs are perfectly consistent, and well-defined, on this
matter. See http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html.
Although any given API may request an ID or name that isn't part of the
ARN, the ARN is always sufficiently complete and lossless to enable
recovery of the information needed. And furthermore, even if an API
doesn't want the full ARN, the resource name component of the resource's
ARN is easily parseable and useable in this role.
To facilitate this, I've created a new `arn` package that has a number
of factory and parsing helpers.
Overall, some things are more verbose -- e.g., we must always translate
from ARN space to names and vice versa -- however, using the ARN is
clarifying and guarantees that we always have a way to get the information
we need. And in general, thanks to the removal of several workarounds in
the code, I think we come out ahead in general code delta-wise.
This change modifies the existing resource provider RPC interface slightly.
Instead of the Create API returning the bag of output properties, we will
rely on the Get API to do so. As a result, this change takes an initial
whack at implementing Get on all existing AWS resources. The Get API needs
to return a fully populated structure containing all inputs and outputs.
Believe it or not, this is actually part of pulumi/lumi#90.
This was done because just returning output properties is insufficient.
Any input properties that weren't supplied may have default values, for
example, and it is wholly reasonable to expect Lumi scripts to depend on
those values in addition to output values.
This isn't fully functional in its current form, because doing this
change turned up many other related changes required to enable output
properties. For instance, at the moment resource properties are defined
in terms of `resource.URN`s, and yet unfortunately the provider side
knows nothing of URNs (instead preferring to deal in `resource.ID`s).
I am going to handle that in a subsequent isolated change, since it will
have far-reaching implications beyond just modifying create and get.
This change adds a @lumi.out decorator and modifies LumIDL to emit it on
output properties of resource types. There still isn't any runtime
awareness, however, this is an isolated change that will facilitate it.
This change includes approximately 1/3rd of the change necessary
to support output properties, as per pulumi/lumi#90.
In short, the runtime now has a new hidden type, Latent<T>, which
represents a "speculative" value, whose eventual type will be T,
that we can use during evaluation in various ways. Namely,
operations against Latent<T>s generally produce new Latent<U>s.
During planning, any Latent<T>s that end up in resource properties
are transformed into "unknown" property values. An unknown property
value is legal only during planning-time activities, such as Check,
Name, and InspectChange. As a result, those RPC interfaces have
been updated to include lookaside maps indicating which properties
have unknown values. My intent is to add some helper functions to
make dealing with this circumstance more correct-by-construction.
For now, using an unresolved Latent<T> in a conditional will lead
to an error. See pulumi/lumi#67. Speculating beyond these -- by
supporting iterative planning and application -- is something we
want to support eventually, but it makes sense to do that as an
additive change beyond this initial support. That is a missing 1/3.
Finally, the other missing 1/3rd which will happen much sooner
than the rest is restructuing plan application so that it will
correctly observe resolution of Latent<T> values. Right now, the
evaluation happens in one single pass, prior to the application, and
so Latent<T>s never actually get witnessed in a resolved state.
This new package is similar to the AWS Serverless Application Model, offering
higher-level interfaces to manage serverless resources. This will be a candidate
for moving into its own package in the future.
The FunctionX class has been moved into this module, and a new API class has
been added
The API class manages a collection of an API Gateway RestAPI, Stage and Deployment,
based on a collection of routes linked to Functions. On changes to the API specification,
it updates the RestAPI, replaces the Deployment and updates the Stage to point to
the updated Deployment.
This change also reorganizes some of the intrinsics.
Adds support for global secondary indexes on DynamoDB Tables.
Also adds a HashSet API to the AWS provider library. This handles part of #178,
providing a standard way for AWS provider implementations to compute set-based
diffs. This new API is used in both aws.dynamodb.Table and aws.elasticbeanstalk.Environment
currently.
Resolves#137.
This is an initial pass for supporting JavaScript lambda syntax for defining an AWS Lambda Function.
A higher level API for defining AWS Lambda Function objects `aws.lambda.FunctionX` is added which accepts a Lumi lambda as an argument, and uses that lambda to generate the AWS Lambda Function code package.
LumiJS lambdas are serialized as the JavaScript text of the lambda body, along with a serialized version of the environment that is deserialized at runtime and used as the context for the body of the lambda.
Remaining work to further improve support for lambdas is being tracked in #173, #174, #175, and #177.
Unifies the notion of BuiltinFunctions with the existing Intrinsic.
Intrinsic is now only a wrapper type, used to indicate the need to lookup the symbol in the
eval pacakges table of registered intrinsics. It does not carry the invoker function used
to eval the intrinsic.
We now support the `ManagedPolicyARNs` property on `aws.iam.Role`, enabling pre-defined policies to be attaced to an IAM role as part of it's definition.
To make it easier to discover and work with AWS managed policies, constants for all currently defined AWS managed policy ARNs are provided in the IAM module.
Inline policies are still not yet supported.
This is triggering the need for all downstream consumers of the Lumi
library to enable experimental decorators support. Given that it's not
even clear where we'll land on this (pending pulumi/lumi#137), and in
fact are actively revisiting the relationship between functions and
assets, I'm removing this annotation.
If a resource has no required properties, there's no need for an
argument object. (In the extreme case, perhaps the resource has
*no* properties.) This is a minor usability thing, but it's far
nicer to write code like
let buck = new Bucket("images");
than it is to write code like
let buck = new Bucket("images", {});
This change keeps the lumi prefix on our CLI tools.
As @lukehoban pointed out in person, as soon as we do pulumi/coconut#98,
most people (other than compiler authors themselves) won't actually be
typing the commands. And, furthermore, the commands aren't all that bad.
Eventually I assume we'll want something like `lumi-js`, or
`lumi-js-compiler`, so that binaries are discovered dynamically in a way
that is extensible for future languages. We can tackle this during #98.
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets. However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.
In this change, we introduce the concept of an Archive. An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).
There is a fair bit of library support for manipulating Archives as a
logical collection of Assets. I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets). This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
This reverts back to the old style of having the resource name as its
first parameter in the generated package. Stylistically, this reads a
little nicer, and also ensures we don't need to rewrite all our existing
samples/test cases, etc.
In a few places, an IDL type will be a pointer, but the resulting
RPC code would, ideally, be the naked type. Namely, in both resource
and asset cases, they are required to be pointers in the IDL (because
they are by-pointer by nature), but the marshaled representations need
not be pointers. This change depointerizes such types in the RPC
unless, of course, they are optional in which case pointers still make
sense. This avoids some annoying dereferencing and is the kind of thing
we want to do sooner before seeing widespread use.
This triggers complicates with local development, due to the way
that Glide vendors dependencies. After we move to distinct repos
for the various provider packages, we can go back to doing this.
Now that the IDL types encode named resources as a first class concept,
there is no need to do the dynamic overriding dance in the AWS provider.
I should have included this in my final "banking" of the IDL changes.
This change includes all of the CIDLC generated code for the AWS
package. Just as we vendor and version the generated gRPC code,
we will do so for these files also. This allows building of the
full packages without needing to run any special tools, in addition
to letting us keep track of generated code deltas over time.
This checkin contains all the non-generated package files. This includes
the package metadata, utility functions like utils/instanceMaps, and the
module layouts (these being a candidate for auto-generation down the road).
In addition, the install script has been updated fo reflect the new layout.
The old TypeScript resource definitions may now go away in favor of
the new IDL and associated generated code. After this change, I will
check in the generated code, so that the repo is self-contained.
This is just a minor stylistic change. Instead of letting the AWS
package imports take the good names, prefer to use the generated IDL
names (since they contain the strongly typed data structures).
Unfortunately, this wasn't a great name. The old one stunk, but the
new one was misleading at best. The thing is, this isn't about performing
an update -- it's about NOT doing an update, depending on its return value.
Further, it's not just previewing the changes, it is actively making a
decision on what to do in response to them. InspectUpdate seems to convey
this and I've unified the InspectUpdate and Update routines to take a
ChangeRequest, instead of UpdateRequest, to help imply the desired behavior.
This change rejects non-pointer optional fields in the IDL. Although
there is no reason this couldn't work, technically speaking, it's almost
always certainly a mistake. Better to issue an error about it; in the
future, we could consider making this a warning.
This "strongly types" the aws.ARN type. In reality, it is simply an
alias for a string at the moment (as this is what it boils down to on
the wire), but this at least gets us some amount of compile-time safety.
This change includes resource IDL definitions for all of AWS API Gateway's
corresponding CloudFormation resource types. There are no actual provider
implementations yet.
This introduces an http.API abstraction to Mantle and implements it
for Kubernetes Fission. (The AWS implementation will come shortly,
but requires a fair bit of API Gateway elbow grease first.)
This lets you invoke a function in response to an API event:
let func = new mantle.func.Function(
(e, ctx, cb => { ... },
);
let api = new mantle.http.API("/hello", "GET", func);
This change renames two provider methods:
* Read becomes Get.
* UpdateImpact becomes PreviewUpdate.
These just read a whole lot nicer than the old names.
This change flattens the Fission metadata property on the Coconut
side so that it is nicer to consume. Due to struct embedding in Go,
we need to perform translation at the serialization boundary anyway,
so the less-usable approach we had before didn't really buy us anything.
The Fission internal data structures reference one another by
name/URI. Coconut needs the graph with dependencies, however, so
we prefer to model things using real object references in these
situations. (This also makes it much nicer to program.) This
changes to fit the Coconut model better; the provider will then
turn around and present names/URIs to the various APIs.
This change introduces an @asset decorator in addition to a Code
asset type. The idea here is that compilers will recognize the
presence of @asset and react by spewing the code as a textual
asset, rather than an AST and/or actually getting compiled into IL.
This change introduces some resource IDL types for the Fission
project (https://github.com/fission/fission), which is a
functions-as-a-service (FaaS) substrate on top of Kubernetes.
It has a single configuration variable,
`kube-fission:config:controller`, which controls the address of
the Fission controller in the target Kubernetes cluster.
Someday, ideally we would rebuild this atop a more general
Kubernetes resource package, and add some sort of discovery of
this controller, to minimize the amount of manual configuration
required and to make it more resilient to self-healing operations.
This is IDL-only. The provider is coming soon.
This introduces a new experimental cross-cloud package, Mantle. This is part
of getting pulumi/coconut#134 up and running.
For now, there isn't much here, except for configuration of the target cloud and
scheduler, plus the ability to create a cross-cloud function. (OK, it's not truly
cross-cloud yet, since we have only implemented AWS Lambdas.)
Mantle is, of course, a placeholder name. Since this is a higher level
abstraction than the unopinionated core primitives, the name mantle corresponds
to the Earth's upper layers (and I thought "Crust" would be a rather odd name).
This change introduces a bunch of new AWS resource types:
* aws:iam/Group
* aws:iam/Policy
* aws:iam/Role
* aws:iam/User
* aws:kms/Key
* aws:lambda/Function
* aws:sqs/Queue
None of the associated resource providers are included; this is
an IDL-only projection of the types, their properties, and the
associated API documentation. Lacking these is blocking me from
making progress on the cross-platform functions project.
It's a little unfortunate to be digging the hole deeper in advance
of pulumi/coconut#133 landing. However, we are laser-focused on the
pulumi/coconut#134 prototype, so for now I'll just keep going.
This change includes a first basic whack at implementing S3 bucket
objects. It leverages the assets infrastructure put in place in the
last commit, supporting uploads from text, files, or arbitrary URIs.
Most of the interesting object properties remain unsupported for now,
but with this we can upload and delete basic S3 objects, sufficient
for a lot of the lambda functions management we need to implement.
This change introduces the basic concept of assets. It is far from
fully featured, however, it is enough to start adding support for various
storage kinds that require access to I/O-backed data (files, etc).
The challenge is that Coconut is deterministic by design, and so you
cannot simply read a file in an ad-hoc manner and present the bytes to
a resource provider. Instead, we will model "assets" as first class
entities whose data source is described to the system in a more declarative
manner, so that the system and resource providers can manage them.
There are three ways to create an asset at the moment:
1. A constant, in-memory string.
2. A path to a file on the local filesystem.
3. A URI, whose scheme is extensible.
Eventually, we want to support byte blobs, but due to our use of a
"JSON-like" type system, this isn't easily expressible just yet.
The URI scheme is extensible in that file://, http://, and https://
are supported "out of the box", but individual providers are free to
recognize their own schemes and support them. For instance, copying
one S3 object to another will be supported simply by passing a URI
with the s3:// protocol in the usual way.
Many utility functions are yet to be written, but this is a start.
This change includes a simple cpuwatch package, as a demonstration of
how easy it can be to add CPU-level monitoring to an existing stack.
It contains a single API, enableAlarm, that takes an instance and
CPU % threshold; if the CPU ever exceeds that threshold for a sustained
period of time (3 consecutive minutes), an email will be generated.
To use this, first simply install the package as usual, e.g.
$ coco pack get cpuwatch
From there, you will need to configure the email address to send to:
$ coco env config <env> cpuwatch:config:emailAddress joe@pulumi.com
And finally, add an import plus call to enableAlarm for all instances:
import * as cpuwatch from "cpuwatch";
..
let instance = new aws.ec2.Instance(...);
cpuwatch.enableAlarm(instance, 90); // email if >90% CPU utilization.
As part of this, I've added the typing projections for the AWS SNS topic
and CloudWatch alarm resource types (but no providers just yet).
This changes the object mapper infrastructure to offer more fine-grained
reporting of errors, and control over verification, during the mapping from
an untyped payload to a typed one. As a result, we can eliminate a bit of
the explicit unmarshaling goo in the AWS providers (but not all of it; I'm
sure there is more we can, and should, be doing here...)
This change adds the ability to specify analyzers in two ways:
1) By listing them in the project file, for example:
analyzers:
- acmecorp/security
- acmecorp/gitflow
2) By explicitly listing them on the CLI, as a "one off":
$ coco deploy <env> \
--analyzer=acmecorp/security \
--analyzer=acmecorp/gitflow
This closes out pulumi/coconut#119.
This change checks that the AMI referenced in an instance provisioning
is valid. If not, the deployment is rejected before even trying. This
is part of pulumi/coconut#115.
This change fixes up a handful of CocoJS runtime issues preventing
it from compiling; we aren't using much of this yet, but I hope to
resurrect it soon (so more legal ECMAScript code can run).
This changes a few naming things:
* Rename "husk" to "environment" (`coco env` for short).
* Rename NutPack/NutIL to CocoPack/CocoIL.
* Rename the primary Nut.yaml/json project file to Coconut.yaml/json.
* Rename the compiled Nutpack.yaml/json file to Cocopack.yaml/json.
* Rename the package asset directory from nutpack/ to .coconut/.
This change refactors the existing property verification logic for
the AWS provider to use the new checking capabilities. Beyond just
checking for required properties, there isn't much here yet. There
is a single check for security group description length that goes
beyond this, mostly for illustration purposes.
This is part of pulumi/coconut#115, although there is clearly more
to do here for something sufficiently snazzy and demoable.
This change adds a new resource.NewUniqueHex API, that simply generates
a unique hex string with the given prefix, with a specific count of
random bytes, and optionally capped to a maximum length.
This is used in the AWS SecurityGroup resource provider to avoid name
collisions, which is especially important during replacements (otherwise
we cannot possibly create a new instance before deleting the old one).
This resolvespulumi/coconut#108.
This change tracks which updates triggered a replacement. This enables
better output and diagnostics. For example, we now colorize those
properties differently in the output. This makes it easier to diagnose
why an unexpected resource might be getting deleted and recreated.
This change, part of pulumi/coconut#105, rearranges support for
resource replacement. The old model didn't properly account for
the cascading updates and possible replacement of dependencies.
Namely, we need to model a replacement as a creation followed by
a deletion, inserted into the overall DAG correctly so that any
resources that must be updated are updated after the creation but
prior to the deletion. This is done by inserting *three* nodes
into the graph per replacement: a physical creation step, a
physical deletion step, and a logical replacement step. The logical
step simply makes it nicer in the output (the plan output shows
a single "replacement" rather than the fine-grained outputs, unless
they are requested with --show-replace-steps). It also makes it
easier to fold all of the edges into a single linchpin node.
As part of this, the update step no longer gets to choose whether
to recreate the resource. Instead, the engine takes care of
orchestrating the replacement through actual create and delete calls.
