This reverts commit 048c35d428.
I have a pending change that fixes this along with a number of
other issues (including pulumi/lumi#198 and pulumi/lumi#187),
however, it's going to take a little longer and I want to unblock.
This fixespulumi/lumi#200.
The @lumi.out decorator ended up not being required after all.
The engine essentially treats all resource properties as potentially
output properties now, given the way we read back the state from
the provider after essential operations.
This is a good thing, because the evaluator currently doesn't
perform dynamic decoration in a way that would be amenable to
a faithful ECMAScript implementation (see pulumi/lumi#128).
This change eliminates the use of nonstandard tools in our build:
* `go test` automatically uses `GOMAXPROCS` for its parallelism
setting. In modern Go, this is now set to the number of processors.
So, there is no need to set it explicitly using `nproc`.
* We can avoid `realpath` in the `lumijs` executable by making it
a Node.js file and using a relative require import of main.
* We can avoid `realpath` in our Makefiles by just using `pwd`.
This change alters diag.Message to not format strings and, instead,
encourages developers to use the Infof, Errorf, and Warningf varargs
functions. It also tests that arguments are never interepreted as
format strings.
* Makeify more; add a "full build" target
This change uses make for more of our tree. Namely, the AWS provider
and LumiJS compilers each now use make to build and/or install them.
Not only does this bring about some consistency to how we build and
test things, but also made it easy to add a full build target:
$ make all
This target will build, test, and install the core Go tools, the LumiJS
compiler, and the AWS provider, in that order.
Each can be made in isolation, however, which ensures that the inner
loop for those is fast and so that, when it comes to finishing
pulumi/lumi#147, we can easily split them out and make from the top.
There are a few things that annoyed me about the way our CLI works with
directories when loading packages. For example, `lumi pack info some/pack/dir/`
never worked correctly. This is unfortunate when scripting commands.
This change fixes the workspace detection logic to handle these cases.
The AWS tests take a while to run and it's easy to forget. Further,
they are going to start taking quite a bit longer very soon.
So, this change introduces a `make full` target, which is what our
CI tests run. But the ordinary `make` skips the long tests for faster
inner loop verification. Over time, I'm sure we'll get far more
sophisticated with the split between inner vs. outer loop testing,
especially for performance, stress, and so on.
Golint by default doesn't return a non-zero exit code when lint errors
are present. To address this, we need to pass -set_exit_status.
But it gets more complicated: because we are suppressing some lint
warnings, we cannot use -set_exit_status in the lint_quiet target.
For that, we use test -z to fail if any output passes the filter.
Eventually when we do pulumi/lumi#191, we can remove this hack.
If you try to specify egress rules on an EC2-Classic security group,
you get obscure error messages like "The parameter CidrIp is not recognized."
It turns out, the underlying AuthorizeSecurityGroupEgress function
isn't supported on EC2-Classic security groups (those w/out VPCs); see:
https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_AuthorizeSecurityGroupEgress.html
This change adds a check up front for this condition so developers get
a nicer error message.
Changes the Source property on AWS S3 Objects
to be an `in` property.
Adds an AllOptionSettings output property to
store the settings returned from AWS. This
unblocks keeping Beanstalk workign while
we evaluate options for #189.
Silently skip unsupported AWS ElasticBeanstalk
Environment properties in Get operation.
Fixes ARN resource name pair extraction to skip the
"/" between parts of the name pair.
The AssumeRolePolicyDocument property returned by the AWS IAM GetRole API returns
a URL-encoded JSON string, so we need to decode this before JSON unmarshalling.
The Code property returned by AWS Lambda GetFunction provides a pre-signed S3 URL,
which changes on each call, and is of a different format to what is provided by the user.
For now, we'll not store this back into the Function object.
Add additional output properties to AWS Lambda Function that are stable values returned
from GetFunction.
Also corrects a gap where some property delete operations were not being correctly reported.
This change enables parallelism for our tests.
It also introdues a `test_core` Makefile target to just run the
core engine tests, and not the providers, since they take a long time.
This is intended only as part of the inner developer loop.
This change responds to some great feedback from @lukehoban on
https://github.com/pulumi/lumi/pull/186. Namely:
* Relax many assertions in the providers. A failure here is pretty
bad because it takes down the entire provider due to fail-fast.
That said, I'd rather fail in response to a bug than let it go
silently. Nevertheless, a few of them were candidates for dynamic
failures.
* Use `aws.StringValue` and friends in more places.
* Remove a superfluous waitForTableState in DynamoDB.
* Fix some erroneous references to `Read` in some `Get` comments.
This is a minor refactoring to introduce a ProviderHost interface
that is associated with the context and can be swapped in and out for
custom plugin behavior. This is required to write tests that mock
certain aspects, like loading packages from the filesystem.
In theory, this change incurs zero behavioral changes.
Altering an instance's security group list used to imply a replace, but
it must have gotten broken somewhere along the line during the IDL updates.
(We desperately need those test cases...) For now, we will mark the groups
as `replaces, and when we get around to pulumi/lumi#187, we'll fix this.
This change fixes up a few things so that updates correctly deal
with output properties. This involves a few things:
1) All outputs stored on the pre snapshot need to get propagated
to the post snapshot during planning at various points. This
ensures that the diffing logic doesn't need to be special cased
everywhere, including both the Lumi and the provider sides.
2) Names are changed to "input" properties (using a new `lumi` tag
option, `in`). These are properties that providers are expected
to know nothing about, which we must treat with care during diffs.
3) We read back properties, via Get, after doing an Update just like
we do after performing a Create. This ensures that if an update
has a cascading impact on other properties, it will be detected.
4) Inspecting a change, prior to updating, must be done using the
computed property set instead of the real one. This is to avoid
mutating the resource objects ahead of actually applying a plan,
which would be wrong and misleading.
This change remembers which properties were computed as outputs,
or even just read back as default values, during a deployment. This
information is required in the before/after comparison in order to
perform an intelligent diff that doesn't flag, for example, the absence
of "default" values in the after image as deletions (among other things).
As I was in here, I also cleaned up the way the provider interface
works, dealing with concrete resource types, making it feel a little
richer and less like we're doing in-memory RPC.
* The EC2 instance get function needs to return security group ARNs, not raw IDs.
* The EC2 security group rule CRUD operations printed pointers and not the values.
This change makes progress on a few things with respect to properly
receiving properties on the engine side, coming from the provider side,
of the RPC boundary. The issues here are twofold:
1. Properties need to get unmapped using a JSON-tag-sensitive
marshaler, so that they are cased properly, etc. For that, we
have a new mapper.Unmap function (which is ultra lame -- see
pulumi/lumi#138).
2. We have the reverse problem with respect to resource IDs: on
the send side, we must translate from URNs (which the engine
knows about) and provider IDs (which the provider knows about);
similarly, then, on the receive side, we must translate from
provider IDs back into URNs.
As a result of these getting fixed, we can now properly marshal the
resulting properties back into the resource object during the plan
execution, alongside propagating and memoizing its ID.
This change skips printing output<T> properties as we perform a
deployment, instead showing the real values inline after the resource
has been created. (output<T> is still shown during planning, of course.)
The change to flow logging to plugins is nice, however, it can be
annoying because all writes to stderr are interepreted on the Lumi
side as errors. After this change, we will only flow if
--logflow is passed, e.g. as in
$ lumi --logtostderr --logflow -v=9 deploy ...
This change overhauls our AWS resource provder to use ARNs for all
AWS resource IDs. We have gone backwards and forwards on whether to
use the name, ID, or ARN for this purpose. This confusion was largely
driven by the inconsistencies within the AWS APIs themselves: sometimes
a resource's name is the preferred identifier, sometimes its ID,
sometimes its ARN, and there are even cases where it differs based on
context (e.g., nondefault versus default VPC).
Thankfully, ARNs are perfectly consistent, and well-defined, on this
matter. See http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html.
Although any given API may request an ID or name that isn't part of the
ARN, the ARN is always sufficiently complete and lossless to enable
recovery of the information needed. And furthermore, even if an API
doesn't want the full ARN, the resource name component of the resource's
ARN is easily parseable and useable in this role.
To facilitate this, I've created a new `arn` package that has a number
of factory and parsing helpers.
Overall, some things are more verbose -- e.g., we must always translate
from ARN space to names and vice versa -- however, using the ARN is
clarifying and guarantees that we always have a way to get the information
we need. And in general, thanks to the removal of several workarounds in
the code, I think we come out ahead in general code delta-wise.
This change modifies the existing resource provider RPC interface slightly.
Instead of the Create API returning the bag of output properties, we will
rely on the Get API to do so. As a result, this change takes an initial
whack at implementing Get on all existing AWS resources. The Get API needs
to return a fully populated structure containing all inputs and outputs.
Believe it or not, this is actually part of pulumi/lumi#90.
This was done because just returning output properties is insufficient.
Any input properties that weren't supplied may have default values, for
example, and it is wholly reasonable to expect Lumi scripts to depend on
those values in addition to output values.
This isn't fully functional in its current form, because doing this
change turned up many other related changes required to enable output
properties. For instance, at the moment resource properties are defined
in terms of `resource.URN`s, and yet unfortunately the provider side
knows nothing of URNs (instead preferring to deal in `resource.ID`s).
I am going to handle that in a subsequent isolated change, since it will
have far-reaching implications beyond just modifying create and get.
This change prepares for integrating more planning and deployment logic
closer to the runtime itself. For historical reasons, we ended up with these
in the env.go file which really has nothing to do with deployments anymore.
This change introduces the notion of a computed versus an output
property on resources. Technically, output is a subset of computed,
however it is a special kind that we want to treat differently during
the evaluation of a deployment plan. Specifically:
* An output property is any property that is populated by the resource
provider, not code running in the Lumi type system. Because these
values aren't available during planning -- since we have not yet
performed the deployment operations -- they will be latent values in
our runtime and generally missing at the time of a plan. This is no
problem and we just want to avoid marshaling them in inopportune places.
* A computed property, on the other hand, is a different beast altogehter.
Although true one of these is missing a value -- by virtue of the fact
that they too are latent values, bottoming out in some manner on an
output property -- they will appear in serializable input positions.
Not only must we treat them differently during the RPC handshake and
in the resource providers, but we also want to guarantee they are gone
by the time we perform any CRUD operations on a resource. They are
purely a planning-time-only construct.
