This change brings back component outputs to the overall system again.
In doing so, it generally overhauls the way we do resource RPCs a bit:
* Instead of RegisterResource and CompleteResource, we call these
BeginRegisterResource and EndRegisterResource, which begins to model
these as effectively "asynchronous" resource requests. This should also
help with parallelism (https://github.com/pulumi/pulumi/issues/106).
* Flip the CLI/engine a little on its head. Rather than it driving the
planning and deployment process, we move more to a model where it
simply observes it. This is done by implementing an event handler
interface with three events: OnResourceStepPre, OnResourceStepPost,
and OnResourceComplete. The first two are invoked immediately before
and after any step operation, and the latter is invoked whenever a
EndRegisterResource comes in. The reason for the asymmetry here is
that the checkpointing logic in the deployment engine is largely
untouched (intentionally, as this is a sensitive part of the system),
and so the "begin"/"end" nature doesn't flow through faithfully.
* Also make the engine more event-oriented in its terminology and the
way it handles the incoming BeginRegisterResource and
EndRegisterResource events from the language host. This is the first
step down a long road of incrementally refactoring the engine to work
this way, a necessary prerequisite for parallelism.
This changes our resource creation logic to tolerate missing children.
This ultimately shouldn't be necessary, but we appear to have a bug where
sometimes during deletions we end up with dangling child pointers. See
pulumi/pulumi#613; it tracks fixing the bug, and also removing this
workaround and reenabling the assertion.
When merging inputs and defaults in order to construct the set of inputs
for a call to `Create`, we must recursively merge each property value:
the provided defaults may contain nested values that must be present in
the merged result.
This change includes a NewResourceMap API, alongside the existing,
but renamed, NewResourceTree API. This also tidies up the resulting
struct for public consumption. This is required for downstream tests.
* Don't show +s, -s, and ~s deeply. The intended format here looks
more like
+ aws:iam/instanceProfile:InstanceProfile (create)
[urn=urn:pulumi:test::aws/minimal::aws/iam/instanceProfile:InstanceProfile::ip2]
name: "ip2-079a29f428dc9987"
path: "/"
role: "ir-d0a632e3084a0252"
versus
+ aws:iam/instanceProfile:InstanceProfile (create)
+ [urn=urn:pulumi:test::aws/minimal::aws/iam/instanceProfile:InstanceProfile::ip2]
+ name: "ip2-079a29f428dc9987"
+ path: "/"
+ role: "ir-d0a632e3084a0252"
This makes it easier to see the resources modified in the output.
* Print adds/deletes during updates as
- property: "x"
+ property: "y"
rather than
~ property: "x"
~ property: "y"
the latter of which doesn't really tell you what's new/old.
* Show parent indentation on output properties, so they line up correctly.
* Only print stack outputs if not undefined.
This change simply prints a stack's output properties at the end of
running the program, since we now no longer actual record the outputs
as part of component registration. Bringing that back is part of
pulumi/pulumi#340, however it is too risky to add hastily. So, for
now, we will simply special case Stacks.
This change switches from child lists to parent pointers, in the
way resource ancestries are represented. This cleans up a fair bit
of the old parenting logic, including all notion of ambient parent
scopes (and will notably address pulumi/pulumi#435).
This lets us show a more parent/child display in the output when
doing planning and updating. For instance, here is an update of
a lambda's text, which is logically part of a cloud timer:
* cloud:timer:Timer: (same)
[urn=urn:pulumi:malta::lm-cloud:☁️timer:Timer::lm-cts-malta-job-CleanSnapshots]
* cloud:function:Function: (same)
[urn=urn:pulumi:malta::lm-cloud:☁️function:Function::lm-cts-malta-job-CleanSnapshots]
* aws:serverless:Function: (same)
[urn=urn:pulumi:malta::lm-cloud::aws:serverless:Function::lm-cts-malta-job-CleanSnapshots]
~ aws:lambda/function:Function: (modify)
[id=lm-cts-malta-job-CleanSnapshots-fee4f3bf41280741]
[urn=urn:pulumi:malta::lm-cloud::aws:lambda/function:Function::lm-cts-malta-job-CleanSnapshots]
- code : archive(assets:2092f44) {
// etc etc etc
Note that we still get walls of text, but this will be actually
quite nice when combined with pulumi/pulumi#454.
I've also suppressed printing properties that didn't change during
updates when --detailed was not passed, and also suppressed empty
strings and zero-length arrays (since TF uses these as defaults in
many places and it just makes creation and deletion quite verbose).
Note that this is a far cry from everything we can possibly do
here as part of pulumi/pulumi#340 (and even pulumi/pulumi#417).
But it's a good start towards taming some of our output spew.
The first exception relates to how we launch plugins. Plugin paths are
calculated using a well-known set of rules; this makes `gas` suspicious
due to the need to use a variable to store the path of the plugin.
The second and third are in test code and aren't terribly concerning.
The latter exception asks `gas` to ignore the access key we hard-code
into the integration tests for our Pulumi test account.
The fourth exception allows use to use more permissive permissions for
the `.pulumi` directory than `gas` would prefer. We use `755`; `gas`
wants `700` or stricter. `755` is the default for `mkdir` and `.git` and
so seems like a reasonable choice for us.
By default, debugging events are not displayed by `pulumi`; the `-d`
flag must be provided to enable their display. This is necessary e.g.
to view debugging output from Terraform when TF logging is enabled.
These changes add the option to display debug output from `pulumi` to
the integration test framework.
These changes also contain a small fix for the display of component
children.
Parallelize collection of logs at each layer of the resource tree walk. Since almost all cost of log collection is at leaf nodes, this should allow the total time for log collection to be close to the time taken for the single longest leaf node, instead of the sum of all leaf nodes.
We need to clear the resource filter during the resource tree walk to ensure that logs from children of matched resources are collected and aggregated.
We have to reverse engineer the name from the soruce LogGroup information since that is all we got at runtime, but luckily that is sufficient given current name generation approach.
This kind of code is *very* sensitive to any changes to automatic name generation - but that is likely inevitable at this layer.
When working on the `.pulumiignore` support, I added a small command
that would just dump the file we sent to the service (without actually
talking to the service) so I could do some measurements on archive
sizes.
I mentioned this to Chris who said he had hacked up a similar thing
for something he was working on, so it felt worthwhile to check this
command in (hidden behind a flag) to share the implementation.
This command is hidden behind an environment variable, so customers
will not see it unless they opt in.
When deploying a project via the Pulumi.com service, we have to upload
the entire "context" of your project to Pulumi.com. The context of the
program is all files in the directory tree rooted by the `Pulumi.yaml`
file, which will often contain stuff we don't want to upload, but
previously we had no control over what would be updated (and so folks
would do hacky things like delete folders before running `pulumi
update`).
This change adds support for `.pulumiignore` files which should behave
like `.gitignore`. In addition, we were not previously compressing
files when we added them to the zip archive we uploaded and now.
By default, every .pulumiignore file is treated as if it had an
exclusion for `.git/` at the top of the file (users can override this
by adding an explicit `!.git/` to their file) since it is very
unlikely for there to ever be a reason to upload the .git folder to
the service.
Fixespulumi/pulumi-service#122
This change adds back component output properties. Doing so
requires splitting the RPC interface for creating resources in
half, with an initial RegisterResource which contains all of the
input properties, and a final CompleteResource which optionally
contains any output properties synthesized by the component.
This change switches from child lists to parent pointers, in the
way resource ancestries are represented. This cleans up a fair bit
of the old parenting logic, including all notion of ambient parent
scopes (and will notably address pulumi/pulumi#435).
This lets us show a more parent/child display in the output when
doing planning and updating. For instance, here is an update of
a lambda's text, which is logically part of a cloud timer:
* cloud:timer:Timer: (same)
[urn=urn:pulumi:malta::lm-cloud:☁️timer:Timer::lm-cts-malta-job-CleanSnapshots]
* cloud:function:Function: (same)
[urn=urn:pulumi:malta::lm-cloud:☁️function:Function::lm-cts-malta-job-CleanSnapshots]
* aws:serverless:Function: (same)
[urn=urn:pulumi:malta::lm-cloud::aws:serverless:Function::lm-cts-malta-job-CleanSnapshots]
~ aws:lambda/function:Function: (modify)
[id=lm-cts-malta-job-CleanSnapshots-fee4f3bf41280741]
[urn=urn:pulumi:malta::lm-cloud::aws:lambda/function:Function::lm-cts-malta-job-CleanSnapshots]
- code : archive(assets:2092f44) {
// etc etc etc
Note that we still get walls of text, but this will be actually
quite nice when combined with pulumi/pulumi#454.
I've also suppressed printing properties that didn't change during
updates when --detailed was not passed, and also suppressed empty
strings and zero-length arrays (since TF uses these as defaults in
many places and it just makes creation and deletion quite verbose).
Note that this is a far cry from everything we can possibly do
here as part of pulumi/pulumi#340 (and even pulumi/pulumi#417).
But it's a good start towards taming some of our output spew.
