This PR just wires the `Package.Main` field to the Pulumi Service (and in subsequent PRs, the `pulumi-service` and `pulumi-ppc` repos).
@joeduffy , should we just upload the entire `package.Package` type with the `UpdateProgramRequest` type? I'm not sure we want to treat that type as part of part of our public API surface area. But on the other hand, we'll need to mirror relevant fields in N places if we don't.
This change adds prefix +/- change markers for added and deleted
lines, in addition to the existing colorization. It also elides
empty strings from the diff which, due to our newline splitting,
always ended up with extra whitespace in the +/- output.
Previously, we were inconsistent on how we handled argument validation
in the CLI. Many commands used cobra.Command's Args property to
provide a validator if they took arguments, but commands which did not
rarely used cobra.NoArgs to indicate this.
This change does two things:
1. Introduce `cmdutil.ArgsFunc` which works like `cmdutil.RunFunc`, it
wraps an existing cobra type and lets us control the behavior when an
arguments validator fails.
2. Ensure every command sets the Args property with an instance of
cmdutil.ArgsFunc. The cmdutil package defines wrapers for all the
cobra validators we are using, to prevent us from having to spell out
`cmduitl.ArgsFunc(...)` everywhere.
Fixes#588
The prior change was incorrectly handling snapshotting of replacement
operations. Further, in hindsight, the older model of having steps
manage their interaction with the snapshot marking was clearer, so
I've essentially brought that back, merging it with the other changes.
Overhaul component model
This change overhauls our component model pretty significantly. At a top level:
* Instead of child pointers, we use parent pointers. This ensures we have a tree rather
than a graph, which was our intent, by construction. It is also easier from a state
management perspective.
* The construction of components is far less "awkward" than the prior model. Rather
than passing a callback to the parent constructor, the component simply calls done with
its computed outputs when it's finished. This checkpoints the outputs with the engine.
* The engine and CLI are a bit more "event oriented" with respect to how resource
registration occurs. This will help with parallelism when we get there #106.
* Use the component hierarchy to "style" the CLI output a bit more nicely. This entails
indenting children underneath their parents, so that it's easier to follow "why" certain
resources are getting created (in other words, attribution).
There are many, many details. I will defer to the individual checkins for details about them.
This change simplifies the necessary RPC changes for components.
Instead of a Begin/End pair, which complicates the whole system
because now we have the opportunity of a missing End call, we will
simply let RPCs come in that append outputs to existing states.
To install a package, we copied the pacakge to the destination root,
deleted the node_modules folder and then restored just the production
dependencies. Because of how we were using yarn, this meant hitting
the network, which sucks.
Make two changes:
1. Copy over the yarn.lock file we created when we did our initial
restore (e.g from `make ensure`) so yarn can reuse it.
2. Pass --offline to yarn when we do yarn install. Since we already
installed the packages previously (as part of `make ensure`) they will
be present in the cache and hence we do not need network access.
This cuts the time spent in the make install step by half or more.
We need to invoke the post-step event hook *after* updating the
state snapshots, so that it will write out the updated state.
We also need to re-serialize the snapshot again after we receive
updated output properties, otherwise they could be missing if this
happens to be the last resource (e.g., as in Stacks).
This change brings back component outputs to the overall system again.
In doing so, it generally overhauls the way we do resource RPCs a bit:
* Instead of RegisterResource and CompleteResource, we call these
BeginRegisterResource and EndRegisterResource, which begins to model
these as effectively "asynchronous" resource requests. This should also
help with parallelism (https://github.com/pulumi/pulumi/issues/106).
* Flip the CLI/engine a little on its head. Rather than it driving the
planning and deployment process, we move more to a model where it
simply observes it. This is done by implementing an event handler
interface with three events: OnResourceStepPre, OnResourceStepPost,
and OnResourceComplete. The first two are invoked immediately before
and after any step operation, and the latter is invoked whenever a
EndRegisterResource comes in. The reason for the asymmetry here is
that the checkpointing logic in the deployment engine is largely
untouched (intentionally, as this is a sensitive part of the system),
and so the "begin"/"end" nature doesn't flow through faithfully.
* Also make the engine more event-oriented in its terminology and the
way it handles the incoming BeginRegisterResource and
EndRegisterResource events from the language host. This is the first
step down a long road of incrementally refactoring the engine to work
this way, a necessary prerequisite for parallelism.
This changes our resource creation logic to tolerate missing children.
This ultimately shouldn't be necessary, but we appear to have a bug where
sometimes during deletions we end up with dangling child pointers. See
pulumi/pulumi#613; it tracks fixing the bug, and also removing this
workaround and reenabling the assertion.
When merging inputs and defaults in order to construct the set of inputs
for a call to `Create`, we must recursively merge each property value:
the provided defaults may contain nested values that must be present in
the merged result.
This change includes a NewResourceMap API, alongside the existing,
but renamed, NewResourceTree API. This also tidies up the resulting
struct for public consumption. This is required for downstream tests.
* Don't show +s, -s, and ~s deeply. The intended format here looks
more like
+ aws:iam/instanceProfile:InstanceProfile (create)
[urn=urn:pulumi:test::aws/minimal::aws/iam/instanceProfile:InstanceProfile::ip2]
name: "ip2-079a29f428dc9987"
path: "/"
role: "ir-d0a632e3084a0252"
versus
+ aws:iam/instanceProfile:InstanceProfile (create)
+ [urn=urn:pulumi:test::aws/minimal::aws/iam/instanceProfile:InstanceProfile::ip2]
+ name: "ip2-079a29f428dc9987"
+ path: "/"
+ role: "ir-d0a632e3084a0252"
This makes it easier to see the resources modified in the output.
* Print adds/deletes during updates as
- property: "x"
+ property: "y"
rather than
~ property: "x"
~ property: "y"
the latter of which doesn't really tell you what's new/old.
* Show parent indentation on output properties, so they line up correctly.
* Only print stack outputs if not undefined.
This change simply prints a stack's output properties at the end of
running the program, since we now no longer actual record the outputs
as part of component registration. Bringing that back is part of
pulumi/pulumi#340, however it is too risky to add hastily. So, for
now, we will simply special case Stacks.
This change switches from child lists to parent pointers, in the
way resource ancestries are represented. This cleans up a fair bit
of the old parenting logic, including all notion of ambient parent
scopes (and will notably address pulumi/pulumi#435).
This lets us show a more parent/child display in the output when
doing planning and updating. For instance, here is an update of
a lambda's text, which is logically part of a cloud timer:
* cloud:timer:Timer: (same)
[urn=urn:pulumi:malta::lm-cloud:☁️timer:Timer::lm-cts-malta-job-CleanSnapshots]
* cloud:function:Function: (same)
[urn=urn:pulumi:malta::lm-cloud:☁️function:Function::lm-cts-malta-job-CleanSnapshots]
* aws:serverless:Function: (same)
[urn=urn:pulumi:malta::lm-cloud::aws:serverless:Function::lm-cts-malta-job-CleanSnapshots]
~ aws:lambda/function:Function: (modify)
[id=lm-cts-malta-job-CleanSnapshots-fee4f3bf41280741]
[urn=urn:pulumi:malta::lm-cloud::aws:lambda/function:Function::lm-cts-malta-job-CleanSnapshots]
- code : archive(assets:2092f44) {
// etc etc etc
Note that we still get walls of text, but this will be actually
quite nice when combined with pulumi/pulumi#454.
I've also suppressed printing properties that didn't change during
updates when --detailed was not passed, and also suppressed empty
strings and zero-length arrays (since TF uses these as defaults in
many places and it just makes creation and deletion quite verbose).
Note that this is a far cry from everything we can possibly do
here as part of pulumi/pulumi#340 (and even pulumi/pulumi#417).
But it's a good start towards taming some of our output spew.
The first exception relates to how we launch plugins. Plugin paths are
calculated using a well-known set of rules; this makes `gas` suspicious
due to the need to use a variable to store the path of the plugin.
The second and third are in test code and aren't terribly concerning.
The latter exception asks `gas` to ignore the access key we hard-code
into the integration tests for our Pulumi test account.
The fourth exception allows use to use more permissive permissions for
the `.pulumi` directory than `gas` would prefer. We use `755`; `gas`
wants `700` or stricter. `755` is the default for `mkdir` and `.git` and
so seems like a reasonable choice for us.
By default, debugging events are not displayed by `pulumi`; the `-d`
flag must be provided to enable their display. This is necessary e.g.
to view debugging output from Terraform when TF logging is enabled.
These changes add the option to display debug output from `pulumi` to
the integration test framework.
These changes also contain a small fix for the display of component
children.
Parallelize collection of logs at each layer of the resource tree walk. Since almost all cost of log collection is at leaf nodes, this should allow the total time for log collection to be close to the time taken for the single longest leaf node, instead of the sum of all leaf nodes.
We need to clear the resource filter during the resource tree walk to ensure that logs from children of matched resources are collected and aggregated.
We have to reverse engineer the name from the soruce LogGroup information since that is all we got at runtime, but luckily that is sufficient given current name generation approach.
This kind of code is *very* sensitive to any changes to automatic name generation - but that is likely inevitable at this layer.
Outside of `.pulumiignore` we support a few "default" excludes that
try to push folks towards a pit of succes.
Previously, there was no way to opt out of these, which would be bad
if our huristics caused something youto really care about to be
elided. With this change, we add an optional setting in Pulumi.yaml
that allows you to opt out of this behavior.
As part of the work, I changed .git to be one of these "default"
excludes instead of it only happening if you had a .pulumiignore file
in a directory
Previously, we would archive every file in node_modules, however we
only actually needed the production modules. This change adds an
implicit ignorer when walking a directory that has a package.json file
in to exclude stuff under `node_modules` which is not pulled in via an
entry in the `dependencies` section of the package.json.
This change will also cause the archives that we upload to not include
either pulumi or any @pulumi/... packages, since they are not listed
in the dependencies section ofa package.json. This is fine, since we
linked in the versions we have in the cloud anyway. When we move to
using npm for these packages (instead of linking) then they will be
included.
This won't be needed once pulumi/pulumi-ppc#95 has landed and we've
updated our PPCs, but for now always add directory entires (even if
the files would be excluded) for every directory.
When working on the `.pulumiignore` support, I added a small command
that would just dump the file we sent to the service (without actually
talking to the service) so I could do some measurements on archive
sizes.
I mentioned this to Chris who said he had hacked up a similar thing
for something he was working on, so it felt worthwhile to check this
command in (hidden behind a flag) to share the implementation.
This command is hidden behind an environment variable, so customers
will not see it unless they opt in.
When deploying a project via the Pulumi.com service, we have to upload
the entire "context" of your project to Pulumi.com. The context of the
program is all files in the directory tree rooted by the `Pulumi.yaml`
file, which will often contain stuff we don't want to upload, but
previously we had no control over what would be updated (and so folks
would do hacky things like delete folders before running `pulumi
update`).
This change adds support for `.pulumiignore` files which should behave
like `.gitignore`. In addition, we were not previously compressing
files when we added them to the zip archive we uploaded and now.
By default, every .pulumiignore file is treated as if it had an
exclusion for `.git/` at the top of the file (users can override this
by adding an explicit `!.git/` to their file) since it is very
unlikely for there to ever be a reason to upload the .git folder to
the service.
Fixespulumi/pulumi-service#122
