Commit graph

2118 commits

Author SHA1 Message Date
Pat Gavlin f5b35561c6 Recursively merge properties.
When merging inputs and defaults in order to construct the set of inputs
for a call to `Create`, we must recursively merge each property value:
the provided defaults may contain nested values that must be present in
the merged result.
2017-11-28 12:32:37 -08:00
joeduffy 5762f2d0a6 Merge remote-tracking branch 'origin/resource_parenting' into resource_parenting_lite 2017-11-28 11:03:34 -08:00
joeduffy 89839038a0 Update local backend to new tree API 2017-11-28 08:13:40 -08:00
joeduffy fbff100789 Expose operations.NewResourceMap and Tree APIs
This change includes a NewResourceMap API, alongside the existing,
but renamed, NewResourceTree API.  This also tidies up the resulting
struct for public consumption.  This is required for downstream tests.
2017-11-28 07:43:07 -08:00
joeduffy 17ab1120e5 Integrate parent/child pointer changes with ops pkg 2017-11-27 11:08:04 -08:00
joeduffy c788ccfdcf Merge branch 'master' of github.com:pulumi/pulumi into resource_parenting_lite 2017-11-26 12:09:31 -08:00
joeduffy 41ae40249e Make root resources more general 2017-11-26 12:01:13 -08:00
Luke Hoban 1cfa055b2a
Merge pull request #600 from pulumi/logs
Improve logging and overhaul OperationsProvider model
2017-11-26 10:19:55 -08:00
Luke Hoban a79b3ab50f TODO and comment fixup 2017-11-26 09:57:41 -08:00
joeduffy be201739b4 Make some diff formatting changes
* Don't show +s, -s, and ~s deeply.  The intended format here looks
  more like

      + aws:iam/instanceProfile:InstanceProfile (create)
          [urn=urn:pulumi:test::aws/minimal::aws/iam/instanceProfile:InstanceProfile::ip2]
          name: "ip2-079a29f428dc9987"
          path: "/"
          role: "ir-d0a632e3084a0252"

  versus

      + aws:iam/instanceProfile:InstanceProfile (create)
        + [urn=urn:pulumi:test::aws/minimal::aws/iam/instanceProfile:InstanceProfile::ip2]
        + name: "ip2-079a29f428dc9987"
        + path: "/"
        + role: "ir-d0a632e3084a0252"

  This makes it easier to see the resources modified in the output.

* Print adds/deletes during updates as

      - property: "x"
      + property: "y"

  rather than

      ~ property: "x"
      ~ property: "y"

  the latter of which doesn't really tell you what's new/old.

* Show parent indentation on output properties, so they line up correctly.

* Only print stack outputs if not undefined.
2017-11-26 09:39:29 -08:00
joeduffy 8dc9fd027c Add back stack outputs
This change simply prints a stack's output properties at the end of
running the program, since we now no longer actual record the outputs
as part of component registration.  Bringing that back is part of
pulumi/pulumi#340, however it is too risky to add hastily.  So, for
now, we will simply special case Stacks.
2017-11-26 08:14:01 -08:00
joeduffy 86f97de7eb Merge root stack changes with parenting 2017-11-26 08:14:01 -08:00
joeduffy a2ae4accf4 Switch to parent pointers; display components nicely
This change switches from child lists to parent pointers, in the
way resource ancestries are represented.  This cleans up a fair bit
of the old parenting logic, including all notion of ambient parent
scopes (and will notably address pulumi/pulumi#435).

This lets us show a more parent/child display in the output when
doing planning and updating.  For instance, here is an update of
a lambda's text, which is logically part of a cloud timer:

    * cloud:timer:Timer: (same)
          [urn=urn:pulumi:malta::lm-cloud:☁️timer:Timer::lm-cts-malta-job-CleanSnapshots]
        * cloud:function:Function: (same)
              [urn=urn:pulumi:malta::lm-cloud:☁️function:Function::lm-cts-malta-job-CleanSnapshots]
            * aws:serverless:Function: (same)
                  [urn=urn:pulumi:malta::lm-cloud::aws:serverless:Function::lm-cts-malta-job-CleanSnapshots]
                ~ aws:lambda/function:Function: (modify)
                      [id=lm-cts-malta-job-CleanSnapshots-fee4f3bf41280741]
                      [urn=urn:pulumi:malta::lm-cloud::aws:lambda/function:Function::lm-cts-malta-job-CleanSnapshots]
                    - code            : archive(assets:2092f44) {
                        // etc etc etc

Note that we still get walls of text, but this will be actually
quite nice when combined with pulumi/pulumi#454.

I've also suppressed printing properties that didn't change during
updates when --detailed was not passed, and also suppressed empty
strings and zero-length arrays (since TF uses these as defaults in
many places and it just makes creation and deletion quite verbose).

Note that this is a far cry from everything we can possibly do
here as part of pulumi/pulumi#340 (and even pulumi/pulumi#417).
But it's a good start towards taming some of our output spew.
2017-11-26 08:14:01 -08:00
Pat Gavlin be8d604c96
Merge pull request #603 from pulumi/DebugUpdates
Optionally enable debug output in integration tests.
2017-11-24 17:00:45 -08:00
Pat Gavlin ea77c396ff
Merge pull request #604 from pulumi/NoLint
Add a few `gas` exceptions.
2017-11-24 16:28:53 -08:00
Pat Gavlin d72b85c90b Add a few gas exceptions.
The first exception relates to how we launch plugins. Plugin paths are
calculated using a well-known set of rules; this makes `gas` suspicious
due to the need to use a variable to store the path of the plugin.

The second and third are in test code and aren't terribly concerning.
The latter exception asks `gas` to ignore the access key we hard-code
into the integration tests for our Pulumi test account.

The fourth exception allows use to use more permissive permissions for
the `.pulumi` directory than `gas` would prefer. We use `755`; `gas`
wants `700` or stricter. `755` is the default for `mkdir` and `.git` and
so seems like a reasonable choice for us.
2017-11-24 16:14:43 -08:00
Pat Gavlin 503e920198 Optionally enable debug output in integration tests.
By default, debugging events are not displayed by `pulumi`; the `-d`
flag must be provided to enable their display. This is necessary e.g.
to view debugging output from Terraform when TF logging is enabled.
These changes add the option to display debug output from `pulumi` to
the integration test framework.

These changes also contain a small fix for the display of component
children.
2017-11-24 15:21:43 -08:00
Luke Hoban 6c694dd959 Use better names for container logs
Use the URN name instead of the log group name for logs reported up by `cloud:service:Service` and `cloud:task:Task` components.
2017-11-22 21:46:50 -08:00
Luke Hoban 75f7416524 Collect logs in parallel
Parallelize collection of logs at each layer of the resource tree walk.  Since almost all cost of log collection is at leaf nodes, this should allow the total time for log collection to be close to the time taken for the single longest leaf node, instead of the sum of all leaf nodes.
2017-11-22 21:33:36 -08:00
Luke Hoban 88452ff8da Include all children of resources that match resource filter in logging
We need to clear the resource filter during the resource tree walk to ensure that logs from children of matched resources are collected and aggregated.
2017-11-22 21:10:23 -08:00
Luke Hoban 9648444b05 Support for filtering logs by resource 2017-11-22 20:58:46 -08:00
Luke Hoban 8fd11f26b8 Ensure we don't skip logs in --follow mode 2017-11-22 17:18:32 -08:00
Luke Hoban e8a8bfb2c5 Fix lint errors 2017-11-22 13:08:19 -08:00
Luke Hoban 098c90419a Use correct source ID for logs from LogCollector
We have to reverse engineer the name from the soruce LogGroup information since that is all we got at runtime, but luckily that is sufficient given current name generation approach.

This kind of code is *very* sensitive to any changes to automatic name generation - but that is likely inevitable at this layer.
2017-11-22 12:47:40 -08:00
Matt Ellis 4f2c599485 Provide a way to opt out of default ignores
Outside of `.pulumiignore` we support a few "default" excludes that
try to push folks towards a pit of succes.

Previously, there was no way to opt out of these, which would be bad
if our  huristics caused something youto really care about to be
elided. With this change, we add an optional setting in Pulumi.yaml
that allows you to opt out of this behavior.

As part of the work, I changed .git to be one of these "default"
excludes instead of it only happening if you had a .pulumiignore file
in a directory
2017-11-22 12:13:44 -08:00
Matt Ellis e994f8394a Only archive production node_modules
Previously, we would archive every file in node_modules, however we
only actually needed the production modules. This change adds an
implicit ignorer when walking a directory that has a package.json file
in to exclude stuff under `node_modules` which is not pulled in via an
entry in the `dependencies` section of the package.json.

This change will also cause the archives that we upload to not include
either pulumi or any @pulumi/... packages, since they are not listed
in the  dependencies section ofa package.json. This is fine, since we
linked in the versions we have in the cloud anyway. When we move to
using npm for these packages (instead of linking) then they will be
included.
2017-11-21 17:00:49 -08:00
Matt Ellis 71ff079606 Include directory entries in archive
This won't be needed once pulumi/pulumi-ppc#95 has landed and we've
updated our PPCs, but for now always add directory entires (even if
the files would be excluded) for every directory.
2017-11-21 16:20:02 -08:00
Matt Ellis 2f985de9e4 Add developer only pulumi archive command
When working on the `.pulumiignore` support, I added a small command
that would just dump the file we sent to the service (without actually
talking to the service) so I could do some measurements on archive
sizes.

I mentioned this to Chris who said he had hacked up a similar thing
for something he was working on, so it felt worthwhile to check this
command in (hidden behind a flag) to share the implementation.

This command is hidden behind an environment variable, so customers
will not see it unless they opt in.
2017-11-21 16:02:34 -08:00
Matt Ellis f953794363 Support .pulumiignore
When deploying a project via the Pulumi.com service, we have to upload
the entire "context" of your project to Pulumi.com. The context of the
program is all files in the directory tree rooted by the `Pulumi.yaml`
file, which will often contain stuff we don't want to upload, but
previously we had no control over what would be updated (and so folks
would do hacky things like delete folders before running `pulumi
update`).

This change adds support for `.pulumiignore` files which should behave
like `.gitignore`. In addition, we were not previously compressing
files when we added them to the zip archive we uploaded and now.

By default, every .pulumiignore file is treated as if it had an
exclusion for `.git/` at the top of the file (users can override this
by adding an explicit `!.git/` to their file) since it is very
unlikely for there to ever be a reason to upload the .git folder to
the service.

Fixes pulumi/pulumi-service#122
2017-11-21 12:09:18 -08:00
Luke Hoban 512044b407 Simplify paging FilterLogEvents 2017-11-20 23:23:11 -08:00
Luke Hoban bee1b55b0b Get all pages of results 2017-11-20 23:18:47 -08:00
joeduffy 7e48e8726b Add (back) component outputs
This change adds back component output properties.  Doing so
requires splitting the RPC interface for creating resources in
half, with an initial RegisterResource which contains all of the
input properties, and a final CompleteResource which optionally
contains any output properties synthesized by the component.
2017-11-20 17:38:09 -08:00
Luke Hoban c6aac7df89 Support --since on pulumi logs
Adds a `since` flag which gets only longs since some relative offset in the past.
2017-11-20 16:37:41 -08:00
Luke Hoban af34e5cb83 Add container logging support
Also fix bug in de-duping.
2017-11-20 14:55:09 -08:00
Luke Hoban 7555eed3d5 Consolidate AWS operations 2017-11-20 13:44:23 -08:00
Luke Hoban 16ccc67654 Incorporate logs from logCollector
Also allow AWS Lambda Function logs to be projected in raw form, but filtered/formatted by higher level layers.
2017-11-20 13:31:20 -08:00
Matt Ellis e9d13dd40f Fix coverage build 2017-11-20 11:49:44 -08:00
CyrusNajmabadi 269004afb4
Show a nicer diff of our serialized functions when doing a 'pulumi update'
Also refactor and clean up a lot of the diff printing code.
2017-11-20 11:39:49 -08:00
joeduffy 86267b86b9 Merge root stack changes with parenting 2017-11-20 10:08:59 -08:00
joeduffy 5dc4b0b75c Switch to parent pointers; display components nicely
This change switches from child lists to parent pointers, in the
way resource ancestries are represented.  This cleans up a fair bit
of the old parenting logic, including all notion of ambient parent
scopes (and will notably address pulumi/pulumi#435).

This lets us show a more parent/child display in the output when
doing planning and updating.  For instance, here is an update of
a lambda's text, which is logically part of a cloud timer:

    * cloud:timer:Timer: (same)
          [urn=urn:pulumi:malta::lm-cloud:☁️timer:Timer::lm-cts-malta-job-CleanSnapshots]
        * cloud:function:Function: (same)
              [urn=urn:pulumi:malta::lm-cloud:☁️function:Function::lm-cts-malta-job-CleanSnapshots]
            * aws:serverless:Function: (same)
                  [urn=urn:pulumi:malta::lm-cloud::aws:serverless:Function::lm-cts-malta-job-CleanSnapshots]
                ~ aws:lambda/function:Function: (modify)
                      [id=lm-cts-malta-job-CleanSnapshots-fee4f3bf41280741]
                      [urn=urn:pulumi:malta::lm-cloud::aws:lambda/function:Function::lm-cts-malta-job-CleanSnapshots]
                    - code            : archive(assets:2092f44) {
                        // etc etc etc

Note that we still get walls of text, but this will be actually
quite nice when combined with pulumi/pulumi#454.

I've also suppressed printing properties that didn't change during
updates when --detailed was not passed, and also suppressed empty
strings and zero-length arrays (since TF uses these as defaults in
many places and it just makes creation and deletion quite verbose).

Note that this is a far cry from everything we can possibly do
here as part of pulumi/pulumi#340 (and even pulumi/pulumi#417).
But it's a good start towards taming some of our output spew.
2017-11-20 09:07:53 -08:00
Luke Hoban 329d70fba2 Rename cloud operations file 2017-11-19 22:31:30 -08:00
Luke Hoban 06f0559849 Refactoring of OperationsProvider code 2017-11-19 22:28:49 -08:00
Luke Hoban 0079a38ee0 Transition to resource tree 2017-11-19 17:42:54 -08:00
Luke Hoban 3f0d511cd8 Simplify log collection 2017-11-19 17:42:54 -08:00
joeduffy 8e01135572 Log the project and stack names 2017-11-19 10:16:47 -08:00
joeduffy a591775409 Add a missing copyright header 2017-11-19 08:08:30 -08:00
joeduffy 5c4c7064ba Fix a hanging ":" in the output 2017-11-19 08:08:30 -08:00
Matt Ellis 9ee15c277f Fix publishing scripts to include package.json
This is needed because install.sh will run yarn install at install
time, so we need the dependency information this provides.
2017-11-17 15:46:48 -08:00
Luke Hoban 96e4b74b15
Support for stack outputs (#581)
Adds support for top-level exports in the main script of a Pulumi Program to be captured as stack-level output properties.

This create a new `pulumi:pulumi:Stack` component as the root of the resource tree in all Pulumi programs.  That resources has properties for each top-level export in the Node.js script.

Running `pulumi stack` will display the current value of these outputs.
2017-11-17 15:22:41 -08:00
joeduffy d840a86a0a Fix outdated config error message 2017-11-17 08:53:58 -08:00