Commit graph

1194 commits

Author SHA1 Message Date
Matt Ellis 5f23a9837a Permit setting multi line config from stdin
When reading a configuration value from standard in and standard in is
not connected to a terminal, read until EOF and then trim a trailing
newline (if present) to get the value

Fixes #822
2018-02-12 15:13:19 -08:00
Luke Hoban c7adaf107d
Don't include pending-delete resources in tree representation (#902)
Resources in the checkpoint file which are pending-delete represent old versions of resources which are no longer part of the active deployment.  For purposes of constructing the active resource tree, we should skip these resources.
2018-02-12 08:47:09 -08:00
Pat Gavlin 881393146b Remove update event waits. 2018-02-09 09:43:04 -08:00
pat@pulumi.com e8e0ae9bb4 Wait between calls to fetch update logs.
As it stands, we currently hammer the service's update logs endpoint in
a tight loop while waiting for a deployment to complete. This is not
necessary, and can indeed be deletrious to the user experience: it
appears that this may be exacerbating some mysterious 500 responses from
API gateway.

These changes add a brief sleep in the relevant loop that waits for 5
seconds if the last call produced new log entries or 15 seconds if it
did not.

Fixes #844.
2018-02-07 16:57:05 -08:00
CyrusNajmabadi 275670692b
Introduce Output<T> and update Resource construction code to properly handle it. (#834)
This PR adds a new formalisms at the Resource layer.  First all inputs to a Resource are typed as ```Input<T>```.  This is either a T, ```Promise<T>``
2018-02-05 14:44:23 -08:00
Chris Smith c408068075
Add User-Agent header to CLI requests (#881)
Today we don't send any version information with API requests to the service, so we cannot make breaking changes between versions of the backend API while preserving backwards compatibility.

This PR adds a `User-Agent` header with REST requests that sends a CLI version number of "1". If the service were to make a breaking change, it could use this header to determine which response handler to use. (e.g. return a different response for "" or "1" and another for "2".) Obviously we want to avoid being in this situation, but in the event that we need to make a breaking change, we'll need this value.

We send the Pulumi version as well, though the SDK will probably rev much more quickly than the backend API client version.

Fixes #848
2018-02-05 09:38:00 -08:00
Matthew Riley 606c45ffea Keep test files on assertion failure
We were pretty careful to keep the test directory around if the test ever
exited early due to a panic or error return. But if the test ran to
completion and failed -- for example, if ExtraRuntimeValidation caused the
test to fail -- we would end up deleting the test directory.

Fixes #868
2018-02-01 23:35:49 -08:00
Matt Ellis 818246a708 Allow control of uploaded archive root in Pulumi.yaml
Previously, when uploading a projectm to the service, we would only
upload the folder rooted by the Pulumi.yaml for that project. This
worked well, but it meant that customers needed to structure their
code in a way such that Pulumi.yaml was always as the root of their
project, and if they wanted to share common files between two projects
there was no good solution for doing this.

This change introduces an optional piece of metadata, named context,
that can be added to Pulumi.yaml, which allows controlling the root
folder used for computing the root folder to archive from.  When it is
set, it is combined with the location of the Pulumi.yaml file for the
project we are uploading and that folder is uses as the root of what
we upload to the service.

Fixes: #574
2018-01-31 16:22:58 -08:00
Matt Ellis 39dbdc98e9 Clean up colorization logic
The existing logic would flow colorization information into the
engine, so depending on the settings in the CLI, the engine may or may
not have emitted colorized events. This coupling is not great and we
want to start moving to a world where the presentation happens
exclusively at the CLI level.

With this change, the engine will always produce strings that have the
colorization formatting directives (i.e. the directives that
reconquest/loreley understands) and the CLI will apply
colorization (which could mean either running loreley to turn the
directives into ANSI escape codes, or drop them or retain them, for
debuging purposes).

Fixes #742
2018-01-31 15:46:14 -08:00
Matt Ellis b1496f3051 Remove Document and Location
These types are no longer used as pulumi no longer reads and evaluates
source code.

Contributes to #441
2018-01-30 16:42:39 -08:00
Chris Smith 63dfd1e302
Add Manifest to GetStackResponse (#854)
Add Manifest to GetStackResponse.
2018-01-30 14:12:11 -08:00
Luke Hoban 0e4fcc16e3
Pass target Cloud and PPC to integation test Reporter (#859)
Ensure that we capture this information in our test reporting so that we can filter on it in queries and reports.
2018-01-30 13:10:32 -08:00
Matthew Riley b0905d99aa Fix long line 2018-01-30 10:52:42 -08:00
Matthew Riley c74db59231 Keep embedded newlines in Lambda log messages
Set the `s` flag in the regexp so `.` matches `\n`.

Add an explicit `\n` to the regexp to account for the newline inserted by Lambda.
2018-01-29 23:33:59 -08:00
Matt Ellis 07fd36c369 Remove deployOptions.Create
This was entirely unused.

Fixes #813
2018-01-29 16:23:55 -08:00
Matt Ellis 803ba93bf5 Pass --network-concurrency 1 to yarn
We've been hitting issues in CI that look like yarnpkg/yarn#4563 and
our hope is that this addresses the issue
2018-01-29 11:49:42 -08:00
Pat Gavlin 842a16cee4 Always set UpdateOptions.DryRun for previews.
If we do not do this, the engine will assume that it should elide
unknown input/output properties when interacting with the language
and resource providers and we will produce unexpected results.
2018-01-26 16:05:42 -08:00
Chris Smith 001d187c90
Add not-started UpdateResult
Add a "not-started" `UpdateResult` type, for supporting updates tracked by the PPC that have not been applied yet.
2018-01-25 14:23:49 -08:00
Chris Smith 4c217fd358
Add "pulumi history" command (#826)
This PR adds a new `pulumi history` command, which prints the update history for a stack.

The local backend stores the update history in a JSON file on disk, next to the checkpoint file. The cloud backend simply provides the update metadata, and expects to receive all the data from a (NYI) `/history` REST endpoint.

`pkg/backend/updates.go` defines the data that is being persisted. The way the data is wired through the system is adding a new `backend.UpdateMetadata` parameter to a Stack/Backend's `Update` and `Destroy` methods.

I use `tests/integration/stack_outputs/` as the simple app for the related tests, hence the addition to the `.gitignore` and fixing the name in the `Pulumi.yaml`.

Fixes #636.
2018-01-24 18:22:41 -08:00
Matt Ellis ce05cce77f Provide a rudimentary progress spinner
Previously, the `pulumi` tool did not show any indication of progress
when doing a deployment. Combined with the fact that we do not create
resources in parallel it meant that sometime `pulumi` would appear to
hang, when really it was just waiting on some resource to be created
in AWS. In addition, some AWS resources take a long time to create and
CI systems like travis will kill the job if there is no output. This
causes us (and our customers) to have to do crazy dances where we
launch shell scripts that write a dot to the console every once in a
while so we don't get killed. While we plan to overhaul the output
logic (see #617), we take a first step towards interactivity by simply
having a nice little spinner (in the interactive case) and when run
non interactive have `pulumi` print a message that it is still
working.

Fixes #794
2018-01-22 14:21:08 -08:00
Matthew Riley 4bb86116d0
Merge pull request #821 from pulumi/recover-in-test
Avoid resource leaks by recovering from panics in integration tests
2018-01-22 10:08:15 -08:00
Joe Duffy d6c798ee52
Add a commentChars parameter to EmitHeaderWarning (#827)
Now that we're using this to generate multiple languages, this
needs to be parameterized.  Splitting out into its own change.
2018-01-22 09:32:37 -08:00
Chris Smith 6f6fca7592
Export resource change counts from engine (#823)
This PR exports the aggregate resource changes for update and destroy operations. We'll use this information in #636 when summarizing previous updates.

I initially started with a new struct that had fields like `Created`, `Deleted`, `Unchanged`, etc. But it became cumbersome with the seven different type of resource operations we perform. So instead went with the more flexible `map[deploy.StepOp]int`.
2018-01-20 19:15:19 -08:00
Chris Smith a4f087460f
Fix false-positives in login verification (#825)
Surprisingly `pulumi login -c https://google.com` would succeed. This was because we were too lax in our way of validating credentials. We take the provided cloud URL and call the "GetCurrentUserHandler" method. But we were only checking that it returned a successful response, not that it was actually valid JSON.

So in the "https://google.com" case, Google returned HTML describing a 404 error, but since the sever response was 200, the Pulumi CLI assumed things were on the up and up.

We now parse the response as JSON, and confirm the response has a `name` property that is non-nil. This heuristic covers the majority of false-positive cases, but without us needing to move all of the service's API shape for users, which includes organizations, which includes Clouds, etc. into `pulumi`.

Fixes https://github.com/pulumi/pulumi-service/issues/457. As an added bonus, we now return a much more useful error message.
2018-01-20 19:11:38 -08:00
Chris Smith f21c462e1b Report no changes for dry-runs 2018-01-20 12:15:28 -08:00
Chris Smith 7a5ac5ce06 Export resource change counts from engine 2018-01-20 12:07:03 -08:00
Matthew Riley 580bbcb3ca Avoid resource leaks by recovering from panics in integration tests 2018-01-20 11:48:42 -08:00
Matt Ellis cc04cd6581 Use per stack key for local stacks instead of per project
In the Pulumi Cloud, there is no guarantee that two stacks will share
the same encryption key. This means that encrypted config can not be
shared across stacks (in the Pulumi.yaml) file. To mimic this behavior
in the local experience, we now use a unique key per stack.

When upgrading an existing project, for any stack with existing
secrets, we copy the existing key into this stack. Future stacks will
get thier own encryption key. This strikes a balance between
expediency of implementation, the end user UX and not having to make a
breaking change.

As part of this change, I have introduced a CHANGELOG.md file in the
root of the repository and added a small note about the change to it.

Fixes #769
2018-01-19 00:50:59 -08:00
Matt Ellis 22938f07be Pretty-print some JSON files we persist to disk
We did not pretty print either the workspace settings file or the
repository settings file, but pretty print other files like the
credentials file and checkpoints. Now we do.

Fixes #540
2018-01-18 20:35:53 -08:00
Chris Smith 3a3d0698ae
Surface update options to the service (#806)
This PR surfaces the configuration options available to updates, previews, and destroys to the Pulumi Service. As part of this I refactored the options to unify them into a single `engine.UpdateOptions`, since they were all overlapping to various degrees.

With this PR we are adding several new flags to commands, e.g. `--summary` was not available on `pulumi destroy`.

There are also a few minor breaking changes.

- `pulumi destroy --preview` is now `pulumi destroy --dry-run` (to match the actual name of the field).
- The default behavior for "--color" is now `Always`. Previously it was `Always` or `Never` based on the value of a `--debug` flag. (You can specify `--color always` or `--color never` to get the exact behavior.)

Fixes #515, and cleans up the code making some other features slightly easier to add.
2018-01-18 11:10:15 -08:00
Matt Ellis c506549a25 Remove MustFprintf in favor of explicitly dropping errors
In travis, we've seen cases where writes to our standard streams
results in an error like: `/dev/stderr: resource temporarily
unavailable` which causes the tests to panic.

Now, in a perfect world, writes to /dev/stderr would not fail in this
way, but we do not live in a perfect world. Other processes on the
machine may make stderr/stdout non-blocking. We've are now seeing this
failure in Travis more often and it is masking real Pulumi failures
we want to debug.
2018-01-16 18:33:44 -08:00
Joe Duffy 34984ba1cb
Delete the old IDL compiler (#801)
It is moving to a new location: https://github.com/pulumi/pidlc
2018-01-13 15:11:52 -08:00
Joe Duffy 0d107c742a
Restructure test framework to ease multiple languages (#799)
This change restructures the test framework code a bit, to make it
easier to introduce additional languages.  Our knowledge of Yarn and
Node.js project structure, for instance, was previously baked in to
the test logic, in a way that was hard to make, for instance, Yarn
optional.  (In Python, of course, it will not be used.)  To better
support this, I've moved some state onto a new programTester struct
that we can use to lazily find binaries required during the testing
(such as Yarn, Pip, and so on).  I'm committing this separately so
that I can minimize merge conflicts in the Python work.
2018-01-12 17:10:53 -08:00
Matthew Riley 3620cdb243
Merge pull request #792 from pulumi/fail-test
Keep test temp directories around on failure
2018-01-12 12:04:58 -08:00
khyperia 1a0cf9ff1b
Respond to feedback 2018-01-11 15:20:20 -08:00
khyperia 2598090cb7
Formatting 2018-01-11 13:46:34 -08:00
khyperia 08f3c623e6
Fix some style issues 2018-01-11 12:05:08 -08:00
khyperia 0f73b8c395
Move types to pulumi-service 2018-01-11 11:51:31 -08:00
khyperia d4de351b10
Correct some field types 2018-01-11 10:57:25 -08:00
khyperia 916ba6cd39
Add UpdateProgramRequestUntyped 2018-01-11 10:42:43 -08:00
khyperia f455689bd4
Add apitypes from pulumi-ppc 2018-01-10 15:43:44 -08:00
khyperia 3c644d243f
Create apitype package 2018-01-10 15:04:55 -08:00
Matthew Riley 047a6c3c56 Make "delete temp dir" condition explicit
Previously we checked if the test had failed -- but the call to
`assert.NoError` that actually fails the test sometimes hasn't happened yet.
2018-01-09 23:35:26 -08:00
Matthew Riley d688910360 Remove named parameters from CopyTestToTemporaryDirectory
Fix two references to the now-unnamed `dir` that should have been to
other variables.

Check a real condition before the deferred call to RemoveAll instead of
checking the error return.
2018-01-09 23:35:26 -08:00
Pat Gavlin 208b7296be
Merge pull request #791 from pulumi/AllowUnknownsPreview
Supply unknown properties to providers during preview.
2018-01-09 19:15:17 -08:00
Pat Gavlin 88a22515df Supply unknown properties to providers during preview.
My previous change to stop supplying unknown properties to providers
broke `pulumi preview` in the case of unknown inputs. This change
restores the previous behavior for previews only; the new unknown-free
behavior remains for applies.

Fixes #790.
2018-01-09 18:41:47 -08:00
Matthew Riley 3c070d3ad5 Restore support for non-Additive edits
None were used in this repository, but `pulumi-aws` used them.
2018-01-09 17:02:30 -08:00
Matthew Riley 9a869eb2d7 Print output of failed commands 2018-01-09 17:02:30 -08:00
Matthew Riley d9fa8237be Use length of correct string for prefixer 2018-01-09 16:48:04 -08:00
pat@pulumi.com f3cb37ef95 Do not expose unknowns to resource providers.
Before these changes, we were inconsistent in our treatment of unknown
property values across the resource provider RPC interface. `Check` and
`Diff` were retaining unknown properties in inputs and outputs;
`Create`, `Update`, and `Delete` were not. This interacted badly with
recent changes to `Check` to return all provider inputs--i.e. not just
defaults--from that method: if an unknown input was provided, it would
be present in the returned inputs, which would eventually confuse the
differ by giving the appearance of changes where none were present.

These changes remove unknowns from the provider interface entirely:
unknown property values are never passed to a provider, and a provider
must never return an unknown property value.

This is the primary piece of the fix for pulumi/pulumi-terraform#93.
2018-01-09 12:21:47 -08:00
pat@pulumi.com 5f28d52e00 Remove the Engine type entirely.
It no longer carries any state. All of its methods are now package-level
functions.
2018-01-08 14:20:51 -08:00
pat@pulumi.com c56e716c31 Refactor the engine's entrypoints.
These changes refactor the engine's entrypoints--Deploy, Destroy, and
Preview--to be update-centric rather than stack-centric. Each of these
methods now takes a value of a new type, Update, that abstracts away the
vagaries of fetching and maintaining the update's state. This
refactoring also reinforces Pulumi.yaml as a CLI concept rather than an
engine concept; the CLI is now the only reader/writer of this format.

These changes will smooth the way for a few refactorings on the service
side that will aid in update isolation.
2018-01-08 14:15:16 -08:00
Matthew Riley 9e3976513c AssertNoError instead of Assert(err == nil)
This may not be exhaustive, but I replaced all instances I could find.
2018-01-08 13:46:21 -08:00
Matthew Riley ff10b65f3a Add environment variable to keep data for failed tests
Useful to keep command output and maybe-still-relevant checkpoints around.
2018-01-08 13:46:21 -08:00
Matthew Riley 594ccf755a Write command output to log file 2018-01-08 13:46:21 -08:00
Matthew Riley b0eed85871 Remove non-additive edits
Now we know we only create one temporary directory per test
2018-01-08 13:46:21 -08:00
Matthew Riley afb734a712 Unexport a few private utility functions
Makes it bit easier to see the module entrypoints.
2018-01-08 13:46:21 -08:00
Matt Ellis 607b4990f5 Remove stack configuration when deleting a stack
When a stack is removed, we should delete any configuration it had
saved in either the Pulumi.yaml file or the workspace.

Fixes #772
2018-01-07 17:38:40 -08:00
Pat Gavlin 3ca874d3f7
Merge pull request #781 from pulumi/ExportImport
Add the ability to {ex,im}port a stack's deployment.
2018-01-05 17:56:16 -08:00
pat@pulumi.com 7782a83030 Appease the linters. 2018-01-05 17:35:22 -08:00
pat@pulumi.com b96217341f Add the ability to {ex,im}port a stack's deployment.
These changes add the ability to export a stack's latest deployment or
import a new deployment to a stack via the Pulumi CLI. These
capabilities are exposed by two new verbs under `stack`:
- export, which writes the current stack's latest deployment to stdout
- import, which reads a new deployment from stdin and applies it to the
  current stack.

In the local case, this simply involves reading/writing the stack's
latest checkpoint file. In the cloud case, this involves hitting two new
endpoints on the service to perform the export or import.
2018-01-05 16:22:31 -08:00
Chris Smith decf814278
Enable running integration tests against the service (#775)
This PR updates the `pkg/testing/integration` package to support running integration tests against the Pulumi Service if desired. This is done through adding new options to `ProgramTestOptions`. (Generally adding support for providing values to flags that were previously inaccessible.)

I added an integration test to confirm that it all works if the PULUMI_API environment variable is set. These tests aren't run in Travis, only manually. Since we cannot reliably run tests from `master` against the service because of the delay in rolling out updates to the Pulumi SDK, etc.
2018-01-03 21:26:50 -08:00
Matt Ellis c052e24370
Merge pull request #771 from pulumi/no-all-for-secrets
Do not allow encrypted global configuration
2018-01-03 15:04:00 -08:00
joeduffy b3ee139b91 Fix error message for stack select failures 2017-12-28 16:51:53 -08:00
Matt Ellis f510f3c914 Do not allow encrypted global configuration
The cloud backend does not support this because it computes an
encryption key per stack, so we should not support this in the CLI.

Fixes #770
2017-12-27 19:00:55 -08:00
Joe Duffy d419229301
Add additional linting (#768)
This adds additional linting checks.  Most importantly, it will
check calls to our custom format routines for missing arguments.
2017-12-27 17:10:12 -08:00
joeduffy 5ac81c2818 Add missing Infof argument 2017-12-27 14:54:27 -08:00
Joe Duffy 86c1e7ad39
Merge pull request #764 from pulumi/763_wait_for_update_resiliency
Make the CLI's waitForUpdates more resilient to transient failure
2017-12-27 06:34:08 -08:00
joeduffy 87079589f1 Use the retry framework for REST API retries
This change incorporates feedback on https://github.com/pulumi/pulumi/pull/764,
in addition to refactoring the retry logic to use our retry framework rather
than hand-rolling it in the REST API code.  It's a minor improvement, but at
least lets us consolidate some of this logic which we'll undoubtedly use more
of over time.
2017-12-26 10:24:08 -08:00
Luke Hoban 56fd8bc257
Fix REST API calls for encrypt/decrypt (#767)
The merge of two recent changes seems to have led to these calls invoking the API incorrectly.
2017-12-26 09:42:42 -08:00
joeduffy db5318b0a5 Make the CLI's waitForUpdates more resilient to transient failure
We saw an issue where a user was mid-update, and got a networking
error stating `read: operation timed out`.  We believe this was simply
a local client error, due to a flaky network.  We should be resilient
to such things during updates, particularly when there's no way to
"reattach" to an in-progress udpate (see pulumi/pulumi#762).

This change accomplishes this by changing our retry logic in the
cloud backend's waitForUpdates function.  Namely:

* We recognize three types of failure, and react differently:

    - Expected HTTP errors.  For instance, the 504 Gateway Timeouts
      that we already retried in the face of.  In these cases, we will
      silently retry up to 10 times.  After 10 times, we begin warning
      the user just in case this is a persistent condition.

    - Unexpected HTTP errors.  The CLI will quit immediately and issue
      an error to the user, in the usual ways.  This covers
      Unauthorized among other things.  Over time, we may find that we
      want to intentionally move some HTTP errors into the above.

    - Anything else.  This covers the transient networking errors case
      that we have just seen.  I'll admit, it's a wide net, but any
      instance of this error issues a warning and it's up to the user
      to ^C out of it.  We also log the error so that we'll see it if
      the user shares their logs with us.

* We implement backoff logic so that we retry very quickly (100ms)
  on the first failure, and more slowly thereafter (1.5x, up to a max
  of 5 seconds).  This helps to avoid accidentally DoSing our service.
2017-12-26 09:40:51 -08:00
Pat Gavlin a7fbdff7ea
Merge pull request #761 from pulumi/UpdateNoMerge
Do not merge old state and inputs for Update.
2017-12-24 14:47:15 -08:00
Pat Gavlin 1bd1eaff50 Do not merge old state and inputs for Update.
This merging causes similar issues to those it did in `Check`, and
differs from the approach we take to `Diff`. This can causes problems
such as an inability to remove properties.
2017-12-22 18:18:14 -08:00
Luke Hoban bc95745fb6
Update regexp to match changes in AWS generated names (#760)
Also keeps support for older name generation pattern.
2017-12-22 14:49:42 -08:00
Pat Gavlin e4d9eb6fd3 Support secrets for cloud stacks.
Use the new {en,de}crypt endpoints in the Pulumi.com API to secure
secret config values. The ciphertext for a secret config value is bound
to the stack to which it applies and cannot be shared with other stacks
(e.g. by copy/pasting it around in Pulumi.yaml). All secrets will need
to be encrypted once per target stack.
2017-12-22 07:59:27 -08:00
joeduffy af087103b9 Fix more edit directory issues
This fixes a few more edit directory issues, where we didn't
correctly propagate the changes in edit directory required during
subsequent destroy/stack activities.  It also fixes a few error
paths so that we preserve the right directory to be removed.
2017-12-22 06:58:31 -08:00
Luke Hoban 7cc782e856
Add support for pulumi logs --since when targeting cloud backend (#756)
Hooks up the CLI to be able to pass query parameters through to the
service.

Part of pulumi/pulumi-service#431.
2017-12-21 17:16:45 -08:00
Joe Duffy b158e9804b
Fix test edit directories (#755)
The test edit directory logic needs to carry forward the checkpoint
from the *prior edit*, not the *original*, as it sequences through edits.
2017-12-21 11:01:30 -08:00
joeduffy 2eda90d67b Fix a dangling fmt parameter 2017-12-21 09:18:00 -08:00
Joe Duffy bc2cf55463
Implement resource protection (#751)
This change implements resource protection, as per pulumi/pulumi#689.
The overall idea is that a resource can be marked as "protect: true",
which will prevent deletion of that resource for any reason whatsoever
(straight deletion, replacement, etc).  This is expressed in the
program.  To "unprotect" a resource, one must perform an update setting
"protect: false", and then afterwards, they can delete the resource.

For example:

    let res = new MyResource("precious", { .. }, { protect: true });

Afterwards, the resource will display in the CLI with a lock icon, and
any attempts to remove it will fail in the usual ways (in planning or,
worst case, during an actual update).

This was done by adding a new ResourceOptions bag parameter to the
base Resource types.  This is unfortunately a breaking change, but now
is the right time to take this one.  We had been adding new settings
one by one -- like parent and dependsOn -- and this new approach will
set us up to add any number of additional settings down the road,
without needing to worry about breaking anything ever again.

This is related to protected stacks, as described in
pulumi/pulumi-service#399.  Most likely this will serve as a foundational
building block that enables the coarser grained policy management.
2017-12-20 14:31:07 -08:00
Joe Duffy 26021a41a3
Add the ability to do negative tests
This change adds the ability to do very coarse-grained negative
tests in our integration test framework.  Either a test itself,
or one of its edits, may be marked ExpectFailure == true, at which
point either the preview or update MUST fail (and, if one fails
without this being set, we still treat it as an error).
2017-12-20 12:10:46 -08:00
Luke Hoban d5acc3da87
Add option to specify command line flags to pulumi update in integration tests (#749)
And use this in the diff test which needs to apply `--colors=raw`.
2017-12-20 11:17:25 -08:00
Joe Duffy 680a09a164
Add fewer leading prefixes (#741) 2017-12-18 14:51:23 -08:00
Joe Duffy 3311d32c2d
Merge pull request #728 from pulumi/hideGoop
Elide our serialized function preamble and postamble when printing user code
2017-12-18 13:22:13 -08:00
Joe Duffy f0c28db639
Attempt to fix colorization (#740)
Our recent changes to colorization changed from a boolean to a tri-valued
enum (Always, Never, Raw).  The events from the service, however, are still
boolean-valued.  This changes the message payload to carry the full values.
2017-12-18 11:42:32 -08:00
Joe Duffy 662ebcbd94
Merge pull request #666 from pulumi/add-logging-to-stack-reading
Log the error when getting a stack fails
2017-12-18 09:51:29 -08:00
Joe Duffy 6dc16a5548
Make cloud authentication more intuitive (#738)
The prior behavior with cloud authentication was a bit confusing
when authenticating against anything but https://pulumi.com/.  This
change fixes a few aspects of this:

* Improve error messages to differentiate between "authentication
  failed" and "you haven't logged into the target cloud URL."

* Default to the cloud you're currently authenticated with, rather
  than unconditionally selecting https://pulumi.com/.  This ensures

      $ pulumi login -c https://api.moolumi.io
      $ pulumi stack ls

  works, versus what was currently required

      $ pulumi login -c https://api.moolumi.io
      $ pulumi stack ls -c https://api.moolumi.io

  with confusing error messages if you forgot the second -c.

* To do this, our default cloud logic changes to

    1) Prefer the explicit -c if supplied;

    2) Otherwise, pick the "currently authenticated" cloud; this is
       the last cloud to have been targeted with pulumi login, or
       otherwise the single cloud in the list if there is only one;

    3) https://pulumi.com/ otherwise.
2017-12-16 07:49:41 -08:00
Joe Duffy bd2baa5091
Pass planning flag to print resource outputs (#737)
We hadn't previously passed the planning flag when printing resource
outputs, meaning any computed ones now are being printed as "undefined".
Instead, we prefer to see the "computed<string>" type name.
2017-12-16 07:33:58 -08:00
Pat Gavlin 9d038c1d88 Only pass old inputs to Check.
We do not need all of the information in the old state for this call, as
outputs will not be read by the provider during validation or defaults
computation.
2017-12-15 11:04:43 -08:00
joeduffy 668e56f2c7 Use old.All() to be consistent with Update
This changes the inputs to Check/Diff to match what we also
pass to the Update function later on (old.All() vs. old.Outputs).
2017-12-15 07:37:15 -08:00
joeduffy 675fe38269 Add more tracing context to RPC marshaling
This change adds a bit more tracing context to RPC marshaling
logging so that it's easier to attribute certain marshaling calls.
Prior to this, we'd just have a flat list of "marshaled property X"
without any information about what the marshaling pertained to.
2017-12-15 07:22:49 -08:00
joeduffy 8492e6100f Pass old *outputs*, not *inputs*, to Check/Diff
This change passes a resource's old output state, so that it contains
everything -- defaults included -- for purposes of the provider's diffing.
Not doing so can lead the provider into thinking some of the requisite
state is missing.
2017-12-15 07:01:11 -08:00
CyrusNajmabadi a5cf6f25d6
Add a framework for baselining and validating pulumi-update diff output. (#700) 2017-12-14 17:10:05 -08:00
Cyrus Najmabadi 84051e84b3 Make linter happy. 2017-12-14 15:41:53 -08:00
Cyrus Najmabadi 1fc33741cc Strip out even more. 2017-12-14 15:02:13 -08:00
Cyrus Najmabadi 1ab2680df8 Elide our serialized function preamble and postamble when printing user code. 2017-12-14 14:55:27 -08:00
CyrusNajmabadi 9bb9295594
Suppress printing out the variables we capture. (#724) 2017-12-14 14:40:04 -08:00
CyrusNajmabadi e4946a6620
Allow users to control if and how output is colorized. (#718)
Part of the work to make it easier to tests of diff output.  Specifically, we now allow users to pass --color=option for several pulumi commands.  'option' can be one of 'always', 'never', 'raw', and 'auto' (the default).  

The meaning of these flags are:

1. auto: colorize normally, unless in --debug 
2. always: always colorize no matter what
3. never: never colorize no matter what.
4. raw: colorize, but preserve the original "<{%%}>" style control codes and not the translated platform specific codes.   This is for testing purposes and ensures we can have test for this stuff across platform.
2017-12-14 11:53:02 -08:00
Luke Hoban 6f15fa8ed8
Pass more stack info to ExtraRuntimeValidation (#717)
This will allow us to remove a lot of current boilerplate in individual tests, and move it into the test harness.

Note that this will require updating users of the integration test framework.  By moving to a property bag of inputs, we should avoid needing future breaking changes to this API though.
2017-12-13 16:09:14 -08:00
joeduffy 92ea5b5bdd Add a test case for delete-before-recreate 2017-12-13 10:47:18 -08:00
joeduffy 3a13621c32 Add rudimentary delete-before-create support
This change adds rudimentary delete-before-create support (see
pulumi/pulumi#450).  This cannot possibly be complete until we also
implement pulumi/pulumi#624, becuase we may try to delete a resource
while it still has dependent resources (which almost certainly will
fail).  But until then, we can use this to manually unwedge ourselves
for leaf-node resources that do not support old and new resources
living side-by-side.
2017-12-13 10:47:18 -08:00
Joe Duffy 36ab8f0087
Make config a little less error prone
As articulated in #714, the way config defaults to workspace-local
configuration is a bit error prone, especially now with the cloud
workflow being the default.  This change implements several improvements:

* First, --save defaults to true, so that configuration changes will
  persist into your project file.  If you want the old local workspace
  behavior, you can specify --save=false.

* Second, the order in which we applied configuration was a little
  strange, because workspace settings overwrote project settings.
  The order is changed now so that we take most specific over least
  specific configuration.  Per-stack is considered more specific
  than global and project settings are considered more specific
  than workspace.

* We now warn anytime workspace local configuration is used.  This
  is a developer scenario and can have subtle effects.  It is simply
  not safe to use in a team environment.  In fact, I lost an arm
  this morning due to workspace config... and that's why you always
  issue warnings for unsafe things.
2017-12-13 10:46:54 -08:00
Joe Duffy dc83a49a0d
Merge pull request #713 from pulumi/712_missing_error_prop
Add a missing error propagation
2017-12-13 09:23:18 -08:00
joeduffy 664d4f3bfe Add a missing error propagation
If the service returns a 504, we happily keep looping around and
retrying until we get a valid update.  Unfortunately, we missed the
else condition, which is what happens when this isn't a 504, leading
us to swallow real errors (500 and the like).  Trivial fix.

Fixes pulumi/pulumi#712.
2017-12-13 08:33:11 -08:00
Joe Duffy 2a8ab4d38e
Merge pull request #708 from pulumi/fix-660
Do not buffer output during planning
2017-12-12 16:02:48 -08:00
pat@pulumi.com 55df0cc201 PR feedback 2017-12-12 14:32:27 -08:00
pat@pulumi.com fcdc3495a4 Render computeds differently during updates.
Rather than displaying `computed` as we do during previews, display
`undefined` instead, as these values may be intentionally undefined by
the user's program.

Fixes #633.
2017-12-12 14:20:44 -08:00
Matt Ellis 8d998fea81 Do not buffer output during preview
Previously, we would compute all the output from the preview operation
and then display it as a single event. Instead, we should do this how
we handle things during deploy. Each logical print operaton gets a
single event and we stream them back over a channel during planning.

Fixes #660
2017-12-12 12:41:56 -08:00
Joe Duffy b6a386995a
Set pwd for plugins (#706)
This change just flows the project's "main" directory all the way
through to the plugins, fixing #667.  In that work item, we discussed
alternative approaches, such as rewriting the asset paths, but this
is tricky because it's very tough to do without those absolute paths
somehow ending up in the checkpoint files.  Just launching the
processes with the right pwd is far easier and safer, and it turns
out that, conveniently, we set up the plugin context in exactly the
same place that we read the project information.
2017-12-12 12:31:09 -08:00
Pat Gavlin d0df98e03b Close assets while creating ZIP archives.
This was an unintentional omission from my earlier change that
refactored the archive code.

Fixes pulumi/pulumi-ppc#133.
2017-12-12 10:38:27 -08:00
CyrusNajmabadi eea5232cce
Break out additional test helper. (#699) 2017-12-11 15:48:45 -08:00
Luke Hoban 3b461ffcb3
Use AWS access key in operations prvider when available (#697)
If the stack config specifies AWS credentials, these should be used in the operations provider instead of ambient credentials.  This is necessary to ensure that we have access to resources in the target account.

Fixes pulumi/pulumi-service#389
2017-12-11 15:40:39 -08:00
CyrusNajmabadi 9bc9ce1505
More test code refactoring.
* Move code down.

* Extact test code out.

* Extract helper and pattern.

* Remove unnecessary code.

* move comment.
2017-12-11 14:42:42 -08:00
Joe Duffy 5a1b1253ab
Save configuration under the stack by default (#696) 2017-12-11 14:41:57 -08:00
Joe Duffy e93fe05efe
Reparameterize NewUniqueHex/ID
This change adds a randlen parameter to NewUniqueHex/ID so that the
caller can decide how long of a random string to generate.
2017-12-10 07:44:11 -08:00
Luke Hoban 90c7305f55
Update logs after changes in pulumi/cloud names (#684) 2017-12-09 17:27:43 -08:00
CyrusNajmabadi 7234cdef02
Take an options pointer so values can change as a test runs. (#679)
* Take an options pointer so values can change as a test runs.

* Don't pass redundant information.

* Extract initialization routine.

* Fix caller.

* Check return value.

* Extract destruction logic.

* Move preview and update into their own function.

* Inline null check.
2017-12-08 16:59:28 -08:00
Joe Duffy 1681119339
Add a stack output command (#675)
This change adds a `pulumi stack output` command.  When passed no
arguments, it prints all stack output properties, in exactly the
same format as `pulumi stack` does (just without all the other stuff).
More importantly, if you pass a specific output property, a la
`pulumi stack output clusterARN`, just that property will be printed,
in a scriptable-friendly manner.  This will help us automate wiring
multiple layers of stacks together during deployments.

This fixes pulumi/pulumi#659.
2017-12-08 13:14:58 -08:00
CyrusNajmabadi edb7bf849e
Reverts (#676)
* Revert "Make sure we properly update dir so that pulumi-destroy works."

This reverts commit 56bfc57998.

* Revert "Edits needs to continuously pass along the new directory. (#668)"

This reverts commit 8bd1822722.

* Revert "Refactor test code to make it simpler to validate code in the middle. (#662)"

This reverts commit ed65360157.
2017-12-08 12:59:39 -08:00
Joe Duffy d89a2b4e1f
Add a logout --all command (#673)
If a cloud you've previously authenticated with goes away -- as ours
sort of did, because the cloud endpoing in the CLI changed (to actually
be correct) -- then you can't logout without manually editing the
credentials file in your workspace.  This is a little annoying.  So,
rather than that, let's have a `pulumi logout --all` command that just
logs out of all clouds you are presently authenticated with.
2017-12-08 12:14:14 -08:00
CyrusNajmabadi 56bfc57998
Make sure we properly update dir so that pulumi-destroy works. 2017-12-08 11:37:38 -08:00
Chris Smith ebc1776eb9 Absorb breaking change from Service/PPC API 2017-12-08 10:21:25 -08:00
Chris Smith 8c3c2c97f5 Add messages to asserts 2017-12-08 10:21:25 -08:00
Chris Smith 527a1c97e7 Fix panic with nil message entry 2017-12-08 10:21:25 -08:00
CyrusNajmabadi 8bd1822722
Edits needs to continuously pass along the new directory. (#668) 2017-12-08 01:51:35 -08:00
Matt Ellis 94b9489759 Log the error when getting a stack fails 2017-12-07 18:24:51 -08:00
CyrusNajmabadi ed65360157
Refactor test code to make it simpler to validate code in the middle. (#662)
* Refactor test code to make it simpler to validate code in the middle.
2017-12-07 16:29:48 -08:00
Joe Duffy aa7c1598ac
Show root stacks and more outputs (#661)
This changes two things:

1) Always show the root stack, even when unchanged.  This ensures
   that you see the outputs during any updates, etc., which is nice.

2) Always show all outputs after operations.  Before, we did some
   diffing between old/new, which actually doesn't make much sense.
2017-12-07 13:33:16 -08:00
Chris Smith e104ebd7f6
Several fixes relating to Pulumi clouds and auth (#651)
- Fix #644 "Re-enable use of local dev stacks". Rather than trying to be smart about using "pulumi.com" and inferring "api.pulumi.com", we just use whatever cloud URL the user provides. (e.g. "http://localhost:8080") We can improve the user experience later by providing friendly names for these things upon login. So we can show "Pulumi" or "Contoso" instead of the URL -- but let's cross that bridge later.

- Fix #640, "Better error on `pulumi login`". We only provide the more friendly error about needing to login IFF you are trying to use an authenticated Pulumi API without having creds.
2017-12-07 10:59:20 -08:00
Joe Duffy 971f6189f2
Fix pending delete replacement failure (#658)
The two-phase output properties change broke the ability to recover
from a failed replacement that yields pending deletes in the checkpoint.
The issue here is simply that we should remember pending registrations
only for logical operations that *also* have a "new" state (create or
update).  This commit fixes this, and also adds a new step test with
fault injection to probe many interesting combinations of steps.
2017-12-07 09:44:38 -08:00
Joe Duffy 9680fc3604
Issue an error when a project file isn't found 2017-12-06 17:26:45 -08:00
Matt Ellis ffdee8bb23 Relax error check when trying to kill child processes
If the process we are trying to kill has already exited, don't treat
this as an error. This can happen when we snapshot the process tree
before the process exits but it has exited by the time we get to
trying to kill it.

Fixes #654
2017-12-06 09:52:26 -08:00
Joe Duffy 29c2a2e08f
Elide the root stack in parent URNs
Every single resource has a type prefix of

    pulumi:pulumi:Stack$

which makes URNs quite lengthy without adding any value.  Since
they all have this prefix, adding it doesn't help to disambiguate.

This change skips adding the parent URN part when it is the built-in
automatic stack type name.
2017-12-05 13:41:26 -08:00
Joe Duffy 0e1ca4363a
Bring back stack outputs (#650)
At some point, we fixed a bug in the way state is managed for "same"
steps, which meant that we wouldn't see newly added output properties.
This had the effect that, if you had a stack already stood up, and
updated it to have output properties, we would miss them.  (Stacks
stood up from scratch would still have them.)  This fixes that problem,
in addition to two other things: 1) we need to sort output property
names to ensure a deterministic ordering, and 2) we need to also
unconditionally apply the outputs RPC coming in, to ensure that the
resulting resource always has the correct outputs (so that for example
deleting prior output properties actually deletes them).

Also add some testing for this area to make sure we don't break again.

Fixes pulumi/pulumi#631.
2017-12-05 13:01:54 -08:00
Pat Gavlin 94645c313a
Merge pull request #643 from pulumi/LateDecrypt
Decrypt configuration nearer to its use.
2017-12-04 17:27:59 -08:00
pat@pulumi.com 7810c824d6 Decrypt configuration nearer to its use.
These changes push the `config.{Map,Value}` interfaces further down into
the deployment engine so that configuration can be decrypted nearer to
its use.

This is the first part of the fix for pulumi/pulumi-ppc#112.
2017-12-04 17:10:40 -08:00
CyrusNajmabadi 75ee89f2b1
Always add 8 chars of randomness to URN names we create. Error if that exceeds the max length allowed for that resource. (#500)
* Include parent type in urn to better ensure component URN uniqueness.
2017-12-04 14:50:55 -08:00
Pat Gavlin f848090479 Return all computed inputs from Provider.Check.
As documented in issue #616, the inputs/defaults/outputs model we have
today has fundamental problems. The crux of the issue is that our
current design requires that defaults present in the old state of a
resource are applied to the new inputs for that resource.
Unfortunately, it is not possible for the engine to decide which
defaults remain applicable and which do not; only the provider has that
knowledge.

These changes take a more tactical approach to resolving this issue than
that originally proposed in #616 that avoids breaking compatibility with
existing checkpoints. Rather than treating the Pulumi inputs as the
provider input properties for a resource, these inputs are first
translated by `Check`. In order to accommodate provider defaults that
were chosen for the old resource but should not change for the new,
`Check` now takes the old provider inputs as well as the new Pulumi
inputs. Rather than the Pulumi inputs and provider defaults, the
provider inputs returned by `Check` are recorded in the checkpoint file.

Put simply, these changes remove defaults as a first-class concept
(except inasmuch as is required to retain the ability to read old
checkpoint files) and move the responsibilty for manging and
merging defaults into the provider that supplies them.

Fixes #616.
2017-12-03 09:33:16 -08:00
Joe Duffy 8d862b52c5
Merge pull request #632 from pulumi/629_cloud
Improve the overall cloud CLI experience
2017-12-03 08:40:34 -08:00
joeduffy 2eb86b24c2 Make some updates based on CR feedback
This change implements some feedback from @ellismg.

* Make backend.Stack an interface and let backends implement it,
  enabling dynamic type testing/casting to access information
  specific to that backend.  For instance, the cloud.Stack conveys
  the cloud URL, org name, and PPC name, for each stack.

* Similarly expose specialized backend.Backend interfaces,
  local.Backend and cloud.Backend, to convey specific information.

* Redo a bunch of the commands in terms of these.

* Keeping with this theme, turn the CreateStack options into an
  opaque interface{}, and let the specific backends expose their
  own structures with their own settings (like PPC name in cloud).

* Show both the org and PPC names in the cloud column printed in
  the stack ls command, in addition to the Pulumi Cloud URL.

Unrelated, but useful:

* Special case the 401 HTTP response and make a friendly error,
  to tell the developer they must use `pulumi login`.  This is
  better than tossing raw "401: Unauthorized" errors in their face.

* Change the "Updating stack '..' in the Pulumi Cloud" message to
  use the correct action verb ("Previewing", "Destroying", etc).
2017-12-03 08:10:50 -08:00
joeduffy b59b8f2e6e Fix cloud tests 2017-12-03 06:34:06 -08:00
joeduffy 22c1f819e6 Better progress reporting
This change does two things:

    1) Adds progress reporting to our uploads.

    2) Eliminate the sleeps that burned 7 seconds at the front of
       any cloud update, needlessly.  It's actually impressively
       fast without these!
2017-12-02 15:17:59 -08:00
joeduffy 1c4e41b916 Improve the overall cloud CLI experience
This improves the overall cloud CLI experience workflow.

Now whether a stack is local or cloud is inherent to the stack
itself.  If you interact with a cloud stack, we transparently talk
to the cloud; if you interact with a local stack, we just do the
right thing, and perform all operations locally.  Aside from sometimes
seeing a cloud emoji pop-up ☁️, the experience is quite similar.

For example, to initialize a new cloud stack, simply:

    $ pulumi login
    Logging into Pulumi Cloud: https://pulumi.com/
    Enter Pulumi access token: <enter your token>
    $ pulumi stack init my-cloud-stack

Note that you may log into a specific cloud if you'd like.  For
now, this is just for our own testing purposes, but someday when we
support custom clouds (e.g., Enterprise), you can just say:

    $ pulumi login --cloud-url https://corp.acme.my-ppc.net:9873

The cloud is now the default.  If you instead prefer a "fire and
forget" style of stack, you can skip the login and pass `--local`:

    $ pulumi stack init my-faf-stack --local

If you are logged in and run `pulumi`, we tell you as much:

    $ pulumi
    Usage:
      pulumi [command]

    // as before...

    Currently logged into the Pulumi Cloud ☁️
        https://pulumi.com/

And if you list your stacks, we tell you which one is local or not:

    $ pulumi stack ls
    NAME            LAST UPDATE       RESOURCE COUNT   CLOUD URL
    my-cloud-stack  2017-12-01 ...    3                https://pulumi.com/
    my-faf-stack    n/a               0                n/a

And `pulumi stack` by itself prints information like your cloud org,
PPC name, and so on, in addition to the usuals.

I shall write up more details and make sure to document these changes.

This change also fairly significantly refactors the layout of cloud
versus local logic, so that the cmd/ package is resonsible for CLI
things, and the new pkg/backend/ package is responsible for the
backends.  The following is the overall resulting package architecture:

* The backend.Backend interface can be implemented to substitute
  a new backend.  This has operations to get and list stacks,
  perform updates, and so on.

* The backend.Stack struct is a wrapper around a stack that has
  or is being manipulated by a Backend.  It resembles our existing
  Stack notions in the engine, but carries additional metadata
  about its source.  Notably, it offers functions that allow
  operations like updating and deleting on the Backend from which
  it came.

* There is very little else in the pkg/backend/ package.

* A new package, pkg/backend/local/, encapsulates all local state
  management for "fire and forget" scenarios.  It simply implements
  the above logic and contains anything specific to the local
  experience.

* A peer package, pkg/backend/cloud/, encapsulates all logic
  required for the cloud experience.  This includes its subpackage
  apitype/ which contains JSON schema descriptions required for
  REST calls against the cloud backend.  It also contains handy
  functions to list which clouds we have authenticated with.

* A subpackage here, pkg/backend/state/, is not a provider at all.
  Instead, it contains all of the state management functions that
  are currently shared between local and cloud backends.  This
  includes configuration logic -- including encryption -- as well
  as logic pertaining to which stacks are known to the workspace.

This addresses pulumi/pulumi#629 and pulumi/pulumi#494.
2017-12-02 14:34:42 -08:00
Luke Hoban 1ab9ae1dda Fix error handling in GetLogs
Ensure we don't create a multierror unless there is at least one non-nil error.
2017-12-02 13:00:49 -08:00
Joe Duffy 16ade183d8
Add a manifest to checkpoint files (#630)
This change adds a new manifest section to the checkpoint files.
The existing time moves into it, and we add to it the version of
the Pulumi CLI that created it, along with the names, types, and
versions of all plugins used to generate the file.  There is a
magic cookie that we also use during verification.

This is to help keep us sane when debugging problems "in the wild,"
and I'm sure we will add more to it over time (checksum, etc).

For example, after an up, you can now see this in `pulumi stack`:

```
Current stack is demo:
    Last updated at 2017-12-01 13:48:49.815740523 -0800 PST
    Pulumi version v0.8.3-79-g1ab99ad
    Plugin pulumi-provider-aws [resource] version v0.8.3-22-g4363e77
    Plugin pulumi-langhost-nodejs [language] version v0.8.3-79-g77bb6b6
    Checkpoint file is /Users/joeduffy/dev/code/src/github.com/pulumi/pulumi-aws/.pulumi/stacks/webserver/demo.json
```

This addresses pulumi/pulumi#628.
2017-12-01 13:50:32 -08:00
Chris Smith 64705b0a73 Ignore unreasonable lint warning 2017-11-30 18:55:18 -08:00
joeduffy 76b77bdabf Add ability to enable verbose logging in tests 2017-11-30 17:23:58 -08:00
Joe Duffy 70c1cdadaf
Mark state snapshots for components (#627)
We need to mark the state snapshots for components, but were skipping this.
I believe this is the root cause for all occurrences of pulumi/pulumi#613.
2017-11-30 16:37:44 -08:00
Joe Duffy 5b57950da6
Add automatic integrity checking (#625)
This change introduces automatic integrity checking for snapshots.
Hopefully this will help us track down what's going on in
pulumi/pulumi#613.  Eventually we probably want to make this opt-in,
or disable it entirely other than for internal Pulumi debugging, but
until we add more complete DAG verification, it's relatively cheap
and is worthwhile to leave on for now.
2017-11-30 11:13:18 -08:00
joeduffy dff4b7d2fb Fix an error variable mistake 2017-11-30 10:45:49 -08:00