Commit graph

1211 commits

Author SHA1 Message Date
joeduffy
225bfd46b3 Don't block on nil channels
We have had a long-standing bug in here where we waiting on a
stdout channel that never got populated, when the language plugin
fails to load entirely.  This would lead to hung processes.  The
fix is simple: only wait for stdout/stderr channels to drain that
have actually been wired up to enjoy the requisite signaling.
2018-02-19 14:06:15 -08:00
joeduffy
25f5a71568 Add support for project plugins
This adds support for two things:

* Installing all plugins that a project requires with a single command:

    $ pulumi plugin install

* Listing the plugins that this project requires:

    $ pulumi plugin ls --project
    $ pulumi plugin ls -p
2018-02-19 11:24:19 -08:00
joeduffy
932e755305 Move plugins to their own directories
Prior to this change, we had a flat list of files in the
~/.pulumi/plugins directory.  This was simple but unfortunately
too naive, since we in fact have multi-file plugins already.
Dumping them in the same directory increases the risk of a
collision.  Instead, let's put them in their own directories.

This means, for example, you'll see things like

    ~/.pulumi/plugins/
        resource-aws-v0.11.0-dev-8-g57a0d62/
            README.txt
            pulumi-resource-aws

Notice that the binary name stays the same -- e.g., in this
case pulumi-resource-aws -- and does not include the version.
This makes it simple to add it to your $PATH in the usual ways
and have it loaded as a preferred location.
2018-02-19 09:31:00 -08:00
joeduffy
e242601133 Do not use absolute paths for main/proj 2018-02-18 08:08:15 -08:00
joeduffy
90ebca4aea Fix up API URLs
The API/REST logic auto-prepended "/api", which we don't want
for the release downloads.  This change just alters callsites
to specify the full path (which I prefer being explicit anyway).
2018-02-18 08:08:15 -08:00
joeduffy
041e44beff Skip reinstalling existing plugins
This change introduces logic to skip installing plugins that already
exist, unless --reinstall is explicitly passed to `pulumi plugin install`.
2018-02-18 08:08:15 -08:00
joeduffy
548c22d014 Reimplement GetRequiredPlugins in Go
This brings back the Node.js language plugin's GetRequiredPlugins
function, reimplemented in Go now that the language host has been
rewritten from JavaScript.  Fairly rote translation, along with
some random fixes required to get tests passing again.
2018-02-18 08:08:15 -08:00
joeduffy
f2cdc8a9ee Use os.Create when expanding plugins 2018-02-18 08:08:15 -08:00
joeduffy
96088dd56f Implement Node.js GetRequiredPlugins function
This change implements the Node.js language host's GetRequiredPlugins
function.  This merely scans all node_modules/*/package.json files in
the program directory, looking for those that have associated plugins.
It returns a list of any found along with their version numbers.
2018-02-18 08:08:15 -08:00
joeduffy
c04341edb2 Consult the program for its list of plugins
This change adds a GetRequiredPlugins RPC method to the language
host, enabling us to query it for its list of plugin requirements.
This is language-specific because it requires looking at the set
of dependencies (e.g., package.json files).

It also adds a call up front during any update/preview operation
to compute the set of plugins and require that they are present.
These plugins are populated in the cache and will be used for all
subsequent plugin-related operations during the engine's activity.

We now cache the language plugins, so that we may load them
eagerly too, which we never did previously due to the fact that
we needed to pass the monitor address at load time.  This was a
bit bizarre anyhow, since it's really the Run RPC function that
needs this information.  So, to enable caching and eager loading
-- which we need in order to invoke GetRequiredPlugins -- the
"phone home" monitor RPC address is passed at Run time.

In a subsequent change, we will switch to faulting in the plugins
that are missing -- rather than erroring -- in addition to
supporting the `pulumi plugin install` CLI command.
2018-02-18 08:08:15 -08:00
joeduffy
5d16fc936a Add workspace.GetPluginPath, and use it
This change introduces a workspace.GetPluginPath function that probes
the central workspace cache of plugins for a matching plugin binary that
matches the desired kind, name, and, optionally, version.  It also permits
overriding this with $PATH for developer scenarios.

The analyzer, language, and resource plugin logic now uses this function
for deciding which binary path to load at runtime.
2018-02-18 08:08:15 -08:00
joeduffy
c1752d357e Implement basic plugin management
This change implements basic plugin management, but we do not yet
actually use the plugins for anything (that comes next).

Plugins are stored in `~/.pulumi/plugins`, and are expected to be
in the format `pulumi-<KIND>-<NAME>-v<VERSION>[.exe]`.  The KIND is
one of `analyzer`, `language`, or `resource`, the NAME is a hyphen-
delimited name (e.g., `aws` or `foo-bar`), and VERSION is the
plugin's semantic version (e.g., `0.9.11`, `1.3.7-beta.a736cf`, etc).

This commit includes four new CLI commands:

* `pulumi plugin` is the top-level plugin command.  It does nothing
  but show the help text for associated child commands.

* `pulumi plugin install` can be used to install plugins manually.
  If run with no additional arguments, it will compute the set of
  plugins used by the current project, and download them all.  It
  may be run to explicitly download a single plugin, however, by
  invoking it as `pulumi plugin install KIND NAME VERSION`.  For
  example, `pulumi plugin install resource aws v0.9.11`.  By default,
  this command uses the cloud backend in the usual way to perform the
  download, although a separate URL may be given with --cloud-url,
  just like all other commands that interact with our backend service.

* `pulumi plugin ls` lists all plugins currently installed in the
  plugin cache.  It displays some useful statistics, like the size
  of the plugin, when it was installed, when it was last used, and
  so on.  It sorts the display alphabetically by plugin name, and
  for plugins with multiple versions, it shows the newest at the top.
  The command also summarizes how much disk space is currently being
  consumed by the plugin cache.  There are no filtering capabilities yet.

* `pulumi plugin prune` will delete plugins from the cache.  By
  default, when run with no arguments, it will delete everything.
  It may be run with additional arguments, KIND, NAME, and VERSION,
  each one getting more specific about what it will delete.  For
  instance, `pulumi plugin prune resource aws` will delete all AWS
  plugin versions, while `pulumi plugin prune resource aws <0.9`
  will delete all AWS plugins before version 0.9.  Unless --yes is
  passed, the command will confirm the deletion with a count of how
  many plugins will be affected by the command.

We do not yet actually download plugins on demand yet.  That will
come in a subsequent change.
2018-02-18 08:08:15 -08:00
Joe Duffy
776a76dffd
Make some stack-related CLI improvements (#947)
This change includes a handful of stack-related CLI formatting
improvements that I've been noodling on in the background for a while,
based on things that tend to trip up demos and the inner loop workflow.

This includes:

* If `pulumi stack select` is run by itself, use an interactive
  CLI menu to let the user select an existing stack, or choose to
  create a new one.  This looks as follows

      $ pulumi stack select
      Please choose a stack, or choose to create a new one:
        abcdef
        babblabblabble
      > currentlyselected
        defcon
        <create a new stack>

  and is navigated in the usual way (key up, down, enter).

* If a stack name is passed that does not exist, prompt the user
  to ask whether s/he wants to create one on-demand.  This hooks
  interesting moments in time, like `pulumi stack select foo`,
  and cuts down on the need to run additional commands.

* If a current stack is required, but none is currently selected,
  then pop the same interactive menu shown above to select one.
  Depending on the command being run, we may or may not show the
  option to create a new stack (e.g., that doesn't make much sense
  when you're running `pulumi destroy`, but might when you're
  running `pulumi stack`).  This again lets you do with a single
  command what would have otherwise entailed an error with multiple
  commands to recover from it.

* If you run `pulumi stack init` without any additional arguments,
  we interactively prompt for the stack name.  Before, we would
  error and you'd then need to run `pulumi stack init <name>`.

* Colorize some things nicely; for example, now all prompts will
  by default become bright white.
2018-02-16 15:03:54 -08:00
Joe Duffy
55e4dbe835
Update spinner to use modern ASCII/emoji art (#942) 2018-02-15 18:22:17 -08:00
Joe Duffy
902d646215
Rename package to project (#935)
This addresses pulumi/pulumi#446: what we used to call "package" is
now called "project".  This has gotten more confusing over time, now
that we're doing real package management.

Also fixes pulumi/pulumi#426, while in here.
2018-02-14 13:56:16 -08:00
Sean Gillespie
402a599fc7
Don't use shebangs to launch providers and correctly kill child process trees on Unix (#934)
* Don't use shebangs to launch providers and correctly kill child process trees on Unix

* Link to relevant documentation
2018-02-14 13:56:07 -08:00
Matt Ellis
a3608e0c2a Retry a few times when running yarn during integration tests
We've seen yarn fail from time to time during our integration
tests (usually timeouts talking to the npm registry) so let's add a
few retries to all yarn commands.
2018-02-12 15:13:35 -08:00
Matt Ellis
5f23a9837a Permit setting multi line config from stdin
When reading a configuration value from standard in and standard in is
not connected to a terminal, read until EOF and then trim a trailing
newline (if present) to get the value

Fixes #822
2018-02-12 15:13:19 -08:00
Luke Hoban
c7adaf107d
Don't include pending-delete resources in tree representation (#902)
Resources in the checkpoint file which are pending-delete represent old versions of resources which are no longer part of the active deployment.  For purposes of constructing the active resource tree, we should skip these resources.
2018-02-12 08:47:09 -08:00
Pat Gavlin
881393146b Remove update event waits. 2018-02-09 09:43:04 -08:00
pat@pulumi.com
e8e0ae9bb4 Wait between calls to fetch update logs.
As it stands, we currently hammer the service's update logs endpoint in
a tight loop while waiting for a deployment to complete. This is not
necessary, and can indeed be deletrious to the user experience: it
appears that this may be exacerbating some mysterious 500 responses from
API gateway.

These changes add a brief sleep in the relevant loop that waits for 5
seconds if the last call produced new log entries or 15 seconds if it
did not.

Fixes #844.
2018-02-07 16:57:05 -08:00
CyrusNajmabadi
275670692b
Introduce Output<T> and update Resource construction code to properly handle it. (#834)
This PR adds a new formalisms at the Resource layer.  First all inputs to a Resource are typed as ```Input<T>```.  This is either a T, ```Promise<T>``
2018-02-05 14:44:23 -08:00
Chris Smith
c408068075
Add User-Agent header to CLI requests (#881)
Today we don't send any version information with API requests to the service, so we cannot make breaking changes between versions of the backend API while preserving backwards compatibility.

This PR adds a `User-Agent` header with REST requests that sends a CLI version number of "1". If the service were to make a breaking change, it could use this header to determine which response handler to use. (e.g. return a different response for "" or "1" and another for "2".) Obviously we want to avoid being in this situation, but in the event that we need to make a breaking change, we'll need this value.

We send the Pulumi version as well, though the SDK will probably rev much more quickly than the backend API client version.

Fixes #848
2018-02-05 09:38:00 -08:00
Matthew Riley
606c45ffea Keep test files on assertion failure
We were pretty careful to keep the test directory around if the test ever
exited early due to a panic or error return. But if the test ran to
completion and failed -- for example, if ExtraRuntimeValidation caused the
test to fail -- we would end up deleting the test directory.

Fixes #868
2018-02-01 23:35:49 -08:00
Matt Ellis
818246a708 Allow control of uploaded archive root in Pulumi.yaml
Previously, when uploading a projectm to the service, we would only
upload the folder rooted by the Pulumi.yaml for that project. This
worked well, but it meant that customers needed to structure their
code in a way such that Pulumi.yaml was always as the root of their
project, and if they wanted to share common files between two projects
there was no good solution for doing this.

This change introduces an optional piece of metadata, named context,
that can be added to Pulumi.yaml, which allows controlling the root
folder used for computing the root folder to archive from.  When it is
set, it is combined with the location of the Pulumi.yaml file for the
project we are uploading and that folder is uses as the root of what
we upload to the service.

Fixes: #574
2018-01-31 16:22:58 -08:00
Matt Ellis
39dbdc98e9 Clean up colorization logic
The existing logic would flow colorization information into the
engine, so depending on the settings in the CLI, the engine may or may
not have emitted colorized events. This coupling is not great and we
want to start moving to a world where the presentation happens
exclusively at the CLI level.

With this change, the engine will always produce strings that have the
colorization formatting directives (i.e. the directives that
reconquest/loreley understands) and the CLI will apply
colorization (which could mean either running loreley to turn the
directives into ANSI escape codes, or drop them or retain them, for
debuging purposes).

Fixes #742
2018-01-31 15:46:14 -08:00
Matt Ellis
b1496f3051 Remove Document and Location
These types are no longer used as pulumi no longer reads and evaluates
source code.

Contributes to #441
2018-01-30 16:42:39 -08:00
Chris Smith
63dfd1e302
Add Manifest to GetStackResponse (#854)
Add Manifest to GetStackResponse.
2018-01-30 14:12:11 -08:00
Luke Hoban
0e4fcc16e3
Pass target Cloud and PPC to integation test Reporter (#859)
Ensure that we capture this information in our test reporting so that we can filter on it in queries and reports.
2018-01-30 13:10:32 -08:00
Matthew Riley
b0905d99aa Fix long line 2018-01-30 10:52:42 -08:00
Matthew Riley
c74db59231 Keep embedded newlines in Lambda log messages
Set the `s` flag in the regexp so `.` matches `\n`.

Add an explicit `\n` to the regexp to account for the newline inserted by Lambda.
2018-01-29 23:33:59 -08:00
Matt Ellis
07fd36c369 Remove deployOptions.Create
This was entirely unused.

Fixes #813
2018-01-29 16:23:55 -08:00
Matt Ellis
803ba93bf5 Pass --network-concurrency 1 to yarn
We've been hitting issues in CI that look like yarnpkg/yarn#4563 and
our hope is that this addresses the issue
2018-01-29 11:49:42 -08:00
Pat Gavlin
842a16cee4 Always set UpdateOptions.DryRun for previews.
If we do not do this, the engine will assume that it should elide
unknown input/output properties when interacting with the language
and resource providers and we will produce unexpected results.
2018-01-26 16:05:42 -08:00
Chris Smith
001d187c90
Add not-started UpdateResult
Add a "not-started" `UpdateResult` type, for supporting updates tracked by the PPC that have not been applied yet.
2018-01-25 14:23:49 -08:00
Chris Smith
4c217fd358
Add "pulumi history" command (#826)
This PR adds a new `pulumi history` command, which prints the update history for a stack.

The local backend stores the update history in a JSON file on disk, next to the checkpoint file. The cloud backend simply provides the update metadata, and expects to receive all the data from a (NYI) `/history` REST endpoint.

`pkg/backend/updates.go` defines the data that is being persisted. The way the data is wired through the system is adding a new `backend.UpdateMetadata` parameter to a Stack/Backend's `Update` and `Destroy` methods.

I use `tests/integration/stack_outputs/` as the simple app for the related tests, hence the addition to the `.gitignore` and fixing the name in the `Pulumi.yaml`.

Fixes #636.
2018-01-24 18:22:41 -08:00
Matt Ellis
ce05cce77f Provide a rudimentary progress spinner
Previously, the `pulumi` tool did not show any indication of progress
when doing a deployment. Combined with the fact that we do not create
resources in parallel it meant that sometime `pulumi` would appear to
hang, when really it was just waiting on some resource to be created
in AWS. In addition, some AWS resources take a long time to create and
CI systems like travis will kill the job if there is no output. This
causes us (and our customers) to have to do crazy dances where we
launch shell scripts that write a dot to the console every once in a
while so we don't get killed. While we plan to overhaul the output
logic (see #617), we take a first step towards interactivity by simply
having a nice little spinner (in the interactive case) and when run
non interactive have `pulumi` print a message that it is still
working.

Fixes #794
2018-01-22 14:21:08 -08:00
Matthew Riley
4bb86116d0
Merge pull request #821 from pulumi/recover-in-test
Avoid resource leaks by recovering from panics in integration tests
2018-01-22 10:08:15 -08:00
Joe Duffy
d6c798ee52
Add a commentChars parameter to EmitHeaderWarning (#827)
Now that we're using this to generate multiple languages, this
needs to be parameterized.  Splitting out into its own change.
2018-01-22 09:32:37 -08:00
Chris Smith
6f6fca7592
Export resource change counts from engine (#823)
This PR exports the aggregate resource changes for update and destroy operations. We'll use this information in #636 when summarizing previous updates.

I initially started with a new struct that had fields like `Created`, `Deleted`, `Unchanged`, etc. But it became cumbersome with the seven different type of resource operations we perform. So instead went with the more flexible `map[deploy.StepOp]int`.
2018-01-20 19:15:19 -08:00
Chris Smith
a4f087460f
Fix false-positives in login verification (#825)
Surprisingly `pulumi login -c https://google.com` would succeed. This was because we were too lax in our way of validating credentials. We take the provided cloud URL and call the "GetCurrentUserHandler" method. But we were only checking that it returned a successful response, not that it was actually valid JSON.

So in the "https://google.com" case, Google returned HTML describing a 404 error, but since the sever response was 200, the Pulumi CLI assumed things were on the up and up.

We now parse the response as JSON, and confirm the response has a `name` property that is non-nil. This heuristic covers the majority of false-positive cases, but without us needing to move all of the service's API shape for users, which includes organizations, which includes Clouds, etc. into `pulumi`.

Fixes https://github.com/pulumi/pulumi-service/issues/457. As an added bonus, we now return a much more useful error message.
2018-01-20 19:11:38 -08:00
Chris Smith
f21c462e1b Report no changes for dry-runs 2018-01-20 12:15:28 -08:00
Chris Smith
7a5ac5ce06 Export resource change counts from engine 2018-01-20 12:07:03 -08:00
Matthew Riley
580bbcb3ca Avoid resource leaks by recovering from panics in integration tests 2018-01-20 11:48:42 -08:00
Matt Ellis
cc04cd6581 Use per stack key for local stacks instead of per project
In the Pulumi Cloud, there is no guarantee that two stacks will share
the same encryption key. This means that encrypted config can not be
shared across stacks (in the Pulumi.yaml) file. To mimic this behavior
in the local experience, we now use a unique key per stack.

When upgrading an existing project, for any stack with existing
secrets, we copy the existing key into this stack. Future stacks will
get thier own encryption key. This strikes a balance between
expediency of implementation, the end user UX and not having to make a
breaking change.

As part of this change, I have introduced a CHANGELOG.md file in the
root of the repository and added a small note about the change to it.

Fixes #769
2018-01-19 00:50:59 -08:00
Matt Ellis
22938f07be Pretty-print some JSON files we persist to disk
We did not pretty print either the workspace settings file or the
repository settings file, but pretty print other files like the
credentials file and checkpoints. Now we do.

Fixes #540
2018-01-18 20:35:53 -08:00
Chris Smith
3a3d0698ae
Surface update options to the service (#806)
This PR surfaces the configuration options available to updates, previews, and destroys to the Pulumi Service. As part of this I refactored the options to unify them into a single `engine.UpdateOptions`, since they were all overlapping to various degrees.

With this PR we are adding several new flags to commands, e.g. `--summary` was not available on `pulumi destroy`.

There are also a few minor breaking changes.

- `pulumi destroy --preview` is now `pulumi destroy --dry-run` (to match the actual name of the field).
- The default behavior for "--color" is now `Always`. Previously it was `Always` or `Never` based on the value of a `--debug` flag. (You can specify `--color always` or `--color never` to get the exact behavior.)

Fixes #515, and cleans up the code making some other features slightly easier to add.
2018-01-18 11:10:15 -08:00
Matt Ellis
c506549a25 Remove MustFprintf in favor of explicitly dropping errors
In travis, we've seen cases where writes to our standard streams
results in an error like: `/dev/stderr: resource temporarily
unavailable` which causes the tests to panic.

Now, in a perfect world, writes to /dev/stderr would not fail in this
way, but we do not live in a perfect world. Other processes on the
machine may make stderr/stdout non-blocking. We've are now seeing this
failure in Travis more often and it is masking real Pulumi failures
we want to debug.
2018-01-16 18:33:44 -08:00
Joe Duffy
34984ba1cb
Delete the old IDL compiler (#801)
It is moving to a new location: https://github.com/pulumi/pidlc
2018-01-13 15:11:52 -08:00
Joe Duffy
0d107c742a
Restructure test framework to ease multiple languages (#799)
This change restructures the test framework code a bit, to make it
easier to introduce additional languages.  Our knowledge of Yarn and
Node.js project structure, for instance, was previously baked in to
the test logic, in a way that was hard to make, for instance, Yarn
optional.  (In Python, of course, it will not be used.)  To better
support this, I've moved some state onto a new programTester struct
that we can use to lazily find binaries required during the testing
(such as Yarn, Pip, and so on).  I'm committing this separately so
that I can minimize merge conflicts in the Python work.
2018-01-12 17:10:53 -08:00
Matthew Riley
3620cdb243
Merge pull request #792 from pulumi/fail-test
Keep test temp directories around on failure
2018-01-12 12:04:58 -08:00
khyperia
1a0cf9ff1b
Respond to feedback 2018-01-11 15:20:20 -08:00
khyperia
2598090cb7
Formatting 2018-01-11 13:46:34 -08:00
khyperia
08f3c623e6
Fix some style issues 2018-01-11 12:05:08 -08:00
khyperia
0f73b8c395
Move types to pulumi-service 2018-01-11 11:51:31 -08:00
khyperia
d4de351b10
Correct some field types 2018-01-11 10:57:25 -08:00
khyperia
916ba6cd39
Add UpdateProgramRequestUntyped 2018-01-11 10:42:43 -08:00
khyperia
f455689bd4
Add apitypes from pulumi-ppc 2018-01-10 15:43:44 -08:00
khyperia
3c644d243f
Create apitype package 2018-01-10 15:04:55 -08:00
Matthew Riley
047a6c3c56 Make "delete temp dir" condition explicit
Previously we checked if the test had failed -- but the call to
`assert.NoError` that actually fails the test sometimes hasn't happened yet.
2018-01-09 23:35:26 -08:00
Matthew Riley
d688910360 Remove named parameters from CopyTestToTemporaryDirectory
Fix two references to the now-unnamed `dir` that should have been to
other variables.

Check a real condition before the deferred call to RemoveAll instead of
checking the error return.
2018-01-09 23:35:26 -08:00
Pat Gavlin
208b7296be
Merge pull request #791 from pulumi/AllowUnknownsPreview
Supply unknown properties to providers during preview.
2018-01-09 19:15:17 -08:00
Pat Gavlin
88a22515df Supply unknown properties to providers during preview.
My previous change to stop supplying unknown properties to providers
broke `pulumi preview` in the case of unknown inputs. This change
restores the previous behavior for previews only; the new unknown-free
behavior remains for applies.

Fixes #790.
2018-01-09 18:41:47 -08:00
Matthew Riley
3c070d3ad5 Restore support for non-Additive edits
None were used in this repository, but `pulumi-aws` used them.
2018-01-09 17:02:30 -08:00
Matthew Riley
9a869eb2d7 Print output of failed commands 2018-01-09 17:02:30 -08:00
Matthew Riley
d9fa8237be Use length of correct string for prefixer 2018-01-09 16:48:04 -08:00
pat@pulumi.com
f3cb37ef95 Do not expose unknowns to resource providers.
Before these changes, we were inconsistent in our treatment of unknown
property values across the resource provider RPC interface. `Check` and
`Diff` were retaining unknown properties in inputs and outputs;
`Create`, `Update`, and `Delete` were not. This interacted badly with
recent changes to `Check` to return all provider inputs--i.e. not just
defaults--from that method: if an unknown input was provided, it would
be present in the returned inputs, which would eventually confuse the
differ by giving the appearance of changes where none were present.

These changes remove unknowns from the provider interface entirely:
unknown property values are never passed to a provider, and a provider
must never return an unknown property value.

This is the primary piece of the fix for pulumi/pulumi-terraform#93.
2018-01-09 12:21:47 -08:00
pat@pulumi.com
5f28d52e00 Remove the Engine type entirely.
It no longer carries any state. All of its methods are now package-level
functions.
2018-01-08 14:20:51 -08:00
pat@pulumi.com
c56e716c31 Refactor the engine's entrypoints.
These changes refactor the engine's entrypoints--Deploy, Destroy, and
Preview--to be update-centric rather than stack-centric. Each of these
methods now takes a value of a new type, Update, that abstracts away the
vagaries of fetching and maintaining the update's state. This
refactoring also reinforces Pulumi.yaml as a CLI concept rather than an
engine concept; the CLI is now the only reader/writer of this format.

These changes will smooth the way for a few refactorings on the service
side that will aid in update isolation.
2018-01-08 14:15:16 -08:00
Matthew Riley
9e3976513c AssertNoError instead of Assert(err == nil)
This may not be exhaustive, but I replaced all instances I could find.
2018-01-08 13:46:21 -08:00
Matthew Riley
ff10b65f3a Add environment variable to keep data for failed tests
Useful to keep command output and maybe-still-relevant checkpoints around.
2018-01-08 13:46:21 -08:00
Matthew Riley
594ccf755a Write command output to log file 2018-01-08 13:46:21 -08:00
Matthew Riley
b0eed85871 Remove non-additive edits
Now we know we only create one temporary directory per test
2018-01-08 13:46:21 -08:00
Matthew Riley
afb734a712 Unexport a few private utility functions
Makes it bit easier to see the module entrypoints.
2018-01-08 13:46:21 -08:00
Matt Ellis
607b4990f5 Remove stack configuration when deleting a stack
When a stack is removed, we should delete any configuration it had
saved in either the Pulumi.yaml file or the workspace.

Fixes #772
2018-01-07 17:38:40 -08:00
Pat Gavlin
3ca874d3f7
Merge pull request #781 from pulumi/ExportImport
Add the ability to {ex,im}port a stack's deployment.
2018-01-05 17:56:16 -08:00
pat@pulumi.com
7782a83030 Appease the linters. 2018-01-05 17:35:22 -08:00
pat@pulumi.com
b96217341f Add the ability to {ex,im}port a stack's deployment.
These changes add the ability to export a stack's latest deployment or
import a new deployment to a stack via the Pulumi CLI. These
capabilities are exposed by two new verbs under `stack`:
- export, which writes the current stack's latest deployment to stdout
- import, which reads a new deployment from stdin and applies it to the
  current stack.

In the local case, this simply involves reading/writing the stack's
latest checkpoint file. In the cloud case, this involves hitting two new
endpoints on the service to perform the export or import.
2018-01-05 16:22:31 -08:00
Chris Smith
decf814278
Enable running integration tests against the service (#775)
This PR updates the `pkg/testing/integration` package to support running integration tests against the Pulumi Service if desired. This is done through adding new options to `ProgramTestOptions`. (Generally adding support for providing values to flags that were previously inaccessible.)

I added an integration test to confirm that it all works if the PULUMI_API environment variable is set. These tests aren't run in Travis, only manually. Since we cannot reliably run tests from `master` against the service because of the delay in rolling out updates to the Pulumi SDK, etc.
2018-01-03 21:26:50 -08:00
Matt Ellis
c052e24370
Merge pull request #771 from pulumi/no-all-for-secrets
Do not allow encrypted global configuration
2018-01-03 15:04:00 -08:00
joeduffy
b3ee139b91 Fix error message for stack select failures 2017-12-28 16:51:53 -08:00
Matt Ellis
f510f3c914 Do not allow encrypted global configuration
The cloud backend does not support this because it computes an
encryption key per stack, so we should not support this in the CLI.

Fixes #770
2017-12-27 19:00:55 -08:00
Joe Duffy
d419229301
Add additional linting (#768)
This adds additional linting checks.  Most importantly, it will
check calls to our custom format routines for missing arguments.
2017-12-27 17:10:12 -08:00
joeduffy
5ac81c2818 Add missing Infof argument 2017-12-27 14:54:27 -08:00
Joe Duffy
86c1e7ad39
Merge pull request #764 from pulumi/763_wait_for_update_resiliency
Make the CLI's waitForUpdates more resilient to transient failure
2017-12-27 06:34:08 -08:00
joeduffy
87079589f1 Use the retry framework for REST API retries
This change incorporates feedback on https://github.com/pulumi/pulumi/pull/764,
in addition to refactoring the retry logic to use our retry framework rather
than hand-rolling it in the REST API code.  It's a minor improvement, but at
least lets us consolidate some of this logic which we'll undoubtedly use more
of over time.
2017-12-26 10:24:08 -08:00
Luke Hoban
56fd8bc257
Fix REST API calls for encrypt/decrypt (#767)
The merge of two recent changes seems to have led to these calls invoking the API incorrectly.
2017-12-26 09:42:42 -08:00
joeduffy
db5318b0a5 Make the CLI's waitForUpdates more resilient to transient failure
We saw an issue where a user was mid-update, and got a networking
error stating `read: operation timed out`.  We believe this was simply
a local client error, due to a flaky network.  We should be resilient
to such things during updates, particularly when there's no way to
"reattach" to an in-progress udpate (see pulumi/pulumi#762).

This change accomplishes this by changing our retry logic in the
cloud backend's waitForUpdates function.  Namely:

* We recognize three types of failure, and react differently:

    - Expected HTTP errors.  For instance, the 504 Gateway Timeouts
      that we already retried in the face of.  In these cases, we will
      silently retry up to 10 times.  After 10 times, we begin warning
      the user just in case this is a persistent condition.

    - Unexpected HTTP errors.  The CLI will quit immediately and issue
      an error to the user, in the usual ways.  This covers
      Unauthorized among other things.  Over time, we may find that we
      want to intentionally move some HTTP errors into the above.

    - Anything else.  This covers the transient networking errors case
      that we have just seen.  I'll admit, it's a wide net, but any
      instance of this error issues a warning and it's up to the user
      to ^C out of it.  We also log the error so that we'll see it if
      the user shares their logs with us.

* We implement backoff logic so that we retry very quickly (100ms)
  on the first failure, and more slowly thereafter (1.5x, up to a max
  of 5 seconds).  This helps to avoid accidentally DoSing our service.
2017-12-26 09:40:51 -08:00
Pat Gavlin
a7fbdff7ea
Merge pull request #761 from pulumi/UpdateNoMerge
Do not merge old state and inputs for Update.
2017-12-24 14:47:15 -08:00
Pat Gavlin
1bd1eaff50 Do not merge old state and inputs for Update.
This merging causes similar issues to those it did in `Check`, and
differs from the approach we take to `Diff`. This can causes problems
such as an inability to remove properties.
2017-12-22 18:18:14 -08:00
Luke Hoban
bc95745fb6
Update regexp to match changes in AWS generated names (#760)
Also keeps support for older name generation pattern.
2017-12-22 14:49:42 -08:00
Pat Gavlin
e4d9eb6fd3 Support secrets for cloud stacks.
Use the new {en,de}crypt endpoints in the Pulumi.com API to secure
secret config values. The ciphertext for a secret config value is bound
to the stack to which it applies and cannot be shared with other stacks
(e.g. by copy/pasting it around in Pulumi.yaml). All secrets will need
to be encrypted once per target stack.
2017-12-22 07:59:27 -08:00
joeduffy
af087103b9 Fix more edit directory issues
This fixes a few more edit directory issues, where we didn't
correctly propagate the changes in edit directory required during
subsequent destroy/stack activities.  It also fixes a few error
paths so that we preserve the right directory to be removed.
2017-12-22 06:58:31 -08:00
Luke Hoban
7cc782e856
Add support for pulumi logs --since when targeting cloud backend (#756)
Hooks up the CLI to be able to pass query parameters through to the
service.

Part of pulumi/pulumi-service#431.
2017-12-21 17:16:45 -08:00
Joe Duffy
b158e9804b
Fix test edit directories (#755)
The test edit directory logic needs to carry forward the checkpoint
from the *prior edit*, not the *original*, as it sequences through edits.
2017-12-21 11:01:30 -08:00
joeduffy
2eda90d67b Fix a dangling fmt parameter 2017-12-21 09:18:00 -08:00
Joe Duffy
bc2cf55463
Implement resource protection (#751)
This change implements resource protection, as per pulumi/pulumi#689.
The overall idea is that a resource can be marked as "protect: true",
which will prevent deletion of that resource for any reason whatsoever
(straight deletion, replacement, etc).  This is expressed in the
program.  To "unprotect" a resource, one must perform an update setting
"protect: false", and then afterwards, they can delete the resource.

For example:

    let res = new MyResource("precious", { .. }, { protect: true });

Afterwards, the resource will display in the CLI with a lock icon, and
any attempts to remove it will fail in the usual ways (in planning or,
worst case, during an actual update).

This was done by adding a new ResourceOptions bag parameter to the
base Resource types.  This is unfortunately a breaking change, but now
is the right time to take this one.  We had been adding new settings
one by one -- like parent and dependsOn -- and this new approach will
set us up to add any number of additional settings down the road,
without needing to worry about breaking anything ever again.

This is related to protected stacks, as described in
pulumi/pulumi-service#399.  Most likely this will serve as a foundational
building block that enables the coarser grained policy management.
2017-12-20 14:31:07 -08:00
Joe Duffy
26021a41a3
Add the ability to do negative tests
This change adds the ability to do very coarse-grained negative
tests in our integration test framework.  Either a test itself,
or one of its edits, may be marked ExpectFailure == true, at which
point either the preview or update MUST fail (and, if one fails
without this being set, we still treat it as an error).
2017-12-20 12:10:46 -08:00
Luke Hoban
d5acc3da87
Add option to specify command line flags to pulumi update in integration tests (#749)
And use this in the diff test which needs to apply `--colors=raw`.
2017-12-20 11:17:25 -08:00
Joe Duffy
680a09a164
Add fewer leading prefixes (#741) 2017-12-18 14:51:23 -08:00
Joe Duffy
3311d32c2d
Merge pull request #728 from pulumi/hideGoop
Elide our serialized function preamble and postamble when printing user code
2017-12-18 13:22:13 -08:00
Joe Duffy
f0c28db639
Attempt to fix colorization (#740)
Our recent changes to colorization changed from a boolean to a tri-valued
enum (Always, Never, Raw).  The events from the service, however, are still
boolean-valued.  This changes the message payload to carry the full values.
2017-12-18 11:42:32 -08:00
Joe Duffy
662ebcbd94
Merge pull request #666 from pulumi/add-logging-to-stack-reading
Log the error when getting a stack fails
2017-12-18 09:51:29 -08:00
Joe Duffy
6dc16a5548
Make cloud authentication more intuitive (#738)
The prior behavior with cloud authentication was a bit confusing
when authenticating against anything but https://pulumi.com/.  This
change fixes a few aspects of this:

* Improve error messages to differentiate between "authentication
  failed" and "you haven't logged into the target cloud URL."

* Default to the cloud you're currently authenticated with, rather
  than unconditionally selecting https://pulumi.com/.  This ensures

      $ pulumi login -c https://api.moolumi.io
      $ pulumi stack ls

  works, versus what was currently required

      $ pulumi login -c https://api.moolumi.io
      $ pulumi stack ls -c https://api.moolumi.io

  with confusing error messages if you forgot the second -c.

* To do this, our default cloud logic changes to

    1) Prefer the explicit -c if supplied;

    2) Otherwise, pick the "currently authenticated" cloud; this is
       the last cloud to have been targeted with pulumi login, or
       otherwise the single cloud in the list if there is only one;

    3) https://pulumi.com/ otherwise.
2017-12-16 07:49:41 -08:00
Joe Duffy
bd2baa5091
Pass planning flag to print resource outputs (#737)
We hadn't previously passed the planning flag when printing resource
outputs, meaning any computed ones now are being printed as "undefined".
Instead, we prefer to see the "computed<string>" type name.
2017-12-16 07:33:58 -08:00
Pat Gavlin
9d038c1d88 Only pass old inputs to Check.
We do not need all of the information in the old state for this call, as
outputs will not be read by the provider during validation or defaults
computation.
2017-12-15 11:04:43 -08:00
joeduffy
668e56f2c7 Use old.All() to be consistent with Update
This changes the inputs to Check/Diff to match what we also
pass to the Update function later on (old.All() vs. old.Outputs).
2017-12-15 07:37:15 -08:00
joeduffy
675fe38269 Add more tracing context to RPC marshaling
This change adds a bit more tracing context to RPC marshaling
logging so that it's easier to attribute certain marshaling calls.
Prior to this, we'd just have a flat list of "marshaled property X"
without any information about what the marshaling pertained to.
2017-12-15 07:22:49 -08:00
joeduffy
8492e6100f Pass old *outputs*, not *inputs*, to Check/Diff
This change passes a resource's old output state, so that it contains
everything -- defaults included -- for purposes of the provider's diffing.
Not doing so can lead the provider into thinking some of the requisite
state is missing.
2017-12-15 07:01:11 -08:00
CyrusNajmabadi
a5cf6f25d6
Add a framework for baselining and validating pulumi-update diff output. (#700) 2017-12-14 17:10:05 -08:00
Cyrus Najmabadi
84051e84b3 Make linter happy. 2017-12-14 15:41:53 -08:00
Cyrus Najmabadi
1fc33741cc Strip out even more. 2017-12-14 15:02:13 -08:00
Cyrus Najmabadi
1ab2680df8 Elide our serialized function preamble and postamble when printing user code. 2017-12-14 14:55:27 -08:00
CyrusNajmabadi
9bb9295594
Suppress printing out the variables we capture. (#724) 2017-12-14 14:40:04 -08:00
CyrusNajmabadi
e4946a6620
Allow users to control if and how output is colorized. (#718)
Part of the work to make it easier to tests of diff output.  Specifically, we now allow users to pass --color=option for several pulumi commands.  'option' can be one of 'always', 'never', 'raw', and 'auto' (the default).  

The meaning of these flags are:

1. auto: colorize normally, unless in --debug 
2. always: always colorize no matter what
3. never: never colorize no matter what.
4. raw: colorize, but preserve the original "<{%%}>" style control codes and not the translated platform specific codes.   This is for testing purposes and ensures we can have test for this stuff across platform.
2017-12-14 11:53:02 -08:00
Luke Hoban
6f15fa8ed8
Pass more stack info to ExtraRuntimeValidation (#717)
This will allow us to remove a lot of current boilerplate in individual tests, and move it into the test harness.

Note that this will require updating users of the integration test framework.  By moving to a property bag of inputs, we should avoid needing future breaking changes to this API though.
2017-12-13 16:09:14 -08:00
joeduffy
92ea5b5bdd Add a test case for delete-before-recreate 2017-12-13 10:47:18 -08:00
joeduffy
3a13621c32 Add rudimentary delete-before-create support
This change adds rudimentary delete-before-create support (see
pulumi/pulumi#450).  This cannot possibly be complete until we also
implement pulumi/pulumi#624, becuase we may try to delete a resource
while it still has dependent resources (which almost certainly will
fail).  But until then, we can use this to manually unwedge ourselves
for leaf-node resources that do not support old and new resources
living side-by-side.
2017-12-13 10:47:18 -08:00
Joe Duffy
36ab8f0087
Make config a little less error prone
As articulated in #714, the way config defaults to workspace-local
configuration is a bit error prone, especially now with the cloud
workflow being the default.  This change implements several improvements:

* First, --save defaults to true, so that configuration changes will
  persist into your project file.  If you want the old local workspace
  behavior, you can specify --save=false.

* Second, the order in which we applied configuration was a little
  strange, because workspace settings overwrote project settings.
  The order is changed now so that we take most specific over least
  specific configuration.  Per-stack is considered more specific
  than global and project settings are considered more specific
  than workspace.

* We now warn anytime workspace local configuration is used.  This
  is a developer scenario and can have subtle effects.  It is simply
  not safe to use in a team environment.  In fact, I lost an arm
  this morning due to workspace config... and that's why you always
  issue warnings for unsafe things.
2017-12-13 10:46:54 -08:00
Joe Duffy
dc83a49a0d
Merge pull request #713 from pulumi/712_missing_error_prop
Add a missing error propagation
2017-12-13 09:23:18 -08:00
joeduffy
664d4f3bfe Add a missing error propagation
If the service returns a 504, we happily keep looping around and
retrying until we get a valid update.  Unfortunately, we missed the
else condition, which is what happens when this isn't a 504, leading
us to swallow real errors (500 and the like).  Trivial fix.

Fixes pulumi/pulumi#712.
2017-12-13 08:33:11 -08:00
Joe Duffy
2a8ab4d38e
Merge pull request #708 from pulumi/fix-660
Do not buffer output during planning
2017-12-12 16:02:48 -08:00
pat@pulumi.com
55df0cc201 PR feedback 2017-12-12 14:32:27 -08:00
pat@pulumi.com
fcdc3495a4 Render computeds differently during updates.
Rather than displaying `computed` as we do during previews, display
`undefined` instead, as these values may be intentionally undefined by
the user's program.

Fixes #633.
2017-12-12 14:20:44 -08:00
Matt Ellis
8d998fea81 Do not buffer output during preview
Previously, we would compute all the output from the preview operation
and then display it as a single event. Instead, we should do this how
we handle things during deploy. Each logical print operaton gets a
single event and we stream them back over a channel during planning.

Fixes #660
2017-12-12 12:41:56 -08:00
Joe Duffy
b6a386995a
Set pwd for plugins (#706)
This change just flows the project's "main" directory all the way
through to the plugins, fixing #667.  In that work item, we discussed
alternative approaches, such as rewriting the asset paths, but this
is tricky because it's very tough to do without those absolute paths
somehow ending up in the checkpoint files.  Just launching the
processes with the right pwd is far easier and safer, and it turns
out that, conveniently, we set up the plugin context in exactly the
same place that we read the project information.
2017-12-12 12:31:09 -08:00
Pat Gavlin
d0df98e03b Close assets while creating ZIP archives.
This was an unintentional omission from my earlier change that
refactored the archive code.

Fixes pulumi/pulumi-ppc#133.
2017-12-12 10:38:27 -08:00
CyrusNajmabadi
eea5232cce
Break out additional test helper. (#699) 2017-12-11 15:48:45 -08:00
Luke Hoban
3b461ffcb3
Use AWS access key in operations prvider when available (#697)
If the stack config specifies AWS credentials, these should be used in the operations provider instead of ambient credentials.  This is necessary to ensure that we have access to resources in the target account.

Fixes pulumi/pulumi-service#389
2017-12-11 15:40:39 -08:00
CyrusNajmabadi
9bc9ce1505
More test code refactoring.
* Move code down.

* Extact test code out.

* Extract helper and pattern.

* Remove unnecessary code.

* move comment.
2017-12-11 14:42:42 -08:00
Joe Duffy
5a1b1253ab
Save configuration under the stack by default (#696) 2017-12-11 14:41:57 -08:00
Joe Duffy
e93fe05efe
Reparameterize NewUniqueHex/ID
This change adds a randlen parameter to NewUniqueHex/ID so that the
caller can decide how long of a random string to generate.
2017-12-10 07:44:11 -08:00
Luke Hoban
90c7305f55
Update logs after changes in pulumi/cloud names (#684) 2017-12-09 17:27:43 -08:00
CyrusNajmabadi
7234cdef02
Take an options pointer so values can change as a test runs. (#679)
* Take an options pointer so values can change as a test runs.

* Don't pass redundant information.

* Extract initialization routine.

* Fix caller.

* Check return value.

* Extract destruction logic.

* Move preview and update into their own function.

* Inline null check.
2017-12-08 16:59:28 -08:00
Joe Duffy
1681119339
Add a stack output command (#675)
This change adds a `pulumi stack output` command.  When passed no
arguments, it prints all stack output properties, in exactly the
same format as `pulumi stack` does (just without all the other stuff).
More importantly, if you pass a specific output property, a la
`pulumi stack output clusterARN`, just that property will be printed,
in a scriptable-friendly manner.  This will help us automate wiring
multiple layers of stacks together during deployments.

This fixes pulumi/pulumi#659.
2017-12-08 13:14:58 -08:00
CyrusNajmabadi
edb7bf849e
Reverts (#676)
* Revert "Make sure we properly update dir so that pulumi-destroy works."

This reverts commit 56bfc57998.

* Revert "Edits needs to continuously pass along the new directory. (#668)"

This reverts commit 8bd1822722.

* Revert "Refactor test code to make it simpler to validate code in the middle. (#662)"

This reverts commit ed65360157.
2017-12-08 12:59:39 -08:00
Joe Duffy
d89a2b4e1f
Add a logout --all command (#673)
If a cloud you've previously authenticated with goes away -- as ours
sort of did, because the cloud endpoing in the CLI changed (to actually
be correct) -- then you can't logout without manually editing the
credentials file in your workspace.  This is a little annoying.  So,
rather than that, let's have a `pulumi logout --all` command that just
logs out of all clouds you are presently authenticated with.
2017-12-08 12:14:14 -08:00
CyrusNajmabadi
56bfc57998
Make sure we properly update dir so that pulumi-destroy works. 2017-12-08 11:37:38 -08:00
Chris Smith
ebc1776eb9 Absorb breaking change from Service/PPC API 2017-12-08 10:21:25 -08:00
Chris Smith
8c3c2c97f5 Add messages to asserts 2017-12-08 10:21:25 -08:00
Chris Smith
527a1c97e7 Fix panic with nil message entry 2017-12-08 10:21:25 -08:00
CyrusNajmabadi
8bd1822722
Edits needs to continuously pass along the new directory. (#668) 2017-12-08 01:51:35 -08:00
Matt Ellis
94b9489759 Log the error when getting a stack fails 2017-12-07 18:24:51 -08:00
CyrusNajmabadi
ed65360157
Refactor test code to make it simpler to validate code in the middle. (#662)
* Refactor test code to make it simpler to validate code in the middle.
2017-12-07 16:29:48 -08:00
Joe Duffy
aa7c1598ac
Show root stacks and more outputs (#661)
This changes two things:

1) Always show the root stack, even when unchanged.  This ensures
   that you see the outputs during any updates, etc., which is nice.

2) Always show all outputs after operations.  Before, we did some
   diffing between old/new, which actually doesn't make much sense.
2017-12-07 13:33:16 -08:00
Chris Smith
e104ebd7f6
Several fixes relating to Pulumi clouds and auth (#651)
- Fix #644 "Re-enable use of local dev stacks". Rather than trying to be smart about using "pulumi.com" and inferring "api.pulumi.com", we just use whatever cloud URL the user provides. (e.g. "http://localhost:8080") We can improve the user experience later by providing friendly names for these things upon login. So we can show "Pulumi" or "Contoso" instead of the URL -- but let's cross that bridge later.

- Fix #640, "Better error on `pulumi login`". We only provide the more friendly error about needing to login IFF you are trying to use an authenticated Pulumi API without having creds.
2017-12-07 10:59:20 -08:00
Joe Duffy
971f6189f2
Fix pending delete replacement failure (#658)
The two-phase output properties change broke the ability to recover
from a failed replacement that yields pending deletes in the checkpoint.
The issue here is simply that we should remember pending registrations
only for logical operations that *also* have a "new" state (create or
update).  This commit fixes this, and also adds a new step test with
fault injection to probe many interesting combinations of steps.
2017-12-07 09:44:38 -08:00
Joe Duffy
9680fc3604
Issue an error when a project file isn't found 2017-12-06 17:26:45 -08:00
Matt Ellis
ffdee8bb23 Relax error check when trying to kill child processes
If the process we are trying to kill has already exited, don't treat
this as an error. This can happen when we snapshot the process tree
before the process exits but it has exited by the time we get to
trying to kill it.

Fixes #654
2017-12-06 09:52:26 -08:00
Joe Duffy
29c2a2e08f
Elide the root stack in parent URNs
Every single resource has a type prefix of

    pulumi:pulumi:Stack$

which makes URNs quite lengthy without adding any value.  Since
they all have this prefix, adding it doesn't help to disambiguate.

This change skips adding the parent URN part when it is the built-in
automatic stack type name.
2017-12-05 13:41:26 -08:00