Commit graph

210 commits

Author SHA1 Message Date
Matt Ellis 2669f65b39 Don't dirty work tree when running tests
Because we run our golang integration tests "in tree" (due to
the need to be under $GOPATH and have a vendor folder around), the
"command-output" folder was getting left behind, dirtying the worktree
after building.

This change does two things:

1. On a succecssful run, remove the folder.
2. Ignore the folder via .gitignore (this way if a test fails and you
do `git add .` you don't end up commiting this folder).
2018-06-14 15:58:37 -07:00
Pat Gavlin dc0c604ae7
Enable fork builds. (#1495)
If we're building in CI and do not have an access token, skip any
lifecycle tests.
2018-06-11 16:01:04 -07:00
Joe Duffy b69ad03625
Merge pull request #1491 from pulumi/go_testing
Fix a few things in the Go test harness
2018-06-11 14:44:07 -07:00
joeduffy c81a5152f0 Skip copying to a temp dir for Go projects
Due to the way GOPATH and vendoring works, copying Go tests out to a
random temp directory simply will not work.  This is largely a holdover
to the days of when .pulumi/ directories would pollute the directory,
but is also done for languages like Node.js whose preparatory and build
steps also pollute the working directory (with bin/ and node_modules/
directories).  Go does not have this problem, so we can safely skip.
2018-06-11 14:24:29 -07:00
joeduffy 6599240861 Default to $HOME/go for GOPATH during tests 2018-06-11 13:08:46 -07:00
Matt Ellis 9d4e24c165 Allow tests to be run by users outside of the Pulumi organization
Today we defaulted our tests to create stacks in the `pulumi`
organization. We did this because our tests run with `pulumi-bot` and
we'd rather create the stacks in our shared organization, so any
Pulumi developer can see them.

Of course, as we prepare to have folks outside of the Pulumi
organization write and run tests, this has now become a bad default.

Remove the ability to explicitly set an owner in
ProgramTestOptions (since that would more or less only lead to pain
going forward) and default to just creating the stacks in whatever
account is currently logged in. In CI, we'll set a new environment
variable "PULUMI_TEST_OWNER" which controls the owner of the stacks,
which we'll set to `pulumi`.

Impact to day to day developers is during test runs locally you'll see
stacks in your list of stacks. If any of the tests fail to clean up,
you'll see these lingering stacks (but you can go clean them up).
2018-06-11 11:03:17 -06:00
joeduffy b28f643164 Add integration test support for Go
This adds integration test support framework for Go.

It also adds a test case for the basic empty Pulumi Go program.
2018-06-10 09:17:19 -07:00
Matt Ellis 8a6ffd08b2 Use crypto/rand when generating a stack suffix
math/rand uses a fixed seed, meaning that across runs the Kth call to
`rand.Int63()` will always return the same value.

Given that we want to provide a unique suffix across multiple
concurrent runs, this isn't the behavior we want.

I saw an instance fail in CI where all three legs ran the test
concurrently and they raced on creating the test stack, since they all
generated the same name.
2018-06-05 11:06:01 -07:00
Matt Ellis 4ac1a2f355 Pull tracing and reporting information from the environment
This lets us set these values globaly, in our Travis and AppVeyor
configurations instead of forcing every test to opt-in. It also means
that by default, local builds will not report any of this data (and
will not need access to these endpoints).
2018-06-04 14:28:52 -07:00
Pat Gavlin 38d3b83494
Allow tests to perform changes in empty updates. (#1411)
This may be a useful knob to have.
2018-05-23 09:17:10 -07:00
joeduffy 5967259795 Add license headers 2018-05-22 15:02:47 -07:00
Pat Gavlin 05ef4fecb9 Validate empty previews and updates in tests.
Fixes #1154.
2018-05-15 18:03:30 -07:00
Pat Gavlin 016ae4acba
Add pre/post-update hooks to the test framework. (#1369)
These changes add support for pre- and post-`pulumi` callbacks to the
integration test framework. These callbacks will be invoked immediately
before and after (respectively) running a `pulumi` command.
2018-05-15 09:48:56 -07:00
joeduffy 7c7f6d3ed7 Bring back preview, swizzle some flags
This changes the CLI interface in a few ways:

* `pulumi preview` is back!  The alternative of saying
  `pulumi update --preview` just felt awkward, and it's a common
  operation to want to perform.  Let's just make it work.

* There are two flags consistent across all update commands,
  `update`, `refresh`, and `destroy`:

    - `--skip-preview` will skip the preview step.  Note that this
      does *not* skip the prompt to confirm that you'd like to proceed.
      Indeed, it will still prompt, with a little warning text about
      the fact that the preview has been skipped.

    * `--yes` will auto-approve the updates.

This lands us in a simpler and more intuitive spot for common scenarios.
2018-05-06 13:55:39 -07:00
joeduffy 6ad785d5c4 Revise the way previews are controlled
I found the flag --force to be a strange name for skipping a preview,
since that name is usually reserved for operations that might be harmful
and yet you're coercing a tool to do it anyway, knowing there's a chance
you're going to shoot yourself in the foot.

I also found that what I almost always want in the situation where
--force was being used is to actually just run a preview and have the
confirmation auto-accepted.  Going straight to --force isn't the right
thing in a CI scenario, where you actually want to run a preview first,
just to ensure there aren't any issues, before doing the update.

In a sense, there are four options here:

1. Run a preview, ask for confirmation, then do an update (the default).
2. Run a preview, auto-accept, and then do an update (the CI scenario).
3. Just run a preview with neither a confirmation nor an update (dry run).
4. Just do an update, without performing a preview beforehand (rare).

This change enables all four workflows in our CLI.

Rather than have an explosion of flags, we have a single flag,
--preview, which can specify the mode that we're operating in.  The
following are the values which correlate to the above four modes:

1. "": default (no --preview specified)
2. "auto": auto-accept preview confirmation
3. "only": only run a preview, don't confirm or update
4. "skip": skip the preview altogether

As part of this change, I redid a bit of how the preview modes
were specified.  Rather than booleans, which had some illegal
combinations, this change introduces a new enum type.  Furthermore,
because the engine is wholly ignorant of these flags -- and only the
backend understands them -- it was confusing to me that
engine.UpdateOptions stored this flag, especially given that all
interesting engine options _also_ accepted a dryRun boolean.  As of
this change, the backend.PreviewBehavior controls the preview options.
2018-05-06 13:55:04 -07:00
Pat Gavlin c0ed11af14 Enable tracing of integration tests.
These changes add a new option to the integration test framework that
allows the specification of a tracing endpoint for Pulumi invocations.
2018-05-03 13:44:21 -07:00
CyrusNajmabadi fd3ddda917
Disable interactive mode for a CI/CD server. (#1297) 2018-04-30 15:27:53 -07:00
CyrusNajmabadi 3b13803c71
Add a hidden --no-interactive flag so that we can reduce interactive noise when running jenkins. (#1262) 2018-04-24 14:23:08 -07:00
Matt Ellis 15e2ad27fe Address code review feedback 2018-04-20 02:34:10 -04:00
Matt Ellis 56d7f8eb24 Support new stack identity for the cloud backend
This change introduces support for using the cloud backend when
`pulumi init` has not been run. When this is the case, we use the new
identity model, where a stack is referenced by an owner and a stack
name only.

There are a few things going on here:

- We add a new `--owner` flag to `pulumi stack init` that lets you
  control what account a stack is created in.

- When listing stacks, we show stacks owned by you and any
  organizations you are a member of. So, for example, I can do:

  * `pulumi stack init my-great-stack`
  * `pulumi stack init --owner pulumi my-great-stack`

  To create a stack owned by my user and one owned by my
  organization. When `pulumi stack ls` is run, you'll see both
  stacks (since they are part of the same project).

- When spelling a stack on the CLI, an owner can be optionally
  specified by prefixing the stack name with an owner name. For
  example `my-great-stack` means the stack `my-great-stack` owned by
  the current logged in user, where-as `pulumi/my-great-stack` would
  be the stack owned by the `pulumi` organization

- `--all` can be passed to `pulumi stack ls` to see *all* stacks you
  have access to, not just stacks tied to the current project.
2018-04-18 04:54:02 -07:00
Matt Ellis 50982e8763 Add some randomness to stack names in our integration tests
To prepare for a world where stack names must be unique across an
owner, add some randomness to the names we use for stacks as part of
our integration tests.
2018-04-18 04:54:02 -07:00
CyrusNajmabadi f2b9bd4b13
Remove the explicit 'pulumi preview' command. (#1170)
Old command still exists, but tells you to run "pulumi update --preview".
2018-04-13 22:26:01 -07:00
Luke Hoban 880fc2d9f9
Ensure that we create legal stack names for integration tests. (#1152) 2018-04-11 12:17:31 -07:00
Matt Ellis d3240fdc64 Require pulumi login before commands that need a backend
This change does three major things:

1. Removes the ability to be logged into multiple clouds at the same
time. Previously, we supported being logged into multiple clouds at
the same time and the CLI would fan out requests and join responses
when needed. In general, this was only useful for Pulumi employees
that wanted run against multiple copies of the service (say production
and staging) but overall was very confusing (for example in the old
world a stack with the same identity could appear twice (since it was
in two backends) which the CLI didn't handle very well).

2. Stops treating the "local" backend as a special thing, from the
point of view of the CLI. Previouly we'd always connect to the local
backend and merge that data with whatever was in clouds we were
connected to. We had gestures like `--local` in `pulumi stack init`
that meant "use the local mode". Instead, to use the local mode now
you run `pulumi login --cloud-url local://` and then you are logged in
the local backend. Since you can only ever be logged into a single
backend, we can remove the `--local` and `--remote` flags from `pulumi
stack init`, it just now requires you to be logged in and creates a
stack in whatever back end you were logged into. When logging into the
local backend, you are not prompted for an access key.

3. Prompt for login in places where you have to log in, if you are not
already logged in.
2018-04-05 10:19:41 -07:00
Luke Hoban 5ede33e03d
Run tests against managed stacks backend instead of FnF (#1092)
Tests now target managed stacks instead of local stacks.

The existing logged in user and target backend API are used unless PULUMI_ACCES_TOKEN is defined, in which case tests are run under that access token and against the PULUMI_API backend.

For developer machines, we will now need to be logged in to Pulumi to run tests, and whichever default API backend is logged in (the one listed as current in ~/.pulumi/credentials.json) will be used. If you need to override these, provide PULUMI_ACCESS_TOKEN and possibly PULUMI_API.

For Travis, we currently target the staging service using the Pulumi Bot user.

We have decided to run tests in the pulumi organization. This can be overridden for local testing (or in Travis in the future) by defining PULUMI_API_OWNER_ORGANIZATION and using an access token with access to that organization.

Part of pulumi/home#195.
2018-04-02 21:34:54 -07:00
CyrusNajmabadi 0caa5cc5fa
Making linter happy. (#1084) 2018-03-29 15:15:52 -07:00
joeduffy 8b5874dab5 General prep work for refresh
This change includes a bunch of refactorings I made in prep for
doing refresh (first, the command, see pulumi/pulumi#1081):

* The primary change is to change the way the engine's core update
  functionality works with respect to deploy.Source.  This is the
  way we can plug in new sources of resource information during
  planning (and, soon, diffing).  The way I intend to model refresh
  is by having a new kind of source, deploy.RefreshSource, which
  will let us do virtually everything about an update/diff the same
  way with refreshes, which avoid otherwise duplicative effort.

  This includes changing the planOptions (nee deployOptions) to
  take a new SourceFunc callback, which is responsible for creating
  a source specific to the kind of plan being requested.

  Preview, Update, and Destroy now are primarily differentiated by
  the kind of deploy.Source that they return, rather than sprinkling
  things like `if Destroying` throughout.  This tidies up some logic
  and, more importantly, gives us precisely the refresh hook we need.

* Originally, we used the deploy.NullSource for Destroy operations.
  This simply returns nothing, which is how Destroy works.  For some
  reason, we were no longer doing this, and instead had some
  `if Destroying` cases sprinkled throughout the deploy.EvalSource.
  I think this is a vestige of some old way we did configuration, at
  least judging by a comment, which is apparently no longer relevant.

* Move diff and diff-printing logic within the engine into its own
  pkg/engine/diff.go file, to prepare for upcoming work.

* I keep noticing benign diffs anytime I regenerate protobufs.  I
  suspect this is because we're also on different versions.  I changed
  generate.sh to also dump the version into grpc_version.txt.  At
  least we can understand where the diffs are coming from, decide
  whether to take them (i.e., a newer version), and ensure that as
  a team we are monotonically increasing, and not going backwards.

* I also tidied up some tiny things I noticed while in there, like
  comments, incorrect types, lint suppressions, and so on.
2018-03-28 07:45:23 -07:00
Pat Gavlin a23b10a9bf
Update the copyright end date to 2018. (#1068)
Just what it says on the tin.
2018-03-21 12:43:21 -07:00
Matt Ellis 936cab0c22 Add a version property to checkpoints
This takes the existing `apitype.Checkpoint` type and renames it to
`apitype.CheckpointV1` locking in the shape. In addition, we introduce
a `apitype.VersionedCheckpoint` type, which holds a version number and
a json document representing a checkpoint at that version. Now, when
reading a checkpoint, the CLI can determine if it's in a format it
understands, and fail gracefully if it is not.

While the CLI understands the older checkpoint version, it always
writes the newest version format, meaning that if you manage a
fire-and-forget stack with this version of the CLI, it will be
un-readable by previous versions.

Stacks managed by Pulumi.com are not impacted by this change.

Fixes: #887
2018-03-10 13:03:05 -08:00
Matthew Riley c0a69a5131 Honor PULUMI_FAILED_TESTS_DIR when destroy fails
We weren't keeping the test directory around when the deferred function
failed the test because the failure happened after we set `testFinished`.
2018-03-07 11:26:42 -08:00
Matt Ellis 70633c2a06 Copy stack config for non aditive tests
The change to refactor out where we store configuration data broke our
old strategy, which we discovered when we tried to take this payload
into pulumi-aws.
2018-03-06 17:32:36 -08:00
Joe Duffy 09cceb4e9e
Remove a few outdated references (#997) 2018-03-04 13:34:20 -08:00
joeduffy 2362d45a5c Eliminate type redundancy
Despite our good progress moving towards having an apitype package,
where our exchange types live and can be shared among the engine and
our services, there were a few major types that were still duplciated.
Resource was the biggest example -- and indeed, the apitype varirant
was missing the new Dependencies property -- but there were others,
like Manfiest, PluginInfo, etc.  These too had semi-random omissions.

This change merges all of these types into the apitype package.  This
not only cleans up the redundancy and missing properties, but will
"force the issue" with respect to keeping them in sync and properly
versioning the information in a backwards compatible way.

The resource/stack package still exists as a simple marshaling layer
to and from the engine's core data types.

Finally, I've made the controversial change to share the actual
Deployment data structure at the apitype layer also.  This will force
us to confront differences in that data structure similarly, and will
allow us to leverage the strong typing throughout to catch issues.
2018-02-28 12:44:55 -08:00
joeduffy 74563afdc8 Get the empty Python program working
This change gets enough of the Python SDK up and running that the
empty Python program will work.  Mostly just scaffolding, but the
basic structure is now in place.  The primary remaining work is to
wire up resource creation to the gRPC interfaces.

In summary:

* The basic structure is as follows:

    - Everything goes into sdk/python/.

    - sdk/python/cmd/pulumi-langhost-python is a Go language host
      that simply knows how to spawn Python processes to run out
      entrypoint in response to requests by the engine.

    - sdk/python/cmd/pulumi-langhost-python-exec is a little Python
      shim that is invoked by the language host to run Python programs,
      and is responsible for setting up the minimal goo before we can
      do so (RPC connections and the like).

    - sdk/python/lib/ contains a Python Pip package suitable for PyPi.

    - In there, we have two packages: the root pulumi package that
      contains all of the basic Pulumi programming model abstractions,
      and pulumi.runtime, which contains the implementation of
      resource registration, RPC interfacing with the engine, and so on.

* Add logic in our test framework to conditionalize on the language
  type and react accordingly.  This will allow us to skip Yarn for
  Python projects and eventually run Pip if there's a requirements.txt.

* Created the basic project structure, including all of the usual
  Make targets for installing into the proper places.

* Building also runs Pylint and we are clean.

There are a few other minor things in here:

* Add an "empty" test for both Node.js and Python.  These pass.

* Fix an existing bug in plugin shutdown logic.  At some point, we
  started waiting for stderr/stdout to flush before shutting down
  the plugin; but if certain failures happen "early" during the
  plugin launch process, these channels will never get initialized
  and so waiting for them deadlocks.

* Recently we seem to have added logic to delete test temp
  directories  if a failure happened during initialization of said
  temp directories.  This is unfortunate, because you often need to
  look at the temp directory to see what failed.  We already clean
  them up elsewhere after the full test completes successfully, so
  I don't think we need to be doing this, and I've removed it.

Still many loose ends (config, resources, etc), but it's a start!
2018-02-23 19:33:02 -08:00
Justin Van Patten ed9716f6ef
Create backups of all local stack checkpoint files (#949)
Backup copies of local stack checkpoints are now saved to the
user's home directory (`~/.pulumi/backups`) by default.

This enables users to recover after accidentally deleting their
local `.pulumi` directory (e.g. via `git clean`).

The behavior can be disabled by setting the
PULUMI_DISABLE_CHECKPOINT_BACKUPS environment variable, which
we use to disable backups when running all tests other than the
test for this functionality.
2018-02-20 21:05:57 -08:00
Joe Duffy 902d646215
Rename package to project (#935)
This addresses pulumi/pulumi#446: what we used to call "package" is
now called "project".  This has gotten more confusing over time, now
that we're doing real package management.

Also fixes pulumi/pulumi#426, while in here.
2018-02-14 13:56:16 -08:00
Matt Ellis a3608e0c2a Retry a few times when running yarn during integration tests
We've seen yarn fail from time to time during our integration
tests (usually timeouts talking to the npm registry) so let's add a
few retries to all yarn commands.
2018-02-12 15:13:35 -08:00
Matthew Riley 606c45ffea Keep test files on assertion failure
We were pretty careful to keep the test directory around if the test ever
exited early due to a panic or error return. But if the test ran to
completion and failed -- for example, if ExtraRuntimeValidation caused the
test to fail -- we would end up deleting the test directory.

Fixes #868
2018-02-01 23:35:49 -08:00
Luke Hoban 0e4fcc16e3
Pass target Cloud and PPC to integation test Reporter (#859)
Ensure that we capture this information in our test reporting so that we can filter on it in queries and reports.
2018-01-30 13:10:32 -08:00
Matt Ellis 803ba93bf5 Pass --network-concurrency 1 to yarn
We've been hitting issues in CI that look like yarnpkg/yarn#4563 and
our hope is that this addresses the issue
2018-01-29 11:49:42 -08:00
Chris Smith 4c217fd358
Add "pulumi history" command (#826)
This PR adds a new `pulumi history` command, which prints the update history for a stack.

The local backend stores the update history in a JSON file on disk, next to the checkpoint file. The cloud backend simply provides the update metadata, and expects to receive all the data from a (NYI) `/history` REST endpoint.

`pkg/backend/updates.go` defines the data that is being persisted. The way the data is wired through the system is adding a new `backend.UpdateMetadata` parameter to a Stack/Backend's `Update` and `Destroy` methods.

I use `tests/integration/stack_outputs/` as the simple app for the related tests, hence the addition to the `.gitignore` and fixing the name in the `Pulumi.yaml`.

Fixes #636.
2018-01-24 18:22:41 -08:00
Matthew Riley 580bbcb3ca Avoid resource leaks by recovering from panics in integration tests 2018-01-20 11:48:42 -08:00
Matt Ellis c506549a25 Remove MustFprintf in favor of explicitly dropping errors
In travis, we've seen cases where writes to our standard streams
results in an error like: `/dev/stderr: resource temporarily
unavailable` which causes the tests to panic.

Now, in a perfect world, writes to /dev/stderr would not fail in this
way, but we do not live in a perfect world. Other processes on the
machine may make stderr/stdout non-blocking. We've are now seeing this
failure in Travis more often and it is masking real Pulumi failures
we want to debug.
2018-01-16 18:33:44 -08:00
Joe Duffy 0d107c742a
Restructure test framework to ease multiple languages (#799)
This change restructures the test framework code a bit, to make it
easier to introduce additional languages.  Our knowledge of Yarn and
Node.js project structure, for instance, was previously baked in to
the test logic, in a way that was hard to make, for instance, Yarn
optional.  (In Python, of course, it will not be used.)  To better
support this, I've moved some state onto a new programTester struct
that we can use to lazily find binaries required during the testing
(such as Yarn, Pip, and so on).  I'm committing this separately so
that I can minimize merge conflicts in the Python work.
2018-01-12 17:10:53 -08:00
Matthew Riley 047a6c3c56 Make "delete temp dir" condition explicit
Previously we checked if the test had failed -- but the call to
`assert.NoError` that actually fails the test sometimes hasn't happened yet.
2018-01-09 23:35:26 -08:00
Matthew Riley d688910360 Remove named parameters from CopyTestToTemporaryDirectory
Fix two references to the now-unnamed `dir` that should have been to
other variables.

Check a real condition before the deferred call to RemoveAll instead of
checking the error return.
2018-01-09 23:35:26 -08:00
Matthew Riley 3c070d3ad5 Restore support for non-Additive edits
None were used in this repository, but `pulumi-aws` used them.
2018-01-09 17:02:30 -08:00
Matthew Riley 9a869eb2d7 Print output of failed commands 2018-01-09 17:02:30 -08:00
Matthew Riley d9fa8237be Use length of correct string for prefixer 2018-01-09 16:48:04 -08:00
pat@pulumi.com c56e716c31 Refactor the engine's entrypoints.
These changes refactor the engine's entrypoints--Deploy, Destroy, and
Preview--to be update-centric rather than stack-centric. Each of these
methods now takes a value of a new type, Update, that abstracts away the
vagaries of fetching and maintaining the update's state. This
refactoring also reinforces Pulumi.yaml as a CLI concept rather than an
engine concept; the CLI is now the only reader/writer of this format.

These changes will smooth the way for a few refactorings on the service
side that will aid in update isolation.
2018-01-08 14:15:16 -08:00
Matthew Riley 9e3976513c AssertNoError instead of Assert(err == nil)
This may not be exhaustive, but I replaced all instances I could find.
2018-01-08 13:46:21 -08:00
Matthew Riley ff10b65f3a Add environment variable to keep data for failed tests
Useful to keep command output and maybe-still-relevant checkpoints around.
2018-01-08 13:46:21 -08:00
Matthew Riley 594ccf755a Write command output to log file 2018-01-08 13:46:21 -08:00
Matthew Riley b0eed85871 Remove non-additive edits
Now we know we only create one temporary directory per test
2018-01-08 13:46:21 -08:00
Matthew Riley afb734a712 Unexport a few private utility functions
Makes it bit easier to see the module entrypoints.
2018-01-08 13:46:21 -08:00
Chris Smith decf814278
Enable running integration tests against the service (#775)
This PR updates the `pkg/testing/integration` package to support running integration tests against the Pulumi Service if desired. This is done through adding new options to `ProgramTestOptions`. (Generally adding support for providing values to flags that were previously inaccessible.)

I added an integration test to confirm that it all works if the PULUMI_API environment variable is set. These tests aren't run in Travis, only manually. Since we cannot reliably run tests from `master` against the service because of the delay in rolling out updates to the Pulumi SDK, etc.
2018-01-03 21:26:50 -08:00
Matt Ellis f510f3c914 Do not allow encrypted global configuration
The cloud backend does not support this because it computes an
encryption key per stack, so we should not support this in the CLI.

Fixes #770
2017-12-27 19:00:55 -08:00
joeduffy af087103b9 Fix more edit directory issues
This fixes a few more edit directory issues, where we didn't
correctly propagate the changes in edit directory required during
subsequent destroy/stack activities.  It also fixes a few error
paths so that we preserve the right directory to be removed.
2017-12-22 06:58:31 -08:00
Joe Duffy b158e9804b
Fix test edit directories (#755)
The test edit directory logic needs to carry forward the checkpoint
from the *prior edit*, not the *original*, as it sequences through edits.
2017-12-21 11:01:30 -08:00
joeduffy 2eda90d67b Fix a dangling fmt parameter 2017-12-21 09:18:00 -08:00
Joe Duffy 26021a41a3
Add the ability to do negative tests
This change adds the ability to do very coarse-grained negative
tests in our integration test framework.  Either a test itself,
or one of its edits, may be marked ExpectFailure == true, at which
point either the preview or update MUST fail (and, if one fails
without this being set, we still treat it as an error).
2017-12-20 12:10:46 -08:00
Luke Hoban d5acc3da87
Add option to specify command line flags to pulumi update in integration tests (#749)
And use this in the diff test which needs to apply `--colors=raw`.
2017-12-20 11:17:25 -08:00
CyrusNajmabadi a5cf6f25d6
Add a framework for baselining and validating pulumi-update diff output. (#700) 2017-12-14 17:10:05 -08:00
Luke Hoban 6f15fa8ed8
Pass more stack info to ExtraRuntimeValidation (#717)
This will allow us to remove a lot of current boilerplate in individual tests, and move it into the test harness.

Note that this will require updating users of the integration test framework.  By moving to a property bag of inputs, we should avoid needing future breaking changes to this API though.
2017-12-13 16:09:14 -08:00
CyrusNajmabadi eea5232cce
Break out additional test helper. (#699) 2017-12-11 15:48:45 -08:00
CyrusNajmabadi 9bc9ce1505
More test code refactoring.
* Move code down.

* Extact test code out.

* Extract helper and pattern.

* Remove unnecessary code.

* move comment.
2017-12-11 14:42:42 -08:00
CyrusNajmabadi 7234cdef02
Take an options pointer so values can change as a test runs. (#679)
* Take an options pointer so values can change as a test runs.

* Don't pass redundant information.

* Extract initialization routine.

* Fix caller.

* Check return value.

* Extract destruction logic.

* Move preview and update into their own function.

* Inline null check.
2017-12-08 16:59:28 -08:00
CyrusNajmabadi edb7bf849e
Reverts (#676)
* Revert "Make sure we properly update dir so that pulumi-destroy works."

This reverts commit 56bfc57998.

* Revert "Edits needs to continuously pass along the new directory. (#668)"

This reverts commit 8bd1822722.

* Revert "Refactor test code to make it simpler to validate code in the middle. (#662)"

This reverts commit ed65360157.
2017-12-08 12:59:39 -08:00
CyrusNajmabadi 56bfc57998
Make sure we properly update dir so that pulumi-destroy works. 2017-12-08 11:37:38 -08:00
CyrusNajmabadi 8bd1822722
Edits needs to continuously pass along the new directory. (#668) 2017-12-08 01:51:35 -08:00
CyrusNajmabadi ed65360157
Refactor test code to make it simpler to validate code in the middle. (#662)
* Refactor test code to make it simpler to validate code in the middle.
2017-12-07 16:29:48 -08:00
Joe Duffy 971f6189f2
Fix pending delete replacement failure (#658)
The two-phase output properties change broke the ability to recover
from a failed replacement that yields pending deletes in the checkpoint.
The issue here is simply that we should remember pending registrations
only for logical operations that *also* have a "new" state (create or
update).  This commit fixes this, and also adds a new step test with
fault injection to probe many interesting combinations of steps.
2017-12-07 09:44:38 -08:00
joeduffy b59b8f2e6e Fix cloud tests 2017-12-03 06:34:06 -08:00
joeduffy 1c4e41b916 Improve the overall cloud CLI experience
This improves the overall cloud CLI experience workflow.

Now whether a stack is local or cloud is inherent to the stack
itself.  If you interact with a cloud stack, we transparently talk
to the cloud; if you interact with a local stack, we just do the
right thing, and perform all operations locally.  Aside from sometimes
seeing a cloud emoji pop-up ☁️, the experience is quite similar.

For example, to initialize a new cloud stack, simply:

    $ pulumi login
    Logging into Pulumi Cloud: https://pulumi.com/
    Enter Pulumi access token: <enter your token>
    $ pulumi stack init my-cloud-stack

Note that you may log into a specific cloud if you'd like.  For
now, this is just for our own testing purposes, but someday when we
support custom clouds (e.g., Enterprise), you can just say:

    $ pulumi login --cloud-url https://corp.acme.my-ppc.net:9873

The cloud is now the default.  If you instead prefer a "fire and
forget" style of stack, you can skip the login and pass `--local`:

    $ pulumi stack init my-faf-stack --local

If you are logged in and run `pulumi`, we tell you as much:

    $ pulumi
    Usage:
      pulumi [command]

    // as before...

    Currently logged into the Pulumi Cloud ☁️
        https://pulumi.com/

And if you list your stacks, we tell you which one is local or not:

    $ pulumi stack ls
    NAME            LAST UPDATE       RESOURCE COUNT   CLOUD URL
    my-cloud-stack  2017-12-01 ...    3                https://pulumi.com/
    my-faf-stack    n/a               0                n/a

And `pulumi stack` by itself prints information like your cloud org,
PPC name, and so on, in addition to the usuals.

I shall write up more details and make sure to document these changes.

This change also fairly significantly refactors the layout of cloud
versus local logic, so that the cmd/ package is resonsible for CLI
things, and the new pkg/backend/ package is responsible for the
backends.  The following is the overall resulting package architecture:

* The backend.Backend interface can be implemented to substitute
  a new backend.  This has operations to get and list stacks,
  perform updates, and so on.

* The backend.Stack struct is a wrapper around a stack that has
  or is being manipulated by a Backend.  It resembles our existing
  Stack notions in the engine, but carries additional metadata
  about its source.  Notably, it offers functions that allow
  operations like updating and deleting on the Backend from which
  it came.

* There is very little else in the pkg/backend/ package.

* A new package, pkg/backend/local/, encapsulates all local state
  management for "fire and forget" scenarios.  It simply implements
  the above logic and contains anything specific to the local
  experience.

* A peer package, pkg/backend/cloud/, encapsulates all logic
  required for the cloud experience.  This includes its subpackage
  apitype/ which contains JSON schema descriptions required for
  REST calls against the cloud backend.  It also contains handy
  functions to list which clouds we have authenticated with.

* A subpackage here, pkg/backend/state/, is not a provider at all.
  Instead, it contains all of the state management functions that
  are currently shared between local and cloud backends.  This
  includes configuration logic -- including encryption -- as well
  as logic pertaining to which stacks are known to the workspace.

This addresses pulumi/pulumi#629 and pulumi/pulumi#494.
2017-12-02 14:34:42 -08:00
Chris Smith 64705b0a73 Ignore unreasonable lint warning 2017-11-30 18:55:18 -08:00
joeduffy 76b77bdabf Add ability to enable verbose logging in tests 2017-11-30 17:23:58 -08:00
Chris Smith 454f946e8c
Wire Package.Main to the Pulumi Service. (#615)
This PR just wires the `Package.Main` field to the Pulumi Service (and in subsequent PRs, the `pulumi-service` and `pulumi-ppc` repos).

@joeduffy , should we just upload the entire `package.Package` type with the `UpdateProgramRequest` type? I'm not sure we want to treat that type as part of part of our public API surface area. But on the other hand, we'll need to mirror relevant fields in N places if we don't.
2017-11-30 08:14:47 -08:00
joeduffy 5762f2d0a6 Merge remote-tracking branch 'origin/resource_parenting' into resource_parenting_lite 2017-11-28 11:03:34 -08:00
Pat Gavlin 503e920198 Optionally enable debug output in integration tests.
By default, debugging events are not displayed by `pulumi`; the `-d`
flag must be provided to enable their display. This is necessary e.g.
to view debugging output from Terraform when TF logging is enabled.
These changes add the option to display debug output from `pulumi` to
the integration test framework.

These changes also contain a small fix for the display of component
children.
2017-11-24 15:21:43 -08:00
Matt Ellis 5fc226a952 Change configuration verbs for getting and setting values
A handful of UX improvments for config:

 - `pulumi config ls` has been removed. Now, `pulumi config` with no
   arguments prints the table of configuration values for a stack and
   a new command `pulumi config get <key>` prints the value for a
   single configuration key (useful for scripting).
 - `pulumi config text` and `pulumi config secret` have been merged
   into a single command `pulumi config set`. The flag `--secret` can
   be used to encrypt the value we store (like `pulumi config secret`
   used to do).
 - To make it obvious that setting a value with `pulumi config set` is
   in plan text, we now echo a message back to the user saying we
   added the configuration value in plaintext.

Fixes #552
2017-11-16 11:39:28 -08:00
Joe Duffy 77460a7dc0
Plumb the project name correctly (#583)
This change fixes getProject to return the project name, as
originally intended.  (One line was missing.)

It also adds an integration test for this.

Fixes pulumi/pulumi#580.
2017-11-16 08:15:56 -08:00
Joe Duffy 98ef0c4bb5
Allow overriding a Pulumi.yaml's entrypoint (#582)
Because the Pulumi.yaml file demarcates the boundary used when
uploading a program to the Pulumi.com service at the moment, we
have trouble when a Pulumi program uses "up and over" references.
For instance, our customer wants to build a Dockerfile located
in some relative path, such as `../../elsewhere/`.

To support this, we will allow the Pulumi.yaml file to live
somewhere other than the main Pulumi entrypoint.  For example,
it can live at the root of the repo, while the Pulumi program
lives in, say, `infra/`:

    Pulumi.yaml:
    name: as-before
    main: infra/

This fixes pulumi/pulumi#575.  Further work can be done here to
provide even more flexibility; see pulumi/pulumi#574.
2017-11-16 07:49:07 -08:00
Luke Hoban 4730ab5a0b
Add integration test support for relative work dir (#567)
Adds the ability to use a working directory nested within an example folder.
2017-11-14 21:02:47 -08:00
Luke Hoban a350b1654f
Report test run data and timing to S3 (#560)
Add the ability to upload data and timing for test runs to S3. Uploaded data is designed to be queried via a service like AWS Athena and these queries can then be imported into BI tools (Excel, QuickSight, PowerBI, etc.)

Initially hook this up to the `minimal` test as a baseline.
2017-11-14 09:33:22 -08:00
pat@pulumi.com 0b6b211e69 Do not delete temp directories during tests.
This interacts badly with the deferred destroy.
2017-11-08 16:51:55 -08:00
Matt Ellis 098b2b8401 Remove integration test working folder if tests pass 2017-11-06 11:50:21 -08:00
Matt Ellis aa80b7dee9 Don't pass PULUMI_API when running integration tests
We are not ready to start running integration tests against
Pulumi.com, but `pulumi` uses `PULUMI_API` to decide what backend to
target. Remove it when running `pulumi` in the integration tests so we
always get the local behavior.

pulumi/pulumi#471 tracks the work to actually be able to choose what
backend you want when an integration test runs.
2017-11-06 11:50:16 -08:00
Joe Duffy d5ef7a8538
Exit early during integration tests (#530)
This changes the overall flow to exit early when a failure occurs.
We will try to clean up if we believe to have gotten far enough to
actually have provisioned any resources.
2017-11-06 09:04:38 -08:00
joeduffy 8fd4770dcf Cap test/host names at 10 chars, to keep names reasonable 2017-11-06 08:22:45 -08:00
joeduffy 145cdf1f71 Also use stack names as temp dir names 2017-11-06 07:50:28 -08:00
Joe Duffy 68e23f03c1
Add hostname and testdir to integration test stack names (#529)
This should help us attribute resources allocated (and orphaned) by
tests a little easier, including tracking down any leaking resources.
2017-11-06 07:23:07 -08:00
joeduffy 4d19a71efb Remove extra newline to shorten logs 2017-11-05 17:52:40 -08:00
joeduffy 7b045bd7af Retain stdout/stderr for integration test commands
...and only print it upon error, as part of pulumi/pulumi#518.
2017-11-05 17:28:12 -08:00
joeduffy 6d74b3ca27 Add minimal runtime verification test
This adds a minimal runtime verification test to our basic
test suite, to at least exercise the portions of the integration
test library that load up and parse checkpoint files.
2017-10-27 20:03:38 -07:00
joeduffy 7835305b82 Fix where integration tests look for checkpoints 2017-10-27 19:42:17 -07:00
Matt Ellis 3f1197ef84 Move .pulumi to root of a repository
Now, instead of having a .pulumi folder next to each project, we have
a single .pulumi folder in the root of the repository. This is created
by running `pulumi init`.

When run in a git repository, `pulumi init` will place the .pulumi
file next to the .git folder, so it can be shared across all projects
in a repository. When not in a git repository, it will be created in
the current working directory.

We also start tracking information about the repository itself, in a
new `repo.json` file stored in the root of the .pulumi folder. The
information we track are "owner" and "name" which map to information
we use on pulumi.com.

When run in a git repository with a remote named origin pointing to a
GitHub project, we compute the owner and name by deconstructing
information from the remote's URL. Otherwise, we just use the current
user's username and the name of the current working directory as the
owner and name, respectively.
2017-10-27 11:46:21 -07:00
CyrusNajmabadi 44d95666e7 Make it possible to do extra validation tests after applying an edit. (#476)
* Make it possible to do extra validation tests after applying an edit.

* Chane name.
2017-10-26 16:01:28 -07:00
Matt Ellis ade366544e Encrypt secrets in Pulumi.yaml
We now encrypt secrets at rest based on a key derived from a user
suplied passphrase.

The system is designed in a way such that we should be able to have a
different decrypter (either using a local key or some remote service
in the Pulumi.com case in the future).

Care is taken to ensure that we do not leak decrypted secrets into the
"info" section of the checkpoint file (since we currently store the
config there).

In addtion, secrets are "pay for play", a passphrase is only needed
when dealing with a value that's encrypted. If secure config values
are not used, `pulumi` will never prompt you for a
passphrase. Otherwise, we only prompt if we know we are going to need
to decrypt the value. For example, `pulumi config <key>` only prompts
if `<key>` is encrypted and `pulumi deploy` and friends only prompt if
you are targeting a stack that has secure configuration assoicated
with it.

Secure values show up as unecrypted config values inside the language
hosts and providers.
2017-10-24 16:48:12 -07:00
Matt Ellis 25b8111eea Retain historical checkpoints
When `PULUMI_RETAIN_CHECKPOINTS` is set in the environment, also write
the checkpoint file to <stack-name>.<ext>.<timestamp>.

This ensures we have historical information about every snapshot, which
would aid in debugging issues like #451. We set this to true for our
integration tests.

Fixes #453
2017-10-24 11:48:33 -07:00
joeduffy 0248b6c309 Fix an err shadowing lint error 2017-10-23 05:55:34 -07:00
joeduffy a7d99a0c80 Preserve Pulumi.yaml while applying edits
Now that config is stored in Pulumi.yaml, we need to mimic the behavior
around .pulumi/ while edits are applied.  This will ensure that config
values carry forward from the original program settings.

This fixes pulumi/pulumi-aws#48.
2017-10-23 05:27:26 -07:00
joeduffy c6ee323ce1 Add some "still running..." messages to Pulumi tests 2017-10-22 18:54:29 -07:00
Matt Ellis b04732f76b Rename env folder to stacks 2017-10-17 17:44:53 -07:00
Matt Ellis 8fda24a776 Write a .yarnrc file when running tests to enable mutex
In #411 we started to run tests in parallel again. To support that, we
added a .yarnrc file in the root of our repository to pass --mutex
network to all yarn invocations, because tests may run yarn commands
concurrently with one another and yarn is not safe to run
concurrently.

However, when we run the integration tests, we actually copy them into
a folder under `/tmp` and so yarn's logic to walk up the directory to
tree to find a `.yarnrc` will not see this `.yarnrc` and we're back
where we started.

Have the testing package explicitly write this file, which should
prevent this issue from happening in the future.
2017-10-16 16:08:50 -07:00
Matt Ellis 22c9e0471c Use Stack over Environment to describe a deployment target
Previously we used the word "Environment" as the term for a deployment
target, but since then we've started to use the term Stack. Adopt this
across the CLI.

From a user's point of view, there are a few changes:

1. The `env` verb has been renamed to `stack`
2. The `-e` and `--env` options to commands which operate on an
environment now take `-s` or `--stack` instead.
3. Becase of (2), the commands that used `-s` to display a summary now
only support passing the full option name (`--summary`).

On the local file system, we still store checkpoint data in the `env`
sub-folder under `.pulumi` (so we can reuse existing checkpoint files
that were written to the old folder)
2017-10-16 13:04:20 -07:00
Matt Ellis 6bab1dbad4 Pass $(YARNFLAGS) to all yarn invocations
This let's you set things like YARNFLAGS==--offline which is helpful
when you are on an airplane. Yarn can still pick up stuff that you had
pulled down recently from its local cache
2017-10-09 18:21:55 -07:00
Cyrus Najmabadi bdd819379e Extract out test helper functionality for use from pulumi-cloud. 2017-09-25 14:14:09 -07:00
pat@pulumi.com 69341fa7c8 push is dead; long live update.
After discussion with Joe and Luke, we've decided to use `update` instead
of `push` as it more intuitively fits the operation being performed.
2017-09-22 17:23:40 -07:00
pat@pulumi.com 597db186ec Renames: plan -> preview, deploy -> push.
Part of #353.

These changes also remove all command aliases from the `pulumi` command.
2017-09-22 15:28:03 -07:00
Joe Duffy f6e694c72b Rename pulumi-fabric to pulumi
This includes a few changes:

* The repo name -- and hence the Go modules -- changes from pulumi-fabric to pulumi.

* The Node.js SDK package changes from @pulumi/pulumi-fabric to just pulumi.

* The CLI is renamed from lumi to pulumi.
2017-09-21 19:18:21 -07:00
Renamed from pkg/testing/integration/lumiprogram.go (Browse further)