Commit graph

40 commits

Author SHA1 Message Date
Sean Gillespie 491bcdc602
Add a list of in-flight operations to the deployment (#1759)
* Add a list of in-flight operations to the deployment

This commit augments 'DeploymentV2' with a list of operations that are
currently in flight. This information is used by the engine to keep
track of whether or not a particular deployment is in a valid state.

The SnapshotManager is responsible for inserting and removing operations
from the in-flight operation list. When the engine registers an intent
to perform an operation, SnapshotManager inserts an Operation into this
list and saves it to the snapshot. When an operation completes, the
SnapshotManager removes it from the snapshot. From this, the engine can
infer that if it ever sees a deployment with pending operations, the
Pulumi CLI must have crashed or otherwise abnormally terminated before
seeing whether or not an operation completed successfully.

To remedy this state, this commit also adds code to 'pulumi stack
import' that clears all pending operations from a deployment, as well as
code to plan generation that will reject any deployments that have
pending operations present.

At the CLI level, if we see that we are in a state where pending
operations were in-flight when the engine died, we'll issue a
human-friendly error message that indicates which resources are in a bad
state and how to recover their stack.

* CR: Multi-line string literals, renaming in-flight -> pending

* CR: Add enum to apitype for operation type, also name status -> type for clarity

* Fix the yaml type

* Fix missed renames

* Add implementation for lifecycle_test.go

* Rebase against master
2018-08-10 21:39:59 -07:00
Pat Gavlin a222705143
Implement first-class providers. (#1695)
### First-Class Providers
These changes implement support for first-class providers. First-class
providers are provider plugins that are exposed as resources via the
Pulumi programming model so that they may be explicitly and multiply
instantiated. Each instance of a provider resource may be configured
differently, and configuration parameters may be source from the
outputs of other resources.

### Provider Plugin Changes
In order to accommodate the need to verify and diff provider
configuration and configure providers without complete configuration
information, these changes adjust the high-level provider plugin
interface. Two new methods for validating a provider's configuration
and diffing changes to the same have been added (`CheckConfig` and
`DiffConfig`, respectively), and the type of the configuration bag
accepted by `Configure` has been changed to a `PropertyMap`.

These changes have not yet been reflected in the provider plugin gRPC
interface. We will do this in a set of follow-up changes. Until then,
these methods are implemented by adapters:
- `CheckConfig` validates that all configuration parameters are string
  or unknown properties. This is necessary because existing plugins
  only accept string-typed configuration values.
- `DiffConfig` either returns "never replace" if all configuration
  values are known or "must replace" if any configuration value is
  unknown. The justification for this behavior is given
  [here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106)
- `Configure` converts the config bag to a legacy config map and
  configures the provider plugin if all config values are known. If any
  config value is unknown, the underlying plugin is not configured and
  the provider may only perform `Check`, `Read`, and `Invoke`, all of
  which return empty results. We justify this behavior becuase it is
  only possible during a preview and provides the best experience we
  can manage with the existing gRPC interface.

### Resource Model Changes
Providers are now exposed as resources that participate in a stack's
dependency graph. Like other resources, they are explicitly created,
may have multiple instances, and may have dependencies on other
resources. Providers are referred to using provider references, which
are a combination of the provider's URN and its ID. This design
addresses the need during a preview to refer to providers that have not
yet been physically created and therefore have no ID.

All custom resources that are not themselves providers must specify a
single provider via a provider reference. The named provider will be
used to manage that resource's CRUD operations. If a resource's
provider reference changes, the resource must be replaced. Though its
URN is not present in the resource's dependency list, the provider
should be treated as a dependency of the resource when topologically
sorting the dependency graph.

Finally, `Invoke` operations must now specify a provider to use for the
invocation via a provider reference.

### Engine Changes
First-class providers support requires a few changes to the engine:
- The engine must have some way to map from provider references to
  provider plugins. It must be possible to add providers from a stack's
  checkpoint to this map and to register new/updated providers during
  the execution of a plan in response to CRUD operations on provider
  resources.
- In order to support updating existing stacks using existing Pulumi
  programs that may not explicitly instantiate providers, the engine
  must be able to manage the "default" providers for each package
  referenced by a checkpoint or Pulumi program. The configuration for
  a "default" provider is taken from the stack's configuration data.

The former need is addressed by adding a provider registry type that is
responsible for managing all of the plugins required by a plan. In
addition to loading plugins froma checkpoint and providing the ability
to map from a provider reference to a provider plugin, this type serves
as the provider plugin for providers themselves (i.e. it is the
"provider provider").

The latter need is solved via two relatively self-contained changes to
plan setup and the eval source.

During plan setup, the old checkpoint is scanned for custom resources
that do not have a provider reference in order to compute the set of
packages that require a default provider. Once this set has been
computed, the required default provider definitions are conjured and
prepended to the checkpoint's resource list. Each resource that
requires a default provider is then updated to refer to the default
provider for its package.

While an eval source is running, each custom resource registration,
resource read, and invoke that does not name a provider is trapped
before being returned by the source iterator. If no default provider
for the appropriate package has been registered, the eval source
synthesizes an appropriate registration, waits for it to complete, and
records the registered provider's reference. This reference is injected
into the original request, which is then processed as usual. If a
default provider was already registered, the recorded reference is
used and no new registration occurs.

### SDK Changes
These changes only expose first-class providers from the Node.JS SDK.
- A new abstract class, `ProviderResource`, can be subclassed and used
  to instantiate first-class providers.
- A new field in `ResourceOptions`, `provider`, can be used to supply
  a particular provider instance to manage a `CustomResource`'s CRUD
  operations.
- A new type, `InvokeOptions`, can be used to specify options that
  control the behavior of a call to `pulumi.runtime.invoke`. This type
  includes a `provider` field that is analogous to
  `ResourceOptions.provider`.
2018-08-06 17:50:29 -07:00
Sean Gillespie 48aa5e73f8
Save resources obtained from ".get" in the snapshot (#1654)
* Protobuf changes to record dependencies for read resources

* Add a number of tests for read resources, especially around replacement

* Place read resources in the snapshot with "external" bit set

Fixes pulumi/pulumi#1521. This commit introduces two new step ops: Read
and ReadReplacement. The engine generates Read and ReadReplacement steps
when servicing ReadResource RPC calls from the language host.

* Fix an omission of OpReadReplace from the step list

* Rebase against master

* Transition to use V2 Resources by default

* Add a semantic "relinquish" operation to the engine

If the engine observes that a resource is read and also that the
resource exists in the snapshot as a non-external resource, it will not
delete the resource if the IDs of the old and new resources match.

* Typo fix

* CR: add missing comments, DeserializeDeployment -> DeserializeDeploymentV2, ID check
2018-08-03 14:06:00 -07:00
Sean Gillespie 48d095d66f Remove "down" operation for migration - it is not semantically valid in
general.
2018-07-27 14:52:28 -07:00
Alex Clemmer f037c7d143 Checkpoint resource initialization errors
When a resource fails to initialize (i.e., it is successfully created,
but fails to transition to a fully-initialized state), and a user
subsequently runs `pulumi update` without changing that resource, our
CLI will fail to warn the user that this resource is not initialized.

This commit begins the process of allowing our CLI to report this by
storing a list of initialization errors in the checkpoint.
2018-07-20 17:59:06 -07:00
Sean Gillespie 80c28c00d2
Add a migrate package for migrating to and from differently-versioned API types (#1647)
* Add a migrate package for migrating to and from differently-versioned
API types

* travis: gofmt -s deployment_test.go
2018-07-20 13:31:41 -07:00
Sean Gillespie 924c49d7e0
Fail fast when attempting to load a too-new or too-old deployment (#1382)
* Error when loading a deployment that is not a version that the CLI understands

* Add a test for 'pulumi stack import' on a badly-versioned deployment

* Move current deployment version to 'apitype'

* Rebase against master

* CR: emit CLI-friendly error message at the two points outside of the engine calling 'DeserializeDeployment'
2018-05-25 13:29:59 -07:00
joeduffy 5967259795 Add license headers 2018-05-22 15:02:47 -07:00
Sean Gillespie 7b7870cdaa
Remove unused stack name from deploy.Snapshot (#1386) 2018-05-18 11:15:35 -07:00
Matt Ellis bac02f1df1 Remove the need to pulumi init for the local backend
This change removes the need to `pulumi init` when targeting the local
backend. A fair amount of the change lays the foundation that the next
set of changes to stop having `pulumi init` be used for cloud stacks
as well.

Previously, `pulumi init` logically did two things:

1. It created the bookkeeping directory for local stacks, this was
stored in `<repository-root>/.pulumi`, where `<repository-root>` was
the path to what we belived the "root" of your project was. In the
case of git repositories, this was the directory that contained your
`.git` folder.

2. It recorded repository information in
`<repository-root>/.pulumi/repository.json`. This was used by the
cloud backend when computing what project to interact with on
Pulumi.com

The new identity model will remove the need for (2), since we only
need an owner and stack name to fully qualify a stack on
pulumi.com, so it's easy enough to stop creating a folder just for
that.

However, for the local backend, we need to continue to retain some
information about stacks (e.g. checkpoints, history, etc). In
addition, we need to store our workspace settings (which today just
contains the selected stack) somehere.

For state stored by the local backend, we change the URL scheme from
`local://` to `local://<optional-root-path>`. When
`<optional-root-path>` is unset, it defaults to `$HOME`. We create our
`.pulumi` folder in that directory. This is important because stack
names now must be unique within the backend, but we have some tests
using local stacks which use fixed stack names, so each integration
test really wants its own "view" of the world.

For the workspace settings, we introduce a new `workspaces` directory
in `~/.pulumi`. In this folder we write the workspace settings file
for each project. The file name is the name of the project, combined
with the SHA1 of the path of the project file on disk, to ensure that
multiple pulumi programs with the same project name have different
workspace settings.

This does mean that moving a project's location on disk will cause the
CLI to "forget" what the selected stack was, which is unfortunate, but
not the end of the world. If this ends up being a big pain point, we
can certianly try to play games in the future (for example, if we saw
a .git folder in a parent folder, we could store data in there).

With respect to compatibility, we don't attempt to migrate older files
to their newer locations. For long lived stacks managed using the
local backend, we can provide information on where to move things
to. For all stacks (regardless of backend) we'll require the user to
`pulumi stack select` their stack again, but that seems like the
correct trade-off vs writing complicated upgrade code.
2018-04-18 04:53:49 -07:00
Sean Gillespie 55711e4ca3
Revert "Lift snapshot management out of the engine and serialize writes to snapshot (#1069)" (#1216)
This reverts commit 2c479c172d.
2018-04-16 23:04:56 -07:00
Sean Gillespie 2c479c172d
Lift snapshot management out of the engine and serialize writes to snapshot (#1069)
* Lift snapshot management out of the engine

This PR is a prerequisite for parallelism by addressing a major problem
that the engine has to deal with when performing parallel resource
construction: parallel mutation of the global snapshot. This PR adds
a `SnapshotManager` type that is responsible for maintaining and
persisting the current resource snapshot. It serializes all reads and
writes to the global snapshot and persists the snapshot to persistent
storage upon every write.

As a side-effect of this, the core engine no longer needs to know about
snapshot management at all; all snapshot operations can be handled as
callbacks on deployment events. This will greatly simplify the
parallelization of the core engine.

Worth noting is that the core engine will still need to be able to read
the current snapshot, since it is interested in the dependency graphs
contained within. The full implications of that are out of scope of this
PR.

Remove dead code, Steps no longer need a reference to the plan iterator that created them

Fixing various issues that arise when bringing up pulumi-aws

Line length broke the build

Code review: remove dead field, fix yaml name error

Rebase against master, provide implementation of StackPersister for cloud backend

Code review feedback: comments on MutationStatus, style in snapshot.go

Code review feedback: move SnapshotManager to pkg/backend, change engine to use an interface SnapshotManager

Code review feedback: use a channel for synchronization

Add a comment and a new test

* Maintain two checkpoints, an immutable base and a mutable delta, and
periodically merge the two to produce snapshots

* Add a lot of tests - covers all of the non-error paths of BeginMutation and End

* Fix a test resource provider

* Add a few tests, fix a few issues

* Rebase against master, fixed merge
2018-04-12 09:55:34 -07:00
Pat Gavlin a23b10a9bf
Update the copyright end date to 2018. (#1068)
Just what it says on the tin.
2018-03-21 12:43:21 -07:00
Matt Ellis 936cab0c22 Add a version property to checkpoints
This takes the existing `apitype.Checkpoint` type and renames it to
`apitype.CheckpointV1` locking in the shape. In addition, we introduce
a `apitype.VersionedCheckpoint` type, which holds a version number and
a json document representing a checkpoint at that version. Now, when
reading a checkpoint, the CLI can determine if it's in a format it
understands, and fail gracefully if it is not.

While the CLI understands the older checkpoint version, it always
writes the newest version format, meaning that if you manage a
fire-and-forget stack with this version of the CLI, it will be
un-readable by previous versions.

Stacks managed by Pulumi.com are not impacted by this change.

Fixes: #887
2018-03-10 13:03:05 -08:00
joeduffy 2362d45a5c Eliminate type redundancy
Despite our good progress moving towards having an apitype package,
where our exchange types live and can be shared among the engine and
our services, there were a few major types that were still duplciated.
Resource was the biggest example -- and indeed, the apitype varirant
was missing the new Dependencies property -- but there were others,
like Manfiest, PluginInfo, etc.  These too had semi-random omissions.

This change merges all of these types into the apitype package.  This
not only cleans up the redundancy and missing properties, but will
"force the issue" with respect to keeping them in sync and properly
versioning the information in a backwards compatible way.

The resource/stack package still exists as a simple marshaling layer
to and from the engine's core data types.

Finally, I've made the controversial change to share the actual
Deployment data structure at the apitype layer also.  This will force
us to confront differences in that data structure similarly, and will
allow us to leverage the strong typing throughout to catch issues.
2018-02-28 12:44:55 -08:00
Sean Gillespie 9757d069ed
Merge pull request #972 from pulumi/swgillespie/dependency-view
Save resource dependency information in the checkpoint file
2018-02-23 11:19:38 -08:00
Sean Gillespie b84320b45e
Code review feedback:
1. Various idiomatic Go and TypeScript fixes
    2. Add an integration test that end-to-end roundtrips dependency
    information for a simple Pulumi program
    3. Add an additional test assert that tests that dependency information
    comes from the language host as expected
2018-02-22 13:33:50 -08:00
Sean Gillespie ad06e9b0d8
Save resource dependency information in the checkpoint file
This commit does two things:
    1. All dependencies of a resource, both implicit and explicit, are
    communicated directly to the engine when registering a resource. The
    engine keeps track of these dependencies and ultimately serializes
    them out to the checkpoint file upon successful deployment.
    2. Once a successful deployment is done, the new `pulumi stack
    graph` command reads the checkpoint file and outputs the dependency
    information within in the DOT format.

Keeping track of dependency information within the checkpoint file is
desirable for a number of reasons, most notably delete-before-create,
where we want to delete resources before we have created their
replacement when performing an update.
2018-02-21 17:49:09 -08:00
Joe Duffy e7af13e144
Capture plugin names in the manifest (#967)
Previously, the checkpoint manifest contained the full path to a plugin
binary, in places of its friendly name.  Now that we must move to a model
where we install plugins in the PPC based on the manifest contents, we
actually need to store the name, in addition to the version (which is
already there).  We still also capture the path for debugging purposes.
2018-02-21 10:32:31 -08:00
joeduffy c04341edb2 Consult the program for its list of plugins
This change adds a GetRequiredPlugins RPC method to the language
host, enabling us to query it for its list of plugin requirements.
This is language-specific because it requires looking at the set
of dependencies (e.g., package.json files).

It also adds a call up front during any update/preview operation
to compute the set of plugins and require that they are present.
These plugins are populated in the cache and will be used for all
subsequent plugin-related operations during the engine's activity.

We now cache the language plugins, so that we may load them
eagerly too, which we never did previously due to the fact that
we needed to pass the monitor address at load time.  This was a
bit bizarre anyhow, since it's really the Run RPC function that
needs this information.  So, to enable caching and eager loading
-- which we need in order to invoke GetRequiredPlugins -- the
"phone home" monitor RPC address is passed at Run time.

In a subsequent change, we will switch to faulting in the plugins
that are missing -- rather than erroring -- in addition to
supporting the `pulumi plugin install` CLI command.
2018-02-18 08:08:15 -08:00
Joe Duffy bc2cf55463
Implement resource protection (#751)
This change implements resource protection, as per pulumi/pulumi#689.
The overall idea is that a resource can be marked as "protect: true",
which will prevent deletion of that resource for any reason whatsoever
(straight deletion, replacement, etc).  This is expressed in the
program.  To "unprotect" a resource, one must perform an update setting
"protect: false", and then afterwards, they can delete the resource.

For example:

    let res = new MyResource("precious", { .. }, { protect: true });

Afterwards, the resource will display in the CLI with a lock icon, and
any attempts to remove it will fail in the usual ways (in planning or,
worst case, during an actual update).

This was done by adding a new ResourceOptions bag parameter to the
base Resource types.  This is unfortunately a breaking change, but now
is the right time to take this one.  We had been adding new settings
one by one -- like parent and dependsOn -- and this new approach will
set us up to add any number of additional settings down the road,
without needing to worry about breaking anything ever again.

This is related to protected stacks, as described in
pulumi/pulumi-service#399.  Most likely this will serve as a foundational
building block that enables the coarser grained policy management.
2017-12-20 14:31:07 -08:00
Luke Hoban 6f15fa8ed8
Pass more stack info to ExtraRuntimeValidation (#717)
This will allow us to remove a lot of current boilerplate in individual tests, and move it into the test harness.

Note that this will require updating users of the integration test framework.  By moving to a property bag of inputs, we should avoid needing future breaking changes to this API though.
2017-12-13 16:09:14 -08:00
Joe Duffy 1681119339
Add a stack output command (#675)
This change adds a `pulumi stack output` command.  When passed no
arguments, it prints all stack output properties, in exactly the
same format as `pulumi stack` does (just without all the other stuff).
More importantly, if you pass a specific output property, a la
`pulumi stack output clusterARN`, just that property will be printed,
in a scriptable-friendly manner.  This will help us automate wiring
multiple layers of stacks together during deployments.

This fixes pulumi/pulumi#659.
2017-12-08 13:14:58 -08:00
CyrusNajmabadi 75ee89f2b1
Always add 8 chars of randomness to URN names we create. Error if that exceeds the max length allowed for that resource. (#500)
* Include parent type in urn to better ensure component URN uniqueness.
2017-12-04 14:50:55 -08:00
Pat Gavlin f848090479 Return all computed inputs from Provider.Check.
As documented in issue #616, the inputs/defaults/outputs model we have
today has fundamental problems. The crux of the issue is that our
current design requires that defaults present in the old state of a
resource are applied to the new inputs for that resource.
Unfortunately, it is not possible for the engine to decide which
defaults remain applicable and which do not; only the provider has that
knowledge.

These changes take a more tactical approach to resolving this issue than
that originally proposed in #616 that avoids breaking compatibility with
existing checkpoints. Rather than treating the Pulumi inputs as the
provider input properties for a resource, these inputs are first
translated by `Check`. In order to accommodate provider defaults that
were chosen for the old resource but should not change for the new,
`Check` now takes the old provider inputs as well as the new Pulumi
inputs. Rather than the Pulumi inputs and provider defaults, the
provider inputs returned by `Check` are recorded in the checkpoint file.

Put simply, these changes remove defaults as a first-class concept
(except inasmuch as is required to retain the ability to read old
checkpoint files) and move the responsibilty for manging and
merging defaults into the provider that supplies them.

Fixes #616.
2017-12-03 09:33:16 -08:00
joeduffy 1c4e41b916 Improve the overall cloud CLI experience
This improves the overall cloud CLI experience workflow.

Now whether a stack is local or cloud is inherent to the stack
itself.  If you interact with a cloud stack, we transparently talk
to the cloud; if you interact with a local stack, we just do the
right thing, and perform all operations locally.  Aside from sometimes
seeing a cloud emoji pop-up ☁️, the experience is quite similar.

For example, to initialize a new cloud stack, simply:

    $ pulumi login
    Logging into Pulumi Cloud: https://pulumi.com/
    Enter Pulumi access token: <enter your token>
    $ pulumi stack init my-cloud-stack

Note that you may log into a specific cloud if you'd like.  For
now, this is just for our own testing purposes, but someday when we
support custom clouds (e.g., Enterprise), you can just say:

    $ pulumi login --cloud-url https://corp.acme.my-ppc.net:9873

The cloud is now the default.  If you instead prefer a "fire and
forget" style of stack, you can skip the login and pass `--local`:

    $ pulumi stack init my-faf-stack --local

If you are logged in and run `pulumi`, we tell you as much:

    $ pulumi
    Usage:
      pulumi [command]

    // as before...

    Currently logged into the Pulumi Cloud ☁️
        https://pulumi.com/

And if you list your stacks, we tell you which one is local or not:

    $ pulumi stack ls
    NAME            LAST UPDATE       RESOURCE COUNT   CLOUD URL
    my-cloud-stack  2017-12-01 ...    3                https://pulumi.com/
    my-faf-stack    n/a               0                n/a

And `pulumi stack` by itself prints information like your cloud org,
PPC name, and so on, in addition to the usuals.

I shall write up more details and make sure to document these changes.

This change also fairly significantly refactors the layout of cloud
versus local logic, so that the cmd/ package is resonsible for CLI
things, and the new pkg/backend/ package is responsible for the
backends.  The following is the overall resulting package architecture:

* The backend.Backend interface can be implemented to substitute
  a new backend.  This has operations to get and list stacks,
  perform updates, and so on.

* The backend.Stack struct is a wrapper around a stack that has
  or is being manipulated by a Backend.  It resembles our existing
  Stack notions in the engine, but carries additional metadata
  about its source.  Notably, it offers functions that allow
  operations like updating and deleting on the Backend from which
  it came.

* There is very little else in the pkg/backend/ package.

* A new package, pkg/backend/local/, encapsulates all local state
  management for "fire and forget" scenarios.  It simply implements
  the above logic and contains anything specific to the local
  experience.

* A peer package, pkg/backend/cloud/, encapsulates all logic
  required for the cloud experience.  This includes its subpackage
  apitype/ which contains JSON schema descriptions required for
  REST calls against the cloud backend.  It also contains handy
  functions to list which clouds we have authenticated with.

* A subpackage here, pkg/backend/state/, is not a provider at all.
  Instead, it contains all of the state management functions that
  are currently shared between local and cloud backends.  This
  includes configuration logic -- including encryption -- as well
  as logic pertaining to which stacks are known to the workspace.

This addresses pulumi/pulumi#629 and pulumi/pulumi#494.
2017-12-02 14:34:42 -08:00
Joe Duffy 16ade183d8
Add a manifest to checkpoint files (#630)
This change adds a new manifest section to the checkpoint files.
The existing time moves into it, and we add to it the version of
the Pulumi CLI that created it, along with the names, types, and
versions of all plugins used to generate the file.  There is a
magic cookie that we also use during verification.

This is to help keep us sane when debugging problems "in the wild,"
and I'm sure we will add more to it over time (checksum, etc).

For example, after an up, you can now see this in `pulumi stack`:

```
Current stack is demo:
    Last updated at 2017-12-01 13:48:49.815740523 -0800 PST
    Pulumi version v0.8.3-79-g1ab99ad
    Plugin pulumi-provider-aws [resource] version v0.8.3-22-g4363e77
    Plugin pulumi-langhost-nodejs [language] version v0.8.3-79-g77bb6b6
    Checkpoint file is /Users/joeduffy/dev/code/src/github.com/pulumi/pulumi-aws/.pulumi/stacks/webserver/demo.json
```

This addresses pulumi/pulumi#628.
2017-12-01 13:50:32 -08:00
joeduffy 7e48e8726b Add (back) component outputs
This change adds back component output properties.  Doing so
requires splitting the RPC interface for creating resources in
half, with an initial RegisterResource which contains all of the
input properties, and a final CompleteResource which optionally
contains any output properties synthesized by the component.
2017-11-20 17:38:09 -08:00
joeduffy 86267b86b9 Merge root stack changes with parenting 2017-11-20 10:08:59 -08:00
joeduffy 5dc4b0b75c Switch to parent pointers; display components nicely
This change switches from child lists to parent pointers, in the
way resource ancestries are represented.  This cleans up a fair bit
of the old parenting logic, including all notion of ambient parent
scopes (and will notably address pulumi/pulumi#435).

This lets us show a more parent/child display in the output when
doing planning and updating.  For instance, here is an update of
a lambda's text, which is logically part of a cloud timer:

    * cloud:timer:Timer: (same)
          [urn=urn:pulumi:malta::lm-cloud:☁️timer:Timer::lm-cts-malta-job-CleanSnapshots]
        * cloud:function:Function: (same)
              [urn=urn:pulumi:malta::lm-cloud:☁️function:Function::lm-cts-malta-job-CleanSnapshots]
            * aws:serverless:Function: (same)
                  [urn=urn:pulumi:malta::lm-cloud::aws:serverless:Function::lm-cts-malta-job-CleanSnapshots]
                ~ aws:lambda/function:Function: (modify)
                      [id=lm-cts-malta-job-CleanSnapshots-fee4f3bf41280741]
                      [urn=urn:pulumi:malta::lm-cloud::aws:lambda/function:Function::lm-cts-malta-job-CleanSnapshots]
                    - code            : archive(assets:2092f44) {
                        // etc etc etc

Note that we still get walls of text, but this will be actually
quite nice when combined with pulumi/pulumi#454.

I've also suppressed printing properties that didn't change during
updates when --detailed was not passed, and also suppressed empty
strings and zero-length arrays (since TF uses these as defaults in
many places and it just makes creation and deletion quite verbose).

Note that this is a far cry from everything we can possibly do
here as part of pulumi/pulumi#340 (and even pulumi/pulumi#417).
But it's a good start towards taming some of our output spew.
2017-11-20 09:07:53 -08:00
Luke Hoban 96e4b74b15
Support for stack outputs (#581)
Adds support for top-level exports in the main script of a Pulumi Program to be captured as stack-level output properties.

This create a new `pulumi:pulumi:Stack` component as the root of the resource tree in all Pulumi programs.  That resources has properties for each top-level export in the Node.js script.

Running `pulumi stack` will display the current value of these outputs.
2017-11-17 15:22:41 -08:00
joeduffy 7835305b82 Fix where integration tests look for checkpoints 2017-10-27 19:42:17 -07:00
Matt Ellis ade366544e Encrypt secrets in Pulumi.yaml
We now encrypt secrets at rest based on a key derived from a user
suplied passphrase.

The system is designed in a way such that we should be able to have a
different decrypter (either using a local key or some remote service
in the Pulumi.com case in the future).

Care is taken to ensure that we do not leak decrypted secrets into the
"info" section of the checkpoint file (since we currently store the
config there).

In addtion, secrets are "pay for play", a passphrase is only needed
when dealing with a value that's encrypted. If secure config values
are not used, `pulumi` will never prompt you for a
passphrase. Otherwise, we only prompt if we know we are going to need
to decrypt the value. For example, `pulumi config <key>` only prompts
if `<key>` is encrypted and `pulumi deploy` and friends only prompt if
you are targeting a stack that has secure configuration assoicated
with it.

Secure values show up as unecrypted config values inside the language
hosts and providers.
2017-10-24 16:48:12 -07:00
joeduffy c61bce3e41 Permit undefined in more places
The prior code was a little too aggressive in rejected undefined
properties, because it assumed any occurrence indicated a resource
that was unavailable due to planning.  This is a by-produt of our
relatively recent decision to flow undefineds freely during planning.

The problem is, it's entirely legitimate to have undefined values
deep down in JavaScript structures, entirely unrelated to resources
whose property values are unknown due to planning.

This change flows undefined more freely.  There really are no
negative consequences of doing so, and avoids hitting some overly
aggressive assertion failures in some important scenarios.  Ideally
we would have a way to know statically whether something is a resource
property, and tighten up the assertions just to catch possible bugs
in the system, but because this is JavaScript, and all the assertions
are happening at runtime, we simply lack the necessary metadata to do so.
2017-10-23 16:02:28 -07:00
Joe Duffy 4a493292b1 Tolerate missing hashes 2017-10-22 15:54:44 -07:00
Joe Duffy 69f7f51375 Many asset improvements
This improves a few things about assets:

* Compute and store hashes as input properties, so that changes on
  disk are recognized and trigger updates (pulumi/pulumi#153).

* Issue explicit and prompt diagnostics when an asset is missing or
  of an unexpected kind, rather than failing late (pulumi/pulumi#156).

* Permit raw directories to be passed as archives, in addition to
  archive formats like tar, zip, etc. (pulumi/pulumi#240).

* Permit not only assets as elements of an archive's member list, but
  also other archives themselves (pulumi/pulumi#280).
2017-10-22 13:39:21 -07:00
Matt Ellis a749ac1102 Use go-yaml directly
Instead of doing the logic to see if a type has YAML tags and then
dispatching based on that to use either the direct go-yaml marshaller
or the one that works in terms of JSON tags, let's just say that we
always add YAML tags as well, and use go-yaml directly.
2017-10-20 14:01:37 -07:00
pat@pulumi.com 23864b9459 PR feedback. 2017-10-19 10:34:23 -07:00
pat@pulumi.com 6b66437fae Track resources that are pending deletion in checkpoints.
During the course of a `pulumi update`, it is possible for a resource to
become slated for deletion. In the case that this deletion is part of a
replacement, another resource with the same URN as the to-be-deleted
resource will have been created earlier. If the `update` fails after the
replacement resource is created but before the original resource has been
deleted, the snapshot must capture that the original resource still exists
and should be deleted in a future update without losing track of the order
in which the deletion must occur relative to other deletes. Currently, we
are unable to track this information because the our checkpoints require
that no two resources have the same URN.

To fix this, these changes introduce to the update engine the notion of a
resource that is pending deletion and change checkpoint serialization to
use an array of resources rather than a map. The meaning of the former is
straightforward: a resource that is pending deletion should be deleted
during the next update.

This is a fairly major breaking change to our checkpoint files, as the
map of resources is no more. Happily, though, it makes our checkpoint
files a bit more "obvious" to any tooling that might want to grovel
or rewrite them.

Fixes #432, #387.
2017-10-18 17:09:00 -07:00
Matt Ellis 22c9e0471c Use Stack over Environment to describe a deployment target
Previously we used the word "Environment" as the term for a deployment
target, but since then we've started to use the term Stack. Adopt this
across the CLI.

From a user's point of view, there are a few changes:

1. The `env` verb has been renamed to `stack`
2. The `-e` and `--env` options to commands which operate on an
environment now take `-s` or `--stack` instead.
3. Becase of (2), the commands that used `-s` to display a summary now
only support passing the full option name (`--summary`).

On the local file system, we still store checkpoint data in the `env`
sub-folder under `.pulumi` (so we can reuse existing checkpoint files
that were written to the old folder)
2017-10-16 13:04:20 -07:00