Commit graph

687 commits

Author SHA1 Message Date
evanboyle c3f6ae2451 move pkg/util/logging -> sdk/go/common/util/logging 2020-03-18 15:34:58 -07:00
evanboyle 30df499838 move pkg/version -> sdk/go/common/version 2020-03-18 15:25:25 -07:00
evanboyle 8df534a71e move pkg/diag -> sdk/go/common/diag 2020-03-18 15:09:29 -07:00
evanboyle f754b486b8 move pkg/resource/config -> sdk/go/common/resource/config 2020-03-18 15:03:37 -07:00
evanboyle 67cb405c93 move pkg/apitype -> sdk/common/apitype 2020-03-18 15:00:30 -07:00
evanboyle 70f386a967 move pkg/tokens -> sdk/go/common/tokens 2020-03-18 14:49:56 -07:00
evanboyle fccf301d14 move pkg/util/contract -> sdk/go/common/util/contract 2020-03-18 14:40:07 -07:00
evanboyle 8fb3f428b0 move pkg/workspace -> sdk/go/common/workspace 2020-03-18 14:35:53 -07:00
evanboyle dfab571aac move pkg/resource/plugin -> sdk/go/common/resource/plugin 2020-03-18 14:26:24 -07:00
evanboyle fba783caf9 move pkg/resource -> sdk/go/common/resource, but leave nested resource packages 2020-03-18 13:36:19 -07:00
Luke Hoban 74ffbfd9ba
Ensure new provider is registered when provider diff is unknown (#4051)
The changes in #4004 caused old provider configuration to be used even when a provider was different between inputs and outputs, in the case that the diff returned DiffUnkown.

To better handle that case, we compute a more accurate (but still conservative) DiffNone or DiffSome so that we can ensure we conservatively update to a new provider when needed, but retain the performance benefit of not creating and configuring a new provider as much as possible.

Part of https://github.com/pulumi/pulumi-aws/issues/814.
2020-03-10 19:40:25 -07:00
Justin Van Patten 80f6c61310
Initial support for configuring policies (#4015) 2020-03-08 14:11:55 -07:00
Luke Hoban 2067e27ee6
Avoid configuring providers twice during preview (#4004)
In the very common case where provider configuration does not change, during preview we were calling `Configure` on the cloud provider twice - once for the "old" configuration, and once for the "new" configuration.

This is not necessary, and we can just avoid using the new provider when configuration has not changed, since we will have configured the old provider very early so if we can use that we should.

Note that this technically doesn't prevent the second call to `Configure` from being made, but it prevents us from ever waiting on it.  We may want to go further and avoid even calling `Configure` on the provider in this case.

Part of #3671.
2020-03-01 14:41:46 -08:00
Chris Smith 9d1edad65c
Improve error message for policy pack not found (#3967) 2020-02-28 12:48:53 -08:00
Pat Gavlin ccde31b713
Add support for extracting schemas from providers. (#3984)
These changes add a new method to the resource provider gRPC interface,
`GetSchema`, that allows consumers of these providers to extract
JSON-serialized schema information for the provider's types, resources,
and functions.
2020-02-27 16:10:47 -08:00
Evan Boyle 930adc0504
Add support for secrets in Go SDK (#3938) 2020-02-25 17:45:36 -08:00
Erin Krengel e660937bab
use version tag (#3961) 2020-02-24 17:11:56 -08:00
Justin Van Patten e48a585b1a
Fix panic on failed stack policy (#3960)
Avoid panic during stack validations. This regressed recently when the PAC error output was modified.
2020-02-24 07:04:35 -08:00
Evan Boyle 67838d8711
Marshal non-string provider config (#3948) 2020-02-21 12:55:27 -08:00
Justin Van Patten 3bf9067bac
Expose options, parent, deps, and provider config to policies (#3862) 2020-02-07 16:11:34 -08:00
Pat Gavlin c4cbff072b
Handle failures from plugins' CheckConfig. (#3813) 2020-01-26 13:19:13 -08:00
Justin Van Patten 10a960ea4b
PaC: Support Config/getProject/getStack/isDryRun (#3612)
Add support for using `Config`, `getProject()`, `getStack()`, and
`isDryRun()` from Policy Packs.
2019-12-16 22:51:02 +00:00
Justin Van Patten 5cb0731eba
Show errors when analyzing the stack (#3624)
Actually show the error that occurred while analyzing the stack. This is consistent with what we show when analyzing individual resources.
2019-12-12 22:26:27 +00:00
Evan Boyle 1ca50d4b89
Propagate parent and providers for go SDK calls (#3563) 2019-11-26 13:23:34 -08:00
Pat Gavlin 0e9c4ed2db
Serialize null property values. (#3561)
Eliding these values prevents us from properly round-tripping resource
states that include null property values.

This is part of the fix for
https://github.com/pulumi/pulumi-azure/issues/383.
2019-11-22 11:03:02 -08:00
Justin Van Patten 3c1bdff5fd
Allow relative paths to --policy-pack (#3565)
A regression was introduced when we added support for non-Node.js Pulumi programs to run Policy Packs. With that change, we now pass the Policy Pack's full path as the plugin's pwd (so that it would load the `@pulumi/pulumi/cmd/run-policy-pack` Node module from the Policy Pack's node_modules rather than the program's node_modules), but we also pass the path to the policy pack as well. If the path is a full rooted path, this would work fine, and that's what our tests do. However, if a relative path is specified, then it will be looking to load the Policy Pack relative to the pwd, which doesn't produce a correct path leading to failures trying to load the Policy Pack.

Since the pwd is the policy pack path, we can simply pass the path as `"."` to the analyzer plugin, and it will load the policy pack in its pwd.
2019-11-22 17:24:35 +00:00
Chris Smith 2ff0e6dee3
Persist computed values (#3558) 2019-11-21 14:58:30 -08:00
Justin Van Patten 6c84b008d8
Send resource URN and name to analyzer (#3554)
More information we want to make available to policy packs.
2019-11-21 21:01:15 +00:00
Chris Smith 4e59263a9c
Add tests for serializing PropertyMaps (#3533)
* WIP - Add tests for serializing PropertyMaps

* Apply suggestions from code review

Co-Authored-By: Pat Gavlin <pat@pulumi.com>

* Cleanup tests
2019-11-19 21:10:51 -08:00
CyrusNajmabadi d4aa5fe20d Switch to 'console.log' for our hang warning. Add warning to synchronous StackReference calls. (#3456)
Codepaths which could result in a hang will print a message to the console indicating the problem, along with a link to documentation on how to restructure code to best address it.

`StackReference.getOutputSync` and `requireOutputSync` have been deprecated as they may cause hangs on some combinations of Node and certain OS platforms. `StackReference.getOutput` and `requireOutput` should be used instead.
2019-11-19 12:51:14 -08:00
Justin Van Patten 83c5e766a4
Provide a more helpful PaC error (#3525)
To allow Policy Packs to run against Pulumi programs written in all languages, we now look for the `@pulumi/pulumi/cmd/run-policy-pack` module in the Policy Pack's node_modules (instead of in the Pulumi program's node_modules; which doesn't exist for non-node languages). The `@pulumi/policy` library that a Policy Pack will depend on should already depend on a recent enough version of `@pulumi/pulumi`. When we can't find the module, it's more likely it's due to the dependencies for the Policy Pack not being installed. Provide a more helpful error message in this case.
2019-11-19 18:47:47 +00:00
CyrusNajmabadi 1908a18d20 Loosen resource targeting restrictions. (#3426)
- If an untargeted create would not affect the inputs of any targeted
  resources, do not fail the update. Untargeted creates that are
  directly dependend on by targeted resources will still cause failures
  that inform the user to add the untargeted resources to the --target
  list.
- Users may now pass the `--target-dependents` flag to allow targeted
  destroys to automatically target dependents that must be destroyed in
  order to destroy an explicitly targeted resource.
2019-11-18 20:28:25 -08:00
Justin Van Patten 022826bac5
Enable policies to run for all languages (#3524) 2019-11-18 18:29:12 +00:00
Alex Clemmer b15937aabd Don't send spurious empty stream invoke response 2019-11-12 13:51:19 -08:00
Pat Gavlin 137fd54f1c
Propagate inputs to outputs during preview. (#3327)
These changes restore a more-correct version of the behavior that was
disabled with #3014. The original implementation of this behavior was
done in the SDKs, which do not have access to the complete inputs for a
resource (in particular, default values filled in by the provider during
`Check` are not exposed to the SDK). This lack of information meant that
the resolved output values could disagree with the typings present in
a provider SDK. Exacerbating this problem was the fact that unknown
values were dropped entirely, causing `undefined` values to appear in
unexpected places.

By doing this in the engine and allowing unknown values to be
represented in a first-class manner in the SDK, we can attack both of
these issues.

Although this behavior is not _strictly_ consistent with respect to the
resource model--in an update, a resource's output properties will come
from its provider and may differ from its input properties--this
behavior was present in the product for a fairly long time without
significant issues. In the future, we may be able to improve the
accuracy of resource outputs during a preview by allowing the provider
to dry-run CRUD operations and return partially-known values where
possible.

These changes also introduce new APIs in the Node and Python SDKs
that work with unknown values in a first-class fashion:
- A new parameter to the `apply` function that indicates that the
  callback should be run even if the result of the apply contains
  unknown values
- `containsUnknowns` and `isUnknown`, which return true if a value
  either contains nested unknown values or is exactly an unknown value
- The `Unknown` type, which represents unknown values

The primary use case for these APIs is to allow nested, properties with
known values to be accessed via the lifted property accessor even when
the containing property is not fully know. A common example of this
pattern is the `metadata.name` property of a Kubernetes `Namespace`
object: while other properties of the `metadata` bag may be unknown,
`name` is often known. These APIs allow `ns.metadata.name` to return a
known value in this case.

In order to avoid exposing downlevel SDKs to unknown values--a change
which could break user code by exposing it to unexpected values--a
language SDK must indicate whether or not it supports first-class
unknown values as part of each `RegisterResourceRequest`.

These changes also allow us to avoid breaking user code with the new
behavior introduced by the prior commit.

Fixes #3190.
2019-11-11 12:09:34 -08:00
Evan Boyle 9506b69c8b
error instead of panic when different resources use the same alias (#3457) 2019-11-06 08:49:13 -08:00
Alex Clemmer c0490ec164 Clean up language and resource providers in query 2019-11-05 10:47:48 -08:00
Alex Clemmer f195cc0d4d Implement StreamInvoke 2019-11-05 10:47:48 -08:00
Justin Van Patten c08714ffb4
Support lists and maps in config (#3342)
This change adds support for lists and maps in config. We now allow
lists/maps (and nested structures) in `Pulumi.<stack>.yaml` (or
`Pulumi.<stack>.json`; yes, we currently support that).

For example:

```yaml
config:
  proj:blah:
  - a
  - b
  - c
  proj:hello: world
  proj:outer:
    inner: value
  proj:servers:
  - port: 80
```

While such structures could be specified in the `.yaml` file manually,
we support setting values in maps/lists from the command line.

As always, you can specify single values with:

```shell
$ pulumi config set hello world
```

Which results in the following YAML:

```yaml
proj:hello world
```

And single value secrets via:

```shell
$ pulumi config set --secret token shhh
```

Which results in the following YAML:

```yaml
proj:token:
  secure: v1:VZAhuroR69FkEPTk:isKafsoZVMWA9pQayGzbWNynww==
```

Values in a list can be set from the command line using the new
`--path` flag, which indicates the config key contains a path to a
property in a map or list:

```shell
$ pulumi config set --path names[0] a
$ pulumi config set --path names[1] b
$ pulumi config set --path names[2] c
```

Which results in:

```yaml
proj:names
- a
- b
- c
```

Values can be obtained similarly:

```shell
$ pulumi config get --path names[1]
b
```

Or setting values in a map:

```shell
$ pulumi config set --path outer.inner value
```

Which results in:

```yaml
proj:outer:
  inner: value
```

Of course, setting values in nested structures is supported:

```shell
$ pulumi config set --path servers[0].port 80
```

Which results in:

```yaml
proj:servers:
- port: 80
```

If you want to include a period in the name of a property, it can be
specified as:

```
$ pulumi config set --path 'nested["foo.bar"]' baz
```

Which results in:

```yaml
proj:nested:
  foo.bar: baz
```

Examples of valid paths:

- root
- root.nested
- 'root["nested"]'
- root.double.nest
- 'root["double"].nest'
- 'root["double"]["nest"]'
- root.array[0]
- root.array[100]
- root.array[0].nested
- root.array[0][1].nested
- root.nested.array[0].double[1]
- 'root["key with \"escaped\" quotes"]'
- 'root["key with a ."]'
- '["root key with \"escaped\" quotes"].nested'
- '["root key with a ."][100]'

Note: paths that contain quotes can be surrounded by single quotes.

When setting values with `--path`, if the value is `"false"` or
`"true"`, it will be saved as the boolean value, and if it is
convertible to an integer, it will be saved as an integer.

Secure values are supported in lists/maps as well:

```shell
$ pulumi config set --path --secret tokens[0] shh
```

Will result in:

```yaml
proj:tokens:
- secure: v1:wpZRCe36sFg1RxwG:WzPeQrCn4n+m4Ks8ps15MxvFXg==
```

Note: maps of length 1 with a key of “secure” and string value are
reserved for storing secret values. Attempting to create such a value
manually will result in an error:

```shell
$ pulumi config set --path parent.secure foo
error: "secure" key in maps of length 1 are reserved
```

**Accessing config values from the command line with JSON**

```shell
$ pulumi config --json
```

Will output:

```json
{
  "proj:hello": {
    "value": "world",
    "secret": false,
    "object": false
  },
  "proj:names": {
    "value": "[\"a\",\"b\",\"c\"]",
    "secret": false,
    "object": true,
    "objectValue": [
      "a",
      "b",
      "c"
    ]
  },
  "proj:nested": {
    "value": "{\"foo.bar\":\"baz\"}",
    "secret": false,
    "object": true,
    "objectValue": {
      "foo.bar": "baz"
    }
  },
  "proj:outer": {
    "value": "{\"inner\":\"value\"}",
    "secret": false,
    "object": true,
    "objectValue": {
      "inner": "value"
    }
  },
  "proj:servers": {
    "value": "[{\"port\":80}]",
    "secret": false,
    "object": true,
    "objectValue": [
      {
        "port": 80
      }
    ]
  },
  "proj:token": {
    "secret": true,
    "object": false
  },
  "proj:tokens": {
    "secret": true,
    "object": true
  }
}
```

If the value is a map or list, `"object"` will be `true`. `"value"` will
contain the object as serialized JSON and a new `"objectValue"` property
will be available containing the value of the object.

If the object contains any secret values, `"secret"` will be `true`, and
just like with scalar values, the value will not be outputted unless
`--show-secrets` is specified.

**Accessing config values from Pulumi programs**

Map/list values are available to Pulumi programs as serialized JSON, so
the existing
`getObject`/`requireObject`/`getSecretObject`/`requireSecretObject`
functions can be used to retrieve such values, e.g.:

```typescript
import * as pulumi from "@pulumi/pulumi";

interface Server {
    port: number;
}

const config = new pulumi.Config();

const names = config.requireObject<string[]>("names");
for (const n of names) {
    console.log(n);
}

const servers = config.requireObject<Server[]>("servers");
for (const s of servers) {
    console.log(s.port);
}
```
2019-11-01 13:41:27 -07:00
Pat Gavlin 23a84df254
Add targeted replaces to update. (#3418)
Allow the user to specify a set of resources to replace via the
`--replace` flag on the CLI. This can be combined with `--target` to
replace a specific set of resources without changing any other
resources. `--target-replace` is shorthand for `--replace urn --target urn`.

Fixes #2643.
2019-10-30 17:16:55 -07:00
CyrusNajmabadi fcc57f2e0f
Fix crash when specifying update-target that doesn't exist. (#3408) 2019-10-28 11:41:43 -07:00
Chris Smith d2805fcb3f
Add support for aggregate resource analysis (#3366)
* Add AnalyzeStack method to Analyze service

* Protobuf generated code

* Hook up AnalyzeStack method

* Address PR feedback

* Address PR feedback
2019-10-25 08:29:02 -07:00
Alex Clemmer ae6cd64dc8 Factor queryResmon to share more code with resmon 2019-10-23 15:14:56 -07:00
Alex Clemmer 34093b1361 Allow provider loading in query mode
This commit will introduce the ability to load providers in `query`
mode.

Previously, `query` mode has been effectively a stand-alone execution
environment for language hosts, running without (e.g.) the
`StepExecutor` and similar engine facilities, but with some minimal
constructs hooked up, notably the ability to retrieve stack snapshots
from the backend for querying.

This commit extends this functionality somewhat by allowing `query` to
load Pulumi resource providers, and to run `Invoke` on them. This will
allow us, in the future, to "query" resource providers in the same way
we can query stack snapshots.
2019-10-23 15:14:56 -07:00
Alex Clemmer 91f50eeb51 Report pulumi query as not supporting secrets 2019-10-23 15:14:56 -07:00
Alex Clemmer 9c16485152 Don't require a stack to run pulumi query 2019-10-23 15:14:56 -07:00
Pat Gavlin f14eba46b5
Add a URN validation method. (#3386)
This method can be used to check whether or not a URN is well-formed.
This is used by the provider reference parser to avoid panicking on
malformed URNs.
2019-10-21 19:09:39 -07:00
CyrusNajmabadi d72872c49a
Fix error message (#3344) 2019-10-15 13:29:32 -07:00
Pat Gavlin d18b59e9c6
Fix a dependency graph bug during DBR. (#3329)
The dependency graph used to determine the set of resources that
depend on a resource being DBR'd is constructured from the list of
resource states present in the old snapshot. However, the dependencies
of resources that are present in both the old snapshot and the current
plan can be different, which in turn can cause the engine to make
incorrect decisions during DBR with respect to which resources need to
be replaced. For example, consider the following program:

```
var resA = new Resource("a", {dbr: "foo"});
var resB = new Resource("b", {dbr: resA.prop});
```

If this program is then changed to:
```
var resB = new Resource("b", {dbr: "<literal value of resA.prop>"});
var resA = new Resource("a", {dbr: "bar"});
```

The engine will first decide to make no changes to "b", as its input
property values have not changed. "b" has changed, however, such that it
no longer has a dependency on "a".

The engine will then decide to DBR "a". In the process, it will
determine that it first needs to delete "b", because the state for "b"
that is used when calculating "a"'s dependents does not reflect the
changes made during the plan.

To fix this issue, we rely on the observation that dependents can only
have been _removed_ from the base dependency graph: for a dependent to
have been added, it would have had to have been registered prior to the
root--a resource it depends on--which is not a valid operation. This
means that any resources that depend on the root must not yet have
been registered, which in turn implies that resources that have already
been registered must not depend on the root. Thus, we ignore these
resources if they are encountered while walking the old dependency graph
to determine the set of dependents.
2019-10-12 17:22:13 -07:00
Pat Gavlin 834e583c95
Revert "Propagate inputs to outputs during preview. (#3245)" (#3324)
This reverts commit 80504bf0bc.
2019-10-10 10:33:05 -07:00
Chris Smith bec608d0d3
Improve error message when using PAC on out-of-date SDK (#3307)
* Fix lint warnings

* Improve error message when using PAC on old SDK

* Update pkg/resource/plugin/analyzer_plugin.go

Co-Authored-By: Justin Van Patten <jvp@justinvp.com>
2019-10-09 13:51:10 -07:00
Chris Smith 0e52e965db
Add local policy packs to summary event (#3308) 2019-10-09 13:50:28 -07:00
Pat Gavlin 5ad307d457
Close test language runtime connections. (#3289)
This should help address the issues we've begun seeing with too many
open files when running tests on OS X.
2019-10-01 12:12:06 -07:00
CyrusNajmabadi 52884096e9
Add support for updating a subset of resources in the stack (i.e. --target) (#3251) 2019-09-30 23:41:56 -07:00
Pat Gavlin 80504bf0bc
Propagate inputs to outputs during preview. (#3245)
These changes restore a more-correct version of the behavior that was
disabled with #3014. The original implementation of this behavior was
done in the SDKs, which do not have access to the complete inputs for a
resource (in particular, default values filled in by the provider during
`Check` are not exposed to the SDK). This lack of information meant that
the resolved output values could disagree with the typings present in
a provider SDK. Exacerbating this problem was the fact that unknown
values were dropped entirely, causing `undefined` values to appear in
unexpected places.

By doing this in the engine and allowing unknown values to be
represented in a first-class manner in the SDK, we can attack both of
these issues.

Although this behavior is not _strictly_ consistent with respect to the
resource model--in an update, a resource's output properties will come
from its provider and may differ from its input properties--this
behavior was present in the product for a fairly long time without
significant issues. In the future, we may be able to improve the
accuracy of resource outputs during a preview by allowing the provider
to dry-run CRUD operations and return partially-known values where
possible.

These changes also introduce new APIs in the Node and Python SDKs
that work with unknown values in a first-class fashion:
- A new parameter to the `apply` function that indicates that the
  callback should be run even if the result of the apply contains
  unknown values
- `containsUnknowns` and `isUnknown`, which return true if a value
  either contains nested unknown values or is exactly an unknown value
- The `Unknown` type, which represents unknown values

The primary use case for these APIs is to allow nested, properties with
known values to be accessed via the lifted property accessor even when
the containing property is not fully know. A common example of this
pattern is the `metadata.name` property of a Kubernetes `Namespace`
object: while other properties of the `metadata` bag may be unknown,
`name` is often known. These APIs allow `ns.metadata.name` to return a
known value in this case.

In order to avoid exposing downlevel SDKs to unknown values--a change
which could break user code by exposing it to unexpected values--a
language SDK must indicate whether or not it supports first-class
unknown values as part of each `RegisterResourceRequest`.

These changes also allow us to avoid breaking user code with the new
behavior introduced by the prior commit.

Fixes #3190.
2019-09-30 11:03:58 -07:00
Luke Hoban 9374c374c3
Transformations (#3174)
Adds the ability to provide `transformations` to modify the properties and resource options that will be used for any child resource of a component or stack.

This offers an "escape hatch" to modify the behaviour of a component by peeking behind it's abstraction.  For example, it can be used to add a resource option (`additionalSecretOutputs`, `aliases`, `protect`, etc.) to a specific known child of a component, or to modify some input property to a child resource if the component does not (yet) expose the ability to control that input directly.  It could also be used for more interesting scenarios - such as:
1. Automatically applying tags to all resources that support them in a stack (or component)
2. Injecting real dependencies between stringly-referenced  resources in a Helm Chart 
3. Injecting explicit names using a preferred naming convention across all resources in a stack
4. Injecting `import` onto all resources by doing a lookup into a name=>id mapping

Because this feature makes it possible to peek behind a component abstraction, it must be used with care in cases where the component is versioned independently of the use of transformations.  Also, this can result in "spooky action at a distance", so should be used judiciously.  That said - this can be used as an escape hatch to unblock a wide variety of common use cases without waiting on changes to be made in a component implementation.  

Each transformation is passed the `resource`, `name`, `type`, `props` and `opts` that are passed into the `Resource` constructor for any resource descended from the resource that has the transformation applied.  The transformation callback can optionally return alternate versions of the `props` and `opts` to be used in place of the original values provided to the resource constructor.

Fixes #2068.
2019-09-29 11:27:37 -07:00
Alex Clemmer ec85408eee Don't throw away secrets in calls to Analyze 2019-09-27 15:26:13 -07:00
Chris Smith 2801bfe0c2
Update apitype.StackRenameRequest // Support project renames (#3273)
* Update apitype.StackRenameRequest

* Add support for renaming projects in Snapshots

* Address PR feedback
2019-09-26 15:23:09 -07:00
CyrusNajmabadi ef7b7d0bd1
Share target checking code. (#3252) 2019-09-20 17:50:44 -07:00
CyrusNajmabadi c1ff9c37f8
Delete specific target (#3244) 2019-09-19 19:28:14 -07:00
Pat Gavlin 40b0f8cbab
Cache ciphertext for secret properties. (#3183)
This caching is enabled by wrapping the `secrets.Manager` returned by
`DefaultSecretsProvider.OfType` in an outer `secrets.Manager` that
cooperates with `stack.{Serialize,Deserialize}PropertyValue`. Ciphertext
is cached on a per-secret-instance basis (i.e. not a per-plaintext-value
basis). Cached ciphertext is only reused if the plaintext for the secret
value has not changed. Entries are inserted into the cache upon both
encryption and decryption so that values that originated from ciphertext
and that have not changed can aoid re-encryption.

Contributes to #3178.
2019-09-18 15:52:31 -07:00
CyrusNajmabadi c53c0b6c15
Produce appropriate diagnostic when asked to refresh a resource that we cannot find. (#3246) 2019-09-18 14:28:42 -07:00
CyrusNajmabadi 1387afec8f
Color 'reads' as cyan so they don't look like 'creates'. (#3236) 2019-09-18 09:49:13 -07:00
CyrusNajmabadi f788eb8fc1
Add support for refreshing specific targets. (#3225) 2019-09-17 18:14:10 -07:00
Pat Gavlin 82204230e1
Improve tracing support. (#3238)
* Fix some tracing issues.

- Add endpoints for `startUpdate` and `postEngineEventsBatch` so that
  spans for these invocations have proper names
- Inject a tracing span when walking a plan so that resource operations
  are properly parented
- When handling gRPC calls, inject a tracing span into the call's
  metadata if no span is already present so that resource monitor and
  engine spans are properly parented
- Do not trace client gRPC invocations of the empty method so that these
  calls (which are used to determine server availability) do not muddy
  the trace. Note that I tried parenting these spans appropriately, but
  doing so broke the trace entirely.

With these changes, the only unparented span in a typical Pulumi
invocation is a single call to `getUser`. This span is unparented
because that call does not have a context available. Plumbing a context
into that particular call is surprisingly tricky, as it is often called
by other context-less functions.

* Make tracing support more flexible.

- Add support for writing trace data to a local file using Appdash
- Add support for viewing Appdash traces via the CLI
2019-09-16 14:16:43 -07:00
Paul Stack 8d1b725840
GH-2319: Increase the grpc.MaxCallRecvMsgSize (#3201)
Fixes: #2319

In #2319, a user is hitting the gRPC limit on the message size the
server can receive when uploading ec2 user-data

This commit doubles the limit that can be sent from `1024*1024*4` to
`1024*1024*8`
2019-09-09 21:31:54 +03:00
Alex Clemmer 99d70e4610 Introduce MarshalOptions.{RejectAsset, RejectArchive}
Not all resource providers support Pulumi's Asset and Archive types. In
particular, the Kubernetes provider should reject any resource
definition that contains either of these types.

This commit will introduce two MarshalOptions that will make it easy for
the Kubernetes provider to guarantee that no properties of this type are
in a resource request, as it's deserializing the request from the
engine.
2019-08-26 15:19:14 -07:00
Pat Gavlin 2455564ddc
Allow IDs to change during import. (#3133)
This is necessary for resources like `aws.ec2.RouteTableAssociation`.

Part of https://github.com/pulumi/pulumi-aws/issues/708.
2019-08-23 15:00:24 -07:00
Pat Gavlin 42fc75fffe
Fail read steps with missing resources. (#3123)
Just what it says on the tin.

Fixes #262.
2019-08-21 10:09:02 -07:00
Pat Gavlin 8745440c1b
Allow users to explicitly disable delete-before-replace. (#3118)
With these changes, a user may explicitly set `deleteBeforeReplace` to
`false` in order to disable DBR behavior for a particular resource. This
is the SDK + CLI escape hatch for cases where the changes in
https://github.com/pulumi/pulumi-terraform/pull/465 cause undesirable
behavior.
2019-08-20 15:51:02 -07:00
Paul Stack 2c0510a23e
Update the JSON representation of customTimeouts in state (#3101)
Omit them when empty and change the casing of Create, Update and
Delete
2019-08-21 01:01:27 +03:00
Matt Ellis 342f8311a1 Fix renaming a freshly created stack using the local backend
Attempting to `pulumi stack rename` a stack which had been created but
never updated, when using the local backend, was broken because
code-paths were not hardened against the snapshot being `nil` (which
is the case for a stack before the initial deployment had been done).

Fixes #2654
2019-08-16 13:39:34 -07:00
Paul Stack f8db8e4209
Allow resource IDs to change on reresh steps (#3087)
* Allow resource IDs to change on reresh steps

This is a requirement for us to be able to move forward with
versions of the Terraform Azurerm provider. In v1.32.1, there was
a state migration that changed the ID format of the azure table
storage resource

We used to have a check in place for old ID being equal to new ID.
This has been changed now and we allow the change of ID to happen
in the RefreshStep

* Update pkg/resource/deploy/step.go

Co-Authored-By: Pat Gavlin <pat@pulumi.com>
2019-08-16 21:04:03 +03:00
Matt Ellis 0bb4e6d70b Respond to PR feedback
Address post commit feedback from Cyrus on
pulumi/pulumi#3071
2019-08-15 12:42:51 -07:00
Matt Ellis 9308246114 Do not taint all stack outputs as secrets if just one is
When using StackReference, if the stack you reference contains any
secret outputs, we have to mark the entire `outputs` member as a
secret output. This is because we only track secretness on a per
`Output<T>` basis.

For `getSecret` and friends, however, we know the name of the output
you are looking up and we can be smarter about if the returned
`Output<T>` should be treated as a secret or not.

This change augments the provider for StackReference such that it also
returns a list of top level stack output names who's values contain
secrets. In the language SDKs, we use this information, when present,
to decide if we should return an `Output<T>` that is marked as a
secret or not. Since the SDK and CLI are independent components, care
is taken to ensure that when the CLI does not return this information,
we behave as we did before (i.e. if any output is a secret, we treat
every output as a secret).

Fixes #2744
2019-08-13 16:11:38 -07:00
Alex Clemmer 6b4832b416 Emit stderr of npm install
If we don't process and report the stderr of `npm install`, the output
is "orphaned" during error condition, and only something like "exit code
1" is reported.
2019-08-13 12:48:16 -07:00
Alex Clemmer ef8cc236c4 Implement --policy-pack flag on up and preview
Fixes pulumi/pulumi-policy#43.
2019-08-12 12:45:48 -07:00
Pat Gavlin 6983e2639b
Fix conversion of empty array properties. (#3047)
Empty `[]interface{}` values were being converted to array property
values with a `nil` element, and empty array property values were being
coverted to `nil` `[]interface{}` values. These changes fix the
converters to return empty but non-nil values in both cases.

This is part of the fix for
https://github.com/pulumi/pulumi-kubernetes/issues/693.
2019-08-07 11:42:40 -07:00
Pat Gavlin 62189e6053
Harden asset and archive deserialization. (#3042)
- Ensure that type assertions are guarded, and that incorrectly-typed
  properties return errors rather than panicking
- Expand the asset/archive tests in the Node SDK to ensure that eventual
  archives and assets serialize and deserialize correctly

Fixes #2836.
Fixes #3016.
2019-08-06 16:32:05 -07:00
Alex Clemmer 6360cba588 Improve "project not found" error messages
Fixes pulumi/pulumi-policy#36.
Fixes pulumi/pulumi-policy#37.
2019-08-05 14:14:20 -07:00
Luke Hoban 6ed4bac5af
Support additional cloud secrets providers (#2994)
Adds support for additional cloud secrets providers (AWS KMS, Azure KeyVault, Google Cloud KMS, and HashiCorp Vault) as the encryption backend for Pulumi secrets. This augments the previous choice between using the app.pulumi.com-managed secrets encryption or a fully-client-side local passphrase encryption.

This is implemented using the Go Cloud Development Kit support for pluggable secrets providers.

Like our cloud storage backend support which also uses Go Cloud Development Kit, this PR also bleeds through to users the URI scheme's that the Go CDK defines for specifying each of secrets providers - like `awskms://alias/LukeTesting?region=us-west-2` or `azurekeyvault://mykeyvaultname.vault.azure.net/keys/mykeyname`.

Also like our cloud storage backend support, this PR doesn't solve for how to configure the cloud provider client used to resolve the URIs above - the standard ambient credentials are used in both cases. Eventually, we will likely need to provide ways for both of these features to be configured independently of each other and of the providers used for resource provisioning.
2019-08-02 16:12:16 -07:00
Chris Smith 17ee050abe
Refactor the way secrets managers are provided (#3001) 2019-08-01 10:33:52 -07:00
Pat Gavlin 67ec74bdc5
Pass ignoreChanges to providers. (#3005)
These changes add support for passing `ignoreChanges` paths to resource
providers. This is intended to accommodate providers that perform diffs
between resource inputs and resource state (e.g. all Terraform-based
providers, the k8s provider when using API server dry-runs). These paths
are specified using the same syntax as the paths used in detailed diffs.

In addition to passing these paths to providers, the existing support
for `ignoreChanges` in inputs has been extended to accept paths rather
than top-level keys. It is an error to specify a path that is missing
one or more component in the old or new inputs.

Fixes #2936, #2663.
2019-07-31 11:39:07 -05:00
Pat Gavlin c6916051f0
Use a bag for misc. resource options in deploytest (#2977)
Most of these options are typically left unset. In order to make it
easier to update the lifecycle test when adding new options, collect
them in a bag s.t. most callsites can go without being updated.
2019-07-25 11:18:40 -07:00
Pat Gavlin fa05e5cb05
Migrate old providers without outputs. (#2973)
If we encounter a provider with old inputs but no old outputs when reading
a checkpoint file, use the old inputs as the old outputs. This handles the
scenario where the CLI is being upgraded from a version that did not
reflect provider inputs to provider outputs, and a provider is being
upgraded from a version that did not implement `DiffConfig` to a version
that does.

Fixes https://github.com/pulumi/pulumi-kubernetes/issues/645.
2019-07-23 13:39:21 -07:00
Alex Clemmer ed5b8437d1 Batch policy violation reporting for pulumi preview
Currently, `pulumi preview` fails immediately when any resource
definition in a Pulumi app is found to be in violation of a resource
policy. But, users would like `preview` to report as many policy
violations as it can before terminating with an error, so that they can
fix many of them before running `preview` again.

This commit will thus change `pulumi preview` to do this sort of
"batching" for policy violations. The engine will attempt to run the
entire preview step, validating every resource definition with the
relevant known resource policies, before finally reporting an error if
any violations are detected.

Fixes pulumi/pulumi-policy#31
2019-07-22 20:42:17 -07:00
Alex Clemmer 716a69ced2 Keep unknowns when marshalling resources in call to Analyze
Fixes pulumi/pulumi-aws#661.
2019-07-19 12:23:36 -07:00
Alex Clemmer 0850c88a97 Address comments 2019-07-16 00:58:33 -07:00
Alex Clemmer 4c069d5cf6 Address lint warnings 2019-07-16 00:58:33 -07:00
Alex Clemmer 9f809b9122 Run required policies as part of all updates 2019-07-16 00:58:33 -07:00
Alex Clemmer 3fc03167c5 Implement cmd/run-policy-pack
This command will cause `pulumi policy publish` to behave in much the
same way `pulumi up` does -- if the policy program is in TypeScript, we
will use ts-node to attempt to compile in-process before executing, and
fall back to plain-old node.

We accomplish this by moving `cmd/run/run.ts` into a generic helper
package, `runtime/run.ts`, which slightly generalizes the use cases
supported (notably, allowing us to exec some program outside of the
context of a Pulumi stack).

This new package is then called by both `cmd/run/index.ts` and
`cmd/run-policy-pack/index.ts`.
2019-07-16 00:58:33 -07:00
Alex Clemmer fc80eaaa3d Implement GetAnalyzerInfo in analyzer plugin 2019-07-16 00:58:33 -07:00
Paul Stack 02ffff8840
Addition of Custom Timeouts (#2885)
* Plumbing the custom timeouts from the engine to the providers

* Plumbing the CustomTimeouts through to the engine and adding test to show this

* Change the provider proto to include individual timeouts

* Plumbing the CustomTimeouts from the engine through to the Provider RPC interface

* Change how the CustomTimeouts are sent across RPC

These errors were spotted in testing. We can now see that the timeout
information is arriving in the RegisterResourceRequest

```
req=&pulumirpc.RegisterResourceRequest{
           Type:                    "aws:s3/bucket:Bucket",
           Name:                    "my-bucket",
           Parent:                  "urn:pulumi:dev::aws-vpc::pulumi:pulumi:Stack::aws-vpc-dev",
           Custom:                  true,
           Object:                  &structpb.Struct{},
           Protect:                 false,
           Dependencies:            nil,
           Provider:                "",
           PropertyDependencies:    {},
           DeleteBeforeReplace:     false,
           Version:                 "",
           IgnoreChanges:           nil,
           AcceptSecrets:           true,
           AdditionalSecretOutputs: nil,
           Aliases:                 nil,
           CustomTimeouts:          &pulumirpc.RegisterResourceRequest_CustomTimeouts{
               Create:               300,
               Update:               400,
               Delete:               500,
               XXX_NoUnkeyedLiteral: struct {}{},
               XXX_unrecognized:     nil,
               XXX_sizecache:        0,
           },
           XXX_NoUnkeyedLiteral: struct {}{},
           XXX_unrecognized:     nil,
           XXX_sizecache:        0,
       }
```

* Changing the design to use strings

* CHANGELOG entry to include the CustomTimeouts work

* Changing custom timeouts to be passed around the engine as converted value

We don't want to pass around strings - the user can provide it but we want
to make the engine aware of the timeout in seconds as a float64
2019-07-16 00:26:28 +03:00
Pat Gavlin e1a52693dc
Add support for importing existing resources. (#2893)
A resource can be imported by setting the `import` property in the
resource options bag when instantiating a resource. In order to
successfully import a resource, its desired configuration (i.e. its
inputs) must not differ from its actual configuration (i.e. its state)
as calculated by the resource's provider.

There are a few interesting state transitions hiding here when importing
a resource:
1. No prior resource exists in the checkpoint file. In this case, the
   resource is simply imported.
2. An external resource exists in the checkpoint file. In this case, the
   resource is imported and the old external state is discarded.
3. A non-external resource exists in the checkpoint file and its ID is
   different from the ID to import. In this case, the new resource is
   imported and the old resource is deleted.
4. A non-external resource exists in the checkpoint file, but the ID is
   the same as the ID to import. In this case, the import ID is ignored
   and the resource is treated as it would be in all cases except for
   changes that would replace the resource. In that case, the step
   generator issues an error that indicates that the import ID should be
   removed: were we to move forward with the replace, the new state of
   the stack would fall under case (3), which is almost certainly not
   what the user intends.

Fixes #1662.
2019-07-12 11:12:01 -07:00
Mikhail Shilkov e30e6208a0 Normalize Windows paths for directory archive (#2887)
* Normalize Windows paths for directory archive

* Changelog

* Remove the redundant check
2019-07-02 00:04:24 +03:00
Pat Gavlin 6e5c4a38d8
Defer all diffs to resource providers. (#2849)
Thse changes make a subtle but critical adjustment to the process the
Pulumi engine uses to determine whether or not a difference exists
between a resource's actual and desired states, and adjusts the way this
difference is calculated and displayed accordingly.

Today, the Pulumi engine get the first chance to decide whether or not
there is a difference between a resource's actual and desired states. It
does this by comparing the current set of inputs for a resource (i.e.
the inputs from the running Pulumi program) with the last set of inputs
used to update the resource. If there is no difference between the old
and new inputs, the engine decides that no change is necessary without
consulting the resource's provider. Only if there are changes does the
engine consult the resource's provider for more information about the
difference. This can be problematic for a number of reasons:

- Not all providers do input-input comparison; some do input-state
  comparison
- Not all providers are able to update the last deployed set of inputs
  when performing a refresh
- Some providers--either intentionally or due to bugs--may see changes
  in resources whose inputs have not changed

All of these situations are confusing at the very least, and the first
is problematic with respect to correctness. Furthermore, the display
code only renders diffs it observes rather than rendering the diffs
observed by the provider, which can obscure the actual changes detected
at runtime.

These changes address both of these issues:
- Rather than comparing the current inputs against the last inputs
  before calling a resource provider's Diff function, the engine calls
  the Diff function in all cases.
- Providers may now return a list of properties that differ between the
  requested and actual state and the way in which they differ. This
  information will then be used by the CLI to render the diff
  appropriately. A provider may also indicate that a particular diff is
  between old and new inputs rather than old state and new inputs.

Fixes #2453.
2019-07-01 12:34:19 -07:00
CyrusNajmabadi 7b8421f0b2
Fix crash when there were multiple duplicate aliases to the same resource. (#2865) 2019-06-23 02:16:18 -07:00
Matt Ellis eb3a7d0a7a Fix up some spelling errors
@keen99 pointed out that newer versions of golangci-lint were failing
due to some spelling errors. This change fixes them up.  We have also
now have a work item to track moving to a newer golangci-lint tool in
the future.

Fixes #2841
2019-06-18 15:30:25 -07:00
Alex Clemmer 0fc4bc7885 Remove policy ID from policy API 2019-06-13 17:39:30 -07:00
Alex Clemmer 8b7d329c69 Use Analyzer PB in analyzer code 2019-06-13 16:04:13 -07:00