Commit graph

86 commits

Author SHA1 Message Date
Pat Gavlin eaac9280f5
Refactor lifecycle tests. (#5575)
Move these tests to a new package, `lifecycletest`, that also exposes
APIs that allow consumers to implement their own lifecycle tests. This
is intended to ease the burden of testing plugin implementations and to
set the stage for cleaning up the lifecycle tests themselves.

This involves two changes to the public API, only one of which is
strictly necessary:

- The `host` field of `UpdateOptions` is now exported
- The `Journal` type has been moved from test-only code to the package
  proper

The former change is necessary, as it is the mechanism by which package
consumers may inject their own plugin loaders. I was reluctant to expose
this field originally because I wanted to ensure that the behavior of
packages that embed Pulumi is consistent with that of the Pulumi CLI
with respect to plugin loading. I now believe that the risk of consumers
changing this behavior outside of test scenarios is low enough that we
can expose this field. This may also be useful for future scenarios,
e.g. statically linking providers and Pulumi programs.

The latter change is not necessary, but fleshes out the engine package
into a more complete toolkit. Downstream consumers may use the Journal
type to conveniently implement snapshotting.
2020-10-15 10:35:09 -07:00
Pat Gavlin 249140242e
Add support for provider-side preview. (#5443)
These changes add support for provider-side previews of create and
update operations, which allows resource providers to supply output
property values for resources that are being created or updated during a
preview.

If a plugin supports provider-side preview, its create/update methods
will be invoked during previews with the `preview` property set to true.
It is the responsibility of the provider to fill in any output
properties that are known before returning. It is a best practice for
providers to only fill in property values that are guaranteed to be
identical if the preview were instead an update (i.e. only those output
properties whose values can be conclusively determined without
actually performing the create/update operation should be populated).
Providers that support previews must accept unknown values in their
create and update methods.

If a plugin does not support provider-side preview, the inputs to a
create or update operation will be propagated to the outputs as they are
today.

Fixes #4992.
2020-10-09 13:13:55 -07:00
Pat Gavlin 855f1fd1cd
Revise host mode. (#5317)
* Revise host mode.

The current implementation of host mode uses a `pulumi host` command and
an ad-hoc communication protocol between the engine and client to
connect a language host after the host has begun listening. The most
significant disadvantages of this approach are the communication
protocol (which currently requires the use of stdout), the host-specific
command, and the difficulty of accommodating the typical program-bound
lifetime for an update.

These changes reimplement host mode by adding engine support for
connecting to an existing language runtime service rather than launching
a plugin. This capability is provided via an engine-specific language
runtime, `client`, which accepts the address of the existing languge
runtime service as a runtime option. The CLI exposes this runtime via
the `--client` flag to the `up` and `preview` commands, which similarly
accepts the address of an existing language runtime service as an
argument. These changes also adjust the automation API to consume the
new host mode implementation.
2020-09-14 17:40:17 -07:00
Pat Gavlin 2585b86aa4
Initial support for remote component construction. (#5280)
These changes add initial support for the construction of remote
components. For now, this support is limited to the NodeJS SDK;
follow-up changes will implement support for the other SDKs.

Remote components are component resources that are constructed and
managed by plugins rather than by Pulumi programs. In this sense, they
are a bit like cloud resources, and are supported by the same
distribution and plugin loading mechanisms and described by the same
schema system.

The construction of a remote component is initiated by a
`RegisterResourceRequest` with the new `remote` field set to `true`.
When the resource monitor receives such a request, it loads the plugin
that implements the component resource and calls the `Construct`
method added to the resource provider interface as part of these
changes. This method accepts the information necessary to construct the
component and its children: the component's name, type, resource
options, inputs, and input dependencies. It is responsible for
dispatching to the appropriate component factory to create the
component, then returning its URN, resolved output properties, and
output property dependencies. The dependency information is necessary to
support features such as delete-before-replace, which rely on precise
dependency information for custom resources.

These changes also add initial support for more conveniently
implementing resource providers in NodeJS. The interface used to
implement such a provider is similar to the dynamic provider interface
(and may be unified with that interface in the future).

An example of a NodeJS program constructing a remote component resource
also implemented in NodeJS can be found in
`tests/construct_component/nodejs`.

This is the core of #2430.
2020-09-07 19:33:55 -07:00
Pat Gavlin 418e2291a2
Protect against engine event mutation. (#5003)
Certain operations in `engine/diff` mutate engine events during display.
This mutation can occur concurrently with the serialization of the event
for persistence, which causes a panic in the CLI. These changes fix the
offending code and add code that copies each engine event before
persisteing it in order to guard against future issues.
2020-07-16 23:52:31 -07:00
Paul Stack 0824fc9a8b
Revert "Narrow a few interfaces." (#4987) 2020-07-09 15:19:12 +01:00
Pat Gavlin 45d2fa95d6
Narrow a few interfaces. (#4934)
- Remove `Info` from `Source`. This method was not used.
- Remove `Stack` from `EvalSource`. This method was not used.
- Remove `Type` and `URN` from `Step`. These values are available via
  `Res().URN.Type()` and `Res().URN`, respectively. This removes the
  possibility of inconsistencies between the type, URN, and state of the
  resource associated with a `Step`.
- Remove URN from StepEventMetadata.
2020-07-01 15:32:50 -07:00
Justin Van Patten 7f27618e2d
Avoid replace on second update with import applied (#4403)
After importing some resources, and running a second update with the
import still applied, an unexpected replace would occur. This wouldn't
happen for the vast majority of resources, but for some it would.

It turns out that the resources that trigger this are ones that use a
different format of identifier for the import input than they do for the
ID property.

Before this change, we would trigger an import-replacement when an
existing resource's ID property didn't match the import property, which
would be the case for the small set of resources where the input
identifier is different than the ID property.

To avoid this, we now store the `importID` in the statefile, and
compare that to the import property instead of comparing the ID.
2020-04-15 18:52:40 -07:00
CyrusNajmabadi 66bd3f4aa8
Breaking changes due to Feature 2.0 work
* Make `async:true` the default for `invoke` calls (#3750)

* Switch away from native grpc impl. (#3728)

* Remove usage of the 'deasync' library from @pulumi/pulumi. (#3752)

* Only retry as long as we get unavailable back.  Anything else continues. (#3769)

* Handle all errors for now. (#3781)


* Do not assume --yes was present when using pulumi in non-interactive mode (#3793)

* Upgrade all paths for sdk and pkg to v2

* Backport C# invoke classes and other recent gen changes (#4288)

Adjust C# generation

* Replace IDeployment with a sealed class (#4318)

Replace IDeployment with a sealed class

* .NET: default to args subtype rather than Args.Empty (#4320)

* Adding system namespace for Dotnet code gen

This is required for using Obsolute attributes for deprecations

```
Iam/InstanceProfile.cs(142,10): error CS0246: The type or namespace name 'ObsoleteAttribute' could not be found (are you missing a using directive or an assembly reference?) [/Users/stack72/code/go/src/github.com/pulumi/pulumi-aws/sdk/dotnet/Pulumi.Aws.csproj]
Iam/InstanceProfile.cs(142,10): error CS0246: The type or namespace name 'Obsolete' could not be found (are you missing a using directive or an assembly reference?) [/Users/stack72/code/go/src/github.com/pulumi/pulumi-aws/sdk/dotnet/Pulumi.Aws.csproj]
```

* Fix the nullability of config type properties in C# codegen (#4379)
2020-04-14 09:30:25 +01:00
evanboyle d3f5bbce48 go fmt 2020-03-18 17:27:02 -07:00
evanboyle c1440e48d4 move pkg/util/result -> sdk/go/common/util 2020-03-18 15:45:42 -07:00
evanboyle 7ff46cb4fa move pkg/util/rpcutil -> sdk/go/common/util/rpcutil 2020-03-18 15:37:13 -07:00
evanboyle c3f6ae2451 move pkg/util/logging -> sdk/go/common/util/logging 2020-03-18 15:34:58 -07:00
evanboyle 8df534a71e move pkg/diag -> sdk/go/common/diag 2020-03-18 15:09:29 -07:00
evanboyle f754b486b8 move pkg/resource/config -> sdk/go/common/resource/config 2020-03-18 15:03:37 -07:00
evanboyle 70f386a967 move pkg/tokens -> sdk/go/common/tokens 2020-03-18 14:49:56 -07:00
evanboyle fccf301d14 move pkg/util/contract -> sdk/go/common/util/contract 2020-03-18 14:40:07 -07:00
evanboyle 8fb3f428b0 move pkg/workspace -> sdk/go/common/workspace 2020-03-18 14:35:53 -07:00
evanboyle dfab571aac move pkg/resource/plugin -> sdk/go/common/resource/plugin 2020-03-18 14:26:24 -07:00
evanboyle fba783caf9 move pkg/resource -> sdk/go/common/resource, but leave nested resource packages 2020-03-18 13:36:19 -07:00
Tasia Halim c96271b7a3
Support transformations in Go (#3978)
* started transformations for go sdk

* added first basic test

* added second test with child

* added RegisterStackTransformation

* added a couple tests to lifecycle_test

* update CHANGELOG and test

* included TODO for #3846
2020-03-02 13:59:11 -08:00
Pat Gavlin f168bdc1c2
Redesign the Go SDK resource/input/output system. (#3506)
The redesign is focused around providing better static typings and
improved ease-of-use for the Go SDK. Most of the redesign revolves
around three pivots:
- Strongly-typed inputs, especially for nested types
- Struct-based resource and invoke APIs
- Ease-of-use of Apply

1. Strongly-typed inputs

Input is the type of a generic input value for a Pulumi resource.
This type is used in conjunction with Output to provide polymorphism
over strongly-typed input values.

The intended pattern for nested Pulumi value types is to define an
input interface and a plain, input, and output variant of the value
type that implement the input interface.

For example, given a nested Pulumi value type with the following shape:

```
type Nested struct {
    Foo int
    Bar string
}
```

We would define the following:

```
var nestedType = reflect.TypeOf((*Nested)(nil)).Elem()

type NestedInput interface {
    pulumi.Input

    ToNestedOutput() NestedOutput
    ToNestedOutputWithContext(context.Context) NestedOutput
}

type Nested struct {
    Foo int `pulumi:"foo"`
    Bar string `pulumi:"bar"`
}

type NestedInputValue struct {
    Foo pulumi.IntInput `pulumi:"foo"`
    Bar pulumi.StringInput `pulumi:"bar"`
}

func (NestedInputValue) ElementType() reflect.Type {
    return nestedType
}

func (v NestedInputValue) ToNestedOutput() NestedOutput {
    return pulumi.ToOutput(v).(NestedOutput)
}

func (v NestedInputValue) ToNestedOutputWithContext(ctx context.Context) NestedOutput {
    return pulumi.ToOutputWithContext(ctx, v).(NestedOutput)
}

type NestedOutput struct { *pulumi.OutputState }

func (NestedOutput) ElementType() reflect.Type {
    return nestedType
}

func (o NestedOutput) ToNestedOutput() NestedOutput {
    return o
}

func (o NestedOutput) ToNestedOutputWithContext(ctx context.Context) NestedOutput {
    return o
}

func (o NestedOutput) Foo() pulumi.IntOutput {
    return o.Apply(func (v Nested) int {
        return v.Foo
    }).(pulumi.IntOutput)
}

func (o NestedOutput) Bar() pulumi.StringOutput {
    return o.Apply(func (v Nested) string {
        return v.Bar
    }).(pulumi.StringOutput)
}
```

The SDK provides input and output types for primitives, arrays, and
maps.

2. Struct-based APIs

Instead of providing expected output properties in the input map passed
to {Read,Register}Resource and returning the outputs as a map, the user
now passes a pointer to a struct that implements one of the Resource
interfaces and has appropriately typed and tagged fields that represent
its output properties.

For example, given a custom resource with an int-typed output "foo" and
a string-typed output "bar", we would define the following
CustomResource type:

```
type MyResource struct {
    pulumi.CustomResourceState

    Foo pulumi.IntOutput    `pulumi:"foo"`
    Bar pulumi.StringOutput `pulumi:"bar"`
}
```

And invoke RegisterResource like so:

```
var resource MyResource
err := ctx.RegisterResource(tok, name, props, &resource, opts...)
```

Invoke arguments and results are also provided via structs, but use
plain-old Go types for their fields:

```
type MyInvokeArgs struct {
    Foo int `pulumi:"foo"`
}

type MyInvokeResult struct {
    Bar string `pulumi:"bar"`
}

var result MyInvokeResult
err := ctx.Invoke(tok, MyInvokeArgs{Foo: 42}, &result, opts...)
```

3. Ease-of-use of Apply

All `Apply` methods now accept an interface{} as the callback type.
The provided callback value must have one of the following signatures:

	func (v T) U
	func (v T) (U, error)
	func (ctx context.Context, v T) U
	func (ctx context.Context, v T) (U, error)

T must be assignable from the ElementType of the Output. If U is a type
that has a registered Output type, the result of the Apply will be the
corresponding Output type. Otherwise, the result of the Apply will be
AnyOutput.

Fixes https://github.com/pulumi/pulumi/issues/2149.
Fixes https://github.com/pulumi/pulumi/issues/3488.
Fixes https://github.com/pulumi/pulumi/issues/3487.
Fixes https://github.com/pulumi/pulumi-aws/issues/248.
Fixes https://github.com/pulumi/pulumi/issues/3492.
Fixes https://github.com/pulumi/pulumi/issues/3491.
Fixes https://github.com/pulumi/pulumi/issues/3562.
2020-01-18 10:08:37 -05:00
Evan Boyle 1ca50d4b89
Propagate parent and providers for go SDK calls (#3563) 2019-11-26 13:23:34 -08:00
Evan Boyle c83e4f9ca6
Fix go SDK ReadResource (#3581) 2019-11-25 15:31:12 -08:00
Evan Boyle a47103b49d
fix go sdk delete before replace implementation (#3572) 2019-11-25 14:10:06 -08:00
CyrusNajmabadi 1908a18d20 Loosen resource targeting restrictions. (#3426)
- If an untargeted create would not affect the inputs of any targeted
  resources, do not fail the update. Untargeted creates that are
  directly dependend on by targeted resources will still cause failures
  that inform the user to add the untargeted resources to the --target
  list.
- Users may now pass the `--target-dependents` flag to allow targeted
  destroys to automatically target dependents that must be destroyed in
  order to destroy an explicitly targeted resource.
2019-11-18 20:28:25 -08:00
Evan Boyle 3ac8dd5285
Add support to the go sdk for IgnoreChanges (#3514) 2019-11-18 16:47:19 -08:00
Pat Gavlin a7f61a59b0
Reimplement Output for Go. (#3496)
- Use a mutex + condition variable instead of a channel for
  synchronizaiton in order to allow multiple calls to resolve/reject
- Properly handle outputs that are resolved to other outputs, especially
  if those outputs are not of exactly type Output
- Remove the Value() methods that allowed prompt access to output values
- Add variants of `Apply` that take a context parameter
- Ensure that resource outputs properly incorporate their resource as
  a dependency
- Make `Output` a plain struct. Uninitialized outputs will be treated as
   resolved and unknown. This makes conversions between output
   types more ergonomic.

Contributes to #3492.
2019-11-12 14:20:06 -08:00
Pat Gavlin 137fd54f1c
Propagate inputs to outputs during preview. (#3327)
These changes restore a more-correct version of the behavior that was
disabled with #3014. The original implementation of this behavior was
done in the SDKs, which do not have access to the complete inputs for a
resource (in particular, default values filled in by the provider during
`Check` are not exposed to the SDK). This lack of information meant that
the resolved output values could disagree with the typings present in
a provider SDK. Exacerbating this problem was the fact that unknown
values were dropped entirely, causing `undefined` values to appear in
unexpected places.

By doing this in the engine and allowing unknown values to be
represented in a first-class manner in the SDK, we can attack both of
these issues.

Although this behavior is not _strictly_ consistent with respect to the
resource model--in an update, a resource's output properties will come
from its provider and may differ from its input properties--this
behavior was present in the product for a fairly long time without
significant issues. In the future, we may be able to improve the
accuracy of resource outputs during a preview by allowing the provider
to dry-run CRUD operations and return partially-known values where
possible.

These changes also introduce new APIs in the Node and Python SDKs
that work with unknown values in a first-class fashion:
- A new parameter to the `apply` function that indicates that the
  callback should be run even if the result of the apply contains
  unknown values
- `containsUnknowns` and `isUnknown`, which return true if a value
  either contains nested unknown values or is exactly an unknown value
- The `Unknown` type, which represents unknown values

The primary use case for these APIs is to allow nested, properties with
known values to be accessed via the lifted property accessor even when
the containing property is not fully know. A common example of this
pattern is the `metadata.name` property of a Kubernetes `Namespace`
object: while other properties of the `metadata` bag may be unknown,
`name` is often known. These APIs allow `ns.metadata.name` to return a
known value in this case.

In order to avoid exposing downlevel SDKs to unknown values--a change
which could break user code by exposing it to unexpected values--a
language SDK must indicate whether or not it supports first-class
unknown values as part of each `RegisterResourceRequest`.

These changes also allow us to avoid breaking user code with the new
behavior introduced by the prior commit.

Fixes #3190.
2019-11-11 12:09:34 -08:00
Evan Boyle 9506b69c8b
error instead of panic when different resources use the same alias (#3457) 2019-11-06 08:49:13 -08:00
Pat Gavlin 23a84df254
Add targeted replaces to update. (#3418)
Allow the user to specify a set of resources to replace via the
`--replace` flag on the CLI. This can be combined with `--target` to
replace a specific set of resources without changing any other
resources. `--target-replace` is shorthand for `--replace urn --target urn`.

Fixes #2643.
2019-10-30 17:16:55 -07:00
CyrusNajmabadi fcc57f2e0f
Fix crash when specifying update-target that doesn't exist. (#3408) 2019-10-28 11:41:43 -07:00
Pat Gavlin d18b59e9c6
Fix a dependency graph bug during DBR. (#3329)
The dependency graph used to determine the set of resources that
depend on a resource being DBR'd is constructured from the list of
resource states present in the old snapshot. However, the dependencies
of resources that are present in both the old snapshot and the current
plan can be different, which in turn can cause the engine to make
incorrect decisions during DBR with respect to which resources need to
be replaced. For example, consider the following program:

```
var resA = new Resource("a", {dbr: "foo"});
var resB = new Resource("b", {dbr: resA.prop});
```

If this program is then changed to:
```
var resB = new Resource("b", {dbr: "<literal value of resA.prop>"});
var resA = new Resource("a", {dbr: "bar"});
```

The engine will first decide to make no changes to "b", as its input
property values have not changed. "b" has changed, however, such that it
no longer has a dependency on "a".

The engine will then decide to DBR "a". In the process, it will
determine that it first needs to delete "b", because the state for "b"
that is used when calculating "a"'s dependents does not reflect the
changes made during the plan.

To fix this issue, we rely on the observation that dependents can only
have been _removed_ from the base dependency graph: for a dependent to
have been added, it would have had to have been registered prior to the
root--a resource it depends on--which is not a valid operation. This
means that any resources that depend on the root must not yet have
been registered, which in turn implies that resources that have already
been registered must not depend on the root. Thus, we ignore these
resources if they are encountered while walking the old dependency graph
to determine the set of dependents.
2019-10-12 17:22:13 -07:00
Pat Gavlin 834e583c95
Revert "Propagate inputs to outputs during preview. (#3245)" (#3324)
This reverts commit 80504bf0bc.
2019-10-10 10:33:05 -07:00
CyrusNajmabadi 52884096e9
Add support for updating a subset of resources in the stack (i.e. --target) (#3251) 2019-09-30 23:41:56 -07:00
Pat Gavlin 80504bf0bc
Propagate inputs to outputs during preview. (#3245)
These changes restore a more-correct version of the behavior that was
disabled with #3014. The original implementation of this behavior was
done in the SDKs, which do not have access to the complete inputs for a
resource (in particular, default values filled in by the provider during
`Check` are not exposed to the SDK). This lack of information meant that
the resolved output values could disagree with the typings present in
a provider SDK. Exacerbating this problem was the fact that unknown
values were dropped entirely, causing `undefined` values to appear in
unexpected places.

By doing this in the engine and allowing unknown values to be
represented in a first-class manner in the SDK, we can attack both of
these issues.

Although this behavior is not _strictly_ consistent with respect to the
resource model--in an update, a resource's output properties will come
from its provider and may differ from its input properties--this
behavior was present in the product for a fairly long time without
significant issues. In the future, we may be able to improve the
accuracy of resource outputs during a preview by allowing the provider
to dry-run CRUD operations and return partially-known values where
possible.

These changes also introduce new APIs in the Node and Python SDKs
that work with unknown values in a first-class fashion:
- A new parameter to the `apply` function that indicates that the
  callback should be run even if the result of the apply contains
  unknown values
- `containsUnknowns` and `isUnknown`, which return true if a value
  either contains nested unknown values or is exactly an unknown value
- The `Unknown` type, which represents unknown values

The primary use case for these APIs is to allow nested, properties with
known values to be accessed via the lifted property accessor even when
the containing property is not fully know. A common example of this
pattern is the `metadata.name` property of a Kubernetes `Namespace`
object: while other properties of the `metadata` bag may be unknown,
`name` is often known. These APIs allow `ns.metadata.name` to return a
known value in this case.

In order to avoid exposing downlevel SDKs to unknown values--a change
which could break user code by exposing it to unexpected values--a
language SDK must indicate whether or not it supports first-class
unknown values as part of each `RegisterResourceRequest`.

These changes also allow us to avoid breaking user code with the new
behavior introduced by the prior commit.

Fixes #3190.
2019-09-30 11:03:58 -07:00
CyrusNajmabadi c1ff9c37f8
Delete specific target (#3244) 2019-09-19 19:28:14 -07:00
CyrusNajmabadi f788eb8fc1
Add support for refreshing specific targets. (#3225) 2019-09-17 18:14:10 -07:00
Pat Gavlin 2455564ddc
Allow IDs to change during import. (#3133)
This is necessary for resources like `aws.ec2.RouteTableAssociation`.

Part of https://github.com/pulumi/pulumi-aws/issues/708.
2019-08-23 15:00:24 -07:00
Pat Gavlin 42fc75fffe
Fail read steps with missing resources. (#3123)
Just what it says on the tin.

Fixes #262.
2019-08-21 10:09:02 -07:00
Pat Gavlin 8745440c1b
Allow users to explicitly disable delete-before-replace. (#3118)
With these changes, a user may explicitly set `deleteBeforeReplace` to
`false` in order to disable DBR behavior for a particular resource. This
is the SDK + CLI escape hatch for cases where the changes in
https://github.com/pulumi/pulumi-terraform/pull/465 cause undesirable
behavior.
2019-08-20 15:51:02 -07:00
Paul Stack f8db8e4209
Allow resource IDs to change on reresh steps (#3087)
* Allow resource IDs to change on reresh steps

This is a requirement for us to be able to move forward with
versions of the Terraform Azurerm provider. In v1.32.1, there was
a state migration that changed the ID format of the azure table
storage resource

We used to have a check in place for old ID being equal to new ID.
This has been changed now and we allow the change of ID to happen
in the RefreshStep

* Update pkg/resource/deploy/step.go

Co-Authored-By: Pat Gavlin <pat@pulumi.com>
2019-08-16 21:04:03 +03:00
Pat Gavlin 67ec74bdc5
Pass ignoreChanges to providers. (#3005)
These changes add support for passing `ignoreChanges` paths to resource
providers. This is intended to accommodate providers that perform diffs
between resource inputs and resource state (e.g. all Terraform-based
providers, the k8s provider when using API server dry-runs). These paths
are specified using the same syntax as the paths used in detailed diffs.

In addition to passing these paths to providers, the existing support
for `ignoreChanges` in inputs has been extended to accept paths rather
than top-level keys. It is an error to specify a path that is missing
one or more component in the old or new inputs.

Fixes #2936, #2663.
2019-07-31 11:39:07 -05:00
Pat Gavlin c6916051f0
Use a bag for misc. resource options in deploytest (#2977)
Most of these options are typically left unset. In order to make it
easier to update the lifecycle test when adding new options, collect
them in a bag s.t. most callsites can go without being updated.
2019-07-25 11:18:40 -07:00
Pat Gavlin fa05e5cb05
Migrate old providers without outputs. (#2973)
If we encounter a provider with old inputs but no old outputs when reading
a checkpoint file, use the old inputs as the old outputs. This handles the
scenario where the CLI is being upgraded from a version that did not
reflect provider inputs to provider outputs, and a provider is being
upgraded from a version that did not implement `DiffConfig` to a version
that does.

Fixes https://github.com/pulumi/pulumi-kubernetes/issues/645.
2019-07-23 13:39:21 -07:00
Paul Stack 02ffff8840
Addition of Custom Timeouts (#2885)
* Plumbing the custom timeouts from the engine to the providers

* Plumbing the CustomTimeouts through to the engine and adding test to show this

* Change the provider proto to include individual timeouts

* Plumbing the CustomTimeouts from the engine through to the Provider RPC interface

* Change how the CustomTimeouts are sent across RPC

These errors were spotted in testing. We can now see that the timeout
information is arriving in the RegisterResourceRequest

```
req=&pulumirpc.RegisterResourceRequest{
           Type:                    "aws:s3/bucket:Bucket",
           Name:                    "my-bucket",
           Parent:                  "urn:pulumi:dev::aws-vpc::pulumi:pulumi:Stack::aws-vpc-dev",
           Custom:                  true,
           Object:                  &structpb.Struct{},
           Protect:                 false,
           Dependencies:            nil,
           Provider:                "",
           PropertyDependencies:    {},
           DeleteBeforeReplace:     false,
           Version:                 "",
           IgnoreChanges:           nil,
           AcceptSecrets:           true,
           AdditionalSecretOutputs: nil,
           Aliases:                 nil,
           CustomTimeouts:          &pulumirpc.RegisterResourceRequest_CustomTimeouts{
               Create:               300,
               Update:               400,
               Delete:               500,
               XXX_NoUnkeyedLiteral: struct {}{},
               XXX_unrecognized:     nil,
               XXX_sizecache:        0,
           },
           XXX_NoUnkeyedLiteral: struct {}{},
           XXX_unrecognized:     nil,
           XXX_sizecache:        0,
       }
```

* Changing the design to use strings

* CHANGELOG entry to include the CustomTimeouts work

* Changing custom timeouts to be passed around the engine as converted value

We don't want to pass around strings - the user can provide it but we want
to make the engine aware of the timeout in seconds as a float64
2019-07-16 00:26:28 +03:00
Pat Gavlin e1a52693dc
Add support for importing existing resources. (#2893)
A resource can be imported by setting the `import` property in the
resource options bag when instantiating a resource. In order to
successfully import a resource, its desired configuration (i.e. its
inputs) must not differ from its actual configuration (i.e. its state)
as calculated by the resource's provider.

There are a few interesting state transitions hiding here when importing
a resource:
1. No prior resource exists in the checkpoint file. In this case, the
   resource is simply imported.
2. An external resource exists in the checkpoint file. In this case, the
   resource is imported and the old external state is discarded.
3. A non-external resource exists in the checkpoint file and its ID is
   different from the ID to import. In this case, the new resource is
   imported and the old resource is deleted.
4. A non-external resource exists in the checkpoint file, but the ID is
   the same as the ID to import. In this case, the import ID is ignored
   and the resource is treated as it would be in all cases except for
   changes that would replace the resource. In that case, the step
   generator issues an error that indicates that the import ID should be
   removed: were we to move forward with the replace, the new state of
   the stack would fall under case (3), which is almost certainly not
   what the user intends.

Fixes #1662.
2019-07-12 11:12:01 -07:00
Pat Gavlin 6e5c4a38d8
Defer all diffs to resource providers. (#2849)
Thse changes make a subtle but critical adjustment to the process the
Pulumi engine uses to determine whether or not a difference exists
between a resource's actual and desired states, and adjusts the way this
difference is calculated and displayed accordingly.

Today, the Pulumi engine get the first chance to decide whether or not
there is a difference between a resource's actual and desired states. It
does this by comparing the current set of inputs for a resource (i.e.
the inputs from the running Pulumi program) with the last set of inputs
used to update the resource. If there is no difference between the old
and new inputs, the engine decides that no change is necessary without
consulting the resource's provider. Only if there are changes does the
engine consult the resource's provider for more information about the
difference. This can be problematic for a number of reasons:

- Not all providers do input-input comparison; some do input-state
  comparison
- Not all providers are able to update the last deployed set of inputs
  when performing a refresh
- Some providers--either intentionally or due to bugs--may see changes
  in resources whose inputs have not changed

All of these situations are confusing at the very least, and the first
is problematic with respect to correctness. Furthermore, the display
code only renders diffs it observes rather than rendering the diffs
observed by the provider, which can obscure the actual changes detected
at runtime.

These changes address both of these issues:
- Rather than comparing the current inputs against the last inputs
  before calling a resource provider's Diff function, the engine calls
  the Diff function in all cases.
- Providers may now return a list of properties that differ between the
  requested and actual state and the way in which they differ. This
  information will then be used by the CLI to render the diff
  appropriately. A provider may also indicate that a particular diff is
  between old and new inputs rather than old state and new inputs.

Fixes #2453.
2019-07-01 12:34:19 -07:00
Sean Gillespie 2870518a64 Refine resource replacement logic for providers (#2767)
This commit touches an intersection of a few different provider-oriented
features that combined to cause a particularly severe bug that made it
impossible for users to upgrade provider versions without seeing
replacements with their resources.

For some context, Pulumi models all providers as resources and places
them in the snapshot like any other resource. Every resource has a
reference to the provider that created it. If a Pulumi program does not
specify a particular provider to use when performing a resource
operation, the Pulumi engine injects one automatically; these are called
"default providers" and are the most common ways that users end up with
providers in their snapshot. Default providers can be identified by
their name, which is always prefixed with "default".

Recently, in an effort to make the Pulumi engine more flexible with
provider versions, it was made possible for the engine to have multiple
default providers active for a provider of a particular type, which was
previously not possible. Because a provider is identified as a tuple of
package name and version, it was difficult to find a name for these
duplicate default providers that did not cause additional problems. The
provider versioning PR gave these default providers a name that was
derived from the version of the package. This proved to be a problem,
because when users upgraded from one version of a package to another,
this changed the name of their default provider which in turn caused all
of their resources created using that provider (read: everything) to be
replaced.

To combat this, this PR introduces a rule that the engine will apply
when diffing a resource to determine whether or not it needs to be
replaced: "If a resource's provider changes, and both old and new
providers are default providers whose properties do not require
replacement, proceed as if there were no diff." This allows the engine
to gracefully recognize and recover when a resource's default provider changes
names, as long as the provider's config has not changed.
2019-06-03 12:16:31 -07:00
Luke Hoban 15e924b5cf
Support aliases for renaming, re-typing, or re-parenting resources (#2774)
Adds a new resource option `aliases` which can be used to rename a resource.  When making a breaking change to the name or type of a resource or component, the old name can be added to the list of `aliases` for a resource to ensure that existing resources will be migrated to the new name instead of being deleted and replaced with the new named resource.

There are two key places this change is implemented. 

The first is the step generator in the engine.  When computing whether there is an old version of a registered resource, we now take into account the aliases specified on the registered resource.  That is, we first look up the resource by its new URN in the old state, and then by any aliases provided (in order).  This can allow the resource to be matched as a (potential) update to an existing resource with a different URN.

The second is the core `Resource` constructor in the JavaScript (and soon Python) SDKs.  This change ensures that when a parent resource is aliased, that all children implicitly inherit corresponding aliases.  It is similar to how many other resource options are "inherited" implicitly from the parent.

Four specific scenarios are explicitly tested as part of this PR:
1. Renaming a resource
2. Adopting a resource into a component (as the owner of both component and consumption codebases)
3. Renaming a component instance (as the owner of the consumption codebase without changes to the component)
4. Changing the type of a component (as the owner of the component codebase without changes to the consumption codebase)
4. Combining (1) and (3) to make both changes to a resource at the same time
2019-05-31 23:01:01 -07:00