This change adds support for lists and maps in config. We now allow
lists/maps (and nested structures) in `Pulumi.<stack>.yaml` (or
`Pulumi.<stack>.json`; yes, we currently support that).
For example:
```yaml
config:
proj:blah:
- a
- b
- c
proj:hello: world
proj:outer:
inner: value
proj:servers:
- port: 80
```
While such structures could be specified in the `.yaml` file manually,
we support setting values in maps/lists from the command line.
As always, you can specify single values with:
```shell
$ pulumi config set hello world
```
Which results in the following YAML:
```yaml
proj:hello world
```
And single value secrets via:
```shell
$ pulumi config set --secret token shhh
```
Which results in the following YAML:
```yaml
proj:token:
secure: v1:VZAhuroR69FkEPTk:isKafsoZVMWA9pQayGzbWNynww==
```
Values in a list can be set from the command line using the new
`--path` flag, which indicates the config key contains a path to a
property in a map or list:
```shell
$ pulumi config set --path names[0] a
$ pulumi config set --path names[1] b
$ pulumi config set --path names[2] c
```
Which results in:
```yaml
proj:names
- a
- b
- c
```
Values can be obtained similarly:
```shell
$ pulumi config get --path names[1]
b
```
Or setting values in a map:
```shell
$ pulumi config set --path outer.inner value
```
Which results in:
```yaml
proj:outer:
inner: value
```
Of course, setting values in nested structures is supported:
```shell
$ pulumi config set --path servers[0].port 80
```
Which results in:
```yaml
proj:servers:
- port: 80
```
If you want to include a period in the name of a property, it can be
specified as:
```
$ pulumi config set --path 'nested["foo.bar"]' baz
```
Which results in:
```yaml
proj:nested:
foo.bar: baz
```
Examples of valid paths:
- root
- root.nested
- 'root["nested"]'
- root.double.nest
- 'root["double"].nest'
- 'root["double"]["nest"]'
- root.array[0]
- root.array[100]
- root.array[0].nested
- root.array[0][1].nested
- root.nested.array[0].double[1]
- 'root["key with \"escaped\" quotes"]'
- 'root["key with a ."]'
- '["root key with \"escaped\" quotes"].nested'
- '["root key with a ."][100]'
Note: paths that contain quotes can be surrounded by single quotes.
When setting values with `--path`, if the value is `"false"` or
`"true"`, it will be saved as the boolean value, and if it is
convertible to an integer, it will be saved as an integer.
Secure values are supported in lists/maps as well:
```shell
$ pulumi config set --path --secret tokens[0] shh
```
Will result in:
```yaml
proj:tokens:
- secure: v1:wpZRCe36sFg1RxwG:WzPeQrCn4n+m4Ks8ps15MxvFXg==
```
Note: maps of length 1 with a key of “secure” and string value are
reserved for storing secret values. Attempting to create such a value
manually will result in an error:
```shell
$ pulumi config set --path parent.secure foo
error: "secure" key in maps of length 1 are reserved
```
**Accessing config values from the command line with JSON**
```shell
$ pulumi config --json
```
Will output:
```json
{
"proj:hello": {
"value": "world",
"secret": false,
"object": false
},
"proj:names": {
"value": "[\"a\",\"b\",\"c\"]",
"secret": false,
"object": true,
"objectValue": [
"a",
"b",
"c"
]
},
"proj:nested": {
"value": "{\"foo.bar\":\"baz\"}",
"secret": false,
"object": true,
"objectValue": {
"foo.bar": "baz"
}
},
"proj:outer": {
"value": "{\"inner\":\"value\"}",
"secret": false,
"object": true,
"objectValue": {
"inner": "value"
}
},
"proj:servers": {
"value": "[{\"port\":80}]",
"secret": false,
"object": true,
"objectValue": [
{
"port": 80
}
]
},
"proj:token": {
"secret": true,
"object": false
},
"proj:tokens": {
"secret": true,
"object": true
}
}
```
If the value is a map or list, `"object"` will be `true`. `"value"` will
contain the object as serialized JSON and a new `"objectValue"` property
will be available containing the value of the object.
If the object contains any secret values, `"secret"` will be `true`, and
just like with scalar values, the value will not be outputted unless
`--show-secrets` is specified.
**Accessing config values from Pulumi programs**
Map/list values are available to Pulumi programs as serialized JSON, so
the existing
`getObject`/`requireObject`/`getSecretObject`/`requireSecretObject`
functions can be used to retrieve such values, e.g.:
```typescript
import * as pulumi from "@pulumi/pulumi";
interface Server {
port: number;
}
const config = new pulumi.Config();
const names = config.requireObject<string[]>("names");
for (const n of names) {
console.log(n);
}
const servers = config.requireObject<Server[]>("servers");
for (const s of servers) {
console.log(s.port);
}
```
Allow the user to specify a set of resources to replace via the
`--replace` flag on the CLI. This can be combined with `--target` to
replace a specific set of resources without changing any other
resources. `--target-replace` is shorthand for `--replace urn --target urn`.
Fixes#2643.
This commit will introduce the ability to load providers in `query`
mode.
Previously, `query` mode has been effectively a stand-alone execution
environment for language hosts, running without (e.g.) the
`StepExecutor` and similar engine facilities, but with some minimal
constructs hooked up, notably the ability to retrieve stack snapshots
from the backend for querying.
This commit extends this functionality somewhat by allowing `query` to
load Pulumi resource providers, and to run `Invoke` on them. This will
allow us, in the future, to "query" resource providers in the same way
we can query stack snapshots.
This method can be used to check whether or not a URN is well-formed.
This is used by the provider reference parser to avoid panicking on
malformed URNs.
The dependency graph used to determine the set of resources that
depend on a resource being DBR'd is constructured from the list of
resource states present in the old snapshot. However, the dependencies
of resources that are present in both the old snapshot and the current
plan can be different, which in turn can cause the engine to make
incorrect decisions during DBR with respect to which resources need to
be replaced. For example, consider the following program:
```
var resA = new Resource("a", {dbr: "foo"});
var resB = new Resource("b", {dbr: resA.prop});
```
If this program is then changed to:
```
var resB = new Resource("b", {dbr: "<literal value of resA.prop>"});
var resA = new Resource("a", {dbr: "bar"});
```
The engine will first decide to make no changes to "b", as its input
property values have not changed. "b" has changed, however, such that it
no longer has a dependency on "a".
The engine will then decide to DBR "a". In the process, it will
determine that it first needs to delete "b", because the state for "b"
that is used when calculating "a"'s dependents does not reflect the
changes made during the plan.
To fix this issue, we rely on the observation that dependents can only
have been _removed_ from the base dependency graph: for a dependent to
have been added, it would have had to have been registered prior to the
root--a resource it depends on--which is not a valid operation. This
means that any resources that depend on the root must not yet have
been registered, which in turn implies that resources that have already
been registered must not depend on the root. Thus, we ignore these
resources if they are encountered while walking the old dependency graph
to determine the set of dependents.
* Fix lint warnings
* Improve error message when using PAC on old SDK
* Update pkg/resource/plugin/analyzer_plugin.go
Co-Authored-By: Justin Van Patten <jvp@justinvp.com>
These changes restore a more-correct version of the behavior that was
disabled with #3014. The original implementation of this behavior was
done in the SDKs, which do not have access to the complete inputs for a
resource (in particular, default values filled in by the provider during
`Check` are not exposed to the SDK). This lack of information meant that
the resolved output values could disagree with the typings present in
a provider SDK. Exacerbating this problem was the fact that unknown
values were dropped entirely, causing `undefined` values to appear in
unexpected places.
By doing this in the engine and allowing unknown values to be
represented in a first-class manner in the SDK, we can attack both of
these issues.
Although this behavior is not _strictly_ consistent with respect to the
resource model--in an update, a resource's output properties will come
from its provider and may differ from its input properties--this
behavior was present in the product for a fairly long time without
significant issues. In the future, we may be able to improve the
accuracy of resource outputs during a preview by allowing the provider
to dry-run CRUD operations and return partially-known values where
possible.
These changes also introduce new APIs in the Node and Python SDKs
that work with unknown values in a first-class fashion:
- A new parameter to the `apply` function that indicates that the
callback should be run even if the result of the apply contains
unknown values
- `containsUnknowns` and `isUnknown`, which return true if a value
either contains nested unknown values or is exactly an unknown value
- The `Unknown` type, which represents unknown values
The primary use case for these APIs is to allow nested, properties with
known values to be accessed via the lifted property accessor even when
the containing property is not fully know. A common example of this
pattern is the `metadata.name` property of a Kubernetes `Namespace`
object: while other properties of the `metadata` bag may be unknown,
`name` is often known. These APIs allow `ns.metadata.name` to return a
known value in this case.
In order to avoid exposing downlevel SDKs to unknown values--a change
which could break user code by exposing it to unexpected values--a
language SDK must indicate whether or not it supports first-class
unknown values as part of each `RegisterResourceRequest`.
These changes also allow us to avoid breaking user code with the new
behavior introduced by the prior commit.
Fixes#3190.
Adds the ability to provide `transformations` to modify the properties and resource options that will be used for any child resource of a component or stack.
This offers an "escape hatch" to modify the behaviour of a component by peeking behind it's abstraction. For example, it can be used to add a resource option (`additionalSecretOutputs`, `aliases`, `protect`, etc.) to a specific known child of a component, or to modify some input property to a child resource if the component does not (yet) expose the ability to control that input directly. It could also be used for more interesting scenarios - such as:
1. Automatically applying tags to all resources that support them in a stack (or component)
2. Injecting real dependencies between stringly-referenced resources in a Helm Chart
3. Injecting explicit names using a preferred naming convention across all resources in a stack
4. Injecting `import` onto all resources by doing a lookup into a name=>id mapping
Because this feature makes it possible to peek behind a component abstraction, it must be used with care in cases where the component is versioned independently of the use of transformations. Also, this can result in "spooky action at a distance", so should be used judiciously. That said - this can be used as an escape hatch to unblock a wide variety of common use cases without waiting on changes to be made in a component implementation.
Each transformation is passed the `resource`, `name`, `type`, `props` and `opts` that are passed into the `Resource` constructor for any resource descended from the resource that has the transformation applied. The transformation callback can optionally return alternate versions of the `props` and `opts` to be used in place of the original values provided to the resource constructor.
Fixes#2068.
This caching is enabled by wrapping the `secrets.Manager` returned by
`DefaultSecretsProvider.OfType` in an outer `secrets.Manager` that
cooperates with `stack.{Serialize,Deserialize}PropertyValue`. Ciphertext
is cached on a per-secret-instance basis (i.e. not a per-plaintext-value
basis). Cached ciphertext is only reused if the plaintext for the secret
value has not changed. Entries are inserted into the cache upon both
encryption and decryption so that values that originated from ciphertext
and that have not changed can aoid re-encryption.
Contributes to #3178.
* Fix some tracing issues.
- Add endpoints for `startUpdate` and `postEngineEventsBatch` so that
spans for these invocations have proper names
- Inject a tracing span when walking a plan so that resource operations
are properly parented
- When handling gRPC calls, inject a tracing span into the call's
metadata if no span is already present so that resource monitor and
engine spans are properly parented
- Do not trace client gRPC invocations of the empty method so that these
calls (which are used to determine server availability) do not muddy
the trace. Note that I tried parenting these spans appropriately, but
doing so broke the trace entirely.
With these changes, the only unparented span in a typical Pulumi
invocation is a single call to `getUser`. This span is unparented
because that call does not have a context available. Plumbing a context
into that particular call is surprisingly tricky, as it is often called
by other context-less functions.
* Make tracing support more flexible.
- Add support for writing trace data to a local file using Appdash
- Add support for viewing Appdash traces via the CLI
Fixes: #2319
In #2319, a user is hitting the gRPC limit on the message size the
server can receive when uploading ec2 user-data
This commit doubles the limit that can be sent from `1024*1024*4` to
`1024*1024*8`
Not all resource providers support Pulumi's Asset and Archive types. In
particular, the Kubernetes provider should reject any resource
definition that contains either of these types.
This commit will introduce two MarshalOptions that will make it easy for
the Kubernetes provider to guarantee that no properties of this type are
in a resource request, as it's deserializing the request from the
engine.
With these changes, a user may explicitly set `deleteBeforeReplace` to
`false` in order to disable DBR behavior for a particular resource. This
is the SDK + CLI escape hatch for cases where the changes in
https://github.com/pulumi/pulumi-terraform/pull/465 cause undesirable
behavior.
Attempting to `pulumi stack rename` a stack which had been created but
never updated, when using the local backend, was broken because
code-paths were not hardened against the snapshot being `nil` (which
is the case for a stack before the initial deployment had been done).
Fixes#2654
* Allow resource IDs to change on reresh steps
This is a requirement for us to be able to move forward with
versions of the Terraform Azurerm provider. In v1.32.1, there was
a state migration that changed the ID format of the azure table
storage resource
We used to have a check in place for old ID being equal to new ID.
This has been changed now and we allow the change of ID to happen
in the RefreshStep
* Update pkg/resource/deploy/step.go
Co-Authored-By: Pat Gavlin <pat@pulumi.com>
When using StackReference, if the stack you reference contains any
secret outputs, we have to mark the entire `outputs` member as a
secret output. This is because we only track secretness on a per
`Output<T>` basis.
For `getSecret` and friends, however, we know the name of the output
you are looking up and we can be smarter about if the returned
`Output<T>` should be treated as a secret or not.
This change augments the provider for StackReference such that it also
returns a list of top level stack output names who's values contain
secrets. In the language SDKs, we use this information, when present,
to decide if we should return an `Output<T>` that is marked as a
secret or not. Since the SDK and CLI are independent components, care
is taken to ensure that when the CLI does not return this information,
we behave as we did before (i.e. if any output is a secret, we treat
every output as a secret).
Fixes#2744
If we don't process and report the stderr of `npm install`, the output
is "orphaned" during error condition, and only something like "exit code
1" is reported.
Empty `[]interface{}` values were being converted to array property
values with a `nil` element, and empty array property values were being
coverted to `nil` `[]interface{}` values. These changes fix the
converters to return empty but non-nil values in both cases.
This is part of the fix for
https://github.com/pulumi/pulumi-kubernetes/issues/693.
- Ensure that type assertions are guarded, and that incorrectly-typed
properties return errors rather than panicking
- Expand the asset/archive tests in the Node SDK to ensure that eventual
archives and assets serialize and deserialize correctly
Fixes#2836.
Fixes#3016.
Adds support for additional cloud secrets providers (AWS KMS, Azure KeyVault, Google Cloud KMS, and HashiCorp Vault) as the encryption backend for Pulumi secrets. This augments the previous choice between using the app.pulumi.com-managed secrets encryption or a fully-client-side local passphrase encryption.
This is implemented using the Go Cloud Development Kit support for pluggable secrets providers.
Like our cloud storage backend support which also uses Go Cloud Development Kit, this PR also bleeds through to users the URI scheme's that the Go CDK defines for specifying each of secrets providers - like `awskms://alias/LukeTesting?region=us-west-2` or `azurekeyvault://mykeyvaultname.vault.azure.net/keys/mykeyname`.
Also like our cloud storage backend support, this PR doesn't solve for how to configure the cloud provider client used to resolve the URIs above - the standard ambient credentials are used in both cases. Eventually, we will likely need to provide ways for both of these features to be configured independently of each other and of the providers used for resource provisioning.
These changes add support for passing `ignoreChanges` paths to resource
providers. This is intended to accommodate providers that perform diffs
between resource inputs and resource state (e.g. all Terraform-based
providers, the k8s provider when using API server dry-runs). These paths
are specified using the same syntax as the paths used in detailed diffs.
In addition to passing these paths to providers, the existing support
for `ignoreChanges` in inputs has been extended to accept paths rather
than top-level keys. It is an error to specify a path that is missing
one or more component in the old or new inputs.
Fixes#2936, #2663.
Most of these options are typically left unset. In order to make it
easier to update the lifecycle test when adding new options, collect
them in a bag s.t. most callsites can go without being updated.
If we encounter a provider with old inputs but no old outputs when reading
a checkpoint file, use the old inputs as the old outputs. This handles the
scenario where the CLI is being upgraded from a version that did not
reflect provider inputs to provider outputs, and a provider is being
upgraded from a version that did not implement `DiffConfig` to a version
that does.
Fixes https://github.com/pulumi/pulumi-kubernetes/issues/645.
Currently, `pulumi preview` fails immediately when any resource
definition in a Pulumi app is found to be in violation of a resource
policy. But, users would like `preview` to report as many policy
violations as it can before terminating with an error, so that they can
fix many of them before running `preview` again.
This commit will thus change `pulumi preview` to do this sort of
"batching" for policy violations. The engine will attempt to run the
entire preview step, validating every resource definition with the
relevant known resource policies, before finally reporting an error if
any violations are detected.
Fixespulumi/pulumi-policy#31
This command will cause `pulumi policy publish` to behave in much the
same way `pulumi up` does -- if the policy program is in TypeScript, we
will use ts-node to attempt to compile in-process before executing, and
fall back to plain-old node.
We accomplish this by moving `cmd/run/run.ts` into a generic helper
package, `runtime/run.ts`, which slightly generalizes the use cases
supported (notably, allowing us to exec some program outside of the
context of a Pulumi stack).
This new package is then called by both `cmd/run/index.ts` and
`cmd/run-policy-pack/index.ts`.
* Plumbing the custom timeouts from the engine to the providers
* Plumbing the CustomTimeouts through to the engine and adding test to show this
* Change the provider proto to include individual timeouts
* Plumbing the CustomTimeouts from the engine through to the Provider RPC interface
* Change how the CustomTimeouts are sent across RPC
These errors were spotted in testing. We can now see that the timeout
information is arriving in the RegisterResourceRequest
```
req=&pulumirpc.RegisterResourceRequest{
Type: "aws:s3/bucket:Bucket",
Name: "my-bucket",
Parent: "urn:pulumi:dev::aws-vpc::pulumi:pulumi:Stack::aws-vpc-dev",
Custom: true,
Object: &structpb.Struct{},
Protect: false,
Dependencies: nil,
Provider: "",
PropertyDependencies: {},
DeleteBeforeReplace: false,
Version: "",
IgnoreChanges: nil,
AcceptSecrets: true,
AdditionalSecretOutputs: nil,
Aliases: nil,
CustomTimeouts: &pulumirpc.RegisterResourceRequest_CustomTimeouts{
Create: 300,
Update: 400,
Delete: 500,
XXX_NoUnkeyedLiteral: struct {}{},
XXX_unrecognized: nil,
XXX_sizecache: 0,
},
XXX_NoUnkeyedLiteral: struct {}{},
XXX_unrecognized: nil,
XXX_sizecache: 0,
}
```
* Changing the design to use strings
* CHANGELOG entry to include the CustomTimeouts work
* Changing custom timeouts to be passed around the engine as converted value
We don't want to pass around strings - the user can provide it but we want
to make the engine aware of the timeout in seconds as a float64
A resource can be imported by setting the `import` property in the
resource options bag when instantiating a resource. In order to
successfully import a resource, its desired configuration (i.e. its
inputs) must not differ from its actual configuration (i.e. its state)
as calculated by the resource's provider.
There are a few interesting state transitions hiding here when importing
a resource:
1. No prior resource exists in the checkpoint file. In this case, the
resource is simply imported.
2. An external resource exists in the checkpoint file. In this case, the
resource is imported and the old external state is discarded.
3. A non-external resource exists in the checkpoint file and its ID is
different from the ID to import. In this case, the new resource is
imported and the old resource is deleted.
4. A non-external resource exists in the checkpoint file, but the ID is
the same as the ID to import. In this case, the import ID is ignored
and the resource is treated as it would be in all cases except for
changes that would replace the resource. In that case, the step
generator issues an error that indicates that the import ID should be
removed: were we to move forward with the replace, the new state of
the stack would fall under case (3), which is almost certainly not
what the user intends.
Fixes#1662.
Thse changes make a subtle but critical adjustment to the process the
Pulumi engine uses to determine whether or not a difference exists
between a resource's actual and desired states, and adjusts the way this
difference is calculated and displayed accordingly.
Today, the Pulumi engine get the first chance to decide whether or not
there is a difference between a resource's actual and desired states. It
does this by comparing the current set of inputs for a resource (i.e.
the inputs from the running Pulumi program) with the last set of inputs
used to update the resource. If there is no difference between the old
and new inputs, the engine decides that no change is necessary without
consulting the resource's provider. Only if there are changes does the
engine consult the resource's provider for more information about the
difference. This can be problematic for a number of reasons:
- Not all providers do input-input comparison; some do input-state
comparison
- Not all providers are able to update the last deployed set of inputs
when performing a refresh
- Some providers--either intentionally or due to bugs--may see changes
in resources whose inputs have not changed
All of these situations are confusing at the very least, and the first
is problematic with respect to correctness. Furthermore, the display
code only renders diffs it observes rather than rendering the diffs
observed by the provider, which can obscure the actual changes detected
at runtime.
These changes address both of these issues:
- Rather than comparing the current inputs against the last inputs
before calling a resource provider's Diff function, the engine calls
the Diff function in all cases.
- Providers may now return a list of properties that differ between the
requested and actual state and the way in which they differ. This
information will then be used by the CLI to render the diff
appropriately. A provider may also indicate that a particular diff is
between old and new inputs rather than old state and new inputs.
Fixes#2453.
@keen99 pointed out that newer versions of golangci-lint were failing
due to some spelling errors. This change fixes them up. We have also
now have a work item to track moving to a newer golangci-lint tool in
the future.
Fixes#2841
This commit will expose the new `Invoke` routine that lists resource
outputs through the Node.js SDK.
This API is implemented via a new API, `EnumerablePromise`, which is a
collection of simple query primitives built onto the `Promise` API. The
query model is lazy and LINQ-like, and generally intended to make
`Promise` simpler to deal with in query scenarios. See #2601 for more
details.
Fixes#2600.
This commit touches an intersection of a few different provider-oriented
features that combined to cause a particularly severe bug that made it
impossible for users to upgrade provider versions without seeing
replacements with their resources.
For some context, Pulumi models all providers as resources and places
them in the snapshot like any other resource. Every resource has a
reference to the provider that created it. If a Pulumi program does not
specify a particular provider to use when performing a resource
operation, the Pulumi engine injects one automatically; these are called
"default providers" and are the most common ways that users end up with
providers in their snapshot. Default providers can be identified by
their name, which is always prefixed with "default".
Recently, in an effort to make the Pulumi engine more flexible with
provider versions, it was made possible for the engine to have multiple
default providers active for a provider of a particular type, which was
previously not possible. Because a provider is identified as a tuple of
package name and version, it was difficult to find a name for these
duplicate default providers that did not cause additional problems. The
provider versioning PR gave these default providers a name that was
derived from the version of the package. This proved to be a problem,
because when users upgraded from one version of a package to another,
this changed the name of their default provider which in turn caused all
of their resources created using that provider (read: everything) to be
replaced.
To combat this, this PR introduces a rule that the engine will apply
when diffing a resource to determine whether or not it needs to be
replaced: "If a resource's provider changes, and both old and new
providers are default providers whose properties do not require
replacement, proceed as if there were no diff." This allows the engine
to gracefully recognize and recover when a resource's default provider changes
names, as long as the provider's config has not changed.
Previously, when the CLI wanted to install a plugin, it used a special
method, `DownloadPlugin` on the `httpstate` backend to actually fetch
the tarball that had the plugin. The reason for this is largely tied
to history, at one point during a closed beta, we required presenting
an API key to download plugins (as a way to enforce folks outside the
beta could not download them) and because of that it was natural to
bake that functionality into the part of the code that interfaced with
the rest of the API from the Pulumi Service.
The downside here is that it means we need to host all the plugins on
`api.pulumi.com` which prevents community folks from being able to
easily write resource providers, since they have to manually manage
the process of downloading a provider to a machine and getting it on
the `$PATH` or putting it in the plugin cache.
To make this easier, we add a `--server` argument you can pass to
`pulumi plugin install` to control the URL that it attempts to fetch
the tarball from. We still have perscriptive guidence on how the
tarball must be
named (`pulumi-[<type>]-[<provider-name>]-vX.Y.Z.tar.gz`) but the base
URL can now be configured.
Folks publishing packages can use install scripts to run `pulumi
plugin install` passing a custom `--server` argument, if needed.
There are two improvements we can make to provide a nicer end to end
story here:
- We can augment the GetRequiredPlugins method on the language
provider to also return information about an optional server to use
when downloading the provider.
- We can pass information about a server to download plugins from as
part of a resource registration or creation of a first class
provider.
These help out in cases where for one reason or another where `pulumi
plugin install` doesn't get run before an update takes place and would
allow us to either do the right thing ahead of time or provide better
error messages with the correct `--server` argument. But, for now,
this unblocks a majority of the cases we care about and provides a
path forward for folks that want to develop and host their own
resource providers.
Adds a new resource option `aliases` which can be used to rename a resource. When making a breaking change to the name or type of a resource or component, the old name can be added to the list of `aliases` for a resource to ensure that existing resources will be migrated to the new name instead of being deleted and replaced with the new named resource.
There are two key places this change is implemented.
The first is the step generator in the engine. When computing whether there is an old version of a registered resource, we now take into account the aliases specified on the registered resource. That is, we first look up the resource by its new URN in the old state, and then by any aliases provided (in order). This can allow the resource to be matched as a (potential) update to an existing resource with a different URN.
The second is the core `Resource` constructor in the JavaScript (and soon Python) SDKs. This change ensures that when a parent resource is aliased, that all children implicitly inherit corresponding aliases. It is similar to how many other resource options are "inherited" implicitly from the parent.
Four specific scenarios are explicitly tested as part of this PR:
1. Renaming a resource
2. Adopting a resource into a component (as the owner of both component and consumption codebases)
3. Renaming a component instance (as the owner of the consumption codebase without changes to the component)
4. Changing the type of a component (as the owner of the component codebase without changes to the consumption codebase)
4. Combining (1) and (3) to make both changes to a resource at the same time
We model providers as resources in our state file, but we were
neglecting to set Outputs for these resources. This was problematic
when we started to try to run DiffConfig, because when diffing a
resource we compare thed new inputs and the old outputs, but the
resource never had any old outputs, so it was impossible for the
provider to see what the old state of the resource was.
To fix this, we now reflect the inputs we use the create the provider
reference as outputs on the resource.
The Kubernetes provider wanted to return Unimplemented for both
DiffConfig and CheckConfig. However, due to an interaction between the
package we used to construct the error we are returning and the
package we are using to actually construct the gRPC server for the
provider, we ended up in a place where the provider would actually end
up returning an error with code "Unknown", and the /text/ of the
message included information about it being due to the RPC not being
implemented.
So, when we try to call Diff/Check config on the provider, detect this
case as well and treat messages of this shape as if the provider just
returned "Unimplemented".
In 3621c01f4b, we implemented
CheckConfig/DiffConfig incorrectly. We should have explicilty added
the handlers (to supress the warnings we were getting) but returned an
error saying the RPC was not implemented. Instead, we just returned
success but passed back bogus data. This was "fine" at the time
because nothing called these methods.
Now that we are actually calling them, returning incorrect values
leads to errors in grpc. To deal with this we do two things:
1. Adjust the implementations in the dynamic provider to correctly
return not implemented. This allows us to pick up the default engine
behavior going forward.
2. Add some code in CheckConfig/DiffConfig that handle the gRPC error
that is returned when calling methods on the dynamic provider and fall
back to the legacy behavior. This means updating your CLI will not
cause issues for existing resources where the SDK has not been
updated.
For provider plugins, the gRPC interfaces expect that a URN would be
included as part of the DiffConfig/CheckConfig request, which means we
need to flow this value into our Provider interface.
This change does that.
A customer reported an issue where operations would fail with the
following error:
```
error: could not deserialize deployment: unknown secrets provider type
```
The problem here was the customer's deployment had a
`secrets_provider` section which looked like the following:
```
"secrets_providers": {
"type": ""
}
```
And so our code to try to construct a secrets manager from this thing
would fail, as our registry does not contain any information about a
provider with an empty type.
We do two things in this change:
1. When serializing a deployment, if there is no secrets manager,
don't even write the `secrets_provider` block. This helps for cases
where we are roundtripping deployments that did not have a provider
configured (i.e. they were older stacks that did not use secrets)
2. When deserializing, if we see an empty secrets provider like the
above, interpret it to mean "this deployment has no secrets". We set
up a decrypter such that if it ends up haiving secrets, we panic
eagerly (since this is a logical bug in our system somewhere).
Providers from plugins require that configuration value be
strings. This means if we are passing a secret string to a
provider (for example, trying to configure a kubernetes provider based
on some secret kubeconfig) we need to be careful to remove the
"secretness" before actually making the calls into the provider.
Failure to do this resulted in errors saying that the provider
configuration values had to be strings, and of course, the values
logically where, they were just marked as secret strings
Fixes#2741
We have to actually return the value we compute instead of just
dropping it on the floor and treating the underlying values as
primitive.
I ran into this during dogfooding, the added test case would
previously panic.
We adopt a new algoritm for annotating secrets, which works as
follows:
If the source and destinations are both property maps, annotate their
secrets deeply.
Otherwise, if there is an property in both the input and output arrays
with the same name and the value in the inputs has secrets /anywhere/
in it, mark the output itself a secret.
This means, for example, an array in the inputs with a secret value as
one of the elmenets will mean in the outputs the entire array value is
marked as a secret. This is done because arrays often are treated as
sets by providers and so we really shouldn't consider ordering. It
also means that if a value is added to the array as part of the
operation we still mark the new array as an output even though the
values may not be indentical to the inputs.
For providers which do not natively support secrets (which is all of
them today), we annotate output values coming back from the provider
if there is a coresponding secret input in the inputs we passed in.
This logic was not tearing into rich objects, so if you passed a
secret as a member of an array or object into a resource provider, we
would lose the secretness on the way back.
Because of the interaction with Check (where we call Check and then
take the values returned by the provider as inputs for all calls to
Diff/Update), this would apply not only to the Output values of a
resource but also the Inputs (because the secret metadata would not
flow from the inputs of check to the outputs).
This change augments our logic which transfers secrets metadata from
one property map to another to handle these additional cases.
In our system, we model secrets as outputs with an additional bit of
metadata that says they are secret. For Read and Register resource
calls, our RPC interface says if the client side of the interface can
handle secrets being returned (i.e. the language SDK knows how to
sniff for the special signiture and resolve the output with the
special bit set).
For Invoke, we have no such model. Instead, we return a `Promise<T>`
where T's shape has just regular property fields. There's no place
for us to tack the secretness onto, since there are no Outputs.
So, for now, don't even return secret values back across the invoke
channel. We can still take them as arguments (which is good) but we
can't even return secrets as part of invoke calls. This is not ideal,
but given the way we model these sources, there's no way around
this. Fortunately, the result of these invoke calls are not stored in
the checkpoint and since the type is not Output<T> it will be clear
that the underlying value is just present in plaintext. A user that
wants to pass the result of an invoke into a resource can turn an
existing property into a secret via `pulumi.secret`.
We move the implementations of our secrets managers in to
`pkg/secrets` (which is where the base64 one lives) and wire their use
up during deserialization.
It's a little unfortunate that for the passphrase based secrets
manager, we have to require `PULUMI_CONFIG_PASSPHRASE` when
constructing it from state, but we can make more progress with the
changes as they are now, and I think we can come up with some ways to
mitigate this problem a bit (at least make it only a problem for cases
where you are trying to take a stack reference to another stack that
is managed with local encryption).
We have many cases where we want to do the following:
deployment -> snapshot -> process snapshot -> deployment
We now retain information in the snapshot about the secrets manager
that was used to construct it, so in these round trip cases, we can
re-use the existing manager.
When nil, it means no information is retained in the deployment about
the manager (as there is none) and any attempt to persist secret
values fails.
This should only be used in cases where the snapshot is known to not
contain secret values.
Half of the call sites didn't care about these values and with the
secrets work the ergonmics of calling this method when it has to
return serialized ouputs isn't great. Move the serialization for this
into the CLI itself, as it was the only place that cared to do
this (so it could display things to end users).
For cloud backed stacks, this was already returning nil and due to the
fact that we no longer include config in the checkpoint for local
stacks, it was nil there as well.
Removing this helps clean stuff up and is should make some future
refactorings around custom secret managers easier to land.
We can always add it back later if we miss it (and make it actually do
the right thing!)
When constructing a Deployment (which is a plaintext representation of
a Snapshot), ensure that we encrypt secret values. To do so, we
introduce a new type `secrets.Manager` which is able to encrypt and
decrypt values. In addition, it is able to reflect information about
itself that can be stored in the deployment such that we can
deserialize the deployment into a snapshot (decrypting the values in
the process) without external knowledge about how it was encrypted.
The ability to do this is import for allowing stack references to
work, since two stacks may not use the same manager (or they will use
the same type of manager, but have different state).
The state value is stored in plaintext in the deployment, so it **must
not** contain sensitive data.
A sample manager, which just base64 encodes and decodes strings is
provided, as it useful for testing. We will allow it to be varried
soon.
When a provider does not natively understand secrets, we need to pass
inputs as raw values, as to not confuse it.
This leads to a not great experience by default, where we pass raw
values to `Check` and then use the results as the inputs to remaining
operations. This means that by default, we don't end up retaining
information about secrets in the checkpoint, since the call to `Check`
erases all of our information about secrets.
To provide a nicer experience we were don't lose information about
secrets even in cases where providers don't natively understand them,
we take property maps produced by the provider and mark any values in
them that are not listed as secret as secret if the coresponding input
was a secret.
This ensures that any secret property values in the inputs are
reflected back into the outputs, even for providers that don't
understand secrets natively.
A new `Secret` property value is introduced, and plumbed across the
engine.
- When Unmarshalling properties /from/ RPC calls, we instruct the
marshaller to retain secrets, since we now understand them in the
rest of the engine.
- When Marshalling properties /to/ RPC calls, we use or tracked data
to understand if the other side of the connection can accept
secrets. If they can, we marshall them in a similar manner to assets
where we have a special object with a signiture specific for secrets
and an underlying value (which is the /plaintext/ value). In cases
where the other end of the connection does not understand secrets,
we just drop the metadata and marshal the underlying value as we
normally would.
- Any secrets that are passed across the engine events boundary are
presently passed as just `[secret]`.
- When persisting secret values as part of a deployment, we use a rich
object so that we can track the value is a secret, but right now the
underlying value is not actually encrypted.
In previous commits, we have changed the language plugin protocol to
allow the host to communicate that the plugin is meant to boot in "query
mode." In nodejs, this involves not doing things like registering the
default stack resource. This commit will implement this functionality.
This command exposes a new resource `Invoke` operation,
`pulumi:pulumi:readStackResourceOutputs` which retrieves all resource
outputs for some user-specified stack, not including those deleted.
Fixes#2600.
`pulumi query` is designed, essentially, as a souped-up `exec`. We
execute a query program, and add a few convenience constructs (e.g., the
default providers that give you access to things like `getStack`).
Early in the design process, we decided to not re-use the `up`/update
path, both to minimize risk to update operations, and to simplify the
implementation.
This commit will add this "parallel query universe" into the engine
package. In particular, this includes:
* `QuerySource`, which executes the language provider running the query
program, and providing it with some simple constructs, such as the
default provider, which provides access to `getStack`. This is much
like a very simplified `EvalSource`, though notably without any of the
planning/step execution machinery.
* `queryResmon`, which disallows all resource operations, except the
`Invoke` that retrieves the resource outputs of some stack's last
snapshot. This is much like a simplified `resmon`, but without any of
the provider resolution, and without and support for resource
operations generally.
* Various static functions that pull together miscellaneous things
needed to execute a query program. Notably, this includes gathering
language plugins.
Fixes#2277.
Adds a new ignoreChanges resource option that allows specifying a list of property names whose values will be ignored during updates. The property values will be used for Create, but will be ignored for purposes of updates, and as a result also cannot trigger replacements.
This is a feature of the Pulumi engine, not of the resource providers, so no new logic is needed in providers to support this feature. Instead, the engine simply replaces the values of input properties in the goal state with old inputs for properties marked as ignoreChanges.
Currently, only top level properties may be specified in ignoreChanges. In the future, this could be extended to support paths to nested properties (including into array elements) with a JSONPath/JMESPath syntax.
Fixes#2650.
We have historically relied on merging inputs and outputs in several places in the engine. This used to be necessary, as discussed in #2650 (comment), but our core engine model has moved away from depending on this. However, we still have a couple places we do this merge, and those places have triggered several severe issues recently in subtle cases.
We believe that this merging should no longer be needed for a correct interpretation of the current engine model, and indeed that doing the merge actively violates the contract with providers. In this PR we remove the remaining places where this input + output merge was being done. In all three cases, we use just the Outputs, which for most providers will already include the same values as the inputs - but correctly as determined by the provider itself.
* Load specific provider versions if requested
As part of pulumi/pulumi#2389, we need the ability for language hosts to
tell the engine that a particular resource registration, read, or invoke
needs to use a particular version of a resource provider. This was not
previously possible before; the engine prior to this commit loaded
plugins from a default provider map, which was inferred for every
resource provider based on the contents of a user's package.json, and
was itself prone to bugs.
This PR adds the engine support needed for language hosts to request a
particular version of a provider. If this occurs, the source evaluator
specifically records the intent to load a provider with a given version
and produces a "default" provider registration that requests exactly
that version. This allows the source evaluator to produce multiple
default providers for a signle package, which was previously not
possible.
This is accomplished by having the source evaluator deal in the
"ProviderRequest" type, which is a tuple of version and package. A
request to load a provider whose version matches the package of a
previously loaded provider will re-use the existing default provider. If
the version was not previously loaded, a new default provider is
injected.
* CR Feedback: raise error if semver is invalid
* CR: call String() if you want a hash key
* Update pkg/resource/deploy/providers/provider.go
Co-Authored-By: swgillespie <sean@pulumi.com>
Fixes#2633.
Currently when a user runs `refresh` and a resource is in a state of
error, the `refresh` will fail and the resource state will not be
persisted. This can make it vastly harder to incrementally fix
infrastructure. The issue mentioned above explains more of the
historical context, as well as some specific failure modes.
This commit resolves this issue by causing refresh to *not* report an
error in this case, and instead to simply log a warning that the
`refresh` has recognized that the resource is in an unhealthy state
during state sync.
This commit switches from dep to Go 1.12 modules for tracking Pulumi
dependencies. Rather than _building_ using Go modules, we instead use the `go
mod vendor` command to populate a vendor tree in the same way as `dep ensure`
was previously doing.
In order to prevent checksum mismatches, it was necessary to also update CI to
use Go 1.12 instead of 1.11 - which also necessitated fixing some linting errors
which appeared with the upgraded golangci-lint for 1.12.
Our logic for how we handled `.tar.gz` archives meant that any other
type of file that had a dot in the filename would not be detected
correctly.
Fixes#2589
* Install missing plugins on startup
This commit addresses the problem of missing plugins by scanning the
snapshot and language host on startup for the list of required plugins
and, if there are any plugins that are required but not installed,
installs them. The mechanism by which plugins are installed is exactly
the same as 'pulumi plugin install'.
The installation of missing plugins is best-effort and, if it fails,
will not fail the update.
This commit addresses pulumi/pulumi-azure#200, where users using Pulumi
in CI often found themselves missing plugins.
* Add CHANGELOG
* Skip downloading plugins if no client provided
* Reduce excessive test output
* Update Gopkg.lock
* Update pkg/engine/destroy.go
Co-Authored-By: swgillespie <sean@pulumi.com>
* CR: make pluginSet a newtype
* CR: Assign loop induction var to local var
Various providers use properties that begin with "__" to represent
internal metadata that should not be presented to the user. These
changes look for such properties and elide them when displaying diffs.
This commit re-uses an error reporting mechanism previously used when
the plugin loader fails to locate a plugin that is compatible with the
requested plugin version. In addition to specifying what version we
attempted to load, it also outputs a command that will install the
missing plugin.
`edit.RenameStack` walks a Snapshot and rewrites all of the parts
where a stack name is present (URNs, the ID of the top level Stack
resource, providers)
These changes take advantage of the newly-added support for returning
inputs from Read to update a resource's inputs as part of a refresh.
As a consequence, the Pulumi engine will now properly detect drift
between the actual state of a resource and the desired state described
by the program and generate appropriate update or replace steps.
As part of these changes, a resource's old inputs are now passed to the
provider when performing a refresh. The provider can take advantage of
this to maintain the accuracy of any additional data or metadata in the
resource's inputs that may need to be updated during the refresh.
This is required for the complete implementation of
https://github.com/pulumi/pulumi-terraform/pull/349. Without access to
the old inputs for a resource, TF-based providers would lose all
information about default population during a refresh.
If a provider returns information about the top-level properties that
differ, use those keys to filter the diffs that are rendered to the
user.
Fixes#2453.
We previously logged the number of replaces and changes returned from a call to Diff, but not the actual properties that were forcing replace. Several times we've had to debug issues with unexpected replaces being proposed, and this information is very useful to have access to.
Changes the verbose logging to include the property names for both replaces and changes instead of just the count.
These changes add a new flag to the various `ResourceOptions` types that
indicates that a resource should be deleted before it is replaced, even
if the provider does not require this behavior. The usual
delete-before-replace cascade semantics apply.
Fixes#1620.
In general, a "delete" in Pulumi is destroying an actual physical
resource. In the case of a read resource, however, the delete is
merely removing the resource from the stack; the physical resource
is not affected. These changes attempt to clarify this situation by
using the term "discard" rather than "delete".
Fixes#2015.
- Add support for per-property dependencies to the Go SDK
- Add tests for first-class secret rejection in the checkpoint and RPC
layers and language SDKs
This implements the new algorithm for deciding which resources must be
deleted due to a delete-before-replace operation.
We need to compute the set of resources that may be replaced by a
change to the resource under consideration. We do this by taking the
complete set of transitive dependents on the resource under
consideration and removing any resources that would not be replaced by
changes to their dependencies. We determine whether or not a resource
may be replaced by substituting unknowns for input properties that may
change due to deletion of the resources their value depends on and
calling the resource provider's Diff method.
This is perhaps clearer when described by example. Consider the
following dependency graph:
A
__|__
B C
| _|_
D E F
In this graph, all of B, C, D, E, and F transitively depend on A. It may
be the case, however, that changes to the specific properties of any of
those resources R that would occur if a resource on the path to A were
deleted and recreated may not cause R to be replaced. For example, the
edge from B to A may be a simple dependsOn edge such that a change to
B does not actually influence any of B's input properties. In that case,
neither B nor D would need to be deleted before A could be deleted.
In order to make the above algorithm a reality, the resource monitor
interface has been updated to include a map that associates an input
property key with the list of resources that input property depends on.
Older clients of the resource monitor will leave this map empty, in
which case all input properties will be treated as depending on all
dependencies of the resource. This is probably overly conservative, but
it is less conservative than what we currently implement, and is
certainly correct.
Resources gain two new fields: `PropertyDependencies` and
`PendingReplacement`. The former maps an input property's name to the
dependencies that may affect the value of that property. The latter is
used to track resources that have been deleted as part of a
delete-before-replace operation but have not yet been recreated.
In addition to the new fields, resource properties may now contain
encrypted first-class secret values. These values are of type `SecretV1`,
where the `Sig` field is set to `resource.SecretSig`.
Finally, the deployment type gains a new field, `SecretsProviders`,
which contains any configuration necessary to handle secrets that may be
present in resource properties.
When constructing an Archive based off a directory path, we would
ignore any symlinks that we saw while walking the file system
collecting files to include in the archive.
A user reported an issue where they were unable to use the
[sharp](https://www.npmjs.com/package/sharp) library from NPM with a
lambda deployed via Pulumi. The problem was that the library includes
native components, and these native components include a bunch of
`*.so` files. As is common, there's a regular file with a name like
`foo.so.1.2.3` and then symlinks to that file with the names
`foo.so.1.2`, `foo.so.1` and `foo.so`. Consumers of these SOs will
try to load the shorter names, i.e. `foo.so` and expect the symlink to
resolve to the actual library.
When these links are not present, upstack code fails to load.
This changes modifies our logic such that if we have a symlink and it
points to a regular file, we include it in the archive. At this time,
we don't try to add it to the archive as a symlink, instead we just
add it as another copy of the regular file. We could explore trying to
include these things as symlinks in archive formats that allow
them (While zip does support this, I'm less sure doing this for zip
files will be a great idea, given the set of tricks we already play to
ensure our zip files are usable across many cloud vendors serverless
offerings, it feels like throwing symlinks into the mix may end up
causing even more downstream weirdness).
We continue to ignore symlinks which point at directories. In
practice, these seem fairly uncommon and doing so lets us not worry
about trying to deal with ensuring we don't end up chasing our tail in
cases where there are circular references.
While this change is in pulumi/pulumi, the downstream resource
providers will need to update their vendored dependencies in order to
pick this up and have it work end to end.
Fixes#2077
This ensures that the gRPC server is properly shut down. This fixes an
issue in which a resource plugin that is still configuring could report
log messages to the plugin host, which would in turn attempt to send
diagnostic packets over a closed channel, causing a panic.
Fixes#2170.
After #2088, we began calling `Diff` on providers that are not configured
due to unknown configuration values. This hit an assertion intended to
detect exactly this scenario, which was previously unexpected.
These changes adjust `Diff` to indicate that a Diff is unavailable and
return an error message that describes why. The step generator then
interprets the diff as indicating a normal update and issues the error
message to the diagnostic stream.
Fixes#2223.
Configuration keys are simple namespace/name pairs, delimited by
":". For compatability, we also allow
"<namespace>:config:<name>", but we always record the "nice" name in
`Pulumi.<stack-name>.yaml`.
While `pulumi config` and friends would block setting a key like
`a🅱️c` (where the "name" has a colon in it), it would allow
`a:config:b:c`. However, this would be recorded as `a🅱️c` in
`Pulumi.<stack-name>.yaml`, which meant we'd error when parsing the
configuration file later.
To work around this, disallow ":" in the "name" part of a
configuration key. With this change the following all work:
```
keyName
my-project:keyName
my-project:config:keyName
```
However, both
`my-project:keyName:subKey`
`my-project:config:keyName:subKey`
are now disallowed.
I considered allowing colons in subkeys, but I think it adds more
confusion (due to the interaction with how we allow you elide the
project name in the default case) than is worthwhile at this point.
Fixes#2171
These changes add a new resource to the Pulumi SDK,
`pulumi.StackReference`, that represents a reference to another stack.
This resource has an output property, `outputs`, that contains the
complete set of outputs for the referenced stack. The Pulumi account
performing the deployment that creates a `StackReference` must have
access to the referenced stack or the call will fail.
This resource is implemented by a builtin provider managed by the engine.
This provider will be used for any custom resources and invokes inside
the `pulumi:pulumi` module. Currently this provider supports only the
`pulumi:pulumi:StackReference` resource.
Fixes#109.
We run the same suite of changes that we did on gometalinter. This
ended up catching a few new issues, some of which were addressed and
some of which were baselined.
In preparation for some workspace restructuring, I decided to scratch a
few itches of my own in the code:
* Change project's RuntimeInfo field to just Runtime, to match the
serialized name in JSON/YAML.
* Eliminate the no-longer-used Context and NoDefaultIgnores fields on
project, and all of the associated legacy PPC-related code.
* Eliminate the no-longer-used IgnoreFile constant.
* Remove a bunch of "// nolint: lll" annotations, and simply format
the structures with comments on dedicated lines, to avoid overly
lengthy lines and lint suppressions.
* Mark Dependencies and InitErrors as `omitempty` in the JSON
serialization directives for CheckpointV2 files. This was done for
the YAML directives, but (presumably accidentally) omitted for JSON.
Go 1.10 made some breaking changes to the headers in archive/tar [1] and
archive/zip [2], breaking the expected values in tests. In order to keep
tests passing with both, wherever a hardcoded hash is expect we switch
on `runtime.Version()` to select whether we want the Go 1.9 (currently
supported Go version) or later version of the hash.
Eventually these switches should be removed in favour of using the later
version only, so they are liberally commented to explain the reasoning.
[1]: https://golang.org/doc/go1.10#archive/tar
[2]: https://golang.org/doc/go1.10#archive/zip
Downlevel versions of the Pulumi Node SDK assumed that a parallelism
level of zero implied serial execution, which current CLIs use to signal
unbounded parallelism. This commit works around the downlevel issue by
using math.MaxInt32 to signal unbounded parallelism.
Providers with unknown properties are currently considered to require
replacement. This was intended to indicate that we could not be sure
whether or not replacement was reqiuired. Unfortunately, this was not a
good user experience, as replacement would never be required at runtime.
This caused quite a bit of confusion--never proposing replacement seems
to be the better option.
The provider registry was checking for a `nil` provider instance before
checking for a non-nil error. This caused the CLI to fail to report
important errors during the plugin load process (e.g. invalid checkpoint
errors) and instead report a failure to find a matching plugin.
Some providers (namely Kubernetes) require unbounded parallelism in
order to function correctly. This commit enables the engine to operate
in a mode with unbounded parallelism and switches to that mode by
default.
* Add new 'pulumi state' command for editing state
This commit adds 'pulumi state unprotect' and 'pulumi state delete', two
commands that can be used to unprotect and delete resources from a
stack's state, respectively.
* Simplify LocateResource
* CR: Print yellow 'warning' before editing state
* Lots of CR feedback
* CR: Only delete protected resources when asked with --force
The preview will proceed as if the operations had not been issued (i.e.
we will not speculate on a new state for the stack). This is consistent
with our behavior prior to the changes that added pending operations to
the checkpoint.
* Process deletions conservatively in parallel
This commit allows the engine to conservatively delete resources in
parallel when it is sure that it is legal to do so. In the absence of a
true data-flow oriented step scheduler, this approach provides a
significant improvement over the existing serial deletion mechanism.
Instead of processing deletes serially, this commit will partition the
set of condemned resources into sets of resources that are known to be
legally deletable in parallel. The step executor will then execute those
independent lists of steps one-by-one until all steps are complete.
* CR: Make ResourceSet a normal map
* Only use the dependency graph if we can trust it
* Reverse polarity of pendingDeletesAreReplaces
* CR: un-export a few types
* CR: simplify control flow in step generator when scheduling
* CR: parents are dependencies, fix loop index
* CR: Remove ParentOf, add new test for parent dependencies
Since I was digging around over the weekend after the change to move
away from light black, and the impact it had on less important
information showing more prominently than it used to, I took a step
back and did a deeper tidying up of things. Another side goal of this
exercise was to be a little more respectful of terminal width; when
we could say things with fewer words, I did so.
* Stylize the preview/update summary differently, so that it stands
out as a section. Also highlight the total changes with bold -- it
turns out this has a similar effect to the bright white colorization,
just without the negative effects on e.g. white terminals.
* Eliminate some verbosity in the phrasing of change summaries.
* Make all heading sections stylized consistently. This includes
the color (bright magenta) and the vertical spacing (always a newline
separating headings). We were previously inconsistent on this (e.g.,
outputs were under "---outputs---"). Now the headings are:
Previewing (etc), Diagnostics, Outputs, Resources, Duration, and Permalink.
* Fix an issue where we'd parent things to "global" until the stack
object later showed up. Now we'll simply mock up a stack resource.
* Don't show messages like "no change" or "unchanged". Prior to the
light black removal, these faded into the background of the terminal.
Now they just clutter up the display. Similar to the elision of "*"
for OpSames in a prior commit, just leave these out. Now anything
that's written is actually a meaningful status for the user to note.
* Don't show the "3 info messages," etc. summaries in the Info column
while an update is ongoing. Instead, just show the latest line. This
is more respectful of width -- I often find that the important
messages scroll off the right of my screen before this change.
For discussion:
- I actually wonder if we should eliminate the summary
altogether and always just show the latest line. Or even
blank it out. The summary feels better suited for the
Diagnostics section, and the Status concisely tells us
how a resource's update ended up (failed, succeeded, etc).
- Similarly, I question the idea of showing only the "worst"
message. I'd vote for always showing the latest, and again
leaving it to the Status column for concisely telling the
user about the final state a resource ended up in.
* Stop prepending "info: " to every stdout/stderr message. It adds
no value, clutters up the display, and worsens horizontal usage.
* Lessen the verbosity of update headline messages, so we now instead
of e.g. "Previewing update of stack 'x':", we just say
"Previewing update (x):".
* Eliminate vertical whitespace in the Diagnostics section. Every
independent console.out previously was separated by an entire newline,
which made the section look cluttered to my eyes. These are just
streams of logs, there's no reason for the extra newlines.
* Colorize the resource headers in the Diagnostic section light blue.
Note that this will change various test baselines, which I will
update next. I didn't want those in the same commit.
Recently, we eliminated bright black text, which IMHO makes the
"same" lines really stand out more than we want them to. This is
partly just due to the heavyweight nature of the "*" character,
which we precede every line with. This has the effect of making it
toughter to scan the update to see what's going to happen. The goal
of SpecUnimportant (bright black) was that we wanted to draw less
attention to certain elements of the CLI text -- and have them fade
into the background (apparently it was too successful at this ;-))
So, this change eliminates the "*" prefix for same operations
altogether. It reads better to my eyes and keeps the original intent.
- Attempting to read an archive with an unknown type now returns an
error
- Attempting to read a path archive that is neither a known archive
format or a directory now returns an error
Fixes#1529.
Fixes#1953.
