Thse changes make a subtle but critical adjustment to the process the
Pulumi engine uses to determine whether or not a difference exists
between a resource's actual and desired states, and adjusts the way this
difference is calculated and displayed accordingly.
Today, the Pulumi engine get the first chance to decide whether or not
there is a difference between a resource's actual and desired states. It
does this by comparing the current set of inputs for a resource (i.e.
the inputs from the running Pulumi program) with the last set of inputs
used to update the resource. If there is no difference between the old
and new inputs, the engine decides that no change is necessary without
consulting the resource's provider. Only if there are changes does the
engine consult the resource's provider for more information about the
difference. This can be problematic for a number of reasons:
- Not all providers do input-input comparison; some do input-state
comparison
- Not all providers are able to update the last deployed set of inputs
when performing a refresh
- Some providers--either intentionally or due to bugs--may see changes
in resources whose inputs have not changed
All of these situations are confusing at the very least, and the first
is problematic with respect to correctness. Furthermore, the display
code only renders diffs it observes rather than rendering the diffs
observed by the provider, which can obscure the actual changes detected
at runtime.
These changes address both of these issues:
- Rather than comparing the current inputs against the last inputs
before calling a resource provider's Diff function, the engine calls
the Diff function in all cases.
- Providers may now return a list of properties that differ between the
requested and actual state and the way in which they differ. This
information will then be used by the CLI to render the diff
appropriately. A provider may also indicate that a particular diff is
between old and new inputs rather than old state and new inputs.
Fixes#2453.
@keen99 pointed out that newer versions of golangci-lint were failing
due to some spelling errors. This change fixes them up. We have also
now have a work item to track moving to a newer golangci-lint tool in
the future.
Fixes#2841
Currently if you log into s3://bucket/subdirectory, Pulumi will write
files to s3://bucket/.pulumi and not s3://bucket/subdirectory/.pulumi,
this corrects the error.
Recent changes to default provider semantics and the addition of
resource aliases allow a resource's provider reference to change even if
the resource itself is considered to have no diffs. `mustWrite` did not
expect this scenario, and indeed asserted against it. These changes
update `mustWrite` to detect such changes and require that the
checkpoint be written if and when they occur.
Fixes#2804.
This commit will expose the new `Invoke` routine that lists resource
outputs through the Node.js SDK.
This API is implemented via a new API, `EnumerablePromise`, which is a
collection of simple query primitives built onto the `Promise` API. The
query model is lazy and LINQ-like, and generally intended to make
`Promise` simpler to deal with in query scenarios. See #2601 for more
details.
Fixes#2600.
This commit touches an intersection of a few different provider-oriented
features that combined to cause a particularly severe bug that made it
impossible for users to upgrade provider versions without seeing
replacements with their resources.
For some context, Pulumi models all providers as resources and places
them in the snapshot like any other resource. Every resource has a
reference to the provider that created it. If a Pulumi program does not
specify a particular provider to use when performing a resource
operation, the Pulumi engine injects one automatically; these are called
"default providers" and are the most common ways that users end up with
providers in their snapshot. Default providers can be identified by
their name, which is always prefixed with "default".
Recently, in an effort to make the Pulumi engine more flexible with
provider versions, it was made possible for the engine to have multiple
default providers active for a provider of a particular type, which was
previously not possible. Because a provider is identified as a tuple of
package name and version, it was difficult to find a name for these
duplicate default providers that did not cause additional problems. The
provider versioning PR gave these default providers a name that was
derived from the version of the package. This proved to be a problem,
because when users upgraded from one version of a package to another,
this changed the name of their default provider which in turn caused all
of their resources created using that provider (read: everything) to be
replaced.
To combat this, this PR introduces a rule that the engine will apply
when diffing a resource to determine whether or not it needs to be
replaced: "If a resource's provider changes, and both old and new
providers are default providers whose properties do not require
replacement, proceed as if there were no diff." This allows the engine
to gracefully recognize and recover when a resource's default provider changes
names, as long as the provider's config has not changed.
Previously, when the CLI wanted to install a plugin, it used a special
method, `DownloadPlugin` on the `httpstate` backend to actually fetch
the tarball that had the plugin. The reason for this is largely tied
to history, at one point during a closed beta, we required presenting
an API key to download plugins (as a way to enforce folks outside the
beta could not download them) and because of that it was natural to
bake that functionality into the part of the code that interfaced with
the rest of the API from the Pulumi Service.
The downside here is that it means we need to host all the plugins on
`api.pulumi.com` which prevents community folks from being able to
easily write resource providers, since they have to manually manage
the process of downloading a provider to a machine and getting it on
the `$PATH` or putting it in the plugin cache.
To make this easier, we add a `--server` argument you can pass to
`pulumi plugin install` to control the URL that it attempts to fetch
the tarball from. We still have perscriptive guidence on how the
tarball must be
named (`pulumi-[<type>]-[<provider-name>]-vX.Y.Z.tar.gz`) but the base
URL can now be configured.
Folks publishing packages can use install scripts to run `pulumi
plugin install` passing a custom `--server` argument, if needed.
There are two improvements we can make to provide a nicer end to end
story here:
- We can augment the GetRequiredPlugins method on the language
provider to also return information about an optional server to use
when downloading the provider.
- We can pass information about a server to download plugins from as
part of a resource registration or creation of a first class
provider.
These help out in cases where for one reason or another where `pulumi
plugin install` doesn't get run before an update takes place and would
allow us to either do the right thing ahead of time or provide better
error messages with the correct `--server` argument. But, for now,
this unblocks a majority of the cases we care about and provides a
path forward for folks that want to develop and host their own
resource providers.
Adds a new resource option `aliases` which can be used to rename a resource. When making a breaking change to the name or type of a resource or component, the old name can be added to the list of `aliases` for a resource to ensure that existing resources will be migrated to the new name instead of being deleted and replaced with the new named resource.
There are two key places this change is implemented.
The first is the step generator in the engine. When computing whether there is an old version of a registered resource, we now take into account the aliases specified on the registered resource. That is, we first look up the resource by its new URN in the old state, and then by any aliases provided (in order). This can allow the resource to be matched as a (potential) update to an existing resource with a different URN.
The second is the core `Resource` constructor in the JavaScript (and soon Python) SDKs. This change ensures that when a parent resource is aliased, that all children implicitly inherit corresponding aliases. It is similar to how many other resource options are "inherited" implicitly from the parent.
Four specific scenarios are explicitly tested as part of this PR:
1. Renaming a resource
2. Adopting a resource into a component (as the owner of both component and consumption codebases)
3. Renaming a component instance (as the owner of the consumption codebase without changes to the component)
4. Changing the type of a component (as the owner of the component codebase without changes to the consumption codebase)
4. Combining (1) and (3) to make both changes to a resource at the same time
We model providers as resources in our state file, but we were
neglecting to set Outputs for these resources. This was problematic
when we started to try to run DiffConfig, because when diffing a
resource we compare thed new inputs and the old outputs, but the
resource never had any old outputs, so it was impossible for the
provider to see what the old state of the resource was.
To fix this, we now reflect the inputs we use the create the provider
reference as outputs on the resource.
* Introduce a new package under ciutil for individual CI systems. Split-out each CI system with env var detection for each.
* Add Bitbucket Piplines env var detection.
* Update changelog with note about adding Bitbucket Pipelines detection.
* Rename the CI system structs.
* Move files from ciutil/systems to ciutil. Un-export some types that don't need visibility beyond the ciutil package.
* Un-export DetectSystem function and the System type.
* Add a test for CI systems which we only know by name and nothing else, i.e. those with just a baseCI implementation.
The Kubernetes provider wanted to return Unimplemented for both
DiffConfig and CheckConfig. However, due to an interaction between the
package we used to construct the error we are returning and the
package we are using to actually construct the gRPC server for the
provider, we ended up in a place where the provider would actually end
up returning an error with code "Unknown", and the /text/ of the
message included information about it being due to the RPC not being
implemented.
So, when we try to call Diff/Check config on the provider, detect this
case as well and treat messages of this shape as if the provider just
returned "Unimplemented".
If --suppress-outputs is passed to `pulumi preview --json`, we
should not emit the stack outputs. This change fixespulumi/pulumi#2765.
Also adds a test case for this plus some variants of updates.
In 3621c01f4b, we implemented
CheckConfig/DiffConfig incorrectly. We should have explicilty added
the handlers (to supress the warnings we were getting) but returned an
error saying the RPC was not implemented. Instead, we just returned
success but passed back bogus data. This was "fine" at the time
because nothing called these methods.
Now that we are actually calling them, returning incorrect values
leads to errors in grpc. To deal with this we do two things:
1. Adjust the implementations in the dynamic provider to correctly
return not implemented. This allows us to pick up the default engine
behavior going forward.
2. Add some code in CheckConfig/DiffConfig that handle the gRPC error
that is returned when calling methods on the dynamic provider and fall
back to the legacy behavior. This means updating your CLI will not
cause issues for existing resources where the SDK has not been
updated.
For provider plugins, the gRPC interfaces expect that a URN would be
included as part of the DiffConfig/CheckConfig request, which means we
need to flow this value into our Provider interface.
This change does that.
A customer reported an issue where operations would fail with the
following error:
```
error: could not deserialize deployment: unknown secrets provider type
```
The problem here was the customer's deployment had a
`secrets_provider` section which looked like the following:
```
"secrets_providers": {
"type": ""
}
```
And so our code to try to construct a secrets manager from this thing
would fail, as our registry does not contain any information about a
provider with an empty type.
We do two things in this change:
1. When serializing a deployment, if there is no secrets manager,
don't even write the `secrets_provider` block. This helps for cases
where we are roundtripping deployments that did not have a provider
configured (i.e. they were older stacks that did not use secrets)
2. When deserializing, if we see an empty secrets provider like the
above, interpret it to mean "this deployment has no secrets". We set
up a decrypter such that if it ends up haiving secrets, we panic
eagerly (since this is a logical bug in our system somewhere).
We were not actually calling our colorization routines, which lead to
printing this very confusing text:
```
<{%reset%}> --outputs:--<{%reset%}>
```
When running updates with `--diff` or when drilling into details of a
proposed operation, like a refresh.
Providers from plugins require that configuration value be
strings. This means if we are passing a secret string to a
provider (for example, trying to configure a kubernetes provider based
on some secret kubeconfig) we need to be careful to remove the
"secretness" before actually making the calls into the provider.
Failure to do this resulted in errors saying that the provider
configuration values had to be strings, and of course, the values
logically where, they were just marked as secret strings
Fixes#2741
We have to actually return the value we compute instead of just
dropping it on the floor and treating the underlying values as
primitive.
I ran into this during dogfooding, the added test case would
previously panic.
We adopt a new algoritm for annotating secrets, which works as
follows:
If the source and destinations are both property maps, annotate their
secrets deeply.
Otherwise, if there is an property in both the input and output arrays
with the same name and the value in the inputs has secrets /anywhere/
in it, mark the output itself a secret.
This means, for example, an array in the inputs with a secret value as
one of the elmenets will mean in the outputs the entire array value is
marked as a secret. This is done because arrays often are treated as
sets by providers and so we really shouldn't consider ordering. It
also means that if a value is added to the array as part of the
operation we still mark the new array as an output even though the
values may not be indentical to the inputs.
For providers which do not natively support secrets (which is all of
them today), we annotate output values coming back from the provider
if there is a coresponding secret input in the inputs we passed in.
This logic was not tearing into rich objects, so if you passed a
secret as a member of an array or object into a resource provider, we
would lose the secretness on the way back.
Because of the interaction with Check (where we call Check and then
take the values returned by the provider as inputs for all calls to
Diff/Update), this would apply not only to the Output values of a
resource but also the Inputs (because the secret metadata would not
flow from the inputs of check to the outputs).
This change augments our logic which transfers secrets metadata from
one property map to another to handle these additional cases.
The change does two things:
- Reorders some calls in the CLI to prevent trying to create a secrets
manager twice (which would end up prompting twice).
- Adds a cache inside the passphrase secrets manager such that when
decrypting a deployment, we can re-use the one created earlier in
the update. This is sort of a hack, but is needed because otherwise
we would fail to decrypt the deployment, meaning that if you had a
secret value in your deployment *and* you were using local
passphrase encryption *and* you had not set PULUMI_CONFIG_PASSPHRASE
you would get an error asking you to do so.
Fixes#2729
This is helpful some round trip cases where we many not be able to
build the encrypter or decrypter but we will end up not needing
them. When we fail to load the manager, we return a manager that has
the correct state, but will error when it tries to preform any
operations. However, if there are no secrets in the deployment, these
methods will never be called and we'll be able to correctly roundtrip
checkpoints even without having access to the password (since there
were no secret values to decrypt or encrypt).
We were dropping new and old states on the floor instead of including
them as part of the previewed operation due to a logic error (we want
to append them when there are no errors from serialization, vs when
there are errors).
When creating a new stack using the local backend, the default
checkpoint has no deployment. That means there's a nil snapshot
created, which means our strategy of using the base snapshot's secrets
manager was not going to work. Trying to do so would result in a panic
because the baseSnapshot is nil in this case.
Using the secrets manager we are going to use to persist the snapshot
is a better idea anyhow, as that's what's actually going to be burned
into the deployment when we serialize the snapshot, so let's use that
instead.
