This change allows using the passphrase secrets manager when creating
a stack managed by the Pulumi service. `pulumi stack init`, `pulumi
new` and `pulumi up` all learned a new optional argument
`--secrets-provider` which can be set to "passphrase" to force the
passphrase based secrets provider to be used. When unset the default
secrets provider is used based on the backend (for local stacks this
is passphrase, for remote stacks, it is the key managed by the pulumi
service).
As part of this change, we also initialize the secrets manager when a
stack is created, instead of waiting for the first time a secret
config value is stored. We do this so that if an update is run using
`pulumi.secret` before any secret configuration values are used, we
already have the correct encryption method selected for a stack.
Logs are no longer provided by the service (this is a holdover from
the PPC days where service deployments where done in the cloud and it
handled collecting logs).
Removing this breaks another cycle that would be introduced with the
next change (in our test code)
The next change is going to do some code motion that would create some
circular imports if we did not do this. There was nothing that
required the members we were moving be in the backend package, so it
was easy enough to pull them out.
When preforming an update, require that a secrets manager is passed in
as part of the `backend.UpdateOperation` bag and use it. The CLI now
passes this in (it still uses the default base64 secrets manager, so
this is just code motion into a high layer, since the CLI will be the
one to choose what secrets manager to use based on project settings).
There are a few operations we do (stack rename, importing and edits)
where we will materialize a `deploy.Snapshot` from an existing
deployment, mutate it in somewhay, and then store it.
In these cases, we will just re-use the secrets manager that was used
to build the snapshot when we re-serialize it. This is less than ideal
in some cases, because many of these operations could run on an
"encrypted" copy of the Snapshot, where Inputs and Outputs have not
been decrypted.
Unfortunately, our system now is not set up in a great way to support
this and adding something like a `deploy.EncryptedSnapshot` would
require large scale code duplications.
So, for now, we'll take the hit of decrypting and re-encrypting, but
long term introducing a `deploy.EncryptedSnapshot` may be nice as it
would let us elide the encryption/decryption steps in some places and
would also make it clear what parts of our system have access to the
plaintext values of secrets.
We have many cases where we want to do the following:
deployment -> snapshot -> process snapshot -> deployment
We now retain information in the snapshot about the secrets manager
that was used to construct it, so in these round trip cases, we can
re-use the existing manager.
When nil, it means no information is retained in the deployment about
the manager (as there is none) and any attempt to persist secret
values fails.
This should only be used in cases where the snapshot is known to not
contain secret values.
Half of the call sites didn't care about these values and with the
secrets work the ergonmics of calling this method when it has to
return serialized ouputs isn't great. Move the serialization for this
into the CLI itself, as it was the only place that cared to do
this (so it could display things to end users).
The previous changes to remove config loading out of the backend means
that the backends no longer need to track this information, as they
never use it.
As part of the pluggable secrets work, the crypter's used for secrets
are no longer tied to a backend. To enforce this, we remove the
`backend.GetStackCrypter` function and then have the relevent logic to
construct one live inside the CLI itself.
Right now the CLI still uses the backend type to decide what Crypter
to build, but we'll change that shortly.
We require configuration to preform updates (as well as previews,
destroys and refreshes). Because of how everything evolved, loading
this configuration (and finding the coresponding decrypter) was
implemented in both the file and http backends, which wasn't great.
Refactor things such that the CLI itself builds out this information
and passes it along to the backend to preform operations. This means
less code duplicated between backends and less places the backend
assume things about the existence of `Pulumi.yaml` files and in
general makes the interface more plesent to use for others uses.
For cloud backed stacks, this was already returning nil and due to the
fact that we no longer include config in the checkpoint for local
stacks, it was nil there as well.
Removing this helps clean stuff up and is should make some future
refactorings around custom secret managers easier to land.
We can always add it back later if we miss it (and make it actually do
the right thing!)
When constructing a Deployment (which is a plaintext representation of
a Snapshot), ensure that we encrypt secret values. To do so, we
introduce a new type `secrets.Manager` which is able to encrypt and
decrypt values. In addition, it is able to reflect information about
itself that can be stored in the deployment such that we can
deserialize the deployment into a snapshot (decrypting the values in
the process) without external knowledge about how it was encrypted.
The ability to do this is import for allowing stack references to
work, since two stacks may not use the same manager (or they will use
the same type of manager, but have different state).
The state value is stored in plaintext in the deployment, so it **must
not** contain sensitive data.
A sample manager, which just base64 encodes and decodes strings is
provided, as it useful for testing. We will allow it to be varried
soon.
* Add a var for PRNumber. Add an environment metadata key for PR number.
* Move the detection of PULUMI_CI_SYSTEM into vars.DetectVars(). Set the PRNumber CI property based on respective env vars from each CI system.
* Add Azure Pipelines build variables.
* Add tests for DetectVars.
* Added changelog entry for Azure Pipelines.
* Capture the value of env var being modified for the ciutil unit test, and restore their values at the end of them.
* Simplify the DetectVars function by moving the Pulumi CI system code into the switch-case expression.
* Rename the Pulumi CI system to Generic CI. Include the GenericCI system in the test case for DetectVars.
This command exposes a new resource `Invoke` operation,
`pulumi:pulumi:readStackResourceOutputs` which retrieves all resource
outputs for some user-specified stack, not including those deleted.
Fixes#2600.
Because `pulumi query` is not implemented with the update
infrastructure, it is important that we *not* do things like open an
update when the query program runs.
This commit will thus implement the "query" path in the state backend in
a completely parallel universe. Conceptually, this is much like the
update path, but with a conspicuous lack of any connection to the
backend service.
The Pulumi CLI currently has two "display modes" -- one for rendering
diffs and one for rendering program updates. Neither of these is
particularly well-suited to `pulumi query`, which essentially needs to
render stdout from the query program verbatim.
This commit will add a separate display mode for this purpose:
* In interactive mode, `pulumi query` will use the display spinner to
say "running query". In non-interactive mode, this will be omitted.
* Query mode will print stdout from the query program verbatim, but
ignore `diag.Infoerr` so that they're not double-printed when they're
emitted again as error events.
* Query mode also does not double-print newlines at the end of diag
events.
This change adds a --json flag to the preview command, enabling
basic JSON serialization of preview plans. This effectively flattens
the engine event stream into a preview structure that contains a list
of steps, diagnostics, and summary information. Each step contains
the deep serialization of resource state, in addition to metadata about
the step, such as what kind of operation it entails.
This is a partial implementation of pulumi/pulumi#2390. In particular,
we only support --json on the `preview` command itself, and not `up`,
meaning that it isn't possible to serialize the result of an actual
deployment yet (thereby limiting what you can do with outputs, etc).
This change adds a --json flag to the preview command, enabling
basic JSON serialization of preview plans. This effectively flattens
the engine event stream into a preview structure that contains a list
of steps, diagnostics, and summary information. Each step contains
the deep serialization of resource state, in addition to metadata about
the step, such as what kind of operation it entails.
This is a partial implementation of pulumi/pulumi#2390. In particular,
we only support --json on the `preview` command itself, and not `up`,
meaning that it isn't possible to serialize the result of an actual
deployment yet (thereby limiting what you can do with outputs, etc).
This commit switches from dep to Go 1.12 modules for tracking Pulumi
dependencies. Rather than _building_ using Go modules, we instead use the `go
mod vendor` command to populate a vendor tree in the same way as `dep ensure`
was previously doing.
In order to prevent checksum mismatches, it was necessary to also update CI to
use Go 1.12 instead of 1.11 - which also necessitated fixing some linting errors
which appeared with the upgraded golangci-lint for 1.12.
A linter was correctly detecting a case where we were we were doing an
unneeded nil check on `err`. The previous clause in the if/else block
ensures that `err` is non nil.
Use `result.Result` in more places, so when a confirmation prompt is
declined, we just return `result.Bail()` after printing a message
without the `error: ` prefix.
Fixes#2070
`pulumi stack rename` allows you to change the name of an existing
stack. This operation is non-distructive, however it is possible that
the next update will show additional changes to resources, if the
pulumi program uses the value of `getStack()` as part of a resource
name.
* Install missing plugins on startup
This commit addresses the problem of missing plugins by scanning the
snapshot and language host on startup for the list of required plugins
and, if there are any plugins that are required but not installed,
installs them. The mechanism by which plugins are installed is exactly
the same as 'pulumi plugin install'.
The installation of missing plugins is best-effort and, if it fails,
will not fail the update.
This commit addresses pulumi/pulumi-azure#200, where users using Pulumi
in CI often found themselves missing plugins.
* Add CHANGELOG
* Skip downloading plugins if no client provided
* Reduce excessive test output
* Update Gopkg.lock
* Update pkg/engine/destroy.go
Co-Authored-By: swgillespie <sean@pulumi.com>
* CR: make pluginSet a newtype
* CR: Assign loop induction var to local var
If a provider returns information about the top-level properties that
differ, use those keys to filter the diffs that are rendered to the
user.
Fixes#2453.
* Decrease log level for HTTP requests and responses
Logging each HTTP request and response can get quite chatty, especially
when publishing a lot of events. This increases the verbosity level of
these logs so that they don't get emitted at level 9, which is the
general level that providers use when issuing verbose logs.
* Appease linter
When `pulumi stack rm` is run against a stack with resources, the
service will respond with an error if `--force` is not
passed. Previously we would just dump the contents of this error and
it looked something like:
`error: [400] Bad Request: Stack still has resources.`
We now handle this case more gracefully, showing our usual "this stack
still has resources" error like we would for the local backend.
Fixes#2431
In general, a "delete" in Pulumi is destroying an actual physical
resource. In the case of a read resource, however, the delete is
merely removing the resource from the stack; the physical resource
is not affected. These changes attempt to clarify this situation by
using the term "discard" rather than "delete".
Fixes#2015.
Because of the change to include a stack's project as part of its
identity in the service, the names passed to StackReference now
require the project name as well.
Improve the error message when they do not include them.
This implements the new algorithm for deciding which resources must be
deleted due to a delete-before-replace operation.
We need to compute the set of resources that may be replaced by a
change to the resource under consideration. We do this by taking the
complete set of transitive dependents on the resource under
consideration and removing any resources that would not be replaced by
changes to their dependencies. We determine whether or not a resource
may be replaced by substituting unknowns for input properties that may
change due to deletion of the resources their value depends on and
calling the resource provider's Diff method.
This is perhaps clearer when described by example. Consider the
following dependency graph:
A
__|__
B C
| _|_
D E F
In this graph, all of B, C, D, E, and F transitively depend on A. It may
be the case, however, that changes to the specific properties of any of
those resources R that would occur if a resource on the path to A were
deleted and recreated may not cause R to be replaced. For example, the
edge from B to A may be a simple dependsOn edge such that a change to
B does not actually influence any of B's input properties. In that case,
neither B nor D would need to be deleted before A could be deleted.
In order to make the above algorithm a reality, the resource monitor
interface has been updated to include a map that associates an input
property key with the list of resources that input property depends on.
Older clients of the resource monitor will leave this map empty, in
which case all input properties will be treated as depending on all
dependencies of the resource. This is probably overly conservative, but
it is less conservative than what we currently implement, and is
certainly correct.
The service also does this filtering on requests, because we'll need
to support older clients, but it would be nice if the CLI itself also
cleaned things up.
This change starts to use a stack's project name as part of it's
identity when talking to the cloud backend, which the Pulumi Service
now supports.
When displaying or parsing stack names for the cloud backend, we now
support the following schemes:
`<stack-name>`
`<owner-name>/<stack-name>`
`<owner-name>/<project-name>/<stack-name>`
When the owner is not specificed, we assume the currently logged in
user (as we did before). When the project name is not specificed, we
use the current project (and fail if we can't find a `Pulumi.yaml`)
Fixes#2039
We continue to do so when `--debug` has been passed, similar to how
these events are elided from the local display when you are not in a
debug context.
We're changing /account to redirect to /account/profile instead of
/account/tokens as the user profile settings are a more natural place
to land when going into account settings.
This commit changes the CLI to link
directly to /account/tokens, avoiding having to click on
"Access Tokens" to go to the tokens page to get an access token when
coming from the URL outputted by the CLI.
When returning immediately from the loop, we are closing the `done` channel early. This signals that we have finished processing every engine event, however that isn't true. Since some events may still be in-flight in `recordEngineEvent`. (This could potentially lead to a race condition in the `diag.Sink` passed to the API client used to record the call.)
