- Cache the username and last verified time associated with each logged-in
backend
- In the HTTP backend, verify the access token explicitly at most once
per hour
This trades off a little bit of usability for improved inner-loop
latency: if a user's API token becomes invalid less than an hour after
it was last verified, the user will see 4xx errors when attempting stack
operations rather than seeing the login prompt.
The current pattern of using backend.Stack values in the CLI but
accepting backend.StackReference values in backend methods leads to
frequent transitions between these types. In the case of the HTTP
backend, the transition from a StackReference to a Stack requires an API
call. These changes refactor the backend.Backend API such that most of
its methods accept backend.Stack values in place of
backend.StackReference values, which avoids these transitions.
This removes two calls to the getStack API on the startup path of a
`pulumi preview`.
* Add support for filtering stacks by organization, tag
* Update CHANGELOG.md
* Address PR feedback
* Address even more PR feedback
* Support empty-string filters
Since the filestate backend is now written using the go-cloud blob
abstraction, places where we were using functions from the `os`
package were very unlikely to be correct.
This change removes their uses in favor of APIs provided by
go-cloud (which sometimes requires more work than before).
For historical reasons, we used to need to require to load an existing
checkpoint to copy some data from it into the snapshot when saving a
new snapshot. The need for this was removed as part of the general
work in #2678, but we continued to load the checkpoint and then just
disregard the data that was returned (unless there was an error and
that error was not FileNotFound, in which case we would fail).
Our logic for checking if something was FileNotFound was correct when
we wrote it, but when we adopted go-cloud in order to have our
filestate backend also write to blob storage backends like S3, we
forgot that we had checks like `os.IsNotExists()` floating around
which were now incorrect. That meant if the file did not exist for
some reason, instead of going along as planned, we'd error out now
with an error saying something wasn't found.
When we write a checkpoint, we first "backed up" the initial version
by renaming it to include a `.bak` suffix, then we write the new file
in place. However, this can run afoul of eventual consistency models
like S3, since there will be a period of time in which a caller may
observe that the object is missing, even after a new version is
written (based on my understanding of [S3's consistency
model](https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyModel)
Since we no longer need to actually copy any information from the
previous checkpoint, we can simply remove the call entirely to load
it.
As a follow up, we need to audit places inside the filebased backend
that assume `os.*` functions are going to do what we want them to do,
since in general they will not.
Fixes#2714
Attempting to `pulumi stack rename` a stack which had been created but
never updated, when using the local backend, was broken because
code-paths were not hardened against the snapshot being `nil` (which
is the case for a stack before the initial deployment had been done).
Fixes#2654
* Fix issue when logging out of local backend
* Put actual cloud URL into credentials.json
* Fix test on Windows; disable newly-enabled tests on Appveyor
* Keep track of the URL used to initialize local backend
This commit will implement the core business logic of `pulumi policy
publish` -- code to boot an analyzer, ask it for metadata about the
policies it contains, pack the code, and transmit all of this to the
Pulumi service.
Currently if you log into s3://bucket/subdirectory, Pulumi will write
files to s3://bucket/.pulumi and not s3://bucket/subdirectory/.pulumi,
this corrects the error.
When creating a new stack using the local backend, the default
checkpoint has no deployment. That means there's a nil snapshot
created, which means our strategy of using the base snapshot's secrets
manager was not going to work. Trying to do so would result in a panic
because the baseSnapshot is nil in this case.
Using the secrets manager we are going to use to persist the snapshot
is a better idea anyhow, as that's what's actually going to be burned
into the deployment when we serialize the snapshot, so let's use that
instead.
The next change is going to do some code motion that would create some
circular imports if we did not do this. There was nothing that
required the members we were moving be in the backend package, so it
was easy enough to pull them out.
When preforming an update, require that a secrets manager is passed in
as part of the `backend.UpdateOperation` bag and use it. The CLI now
passes this in (it still uses the default base64 secrets manager, so
this is just code motion into a high layer, since the CLI will be the
one to choose what secrets manager to use based on project settings).
There are a few operations we do (stack rename, importing and edits)
where we will materialize a `deploy.Snapshot` from an existing
deployment, mutate it in somewhay, and then store it.
In these cases, we will just re-use the secrets manager that was used
to build the snapshot when we re-serialize it. This is less than ideal
in some cases, because many of these operations could run on an
"encrypted" copy of the Snapshot, where Inputs and Outputs have not
been decrypted.
Unfortunately, our system now is not set up in a great way to support
this and adding something like a `deploy.EncryptedSnapshot` would
require large scale code duplications.
So, for now, we'll take the hit of decrypting and re-encrypting, but
long term introducing a `deploy.EncryptedSnapshot` may be nice as it
would let us elide the encryption/decryption steps in some places and
would also make it clear what parts of our system have access to the
plaintext values of secrets.
We have many cases where we want to do the following:
deployment -> snapshot -> process snapshot -> deployment
We now retain information in the snapshot about the secrets manager
that was used to construct it, so in these round trip cases, we can
re-use the existing manager.
When nil, it means no information is retained in the deployment about
the manager (as there is none) and any attempt to persist secret
values fails.
This should only be used in cases where the snapshot is known to not
contain secret values.
The previous changes to remove config loading out of the backend means
that the backends no longer need to track this information, as they
never use it.
As part of the pluggable secrets work, the crypter's used for secrets
are no longer tied to a backend. To enforce this, we remove the
`backend.GetStackCrypter` function and then have the relevent logic to
construct one live inside the CLI itself.
Right now the CLI still uses the backend type to decide what Crypter
to build, but we'll change that shortly.
We require configuration to preform updates (as well as previews,
destroys and refreshes). Because of how everything evolved, loading
this configuration (and finding the coresponding decrypter) was
implemented in both the file and http backends, which wasn't great.
Refactor things such that the CLI itself builds out this information
and passes it along to the backend to preform operations. This means
less code duplicated between backends and less places the backend
assume things about the existence of `Pulumi.yaml` files and in
general makes the interface more plesent to use for others uses.
For cloud backed stacks, this was already returning nil and due to the
fact that we no longer include config in the checkpoint for local
stacks, it was nil there as well.
Removing this helps clean stuff up and is should make some future
refactorings around custom secret managers easier to land.
We can always add it back later if we miss it (and make it actually do
the right thing!)
When constructing a Deployment (which is a plaintext representation of
a Snapshot), ensure that we encrypt secret values. To do so, we
introduce a new type `secrets.Manager` which is able to encrypt and
decrypt values. In addition, it is able to reflect information about
itself that can be stored in the deployment such that we can
deserialize the deployment into a snapshot (decrypting the values in
the process) without external knowledge about how it was encrypted.
The ability to do this is import for allowing stack references to
work, since two stacks may not use the same manager (or they will use
the same type of manager, but have different state).
The state value is stored in plaintext in the deployment, so it **must
not** contain sensitive data.
A sample manager, which just base64 encodes and decodes strings is
provided, as it useful for testing. We will allow it to be varried
soon.
Because `pulumi query` is not implemented with the update
infrastructure, it is important that we *not* do things like open an
update when the query program runs.
This commit will thus implement the "query" path in the state backend in
a completely parallel universe. Conceptually, this is much like the
update path, but with a conspicuous lack of any connection to the
backend service.
This change adds a --json flag to the preview command, enabling
basic JSON serialization of preview plans. This effectively flattens
the engine event stream into a preview structure that contains a list
of steps, diagnostics, and summary information. Each step contains
the deep serialization of resource state, in addition to metadata about
the step, such as what kind of operation it entails.
This is a partial implementation of pulumi/pulumi#2390. In particular,
we only support --json on the `preview` command itself, and not `up`,
meaning that it isn't possible to serialize the result of an actual
deployment yet (thereby limiting what you can do with outputs, etc).
This change adds a --json flag to the preview command, enabling
basic JSON serialization of preview plans. This effectively flattens
the engine event stream into a preview structure that contains a list
of steps, diagnostics, and summary information. Each step contains
the deep serialization of resource state, in addition to metadata about
the step, such as what kind of operation it entails.
This is a partial implementation of pulumi/pulumi#2390. In particular,
we only support --json on the `preview` command itself, and not `up`,
meaning that it isn't possible to serialize the result of an actual
deployment yet (thereby limiting what you can do with outputs, etc).
A linter was correctly detecting a case where we were we were doing an
unneeded nil check on `err`. The previous clause in the if/else block
ensures that `err` is non nil.
`pulumi stack rename` allows you to change the name of an existing
stack. This operation is non-distructive, however it is possible that
the next update will show additional changes to resources, if the
pulumi program uses the value of `getStack()` as part of a resource
name.
This implements the new algorithm for deciding which resources must be
deleted due to a delete-before-replace operation.
We need to compute the set of resources that may be replaced by a
change to the resource under consideration. We do this by taking the
complete set of transitive dependents on the resource under
consideration and removing any resources that would not be replaced by
changes to their dependencies. We determine whether or not a resource
may be replaced by substituting unknowns for input properties that may
change due to deletion of the resources their value depends on and
calling the resource provider's Diff method.
This is perhaps clearer when described by example. Consider the
following dependency graph:
A
__|__
B C
| _|_
D E F
In this graph, all of B, C, D, E, and F transitively depend on A. It may
be the case, however, that changes to the specific properties of any of
those resources R that would occur if a resource on the path to A were
deleted and recreated may not cause R to be replaced. For example, the
edge from B to A may be a simple dependsOn edge such that a change to
B does not actually influence any of B's input properties. In that case,
neither B nor D would need to be deleted before A could be deleted.
In order to make the above algorithm a reality, the resource monitor
interface has been updated to include a map that associates an input
property key with the list of resources that input property depends on.
Older clients of the resource monitor will leave this map empty, in
which case all input properties will be treated as depending on all
dependencies of the resource. This is probably overly conservative, but
it is less conservative than what we currently implement, and is
certainly correct.
The event rendering goroutine in the remote backend was not properly
synchronizing with the goroutine that created it, and could continue
executing after its creator finished. I believe that this is the root
cause of #1850.
This option allows the user to override the file used to fetch and store
configuration information for a stack. It is available for the config,
destroy, logs, preview, refresh, and up commands.
Note that this option is not persistent: if it is not specified, the
stack's default configuration will be used. If an alternate config file
is used exclusively for a stack, it must be specified to all commands
that interact with that stack.
This option can be used to share plaintext configuration across multiple
stacks. It cannot be used to share secret configuration, as secrets are
associated with a particular stack and cannot be decryptex by other
stacks.
These changes add a new resource to the Pulumi SDK,
`pulumi.StackReference`, that represents a reference to another stack.
This resource has an output property, `outputs`, that contains the
complete set of outputs for the referenced stack. The Pulumi account
performing the deployment that creates a `StackReference` must have
access to the referenced stack or the call will fail.
This resource is implemented by a builtin provider managed by the engine.
This provider will be used for any custom resources and invokes inside
the `pulumi:pulumi` module. Currently this provider supports only the
`pulumi:pulumi:StackReference` resource.
Fixes#109.
Last-minute events coming through the engine could cause the goroutine
iterating over engineEvents to write to displayEvents after it has
already been closed by the main goroutine.
Since I was digging around over the weekend after the change to move
away from light black, and the impact it had on less important
information showing more prominently than it used to, I took a step
back and did a deeper tidying up of things. Another side goal of this
exercise was to be a little more respectful of terminal width; when
we could say things with fewer words, I did so.
* Stylize the preview/update summary differently, so that it stands
out as a section. Also highlight the total changes with bold -- it
turns out this has a similar effect to the bright white colorization,
just without the negative effects on e.g. white terminals.
* Eliminate some verbosity in the phrasing of change summaries.
* Make all heading sections stylized consistently. This includes
the color (bright magenta) and the vertical spacing (always a newline
separating headings). We were previously inconsistent on this (e.g.,
outputs were under "---outputs---"). Now the headings are:
Previewing (etc), Diagnostics, Outputs, Resources, Duration, and Permalink.
* Fix an issue where we'd parent things to "global" until the stack
object later showed up. Now we'll simply mock up a stack resource.
* Don't show messages like "no change" or "unchanged". Prior to the
light black removal, these faded into the background of the terminal.
Now they just clutter up the display. Similar to the elision of "*"
for OpSames in a prior commit, just leave these out. Now anything
that's written is actually a meaningful status for the user to note.
* Don't show the "3 info messages," etc. summaries in the Info column
while an update is ongoing. Instead, just show the latest line. This
is more respectful of width -- I often find that the important
messages scroll off the right of my screen before this change.
For discussion:
- I actually wonder if we should eliminate the summary
altogether and always just show the latest line. Or even
blank it out. The summary feels better suited for the
Diagnostics section, and the Status concisely tells us
how a resource's update ended up (failed, succeeded, etc).
- Similarly, I question the idea of showing only the "worst"
message. I'd vote for always showing the latest, and again
leaving it to the Status column for concisely telling the
user about the final state a resource ended up in.
* Stop prepending "info: " to every stdout/stderr message. It adds
no value, clutters up the display, and worsens horizontal usage.
* Lessen the verbosity of update headline messages, so we now instead
of e.g. "Previewing update of stack 'x':", we just say
"Previewing update (x):".
* Eliminate vertical whitespace in the Diagnostics section. Every
independent console.out previously was separated by an entire newline,
which made the section look cluttered to my eyes. These are just
streams of logs, there's no reason for the extra newlines.
* Colorize the resource headers in the Diagnostic section light blue.
Note that this will change various test baselines, which I will
update next. I didn't want those in the same commit.
Now that we're showing SpecUnimportant as regular text, the extra
"Previewing"/"Updating" line that we show really stands out as being
superfluous. For example, we previously said:
Updating stack 'docker-images'
Performing changes:
...
This change eliminates that second line, so we just have:
Updating stack 'docker-images':
...
* Have backend.ListStacks return a new StackSummary interface
* Update filestake backend to use new type
* Update httpstate backend to use new type
* Update commands to use new type
* lint
* Address PR feedback
* Lint
* Close cancellation source before closing events
The cancellation source logs cancellation messages to the engine event
channel, so we must first close the cancellation source before closing
the channel.
* CR: Fix race in shutdown of signal goroutine
This change implements the same preview behavior we have for
cloud stacks, in pkg/backend/httpbe, for local stacks, in
pkg/backend/filebe. This mostly required just refactoring bits
and pieces so that we can share more of the code, although it
does still entail quite a bit of redundancy. In particular, the
apply functions for both backends are now so close to being
unified, but still require enough custom logic that it warrants
keeping them separate (for now...)
This simply refactors all the display logic out of the
pkg/backend/filestate package. This helps to gear us up to better unify
this logic between the filestate and httpstate backends.
Furthermore, this really ought to be in its own non-backend,
CLI-specific package, but I'm taking one step at a time here.
This renames the backend packages to more closely align with the
new direction for them. Namely, pkg/backend/cloud becomes
pkg/backend/httpstate and pkg/backend/local becomes
pkg/backend/filestate. This also helps to clarify that these are meant
to be around state management and so the upcoming refactoring required
to split out (e.g.) the display logic (amongst other things) will make
more sense, and we'll need better package names for those too.
