This changes two things:
1) Eliminates the fact that we had two kinds of previews in our engine.
2) Always initialize the plugin.Events, to ensure that all plugin loads
are persisted no matter the update type (update, refresh, destroy),
and skip initializing it when dryRun == true, since we won't save them.
The PluginEvents will now try to register loaded plugins which,
during a refresh preview, will result in attempting to save mutations
when a token is missing. This change mirrors the changes made to
destroy which avoid it panicing similarly, by simply leaving
PluginEvents unset. Also adds a bit of tracing that was helpful to
me as I debugged through the underlying issues.
Fixes#1377.
In pulumi/pulumi#1356, we observed that we can fail during a destroy
because we attempt to load the language plugin, which now eagerly looks
for the @pulumi/pulumi package.
This is also blocking ingestion of the latest engine bits into the PPC.
It turns out that for destroy (and refresh), we have no need for the
language plugin. So, let's skip loading it when appropriate.
This changes the CLI interface in a few ways:
* `pulumi preview` is back! The alternative of saying
`pulumi update --preview` just felt awkward, and it's a common
operation to want to perform. Let's just make it work.
* There are two flags consistent across all update commands,
`update`, `refresh`, and `destroy`:
- `--skip-preview` will skip the preview step. Note that this
does *not* skip the prompt to confirm that you'd like to proceed.
Indeed, it will still prompt, with a little warning text about
the fact that the preview has been skipped.
* `--yes` will auto-approve the updates.
This lands us in a simpler and more intuitive spot for common scenarios.
I found the flag --force to be a strange name for skipping a preview,
since that name is usually reserved for operations that might be harmful
and yet you're coercing a tool to do it anyway, knowing there's a chance
you're going to shoot yourself in the foot.
I also found that what I almost always want in the situation where
--force was being used is to actually just run a preview and have the
confirmation auto-accepted. Going straight to --force isn't the right
thing in a CI scenario, where you actually want to run a preview first,
just to ensure there aren't any issues, before doing the update.
In a sense, there are four options here:
1. Run a preview, ask for confirmation, then do an update (the default).
2. Run a preview, auto-accept, and then do an update (the CI scenario).
3. Just run a preview with neither a confirmation nor an update (dry run).
4. Just do an update, without performing a preview beforehand (rare).
This change enables all four workflows in our CLI.
Rather than have an explosion of flags, we have a single flag,
--preview, which can specify the mode that we're operating in. The
following are the values which correlate to the above four modes:
1. "": default (no --preview specified)
2. "auto": auto-accept preview confirmation
3. "only": only run a preview, don't confirm or update
4. "skip": skip the preview altogether
As part of this change, I redid a bit of how the preview modes
were specified. Rather than booleans, which had some illegal
combinations, this change introduces a new enum type. Furthermore,
because the engine is wholly ignorant of these flags -- and only the
backend understands them -- it was confusing to me that
engine.UpdateOptions stored this flag, especially given that all
interesting engine options _also_ accepted a dryRun boolean. As of
this change, the backend.PreviewBehavior controls the preview options.
* Refactor the SnapshotManager interface
Lift snapshot management out of the engine by delegating it to the
SnapshotManager implementation in pkg/backend.
* Add a event interface for plugin loads and use that interface to record plugins in the snapshot
* Remove dead code
* Add comments to Events
* Add a number of tests for SnapshotManager
* CR feedback: use a successful bit on 'End' instead of having a separate 'Abort' API
* CR feedback
* CR feedback: register plugins one-at-a-time instead of the entire state at once
* Re-introduce interface for snapshot management
Snapshot management was done through the Update interface; this commit
splits it into a separate interface
* Put the SnapshotManager instance onto the engine context
* Remove SnapshotManager from planContext and updateActions now that it can be accessed by engine Context
Do not fire a "resource outputs" display event for component resources
after their initial registration. Instead, defer this event until the
component's `RegisterResourceOutputs` call arrives.
hese changes plumb basic support for cancellation through the engine.
Two types of cancellation are supported for all engine operations:
- Cancellation, which waits for the operation to drive itself to a safe
point before the operation returns, and
- Termination, which does not wait for the operation to drive itself
to a safe opint for the operation returns.
When updating local or managed stacks, a single ^C triggers cancellation
of any running operation; a second ^C will trigger termination.
Fixes#513, #1077.
This change implements a `pulumi refresh` command. It operates a bit
like `pulumi update`, and friends, in that it supports `--preview` and
`--diff`, along with the usual flags, and will update your checkpoint.
It works through substitution of the deploy.Source abstraction, which
generates a sequence of resource registration events. This new
deploy.RefreshSource takes in a prior checkpoint and will walk it,
refreshing the state via the associated resource providers by invoking
Read for each resource encountered, and merging the resulting state with
the prior checkpoint, to yield a new resource.Goal state. This state is
then fed through the engine in the usual ways with a few minor caveats:
namely, although the engine must generate steps for the logical
operations (permitting us to get nice summaries, progress, and diffs),
it mustn't actually carry them out because the state being imported
already reflects reality (a deleted resource has *already* been deleted,
so of course the engine need not perform the deletion). The diffing
logic also needs to know how to treat the case of refresh slightly
differently, because we are going to be diffing outputs and not inputs.
Note that support for managed stacks is not yet complete, since that
requires updates to the service to support a refresh endpoint. That
will be coming soon ...
* Lift snapshot management out of the engine
This PR is a prerequisite for parallelism by addressing a major problem
that the engine has to deal with when performing parallel resource
construction: parallel mutation of the global snapshot. This PR adds
a `SnapshotManager` type that is responsible for maintaining and
persisting the current resource snapshot. It serializes all reads and
writes to the global snapshot and persists the snapshot to persistent
storage upon every write.
As a side-effect of this, the core engine no longer needs to know about
snapshot management at all; all snapshot operations can be handled as
callbacks on deployment events. This will greatly simplify the
parallelization of the core engine.
Worth noting is that the core engine will still need to be able to read
the current snapshot, since it is interested in the dependency graphs
contained within. The full implications of that are out of scope of this
PR.
Remove dead code, Steps no longer need a reference to the plan iterator that created them
Fixing various issues that arise when bringing up pulumi-aws
Line length broke the build
Code review: remove dead field, fix yaml name error
Rebase against master, provide implementation of StackPersister for cloud backend
Code review feedback: comments on MutationStatus, style in snapshot.go
Code review feedback: move SnapshotManager to pkg/backend, change engine to use an interface SnapshotManager
Code review feedback: use a channel for synchronization
Add a comment and a new test
* Maintain two checkpoints, an immutable base and a mutable delta, and
periodically merge the two to produce snapshots
* Add a lot of tests - covers all of the non-error paths of BeginMutation and End
* Fix a test resource provider
* Add a few tests, fix a few issues
* Rebase against master, fixed merge
This change includes a bunch of refactorings I made in prep for
doing refresh (first, the command, see pulumi/pulumi#1081):
* The primary change is to change the way the engine's core update
functionality works with respect to deploy.Source. This is the
way we can plug in new sources of resource information during
planning (and, soon, diffing). The way I intend to model refresh
is by having a new kind of source, deploy.RefreshSource, which
will let us do virtually everything about an update/diff the same
way with refreshes, which avoid otherwise duplicative effort.
This includes changing the planOptions (nee deployOptions) to
take a new SourceFunc callback, which is responsible for creating
a source specific to the kind of plan being requested.
Preview, Update, and Destroy now are primarily differentiated by
the kind of deploy.Source that they return, rather than sprinkling
things like `if Destroying` throughout. This tidies up some logic
and, more importantly, gives us precisely the refresh hook we need.
* Originally, we used the deploy.NullSource for Destroy operations.
This simply returns nothing, which is how Destroy works. For some
reason, we were no longer doing this, and instead had some
`if Destroying` cases sprinkled throughout the deploy.EvalSource.
I think this is a vestige of some old way we did configuration, at
least judging by a comment, which is apparently no longer relevant.
* Move diff and diff-printing logic within the engine into its own
pkg/engine/diff.go file, to prepare for upcoming work.
* I keep noticing benign diffs anytime I regenerate protobufs. I
suspect this is because we're also on different versions. I changed
generate.sh to also dump the version into grpc_version.txt. At
least we can understand where the diffs are coming from, decide
whether to take them (i.e., a newer version), and ensure that as
a team we are monotonically increasing, and not going backwards.
* I also tidied up some tiny things I noticed while in there, like
comments, incorrect types, lint suppressions, and so on.
This change uses the prior checkpoint's deployment manifest to pre-
populate all plugins required to complete the destroy operation. This
allows for subsequent attempts to load a resource's plugin to match the
already-loaded version. This approach obviously doesn't work in a
hypothetical future world where plugins for the same resource provider
are loaded side-by-side, but we already know that.
Also, rename/cleanup a bunch of serialization code.
Also, generate better environment names in the serialized closure code. Thsi code should be much easier to make sense of as hte names will better track to the original names in the user code.
Also, dedupe simple non-capturing functions. This helps ensure we don't spit out N copies of __awaiter (one per file it is declared in).
* Improve error messages output by the CLI
This fixes a couple known issues with the way that we present errors
from the Pulumi CLI:
1. Any errors from RPC endpoints were bubbling up as they were to
the top-level, which was unfortunate because they contained
RPC-specific noise that we don't want to present to the user. This
commit unwraps errors from resource providers.
2. The "catastrophic error" message often got printed twice
3. Fatal errors are often printed twice, because our CLI top-level
prints out the fatal error that it receives before exiting. A lot of
the time this error has already been printed.
4. Errors were prefixed by PU####.
* Feedback: Omit the 'catastrophic' error message and use a less verbose error message as the final error
* Code review feedback: interpretRPCError -> resourceStateAndError
* Code review feedback: deleting some commented-out code, error capitalization
* Cleanup after rebase
When a stack has secrets, we now take the secret values and construct
a regular expression which is just an alternation of all the secret
values. Then, before pushing any string data into an Event, we run the
regular expression and replace all matches with '[secret]'.
Fixes#747
The engine now emits events with richer metadata during the
ResourceOutputs and ResourcePre callbacks. The CLI can then use this
information to decide if it should display the event or not and how
much of the event to display.
Options dealing with what to display and how to display it have moved
into the CLI and the engine now emits all information for each event.
I believe because of the way we have structured the code, it is
impossible to know a resource's parent but not printed it. I've
changed the test which would print the parent resource to an assert
that ensure we have printed it.
The next commit is going to remove the shown array because we no
longer need it, but this commit is here so that if there are display
bugs as part of the larger refactoring in how we display events, we
can bisect back and see this failure.
The `shouldShow` method always marked a step as seen, and having the
side effect there is a little confusing. Because we call shouldShow in
the StepPre, StepPost and Output handlers, its also hard to ensure an
invarant I think we want, which is that in the Post and Output
handlers, we've already seen the event.
So, let's move the marking out of `shouldShow` and into
`OnResourceStepPre` and then assert we've already seen it in
`OnResourceStepPre` and `OnResourceOutputs` handlers.
This means that shouldShow is now a pure function and makes it easier
to move the decision on if we should print information about a step
out of the engine and into the CLI.
The engine now unconditionally emits a new type of event, a
PreludeEvent, which contains the configuration for a stack as well as
an indication if the stack is being previewed or updated. The
responsibility for interpreting the --show-config flag on the command
line is now handled by the CLI, which uses this to decide if it should
print the configuration or not, and then writes the "Previewing
changes" or "Deploying chanages" header.
This value was unused across all of our display code. We did thread it
everywhere, but we never actually used the value to make any
decisions. Since we want to move to a model where the engine does not
decide *what* to display, it's helpful to remove this policy stuff
anyway.
config.Key has become a pair of namespace and name. Because the whole
world has not changed yet, there continues to be a way to convert
between a tokens.ModuleMember and config.Key, however now sometime the
conversion from tokens.ModuleMember can fail (when the module member
is not of the form `<package>:config:<name>`).
Right now, config.Key is a type alias for tokens.ModuleMember. I did a
pass over the codebase such that we use config.Key everywhere it
looked like the value did not leak to some external process (e.g a
resource provider or a langhost).
Doing this makes it a little clearer (hopefully) where code is
depending on a module member structure (e.g. <package>:config:<value>)
instead of just an opaque type.
As it stands, we only configure those providers for which configuration
is present. This can lead to surprising failure modes if those providers
are then used to create resources. These changes ensure that all
resource providers that are not configured during plan initialization
are configured upon first load.
Fixes#758.
