The previous changes to remove config loading out of the backend means
that the backends no longer need to track this information, as they
never use it.
As part of the pluggable secrets work, the crypter's used for secrets
are no longer tied to a backend. To enforce this, we remove the
`backend.GetStackCrypter` function and then have the relevent logic to
construct one live inside the CLI itself.
Right now the CLI still uses the backend type to decide what Crypter
to build, but we'll change that shortly.
We require configuration to preform updates (as well as previews,
destroys and refreshes). Because of how everything evolved, loading
this configuration (and finding the coresponding decrypter) was
implemented in both the file and http backends, which wasn't great.
Refactor things such that the CLI itself builds out this information
and passes it along to the backend to preform operations. This means
less code duplicated between backends and less places the backend
assume things about the existence of `Pulumi.yaml` files and in
general makes the interface more plesent to use for others uses.
For cloud backed stacks, this was already returning nil and due to the
fact that we no longer include config in the checkpoint for local
stacks, it was nil there as well.
Removing this helps clean stuff up and is should make some future
refactorings around custom secret managers easier to land.
We can always add it back later if we miss it (and make it actually do
the right thing!)
When constructing a Deployment (which is a plaintext representation of
a Snapshot), ensure that we encrypt secret values. To do so, we
introduce a new type `secrets.Manager` which is able to encrypt and
decrypt values. In addition, it is able to reflect information about
itself that can be stored in the deployment such that we can
deserialize the deployment into a snapshot (decrypting the values in
the process) without external knowledge about how it was encrypted.
The ability to do this is import for allowing stack references to
work, since two stacks may not use the same manager (or they will use
the same type of manager, but have different state).
The state value is stored in plaintext in the deployment, so it **must
not** contain sensitive data.
A sample manager, which just base64 encodes and decodes strings is
provided, as it useful for testing. We will allow it to be varried
soon.
* Add a var for PRNumber. Add an environment metadata key for PR number.
* Move the detection of PULUMI_CI_SYSTEM into vars.DetectVars(). Set the PRNumber CI property based on respective env vars from each CI system.
* Add Azure Pipelines build variables.
* Add tests for DetectVars.
* Added changelog entry for Azure Pipelines.
* Capture the value of env var being modified for the ciutil unit test, and restore their values at the end of them.
* Simplify the DetectVars function by moving the Pulumi CI system code into the switch-case expression.
* Rename the Pulumi CI system to Generic CI. Include the GenericCI system in the test case for DetectVars.
This command exposes a new resource `Invoke` operation,
`pulumi:pulumi:readStackResourceOutputs` which retrieves all resource
outputs for some user-specified stack, not including those deleted.
Fixes#2600.
Because `pulumi query` is not implemented with the update
infrastructure, it is important that we *not* do things like open an
update when the query program runs.
This commit will thus implement the "query" path in the state backend in
a completely parallel universe. Conceptually, this is much like the
update path, but with a conspicuous lack of any connection to the
backend service.
The Pulumi CLI currently has two "display modes" -- one for rendering
diffs and one for rendering program updates. Neither of these is
particularly well-suited to `pulumi query`, which essentially needs to
render stdout from the query program verbatim.
This commit will add a separate display mode for this purpose:
* In interactive mode, `pulumi query` will use the display spinner to
say "running query". In non-interactive mode, this will be omitted.
* Query mode will print stdout from the query program verbatim, but
ignore `diag.Infoerr` so that they're not double-printed when they're
emitted again as error events.
* Query mode also does not double-print newlines at the end of diag
events.
This change adds a --json flag to the preview command, enabling
basic JSON serialization of preview plans. This effectively flattens
the engine event stream into a preview structure that contains a list
of steps, diagnostics, and summary information. Each step contains
the deep serialization of resource state, in addition to metadata about
the step, such as what kind of operation it entails.
This is a partial implementation of pulumi/pulumi#2390. In particular,
we only support --json on the `preview` command itself, and not `up`,
meaning that it isn't possible to serialize the result of an actual
deployment yet (thereby limiting what you can do with outputs, etc).
This change adds a --json flag to the preview command, enabling
basic JSON serialization of preview plans. This effectively flattens
the engine event stream into a preview structure that contains a list
of steps, diagnostics, and summary information. Each step contains
the deep serialization of resource state, in addition to metadata about
the step, such as what kind of operation it entails.
This is a partial implementation of pulumi/pulumi#2390. In particular,
we only support --json on the `preview` command itself, and not `up`,
meaning that it isn't possible to serialize the result of an actual
deployment yet (thereby limiting what you can do with outputs, etc).
This commit switches from dep to Go 1.12 modules for tracking Pulumi
dependencies. Rather than _building_ using Go modules, we instead use the `go
mod vendor` command to populate a vendor tree in the same way as `dep ensure`
was previously doing.
In order to prevent checksum mismatches, it was necessary to also update CI to
use Go 1.12 instead of 1.11 - which also necessitated fixing some linting errors
which appeared with the upgraded golangci-lint for 1.12.
A linter was correctly detecting a case where we were we were doing an
unneeded nil check on `err`. The previous clause in the if/else block
ensures that `err` is non nil.
Use `result.Result` in more places, so when a confirmation prompt is
declined, we just return `result.Bail()` after printing a message
without the `error: ` prefix.
Fixes#2070
`pulumi stack rename` allows you to change the name of an existing
stack. This operation is non-distructive, however it is possible that
the next update will show additional changes to resources, if the
pulumi program uses the value of `getStack()` as part of a resource
name.
* Install missing plugins on startup
This commit addresses the problem of missing plugins by scanning the
snapshot and language host on startup for the list of required plugins
and, if there are any plugins that are required but not installed,
installs them. The mechanism by which plugins are installed is exactly
the same as 'pulumi plugin install'.
The installation of missing plugins is best-effort and, if it fails,
will not fail the update.
This commit addresses pulumi/pulumi-azure#200, where users using Pulumi
in CI often found themselves missing plugins.
* Add CHANGELOG
* Skip downloading plugins if no client provided
* Reduce excessive test output
* Update Gopkg.lock
* Update pkg/engine/destroy.go
Co-Authored-By: swgillespie <sean@pulumi.com>
* CR: make pluginSet a newtype
* CR: Assign loop induction var to local var
If a provider returns information about the top-level properties that
differ, use those keys to filter the diffs that are rendered to the
user.
Fixes#2453.
* Decrease log level for HTTP requests and responses
Logging each HTTP request and response can get quite chatty, especially
when publishing a lot of events. This increases the verbosity level of
these logs so that they don't get emitted at level 9, which is the
general level that providers use when issuing verbose logs.
* Appease linter
When `pulumi stack rm` is run against a stack with resources, the
service will respond with an error if `--force` is not
passed. Previously we would just dump the contents of this error and
it looked something like:
`error: [400] Bad Request: Stack still has resources.`
We now handle this case more gracefully, showing our usual "this stack
still has resources" error like we would for the local backend.
Fixes#2431
In general, a "delete" in Pulumi is destroying an actual physical
resource. In the case of a read resource, however, the delete is
merely removing the resource from the stack; the physical resource
is not affected. These changes attempt to clarify this situation by
using the term "discard" rather than "delete".
Fixes#2015.
Because of the change to include a stack's project as part of its
identity in the service, the names passed to StackReference now
require the project name as well.
Improve the error message when they do not include them.
This implements the new algorithm for deciding which resources must be
deleted due to a delete-before-replace operation.
We need to compute the set of resources that may be replaced by a
change to the resource under consideration. We do this by taking the
complete set of transitive dependents on the resource under
consideration and removing any resources that would not be replaced by
changes to their dependencies. We determine whether or not a resource
may be replaced by substituting unknowns for input properties that may
change due to deletion of the resources their value depends on and
calling the resource provider's Diff method.
This is perhaps clearer when described by example. Consider the
following dependency graph:
A
__|__
B C
| _|_
D E F
In this graph, all of B, C, D, E, and F transitively depend on A. It may
be the case, however, that changes to the specific properties of any of
those resources R that would occur if a resource on the path to A were
deleted and recreated may not cause R to be replaced. For example, the
edge from B to A may be a simple dependsOn edge such that a change to
B does not actually influence any of B's input properties. In that case,
neither B nor D would need to be deleted before A could be deleted.
In order to make the above algorithm a reality, the resource monitor
interface has been updated to include a map that associates an input
property key with the list of resources that input property depends on.
Older clients of the resource monitor will leave this map empty, in
which case all input properties will be treated as depending on all
dependencies of the resource. This is probably overly conservative, but
it is less conservative than what we currently implement, and is
certainly correct.
The service also does this filtering on requests, because we'll need
to support older clients, but it would be nice if the CLI itself also
cleaned things up.
This change starts to use a stack's project name as part of it's
identity when talking to the cloud backend, which the Pulumi Service
now supports.
When displaying or parsing stack names for the cloud backend, we now
support the following schemes:
`<stack-name>`
`<owner-name>/<stack-name>`
`<owner-name>/<project-name>/<stack-name>`
When the owner is not specificed, we assume the currently logged in
user (as we did before). When the project name is not specificed, we
use the current project (and fail if we can't find a `Pulumi.yaml`)
Fixes#2039
We continue to do so when `--debug` has been passed, similar to how
these events are elided from the local display when you are not in a
debug context.
We're changing /account to redirect to /account/profile instead of
/account/tokens as the user profile settings are a more natural place
to land when going into account settings.
This commit changes the CLI to link
directly to /account/tokens, avoiding having to click on
"Access Tokens" to go to the tokens page to get an access token when
coming from the URL outputted by the CLI.
When returning immediately from the loop, we are closing the `done` channel early. This signals that we have finished processing every engine event, however that isn't true. Since some events may still be in-flight in `recordEngineEvent`. (This could potentially lead to a race condition in the `diag.Sink` passed to the API client used to record the call.)
The event rendering goroutine in the remote backend was not properly
synchronizing with the goroutine that created it, and could continue
executing after its creator finished. I believe that this is the root
cause of #1850.
This option allows the user to override the file used to fetch and store
configuration information for a stack. It is available for the config,
destroy, logs, preview, refresh, and up commands.
Note that this option is not persistent: if it is not specified, the
stack's default configuration will be used. If an alternate config file
is used exclusively for a stack, it must be specified to all commands
that interact with that stack.
This option can be used to share plaintext configuration across multiple
stacks. It cannot be used to share secret configuration, as secrets are
associated with a particular stack and cannot be decryptex by other
stacks.
These changes add a new resource to the Pulumi SDK,
`pulumi.StackReference`, that represents a reference to another stack.
This resource has an output property, `outputs`, that contains the
complete set of outputs for the referenced stack. The Pulumi account
performing the deployment that creates a `StackReference` must have
access to the referenced stack or the call will fail.
This resource is implemented by a builtin provider managed by the engine.
This provider will be used for any custom resources and invokes inside
the `pulumi:pulumi` module. Currently this provider supports only the
`pulumi:pulumi:StackReference` resource.
Fixes#109.
We run the same suite of changes that we did on gometalinter. This
ended up catching a few new issues, some of which were addressed and
some of which were baselined.
In preparation for some workspace restructuring, I decided to scratch a
few itches of my own in the code:
* Change project's RuntimeInfo field to just Runtime, to match the
serialized name in JSON/YAML.
* Eliminate the no-longer-used Context and NoDefaultIgnores fields on
project, and all of the associated legacy PPC-related code.
* Eliminate the no-longer-used IgnoreFile constant.
* Remove a bunch of "// nolint: lll" annotations, and simply format
the structures with comments on dedicated lines, to avoid overly
lengthy lines and lint suppressions.
* Mark Dependencies and InitErrors as `omitempty` in the JSON
serialization directives for CheckpointV2 files. This was done for
the YAML directives, but (presumably accidentally) omitted for JSON.
In the past, we had a mode where the CLI would upload the Pulumi
program, as well as its contents and do the execution remotely.
We've since stopped supporting that, but all the supporting code has
been left in the CLI.
This change removes the code we had to support the above case,
including the `pulumi archive` command, which was a debugging tool to
generate the archive we would have uploaded (which was helpful in the
past to understand why behavior differed between local execution and
remote execution.)
If you took the time to type out `--skip-preview`, we should have high
confidence that you don't want a preview and you're okay with the
operation just happening without a prompt.
Fixes#1448
* Remove TODO for issue since fixed in PPCs.
* Update issue reference to source
* Update comment wording
* Remove --ppc arg of stack init
* Remove PPC references in int. testing fx
* Remove vestigial PPC API types
* Introduce new metadata keys `vcs.repo`, `vcs.kind` and `vcs.owner` to keep the keys generic for any vcs. Expanded the git SSH regex to account for bitbucket's .org domain.
* Introduce new stack tags keys with the same theme of detecting the vcs.
* Enable gzip compression on the wire
This change allows the Pulumi API client to gzip requests sent to the
Pulumi service if requested using the 'GzipCompress' http option.
This change also sets the Accept-Encoding: gzip header for all requests
originating from the CLI, indicating to the service that it is free to
gzip responses. The 'readBody' function is used in the API client to
read a response's body, regardless of how it is encoded.
Finally, this change sets GzipCompress: true on the
'PatchUpdateCheckpoint' API call, since JSON payloads in that call tend
to be large and it has become a performance bottleneck.
* spelling
* CR feedback:
1. Clarify and edit comments
2. Close the gzip.Reader when reading bodies
3. Log the payload size when logging compression ratios
Last-minute events coming through the engine could cause the goroutine
iterating over engineEvents to write to displayEvents after it has
already been closed by the main goroutine.
This adds an option, --suppress-outputs, to many commands, to
avoid printing stack outputs for stacks that might contain sensitive
information in their outputs (like key material, and whatnot).
Fixespulumi/pulumi#2028.
There is a seldom-used capability in our CLI, the ability to pass
-m to specify an update message, which we will then show prominently.
At the same time, we already scrape some interesting information from
the Git repo from which an update is performed, like the SHA hash,
committer, and author information. We explicitly didn't want to scrape
the entire message just in case someone put sensitive info inside of it.
It seems safe -- indeed, appealing -- to use just the title portion
as the default update message when no other has been provided (the
majority case). We'll work on displaying it in a better way, but this
strengthens our GitOps/CI/CD story.
Fixespulumi/pulumi#2008.
It's not good practice to dirty up the stdout stream with messages
like this. In fact, I questioned whether we should be emitting
*anything* here, but given that this is often used in unattended
environments, coupled with the fact that it's easy to accidentally set
this and then wonder why `pulumi login` is silently returning, led
me to keep it in (for now, at least).
This change stops attempting to pop a web browser in non-interactive
sessions. Instead, the PULUMI_ACCESS_TOKEN environment variable must
be set. Otherwise, any attempt to use the CLI will yield
$ pulumi --non-interactive preview
error: PULUMI_ACCESS_TOKEN must be set for login during non-interactive CLI sessions
This is the behavior we want for Docker-based invocations of the CLI,
and so is part of pulumi/pulumi#1991.
API calls agains the Pulumi service may start setting a new header,
`X-Pulumi-Warning`. The value of this header should be presented to
the user as a warning.
The Service will use this to provide additional information to the
user without having the CLI have to know about every specific warning
path.
Since I was digging around over the weekend after the change to move
away from light black, and the impact it had on less important
information showing more prominently than it used to, I took a step
back and did a deeper tidying up of things. Another side goal of this
exercise was to be a little more respectful of terminal width; when
we could say things with fewer words, I did so.
* Stylize the preview/update summary differently, so that it stands
out as a section. Also highlight the total changes with bold -- it
turns out this has a similar effect to the bright white colorization,
just without the negative effects on e.g. white terminals.
* Eliminate some verbosity in the phrasing of change summaries.
* Make all heading sections stylized consistently. This includes
the color (bright magenta) and the vertical spacing (always a newline
separating headings). We were previously inconsistent on this (e.g.,
outputs were under "---outputs---"). Now the headings are:
Previewing (etc), Diagnostics, Outputs, Resources, Duration, and Permalink.
* Fix an issue where we'd parent things to "global" until the stack
object later showed up. Now we'll simply mock up a stack resource.
* Don't show messages like "no change" or "unchanged". Prior to the
light black removal, these faded into the background of the terminal.
Now they just clutter up the display. Similar to the elision of "*"
for OpSames in a prior commit, just leave these out. Now anything
that's written is actually a meaningful status for the user to note.
* Don't show the "3 info messages," etc. summaries in the Info column
while an update is ongoing. Instead, just show the latest line. This
is more respectful of width -- I often find that the important
messages scroll off the right of my screen before this change.
For discussion:
- I actually wonder if we should eliminate the summary
altogether and always just show the latest line. Or even
blank it out. The summary feels better suited for the
Diagnostics section, and the Status concisely tells us
how a resource's update ended up (failed, succeeded, etc).
- Similarly, I question the idea of showing only the "worst"
message. I'd vote for always showing the latest, and again
leaving it to the Status column for concisely telling the
user about the final state a resource ended up in.
* Stop prepending "info: " to every stdout/stderr message. It adds
no value, clutters up the display, and worsens horizontal usage.
* Lessen the verbosity of update headline messages, so we now instead
of e.g. "Previewing update of stack 'x':", we just say
"Previewing update (x):".
* Eliminate vertical whitespace in the Diagnostics section. Every
independent console.out previously was separated by an entire newline,
which made the section look cluttered to my eyes. These are just
streams of logs, there's no reason for the extra newlines.
* Colorize the resource headers in the Diagnostic section light blue.
Note that this will change various test baselines, which I will
update next. I didn't want those in the same commit.
Now that we're showing SpecUnimportant as regular text, the extra
"Previewing"/"Updating" line that we show really stands out as being
superfluous. For example, we previously said:
Updating stack 'docker-images'
Performing changes:
...
This change eliminates that second line, so we just have:
Updating stack 'docker-images':
...
* Have backend.ListStacks return a new StackSummary interface
* Update filestake backend to use new type
* Update httpstate backend to use new type
* Update commands to use new type
* lint
* Address PR feedback
* Lint
In order to present a message with a link to a user's access tokens in
the service, we have to convert the API server's URL to the URL for
the console. We understand how to do this for servers that look like
our servers (i.e. their host name is `api.<whatever>`), but in general
we can't do this.
Rewrite the code such that we only print a message about how to find
your access tokens (and offer browser based login) only in cases where
we are able to construct a URL to the console.
Fixes#1930
* Revert "Don't show stack outputs when update fails (#1916)"
This reverts commit e3f89e82aa.
* Be more precise about printing outputs
This commit prints outputs only if they are known to be complete. This
avoids massive red diffs during previews and when component resources
fail to call registerResourceOutputs.
* CR: Clean up large boolean expression and comment
* CR: boolean compromise
* Retire pending deletions at start of plan
Instead of letting pending deletions pile up to be retired at the end of
a plan, this commit eagerly disposes of any pending deletions that were
pending at the end of the previous plan. This is a nice usability win
and also reclaims an invariant that at most one resource with a given
URN is live and at most one is pending deletion at any point in time.
* Rebase against master
* Fix a test issue arising from shared snapshots
* CR feedback
* plan -> replacement
* Use ephemeral statuses to communicate deletions
* Close cancellation source before closing events
The cancellation source logs cancellation messages to the engine event
channel, so we must first close the cancellation source before closing
the channel.
* CR: Fix race in shutdown of signal goroutine
* Don't show stack outputs when update fails
It is basically guaranteed that stack outputs are going to be
meaningless if a plan fails, so this commmit doesn't display them if an
error has occured.
* CR: expand on comment
It is possible for the inputs of a "same" resource to have changed even
if the contents of the input bags are different if the resource's
provider deems the physical change to be semantically irrelevant.
Previously `new` was operating under the assumption that it was always
going to be creating a new project/stack, and would always prompt for
these values. However, we want to be able to use `new` to pull down the
source for an existing stack. This change adds a `--stack` flag to `new`
that can be used to specify an existing stack. If the specified stack
already exists, we won't prompt for the project name/description, and
instead just use the existing stack's values. If `--stack` is specified,
but doesn't already exist, it will just use that as the stack name
(instead of prompting) when creating the stack. `new` also now handles
configuration like `up <url>`: if the stack is a preconfigured empty
stack (i.e. it was created/configured in the Pulumi Console via Pulumi
Button or New Project), we will use the existing stack's config without
prompting. Otherwise we will prompt for config, and just like `up
<url>`, we'll use the existing stack's config values as defaults when
prompting, or if the stack didn't exist, use the defaults from the
template.
Previously `up <url>`'s handling of the project name/description wasn't
correct: it would always automatically use the values from the template
without prompting. Now, just like `new`:
- When updating an existing stack, it will not prompt, and just use the
existing stack's values.
- When creating a new stack, it will prompt for the project
name/description, using the defaults from the template.
This PR consolidates some of the `new`/`up` implementation so it shares
code for this functionality. There's definitely opportunities for a lot
more code reuse, but that cleanup can happen down the line if/when we
have the cycles.
* Don't close eventChannel when panicking
The state of the system is completely unknown when panicking and in
general it's not safe to infer whether or not it is safe to close a
channel when in this tate.
* CR feedback
* Spelling
This change implements the same preview behavior we have for
cloud stacks, in pkg/backend/httpbe, for local stacks, in
pkg/backend/filebe. This mostly required just refactoring bits
and pieces so that we can share more of the code, although it
does still entail quite a bit of redundancy. In particular, the
apply functions for both backends are now so close to being
unified, but still require enough custom logic that it warrants
keeping them separate (for now...)
This simply refactors all the display logic out of the
pkg/backend/filestate package. This helps to gear us up to better unify
this logic between the filestate and httpstate backends.
Furthermore, this really ought to be in its own non-backend,
CLI-specific package, but I'm taking one step at a time here.
This change alters the login prompt slightly, so that it is more
obvious that alternative methods exist.
Before this change, we would say:
$ pulumi login
We need your Pulumi account to identify you.
Enter your access token from https://app.pulumi.com/account
or hit <ENTER> to log in using your browser :
After this change, we say this instead:
$ pulumi login
Manage your Pulumi stacks by logging in.
Run `pulumi login --help` for alternative login options.
Enter your access token from https://app.pulumi.com/account
or hit <ENTER> to log in using your browser :
Also updated the help text to advertise this a bit more prominently.
This renames the backend packages to more closely align with the
new direction for them. Namely, pkg/backend/cloud becomes
pkg/backend/httpstate and pkg/backend/local becomes
pkg/backend/filestate. This also helps to clarify that these are meant
to be around state management and so the upcoming refactoring required
to split out (e.g.) the display logic (amongst other things) will make
more sense, and we'll need better package names for those too.
As part of making the local backend more prominent, this changes a few
aspects of how you use it:
* Simplify how you log into a specific cloud; rather than
`pulumi login --cloud-url <url>`, just say `pulumi login <url>`.
* Use a proper URL scheme to denote local backend usage. We have chosen
file://, since the REST API backend is of course always https://.
This means that you can say `pulumi login file://~` to use the local
backend, with state files stored in your home directory. Similarly,
we support `pulumi login file://.` for the current directory.
* Add a --local flag to the login command, to make local logins a
bit easier in the common case of using your home directory. Just say
`pulumi login --local` and it is sugar for `pulumi login file://~`.
* Print the URL for the backend after logging in; for the cloud,
this is just the user's stacks page, and for the local backend,
this is the path to the user's stacks directory on disk.
* Tidy up the documentation for login a bit to be clearer about this.
This is part of pulumi/pulumi#1818.
The existing message put the URL to visit and some explanation text on
the same line, which makes it a little harder to copy only the URL
into a browser. If this extra text ends up being copied as well as the
URL it can lead to failures later, when we try to decode the query
string as part of the OAuth flow.
It's easy enough to fix by just putting the URL on its own line, split
off from the text itself.
Fixes#1832
This changes two things:
1) Per PR feedback, change the text to "N changes found during refresh."
2) Colorize the parenthetical "No resources will be modified" message;
it looked a little odd being plain colored.
This commit reverts most of #1853 and replaces it with functionally
identical logic, using the notion of status message-specific sinks.
In other words, where the original commit implemented ephemeral status
messages by adding an `isStatus` parameter to most of the logging
methdos in pulumi/pulumi, this implements ephemeral status messages as a
parallel logging sink, which emits _only_ ephemeral status messages.
The original commit message in that PR was:
> Allow log events to be marked "status" events
>
> This commit will introduce a field, IsStatus to LogRequest. A "status"
> logging event will be displayed in the Info column of the main
> display, but will not be printed out at the end, when resource
> operations complete.
>
> For example, for complex resource initialization, we'd like to display
> a series of intermediate results: [1/4] Service object created, for
> example. We'd like these to appear in the Info column, but not at the
> end, where they are not helpful to the user.
- Create all refresh steps before issuing any. This is important as the
state update loop expects all steps to exist.
- Check for cancellation later in the refresher.
This also fixes races in the SnapshotManager and the test journal that
could cause panics during cancellation.
The wording for refresh doesn't accurately convey that the operations
aren't actually mutating your resources, but instead are simply changing
your checkpoint state. This change (hopefully) helps in two ways:
First, put text just before the prompt:
Do you want to perform this refresh?
No resources will be modified as part of this refresh; just your stack's state will be.
Second, alter the summary ever-so-slightly, from:
info: 2 changes performed:
~ 2 resources updated
3 resources unchanged
to:
info: 2 changes refreshed:
~ 2 resources updated
3 resources unchanged
This reads just slightly better, and removes any sense of panic I might
have otherwise had that my refresh just did something wrong.
As I was in here, since I had to pass UpdateKind information to new
places, I cleaned up the situation where we had three mostly-similar
enums (but which actually diverged) and several areas where we were
using untyped strings for this same information. Now there's just one.
This fixespulumi/pulumi#1551.
Replace the Source-based implementation of refresh with a phase that
runs as the first part of plan execution and rewrites the snapshot in-memory.
In order to fit neatly within the existing framework for resource operations,
these changes introduce a new kind of step, RefreshStep, to represent
refreshes. RefreshSteps operate similar to ReadSteps but do not imply that
the resource being read is not managed by Pulumi.
In addition to the refresh reimplementation, these changes incorporate those
from #1394 to run refresh in the integration test framework.
Fixes#1598.
Fixespulumi/pulumi-terraform#165.
Contributes to #1449.
The fact that we include the `:config:` portion of a configuration
name when sending to the service is an artifact of history. It was
needed back when we needed to support deploying using a new CLI
against a PPC that had older packages embeded in it.
We don't have that sort of problem anymore, so we can stop sending
this data.
A checkpoint write is unnecessary if it does not change the semantics of
the data currently stored in the checkpoint. We currently perform
unnecessary checkpoint writes in two cases:
- Same steps where no aspect of the resource's state has changed
- Replace steps, which exist solely for display purposes
The former case is particularly bothersome, as it is rather common to
run updates--especially in CI--that consist largely/entirely of these
same steps.
These changes eliminate the checkpoint writes we perform in these two
cases. Some care is needed to ensure that we continue to write the
checkpoint in the case of same steps that do represent meaningful
changes (e.g. changes to a resource's output properties or
dependencies).
Fixes#1769.
* Add a list of in-flight operations to the deployment
This commit augments 'DeploymentV2' with a list of operations that are
currently in flight. This information is used by the engine to keep
track of whether or not a particular deployment is in a valid state.
The SnapshotManager is responsible for inserting and removing operations
from the in-flight operation list. When the engine registers an intent
to perform an operation, SnapshotManager inserts an Operation into this
list and saves it to the snapshot. When an operation completes, the
SnapshotManager removes it from the snapshot. From this, the engine can
infer that if it ever sees a deployment with pending operations, the
Pulumi CLI must have crashed or otherwise abnormally terminated before
seeing whether or not an operation completed successfully.
To remedy this state, this commit also adds code to 'pulumi stack
import' that clears all pending operations from a deployment, as well as
code to plan generation that will reject any deployments that have
pending operations present.
At the CLI level, if we see that we are in a state where pending
operations were in-flight when the engine died, we'll issue a
human-friendly error message that indicates which resources are in a bad
state and how to recover their stack.
* CR: Multi-line string literals, renaming in-flight -> pending
* CR: Add enum to apitype for operation type, also name status -> type for clarity
* Fix the yaml type
* Fix missed renames
* Add implementation for lifecycle_test.go
* Rebase against master
* Initial support for passing URLs to `new` and `up`
This PR adds initial support for `pulumi new` using Git under the covers
to manage Pulumi templates, providing the same experience as before.
You can now also optionally pass a URL to a Git repository, e.g.
`pulumi new [<url>]`, including subdirectories within the repository,
and arbitrary branches, tags, or commits.
The following commands result in the same behavior from the user's
perspective:
- `pulumi new javascript`
- `pulumi new https://github.com/pulumi/templates/templates/javascript`
- `pulumi new https://github.com/pulumi/templates/tree/master/templates/javascript`
- `pulumi new https://github.com/pulumi/templates/tree/HEAD/templates/javascript`
To specify an arbitrary branch, tag, or commit:
- `pulumi new https://github.com/pulumi/templates/tree/<branch>/templates/javascript`
- `pulumi new https://github.com/pulumi/templates/tree/<tag>/templates/javascript`
- `pulumi new https://github.com/pulumi/templates/tree/<commit>/templates/javascript`
Branches and tags can include '/' separators, and `pulumi` will still
find the right subdirectory.
URLs to Gists are also supported, e.g.:
`pulumi new https://gist.github.com/justinvp/6673959ceb9d2ac5a14c6d536cb871a6`
If the specified subdirectory in the repository does not contain a
`Pulumi.yaml`, it will look for subdirectories within containing
`Pulumi.yaml` files, and prompt the user to choose a template, along the
lines of how `pulumi new` behaves when no template is specified.
The following commands result in the CLI prompting to choose a template:
- `pulumi new`
- `pulumi new https://github.com/pulumi/templates/templates`
- `pulumi new https://github.com/pulumi/templates/tree/master/templates`
- `pulumi new https://github.com/pulumi/templates/tree/HEAD/templates`
Of course, arbitrary branches, tags, or commits can be specified as well:
- `pulumi new https://github.com/pulumi/templates/tree/<branch>/templates`
- `pulumi new https://github.com/pulumi/templates/tree/<tag>/templates`
- `pulumi new https://github.com/pulumi/templates/tree/<commit>/templates`
This PR also includes initial support for passing URLs to `pulumi up`,
providing a streamlined way to deploy installable cloud applications
with Pulumi, without having to manage source code locally before doing
a deployment.
For example, `pulumi up https://github.com/justinvp/aws` can be used to
deploy a sample AWS app. The stack can be updated with different
versions, e.g.
`pulumi up https://github.com/justinvp/aws/tree/v2 -s <stack-to-update>`
Config values can optionally be passed via command line flags, e.g.
`pulumi up https://github.com/justinvp/aws -c aws:region=us-west-2 -c foo:bar=blah`
Gists can also be used, e.g.
`pulumi up https://gist.github.com/justinvp/62fde0463f243fcb49f5a7222e51bc76`
* Fix panic when hitting ^C from "choose template" prompt
* Add description to templates
When running `pulumi new` without specifying a template, include the template description along with the name in the "choose template" display.
```
$ pulumi new
Please choose a template:
aws-go A minimal AWS Go program
aws-javascript A minimal AWS JavaScript program
aws-python A minimal AWS Python program
aws-typescript A minimal AWS TypeScript program
> go A minimal Go program
hello-aws-javascript A simple AWS serverless JavaScript program
javascript A minimal JavaScript program
python A minimal Python program
typescript A minimal TypeScript program
```
* React to changes to the pulumi/templates repo.
We restructured the `pulumi/templates` repo to have all the templates in the root instead of in a `templates` subdirectory, so make the change here to no longer look for templates in `templates`.
This also fixes an issue around using `Depth: 1` that I found while testing this. When a named template is used, we attempt to clone or pull from the `pulumi/templates` repo to `~/.pulumi/templates`. Having it go in this well-known directory allows us to maintain previous behavior around allowing offline use of templates. If we use `Depth: 1` for the initial clone, it will fail when attempting to pull when there are updates to the remote repository. Unfortunately, there's no built-in `--unshallow` support in `go-git` and setting a larger `Depth` doesn't appear to help. There may be a workaround, but for now, if we're cloning the pulumi templates directory to `~/.pulumi/templates`, we won't use `Depth: 1`. For template URLs, we will continue to use `Depth: 1` as we clone those to a temp directory (which gets deleted) that we'll never try to update.
* List available templates in help text
* Address PR Feedback
* Don't show "Installing dependencies" message for `up`
* Fix secrets handling
When prompting for config, if the existing stack value is a secret, keep it a secret and mask the prompt. If the template says it should be secret, make it a secret.
* Fix ${PROJECT} and ${DESCRIPTION} handling for `up`
Templates used with `up` should already have a filled-in project name and description, but if it's a `new`-style template, that has `${PROJECT}` and/or `${DESCRIPTION}`, be helpful and just replace these with better values.
* Fix stack handling
Add a bool `setCurrent` param to `requireStack` to control whether the current stack should be saved in workspace settings. For the `up <url>` case, we don't want to save. Also, split the `up` code into two separate functions: one for the `up <url>` case and another for the normal `up` case where you have workspace in your current directory. While we may be able to combine them back into a single function, right now it's a bit cleaner being separate, even with some small amount of duplication.
* Fix panic due to nil crypter
Lazily get the crypter only if needed inside `promptForConfig`.
* Embellish comment
* Harden isPreconfiguredEmptyStack check
Fix the code to check to make sure the URL specified on the command line matches the URL stored in the `pulumi:template` config value, and that the rest of the config from the stack satisfies the config requirements of the template.
* Serialize SourceEvents coming from the refresh source
The engine requires that a source event coming from a source be "ready
to execute" at the moment that it is sent to the engine. Since the
refresh source sent all goal states eagerly through its source iterator,
the engine assumed that it was legal to execute them all in parallel and
did so. This is a problem for the snapshot, since the snapshot expects
to be in an order that is a legal topological ordering of the dependency
DAG.
This PR fixes the issue by sending refresh source events one-at-a-time
through the refresh source iterator, only unblocking to send the next
step as soon as the previous step completes.
* Fix deadlock in refresh test
* Fix an issue where the engine "completed" steps too early
By signalling that a step is done before committing the step's results
to the snapshot, the engine was left with a race where dependent
resources could find themselves completely executed and committed before
a resource that they depend on has been committed.
Fixespulumi/pulumi#1726
* Fix an issue with Replace steps at the end of a plan
If the last step that was executed successfully was a Replace, we could
end up in a situation where we unintentionally left the snapshot
invalid.
* Add a test
* CR: pass context.Context as first parameter to Iterate
* CR: null->nil
### First-Class Providers
These changes implement support for first-class providers. First-class
providers are provider plugins that are exposed as resources via the
Pulumi programming model so that they may be explicitly and multiply
instantiated. Each instance of a provider resource may be configured
differently, and configuration parameters may be source from the
outputs of other resources.
### Provider Plugin Changes
In order to accommodate the need to verify and diff provider
configuration and configure providers without complete configuration
information, these changes adjust the high-level provider plugin
interface. Two new methods for validating a provider's configuration
and diffing changes to the same have been added (`CheckConfig` and
`DiffConfig`, respectively), and the type of the configuration bag
accepted by `Configure` has been changed to a `PropertyMap`.
These changes have not yet been reflected in the provider plugin gRPC
interface. We will do this in a set of follow-up changes. Until then,
these methods are implemented by adapters:
- `CheckConfig` validates that all configuration parameters are string
or unknown properties. This is necessary because existing plugins
only accept string-typed configuration values.
- `DiffConfig` either returns "never replace" if all configuration
values are known or "must replace" if any configuration value is
unknown. The justification for this behavior is given
[here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106)
- `Configure` converts the config bag to a legacy config map and
configures the provider plugin if all config values are known. If any
config value is unknown, the underlying plugin is not configured and
the provider may only perform `Check`, `Read`, and `Invoke`, all of
which return empty results. We justify this behavior becuase it is
only possible during a preview and provides the best experience we
can manage with the existing gRPC interface.
### Resource Model Changes
Providers are now exposed as resources that participate in a stack's
dependency graph. Like other resources, they are explicitly created,
may have multiple instances, and may have dependencies on other
resources. Providers are referred to using provider references, which
are a combination of the provider's URN and its ID. This design
addresses the need during a preview to refer to providers that have not
yet been physically created and therefore have no ID.
All custom resources that are not themselves providers must specify a
single provider via a provider reference. The named provider will be
used to manage that resource's CRUD operations. If a resource's
provider reference changes, the resource must be replaced. Though its
URN is not present in the resource's dependency list, the provider
should be treated as a dependency of the resource when topologically
sorting the dependency graph.
Finally, `Invoke` operations must now specify a provider to use for the
invocation via a provider reference.
### Engine Changes
First-class providers support requires a few changes to the engine:
- The engine must have some way to map from provider references to
provider plugins. It must be possible to add providers from a stack's
checkpoint to this map and to register new/updated providers during
the execution of a plan in response to CRUD operations on provider
resources.
- In order to support updating existing stacks using existing Pulumi
programs that may not explicitly instantiate providers, the engine
must be able to manage the "default" providers for each package
referenced by a checkpoint or Pulumi program. The configuration for
a "default" provider is taken from the stack's configuration data.
The former need is addressed by adding a provider registry type that is
responsible for managing all of the plugins required by a plan. In
addition to loading plugins froma checkpoint and providing the ability
to map from a provider reference to a provider plugin, this type serves
as the provider plugin for providers themselves (i.e. it is the
"provider provider").
The latter need is solved via two relatively self-contained changes to
plan setup and the eval source.
During plan setup, the old checkpoint is scanned for custom resources
that do not have a provider reference in order to compute the set of
packages that require a default provider. Once this set has been
computed, the required default provider definitions are conjured and
prepended to the checkpoint's resource list. Each resource that
requires a default provider is then updated to refer to the default
provider for its package.
While an eval source is running, each custom resource registration,
resource read, and invoke that does not name a provider is trapped
before being returned by the source iterator. If no default provider
for the appropriate package has been registered, the eval source
synthesizes an appropriate registration, waits for it to complete, and
records the registered provider's reference. This reference is injected
into the original request, which is then processed as usual. If a
default provider was already registered, the recorded reference is
used and no new registration occurs.
### SDK Changes
These changes only expose first-class providers from the Node.JS SDK.
- A new abstract class, `ProviderResource`, can be subclassed and used
to instantiate first-class providers.
- A new field in `ResourceOptions`, `provider`, can be used to supply
a particular provider instance to manage a `CustomResource`'s CRUD
operations.
- A new type, `InvokeOptions`, can be used to specify options that
control the behavior of a call to `pulumi.runtime.invoke`. This type
includes a `provider` field that is analogous to
`ResourceOptions.provider`.
This change lets us set runtime specific options in Pulumi.yaml, which
will flow as arguments to the language hosts. We then teach the nodejs
host that when the `typescript` is set to `true` that it should load
ts-node before calling into user code. This allows using typescript
natively without an explicit compile step outside of Pulumi.
This works even when a tsconfig.json file is not present in the
application and should provide a nicer inner loop for folks writing
typescript (I'm pretty sure everyone has run into the "but I fixed
that bug! Why isn't it getting picked up? Oh, I forgot to run tsc"
problem.
Fixes#958
* Protobuf changes to record dependencies for read resources
* Add a number of tests for read resources, especially around replacement
* Place read resources in the snapshot with "external" bit set
Fixespulumi/pulumi#1521. This commit introduces two new step ops: Read
and ReadReplacement. The engine generates Read and ReadReplacement steps
when servicing ReadResource RPC calls from the language host.
* Fix an omission of OpReadReplace from the step list
* Rebase against master
* Transition to use V2 Resources by default
* Add a semantic "relinquish" operation to the engine
If the engine observes that a resource is read and also that the
resource exists in the snapshot as a non-external resource, it will not
delete the resource if the IDs of the old and new resources match.
* Typo fix
* CR: add missing comments, DeserializeDeployment -> DeserializeDeploymentV2, ID check
This change makes it a little easier to do the style of highlighting
we are doing now with the login prompt, cleaning up some of the
padding calculations that were otherwise complicated due to ANSI
escape sequences.
This change makes our login prompt a little "friendlier", especially
important since this will be the first thing a new user sees.
The new message is:
$ pulumi new
We need your Pulumi account to identify you.
Enter your access token from https://app.pulumi.com/account
or hit <ENTER> to log in using your browser :
This change adds some convenience functions and types, to make strongly
typed outputs more pleasant to interact with. It also includes tests
for output generally, in addition to these new functions and types.
The Version field was inadvertently dropped when sending an import
request to the service. Now that we are requiring that the Version field
be set in deployments, this was causing errors.
Calling `bytes.NewBuffer` gives up ownership of the backing memory and creates
a _mutable_ buffer. We don't need that, so let's use `bytes.NewReader` instead.
* Error when loading a deployment that is not a version that the CLI understands
* Add a test for 'pulumi stack import' on a badly-versioned deployment
* Move current deployment version to 'apitype'
* Rebase against master
* CR: emit CLI-friendly error message at the two points outside of the engine calling 'DeserializeDeployment'
An empty preview is not a guarantee that a subsequent update will cause
no changes. As such, we should always offer the user the option to
perform an update even if the preview proposes no changes.
Note that this indirectly fixes#1410 by masking the effects of #1429.
This change supports displaying manifest information for a stack and
changes the way we handle Snapshots in our backend.
Previously, every call to GetStack would synthesize a Snapshot by
taking the set of resources returned from the
`/api/stacks/<owner>/<name>` endpoint, combined with an empty
manfiest (since the service was not returning the manifest).
This wasn't great for two reasons:
1. We didn't have manifest information, so we couldn't display any of
its information (most important the last updated time).
2. This strategy required that the service return all the resources
for a stack anytime GetStack was called. While the CLI did not
often need this detailed information the fact that we forced the
Service to produce it (which in the case of stack managed PPC would
require the service to talk to yet another service) creates a bunch
of work that we end up ignoring.
I've refactored the code such that `backend.Stack`'s `Snapshot()` method
now lazily requests the information from the service such that we can
construct a `Snapshot()` on demand and only pay the cost when we
actually need it.
I think making more of this stuff lazy is the long term direction we
want to follow.
Unfortunately, right now, it means in cases where we do need this data
we end up fetching it twice. The service does it once when we call
GetStack and then we do it again when we actually need to get at the
Snapshot. However, once we land this change, we can update the
service to no longer return resources on the apistack.Stack type. The
CLI no longer needs this property. We'll likely want to continue in a
direction where `apistack.Stack` can be created quickly by the
service (without expensive database queries or fetching remote
resources) and just add additional endpoints that let us get at the
specific information we want in the specific cases when we want it
instead of forcing us to return a bunch of data that we often ignore.
Fixespulumi/pulumi-service#371
* Delete Before Create
This commit implements the full semantics of delete before create. If a
resource is replaced and requires deletion before creation, the engine
will use the dependency graph saved in the snapshot to delete all
resources that depend on the resource being replaced prior to the
deletion of the resource to be replaced.
* Rebase against master
* CR: Simplify the control flow in makeRegisterResourceSteps
* Run Check on new inputs when re-creating a resource
* Fix an issue where the planner emitted benign but incorrect deletes of DBR-deleted resources
* CR: produce the list of dependent resources in dependency order and iterate over the list in reverse
* CR: deps->dependents, fix an issue with DependingOn where duplicate nodes could be added to the dependent set
* CR: Fix an issue where we were considering old defaults and new inputs
inappropriately when re-creating a deleted resource
* CR: save 'iter.deletes[urn]' as a local, iterate starting at cursorIndex + 1 for dependency graph
At some point the summary and preview parameters to
`engine.GetResourcePropertiesDetails` were flipped s.t. we stopped
rendering computed<> and output<> values properly during previews.
We currently display at most two rows for messages that are not
associated with a URN: one for those that arrive before the stack
resource is first observed and one for those that arrive after the stack
resource is first observed. The former is labeled "global"; the latter
is the row associated with the stack resource. These changes remove the
duplication by reassociating the row used for events with no URN that
arrive before the stack resource with the stack resource once it has
been observed.
Fixes#1385.
Our current strategy for the progress display on non-TTYs causes us to
display multiple identical rows for each resource when the row is a
preview or a no-op. This behavior is not particularly useful, and
generally just makes the display noisier than it needs to be.
These changes avoid displaying resource rows without meaningful output
by supressing the display of resource output events that are delivered
during a preview or that correspond to a no-op update.
Fixes#1384.
This changes two things:
1) Eliminates the fact that we had two kinds of previews in our engine.
2) Always initialize the plugin.Events, to ensure that all plugin loads
are persisted no matter the update type (update, refresh, destroy),
and skip initializing it when dryRun == true, since we won't save them.
The PluginEvents will now try to register loaded plugins which,
during a refresh preview, will result in attempting to save mutations
when a token is missing. This change mirrors the changes made to
destroy which avoid it panicing similarly, by simply leaving
PluginEvents unset. Also adds a bit of tracing that was helpful to
me as I debugged through the underlying issues.
Fixes#1377.
* Graceful RPC shutdown: CLI side
* Handle unavailable resource monitor in language hosts
* Fix a comment
* Don't commit package-lock.json
* fix mangled pylint pragma
* Rebase against master and fix Gopkg.lock
* Code review feedback
* Fix a race between closing the callerEventsOpt channel and terminating a goroutine that writes to it
* glog -> logging
During a deployment, we end up making a bunch of PATCH/PUT/POST style
REST requests. For these calls, it should be safe to retry the
operation if there was a hickup during our REST call, so mark them as
retryable.
These changes add support for adding a tracing header to API requests
made to the Pulumi service. Setting the `PULUMI_TRACING_HEADER`
environment variable or enabling debug commands and passing the
`--tracing-header` will change the value sent in this header. Setting
this value to `1` will request that the service enable distributed
tracing for all requests made by a particular CLI invocation.
The newly added `pulumi config refresh` updates your local copy of the
Pulumi.<stack-name>.yaml file to have the same configuration as the
most recent deployment in the cloud.
This can be used in a varirty of ways. One place we plan to use it is
in automation to clean up "leaked" stacks we have in CI. With the
changes you'll now be able to do the following:
```
$ cd $(mktemp -d)
$ echo -e "name: who-cares\nruntime: nodejs" > Pulumi.yaml
$ pulumi stack select <leaked-stack-name>
$ pulumi config refresh -f
$ pulumi destroy --force
```
Having a simpler gesture for the above is something we'll want to do
long term (we should be able to support `pulumi destory <stack-name>`
from a completely empty folder, today you need a Pulumi.yaml file
present, even if the contents don't matter).
But this gets us a little closer to where we want to be and introduces
a helpful primitive in the system.
Contributes to #814
These changes add support for injecting client tracing spans into HTTP
requests to the Pulumi API. The server can then rematerialize these span
references and attach its own spans for distributed tracing.
This change captures the Git committer and author's login and email
addresses, so that we can display them prominently in the service. At
the moment, we only attribute updates to the identity that performed the
update which, in CI scenarios, is often always the same person for an
organization. This makes Pulumi look like a needlessly lonely place.
These changes enable tracing of Pulumi API calls.
The span with which to associate an API call is passed via a
`context.Context` parameter. This required plumbing a
`context.Context` parameter through a rather large number of APIs,
especially in the backend.
In general, all API calls are associated with a new root span that
exists for essentially the entire lifetime of an invocation of the
Pulumi CLI. There were a few places where the plumbing got a bit hairier
than I was willing to address with these changes; I've used
`context.Background()` in these instances. API calls that receive this
context will create new root spans, but will still be traced.
* Initialize a new stack as part of `pulumi new`
* Prompt for values with defaults preselected
* Install dependencies
* Prompt for default config values
This changes the CLI interface in a few ways:
* `pulumi preview` is back! The alternative of saying
`pulumi update --preview` just felt awkward, and it's a common
operation to want to perform. Let's just make it work.
* There are two flags consistent across all update commands,
`update`, `refresh`, and `destroy`:
- `--skip-preview` will skip the preview step. Note that this
does *not* skip the prompt to confirm that you'd like to proceed.
Indeed, it will still prompt, with a little warning text about
the fact that the preview has been skipped.
* `--yes` will auto-approve the updates.
This lands us in a simpler and more intuitive spot for common scenarios.
I found the flag --force to be a strange name for skipping a preview,
since that name is usually reserved for operations that might be harmful
and yet you're coercing a tool to do it anyway, knowing there's a chance
you're going to shoot yourself in the foot.
I also found that what I almost always want in the situation where
--force was being used is to actually just run a preview and have the
confirmation auto-accepted. Going straight to --force isn't the right
thing in a CI scenario, where you actually want to run a preview first,
just to ensure there aren't any issues, before doing the update.
In a sense, there are four options here:
1. Run a preview, ask for confirmation, then do an update (the default).
2. Run a preview, auto-accept, and then do an update (the CI scenario).
3. Just run a preview with neither a confirmation nor an update (dry run).
4. Just do an update, without performing a preview beforehand (rare).
This change enables all four workflows in our CLI.
Rather than have an explosion of flags, we have a single flag,
--preview, which can specify the mode that we're operating in. The
following are the values which correlate to the above four modes:
1. "": default (no --preview specified)
2. "auto": auto-accept preview confirmation
3. "only": only run a preview, don't confirm or update
4. "skip": skip the preview altogether
As part of this change, I redid a bit of how the preview modes
were specified. Rather than booleans, which had some illegal
combinations, this change introduces a new enum type. Furthermore,
because the engine is wholly ignorant of these flags -- and only the
backend understands them -- it was confusing to me that
engine.UpdateOptions stored this flag, especially given that all
interesting engine options _also_ accepted a dryRun boolean. As of
this change, the backend.PreviewBehavior controls the preview options.
The service has no good way to preview a destroy operation for a stack
managed by a PPC. Until it does, just behave as if --force was passed
to the CLI in this case (i.e. skip the preview).
Fixes#1301
When linking back to the service, use the newer, simpler, URLs for a
stack: `https://pulumi.com/<owner>/<stack-name>` instead of
`https://pulumi.com/<owner>/-/-/<stack-name>`
Refactoring to support "preview" and "update" as part of the same
operation interacted poorly with deploying into a PPC via the service.
Per Pat, this is the quickest fix that gets us off the floor.
Part of #1301
This changes three minor UX things in the CLI's update flow:
1. Use bright blue for column headers, since the dark blue is nearly
invisible on a black background.
2. Move the operation symbol to the far left, as a sort of "bullet
point" for each line of output. This was how they initial symbols
were designed and this helps to glance at the summary to see what's
going on.
3. Shorten some of the column headers that didn't add extra clarity.
Pat ran into a weird error when trying to do some development agains
the testing cloud:
```
$ pulumi logout
$ pulumi login --cloud-url [test-cloud-url]
Logged into [test-cloud-url]
$ pulumi stack ls
Enter your Pulumi access token from https://pulumi.com/account:
```
In his case, we did not have `current` set in our credentials.json
file (likely due to him calling `pulumi logout` at some point) but we
did have stored credentials for that cloud. When he logged in the CLI
noticed we could reuse the stored credentials but did not update the
the current setting to set the current cloud.
While investigating, I also noticed that `logout` did not always do
the right thing when you were logged into a different backend than
pulumi.com
* Refactor the SnapshotManager interface
Lift snapshot management out of the engine by delegating it to the
SnapshotManager implementation in pkg/backend.
* Add a event interface for plugin loads and use that interface to record plugins in the snapshot
* Remove dead code
* Add comments to Events
* Add a number of tests for SnapshotManager
* CR feedback: use a successful bit on 'End' instead of having a separate 'Abort' API
* CR feedback
* CR feedback: register plugins one-at-a-time instead of the entire state at once
- Show Emojis on non-Windows platforms, instead of just macOS
- Change help text for `pulumi logs` to clarify the logs are specific
to a stack, not a project
- Display stack name when showing logs
- Have preamble text show to users for engine operations read a little
nicer
We were generating incorrect URLs for stacks on the new identity
model. When we don't have a repository in our ProjectIdentifier, the
URL's in the service use "-" for the repository and project names.
* Re-introduce interface for snapshot management
Snapshot management was done through the Update interface; this commit
splits it into a separate interface
* Put the SnapshotManager instance onto the engine context
* Remove SnapshotManager from planContext and updateActions now that it can be accessed by engine Context
PPCs are no longer a central concept to our model, but instead a
feature that that pulumi.com provides to some organizations. Let's
remove most mentions of PPCs except for cases where we really need to
talk about them (e.g. when a stack is actually hosted in a PPC instead
of just via the normal pulumi.com service)
Also remove some "in the Pulumi Cloud" messages from the CLI, as using
the Pulumi Cloud is now the only real way to use pulumi.
Fixespulumi/pulumi-service#1117
The cloud backend would fail if `nil` was passed in as the options to
use when creating a stack. However, we passed nil in places where we
allow the user to create stacks interacively. I noticed this while
dogfooding.
In the cloud backend, treat a `nil` opts as if the default set of
options was passed.
hese changes plumb basic support for cancellation through the engine.
Two types of cancellation are supported for all engine operations:
- Cancellation, which waits for the operation to drive itself to a safe
point before the operation returns, and
- Termination, which does not wait for the operation to drive itself
to a safe opint for the operation returns.
When updating local or managed stacks, a single ^C triggers cancellation
of any running operation; a second ^C will trigger termination.
Fixes#513, #1077.
This command cancels a stack's currently running update, if any. It can
be used to recover from the scenario in which an update is aborted
without marking the running update as complete. Once an update has been
cancelled, it is likely that the affected stack will need to be repaired
via an pair of export/import commands before future updates can succeed.
This is part of #1077.
This change implements a `pulumi refresh` command. It operates a bit
like `pulumi update`, and friends, in that it supports `--preview` and
`--diff`, along with the usual flags, and will update your checkpoint.
It works through substitution of the deploy.Source abstraction, which
generates a sequence of resource registration events. This new
deploy.RefreshSource takes in a prior checkpoint and will walk it,
refreshing the state via the associated resource providers by invoking
Read for each resource encountered, and merging the resulting state with
the prior checkpoint, to yield a new resource.Goal state. This state is
then fed through the engine in the usual ways with a few minor caveats:
namely, although the engine must generate steps for the logical
operations (permitting us to get nice summaries, progress, and diffs),
it mustn't actually carry them out because the state being imported
already reflects reality (a deleted resource has *already* been deleted,
so of course the engine need not perform the deletion). The diffing
logic also needs to know how to treat the case of refresh slightly
differently, because we are going to be diffing outputs and not inputs.
Note that support for managed stacks is not yet complete, since that
requires updates to the service to support a refresh endpoint. That
will be coming soon ...
This change introduces support for using the cloud backend when
`pulumi init` has not been run. When this is the case, we use the new
identity model, where a stack is referenced by an owner and a stack
name only.
There are a few things going on here:
- We add a new `--owner` flag to `pulumi stack init` that lets you
control what account a stack is created in.
- When listing stacks, we show stacks owned by you and any
organizations you are a member of. So, for example, I can do:
* `pulumi stack init my-great-stack`
* `pulumi stack init --owner pulumi my-great-stack`
To create a stack owned by my user and one owned by my
organization. When `pulumi stack ls` is run, you'll see both
stacks (since they are part of the same project).
- When spelling a stack on the CLI, an owner can be optionally
specified by prefixing the stack name with an owner name. For
example `my-great-stack` means the stack `my-great-stack` owned by
the current logged in user, where-as `pulumi/my-great-stack` would
be the stack owned by the `pulumi` organization
- `--all` can be passed to `pulumi stack ls` to see *all* stacks you
have access to, not just stacks tied to the current project.
Long term, a stack name alone will not be sufficent to address a
stack. Introduce a new `backend.StackReference` interface that allows
each backend to give an opaque stack reference that can be used across
operations.
This change removes the need to `pulumi init` when targeting the local
backend. A fair amount of the change lays the foundation that the next
set of changes to stop having `pulumi init` be used for cloud stacks
as well.
Previously, `pulumi init` logically did two things:
1. It created the bookkeeping directory for local stacks, this was
stored in `<repository-root>/.pulumi`, where `<repository-root>` was
the path to what we belived the "root" of your project was. In the
case of git repositories, this was the directory that contained your
`.git` folder.
2. It recorded repository information in
`<repository-root>/.pulumi/repository.json`. This was used by the
cloud backend when computing what project to interact with on
Pulumi.com
The new identity model will remove the need for (2), since we only
need an owner and stack name to fully qualify a stack on
pulumi.com, so it's easy enough to stop creating a folder just for
that.
However, for the local backend, we need to continue to retain some
information about stacks (e.g. checkpoints, history, etc). In
addition, we need to store our workspace settings (which today just
contains the selected stack) somehere.
For state stored by the local backend, we change the URL scheme from
`local://` to `local://<optional-root-path>`. When
`<optional-root-path>` is unset, it defaults to `$HOME`. We create our
`.pulumi` folder in that directory. This is important because stack
names now must be unique within the backend, but we have some tests
using local stacks which use fixed stack names, so each integration
test really wants its own "view" of the world.
For the workspace settings, we introduce a new `workspaces` directory
in `~/.pulumi`. In this folder we write the workspace settings file
for each project. The file name is the name of the project, combined
with the SHA1 of the path of the project file on disk, to ensure that
multiple pulumi programs with the same project name have different
workspace settings.
This does mean that moving a project's location on disk will cause the
CLI to "forget" what the selected stack was, which is unfortunate, but
not the end of the world. If this ends up being a big pain point, we
can certianly try to play games in the future (for example, if we saw
a .git folder in a parent folder, we could store data in there).
With respect to compatibility, we don't attempt to migrate older files
to their newer locations. For long lived stacks managed using the
local backend, we can provide information on where to move things
to. For all stacks (regardless of backend) we'll require the user to
`pulumi stack select` their stack again, but that seems like the
correct trade-off vs writing complicated upgrade code.
This field indicates the schema of the serialized deployment. This field
behaves identically to the `Version` field of
`PatchUpdateCheckpointRequest`.
This is part of pulumi/pulumi-service#1046
And make the deployment an opaque JSON message. The verison field
indicates the schema of the deployment. A missing version field will
behave as if the version was set to `1`. A version of `1` indicates that
the serialized deployment has the `DeploymentV1` schema.
This is part of pulumi/pulumi-service#1046.
This makes two minor tweaks to the login prompt:
1. Change the text so that it hyperlinks in most terminals, including
iTerm, in a way that doesn't include excess characters.
2. Disable echoing of the token.
* Lift snapshot management out of the engine
This PR is a prerequisite for parallelism by addressing a major problem
that the engine has to deal with when performing parallel resource
construction: parallel mutation of the global snapshot. This PR adds
a `SnapshotManager` type that is responsible for maintaining and
persisting the current resource snapshot. It serializes all reads and
writes to the global snapshot and persists the snapshot to persistent
storage upon every write.
As a side-effect of this, the core engine no longer needs to know about
snapshot management at all; all snapshot operations can be handled as
callbacks on deployment events. This will greatly simplify the
parallelization of the core engine.
Worth noting is that the core engine will still need to be able to read
the current snapshot, since it is interested in the dependency graphs
contained within. The full implications of that are out of scope of this
PR.
Remove dead code, Steps no longer need a reference to the plan iterator that created them
Fixing various issues that arise when bringing up pulumi-aws
Line length broke the build
Code review: remove dead field, fix yaml name error
Rebase against master, provide implementation of StackPersister for cloud backend
Code review feedback: comments on MutationStatus, style in snapshot.go
Code review feedback: move SnapshotManager to pkg/backend, change engine to use an interface SnapshotManager
Code review feedback: use a channel for synchronization
Add a comment and a new test
* Maintain two checkpoints, an immutable base and a mutable delta, and
periodically merge the two to produce snapshots
* Add a lot of tests - covers all of the non-error paths of BeginMutation and End
* Fix a test resource provider
* Add a few tests, fix a few issues
* Rebase against master, fixed merge
We've seen failures in CI where DNS lookups fail which cause our
operations against the service to fail, as well as other sorts of
timeouts.
Add a set of helper methods in a new httputil package that helps us do
retries on these operations, and then update our client library to use
them when we are doing GET requests. We also provide a way for non GET
requests to be retried, and use this when updating a lease (since it
is safe to retry multiple requests in this case).
This change does three major things:
1. Removes the ability to be logged into multiple clouds at the same
time. Previously, we supported being logged into multiple clouds at
the same time and the CLI would fan out requests and join responses
when needed. In general, this was only useful for Pulumi employees
that wanted run against multiple copies of the service (say production
and staging) but overall was very confusing (for example in the old
world a stack with the same identity could appear twice (since it was
in two backends) which the CLI didn't handle very well).
2. Stops treating the "local" backend as a special thing, from the
point of view of the CLI. Previouly we'd always connect to the local
backend and merge that data with whatever was in clouds we were
connected to. We had gestures like `--local` in `pulumi stack init`
that meant "use the local mode". Instead, to use the local mode now
you run `pulumi login --cloud-url local://` and then you are logged in
the local backend. Since you can only ever be logged into a single
backend, we can remove the `--local` and `--remote` flags from `pulumi
stack init`, it just now requires you to be logged in and creates a
stack in whatever back end you were logged into. When logging into the
local backend, you are not prompted for an access key.
3. Prompt for login in places where you have to log in, if you are not
already logged in.
As part of the new identity model, we're going to use tagging on
stacks to record metadata, let's create a bag for that, as well as a
few well known tag names that map to metadata we know we'll want to set.
This change adds a Pulumi Cloud Console URL at the end of an update
that went through the Pulumi Cloud API. This is very basic, and is
meant to be just the beginning of adding more cross-linking to the
service. I'm still thinking through the phrasing to use here.
