The linter had been warning me for a while that some comments we had
confused it. I fixed this. Then the linter found a place where we
were ignoring a return value. Looking at it, it feels like we want to
continue in this case, so I just `contract.IgnoreError`'d it.
Validate the value is well formed much earlier so you don't end up
seeing you've picked a bad value in the middle of trying to create
your new stack. Update the helptext to list currently supported
values.
Fixes#2727
The change does two things:
- Reorders some calls in the CLI to prevent trying to create a secrets
manager twice (which would end up prompting twice).
- Adds a cache inside the passphrase secrets manager such that when
decrypting a deployment, we can re-use the one created earlier in
the update. This is sort of a hack, but is needed because otherwise
we would fail to decrypt the deployment, meaning that if you had a
secret value in your deployment *and* you were using local
passphrase encryption *and* you had not set PULUMI_CONFIG_PASSPHRASE
you would get an error asking you to do so.
Fixes#2729
This change allows using the passphrase secrets manager when creating
a stack managed by the Pulumi service. `pulumi stack init`, `pulumi
new` and `pulumi up` all learned a new optional argument
`--secrets-provider` which can be set to "passphrase" to force the
passphrase based secrets provider to be used. When unset the default
secrets provider is used based on the backend (for local stacks this
is passphrase, for remote stacks, it is the key managed by the pulumi
service).
As part of this change, we also initialize the secrets manager when a
stack is created, instead of waiting for the first time a secret
config value is stored. We do this so that if an update is run using
`pulumi.secret` before any secret configuration values are used, we
already have the correct encryption method selected for a stack.
We have now done all the work needed such that we can start passing
the passphrase and service secrets managers into the engine to be used
when storing values.
With this change `pulumi up` will now correctly encrypt secrets
instead of just base64 encoding them.
When preforming an update, require that a secrets manager is passed in
as part of the `backend.UpdateOperation` bag and use it. The CLI now
passes this in (it still uses the default base64 secrets manager, so
this is just code motion into a high layer, since the CLI will be the
one to choose what secrets manager to use based on project settings).
Half of the call sites didn't care about these values and with the
secrets work the ergonmics of calling this method when it has to
return serialized ouputs isn't great. Move the serialization for this
into the CLI itself, as it was the only place that cared to do
this (so it could display things to end users).
We require configuration to preform updates (as well as previews,
destroys and refreshes). Because of how everything evolved, loading
this configuration (and finding the coresponding decrypter) was
implemented in both the file and http backends, which wasn't great.
Refactor things such that the CLI itself builds out this information
and passes it along to the backend to preform operations. This means
less code duplicated between backends and less places the backend
assume things about the existence of `Pulumi.yaml` files and in
general makes the interface more plesent to use for others uses.
When constructing a Deployment (which is a plaintext representation of
a Snapshot), ensure that we encrypt secret values. To do so, we
introduce a new type `secrets.Manager` which is able to encrypt and
decrypt values. In addition, it is able to reflect information about
itself that can be stored in the deployment such that we can
deserialize the deployment into a snapshot (decrypting the values in
the process) without external knowledge about how it was encrypted.
The ability to do this is import for allowing stack references to
work, since two stacks may not use the same manager (or they will use
the same type of manager, but have different state).
The state value is stored in plaintext in the deployment, so it **must
not** contain sensitive data.
A sample manager, which just base64 encodes and decodes strings is
provided, as it useful for testing. We will allow it to be varried
soon.
The Pulumi CLI currently has two "display modes" -- one for rendering
diffs and one for rendering program updates. Neither of these is
particularly well-suited to `pulumi query`, which essentially needs to
render stdout from the query program verbatim.
This commit will add a separate display mode for this purpose:
* In interactive mode, `pulumi query` will use the display spinner to
say "running query". In non-interactive mode, this will be omitted.
* Query mode will print stdout from the query program verbatim, but
ignore `diag.Infoerr` so that they're not double-printed when they're
emitted again as error events.
* Query mode also does not double-print newlines at the end of diag
events.
Originally, `pulumi new` did not run `up` after generating a project. To
help users get a deployed stack as quickly as possible, we changed
`pulumi new` to run an initial deployment at the end of its operation.
Users would first see a preview and get to decide whether to proceed
with an initial deployment, and then continue to iterate from there.
Note that this would only happen for nodejs projects
(TypeScript/JavaScript). We would not run `up` for Python projects as we
require the user to run `pip install` in a virtualenv, so we'd print
instructions with the necessary commands the user must run instead.
Running `up` as part of `pulumi new` for nodejs projects has ended up
being more confusing than helpful for new users, and annoying for
experienced users. New users aren't expecting `pulumi new` to run an
initial deployment after generating the project (they haven't even
looked at the project source yet). Experienced users find it frustrating
as you typically want to just generate the project, and don't want to
have to wait for the preview just to decline running the update.
This change reverts `pulumi new` back to not running `up` automatically
for nodejs projects. Instead, like with Python projects, at the end of
the operation, we print instructions to the user to run `pulumi up` to
deploy the project.
Some changes to `pulumi new` to improve the experience:
- Color the default values for config differently to make them stand
out better
- Mention that `new` will also perform an initial deployment
- Add more vertical whitespace between steps in the process
- Print message indicating the "Installing dependencies" step is
complete
- After "project is ready to go", add a note about doing an initial
deployment
- Output follow-up command to run when an update fails
- Go back to showing the `npm install` output as `npm` doesn't always
return an error code when it runs into problems
- Ensure new projects have a project name in line with what we'd like
to enforce going forward
- Do more aggresive validation during the interactive prompts during
`pulumi new`
- Fix an issue where the interactive prompt rendered weridly when
there was a validation error
Contributes to #1988Fixes#1441
We currently leave behind the template section inside Pulumi.yaml after
`pulumi new`. While we may eventually make use of it, we're not
currently using it, so it's cleaner to just remove it for now.
This option allows the user to override the file used to fetch and store
configuration information for a stack. It is available for the config,
destroy, logs, preview, refresh, and up commands.
Note that this option is not persistent: if it is not specified, the
stack's default configuration will be used. If an alternate config file
is used exclusively for a stack, it must be specified to all commands
that interact with that stack.
This option can be used to share plaintext configuration across multiple
stacks. It cannot be used to share secret configuration, as secrets are
associated with a particular stack and cannot be decryptex by other
stacks.
`pulumi up <arg>` does not currently support template names like `pulumi
new`; `up` is hard-coded to only support URLs to templates. This
prevents us from displaying shorter `$ pulumi up ...` commands in the
Pulumi Console when the URL is to one of our standard Pulumi templates.
In such cases, we'd like to be able to show the command with just the
template name instead of a full URL to the template in the
pulumi/templates repo.
This changes `up` to support the standard Pulumi template names
just like `new`.
Also, while making changes here, if a URL specified to `up` contains
multiple templates in subdirectories, allow the user to choose the
template, just like with `new`.
Downlevel versions of the Pulumi Node SDK assumed that a parallelism
level of zero implied serial execution, which current CLIs use to signal
unbounded parallelism. This commit works around the downlevel issue by
using math.MaxInt32 to signal unbounded parallelism.
Some providers (namely Kubernetes) require unbounded parallelism in
order to function correctly. This commit enables the engine to operate
in a mode with unbounded parallelism and switches to that mode by
default.
This adds an option, --suppress-outputs, to many commands, to
avoid printing stack outputs for stacks that might contain sensitive
information in their outputs (like key material, and whatnot).
Fixespulumi/pulumi#2028.
It is easy to continually type `pulumi apply` from years of muscle memory using
Terraform - this commit makes the CLI suggest `up` when the `apply` subcommand
is used, similar to how `plan` suggests `preview`.
Right now, we only support --non-interactive in a few places (up,
refresh, destroy, etc). Over time, we've added it to more (like new).
And now, as we're working on better Docker support (pulumi/pulumi#1991),
we want to support this more globally, so we can, for example, avoid
popping up a web browser inside a Docker contain for logging in.
So, this change makes --non-interactive a global flag. Because it is
a persistent flag, it still works in the old positions, so this isn't
a breaking change to existing commands that use it.
Previously `new` was operating under the assumption that it was always
going to be creating a new project/stack, and would always prompt for
these values. However, we want to be able to use `new` to pull down the
source for an existing stack. This change adds a `--stack` flag to `new`
that can be used to specify an existing stack. If the specified stack
already exists, we won't prompt for the project name/description, and
instead just use the existing stack's values. If `--stack` is specified,
but doesn't already exist, it will just use that as the stack name
(instead of prompting) when creating the stack. `new` also now handles
configuration like `up <url>`: if the stack is a preconfigured empty
stack (i.e. it was created/configured in the Pulumi Console via Pulumi
Button or New Project), we will use the existing stack's config without
prompting. Otherwise we will prompt for config, and just like `up
<url>`, we'll use the existing stack's config values as defaults when
prompting, or if the stack didn't exist, use the defaults from the
template.
Previously `up <url>`'s handling of the project name/description wasn't
correct: it would always automatically use the values from the template
without prompting. Now, just like `new`:
- When updating an existing stack, it will not prompt, and just use the
existing stack's values.
- When creating a new stack, it will prompt for the project
name/description, using the defaults from the template.
This PR consolidates some of the `new`/`up` implementation so it shares
code for this functionality. There's definitely opportunities for a lot
more code reuse, but that cleanup can happen down the line if/when we
have the cycles.
We generally want examples and apps to be authored such that they are
clonable/deployable as-is without using new/up (and want to
encourage this). That means no longer using the ${PROJECT} and
${DESCRIPTION} replacement strings in Pulumi.yaml and other text files.
Instead, good default project names and descriptions should be specified
in Pulumi.yaml and elsewhere.
We'll use the specified values as defaults when prompting the user, and
then directly serialize/save the values to Pulumi.yaml when configuring
the user's project. This does mean that name in package.json (for nodejs
projects) won't be updated if it isn't using ${PROJECT}, but that's OK.
Our templates in the pulumi/templates repo will still use
${PROJECT}/${DESCRIPTION} for now, to continue to work well with v0.15
of the CLI. After that version is no longer in use, we can update the
templates to no longer use the replacement strings and delete the code
in the CLI that deals with it.
This change implements the same preview behavior we have for
cloud stacks, in pkg/backend/httpbe, for local stacks, in
pkg/backend/filebe. This mostly required just refactoring bits
and pieces so that we can share more of the code, although it
does still entail quite a bit of redundancy. In particular, the
apply functions for both backends are now so close to being
unified, but still require enough custom logic that it warrants
keeping them separate (for now...)
This simply refactors all the display logic out of the
pkg/backend/filestate package. This helps to gear us up to better unify
this logic between the filestate and httpstate backends.
Furthermore, this really ought to be in its own non-backend,
CLI-specific package, but I'm taking one step at a time here.
Previously, we were only saving config values specified on the command line (via --config/-c) for the URL case. This change fixes things to save these config values for the non-URL path.
Previously, we only supported config keys that included a ':' delimiter
in config keys specified in the template manifest and in `-c` flags to
`new` and `up`. This prevented the use of project keys in the template
manifest and made it more difficult to pass such keys with `-c`,
effectively preventing the use of `new pulumi.Config()` in project code.
This change fixes this by allowing config keys that don't have a
delimiter in the template manifest and `-c` flags. In such cases, the
project name is automatically prepended behind the scenes, the same as
what `pulumi config set` does.
We already walk through creating a stack and prompting for required
config, and then tell the user to run `pulumi up` to do an initial
deployment. Instead, just proceed with the `up` automatically.
Replace the Source-based implementation of refresh with a phase that
runs as the first part of plan execution and rewrites the snapshot in-memory.
In order to fit neatly within the existing framework for resource operations,
these changes introduce a new kind of step, RefreshStep, to represent
refreshes. RefreshSteps operate similar to ReadSteps but do not imply that
the resource being read is not managed by Pulumi.
In addition to the refresh reimplementation, these changes incorporate those
from #1394 to run refresh in the integration test framework.
Fixes#1598.
Fixespulumi/pulumi-terraform#165.
Contributes to #1449.
* Add a list of in-flight operations to the deployment
This commit augments 'DeploymentV2' with a list of operations that are
currently in flight. This information is used by the engine to keep
track of whether or not a particular deployment is in a valid state.
The SnapshotManager is responsible for inserting and removing operations
from the in-flight operation list. When the engine registers an intent
to perform an operation, SnapshotManager inserts an Operation into this
list and saves it to the snapshot. When an operation completes, the
SnapshotManager removes it from the snapshot. From this, the engine can
infer that if it ever sees a deployment with pending operations, the
Pulumi CLI must have crashed or otherwise abnormally terminated before
seeing whether or not an operation completed successfully.
To remedy this state, this commit also adds code to 'pulumi stack
import' that clears all pending operations from a deployment, as well as
code to plan generation that will reject any deployments that have
pending operations present.
At the CLI level, if we see that we are in a state where pending
operations were in-flight when the engine died, we'll issue a
human-friendly error message that indicates which resources are in a bad
state and how to recover their stack.
* CR: Multi-line string literals, renaming in-flight -> pending
* CR: Add enum to apitype for operation type, also name status -> type for clarity
* Fix the yaml type
* Fix missed renames
* Add implementation for lifecycle_test.go
* Rebase against master
* Initial support for passing URLs to `new` and `up`
This PR adds initial support for `pulumi new` using Git under the covers
to manage Pulumi templates, providing the same experience as before.
You can now also optionally pass a URL to a Git repository, e.g.
`pulumi new [<url>]`, including subdirectories within the repository,
and arbitrary branches, tags, or commits.
The following commands result in the same behavior from the user's
perspective:
- `pulumi new javascript`
- `pulumi new https://github.com/pulumi/templates/templates/javascript`
- `pulumi new https://github.com/pulumi/templates/tree/master/templates/javascript`
- `pulumi new https://github.com/pulumi/templates/tree/HEAD/templates/javascript`
To specify an arbitrary branch, tag, or commit:
- `pulumi new https://github.com/pulumi/templates/tree/<branch>/templates/javascript`
- `pulumi new https://github.com/pulumi/templates/tree/<tag>/templates/javascript`
- `pulumi new https://github.com/pulumi/templates/tree/<commit>/templates/javascript`
Branches and tags can include '/' separators, and `pulumi` will still
find the right subdirectory.
URLs to Gists are also supported, e.g.:
`pulumi new https://gist.github.com/justinvp/6673959ceb9d2ac5a14c6d536cb871a6`
If the specified subdirectory in the repository does not contain a
`Pulumi.yaml`, it will look for subdirectories within containing
`Pulumi.yaml` files, and prompt the user to choose a template, along the
lines of how `pulumi new` behaves when no template is specified.
The following commands result in the CLI prompting to choose a template:
- `pulumi new`
- `pulumi new https://github.com/pulumi/templates/templates`
- `pulumi new https://github.com/pulumi/templates/tree/master/templates`
- `pulumi new https://github.com/pulumi/templates/tree/HEAD/templates`
Of course, arbitrary branches, tags, or commits can be specified as well:
- `pulumi new https://github.com/pulumi/templates/tree/<branch>/templates`
- `pulumi new https://github.com/pulumi/templates/tree/<tag>/templates`
- `pulumi new https://github.com/pulumi/templates/tree/<commit>/templates`
This PR also includes initial support for passing URLs to `pulumi up`,
providing a streamlined way to deploy installable cloud applications
with Pulumi, without having to manage source code locally before doing
a deployment.
For example, `pulumi up https://github.com/justinvp/aws` can be used to
deploy a sample AWS app. The stack can be updated with different
versions, e.g.
`pulumi up https://github.com/justinvp/aws/tree/v2 -s <stack-to-update>`
Config values can optionally be passed via command line flags, e.g.
`pulumi up https://github.com/justinvp/aws -c aws:region=us-west-2 -c foo:bar=blah`
Gists can also be used, e.g.
`pulumi up https://gist.github.com/justinvp/62fde0463f243fcb49f5a7222e51bc76`
* Fix panic when hitting ^C from "choose template" prompt
* Add description to templates
When running `pulumi new` without specifying a template, include the template description along with the name in the "choose template" display.
```
$ pulumi new
Please choose a template:
aws-go A minimal AWS Go program
aws-javascript A minimal AWS JavaScript program
aws-python A minimal AWS Python program
aws-typescript A minimal AWS TypeScript program
> go A minimal Go program
hello-aws-javascript A simple AWS serverless JavaScript program
javascript A minimal JavaScript program
python A minimal Python program
typescript A minimal TypeScript program
```
* React to changes to the pulumi/templates repo.
We restructured the `pulumi/templates` repo to have all the templates in the root instead of in a `templates` subdirectory, so make the change here to no longer look for templates in `templates`.
This also fixes an issue around using `Depth: 1` that I found while testing this. When a named template is used, we attempt to clone or pull from the `pulumi/templates` repo to `~/.pulumi/templates`. Having it go in this well-known directory allows us to maintain previous behavior around allowing offline use of templates. If we use `Depth: 1` for the initial clone, it will fail when attempting to pull when there are updates to the remote repository. Unfortunately, there's no built-in `--unshallow` support in `go-git` and setting a larger `Depth` doesn't appear to help. There may be a workaround, but for now, if we're cloning the pulumi templates directory to `~/.pulumi/templates`, we won't use `Depth: 1`. For template URLs, we will continue to use `Depth: 1` as we clone those to a temp directory (which gets deleted) that we'll never try to update.
* List available templates in help text
* Address PR Feedback
* Don't show "Installing dependencies" message for `up`
* Fix secrets handling
When prompting for config, if the existing stack value is a secret, keep it a secret and mask the prompt. If the template says it should be secret, make it a secret.
* Fix ${PROJECT} and ${DESCRIPTION} handling for `up`
Templates used with `up` should already have a filled-in project name and description, but if it's a `new`-style template, that has `${PROJECT}` and/or `${DESCRIPTION}`, be helpful and just replace these with better values.
* Fix stack handling
Add a bool `setCurrent` param to `requireStack` to control whether the current stack should be saved in workspace settings. For the `up <url>` case, we don't want to save. Also, split the `up` code into two separate functions: one for the `up <url>` case and another for the normal `up` case where you have workspace in your current directory. While we may be able to combine them back into a single function, right now it's a bit cleaner being separate, even with some small amount of duplication.
* Fix panic due to nil crypter
Lazily get the crypter only if needed inside `promptForConfig`.
* Embellish comment
* Harden isPreconfiguredEmptyStack check
Fix the code to check to make sure the URL specified on the command line matches the URL stored in the `pulumi:template` config value, and that the rest of the config from the stack satisfies the config requirements of the template.
