pulumi/cmd/config.go
Justin Van Patten 9d84f2e249
Initial support for passing URLs to new and up (#1727)
* Initial support for passing URLs to `new` and `up`

This PR adds initial support for `pulumi new` using Git under the covers
to manage Pulumi templates, providing the same experience as before.

You can now also optionally pass a URL to a Git repository, e.g.
`pulumi new [<url>]`, including subdirectories within the repository,
and arbitrary branches, tags, or commits.

The following commands result in the same behavior from the user's
perspective:
 - `pulumi new javascript`
 - `pulumi new https://github.com/pulumi/templates/templates/javascript`
 - `pulumi new https://github.com/pulumi/templates/tree/master/templates/javascript`
 - `pulumi new https://github.com/pulumi/templates/tree/HEAD/templates/javascript`

To specify an arbitrary branch, tag, or commit:
 - `pulumi new https://github.com/pulumi/templates/tree/<branch>/templates/javascript`
 - `pulumi new https://github.com/pulumi/templates/tree/<tag>/templates/javascript`
 - `pulumi new https://github.com/pulumi/templates/tree/<commit>/templates/javascript`

Branches and tags can include '/' separators, and `pulumi` will still
find the right subdirectory.

URLs to Gists are also supported, e.g.:
`pulumi new https://gist.github.com/justinvp/6673959ceb9d2ac5a14c6d536cb871a6`

If the specified subdirectory in the repository does not contain a
`Pulumi.yaml`, it will look for subdirectories within containing
`Pulumi.yaml` files, and prompt the user to choose a template, along the
lines of how `pulumi new` behaves when no template is specified.

The following commands result in the CLI prompting to choose a template:
 - `pulumi new`
 - `pulumi new https://github.com/pulumi/templates/templates`
 - `pulumi new https://github.com/pulumi/templates/tree/master/templates`
 - `pulumi new https://github.com/pulumi/templates/tree/HEAD/templates`

Of course, arbitrary branches, tags, or commits can be specified as well:
 - `pulumi new https://github.com/pulumi/templates/tree/<branch>/templates`
 - `pulumi new https://github.com/pulumi/templates/tree/<tag>/templates`
 - `pulumi new https://github.com/pulumi/templates/tree/<commit>/templates`

This PR also includes initial support for passing URLs to `pulumi up`,
providing a streamlined way to deploy installable cloud applications
with Pulumi, without having to manage source code locally before doing
a deployment.

For example, `pulumi up https://github.com/justinvp/aws` can be used to
deploy a sample AWS app. The stack can be updated with different
versions, e.g.
`pulumi up https://github.com/justinvp/aws/tree/v2 -s <stack-to-update>`

Config values can optionally be passed via command line flags, e.g.
`pulumi up https://github.com/justinvp/aws -c aws:region=us-west-2 -c foo:bar=blah`

Gists can also be used, e.g.
`pulumi up https://gist.github.com/justinvp/62fde0463f243fcb49f5a7222e51bc76`

* Fix panic when hitting ^C from "choose template" prompt

* Add description to templates

When running `pulumi new` without specifying a template, include the template description along with the name in the "choose template" display.

```
$ pulumi new
Please choose a template:
  aws-go                  A minimal AWS Go program
  aws-javascript          A minimal AWS JavaScript program
  aws-python              A minimal AWS Python program
  aws-typescript          A minimal AWS TypeScript program
> go                      A minimal Go program
  hello-aws-javascript    A simple AWS serverless JavaScript program
  javascript              A minimal JavaScript program
  python                  A minimal Python program
  typescript              A minimal TypeScript program
```

* React to changes to the pulumi/templates repo.

We restructured the `pulumi/templates` repo to have all the templates in the root instead of in a `templates` subdirectory, so make the change here to no longer look for templates in `templates`.

This also fixes an issue around using `Depth: 1` that I found while testing this. When a named template is used, we attempt to clone or pull from the `pulumi/templates` repo to `~/.pulumi/templates`. Having it go in this well-known directory allows us to maintain previous behavior around allowing offline use of templates. If we use `Depth: 1` for the initial clone, it will fail when attempting to pull when there are updates to the remote repository. Unfortunately, there's no built-in `--unshallow` support in `go-git` and setting a larger `Depth` doesn't appear to help. There may be a workaround, but for now, if we're cloning the pulumi templates directory to `~/.pulumi/templates`, we won't use `Depth: 1`. For template URLs, we will continue to use `Depth: 1` as we clone those to a temp directory (which gets deleted) that we'll never try to update.

* List available templates in help text

* Address PR Feedback

* Don't show "Installing dependencies" message for `up`

* Fix secrets handling

When prompting for config, if the existing stack value is a secret, keep it a secret and mask the prompt. If the template says it should be secret, make it a secret.

* Fix ${PROJECT} and ${DESCRIPTION} handling for `up`

Templates used with `up` should already have a filled-in project name and description, but if it's a `new`-style template, that has `${PROJECT}` and/or `${DESCRIPTION}`, be helpful and just replace these with better values.

* Fix stack handling

Add a bool `setCurrent` param to `requireStack` to control whether the current stack should be saved in workspace settings. For the `up <url>` case, we don't want to save. Also, split the `up` code into two separate functions: one for the `up <url>` case and another for the normal `up` case where you have workspace in your current directory. While we may be able to combine them back into a single function, right now it's a bit cleaner being separate, even with some small amount of duplication.

* Fix panic due to nil crypter

Lazily get the crypter only if needed inside `promptForConfig`.

* Embellish comment

* Harden isPreconfiguredEmptyStack check

Fix the code to check to make sure the URL specified on the command line matches the URL stored in the `pulumi:template` config value, and that the rest of the config from the stack satisfies the config requirements of the template.
2018-08-10 18:08:16 -07:00

451 lines
12 KiB
Go

// Copyright 2016-2018, Pulumi Corporation.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package cmd
import (
"fmt"
"io/ioutil"
"os"
"regexp"
"sort"
"strconv"
"strings"
zxcvbn "github.com/nbutton23/zxcvbn-go"
"github.com/pkg/errors"
"github.com/spf13/cobra"
"golang.org/x/crypto/ssh/terminal"
"github.com/pulumi/pulumi/pkg/backend"
"github.com/pulumi/pulumi/pkg/resource/config"
"github.com/pulumi/pulumi/pkg/tokens"
"github.com/pulumi/pulumi/pkg/util/cmdutil"
"github.com/pulumi/pulumi/pkg/workspace"
)
func newConfigCmd() *cobra.Command {
var stack string
var showSecrets bool
cmd := &cobra.Command{
Use: "config",
Short: "Manage configuration",
Long: "Lists all configuration values for a specific stack. To add a new configuration value, run\n" +
"'pulumi config set', to remove and existing value run 'pulumi config rm'. To get the value of\n" +
"for a specific configuration key, use 'pulumi config get <key-name>'.",
Args: cmdutil.NoArgs,
Run: cmdutil.RunFunc(func(cmd *cobra.Command, args []string) error {
opts := backend.DisplayOptions{
Color: cmdutil.GetGlobalColorization(),
}
stack, err := requireStack(stack, true, opts, true /*setCurrent*/)
if err != nil {
return err
}
return listConfig(stack, showSecrets)
}),
}
cmd.Flags().BoolVar(
&showSecrets, "show-secrets", false,
"Show secret values when listing config instead of displaying blinded values")
cmd.PersistentFlags().StringVarP(
&stack, "stack", "s", "",
"The name of the stack to operate on. Defaults to the current stack")
cmd.AddCommand(newConfigGetCmd(&stack))
cmd.AddCommand(newConfigRmCmd(&stack))
cmd.AddCommand(newConfigSetCmd(&stack))
cmd.AddCommand(newConfigRefreshCmd(&stack))
return cmd
}
func newConfigGetCmd(stack *string) *cobra.Command {
getCmd := &cobra.Command{
Use: "get <key>",
Short: "Get a single configuration value",
Args: cmdutil.SpecificArgs([]string{"key"}),
Run: cmdutil.RunFunc(func(cmd *cobra.Command, args []string) error {
opts := backend.DisplayOptions{
Color: cmdutil.GetGlobalColorization(),
}
s, err := requireStack(*stack, true, opts, true /*setCurrent*/)
if err != nil {
return err
}
key, err := parseConfigKey(args[0])
if err != nil {
return errors.Wrap(err, "invalid configuration key")
}
return getConfig(s, key)
}),
}
return getCmd
}
func newConfigRmCmd(stack *string) *cobra.Command {
rmCmd := &cobra.Command{
Use: "rm <key>",
Short: "Remove configuration value",
Args: cmdutil.SpecificArgs([]string{"key"}),
Run: cmdutil.RunFunc(func(cmd *cobra.Command, args []string) error {
opts := backend.DisplayOptions{
Color: cmdutil.GetGlobalColorization(),
}
s, err := requireStack(*stack, true, opts, true /*setCurrent*/)
if err != nil {
return err
}
key, err := parseConfigKey(args[0])
if err != nil {
return errors.Wrap(err, "invalid configuration key")
}
ps, err := workspace.DetectProjectStack(s.Name().StackName())
if err != nil {
return err
}
if ps.Config != nil {
delete(ps.Config, key)
}
return workspace.SaveProjectStack(s.Name().StackName(), ps)
}),
}
return rmCmd
}
func newConfigRefreshCmd(stack *string) *cobra.Command {
var force bool
refreshCmd := &cobra.Command{
Use: "refresh",
Short: "Update the local configuration based on the most recent deployment of the stack",
Args: cmdutil.NoArgs,
Run: cmdutil.RunFunc(func(cmd *cobra.Command, args []string) error {
opts := backend.DisplayOptions{
Color: cmdutil.GetGlobalColorization(),
}
// Ensure the stack exists.
s, err := requireStack(*stack, false, opts, true /*setCurrent*/)
if err != nil {
return err
}
c, err := backend.GetLatestConfiguration(commandContext(), s)
if err != nil {
return err
}
configPath, err := workspace.DetectProjectStackPath(s.Name().StackName())
if err != nil {
return err
}
ps, err := workspace.LoadProjectStack(configPath)
if err != nil {
return err
}
ps.Config = c
// If the configuration file doesn't exist, or force has been passed, save it in place.
if _, err = os.Stat(configPath); os.IsNotExist(err) || force {
return ps.Save(configPath)
}
// Otherwise we'll create a backup, let's figure out what name to use by adding ".bak" over and over
// until we get to a name not in use.
backupFile := configPath + ".bak"
for {
_, err = os.Stat(backupFile)
if os.IsNotExist(err) {
if err = os.Rename(configPath, backupFile); err != nil {
return errors.Wrap(err, "backing up existing configuration file")
}
fmt.Printf("backed up existing configuration file to %s\n", backupFile)
break
} else if err != nil {
return errors.Wrap(err, "backing up existing configuration file")
}
backupFile = backupFile + ".bak"
}
err = ps.Save(configPath)
if err == nil {
fmt.Printf("refreshed configuration for stack '%s'\n", s.Name().String())
}
return err
}),
}
refreshCmd.PersistentFlags().BoolVarP(
&force, "force", "f", false, "Overwrite configuration file, if it exists, without creating a backup")
return refreshCmd
}
func newConfigSetCmd(stack *string) *cobra.Command {
var plaintext bool
var secret bool
setCmd := &cobra.Command{
Use: "set <key> [value]",
Short: "Set configuration value",
Long: "Configuration values can be accessed when a stack is being deployed and used to configure behavior. \n" +
"If a value is not present on the command line, pulumi will prompt for the value. Multi-line values\n" +
"may be set by piping a file to standard in.",
Args: cmdutil.RangeArgs(1, 2),
Run: cmdutil.RunFunc(func(cmd *cobra.Command, args []string) error {
opts := backend.DisplayOptions{
Color: cmdutil.GetGlobalColorization(),
}
// Ensure the stack exists.
s, err := requireStack(*stack, true, opts, true /*setCurrent*/)
if err != nil {
return err
}
key, err := parseConfigKey(args[0])
if err != nil {
return errors.Wrap(err, "invalid configuration key")
}
var value string
switch {
case len(args) == 2:
value = args[1]
case !terminal.IsTerminal(int(os.Stdin.Fd())):
b, readerr := ioutil.ReadAll(os.Stdin)
if readerr != nil {
return readerr
}
value = cmdutil.RemoveTralingNewline(string(b))
case secret:
value, err = cmdutil.ReadConsoleNoEcho("value")
if err != nil {
return err
}
default:
value, err = cmdutil.ReadConsole("value")
if err != nil {
return err
}
}
// Encrypt the config value if needed.
var v config.Value
if secret {
c, cerr := backend.GetStackCrypter(s)
if cerr != nil {
return cerr
}
enc, eerr := c.EncryptValue(value)
if eerr != nil {
return eerr
}
v = config.NewSecureValue(enc)
} else {
v = config.NewValue(value)
// If we saved a plaintext configuration value, and --plaintext was not passed, warn the user.
if !plaintext && looksLikeSecret(key, value) {
return errors.Errorf(
"config value '%s' looks like a secret; "+
"rerun with --secret to encrypt it, or --plaintext if you meant to store in plaintext",
value)
}
}
ps, err := workspace.DetectProjectStack(s.Name().StackName())
if err != nil {
return err
}
ps.Config[key] = v
return workspace.SaveProjectStack(s.Name().StackName(), ps)
}),
}
setCmd.PersistentFlags().BoolVar(
&plaintext, "plaintext", false,
"Save the value as plaintext (unencrypted)")
setCmd.PersistentFlags().BoolVar(
&secret, "secret", false,
"Encrypt the value instead of storing it in plaintext")
return setCmd
}
func parseConfigKey(key string) (config.Key, error) {
// As a convience, we'll treat any key with no delimiter as if:
// <program-name>:config:<key> had been written instead
if !strings.Contains(key, tokens.TokenDelimiter) {
proj, err := workspace.DetectProject()
if err != nil {
return config.Key{}, err
}
return config.ParseKey(fmt.Sprintf("%s:config:%s", proj.Name, key))
}
return config.ParseKey(key)
}
func prettyKey(k config.Key) string {
proj, err := workspace.DetectProject()
if err != nil {
return fmt.Sprintf("%s:%s", k.Namespace(), k.Name())
}
return prettyKeyForProject(k, proj)
}
func prettyKeyForProject(k config.Key, proj *workspace.Project) string {
if k.Namespace() == string(proj.Name) {
return k.Name()
}
return fmt.Sprintf("%s:%s", k.Namespace(), k.Name())
}
func listConfig(stack backend.Stack, showSecrets bool) error {
ps, err := workspace.DetectProjectStack(stack.Name().StackName())
if err != nil {
return err
}
cfg := ps.Config
// By default, we will use a blinding decrypter to show '******'. If requested, display secrets in plaintext.
var decrypter config.Decrypter
if cfg.HasSecureValue() && showSecrets {
decrypter, err = backend.GetStackCrypter(stack)
if err != nil {
return err
}
} else {
decrypter = config.NewBlindingDecrypter()
}
fullKey := func(k config.Key) string {
return fmt.Sprintf("%s:%s", k.Namespace(), k.Name())
}
// Devote 48 characters to the config key, unless there's a key longer, in which case use that.
maxkey := 48
for key := range cfg {
if len(fullKey(key)) > maxkey {
maxkey = len(fullKey(key))
}
}
fmt.Printf("%-"+strconv.Itoa(maxkey)+"s %-48s\n", "KEY", "VALUE")
var keys config.KeyArray
for key := range cfg {
// Note that we use the fully qualified module member here instead of a `prettyKey`, this lets us ensure
// that all the config values for the current program are displayed next to one another in the output.
keys = append(keys, key)
}
sort.Sort(keys)
for _, key := range keys {
decrypted, err := cfg[key].Value(decrypter)
if err != nil {
return errors.Wrap(err, "could not decrypt configuration value")
}
fmt.Printf("%-"+strconv.Itoa(maxkey)+"s %-48s\n", prettyKey(key), decrypted)
}
return nil
}
func getConfig(stack backend.Stack, key config.Key) error {
ps, err := workspace.DetectProjectStack(stack.Name().StackName())
if err != nil {
return err
}
cfg := ps.Config
if v, ok := cfg[key]; ok {
var d config.Decrypter
if v.Secure() {
var err error
if d, err = backend.GetStackCrypter(stack); err != nil {
return errors.Wrap(err, "could not create a decrypter")
}
} else {
d = config.NewPanicCrypter()
}
raw, err := v.Value(d)
if err != nil {
return errors.Wrap(err, "could not decrypt configuration value")
}
fmt.Printf("%v\n", raw)
return nil
}
return errors.Errorf(
"configuration key '%s' not found for stack '%s'", prettyKey(key), stack.Name())
}
var (
// keyPattern is the regular expression a configuration key must match before we check (and error) if we think
// it is a password
keyPattern = regexp.MustCompile("(?i)passwd|pass|password|pwd|secret|token")
)
const (
// maxEntropyCheckLength is the maximum length of a possible secret for entropy checking.
maxEntropyCheckLength = 16
// entropyThreshold is the total entropy threshold a potential secret needs to pass before being flagged.
entropyThreshold = 80.0
// entropyCharThreshold is the per-char entropy threshold a potential secret needs to pass before being flagged.
entropyPerCharThreshold = 3.0
)
// looksLikeSecret returns true if a configuration value "looks" like a secret. This is always going to be a heuristic
// that suffers from false positives, but is better (a) than our prior approach of unconditionally printing a warning
// for all plaintext values, and (b) to be paranoid about such things. Inspired by the gas linter and securego project.
func looksLikeSecret(k config.Key, v string) bool {
if !keyPattern.MatchString(k.Name()) {
return false
}
if len(v) > maxEntropyCheckLength {
v = v[:maxEntropyCheckLength]
}
// Compute the strength use the resulting entropy to flag whether this looks like a secret.
info := zxcvbn.PasswordStrength(v, nil)
entropyPerChar := info.Entropy / float64(len(v))
return (info.Entropy >= entropyThreshold ||
(info.Entropy >= (entropyThreshold/2) && entropyPerChar >= entropyPerCharThreshold))
}