* Initial support for passing URLs to `new` and `up` This PR adds initial support for `pulumi new` using Git under the covers to manage Pulumi templates, providing the same experience as before. You can now also optionally pass a URL to a Git repository, e.g. `pulumi new [<url>]`, including subdirectories within the repository, and arbitrary branches, tags, or commits. The following commands result in the same behavior from the user's perspective: - `pulumi new javascript` - `pulumi new https://github.com/pulumi/templates/templates/javascript` - `pulumi new https://github.com/pulumi/templates/tree/master/templates/javascript` - `pulumi new https://github.com/pulumi/templates/tree/HEAD/templates/javascript` To specify an arbitrary branch, tag, or commit: - `pulumi new https://github.com/pulumi/templates/tree/<branch>/templates/javascript` - `pulumi new https://github.com/pulumi/templates/tree/<tag>/templates/javascript` - `pulumi new https://github.com/pulumi/templates/tree/<commit>/templates/javascript` Branches and tags can include '/' separators, and `pulumi` will still find the right subdirectory. URLs to Gists are also supported, e.g.: `pulumi new https://gist.github.com/justinvp/6673959ceb9d2ac5a14c6d536cb871a6` If the specified subdirectory in the repository does not contain a `Pulumi.yaml`, it will look for subdirectories within containing `Pulumi.yaml` files, and prompt the user to choose a template, along the lines of how `pulumi new` behaves when no template is specified. The following commands result in the CLI prompting to choose a template: - `pulumi new` - `pulumi new https://github.com/pulumi/templates/templates` - `pulumi new https://github.com/pulumi/templates/tree/master/templates` - `pulumi new https://github.com/pulumi/templates/tree/HEAD/templates` Of course, arbitrary branches, tags, or commits can be specified as well: - `pulumi new https://github.com/pulumi/templates/tree/<branch>/templates` - `pulumi new https://github.com/pulumi/templates/tree/<tag>/templates` - `pulumi new https://github.com/pulumi/templates/tree/<commit>/templates` This PR also includes initial support for passing URLs to `pulumi up`, providing a streamlined way to deploy installable cloud applications with Pulumi, without having to manage source code locally before doing a deployment. For example, `pulumi up https://github.com/justinvp/aws` can be used to deploy a sample AWS app. The stack can be updated with different versions, e.g. `pulumi up https://github.com/justinvp/aws/tree/v2 -s <stack-to-update>` Config values can optionally be passed via command line flags, e.g. `pulumi up https://github.com/justinvp/aws -c aws:region=us-west-2 -c foo:bar=blah` Gists can also be used, e.g. `pulumi up https://gist.github.com/justinvp/62fde0463f243fcb49f5a7222e51bc76` * Fix panic when hitting ^C from "choose template" prompt * Add description to templates When running `pulumi new` without specifying a template, include the template description along with the name in the "choose template" display. ``` $ pulumi new Please choose a template: aws-go A minimal AWS Go program aws-javascript A minimal AWS JavaScript program aws-python A minimal AWS Python program aws-typescript A minimal AWS TypeScript program > go A minimal Go program hello-aws-javascript A simple AWS serverless JavaScript program javascript A minimal JavaScript program python A minimal Python program typescript A minimal TypeScript program ``` * React to changes to the pulumi/templates repo. We restructured the `pulumi/templates` repo to have all the templates in the root instead of in a `templates` subdirectory, so make the change here to no longer look for templates in `templates`. This also fixes an issue around using `Depth: 1` that I found while testing this. When a named template is used, we attempt to clone or pull from the `pulumi/templates` repo to `~/.pulumi/templates`. Having it go in this well-known directory allows us to maintain previous behavior around allowing offline use of templates. If we use `Depth: 1` for the initial clone, it will fail when attempting to pull when there are updates to the remote repository. Unfortunately, there's no built-in `--unshallow` support in `go-git` and setting a larger `Depth` doesn't appear to help. There may be a workaround, but for now, if we're cloning the pulumi templates directory to `~/.pulumi/templates`, we won't use `Depth: 1`. For template URLs, we will continue to use `Depth: 1` as we clone those to a temp directory (which gets deleted) that we'll never try to update. * List available templates in help text * Address PR Feedback * Don't show "Installing dependencies" message for `up` * Fix secrets handling When prompting for config, if the existing stack value is a secret, keep it a secret and mask the prompt. If the template says it should be secret, make it a secret. * Fix ${PROJECT} and ${DESCRIPTION} handling for `up` Templates used with `up` should already have a filled-in project name and description, but if it's a `new`-style template, that has `${PROJECT}` and/or `${DESCRIPTION}`, be helpful and just replace these with better values. * Fix stack handling Add a bool `setCurrent` param to `requireStack` to control whether the current stack should be saved in workspace settings. For the `up <url>` case, we don't want to save. Also, split the `up` code into two separate functions: one for the `up <url>` case and another for the normal `up` case where you have workspace in your current directory. While we may be able to combine them back into a single function, right now it's a bit cleaner being separate, even with some small amount of duplication. * Fix panic due to nil crypter Lazily get the crypter only if needed inside `promptForConfig`. * Embellish comment * Harden isPreconfiguredEmptyStack check Fix the code to check to make sure the URL specified on the command line matches the URL stored in the `pulumi:template` config value, and that the rest of the config from the stack satisfies the config requirements of the template.
525 lines
14 KiB
Go
525 lines
14 KiB
Go
// Copyright 2016-2018, Pulumi Corporation.
|
|
//
|
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
|
// you may not use this file except in compliance with the License.
|
|
// You may obtain a copy of the License at
|
|
//
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
//
|
|
// Unless required by applicable law or agreed to in writing, software
|
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
// See the License for the specific language governing permissions and
|
|
// limitations under the License.
|
|
|
|
package workspace
|
|
|
|
import (
|
|
"fmt"
|
|
"io/ioutil"
|
|
"os"
|
|
"os/user"
|
|
"path/filepath"
|
|
"runtime"
|
|
"strings"
|
|
|
|
git "gopkg.in/src-d/go-git.v4"
|
|
"gopkg.in/src-d/go-git.v4/plumbing"
|
|
|
|
"github.com/pkg/errors"
|
|
"github.com/pulumi/pulumi/pkg/resource/config"
|
|
"github.com/pulumi/pulumi/pkg/tokens"
|
|
"github.com/pulumi/pulumi/pkg/util/contract"
|
|
"github.com/pulumi/pulumi/pkg/util/gitutil"
|
|
)
|
|
|
|
const (
|
|
defaultProjectName = "project"
|
|
|
|
pulumiTemplateGitRepository = "https://github.com/pulumi/templates.git"
|
|
|
|
// This file will be ignored when copying from the template cache to
|
|
// a project directory.
|
|
legacyPulumiTemplateManifestFile = ".pulumi.template.yaml"
|
|
|
|
// pulumiLocalTemplatePathEnvVar is a path to the folder where templates are stored.
|
|
// It is used in sandboxed environments where the classic template folder may not be writable.
|
|
pulumiLocalTemplatePathEnvVar = "PULUMI_TEMPLATE_PATH"
|
|
)
|
|
|
|
// TemplateRepository represents a repository of templates.
|
|
type TemplateRepository struct {
|
|
Root string // The full path to the root directory of the repository.
|
|
SubDirectory string // The full path to the sub directory within the repository.
|
|
ShouldDelete bool // Whether the root directory should be deleted.
|
|
}
|
|
|
|
// Delete deletes the template repository.
|
|
func (repo TemplateRepository) Delete() error {
|
|
if repo.ShouldDelete {
|
|
return os.RemoveAll(repo.Root)
|
|
}
|
|
return nil
|
|
}
|
|
|
|
// Templates lists the templates in the repository.
|
|
func (repo TemplateRepository) Templates() ([]Template, error) {
|
|
path := repo.SubDirectory
|
|
|
|
info, err := os.Stat(path)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
|
|
// If it's a file, look in its directory.
|
|
if !info.IsDir() {
|
|
path = filepath.Dir(path)
|
|
}
|
|
|
|
// See if there's a Pulumi.yaml in the directory.
|
|
template, err := LoadTemplate(path)
|
|
if err != nil && !os.IsNotExist(err) {
|
|
return nil, err
|
|
} else if err == nil {
|
|
return []Template{template}, nil
|
|
}
|
|
|
|
// Otherwise, read all subdirectories to find the ones
|
|
// that contain a Pulumi.yaml.
|
|
infos, err := ioutil.ReadDir(path)
|
|
if err != nil {
|
|
return nil, err
|
|
}
|
|
|
|
var result []Template
|
|
for _, info := range infos {
|
|
if info.IsDir() {
|
|
name := info.Name()
|
|
|
|
// Ignore the .git directory.
|
|
if name == GitDir {
|
|
continue
|
|
}
|
|
|
|
template, err := LoadTemplate(filepath.Join(path, name))
|
|
if err != nil && !os.IsNotExist(err) {
|
|
return nil, err
|
|
} else if err == nil {
|
|
result = append(result, template)
|
|
}
|
|
}
|
|
}
|
|
return result, nil
|
|
}
|
|
|
|
// Template represents a project template.
|
|
type Template struct {
|
|
Dir string // The directory containing Pulumi.yaml.
|
|
Name string // The name of the template.
|
|
Description string // Description of the template.
|
|
Quickstart string // Optional text to be displayed after template creation.
|
|
Config map[config.Key]ProjectTemplateConfigValue // Optional template config.
|
|
|
|
ProjectName string // Name of the project.
|
|
ProjectDescription string // Optional description of the project.
|
|
}
|
|
|
|
// cleanupLegacyTemplateDir deletes an existing ~/.pulumi/templates directory if it isn't a git repository.
|
|
func cleanupLegacyTemplateDir() error {
|
|
templateDir, err := GetTemplateDir()
|
|
if err != nil {
|
|
return err
|
|
}
|
|
|
|
// See if the template directory is a Git repository.
|
|
if _, err = git.PlainOpen(templateDir); err != nil {
|
|
// If the repository doesn't exist, it's a legacy directory.
|
|
// Delete the entire template directory and all children.
|
|
if err == git.ErrRepositoryNotExists {
|
|
return os.RemoveAll(templateDir)
|
|
}
|
|
|
|
return err
|
|
}
|
|
|
|
return nil
|
|
}
|
|
|
|
// IsTemplateURL returns true if templateNameOrURL starts with "https://".
|
|
func IsTemplateURL(templateNameOrURL string) bool {
|
|
return strings.HasPrefix(templateNameOrURL, "https://")
|
|
}
|
|
|
|
// RetrieveTemplates retrieves a "template repository" based on the specified name or URL.
|
|
func RetrieveTemplates(templateNameOrURL string, offline bool) (TemplateRepository, error) {
|
|
if IsTemplateURL(templateNameOrURL) {
|
|
return retrieveURLTemplates(templateNameOrURL, offline)
|
|
}
|
|
return retrievePulumiTemplates(templateNameOrURL, offline)
|
|
}
|
|
|
|
// retrieveURLTemplates retrieves the "template repository" at the specified URL.
|
|
func retrieveURLTemplates(rawurl string, offline bool) (TemplateRepository, error) {
|
|
if offline {
|
|
return TemplateRepository{}, errors.Errorf("cannot use %s offline", rawurl)
|
|
}
|
|
|
|
var err error
|
|
|
|
// Create a temp dir.
|
|
var temp string
|
|
if temp, err = ioutil.TempDir("", "pulumi-template-"); err != nil {
|
|
return TemplateRepository{}, err
|
|
}
|
|
|
|
var fullPath string
|
|
if fullPath, err = RetrieveTemplate(rawurl, temp); err != nil {
|
|
return TemplateRepository{}, err
|
|
}
|
|
|
|
return TemplateRepository{
|
|
Root: temp,
|
|
SubDirectory: fullPath,
|
|
ShouldDelete: true,
|
|
}, nil
|
|
}
|
|
|
|
// retrievePulumiTemplates retrieves the "template repository" for Pulumi templates.
|
|
// Instead of retrieving to a temporary directory, the Pulumi templates are managed from
|
|
// ~/.pulumi/templates.
|
|
func retrievePulumiTemplates(templateName string, offline bool) (TemplateRepository, error) {
|
|
templateName = strings.ToLower(templateName)
|
|
|
|
// Cleanup the template directory.
|
|
if err := cleanupLegacyTemplateDir(); err != nil {
|
|
return TemplateRepository{}, err
|
|
}
|
|
|
|
// Get the template directory.
|
|
templateDir, err := GetTemplateDir()
|
|
if err != nil {
|
|
return TemplateRepository{}, err
|
|
}
|
|
|
|
// Ensure the template directory exists.
|
|
if err := os.MkdirAll(templateDir, 0700); err != nil {
|
|
return TemplateRepository{}, err
|
|
}
|
|
|
|
if !offline {
|
|
// Clone or update the pulumi/templates repo.
|
|
err := gitutil.GitCloneOrPull(pulumiTemplateGitRepository, plumbing.HEAD, templateDir, false /*shallow*/)
|
|
if err != nil {
|
|
return TemplateRepository{}, err
|
|
}
|
|
}
|
|
|
|
subDir := templateDir
|
|
if templateName != "" {
|
|
subDir = filepath.Join(subDir, templateName)
|
|
}
|
|
|
|
return TemplateRepository{
|
|
Root: templateDir,
|
|
SubDirectory: subDir,
|
|
ShouldDelete: false,
|
|
}, nil
|
|
}
|
|
|
|
// RetrieveTemplate downloads the repo to path and returns the full path on disk.
|
|
func RetrieveTemplate(rawurl string, path string) (string, error) {
|
|
url, urlPath, err := gitutil.ParseGitRepoURL(rawurl)
|
|
if err != nil {
|
|
return "", err
|
|
}
|
|
|
|
ref, commit, subDirectory, err := gitutil.GetGitReferenceNameOrHashAndSubDirectory(url, urlPath)
|
|
if err != nil {
|
|
return "", err
|
|
}
|
|
|
|
if ref != "" {
|
|
if cloneErr := gitutil.GitCloneOrPull(url, ref, path, true /*shallow*/); cloneErr != nil {
|
|
return "", cloneErr
|
|
}
|
|
} else {
|
|
if cloneErr := gitutil.GitCloneAndCheckoutCommit(url, commit, path); cloneErr != nil {
|
|
return "", cloneErr
|
|
}
|
|
}
|
|
|
|
// Verify the sub directory exists.
|
|
fullPath := filepath.Join(path, filepath.FromSlash(subDirectory))
|
|
info, err := os.Stat(fullPath)
|
|
if err != nil {
|
|
return "", err
|
|
}
|
|
if !info.IsDir() {
|
|
return "", errors.Errorf("%s is not a directory", fullPath)
|
|
}
|
|
|
|
return fullPath, nil
|
|
}
|
|
|
|
// LoadTemplate returns a template from a path.
|
|
func LoadTemplate(path string) (Template, error) {
|
|
info, err := os.Stat(path)
|
|
if err != nil {
|
|
return Template{}, err
|
|
}
|
|
if !info.IsDir() {
|
|
return Template{}, errors.Errorf("%s is not a directory", path)
|
|
}
|
|
|
|
// TODO handle other extensions like Pulumi.yml and Pulumi.json?
|
|
proj, err := LoadProject(filepath.Join(path, "Pulumi.yaml"))
|
|
if err != nil {
|
|
return Template{}, err
|
|
}
|
|
|
|
template := Template{
|
|
Dir: path,
|
|
Name: filepath.Base(path),
|
|
|
|
ProjectName: proj.Name.String(),
|
|
}
|
|
if proj.Template != nil {
|
|
template.Description = proj.Template.Description
|
|
template.Quickstart = proj.Template.Quickstart
|
|
template.Config = proj.Template.Config
|
|
}
|
|
if proj.Description != nil {
|
|
template.ProjectDescription = *proj.Description
|
|
}
|
|
|
|
return template, nil
|
|
}
|
|
|
|
// CopyTemplateFilesDryRun does a dry run of copying a template to a destination directory,
|
|
// to ensure it won't overwrite any files.
|
|
func (template Template) CopyTemplateFilesDryRun(destDir string) error {
|
|
var existing []string
|
|
if err := walkFiles(template.Dir, destDir, func(info os.FileInfo, source string, dest string) error {
|
|
if destInfo, statErr := os.Stat(dest); statErr == nil && !destInfo.IsDir() {
|
|
existing = append(existing, filepath.Base(dest))
|
|
}
|
|
return nil
|
|
}); err != nil {
|
|
return err
|
|
}
|
|
|
|
if len(existing) > 0 {
|
|
return newExistingFilesError(existing)
|
|
}
|
|
return nil
|
|
}
|
|
|
|
// CopyTemplateFiles does the actual copy operation to a destination directory.
|
|
func (template Template) CopyTemplateFiles(
|
|
destDir string, force bool, projectName string, projectDescription string) error {
|
|
|
|
return walkFiles(template.Dir, destDir, func(info os.FileInfo, source string, dest string) error {
|
|
if info.IsDir() {
|
|
// Create the destination directory.
|
|
return os.Mkdir(dest, 0700)
|
|
}
|
|
|
|
// Read the source file.
|
|
b, err := ioutil.ReadFile(source)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
|
|
// Transform only if it isn't a binary file.
|
|
result := b
|
|
if !isBinary(b) {
|
|
transformed := transform(string(b), projectName, projectDescription)
|
|
result = []byte(transformed)
|
|
}
|
|
|
|
// Write to the destination file.
|
|
err = writeAllBytes(dest, result, force)
|
|
if err != nil {
|
|
// An existing file has shown up in between the dry run and the actual copy operation.
|
|
if os.IsExist(err) {
|
|
return newExistingFilesError([]string{filepath.Base(dest)})
|
|
}
|
|
}
|
|
return err
|
|
})
|
|
}
|
|
|
|
// GetTemplateDir returns the directory in which templates on the current machine are stored.
|
|
func GetTemplateDir() (string, error) {
|
|
// Allow the folder we use to store templates to be overridden.
|
|
dir := os.Getenv(pulumiLocalTemplatePathEnvVar)
|
|
|
|
// Use the classic template directory if there is no override.
|
|
if dir == "" {
|
|
u, err := user.Current()
|
|
if u == nil || err != nil {
|
|
return "", errors.Wrap(err, "getting user home directory")
|
|
}
|
|
dir = filepath.Join(u.HomeDir, BookkeepingDir, TemplateDir)
|
|
}
|
|
|
|
return dir, nil
|
|
}
|
|
|
|
// IsValidProjectName returns true if the project name is a valid name.
|
|
func IsValidProjectName(name string) bool {
|
|
return tokens.IsPackageName(name)
|
|
}
|
|
|
|
// ValueOrSanitizedDefaultProjectName returns the value or a sanitized valid project name
|
|
// based on defaultNameToSanitize.
|
|
func ValueOrSanitizedDefaultProjectName(name string, defaultNameToSanitize string) string {
|
|
if name != "" {
|
|
return name
|
|
}
|
|
return getValidProjectName(defaultNameToSanitize)
|
|
}
|
|
|
|
// ValueOrDefaultProjectDescription returns the value or defaultDescription.
|
|
func ValueOrDefaultProjectDescription(description string, defaultDescription string) string {
|
|
if description != "" {
|
|
return description
|
|
}
|
|
if defaultDescription != "" {
|
|
return defaultDescription
|
|
}
|
|
return ""
|
|
}
|
|
|
|
// getValidProjectName returns a valid project name based on the passed-in name.
|
|
func getValidProjectName(name string) string {
|
|
// If the name is valid, return it.
|
|
if IsValidProjectName(name) {
|
|
return name
|
|
}
|
|
|
|
// Otherwise, try building-up the name, removing any invalid chars.
|
|
var result string
|
|
for i := 0; i < len(name); i++ {
|
|
temp := result + string(name[i])
|
|
if IsValidProjectName(temp) {
|
|
result = temp
|
|
}
|
|
}
|
|
|
|
// If we couldn't come up with a valid project name, fallback to a default.
|
|
if result == "" {
|
|
result = defaultProjectName
|
|
}
|
|
|
|
return result
|
|
}
|
|
|
|
// walkFiles is a helper that walks the directories/files in a source directory
|
|
// and performs an action for each item.
|
|
func walkFiles(sourceDir string, destDir string,
|
|
actionFn func(info os.FileInfo, source string, dest string) error) error {
|
|
|
|
contract.Require(sourceDir != "", "sourceDir")
|
|
contract.Require(destDir != "", "destDir")
|
|
contract.Require(actionFn != nil, "actionFn")
|
|
|
|
infos, err := ioutil.ReadDir(sourceDir)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
for _, info := range infos {
|
|
name := info.Name()
|
|
source := filepath.Join(sourceDir, name)
|
|
dest := filepath.Join(destDir, name)
|
|
|
|
if info.IsDir() {
|
|
// Ignore the .git directory.
|
|
if name == GitDir {
|
|
continue
|
|
}
|
|
|
|
if err := actionFn(info, source, dest); err != nil {
|
|
return err
|
|
}
|
|
|
|
if err := walkFiles(source, dest, actionFn); err != nil {
|
|
return err
|
|
}
|
|
} else {
|
|
// Ignore the legacy template manifest.
|
|
if name == legacyPulumiTemplateManifestFile {
|
|
continue
|
|
}
|
|
|
|
if err := actionFn(info, source, dest); err != nil {
|
|
return err
|
|
}
|
|
}
|
|
}
|
|
|
|
return nil
|
|
}
|
|
|
|
// newExistingFilesError returns a new error from a list of existing file names
|
|
// that would be overwritten.
|
|
func newExistingFilesError(existing []string) error {
|
|
contract.Assert(len(existing) > 0)
|
|
message := "creating this template will make changes to existing files:\n"
|
|
for _, file := range existing {
|
|
message = message + fmt.Sprintf(" overwrite %s\n", file)
|
|
}
|
|
message = message + "\nrerun the command and pass --force to accept and create"
|
|
return errors.New(message)
|
|
}
|
|
|
|
// transform returns a new string with ${PROJECT} and ${DESCRIPTION} replaced by
|
|
// the value of projectName and projectDescription.
|
|
func transform(content string, projectName string, projectDescription string) string {
|
|
// On Windows, we need to replace \n with \r\n because go-git does not currently handle it.
|
|
if runtime.GOOS == "windows" {
|
|
content = strings.Replace(content, "\n", "\r\n", -1)
|
|
}
|
|
content = strings.Replace(content, "${PROJECT}", projectName, -1)
|
|
content = strings.Replace(content, "${DESCRIPTION}", projectDescription, -1)
|
|
return content
|
|
}
|
|
|
|
// writeAllBytes writes the bytes to the specified file, with an option to overwrite.
|
|
func writeAllBytes(filename string, bytes []byte, overwrite bool) error {
|
|
flag := os.O_WRONLY | os.O_CREATE
|
|
if overwrite {
|
|
flag = flag | os.O_TRUNC
|
|
} else {
|
|
flag = flag | os.O_EXCL
|
|
}
|
|
|
|
f, err := os.OpenFile(filename, flag, 0600)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
defer contract.IgnoreClose(f)
|
|
|
|
_, err = f.Write(bytes)
|
|
return err
|
|
}
|
|
|
|
// isBinary returns true if a zero byte occurs within the first
|
|
// 8000 bytes (or the entire length if shorter). This is the
|
|
// same approach that git uses to determine if a file is binary.
|
|
func isBinary(bytes []byte) bool {
|
|
const firstFewBytes = 8000
|
|
|
|
length := len(bytes)
|
|
if firstFewBytes < length {
|
|
length = firstFewBytes
|
|
}
|
|
|
|
for i := 0; i < length; i++ {
|
|
if bytes[i] == 0 {
|
|
return true
|
|
}
|
|
}
|
|
|
|
return false
|
|
}
|