pulumi/pkg/workspace/project.go
Justin Van Patten 9d84f2e249
Initial support for passing URLs to new and up (#1727)
* Initial support for passing URLs to `new` and `up`

This PR adds initial support for `pulumi new` using Git under the covers
to manage Pulumi templates, providing the same experience as before.

You can now also optionally pass a URL to a Git repository, e.g.
`pulumi new [<url>]`, including subdirectories within the repository,
and arbitrary branches, tags, or commits.

The following commands result in the same behavior from the user's
perspective:
 - `pulumi new javascript`
 - `pulumi new https://github.com/pulumi/templates/templates/javascript`
 - `pulumi new https://github.com/pulumi/templates/tree/master/templates/javascript`
 - `pulumi new https://github.com/pulumi/templates/tree/HEAD/templates/javascript`

To specify an arbitrary branch, tag, or commit:
 - `pulumi new https://github.com/pulumi/templates/tree/<branch>/templates/javascript`
 - `pulumi new https://github.com/pulumi/templates/tree/<tag>/templates/javascript`
 - `pulumi new https://github.com/pulumi/templates/tree/<commit>/templates/javascript`

Branches and tags can include '/' separators, and `pulumi` will still
find the right subdirectory.

URLs to Gists are also supported, e.g.:
`pulumi new https://gist.github.com/justinvp/6673959ceb9d2ac5a14c6d536cb871a6`

If the specified subdirectory in the repository does not contain a
`Pulumi.yaml`, it will look for subdirectories within containing
`Pulumi.yaml` files, and prompt the user to choose a template, along the
lines of how `pulumi new` behaves when no template is specified.

The following commands result in the CLI prompting to choose a template:
 - `pulumi new`
 - `pulumi new https://github.com/pulumi/templates/templates`
 - `pulumi new https://github.com/pulumi/templates/tree/master/templates`
 - `pulumi new https://github.com/pulumi/templates/tree/HEAD/templates`

Of course, arbitrary branches, tags, or commits can be specified as well:
 - `pulumi new https://github.com/pulumi/templates/tree/<branch>/templates`
 - `pulumi new https://github.com/pulumi/templates/tree/<tag>/templates`
 - `pulumi new https://github.com/pulumi/templates/tree/<commit>/templates`

This PR also includes initial support for passing URLs to `pulumi up`,
providing a streamlined way to deploy installable cloud applications
with Pulumi, without having to manage source code locally before doing
a deployment.

For example, `pulumi up https://github.com/justinvp/aws` can be used to
deploy a sample AWS app. The stack can be updated with different
versions, e.g.
`pulumi up https://github.com/justinvp/aws/tree/v2 -s <stack-to-update>`

Config values can optionally be passed via command line flags, e.g.
`pulumi up https://github.com/justinvp/aws -c aws:region=us-west-2 -c foo:bar=blah`

Gists can also be used, e.g.
`pulumi up https://gist.github.com/justinvp/62fde0463f243fcb49f5a7222e51bc76`

* Fix panic when hitting ^C from "choose template" prompt

* Add description to templates

When running `pulumi new` without specifying a template, include the template description along with the name in the "choose template" display.

```
$ pulumi new
Please choose a template:
  aws-go                  A minimal AWS Go program
  aws-javascript          A minimal AWS JavaScript program
  aws-python              A minimal AWS Python program
  aws-typescript          A minimal AWS TypeScript program
> go                      A minimal Go program
  hello-aws-javascript    A simple AWS serverless JavaScript program
  javascript              A minimal JavaScript program
  python                  A minimal Python program
  typescript              A minimal TypeScript program
```

* React to changes to the pulumi/templates repo.

We restructured the `pulumi/templates` repo to have all the templates in the root instead of in a `templates` subdirectory, so make the change here to no longer look for templates in `templates`.

This also fixes an issue around using `Depth: 1` that I found while testing this. When a named template is used, we attempt to clone or pull from the `pulumi/templates` repo to `~/.pulumi/templates`. Having it go in this well-known directory allows us to maintain previous behavior around allowing offline use of templates. If we use `Depth: 1` for the initial clone, it will fail when attempting to pull when there are updates to the remote repository. Unfortunately, there's no built-in `--unshallow` support in `go-git` and setting a larger `Depth` doesn't appear to help. There may be a workaround, but for now, if we're cloning the pulumi templates directory to `~/.pulumi/templates`, we won't use `Depth: 1`. For template URLs, we will continue to use `Depth: 1` as we clone those to a temp directory (which gets deleted) that we'll never try to update.

* List available templates in help text

* Address PR Feedback

* Don't show "Installing dependencies" message for `up`

* Fix secrets handling

When prompting for config, if the existing stack value is a secret, keep it a secret and mask the prompt. If the template says it should be secret, make it a secret.

* Fix ${PROJECT} and ${DESCRIPTION} handling for `up`

Templates used with `up` should already have a filled-in project name and description, but if it's a `new`-style template, that has `${PROJECT}` and/or `${DESCRIPTION}`, be helpful and just replace these with better values.

* Fix stack handling

Add a bool `setCurrent` param to `requireStack` to control whether the current stack should be saved in workspace settings. For the `up <url>` case, we don't want to save. Also, split the `up` code into two separate functions: one for the `up <url>` case and another for the normal `up` case where you have workspace in your current directory. While we may be able to combine them back into a single function, right now it's a bit cleaner being separate, even with some small amount of duplication.

* Fix panic due to nil crypter

Lazily get the crypter only if needed inside `promptForConfig`.

* Embellish comment

* Harden isPreconfiguredEmptyStack check

Fix the code to check to make sure the URL specified on the command line matches the URL stored in the `pulumi:template` config value, and that the rest of the config from the stack satisfies the config requirements of the template.
2018-08-10 18:08:16 -07:00

294 lines
8.9 KiB
Go

// Copyright 2016-2018, Pulumi Corporation.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package workspace
import (
"encoding/json"
"io/ioutil"
"os"
"path/filepath"
"github.com/pulumi/pulumi/pkg/resource/config"
"github.com/pulumi/pulumi/pkg/util/contract"
"github.com/pkg/errors"
"github.com/pulumi/pulumi/pkg/encoding"
"github.com/pulumi/pulumi/pkg/tokens"
)
// Analyzers is a list of analyzers to run on this project.
type Analyzers []tokens.QName
// ProjectTemplate is a Pulumi project template manifest.
// nolint: lll
type ProjectTemplate struct {
Description string `json:"description,omitempty" yaml:"description,omitempty"` // an optional description of the template.
Quickstart string `json:"quickstart,omitempty" yaml:"quickstart,omitempty"` // optional text to be displayed after template creation.
Config map[config.Key]ProjectTemplateConfigValue `json:"config,omitempty" yaml:"config,omitempty"` // optional template config.
}
// ProjectTemplateConfigValue is a config value included in the project template manifest.
// nolint: lll
type ProjectTemplateConfigValue struct {
Description string `json:"description,omitempty" yaml:"description,omitempty"` // an optional description for the config value.
Default string `json:"default,omitempty" yaml:"default,omitempty"` // an optional default value for the config value.
Secret bool `json:"secret,omitempty" yaml:"secret,omitempty"` // an optional value indicating whether the config value should be encrypted.
}
// Project is a Pulumi project manifest..
//
// We explicitly add yaml tags (instead of using the default behavior from https://github.com/ghodss/yaml which works
// in terms of the JSON tags) so we can directly marshall and unmarshall this struct using go-yaml an have the fields
// in the serialized object match the order they are defined in this struct.
//
// TODO[pulumi/pulumi#423]: use DOM based marshalling so we can roundtrip the seralized structure perfectly.
// nolint: lll
type Project struct {
Name tokens.PackageName `json:"name" yaml:"name"` // a required fully qualified name.
RuntimeInfo ProjectRuntimeInfo `json:"runtime" yaml:"runtime"` // a required runtime that executes code.
Main string `json:"main,omitempty" yaml:"main,omitempty"` // an optional override for the main program location.
Description *string `json:"description,omitempty" yaml:"description,omitempty"` // an optional informational description.
Author *string `json:"author,omitempty" yaml:"author,omitempty"` // an optional author.
Website *string `json:"website,omitempty" yaml:"website,omitempty"` // an optional website for additional info.
License *string `json:"license,omitempty" yaml:"license,omitempty"` // an optional license governing this project's usage.
Analyzers *Analyzers `json:"analyzers,omitempty" yaml:"analyzers,omitempty"` // any analyzers enabled for this project.
Context string `json:"context,omitempty" yaml:"context,omitempty"` // an optional path (combined with the on disk location of Pulumi.yaml) to control the data uploaded to the service.
NoDefaultIgnores *bool `json:"nodefaultignores,omitempty" yaml:"nodefaultignores,omitempty"` // true if we should only respect .pulumiignore when archiving
Config string `json:"config,omitempty" yaml:"config,omitempty"` // where to store Pulumi.<stack-name>.yaml files, this is combined with the folder Pulumi.yaml is in.
Template *ProjectTemplate `json:"template,omitempty" yaml:"template,omitempty"` // optional template manifest.
}
func (proj *Project) Validate() error {
if proj.Name == "" {
return errors.New("project is missing a 'name' attribute")
}
if proj.RuntimeInfo.Name() == "" {
return errors.New("project is missing a 'runtime' attribute")
}
return nil
}
func (proj *Project) UseDefaultIgnores() bool {
if proj.NoDefaultIgnores == nil {
return true
}
return !(*proj.NoDefaultIgnores)
}
// Save writes a project definition to a file.
func (proj *Project) Save(path string) error {
contract.Require(path != "", "path")
contract.Require(proj != nil, "proj")
contract.Requiref(proj.Validate() == nil, "proj", "Validate()")
m, err := marshallerForPath(path)
if err != nil {
return err
}
b, err := m.Marshal(proj)
if err != nil {
return err
}
return ioutil.WriteFile(path, b, 0644)
}
// ProjectStack holds stack specific information about a project.
// nolint: lll
type ProjectStack struct {
EncryptionSalt string `json:"encryptionsalt,omitempty" yaml:"encryptionsalt,omitempty"` // base64 encoded encryption salt.
Config config.Map `json:"config,omitempty" yaml:"config,omitempty"` // optional config.
}
// Save writes a project definition to a file.
func (ps *ProjectStack) Save(path string) error {
contract.Require(path != "", "path")
contract.Require(ps != nil, "ps")
m, err := marshallerForPath(path)
if err != nil {
return err
}
b, err := m.Marshal(ps)
if err != nil {
return err
}
// nolint: gas, gas prefers 0700 for a directory, but 0755 (so group and world can read it) is what we prefer
if err := os.MkdirAll(filepath.Dir(path), 0755); err != nil {
return err
}
return ioutil.WriteFile(path, b, 0644)
}
type ProjectRuntimeInfo struct {
name string
options map[string]interface{}
}
func NewProjectRuntimeInfo(name string, options map[string]interface{}) ProjectRuntimeInfo {
return ProjectRuntimeInfo{
name: name,
options: options,
}
}
func (info *ProjectRuntimeInfo) Name() string {
return info.name
}
func (info *ProjectRuntimeInfo) Options() map[string]interface{} {
return info.options
}
func (info ProjectRuntimeInfo) MarshalYAML() (interface{}, error) {
if info.options == nil || len(info.options) == 0 {
return info.name, nil
}
return map[string]interface{}{
"name": info.name,
"options": info.options,
}, nil
}
func (info ProjectRuntimeInfo) MarshalJSON() ([]byte, error) {
if info.options == nil || len(info.options) == 0 {
return json.Marshal(info.name)
}
return json.Marshal(map[string]interface{}{
"name": info.name,
"options": info.options,
})
}
func (info *ProjectRuntimeInfo) UnmarshalJSON(data []byte) error {
if err := json.Unmarshal(data, &info.name); err == nil {
return nil
}
var payload struct {
Name string `json:"name"`
Options map[string]interface{} `json:"options"`
}
if err := json.Unmarshal(data, &payload); err == nil {
info.name = payload.Name
info.options = payload.Options
return nil
}
return errors.New("runtime section must be a string or an object with name and options attributes")
}
func (info *ProjectRuntimeInfo) UnmarshalYAML(unmarshal func(interface{}) error) error {
if err := unmarshal(&info.name); err == nil {
return nil
}
var payload struct {
Name string `yaml:"name"`
Options map[string]interface{} `yaml:"options"`
}
if err := unmarshal(&payload); err == nil {
info.name = payload.Name
info.options = payload.Options
return nil
}
return errors.New("runtime section must be a string or an object with name and options attributes")
}
// LoadProject reads a project definition from a file.
func LoadProject(path string) (*Project, error) {
contract.Require(path != "", "path")
m, err := marshallerForPath(path)
if err != nil {
return nil, err
}
b, err := ioutil.ReadFile(path)
if err != nil {
return nil, err
}
var proj Project
err = m.Unmarshal(b, &proj)
if err != nil {
return nil, err
}
err = proj.Validate()
if err != nil {
return nil, err
}
return &proj, err
}
// LoadProjectStack reads a stack definition from a file.
func LoadProjectStack(path string) (*ProjectStack, error) {
contract.Require(path != "", "path")
m, err := marshallerForPath(path)
if err != nil {
return nil, err
}
b, err := ioutil.ReadFile(path)
if os.IsNotExist(err) {
return &ProjectStack{
Config: make(config.Map),
}, nil
} else if err != nil {
return nil, err
}
var ps ProjectStack
err = m.Unmarshal(b, &ps)
if err != nil {
return nil, err
}
if ps.Config == nil {
ps.Config = make(config.Map)
}
return &ps, err
}
func marshallerForPath(path string) (encoding.Marshaler, error) {
ext := filepath.Ext(path)
m, has := encoding.Marshalers[ext]
if !has {
return nil, errors.Errorf("no marshaler found for file format '%v'", ext)
}
return m, nil
}