I found the flag --force to be a strange name for skipping a preview,
since that name is usually reserved for operations that might be harmful
and yet you're coercing a tool to do it anyway, knowing there's a chance
you're going to shoot yourself in the foot.
I also found that what I almost always want in the situation where
--force was being used is to actually just run a preview and have the
confirmation auto-accepted. Going straight to --force isn't the right
thing in a CI scenario, where you actually want to run a preview first,
just to ensure there aren't any issues, before doing the update.
In a sense, there are four options here:
1. Run a preview, ask for confirmation, then do an update (the default).
2. Run a preview, auto-accept, and then do an update (the CI scenario).
3. Just run a preview with neither a confirmation nor an update (dry run).
4. Just do an update, without performing a preview beforehand (rare).
This change enables all four workflows in our CLI.
Rather than have an explosion of flags, we have a single flag,
--preview, which can specify the mode that we're operating in. The
following are the values which correlate to the above four modes:
1. "": default (no --preview specified)
2. "auto": auto-accept preview confirmation
3. "only": only run a preview, don't confirm or update
4. "skip": skip the preview altogether
As part of this change, I redid a bit of how the preview modes
were specified. Rather than booleans, which had some illegal
combinations, this change introduces a new enum type. Furthermore,
because the engine is wholly ignorant of these flags -- and only the
backend understands them -- it was confusing to me that
engine.UpdateOptions stored this flag, especially given that all
interesting engine options _also_ accepted a dryRun boolean. As of
this change, the backend.PreviewBehavior controls the preview options.
We already have a great history viewing experience on
Pulumi.com. `pulumi history` in the CLI today is basically unuseable,
and instead of working on trying to improve it, let's just remove it
for now.
Fixes#965
To prepare for a world where stack names must be unique across an
owner, add some randomness to the names we use for stacks as part of
our integration tests.
This change removes the need to `pulumi init` when targeting the local
backend. A fair amount of the change lays the foundation that the next
set of changes to stop having `pulumi init` be used for cloud stacks
as well.
Previously, `pulumi init` logically did two things:
1. It created the bookkeeping directory for local stacks, this was
stored in `<repository-root>/.pulumi`, where `<repository-root>` was
the path to what we belived the "root" of your project was. In the
case of git repositories, this was the directory that contained your
`.git` folder.
2. It recorded repository information in
`<repository-root>/.pulumi/repository.json`. This was used by the
cloud backend when computing what project to interact with on
Pulumi.com
The new identity model will remove the need for (2), since we only
need an owner and stack name to fully qualify a stack on
pulumi.com, so it's easy enough to stop creating a folder just for
that.
However, for the local backend, we need to continue to retain some
information about stacks (e.g. checkpoints, history, etc). In
addition, we need to store our workspace settings (which today just
contains the selected stack) somehere.
For state stored by the local backend, we change the URL scheme from
`local://` to `local://<optional-root-path>`. When
`<optional-root-path>` is unset, it defaults to `$HOME`. We create our
`.pulumi` folder in that directory. This is important because stack
names now must be unique within the backend, but we have some tests
using local stacks which use fixed stack names, so each integration
test really wants its own "view" of the world.
For the workspace settings, we introduce a new `workspaces` directory
in `~/.pulumi`. In this folder we write the workspace settings file
for each project. The file name is the name of the project, combined
with the SHA1 of the path of the project file on disk, to ensure that
multiple pulumi programs with the same project name have different
workspace settings.
This does mean that moving a project's location on disk will cause the
CLI to "forget" what the selected stack was, which is unfortunate, but
not the end of the world. If this ends up being a big pain point, we
can certianly try to play games in the future (for example, if we saw
a .git folder in a parent folder, we could store data in there).
With respect to compatibility, we don't attempt to migrate older files
to their newer locations. For long lived stacks managed using the
local backend, we can provide information on where to move things
to. For all stacks (regardless of backend) we'll require the user to
`pulumi stack select` their stack again, but that seems like the
correct trade-off vs writing complicated upgrade code.
Our normal lifecycle tests always call pulumi stack rm, but some of
the tests that used the more barebones framework did not do so. This
was "ok" in the past, since all bookkeeping data about a stack was
stored next to the Pulumi program itself and we deleted that folder
once the test passed.
As part of removing `pulumi init` workspace tracking will move to
~/.pulumi/workspaces and so we'd like to have a gesture that actually
removes the stack, which will cause the workspace file to be removed
as well, instead of littering ~/.pulumi/workspaces with tests.
Upcoming work to remove the need for `pulumi init` makes testing the
upgrade code much harder than it did in the past (since workspace data
is moving to a different location on the file system, as well as some
other changes).
Instead of trying to maintain the code and test, let's just remove
it. Folks who haven't migrated (LM and the PPC deployment in the
service) should use the 0.11.3 or earlier CLI to migrate their
projects (simply by logging in and running a pulumi command) or move
things forward by hand.
In cases where we are running against a local service, the CLI does
not print a Permalink line when updating a stack, because we can not
determine what the URL for the link would be. This breaks the diff
tests which need to clean the CLI output and compare them to known
values, since the output now has one less line than expected.
Update the test's cleaning logic to handle this case.
* Lift snapshot management out of the engine
This PR is a prerequisite for parallelism by addressing a major problem
that the engine has to deal with when performing parallel resource
construction: parallel mutation of the global snapshot. This PR adds
a `SnapshotManager` type that is responsible for maintaining and
persisting the current resource snapshot. It serializes all reads and
writes to the global snapshot and persists the snapshot to persistent
storage upon every write.
As a side-effect of this, the core engine no longer needs to know about
snapshot management at all; all snapshot operations can be handled as
callbacks on deployment events. This will greatly simplify the
parallelization of the core engine.
Worth noting is that the core engine will still need to be able to read
the current snapshot, since it is interested in the dependency graphs
contained within. The full implications of that are out of scope of this
PR.
Remove dead code, Steps no longer need a reference to the plan iterator that created them
Fixing various issues that arise when bringing up pulumi-aws
Line length broke the build
Code review: remove dead field, fix yaml name error
Rebase against master, provide implementation of StackPersister for cloud backend
Code review feedback: comments on MutationStatus, style in snapshot.go
Code review feedback: move SnapshotManager to pkg/backend, change engine to use an interface SnapshotManager
Code review feedback: use a channel for synchronization
Add a comment and a new test
* Maintain two checkpoints, an immutable base and a mutable delta, and
periodically merge the two to produce snapshots
* Add a lot of tests - covers all of the non-error paths of BeginMutation and End
* Fix a test resource provider
* Add a few tests, fix a few issues
* Rebase against master, fixed merge
This change does three major things:
1. Removes the ability to be logged into multiple clouds at the same
time. Previously, we supported being logged into multiple clouds at
the same time and the CLI would fan out requests and join responses
when needed. In general, this was only useful for Pulumi employees
that wanted run against multiple copies of the service (say production
and staging) but overall was very confusing (for example in the old
world a stack with the same identity could appear twice (since it was
in two backends) which the CLI didn't handle very well).
2. Stops treating the "local" backend as a special thing, from the
point of view of the CLI. Previouly we'd always connect to the local
backend and merge that data with whatever was in clouds we were
connected to. We had gestures like `--local` in `pulumi stack init`
that meant "use the local mode". Instead, to use the local mode now
you run `pulumi login --cloud-url local://` and then you are logged in
the local backend. Since you can only ever be logged into a single
backend, we can remove the `--local` and `--remote` flags from `pulumi
stack init`, it just now requires you to be logged in and creates a
stack in whatever back end you were logged into. When logging into the
local backend, you are not prompted for an access key.
3. Prompt for login in places where you have to log in, if you are not
already logged in.
Tests now target managed stacks instead of local stacks.
The existing logged in user and target backend API are used unless PULUMI_ACCES_TOKEN is defined, in which case tests are run under that access token and against the PULUMI_API backend.
For developer machines, we will now need to be logged in to Pulumi to run tests, and whichever default API backend is logged in (the one listed as current in ~/.pulumi/credentials.json) will be used. If you need to override these, provide PULUMI_ACCESS_TOKEN and possibly PULUMI_API.
For Travis, we currently target the staging service using the Pulumi Bot user.
We have decided to run tests in the pulumi organization. This can be overridden for local testing (or in Travis in the future) by defining PULUMI_API_OWNER_ORGANIZATION and using an access token with access to that organization.
Part of pulumi/home#195.
If you use --cloud-url, as in
$ pulumi stack init foo --cloud-url https://x.io
we would silently fall back to logic that creates local stacks.
I realize all of this will get better with the new stack identity
model, however in the meantime, let's infer that the user wanted
--remote when --cloud-url is non-"".
Note: This is a minor issue that I didn't get to for M11 that isn't
required for M11 and would be fine merging for post-M11.
When you specify a template name explicitly (e.g.
`pulumi new typescript`), we'll try to download the template tarball
without first downloading the JSON list of available templates. The JSON
includes a description used when replacing the `${DESCRIPTION}` string
in template files. Since we didn't download the JSON, we won't have a
description, so we fallback to a default value (`"A Pulumi project."`).
This also happens when specifying `--offline` to use an existing
template under `~/.pulumi/templates`; we won't have a description for
the template, so we fallback to a default description. The fallback
value happens to be the same as the description for each of our current
templates, so noone will currently notice an issue.
For M11, I included initial support for a template manifest file where
the description (and any future metadata) could be stored, but didn't go
as far as actually reading the file.
This change makes it so the CLI actually reads the description from the
manifest file (if it exists), otherwise falling back to the default
value as is done currently. Some minor related cleanup is included in
this change.
Also, rename/cleanup a bunch of serialization code.
Also, generate better environment names in the serialized closure code. Thsi code should be much easier to make sense of as hte names will better track to the original names in the user code.
Also, dedupe simple non-capturing functions. This helps ensure we don't spit out N copies of __awaiter (one per file it is declared in).
This adds a `pulumi new` command which makes it easy to quickly
automatically create the handful of needed files to get started building
an empty Pulumi project.
Usage:
```
$ pulumi new typescript
```
Or you can leave off the template name, and it will ask you to choose
one:
```
$ pulumi new
Please choose a template:
> javascript
python
typescript
```
config.Key has become a pair of namespace and name. Because the whole
world has not changed yet, there continues to be a way to convert
between a tokens.ModuleMember and config.Key, however now sometime the
conversion from tokens.ModuleMember can fail (when the module member
is not of the form `<package>:config:<name>`).
Right now, config.Key is a type alias for tokens.ModuleMember. I did a
pass over the codebase such that we use config.Key everywhere it
looked like the value did not leak to some external process (e.g a
resource provider or a langhost).
Doing this makes it a little clearer (hopefully) where code is
depending on a module member structure (e.g. <package>:config:<value>)
instead of just an opaque type.
Make many fixes to closure serialization
Primary things that i've done as part of this change:
Added support for cyclic objects.
Properly serialize objects that are shared across different function. previously you would get multiple copies, now you properly reference the same copy.
Remove the usages of 'hashes' for functions. Because we track identity of objects, we no longer need them.
Serialize properties of functions (if they have any).
Handle Objects/Functions with different __proto__s than normal. i.e. classes/constructors. but also anything the user may have done themselves to the object.
Handle generator functions.
Handle functions with 'computed' names.
Handle functions with 'symbol' names.
Handle serializing Promises as Promises.
Removed the dual Closure/AsyncClosure tree. One existed solely so we could have a tree without promises (for use in testing maybe?). Because this all exists in a part of our codebase that is entirely async, it's fine to have promises in the tree, and to await them when serializing the Closure to a string.
Handle serializing class-constructors and methods. Including properly handling 'super' calls.
Migrate configuration from the old model to the new model. The
strategy here is that when we first run `pulumi` we enumerate all of
the stacks from all of the backends we know about and for each stack
get the configuration values from the project and workspace and
promote them into the new file. As we do this, we remove stack
specific config from the workspace and Pulumi.yaml file.
If we are able to upgrade all the stacks we know about, we delete all
global configuration data in the workspace and in Pulumi.yaml as well.
We have a test that ensures upgrades continue to work.
This change updates our configuration model to make it simpler to
understand by removing some features and changing how things are
persisted in files.
Notable changes:
- We've removed the notion of "workspace" vs "project"
config. Now, configuration is always stored in a file next to
`Pulumi.yaml` named `Pulumi.<stack-name>.yaml` (the same file we'd
use for an other stack specific information we would need to persist
in the future).
- We've removed the notion of project wide configuration. Every new
stack gets a completely empty set of configuration and there's no
way to share common values across stacks, instead the common value
has to be set on each stack.
We retain some of the old code for the configuration system so we can
support upgrading a project in place. That will happen with the next
change.
This change fixes some issues and allows us to close some
others (since they are no longer possible).
Fixes#866Closes#872Closes#731
We are going to be changing the configuration model. To begin, let's
take most of the existing stuff and mark it as "deprecated" so we can
keep the existing behavior (to help transition newer code forward)
while making it clear what APIs should not be called in the
implementation of `pulumi` itself.
Despite our good progress moving towards having an apitype package,
where our exchange types live and can be shared among the engine and
our services, there were a few major types that were still duplciated.
Resource was the biggest example -- and indeed, the apitype varirant
was missing the new Dependencies property -- but there were others,
like Manfiest, PluginInfo, etc. These too had semi-random omissions.
This change merges all of these types into the apitype package. This
not only cleans up the redundancy and missing properties, but will
"force the issue" with respect to keeping them in sync and properly
versioning the information in a backwards compatible way.
The resource/stack package still exists as a simple marshaling layer
to and from the engine's core data types.
Finally, I've made the controversial change to share the actual
Deployment data structure at the apitype layer also. This will force
us to confront differences in that data structure similarly, and will
allow us to leverage the strong typing throughout to catch issues.
If currently logged in, `stack init` creates a managed stack. Otherwise, it creates a local
stack. This avoids the need to specify `--local` when not using the service.
As today, `--local` can be passed, which will create a local stack regardless of being logged
in or not.
A new flag, `--remote`, has been added, which can be passed to indicate a managed stack,
used to force an error if not logged into the service.
This change gets enough of the Python SDK up and running that the
empty Python program will work. Mostly just scaffolding, but the
basic structure is now in place. The primary remaining work is to
wire up resource creation to the gRPC interfaces.
In summary:
* The basic structure is as follows:
- Everything goes into sdk/python/.
- sdk/python/cmd/pulumi-langhost-python is a Go language host
that simply knows how to spawn Python processes to run out
entrypoint in response to requests by the engine.
- sdk/python/cmd/pulumi-langhost-python-exec is a little Python
shim that is invoked by the language host to run Python programs,
and is responsible for setting up the minimal goo before we can
do so (RPC connections and the like).
- sdk/python/lib/ contains a Python Pip package suitable for PyPi.
- In there, we have two packages: the root pulumi package that
contains all of the basic Pulumi programming model abstractions,
and pulumi.runtime, which contains the implementation of
resource registration, RPC interfacing with the engine, and so on.
* Add logic in our test framework to conditionalize on the language
type and react accordingly. This will allow us to skip Yarn for
Python projects and eventually run Pip if there's a requirements.txt.
* Created the basic project structure, including all of the usual
Make targets for installing into the proper places.
* Building also runs Pylint and we are clean.
There are a few other minor things in here:
* Add an "empty" test for both Node.js and Python. These pass.
* Fix an existing bug in plugin shutdown logic. At some point, we
started waiting for stderr/stdout to flush before shutting down
the plugin; but if certain failures happen "early" during the
plugin launch process, these channels will never get initialized
and so waiting for them deadlocks.
* Recently we seem to have added logic to delete test temp
directories if a failure happened during initialization of said
temp directories. This is unfortunate, because you often need to
look at the temp directory to see what failed. We already clean
them up elsewhere after the full test completes successfully, so
I don't think we need to be doing this, and I've removed it.
Still many loose ends (config, resources, etc), but it's a start!
1. Various idiomatic Go and TypeScript fixes
2. Add an integration test that end-to-end roundtrips dependency
information for a simple Pulumi program
3. Add an additional test assert that tests that dependency information
comes from the language host as expected
Backup copies of local stack checkpoints are now saved to the
user's home directory (`~/.pulumi/backups`) by default.
This enables users to recover after accidentally deleting their
local `.pulumi` directory (e.g. via `git clean`).
The behavior can be disabled by setting the
PULUMI_DISABLE_CHECKPOINT_BACKUPS environment variable, which
we use to disable backups when running all tests other than the
test for this functionality.
This change includes a handful of stack-related CLI formatting
improvements that I've been noodling on in the background for a while,
based on things that tend to trip up demos and the inner loop workflow.
This includes:
* If `pulumi stack select` is run by itself, use an interactive
CLI menu to let the user select an existing stack, or choose to
create a new one. This looks as follows
$ pulumi stack select
Please choose a stack, or choose to create a new one:
abcdef
babblabblabble
> currentlyselected
defcon
<create a new stack>
and is navigated in the usual way (key up, down, enter).
* If a stack name is passed that does not exist, prompt the user
to ask whether s/he wants to create one on-demand. This hooks
interesting moments in time, like `pulumi stack select foo`,
and cuts down on the need to run additional commands.
* If a current stack is required, but none is currently selected,
then pop the same interactive menu shown above to select one.
Depending on the command being run, we may or may not show the
option to create a new stack (e.g., that doesn't make much sense
when you're running `pulumi destroy`, but might when you're
running `pulumi stack`). This again lets you do with a single
command what would have otherwise entailed an error with multiple
commands to recover from it.
* If you run `pulumi stack init` without any additional arguments,
we interactively prompt for the stack name. Before, we would
error and you'd then need to run `pulumi stack init <name>`.
* Colorize some things nicely; for example, now all prompts will
by default become bright white.
This addresses pulumi/pulumi#446: what we used to call "package" is
now called "project". This has gotten more confusing over time, now
that we're doing real package management.
Also fixespulumi/pulumi#426, while in here.
In order to begin publishing our core SDK package to NPM, we will
need it to be underneath the @pulumi scope so that it may remain
private. Eventually, we can alias pulumi back to it.
This is part of pulumi/pulumi#915.
As it stands, we serialize more than is correct when registering
resources: in addition to serializing the RegisterResource RPC, we also
wait for input properties to resolve in the same context. Unfortunately,
this means that we can create cycles in the promise graph when a
resource A is constructed in an earlier turn than some resource B and
one of B's output properties is an input to resource A. These changes
fix this issue by allowing input properties to resolve *before*
serializing the RegisterResource RPC.
Some integration tests had taken a dependency on the ordering of resources in
either the output of the `pulumi` command or the checkpoint file. The
only test that took a dependency on command output was updated s.t. its
resources have exactly one legal topographical sort (and therefore their
ordering is deterministic). The other tests were updated s.t. their
validation did not depend on resource ordering.
This PR adds a new `pulumi history` command, which prints the update history for a stack.
The local backend stores the update history in a JSON file on disk, next to the checkpoint file. The cloud backend simply provides the update metadata, and expects to receive all the data from a (NYI) `/history` REST endpoint.
`pkg/backend/updates.go` defines the data that is being persisted. The way the data is wired through the system is adding a new `backend.UpdateMetadata` parameter to a Stack/Backend's `Update` and `Destroy` methods.
I use `tests/integration/stack_outputs/` as the simple app for the related tests, hence the addition to the `.gitignore` and fixing the name in the `Pulumi.yaml`.
Fixes#636.
