This change introduces support for using the cloud backend when
`pulumi init` has not been run. When this is the case, we use the new
identity model, where a stack is referenced by an owner and a stack
name only.
There are a few things going on here:
- We add a new `--owner` flag to `pulumi stack init` that lets you
control what account a stack is created in.
- When listing stacks, we show stacks owned by you and any
organizations you are a member of. So, for example, I can do:
* `pulumi stack init my-great-stack`
* `pulumi stack init --owner pulumi my-great-stack`
To create a stack owned by my user and one owned by my
organization. When `pulumi stack ls` is run, you'll see both
stacks (since they are part of the same project).
- When spelling a stack on the CLI, an owner can be optionally
specified by prefixing the stack name with an owner name. For
example `my-great-stack` means the stack `my-great-stack` owned by
the current logged in user, where-as `pulumi/my-great-stack` would
be the stack owned by the `pulumi` organization
- `--all` can be passed to `pulumi stack ls` to see *all* stacks you
have access to, not just stacks tied to the current project.
Long term, a stack name alone will not be sufficent to address a
stack. Introduce a new `backend.StackReference` interface that allows
each backend to give an opaque stack reference that can be used across
operations.
To prepare for a world where stack names must be unique across an
owner, add some randomness to the names we use for stacks as part of
our integration tests.
This change removes the need to `pulumi init` when targeting the local
backend. A fair amount of the change lays the foundation that the next
set of changes to stop having `pulumi init` be used for cloud stacks
as well.
Previously, `pulumi init` logically did two things:
1. It created the bookkeeping directory for local stacks, this was
stored in `<repository-root>/.pulumi`, where `<repository-root>` was
the path to what we belived the "root" of your project was. In the
case of git repositories, this was the directory that contained your
`.git` folder.
2. It recorded repository information in
`<repository-root>/.pulumi/repository.json`. This was used by the
cloud backend when computing what project to interact with on
Pulumi.com
The new identity model will remove the need for (2), since we only
need an owner and stack name to fully qualify a stack on
pulumi.com, so it's easy enough to stop creating a folder just for
that.
However, for the local backend, we need to continue to retain some
information about stacks (e.g. checkpoints, history, etc). In
addition, we need to store our workspace settings (which today just
contains the selected stack) somehere.
For state stored by the local backend, we change the URL scheme from
`local://` to `local://<optional-root-path>`. When
`<optional-root-path>` is unset, it defaults to `$HOME`. We create our
`.pulumi` folder in that directory. This is important because stack
names now must be unique within the backend, but we have some tests
using local stacks which use fixed stack names, so each integration
test really wants its own "view" of the world.
For the workspace settings, we introduce a new `workspaces` directory
in `~/.pulumi`. In this folder we write the workspace settings file
for each project. The file name is the name of the project, combined
with the SHA1 of the path of the project file on disk, to ensure that
multiple pulumi programs with the same project name have different
workspace settings.
This does mean that moving a project's location on disk will cause the
CLI to "forget" what the selected stack was, which is unfortunate, but
not the end of the world. If this ends up being a big pain point, we
can certianly try to play games in the future (for example, if we saw
a .git folder in a parent folder, we could store data in there).
With respect to compatibility, we don't attempt to migrate older files
to their newer locations. For long lived stacks managed using the
local backend, we can provide information on where to move things
to. For all stacks (regardless of backend) we'll require the user to
`pulumi stack select` their stack again, but that seems like the
correct trade-off vs writing complicated upgrade code.
This field indicates the schema of the serialized deployment. This field
behaves identically to the `Version` field of
`PatchUpdateCheckpointRequest`.
This is part of pulumi/pulumi-service#1046
And make the deployment an opaque JSON message. The verison field
indicates the schema of the deployment. A missing version field will
behave as if the version was set to `1`. A version of `1` indicates that
the serialized deployment has the `DeploymentV1` schema.
This is part of pulumi/pulumi-service#1046.
This covers most of the transitive closure of the types that appear in a
checkpoint. A breaking change to any of these types implies a bump in
the checkpoint version number.
This is part of pulumi/pulumi-service#1046.
This makes two minor tweaks to the login prompt:
1. Change the text so that it hyperlinks in most terminals, including
iTerm, in a way that doesn't include excess characters.
2. Disable echoing of the token.
A previous change altered the calculation of plugin sizes broke the
ability to install plugins that weren't backed by a directory. This
fixes that by making getPluginSize act correctly for any kind of path.
Due to an interaction between pointers and go's `range` operator, we
would end up always returning the last plugin in a user's plugin
cache, instead of the right value.
We now save the current plugin into a local, so if we end up taking
the address the right thing will happen.
Fixes#1196
* Lift snapshot management out of the engine
This PR is a prerequisite for parallelism by addressing a major problem
that the engine has to deal with when performing parallel resource
construction: parallel mutation of the global snapshot. This PR adds
a `SnapshotManager` type that is responsible for maintaining and
persisting the current resource snapshot. It serializes all reads and
writes to the global snapshot and persists the snapshot to persistent
storage upon every write.
As a side-effect of this, the core engine no longer needs to know about
snapshot management at all; all snapshot operations can be handled as
callbacks on deployment events. This will greatly simplify the
parallelization of the core engine.
Worth noting is that the core engine will still need to be able to read
the current snapshot, since it is interested in the dependency graphs
contained within. The full implications of that are out of scope of this
PR.
Remove dead code, Steps no longer need a reference to the plan iterator that created them
Fixing various issues that arise when bringing up pulumi-aws
Line length broke the build
Code review: remove dead field, fix yaml name error
Rebase against master, provide implementation of StackPersister for cloud backend
Code review feedback: comments on MutationStatus, style in snapshot.go
Code review feedback: move SnapshotManager to pkg/backend, change engine to use an interface SnapshotManager
Code review feedback: use a channel for synchronization
Add a comment and a new test
* Maintain two checkpoints, an immutable base and a mutable delta, and
periodically merge the two to produce snapshots
* Add a lot of tests - covers all of the non-error paths of BeginMutation and End
* Fix a test resource provider
* Add a few tests, fix a few issues
* Rebase against master, fixed merge
We've seen failures in CI where DNS lookups fail which cause our
operations against the service to fail, as well as other sorts of
timeouts.
Add a set of helper methods in a new httputil package that helps us do
retries on these operations, and then update our client library to use
them when we are doing GET requests. We also provide a way for non GET
requests to be retried, and use this when updating a lease (since it
is safe to retry multiple requests in this case).
The RPC provider interface needs a way to convey back to the engine
that a resource being read no longer exists. To do this, we'll return
the ID property that was read back. If it is empty, it means the
resource is gone. If it is non-empty, we expect it to match the input.
We might create an AWS session with one set of credentials, cache it, then
return it when a later caller asked for a session with *different* creds.
Instead, just cache one default session and Copy other sessions from it.
Less important in the CLI, but critical when the engine is used as a library
in a long-running process.
The code that calculated plugin sizes was incorrect; it would show the
total size consumed by all plugins, for each plugin, which is clearly
busted. We should compute each plugin's size from its own directory.
This change skips unknown IDs during read operations. This can happen
when a read is performed using the output property of another resource
during planning. This is intentionally supported via ID being an
Input<ID> and all we need to do for this to work correctly is skip the
actual provider RPC and the runtime will propagate unknown outputs as
usual.
This change lets plugin versions to float in two ways:
1) If a `pulumi plugin install` detects a newer version is available
already, there's no need to download and install the older version.
2) If the engine attempts to load a plugin at a particular version,
if a newer version is available, it will be accepted without error.
As part of this, we permit $PATH to have the final say when determining
which version to accept. That is, it can always override the choice.
Note that I highly suspect, in the limit, that we'll want to stop doing
this for major version incompatibilities. For now, since we don't
envision any such version changes imminently, this will suffice.
This change does three major things:
1. Removes the ability to be logged into multiple clouds at the same
time. Previously, we supported being logged into multiple clouds at
the same time and the CLI would fan out requests and join responses
when needed. In general, this was only useful for Pulumi employees
that wanted run against multiple copies of the service (say production
and staging) but overall was very confusing (for example in the old
world a stack with the same identity could appear twice (since it was
in two backends) which the CLI didn't handle very well).
2. Stops treating the "local" backend as a special thing, from the
point of view of the CLI. Previouly we'd always connect to the local
backend and merge that data with whatever was in clouds we were
connected to. We had gestures like `--local` in `pulumi stack init`
that meant "use the local mode". Instead, to use the local mode now
you run `pulumi login --cloud-url local://` and then you are logged in
the local backend. Since you can only ever be logged into a single
backend, we can remove the `--local` and `--remote` flags from `pulumi
stack init`, it just now requires you to be logged in and creates a
stack in whatever back end you were logged into. When logging into the
local backend, you are not prompted for an access key.
3. Prompt for login in places where you have to log in, if you are not
already logged in.
This change wires up the new Read RPC method in such a manner that
Pulumi programs can invoke it. This is technically not required for
refreshing state programmatically (as in pulumi/pulumi#1081), however
it's a feature we had eons ago and have wanted since (see
pulumi/pulumi#83), and will allow us to write code like
let vm = aws.ec2.Instance.get("my-vm", "i-07043cd97bd2c9cfc");
// use any property from here on out ...
The way this works is simply by bridging the Pulumi program via its
existing RPC connection to the engine, much like Invoke and
RegisterResource RPC requests already do, and then invoking the proper
resource provider in order to read the state. Note that some resources
cannot be uniquely identified by their ID alone, and so an extra
resource state bag may be provided with just those properties required.
This came almost for free (okay, not exactly) and will come in handy as
we start gaining experience with reading live state from resources.
This commit changes two things about our resource model:
* Stop performing Pulumi Engine-side diffing of resource state.
Instead, we defer to the resource plugins themselves to determine
whether a change was made and, if so, the extent of it. This
manifests as a simple change to the Diff function; it is done in
a backwards compatible way so that we continue with legacy diffing
for existing resource provider plugins.
* Add a Read RPC method for resource providers. It simply takes a
resource's ID and URN, plus an optional bag of further qualifying
state, and it returns the current property state as read back from
the actual live environment. Note that the optional bag of state
must at least include enough additional properties for resources
wherein the ID is insufficient for the provider to perform a lookup.
It may, however, include the full bag of prior state, for instance
in the case of a refresh operation.
This is part of pulumi/pulumi#1108.
* Improve the error message arising from missing required configs for
resource providers
If the resource provider that we are speaking to is new enough, it will send
across a list of keys and their descriptions alongside an error
indicating that the provider we are configuring is missing required
config. This commit packages up the list of missing keys into an error
that can be presented nicely to the user.
* Code review feedback: renaming simplification and correcting errors in comments
Tests now target managed stacks instead of local stacks.
The existing logged in user and target backend API are used unless PULUMI_ACCES_TOKEN is defined, in which case tests are run under that access token and against the PULUMI_API backend.
For developer machines, we will now need to be logged in to Pulumi to run tests, and whichever default API backend is logged in (the one listed as current in ~/.pulumi/credentials.json) will be used. If you need to override these, provide PULUMI_ACCESS_TOKEN and possibly PULUMI_API.
For Travis, we currently target the staging service using the Pulumi Bot user.
We have decided to run tests in the pulumi organization. This can be overridden for local testing (or in Travis in the future) by defining PULUMI_API_OWNER_ORGANIZATION and using an access token with access to that organization.
Part of pulumi/home#195.
