I noticed in our Docker builds, we often end up seeing %(MISSING)!
style messages, which were an indication we were trying to format
them. The reason was the presence of %c's in the stream, and the
fact that we passed said messages to Fprintf. We were careful in
all other layers to use the message on the "right hand side" of
any *f calls, but in this instance, we used Fprintf and passed the
message on the "left hand side", triggering formatting. It turns
out we've already formatted everything by the time we get here,
so there's no need -- we can just use Fprint instead.
Adds `pulumi update` so you can deploy to the Pulumi Console (via PPC on the backend).
As per an earlier discussion (now lost because I rebased/squashed the commits), we want to be more deliberate about how to bifurcate "local" and "cloud" versions of every Pulumi command.
We can block this PR until we do the refactoring to have `pulumi` commands go through a generic "PulumiCloud" interface. But it would be nice to commit this so I can do more refining of the `pulumi` -> Console -> PPC workflow.
Another known area that will need to be revisited is how we render the PPC events on the CLI. Update events from the PPC are generated in a different format than the `engine.Event`, and we'll probably want to change the PPC to emit messages in the same format. (e.g. how we handle coloring, etc.)
We now encrypt secrets at rest based on a key derived from a user
suplied passphrase.
The system is designed in a way such that we should be able to have a
different decrypter (either using a local key or some remote service
in the Pulumi.com case in the future).
Care is taken to ensure that we do not leak decrypted secrets into the
"info" section of the checkpoint file (since we currently store the
config there).
In addtion, secrets are "pay for play", a passphrase is only needed
when dealing with a value that's encrypted. If secure config values
are not used, `pulumi` will never prompt you for a
passphrase. Otherwise, we only prompt if we know we are going to need
to decrypt the value. For example, `pulumi config <key>` only prompts
if `<key>` is encrypted and `pulumi deploy` and friends only prompt if
you are targeting a stack that has secure configuration assoicated
with it.
Secure values show up as unecrypted config values inside the language
hosts and providers.
Trapping these signals hijacks the usual termination behavior for any
program that happens to link in the engine and perform an operation
that starts a gRPC server. These servers already provide a cancellation
mechanism via a `cancel` channel parameter; if the using program wants
to gracefully terminate these servers on some signal, it is responsible
for providing that behavior.
This also fixes a leak in which the goroutine responsible for waiting on
a server's signals and cancellation channel would never exit.
When `PULUMI_RETAIN_CHECKPOINTS` is set in the environment, also write
the checkpoint file to <stack-name>.<ext>.<timestamp>.
This ensures we have historical information about every snapshot, which
would aid in debugging issues like #451. We set this to true for our
integration tests.
Fixes#453
This change fixes a couple bugs with assets:
* We weren't recursing into subdirectories in the new "path as
archive" feature, which meant we missed most of the files.
* We need to make paths relative to the root of the archive
directory itself, otherwise paths end up redundantly including
the asset's root folder path.
* We need to clean the file paths before adding them to the
archive asset map, otherwise they are inconsistent between the
path, tar, tgz, and zip cases.
* Ignore directories when traversing zips, since they aren't
included in the other formats.
* Tolerate io.EOF errors when reading the ZIP contents into blobs.
* Add test cases for the four different archive kinds.
This fixespulumi/pulumi-aws#50.
The prior code was a little too aggressive in rejected undefined
properties, because it assumed any occurrence indicated a resource
that was unavailable due to planning. This is a by-produt of our
relatively recent decision to flow undefineds freely during planning.
The problem is, it's entirely legitimate to have undefined values
deep down in JavaScript structures, entirely unrelated to resources
whose property values are unknown due to planning.
This change flows undefined more freely. There really are no
negative consequences of doing so, and avoids hitting some overly
aggressive assertion failures in some important scenarios. Ideally
we would have a way to know statically whether something is a resource
property, and tighten up the assertions just to catch possible bugs
in the system, but because this is JavaScript, and all the assertions
are happening at runtime, we simply lack the necessary metadata to do so.
Now that config is stored in Pulumi.yaml, we need to mimic the behavior
around .pulumi/ while edits are applied. This will ensure that config
values carry forward from the original program settings.
This fixespulumi/pulumi-aws#48.
The event diagnostic goroutines could error out sometimes during
early program exits, due to a race between the goroutine writing to
the channel and the early exiting goroutine which closed the channel.
This change stops closing the channels entirely on the abrupt exit
paths, since it's not necessary and we want to exit immediately.
This improves a few things about assets:
* Compute and store hashes as input properties, so that changes on
disk are recognized and trigger updates (pulumi/pulumi#153).
* Issue explicit and prompt diagnostics when an asset is missing or
of an unexpected kind, rather than failing late (pulumi/pulumi#156).
* Permit raw directories to be passed as archives, in addition to
archive formats like tar, zip, etc. (pulumi/pulumi#240).
* Permit not only assets as elements of an archive's member list, but
also other archives themselves (pulumi/pulumi#280).
The existing `SnapshotProvider` interface does not sufficiently lend
itself to reliable persistence of snapshot data. For example, consider
the following:
- The deployment engine creates a resource
- The snapshot provider fails to save the updated snapshot
In this scenario, we have no mechanism by which we can discover that the
existing snapshot (if any) does not reflect the actual state of the
resources managed by the stack, and future updates may operate
incorrectly. To address this, these changes split snapshotting into two
phases: the `Begin` phase and the `End` phase. A provider that needs to
be robust against the scenario described above (or any other scenario
that allows for a mutation to the state of the stack that is not
persisted) can use the `Begin` phase to persist the fact that there are
outstanding mutations to the stack. It would then use the `End` phase to
persist the updated snapshot and indicate that the mutation is no longer
outstanding. These steps are somewhat analogous to the prepare and
commit phases of two-phase commit.
If --logtostderr is passed, and an unhandled error occurs that
was produced by the github.com/pkg/errors package, we will now
emit the stack trace. Much easier for debugging purposes.
The change to use a Goroutine for pumping output causes a hang
when an error occurs. This is because we unconditionally block
on the <-done channel, even though the failure means the done
will actually never occur. This changes the logic to only wait
on the channel if we successfully began the operation in question.
Instead of doing the logic to see if a type has YAML tags and then
dispatching based on that to use either the direct go-yaml marshaller
or the one that works in terms of JSON tags, let's just say that we
always add YAML tags as well, and use go-yaml directly.
Previously, config information was stored per stack. With this change,
we now allow config values which apply to every stack a program may
target.
When passed without the `-s <stack>` argument, `pulumi config`
operates on the "global" configuration. Stack specific information can
be modified by passing an explicit stack.
Stack specific configuration overwrites global configuration.
Conside the following Pulumi.yaml:
```
name: hello-world
runtime: nodejs
description: a hello world program
config:
hello-world:config:message Hello, from Pulumi
stacks:
production:
config:
hello-world:config:message Hello, from Production
```
This program contains a single configuration value,
"hello-world:config:message" which has the value "Hello, from Pulumi"
when the program is activated into any stack except for "production"
where the value is "Hello, from Production".
Instead of having information stored in the checkpoint file, save it
in the Pulumi.yaml file. We introduce a new section `stacks` which
holds information specific to a stack.
Next, we'll support adding configuration information that applies
to *all* stacks for a Program and allow the stack specific config to
overwrite or augment it.
This will allow us to marhsall and unmarshall the structure using the
go-yaml package directly, instead of the current way we handle YAML
which is to treat it as JSON and use the JSON marshaller behind the
scenes.
By having the explicit tags and using go-yaml directly, we can ensure
the resulting order of the resulting document's properties matches the
go type instead of just being lexigraphicaly sorted.
Long term, we probably want to stop using go-yaml or other packages in
favor of parsing the YAML file into some sort of DOM that we can use
to retain comments and other formatting in the file.
This PR adds `login` and `logout` commands to the `pulumi` CLI.
Rather than requiring a user name and password like before, we instead require users to login with GitHub credentials on the Pulumi Console website. (You can do this now via https://beta.moolumi.io.) Once there, the account page will show you an "access token" you can use to authenticate against the CLI.
Upon successful login, the user's credentials will be stored in `~/.pulumi/credentials.json`. This credentials file will be automatically read with the credentials added to every call to `PulumiRESTCall`.
This change adds functions, `pulumi.getProject()` and `pulumi.getStack()`,
to fetch the names of the project and stack, respectively. These can be
handy in generating names, specializing areas of the code, etc.
This fixespulumi/pulumi#429.
During the course of a `pulumi update`, it is possible for a resource to
become slated for deletion. In the case that this deletion is part of a
replacement, another resource with the same URN as the to-be-deleted
resource will have been created earlier. If the `update` fails after the
replacement resource is created but before the original resource has been
deleted, the snapshot must capture that the original resource still exists
and should be deleted in a future update without losing track of the order
in which the deletion must occur relative to other deletes. Currently, we
are unable to track this information because the our checkpoints require
that no two resources have the same URN.
To fix this, these changes introduce to the update engine the notion of a
resource that is pending deletion and change checkpoint serialization to
use an array of resources rather than a map. The meaning of the former is
straightforward: a resource that is pending deletion should be deleted
during the next update.
This is a fairly major breaking change to our checkpoint files, as the
map of resources is no more. Happily, though, it makes our checkpoint
files a bit more "obvious" to any tooling that might want to grovel
or rewrite them.
Fixes#432, #387.
A dynamic resource is a resource whose provider is implemented alongside
the resource itself. This provider may close over and use orther
resources in the implementation of its CRUD operations. The provider
itself must be stateless, as each CRUD operation for a particular
dynamic resource type may use an independent instance of the provider.
Changes to the definition of a resource's provider result in replacement
of the resource itself (rather than a simple update), as this allows the
old provider definition to delete the old resource and the new provider
definition to create an appropriate replacement.
If a plugin fails to load after we've set up the goroutines that copy
from its std{out,err} streams, then those goroutines can end up writing
to a closed event channel. This change ensures that we properly drain
those streams in this case.
In #411 we started to run tests in parallel again. To support that, we
added a .yarnrc file in the root of our repository to pass --mutex
network to all yarn invocations, because tests may run yarn commands
concurrently with one another and yarn is not safe to run
concurrently.
However, when we run the integration tests, we actually copy them into
a folder under `/tmp` and so yarn's logic to walk up the directory to
tree to find a `.yarnrc` will not see this `.yarnrc` and we're back
where we started.
Have the testing package explicitly write this file, which should
prevent this issue from happening in the future.
Previously we used the word "Environment" as the term for a deployment
target, but since then we've started to use the term Stack. Adopt this
across the CLI.
From a user's point of view, there are a few changes:
1. The `env` verb has been renamed to `stack`
2. The `-e` and `--env` options to commands which operate on an
environment now take `-s` or `--stack` instead.
3. Becase of (2), the commands that used `-s` to display a summary now
only support passing the full option name (`--summary`).
On the local file system, we still store checkpoint data in the `env`
sub-folder under `.pulumi` (so we can reuse existing checkpoint files
that were written to the old folder)
This changes a few things about "components":
* Rename what was previously ExternalResource to CustomResource,
and all of the related fields and parameters that this implies.
This just seems like a much nicer and expected name for what
these represent. I realize I am stealing a name we had thought
about using elsewhere, but this seems like an appropriate use.
* Introduce ComponentResource, to make initializing resources
that merely aggregate other resources easier to do correctly.
* Add a withParent and parentScope concept to Resource, to make
allocating children less error-prone. Now there's no need to
explicitly adopt children as they are allocated; instead, any
children allocated as part of the withParent callback will
auto-parent to the resource provided. This is used by
ComponentResource's initialization function to make initialization
easier, including the distinction between inputs and outputs.
This change implements core support for "components" in the Pulumi
Fabric. This work is described further in pulumi/pulumi#340, where
we are still discussing some of the finer points.
In a nutshell, resources no longer imply external providers. It's
entirely possible to have a resource that logically represents
something but without having a physical manifestation that needs to
be tracked and managed by our typical CRUD operations.
For example, the aws/serverless/Function helper is one such type.
It aggregates Lambda-related resources and exposes a nice interface.
All of the Pulumi Cloud Framework resources are also examples.
To indicate that a resource does participate in the usual CRUD resource
provider, it simply derives from ExternalResource instead of Resource.
All resources now have the ability to adopt children. This is purely
a metadata/tagging thing, and will help us roll up displays, provide
attribution to the developer, and even hide aspects of the resource
graph as appropriate (e.g., when they are implementation details).
Our use of this capability is ultra limited right now; in fact, the
only place we display children is in the CLI output. For instance:
+ aws:serverless:Function: (create)
[urn=urn:pulumi:demo::serverless::aws:serverless:Function::mylambda]
=> urn:pulumi:demo::serverless::aws:iam/role:Role::mylambda-iamrole
=> urn:pulumi:demo::serverless::aws:iam/rolePolicyAttachment:RolePolicyAttachment::mylambda-iampolicy-0
=> urn:pulumi:demo::serverless::aws:lambda/function:Function::mylambda
The bit indicating whether a resource is external or not is tracked
in the resulting checkpoint file, along with any of its children.
