Per #1481, we unintentionally disallowed empty text assets, simply
because of the way we did discriminated union testing. It's easy
enough to just treat the empty asset like an empty string asset.
This commit adds CLI support for resource providers to provide partial
state upon failure. For resource providers that model resource
operations across multiple API calls, the Provider RPC interface can now
accomodate saving bags of state for resource operations that failed.
This is a common pattern for Terraform-backed providers that try to do
post-creation steps on resource as part of Create or Update resource
operations.
If we've hit the maximum number of retry attempts, return `true` for
`done`. As it stands, it is possible for us to return a `nil` error
(e.g. if the request sends successfully but returns a `50x` status) and
`false` for `done`, which `retry.Until` will interpret as "keep
retrying".
As an alternative we could consider simply returning an empty `Blob`.
I think it's better to fail loudly, though, as this situation can result
from unexpected/invalid archives (which we need to do a better job of
validating).
Fixes#1493.
This allows us to delete the one off export/import test, which is nice
because it failed to run when PULUMI_ACCESS_TOKEN was not set in the
environment (due to an interaction between `pulumi login` and the
bare-bones integration test framework)
Because we run our golang integration tests "in tree" (due to
the need to be under $GOPATH and have a vendor folder around), the
"command-output" folder was getting left behind, dirtying the worktree
after building.
This change does two things:
1. On a succecssful run, remove the folder.
2. Ignore the folder via .gitignore (this way if a test fails and you
do `git add .` you don't end up commiting this folder).
This change makes it a little easier to do the style of highlighting
we are doing now with the login prompt, cleaning up some of the
padding calculations that were otherwise complicated due to ANSI
escape sequences.
This change makes our login prompt a little "friendlier", especially
important since this will be the first thing a new user sees.
The new message is:
$ pulumi new
We need your Pulumi account to identify you.
Enter your access token from https://app.pulumi.com/account
or hit <ENTER> to log in using your browser :
The ZIP format started with MS-DOS dates, which start in 1980. Other dates
have been layered on, but the ZIP file handler used by Azure websites still
relies on the MS-DOS dates.
Using the Unix epoch here (1970) results in ZIP entries that (e.g.)
OSX `unzip` sees as 12-31-1969 (timezones) but Azure websites sees as
01/01/2098.
Due to the way GOPATH and vendoring works, copying Go tests out to a
random temp directory simply will not work. This is largely a holdover
to the days of when .pulumi/ directories would pollute the directory,
but is also done for languages like Node.js whose preparatory and build
steps also pollute the working directory (with bin/ and node_modules/
directories). Go does not have this problem, so we can safely skip.
Previously, we would include information about what git commit a build
came from in the "local" portion of the PEP-440 version. This was a
problem because PyPI does not allow packages to be upload to the
registry if they contain local parts.
So, for now, we'll just never put in the git commit information in the
generate version. We'll continue to add a dirty tag in the local
part. This will be prevent us from publishing dirty builds to PyPI,
but that's in line with what we want.
Today we defaulted our tests to create stacks in the `pulumi`
organization. We did this because our tests run with `pulumi-bot` and
we'd rather create the stacks in our shared organization, so any
Pulumi developer can see them.
Of course, as we prepare to have folks outside of the Pulumi
organization write and run tests, this has now become a bad default.
Remove the ability to explicitly set an owner in
ProgramTestOptions (since that would more or less only lead to pain
going forward) and default to just creating the stacks in whatever
account is currently logged in. In CI, we'll set a new environment
variable "PULUMI_TEST_OWNER" which controls the owner of the stacks,
which we'll set to `pulumi`.
Impact to day to day developers is during test runs locally you'll see
stacks in your list of stacks. If any of the tests fail to clean up,
you'll see these lingering stacks (but you can go clean them up).
This change adds some convenience functions and types, to make strongly
typed outputs more pleasant to interact with. It also includes tests
for output generally, in addition to these new functions and types.
This adds rudimentary support for Pulumi programs written in Go. It
is not complete yet but the basic resource registration works.
Note that, stylistically speaking, Go is a bit different from our other
languages. This made it a bit easier to build this initial prototype,
since what we want is actually a rather thin veneer atop our existing
RPC interfaces. The lack of generics, however, adds some friction and
is something I'm continuing to hammer on; this will most likely lead to
little specialized types (e.g. StringOutput) once the dust settles.
There are two primary components:
1) A new language host, `pulumi-language-go`, which is responsible for
communicating with the engine through the usual gRPC interfaces.
Because Go programs are pre-compiled, it very simply loads a binary
with the same name as the project.
2) A client SDK library that Pulumi programs bind against. This exports
the core resource types -- including assets -- properties -- including
output properties -- and configuration.
Most remaining TODOs are marked as such in the code, and this will not
be merged until they have been addressed, and some better tests written.
math/rand uses a fixed seed, meaning that across runs the Kth call to
`rand.Int63()` will always return the same value.
Given that we want to provide a unique suffix across multiple
concurrent runs, this isn't the behavior we want.
I saw an instance fail in CI where all three legs ran the test
concurrently and they raced on creating the test stack, since they all
generated the same name.
Now that the "config" member of Pulumi.yaml has been deprecated for a
while, we'll change it's meaning. When set, the value is treated as a
path and joined with the path to Pulumi.yaml, and per stack
configuration is stored in that folder
Fixes#1031
This lets us set these values globaly, in our Travis and AppVeyor
configurations instead of forcing every test to opt-in. It also means
that by default, local builds will not report any of this data (and
will not need access to these endpoints).
The Version field was inadvertently dropped when sending an import
request to the service. Now that we are requiring that the Version field
be set in deployments, this was causing errors.
Calling `bytes.NewBuffer` gives up ownership of the backing memory and creates
a _mutable_ buffer. We don't need that, so let's use `bytes.NewReader` instead.
Not sending request body on retry is bad enough, but we keep the original
`Content-Length` and the server ends up expecting a body we never send.
This results in `400 Bad Request` errors from the ALB.
* Error when loading a deployment that is not a version that the CLI understands
* Add a test for 'pulumi stack import' on a badly-versioned deployment
* Move current deployment version to 'apitype'
* Rebase against master
* CR: emit CLI-friendly error message at the two points outside of the engine calling 'DeserializeDeployment'
An empty preview is not a guarantee that a subsequent update will cause
no changes. As such, we should always offer the user the option to
perform an update even if the preview proposes no changes.
Note that this indirectly fixes#1410 by masking the effects of #1429.
This change supports displaying manifest information for a stack and
changes the way we handle Snapshots in our backend.
Previously, every call to GetStack would synthesize a Snapshot by
taking the set of resources returned from the
`/api/stacks/<owner>/<name>` endpoint, combined with an empty
manfiest (since the service was not returning the manifest).
This wasn't great for two reasons:
1. We didn't have manifest information, so we couldn't display any of
its information (most important the last updated time).
2. This strategy required that the service return all the resources
for a stack anytime GetStack was called. While the CLI did not
often need this detailed information the fact that we forced the
Service to produce it (which in the case of stack managed PPC would
require the service to talk to yet another service) creates a bunch
of work that we end up ignoring.
I've refactored the code such that `backend.Stack`'s `Snapshot()` method
now lazily requests the information from the service such that we can
construct a `Snapshot()` on demand and only pay the cost when we
actually need it.
I think making more of this stuff lazy is the long term direction we
want to follow.
Unfortunately, right now, it means in cases where we do need this data
we end up fetching it twice. The service does it once when we call
GetStack and then we do it again when we actually need to get at the
Snapshot. However, once we land this change, we can update the
service to no longer return resources on the apistack.Stack type. The
CLI no longer needs this property. We'll likely want to continue in a
direction where `apistack.Stack` can be created quickly by the
service (without expensive database queries or fetching remote
resources) and just add additional endpoints that let us get at the
specific information we want in the specific cases when we want it
instead of forcing us to return a bunch of data that we often ignore.
Fixespulumi/pulumi-service#371
* Delete Before Create
This commit implements the full semantics of delete before create. If a
resource is replaced and requires deletion before creation, the engine
will use the dependency graph saved in the snapshot to delete all
resources that depend on the resource being replaced prior to the
deletion of the resource to be replaced.
* Rebase against master
* CR: Simplify the control flow in makeRegisterResourceSteps
* Run Check on new inputs when re-creating a resource
* Fix an issue where the planner emitted benign but incorrect deletes of DBR-deleted resources
* CR: produce the list of dependent resources in dependency order and iterate over the list in reverse
* CR: deps->dependents, fix an issue with DependingOn where duplicate nodes could be added to the dependent set
* CR: Fix an issue where we were considering old defaults and new inputs
inappropriately when re-creating a deleted resource
* CR: save 'iter.deletes[urn]' as a local, iterate starting at cursorIndex + 1 for dependency graph
At some point the summary and preview parameters to
`engine.GetResourcePropertiesDetails` were flipped s.t. we stopped
rendering computed<> and output<> values properly during previews.
We currently display at most two rows for messages that are not
associated with a URN: one for those that arrive before the stack
resource is first observed and one for those that arrive after the stack
resource is first observed. The former is labeled "global"; the latter
is the row associated with the stack resource. These changes remove the
duplication by reassociating the row used for events with no URN that
arrive before the stack resource with the stack resource once it has
been observed.
Fixes#1385.
Our current strategy for the progress display on non-TTYs causes us to
display multiple identical rows for each resource when the row is a
preview or a no-op. This behavior is not particularly useful, and
generally just makes the display noisier than it needs to be.
These changes avoid displaying resource rows without meaningful output
by supressing the display of resource output events that are delivered
during a preview or that correspond to a no-op update.
Fixes#1384.
This changes two things:
1) Eliminates the fact that we had two kinds of previews in our engine.
2) Always initialize the plugin.Events, to ensure that all plugin loads
are persisted no matter the update type (update, refresh, destroy),
and skip initializing it when dryRun == true, since we won't save them.
The PluginEvents will now try to register loaded plugins which,
during a refresh preview, will result in attempting to save mutations
when a token is missing. This change mirrors the changes made to
destroy which avoid it panicing similarly, by simply leaving
PluginEvents unset. Also adds a bit of tracing that was helpful to
me as I debugged through the underlying issues.
Fixes#1377.
Before we were reading a boolean variable that was set by another goroutine.
Without any sync primitives, I think Go is within its rights to optimize
that away.
We now require that `ctx` is non-`nil`. This was already true for all callers
and consistent with other Go functions that take contexts.
* Graceful RPC shutdown: CLI side
* Handle unavailable resource monitor in language hosts
* Fix a comment
* Don't commit package-lock.json
* fix mangled pylint pragma
* Rebase against master and fix Gopkg.lock
* Code review feedback
* Fix a race between closing the callerEventsOpt channel and terminating a goroutine that writes to it
* glog -> logging
During a deployment, we end up making a bunch of PATCH/PUT/POST style
REST requests. For these calls, it should be safe to retry the
operation if there was a hickup during our REST call, so mark them as
retryable.
These changes add support for pre- and post-`pulumi` callbacks to the
integration test framework. These callbacks will be invoked immediately
before and after (respectively) running a `pulumi` command.
In pulumi/pulumi#1356, we observed that we can fail during a destroy
because we attempt to load the language plugin, which now eagerly looks
for the @pulumi/pulumi package.
This is also blocking ingestion of the latest engine bits into the PPC.
It turns out that for destroy (and refresh), we have no need for the
language plugin. So, let's skip loading it when appropriate.
These changes add support for adding a tracing header to API requests
made to the Pulumi service. Setting the `PULUMI_TRACING_HEADER`
environment variable or enabling debug commands and passing the
`--tracing-header` will change the value sent in this header. Setting
this value to `1` will request that the service enable distributed
tracing for all requests made by a particular CLI invocation.
The newly added `pulumi config refresh` updates your local copy of the
Pulumi.<stack-name>.yaml file to have the same configuration as the
most recent deployment in the cloud.
This can be used in a varirty of ways. One place we plan to use it is
in automation to clean up "leaked" stacks we have in CI. With the
changes you'll now be able to do the following:
```
$ cd $(mktemp -d)
$ echo -e "name: who-cares\nruntime: nodejs" > Pulumi.yaml
$ pulumi stack select <leaked-stack-name>
$ pulumi config refresh -f
$ pulumi destroy --force
```
Having a simpler gesture for the above is something we'll want to do
long term (we should be able to support `pulumi destory <stack-name>`
from a completely empty folder, today you need a Pulumi.yaml file
present, even if the contents don't matter).
But this gets us a little closer to where we want to be and introduces
a helpful primitive in the system.
Contributes to #814
This change suppresses the warning
warning: resource plugin aws is expected to have version >=0.11.3,
but has 0.11.1-dev-1523506162-g06ec765; the wrong version may
be on your path, or this may be a bug in the plugin
when the PULUMI_DEV envvar is set to a truthy value.
This warning keeps popping up in demos since I'm always using dev builds
and I'd like a way to shut it off, even though this can legitimately
point out a problem. Eventually I'll switch to official buildsa but,
until then, it seems worth having a simple way to suppress.
These changes add support for injecting client tracing spans into HTTP
requests to the Pulumi API. The server can then rematerialize these span
references and attach its own spans for distributed tracing.
This change captures the Git committer and author's login and email
addresses, so that we can display them prominently in the service. At
the moment, we only attribute updates to the identity that performed the
update which, in CI scenarios, is often always the same person for an
organization. This makes Pulumi look like a needlessly lonely place.
These changes enable tracing of Pulumi API calls.
The span with which to associate an API call is passed via a
`context.Context` parameter. This required plumbing a
`context.Context` parameter through a rather large number of APIs,
especially in the backend.
In general, all API calls are associated with a new root span that
exists for essentially the entire lifetime of an invocation of the
Pulumi CLI. There were a few places where the plumbing got a bit hairier
than I was willing to address with these changes; I've used
`context.Background()` in these instances. API calls that receive this
context will create new root spans, but will still be traced.
As it stands, we recompute the workspace for the current directory
potentially many times during some CLI operations, most notably
`stack ls`. These changes add a simple cache based on the complete path
passed to `NewFrom`, and eliminate some lagginess in `stack ls` when
there are multiple stacks.
Another option is to calculate the current workspace once in the CLI and
then fetch it as necessary.
* Initialize a new stack as part of `pulumi new`
* Prompt for values with defaults preselected
* Install dependencies
* Prompt for default config values
Our logic for converting npm style versions to PEP-440 style versions
was not correct in some cases. This change fixes this.
As part of this change we no longer produce a NPM version that would
be just X.Y.Z-dev, instead for development versions we always include
both the timestamp of the commit and the commit hash.
Instead of trying to use a bunch of sed logic to do our conversions,
we now have a small go program that uses a newly added library in
pkg/util. A side effect of this is that we can more easily write tests
to ensure the conversion works as expected.
Fixes#1243
If the file contains a zero byte within the first 8000 bytes (or the entire length of the file if shorter), treat the file as binary and do not do any textual transformations. This is the same approach git uses to determine if a file is binary.
This changes the CLI interface in a few ways:
* `pulumi preview` is back! The alternative of saying
`pulumi update --preview` just felt awkward, and it's a common
operation to want to perform. Let's just make it work.
* There are two flags consistent across all update commands,
`update`, `refresh`, and `destroy`:
- `--skip-preview` will skip the preview step. Note that this
does *not* skip the prompt to confirm that you'd like to proceed.
Indeed, it will still prompt, with a little warning text about
the fact that the preview has been skipped.
* `--yes` will auto-approve the updates.
This lands us in a simpler and more intuitive spot for common scenarios.
I found the flag --force to be a strange name for skipping a preview,
since that name is usually reserved for operations that might be harmful
and yet you're coercing a tool to do it anyway, knowing there's a chance
you're going to shoot yourself in the foot.
I also found that what I almost always want in the situation where
--force was being used is to actually just run a preview and have the
confirmation auto-accepted. Going straight to --force isn't the right
thing in a CI scenario, where you actually want to run a preview first,
just to ensure there aren't any issues, before doing the update.
In a sense, there are four options here:
1. Run a preview, ask for confirmation, then do an update (the default).
2. Run a preview, auto-accept, and then do an update (the CI scenario).
3. Just run a preview with neither a confirmation nor an update (dry run).
4. Just do an update, without performing a preview beforehand (rare).
This change enables all four workflows in our CLI.
Rather than have an explosion of flags, we have a single flag,
--preview, which can specify the mode that we're operating in. The
following are the values which correlate to the above four modes:
1. "": default (no --preview specified)
2. "auto": auto-accept preview confirmation
3. "only": only run a preview, don't confirm or update
4. "skip": skip the preview altogether
As part of this change, I redid a bit of how the preview modes
were specified. Rather than booleans, which had some illegal
combinations, this change introduces a new enum type. Furthermore,
because the engine is wholly ignorant of these flags -- and only the
backend understands them -- it was confusing to me that
engine.UpdateOptions stored this flag, especially given that all
interesting engine options _also_ accepted a dryRun boolean. As of
this change, the backend.PreviewBehavior controls the preview options.
As of this change, the engine will run all Configure calls in parallel.
This improves startup performance, since otherwise, we would block
waiting for all plugins to be configured before proceeding to run a
program. Emperically, this is about 1.5-2s for AWS programs, and
manifests as a delay between the purple "Previewing update of stack"
being printed, and the corresponding grey "Previewing update" message.
This is done simply by using a Goroutine for Configure, and making sure
to synchronize on all actual CRUD operations. I toyed with using double
checked locking to eliminate lock acquisitions -- something we may want
to consider as we add more fine-grained parallelism -- however, I've
kept it simple to avoid all the otherwise implied memory model woes.
I made the judgment call that GetPluginInfo may proceed before
Configure has settled. (Otherwise, we'd immediately call it and block
after loading the plugin, obviating the parallelism benefits.) I also
made the judgment call to do this in the engine, after flip flopping
several times about possibly making it a provider's own decision.
These changes add a new flag to the CLI, `--profiling`, that enables
CPU and heap profiling as well as execution tracing of the CLI itself.
The argument to this flag serves as a prefix for the profile outputs;
the CPU and heap profiles and execution trace are written to
`[filename].[pid].{cpu,mem,trace}`, respectively.
These changes also fix an issue with `cmdutil.RunFunc` wherein any error
would prevent the command's post-run hooks from executing.
The service has no good way to preview a destroy operation for a stack
managed by a PPC. Until it does, just behave as if --force was passed
to the CLI in this case (i.e. skip the preview).
Fixes#1301
