This matches what we tell our customers to do and makes some
downstream testing stuff for beta versions of our python packages
easier (since `pip` has a more straightforward package selection
algorithm than `pipenv`)
This will allow us to install a latest version from PyPi or NPM
and then yarn link / pip install from local machine and test that
no changes are introduced when this occurs
If the flag is not passed, then the dependencies are installed
as before where we prefer a local package to be linked / installed
Adds support for additional cloud secrets providers (AWS KMS, Azure KeyVault, Google Cloud KMS, and HashiCorp Vault) as the encryption backend for Pulumi secrets. This augments the previous choice between using the app.pulumi.com-managed secrets encryption or a fully-client-side local passphrase encryption.
This is implemented using the Go Cloud Development Kit support for pluggable secrets providers.
Like our cloud storage backend support which also uses Go Cloud Development Kit, this PR also bleeds through to users the URI scheme's that the Go CDK defines for specifying each of secrets providers - like `awskms://alias/LukeTesting?region=us-west-2` or `azurekeyvault://mykeyvaultname.vault.azure.net/keys/mykeyname`.
Also like our cloud storage backend support, this PR doesn't solve for how to configure the cloud provider client used to resolve the URIs above - the standard ambient credentials are used in both cases. Eventually, we will likely need to provide ways for both of these features to be configured independently of each other and of the providers used for resource provisioning.
Dynamic providers in Python.
This PR uses [dill](https://pypi.org/project/dill/) for code serialization, along with a customization to help ensure deterministic serialization results.
One notable limitation - which I believe is a general requirement of Python - is that any serialization of Python functions must serialize byte code, and byte code is not safely versioned across Python versions. So any resource created with Python `3.x.y` can only be updated by exactly the same version of Python. This is very constraining, but it's not clear there is any other option within the realm of what "dynamic providers" are as a feature. It is plausible that we could ensure that updates which only update the serialized provider can avoid calling the dynamic provider operations, so that version updates could still be accomplished. We can explore this separately.
```py
from pulumi import ComponentResource, export, Input, Output
from pulumi.dynamic import Resource, ResourceProvider, CreateResult, UpdateResult
from typing import Optional
from github import Github, GithubObject
auth = "<auth token>"
g = Github(auth)
class GithubLabelArgs(object):
owner: Input[str]
repo: Input[str]
name: Input[str]
color: Input[str]
description: Optional[Input[str]]
def __init__(self, owner, repo, name, color, description=None):
self.owner = owner
self.repo = repo
self.name = name
self.color = color
self.description = description
class GithubLabelProvider(ResourceProvider):
def create(self, props):
l = g.get_user(props["owner"]).get_repo(props["repo"]).create_label(
name=props["name"],
color=props["color"],
description=props.get("description", GithubObject.NotSet))
return CreateResult(l.name, {**props, **l.raw_data})
def update(self, id, _olds, props):
l = g.get_user(props["owner"]).get_repo(props["repo"]).get_label(id)
l.edit(name=props["name"],
color=props["color"],
description=props.get("description", GithubObject.NotSet))
return UpdateResult({**props, **l.raw_data})
def delete(self, id, props):
l = g.get_user(props["owner"]).get_repo(props["repo"]).get_label(id)
l.delete()
class GithubLabel(Resource):
name: Output[str]
color: Output[str]
url: Output[str]
description: Output[str]
def __init__(self, name, args: GithubLabelArgs, opts = None):
full_args = {'url':None, 'description':None, 'name':None, 'color':None, **vars(args)}
super().__init__(GithubLabelProvider(), name, full_args, opts)
label = GithubLabel("foo", GithubLabelArgs("lukehoban", "todo", "mylabel", "d94f0b"))
export("label_color", label.color)
export("label_url", label.url)
```
Fixes https://github.com/pulumi/pulumi/issues/2902.
This commit will expose the new `Invoke` routine that lists resource
outputs through the Node.js SDK.
This API is implemented via a new API, `EnumerablePromise`, which is a
collection of simple query primitives built onto the `Promise` API. The
query model is lazy and LINQ-like, and generally intended to make
`Promise` simpler to deal with in query scenarios. See #2601 for more
details.
Fixes#2600.
Adds a new resource option `aliases` which can be used to rename a resource. When making a breaking change to the name or type of a resource or component, the old name can be added to the list of `aliases` for a resource to ensure that existing resources will be migrated to the new name instead of being deleted and replaced with the new named resource.
There are two key places this change is implemented.
The first is the step generator in the engine. When computing whether there is an old version of a registered resource, we now take into account the aliases specified on the registered resource. That is, we first look up the resource by its new URN in the old state, and then by any aliases provided (in order). This can allow the resource to be matched as a (potential) update to an existing resource with a different URN.
The second is the core `Resource` constructor in the JavaScript (and soon Python) SDKs. This change ensures that when a parent resource is aliased, that all children implicitly inherit corresponding aliases. It is similar to how many other resource options are "inherited" implicitly from the parent.
Four specific scenarios are explicitly tested as part of this PR:
1. Renaming a resource
2. Adopting a resource into a component (as the owner of both component and consumption codebases)
3. Renaming a component instance (as the owner of the consumption codebase without changes to the component)
4. Changing the type of a component (as the owner of the component codebase without changes to the consumption codebase)
4. Combining (1) and (3) to make both changes to a resource at the same time
When constructing a Deployment (which is a plaintext representation of
a Snapshot), ensure that we encrypt secret values. To do so, we
introduce a new type `secrets.Manager` which is able to encrypt and
decrypt values. In addition, it is able to reflect information about
itself that can be stored in the deployment such that we can
deserialize the deployment into a snapshot (decrypting the values in
the process) without external knowledge about how it was encrypted.
The ability to do this is import for allowing stack references to
work, since two stacks may not use the same manager (or they will use
the same type of manager, but have different state).
The state value is stored in plaintext in the deployment, so it **must
not** contain sensitive data.
A sample manager, which just base64 encodes and decodes strings is
provided, as it useful for testing. We will allow it to be varried
soon.
We changed the `pulumi update` command to be `pulumi up` a while back
(`update` is an alias of `up`). This change just makes it so we refer to
the actual command, `pulumi up`, instead of the older `pulumi update`.
This implements the new algorithm for deciding which resources must be
deleted due to a delete-before-replace operation.
We need to compute the set of resources that may be replaced by a
change to the resource under consideration. We do this by taking the
complete set of transitive dependents on the resource under
consideration and removing any resources that would not be replaced by
changes to their dependencies. We determine whether or not a resource
may be replaced by substituting unknowns for input properties that may
change due to deletion of the resources their value depends on and
calling the resource provider's Diff method.
This is perhaps clearer when described by example. Consider the
following dependency graph:
A
__|__
B C
| _|_
D E F
In this graph, all of B, C, D, E, and F transitively depend on A. It may
be the case, however, that changes to the specific properties of any of
those resources R that would occur if a resource on the path to A were
deleted and recreated may not cause R to be replaced. For example, the
edge from B to A may be a simple dependsOn edge such that a change to
B does not actually influence any of B's input properties. In that case,
neither B nor D would need to be deleted before A could be deleted.
In order to make the above algorithm a reality, the resource monitor
interface has been updated to include a map that associates an input
property key with the list of resources that input property depends on.
Older clients of the resource monitor will leave this map empty, in
which case all input properties will be treated as depending on all
dependencies of the resource. This is probably overly conservative, but
it is less conservative than what we currently implement, and is
certainly correct.
* Use both a in-proc and out-of-proc pipenv lock
Turns out that flock alone is not sufficient to guarantee exclusive
access to a resource within a single process. To remedy this, a few
FileMutex type wraps both an in-proc mutex and an out-of-proc
file-backed mutex to achieve the goal of exclusive access to a resource
in both in-proc and out-of-proc scenarios.
This commit also uses this lock globally in the integration test
framework in order to globally serialize invocations of pipenv install.
* Remove merge markers
* Use a file lock for serializing pipenv installs
A in-process mutex is not sufficient for serializing pipenv installs
because the 1) go test runner occasionally will split test executions
into multiple processes and 2) each test gets an instance of a
programTester and we'd need to share the mutex globally if we wanted to
successfully serialize access to the pipenv install command.
* Please linter
setuptools's "develop" action is not safe to run concurrently when
targeting the same source tree. In order to work around this, this
commit explicitly serializes package installations.
When the install fails, we end up printing the entire contents of
yarn's stdout to stdout. This output can often be quite long and will
cause Travis to fail in some cases.
The regular error output should be sufficent for us to diagnose any
issues we'll face.
Add a new property to ProgramTestOptions, `Overrides` that allows a
test to request a different version of a package is used instead of
what would be listed in the package.json file.
This will be used by our nightly automation to run everything "at head"
While the lifecycle tests wrote a `.yarnrc` file to ensure that copies
of `yarn` did not race with one another, the more barebones testing
framework did not.
This should address some of the yarn issues we've been seeing in CI
recently
The lockfile is not super interesting when running the lifecycle
tests, since the test project is not a long lived artifact. There was
no real harm in writing it, exepct that in some cases `pipenv` gets
very confused when you have dependencies on pre-release
versions (which is the case right now as we are brining up Python 3).
The packages are still installed correctly, even when we don't write a
lock file.
* Fix Python support in integration test framework
Update the integratino test framework to use pipenv to bootstrap new
virtual environments for tests and use those virtual environments to run
pulumi update and pulumi preview.
Fixespulumi/pulumi#2138
* Install packages via 'Dependencies' field
* Remove code for installing packages from Dependencies
In preparation for some workspace restructuring, I decided to scratch a
few itches of my own in the code:
* Change project's RuntimeInfo field to just Runtime, to match the
serialized name in JSON/YAML.
* Eliminate the no-longer-used Context and NoDefaultIgnores fields on
project, and all of the associated legacy PPC-related code.
* Eliminate the no-longer-used IgnoreFile constant.
* Remove a bunch of "// nolint: lll" annotations, and simply format
the structures with comments on dedicated lines, to avoid overly
lengthy lines and lint suppressions.
* Mark Dependencies and InitErrors as `omitempty` in the JSON
serialization directives for CheckpointV2 files. This was done for
the YAML directives, but (presumably accidentally) omitted for JSON.
* Remove TODO for issue since fixed in PPCs.
* Update issue reference to source
* Update comment wording
* Remove --ppc arg of stack init
* Remove PPC references in int. testing fx
* Remove vestigial PPC API types
This change adds GitLab CI support, by sniffing out the right
variables (equivalent to what we already do for Travis).
I've also restructured the code to share more logic with our
existing CI detection code, now moved to the pkg/util/ciutil
package, and will be fleshing this out more in the days to come.
Add two command-line options to the test framework, "-dirs" and
"list-dirs". The former accepts a regex that is used to select which
integration tests to run. The latter lists available integration tests
without running them.
The pattern we are using here is generally prone to error, so I hope we find a way to move away from this more generally - but for now we need to be able to configure more of these in places we are using `With`.
Also remove some `+""` that are tripping up the linter for me locally.
Often when the test fails, you'll want to use the web console to
investigate the stack, looking at detailed log messages, for
example. But since we always delete the stack today, you often can't.
Stop running `pulumi stack rm` when the test fails, however do run a
`pulumi destroy` so we at least clean up any resources the stack may
have allocated during testing.
Fixes#1722
Replace the Source-based implementation of refresh with a phase that
runs as the first part of plan execution and rewrites the snapshot in-memory.
In order to fit neatly within the existing framework for resource operations,
these changes introduce a new kind of step, RefreshStep, to represent
refreshes. RefreshSteps operate similar to ReadSteps but do not imply that
the resource being read is not managed by Pulumi.
In addition to the refresh reimplementation, these changes incorporate those
from #1394 to run refresh in the integration test framework.
Fixes#1598.
Fixespulumi/pulumi-terraform#165.
Contributes to #1449.
We retain a few tests on the RunBuild plan, with `typescript` set to
false in the runtime options, but for the general case, we remove the
build steps and custom entry points for our programs.
This change lets us set runtime specific options in Pulumi.yaml, which
will flow as arguments to the language hosts. We then teach the nodejs
host that when the `typescript` is set to `true` that it should load
ts-node before calling into user code. This allows using typescript
natively without an explicit compile step outside of Pulumi.
This works even when a tsconfig.json file is not present in the
application and should provide a nicer inner loop for folks writing
typescript (I'm pretty sure everyone has run into the "but I fixed
that bug! Why isn't it getting picked up? Oh, I forgot to run tsc"
problem.
Fixes#958
This allows us to delete the one off export/import test, which is nice
because it failed to run when PULUMI_ACCESS_TOKEN was not set in the
environment (due to an interaction between `pulumi login` and the
bare-bones integration test framework)
Because we run our golang integration tests "in tree" (due to
the need to be under $GOPATH and have a vendor folder around), the
"command-output" folder was getting left behind, dirtying the worktree
after building.
This change does two things:
1. On a succecssful run, remove the folder.
2. Ignore the folder via .gitignore (this way if a test fails and you
do `git add .` you don't end up commiting this folder).
Due to the way GOPATH and vendoring works, copying Go tests out to a
random temp directory simply will not work. This is largely a holdover
to the days of when .pulumi/ directories would pollute the directory,
but is also done for languages like Node.js whose preparatory and build
steps also pollute the working directory (with bin/ and node_modules/
directories). Go does not have this problem, so we can safely skip.
Today we defaulted our tests to create stacks in the `pulumi`
organization. We did this because our tests run with `pulumi-bot` and
we'd rather create the stacks in our shared organization, so any
Pulumi developer can see them.
Of course, as we prepare to have folks outside of the Pulumi
organization write and run tests, this has now become a bad default.
Remove the ability to explicitly set an owner in
ProgramTestOptions (since that would more or less only lead to pain
going forward) and default to just creating the stacks in whatever
account is currently logged in. In CI, we'll set a new environment
variable "PULUMI_TEST_OWNER" which controls the owner of the stacks,
which we'll set to `pulumi`.
Impact to day to day developers is during test runs locally you'll see
stacks in your list of stacks. If any of the tests fail to clean up,
you'll see these lingering stacks (but you can go clean them up).
math/rand uses a fixed seed, meaning that across runs the Kth call to
`rand.Int63()` will always return the same value.
Given that we want to provide a unique suffix across multiple
concurrent runs, this isn't the behavior we want.
I saw an instance fail in CI where all three legs ran the test
concurrently and they raced on creating the test stack, since they all
generated the same name.
This lets us set these values globaly, in our Travis and AppVeyor
configurations instead of forcing every test to opt-in. It also means
that by default, local builds will not report any of this data (and
will not need access to these endpoints).
These changes add support for pre- and post-`pulumi` callbacks to the
integration test framework. These callbacks will be invoked immediately
before and after (respectively) running a `pulumi` command.
This changes the CLI interface in a few ways:
* `pulumi preview` is back! The alternative of saying
`pulumi update --preview` just felt awkward, and it's a common
operation to want to perform. Let's just make it work.
* There are two flags consistent across all update commands,
`update`, `refresh`, and `destroy`:
- `--skip-preview` will skip the preview step. Note that this
does *not* skip the prompt to confirm that you'd like to proceed.
Indeed, it will still prompt, with a little warning text about
the fact that the preview has been skipped.
* `--yes` will auto-approve the updates.
This lands us in a simpler and more intuitive spot for common scenarios.
I found the flag --force to be a strange name for skipping a preview,
since that name is usually reserved for operations that might be harmful
and yet you're coercing a tool to do it anyway, knowing there's a chance
you're going to shoot yourself in the foot.
I also found that what I almost always want in the situation where
--force was being used is to actually just run a preview and have the
confirmation auto-accepted. Going straight to --force isn't the right
thing in a CI scenario, where you actually want to run a preview first,
just to ensure there aren't any issues, before doing the update.
In a sense, there are four options here:
1. Run a preview, ask for confirmation, then do an update (the default).
2. Run a preview, auto-accept, and then do an update (the CI scenario).
3. Just run a preview with neither a confirmation nor an update (dry run).
4. Just do an update, without performing a preview beforehand (rare).
This change enables all four workflows in our CLI.
Rather than have an explosion of flags, we have a single flag,
--preview, which can specify the mode that we're operating in. The
following are the values which correlate to the above four modes:
1. "": default (no --preview specified)
2. "auto": auto-accept preview confirmation
3. "only": only run a preview, don't confirm or update
4. "skip": skip the preview altogether
As part of this change, I redid a bit of how the preview modes
were specified. Rather than booleans, which had some illegal
combinations, this change introduces a new enum type. Furthermore,
because the engine is wholly ignorant of these flags -- and only the
backend understands them -- it was confusing to me that
engine.UpdateOptions stored this flag, especially given that all
interesting engine options _also_ accepted a dryRun boolean. As of
this change, the backend.PreviewBehavior controls the preview options.
This change introduces support for using the cloud backend when
`pulumi init` has not been run. When this is the case, we use the new
identity model, where a stack is referenced by an owner and a stack
name only.
There are a few things going on here:
- We add a new `--owner` flag to `pulumi stack init` that lets you
control what account a stack is created in.
- When listing stacks, we show stacks owned by you and any
organizations you are a member of. So, for example, I can do:
* `pulumi stack init my-great-stack`
* `pulumi stack init --owner pulumi my-great-stack`
To create a stack owned by my user and one owned by my
organization. When `pulumi stack ls` is run, you'll see both
stacks (since they are part of the same project).
- When spelling a stack on the CLI, an owner can be optionally
specified by prefixing the stack name with an owner name. For
example `my-great-stack` means the stack `my-great-stack` owned by
the current logged in user, where-as `pulumi/my-great-stack` would
be the stack owned by the `pulumi` organization
- `--all` can be passed to `pulumi stack ls` to see *all* stacks you
have access to, not just stacks tied to the current project.
To prepare for a world where stack names must be unique across an
owner, add some randomness to the names we use for stacks as part of
our integration tests.
This change does three major things:
1. Removes the ability to be logged into multiple clouds at the same
time. Previously, we supported being logged into multiple clouds at
the same time and the CLI would fan out requests and join responses
when needed. In general, this was only useful for Pulumi employees
that wanted run against multiple copies of the service (say production
and staging) but overall was very confusing (for example in the old
world a stack with the same identity could appear twice (since it was
in two backends) which the CLI didn't handle very well).
2. Stops treating the "local" backend as a special thing, from the
point of view of the CLI. Previouly we'd always connect to the local
backend and merge that data with whatever was in clouds we were
connected to. We had gestures like `--local` in `pulumi stack init`
that meant "use the local mode". Instead, to use the local mode now
you run `pulumi login --cloud-url local://` and then you are logged in
the local backend. Since you can only ever be logged into a single
backend, we can remove the `--local` and `--remote` flags from `pulumi
stack init`, it just now requires you to be logged in and creates a
stack in whatever back end you were logged into. When logging into the
local backend, you are not prompted for an access key.
3. Prompt for login in places where you have to log in, if you are not
already logged in.
Tests now target managed stacks instead of local stacks.
The existing logged in user and target backend API are used unless PULUMI_ACCES_TOKEN is defined, in which case tests are run under that access token and against the PULUMI_API backend.
For developer machines, we will now need to be logged in to Pulumi to run tests, and whichever default API backend is logged in (the one listed as current in ~/.pulumi/credentials.json) will be used. If you need to override these, provide PULUMI_ACCESS_TOKEN and possibly PULUMI_API.
For Travis, we currently target the staging service using the Pulumi Bot user.
We have decided to run tests in the pulumi organization. This can be overridden for local testing (or in Travis in the future) by defining PULUMI_API_OWNER_ORGANIZATION and using an access token with access to that organization.
Part of pulumi/home#195.
This change includes a bunch of refactorings I made in prep for
doing refresh (first, the command, see pulumi/pulumi#1081):
* The primary change is to change the way the engine's core update
functionality works with respect to deploy.Source. This is the
way we can plug in new sources of resource information during
planning (and, soon, diffing). The way I intend to model refresh
is by having a new kind of source, deploy.RefreshSource, which
will let us do virtually everything about an update/diff the same
way with refreshes, which avoid otherwise duplicative effort.
This includes changing the planOptions (nee deployOptions) to
take a new SourceFunc callback, which is responsible for creating
a source specific to the kind of plan being requested.
Preview, Update, and Destroy now are primarily differentiated by
the kind of deploy.Source that they return, rather than sprinkling
things like `if Destroying` throughout. This tidies up some logic
and, more importantly, gives us precisely the refresh hook we need.
* Originally, we used the deploy.NullSource for Destroy operations.
This simply returns nothing, which is how Destroy works. For some
reason, we were no longer doing this, and instead had some
`if Destroying` cases sprinkled throughout the deploy.EvalSource.
I think this is a vestige of some old way we did configuration, at
least judging by a comment, which is apparently no longer relevant.
* Move diff and diff-printing logic within the engine into its own
pkg/engine/diff.go file, to prepare for upcoming work.
* I keep noticing benign diffs anytime I regenerate protobufs. I
suspect this is because we're also on different versions. I changed
generate.sh to also dump the version into grpc_version.txt. At
least we can understand where the diffs are coming from, decide
whether to take them (i.e., a newer version), and ensure that as
a team we are monotonically increasing, and not going backwards.
* I also tidied up some tiny things I noticed while in there, like
comments, incorrect types, lint suppressions, and so on.
This takes the existing `apitype.Checkpoint` type and renames it to
`apitype.CheckpointV1` locking in the shape. In addition, we introduce
a `apitype.VersionedCheckpoint` type, which holds a version number and
a json document representing a checkpoint at that version. Now, when
reading a checkpoint, the CLI can determine if it's in a format it
understands, and fail gracefully if it is not.
While the CLI understands the older checkpoint version, it always
writes the newest version format, meaning that if you manage a
fire-and-forget stack with this version of the CLI, it will be
un-readable by previous versions.
Stacks managed by Pulumi.com are not impacted by this change.
Fixes: #887
The change to refactor out where we store configuration data broke our
old strategy, which we discovered when we tried to take this payload
into pulumi-aws.
Despite our good progress moving towards having an apitype package,
where our exchange types live and can be shared among the engine and
our services, there were a few major types that were still duplciated.
Resource was the biggest example -- and indeed, the apitype varirant
was missing the new Dependencies property -- but there were others,
like Manfiest, PluginInfo, etc. These too had semi-random omissions.
This change merges all of these types into the apitype package. This
not only cleans up the redundancy and missing properties, but will
"force the issue" with respect to keeping them in sync and properly
versioning the information in a backwards compatible way.
The resource/stack package still exists as a simple marshaling layer
to and from the engine's core data types.
Finally, I've made the controversial change to share the actual
Deployment data structure at the apitype layer also. This will force
us to confront differences in that data structure similarly, and will
allow us to leverage the strong typing throughout to catch issues.
This change gets enough of the Python SDK up and running that the
empty Python program will work. Mostly just scaffolding, but the
basic structure is now in place. The primary remaining work is to
wire up resource creation to the gRPC interfaces.
In summary:
* The basic structure is as follows:
- Everything goes into sdk/python/.
- sdk/python/cmd/pulumi-langhost-python is a Go language host
that simply knows how to spawn Python processes to run out
entrypoint in response to requests by the engine.
- sdk/python/cmd/pulumi-langhost-python-exec is a little Python
shim that is invoked by the language host to run Python programs,
and is responsible for setting up the minimal goo before we can
do so (RPC connections and the like).
- sdk/python/lib/ contains a Python Pip package suitable for PyPi.
- In there, we have two packages: the root pulumi package that
contains all of the basic Pulumi programming model abstractions,
and pulumi.runtime, which contains the implementation of
resource registration, RPC interfacing with the engine, and so on.
* Add logic in our test framework to conditionalize on the language
type and react accordingly. This will allow us to skip Yarn for
Python projects and eventually run Pip if there's a requirements.txt.
* Created the basic project structure, including all of the usual
Make targets for installing into the proper places.
* Building also runs Pylint and we are clean.
There are a few other minor things in here:
* Add an "empty" test for both Node.js and Python. These pass.
* Fix an existing bug in plugin shutdown logic. At some point, we
started waiting for stderr/stdout to flush before shutting down
the plugin; but if certain failures happen "early" during the
plugin launch process, these channels will never get initialized
and so waiting for them deadlocks.
* Recently we seem to have added logic to delete test temp
directories if a failure happened during initialization of said
temp directories. This is unfortunate, because you often need to
look at the temp directory to see what failed. We already clean
them up elsewhere after the full test completes successfully, so
I don't think we need to be doing this, and I've removed it.
Still many loose ends (config, resources, etc), but it's a start!
Backup copies of local stack checkpoints are now saved to the
user's home directory (`~/.pulumi/backups`) by default.
This enables users to recover after accidentally deleting their
local `.pulumi` directory (e.g. via `git clean`).
The behavior can be disabled by setting the
PULUMI_DISABLE_CHECKPOINT_BACKUPS environment variable, which
we use to disable backups when running all tests other than the
test for this functionality.
This addresses pulumi/pulumi#446: what we used to call "package" is
now called "project". This has gotten more confusing over time, now
that we're doing real package management.
Also fixespulumi/pulumi#426, while in here.
We've seen yarn fail from time to time during our integration
tests (usually timeouts talking to the npm registry) so let's add a
few retries to all yarn commands.
We were pretty careful to keep the test directory around if the test ever
exited early due to a panic or error return. But if the test ran to
completion and failed -- for example, if ExtraRuntimeValidation caused the
test to fail -- we would end up deleting the test directory.
Fixes#868
This PR adds a new `pulumi history` command, which prints the update history for a stack.
The local backend stores the update history in a JSON file on disk, next to the checkpoint file. The cloud backend simply provides the update metadata, and expects to receive all the data from a (NYI) `/history` REST endpoint.
`pkg/backend/updates.go` defines the data that is being persisted. The way the data is wired through the system is adding a new `backend.UpdateMetadata` parameter to a Stack/Backend's `Update` and `Destroy` methods.
I use `tests/integration/stack_outputs/` as the simple app for the related tests, hence the addition to the `.gitignore` and fixing the name in the `Pulumi.yaml`.
Fixes#636.
In travis, we've seen cases where writes to our standard streams
results in an error like: `/dev/stderr: resource temporarily
unavailable` which causes the tests to panic.
Now, in a perfect world, writes to /dev/stderr would not fail in this
way, but we do not live in a perfect world. Other processes on the
machine may make stderr/stdout non-blocking. We've are now seeing this
failure in Travis more often and it is masking real Pulumi failures
we want to debug.
This change restructures the test framework code a bit, to make it
easier to introduce additional languages. Our knowledge of Yarn and
Node.js project structure, for instance, was previously baked in to
the test logic, in a way that was hard to make, for instance, Yarn
optional. (In Python, of course, it will not be used.) To better
support this, I've moved some state onto a new programTester struct
that we can use to lazily find binaries required during the testing
(such as Yarn, Pip, and so on). I'm committing this separately so
that I can minimize merge conflicts in the Python work.
Fix two references to the now-unnamed `dir` that should have been to
other variables.
Check a real condition before the deferred call to RemoveAll instead of
checking the error return.
These changes refactor the engine's entrypoints--Deploy, Destroy, and
Preview--to be update-centric rather than stack-centric. Each of these
methods now takes a value of a new type, Update, that abstracts away the
vagaries of fetching and maintaining the update's state. This
refactoring also reinforces Pulumi.yaml as a CLI concept rather than an
engine concept; the CLI is now the only reader/writer of this format.
These changes will smooth the way for a few refactorings on the service
side that will aid in update isolation.
This PR updates the `pkg/testing/integration` package to support running integration tests against the Pulumi Service if desired. This is done through adding new options to `ProgramTestOptions`. (Generally adding support for providing values to flags that were previously inaccessible.)
I added an integration test to confirm that it all works if the PULUMI_API environment variable is set. These tests aren't run in Travis, only manually. Since we cannot reliably run tests from `master` against the service because of the delay in rolling out updates to the Pulumi SDK, etc.
This fixes a few more edit directory issues, where we didn't
correctly propagate the changes in edit directory required during
subsequent destroy/stack activities. It also fixes a few error
paths so that we preserve the right directory to be removed.
