We are going to be changing the configuration model. To begin, let's
take most of the existing stuff and mark it as "deprecated" so we can
keep the existing behavior (to help transition newer code forward)
while making it clear what APIs should not be called in the
implementation of `pulumi` itself.
Per pulumi/pulumi#984, we will now issue an error if it appears you're
importing a checkpoint from a different stack. This can be overridden
if you know what you're doing (with --force), but in general this is a
sign that you're doing something very wrong that will be hard to undo.
Despite our good progress moving towards having an apitype package,
where our exchange types live and can be shared among the engine and
our services, there were a few major types that were still duplciated.
Resource was the biggest example -- and indeed, the apitype varirant
was missing the new Dependencies property -- but there were others,
like Manfiest, PluginInfo, etc. These too had semi-random omissions.
This change merges all of these types into the apitype package. This
not only cleans up the redundancy and missing properties, but will
"force the issue" with respect to keeping them in sync and properly
versioning the information in a backwards compatible way.
The resource/stack package still exists as a simple marshaling layer
to and from the engine's core data types.
Finally, I've made the controversial change to share the actual
Deployment data structure at the apitype layer also. This will force
us to confront differences in that data structure similarly, and will
allow us to leverage the strong typing throughout to catch issues.
Previously, we would prefer a plugin on the $PATH which is more or
less always the case for people hacking on `pulumi`. Later, when we
went to check the loaded plugin version matched the one we requested,
we fail.
Now, if we have a version, we'll first consult the local plugin
cache. If that fails, we'll fall back to the $PATH as we used to.
When we are loading a plugin without a version, we continue to use the
one on the $PATH (without testing the cache) on the assumption it is
newer.
In addition, we've turned the "plugin versions are mis-matched" from
an error into a warning. We expect that we'll only ever see this
warning when something strange is going on (since in the normal case,
we'll have found the exact version in the cache) but having it not
hard fail does help in development cases.
Fixes#977
If currently logged in, `stack init` creates a managed stack. Otherwise, it creates a local
stack. This avoids the need to specify `--local` when not using the service.
As today, `--local` can be passed, which will create a local stack regardless of being logged
in or not.
A new flag, `--remote`, has been added, which can be passed to indicate a managed stack,
used to force an error if not logged into the service.
* Produce better error messages when the main module is not found
If we fail to load a program's main module, inspect the program's
package.json and attempt to diagnose why the main module load failed.
* Code review feedback: entrypoint -> entry point, call out npm build explicitly, simplify control flow
* Code review feedback: add a little more levity to the unknown exception error message
Pip is called `pip2.7` in Travis' Mac image. Our script already
had to deal with this, but did so by conditionalizing the name we
use in our scripts. Rather than doing that, let's create a symlink
with the name `pip` so that everything can just use the good name.
This change passes --user to pip install, so that it installs packages
underneath the home directory. This is required because, except for the
"python" image in Travis, all Python and Pip-related directories are
root-owned. The --user approach avoids needing to sudo all over the place.
This change includes a lot more functionality. Enough to actually
run the webserver-py example through previews, updates, and destroys!
* Actually wire up the gRPC connections to the engine/monitor.
* Move the Node.js and Python generated Protobuf/gRPC files underneath
the actual SDK directories to simplify this generally. No more
copying during `make` and, in fact, this was required to give a smoother
experience with good packages/modules for the Python's SDK development.
* Build the Python egg during `make build`.
* Add support for program stacks. Just like with the Node.js runtime,
we will auto-parent any resources without explicit parents to a single
top-level resource component.
* Add support for component resource output properties.
* Add get_project() and get_stack() functions for retrieving the current
project and stack names.
* Properly use UNKNOWN sentinels.
* Add a set_outputs() function on Resource. This is defined by the
code-generator and allows custom logic for output property setting.
This is cleaner than the way we do this in Node.js, and gives us a
way to ensure that output properties are "real" properties, complete
with member documentation. This also gives us a hook to perform
name demangling, which the code-generator typically controls anyway.
* Add package dependencies to setuptools.py and requirements.txt.
The spinner code used \b, but didn't overwrite with spaces, so part
of the message could get left behind when other writes to stdout/err
occurred. This change simply overwrites characters with spaces.
This change gets enough of the Python SDK up and running that the
empty Python program will work. Mostly just scaffolding, but the
basic structure is now in place. The primary remaining work is to
wire up resource creation to the gRPC interfaces.
In summary:
* The basic structure is as follows:
- Everything goes into sdk/python/.
- sdk/python/cmd/pulumi-langhost-python is a Go language host
that simply knows how to spawn Python processes to run out
entrypoint in response to requests by the engine.
- sdk/python/cmd/pulumi-langhost-python-exec is a little Python
shim that is invoked by the language host to run Python programs,
and is responsible for setting up the minimal goo before we can
do so (RPC connections and the like).
- sdk/python/lib/ contains a Python Pip package suitable for PyPi.
- In there, we have two packages: the root pulumi package that
contains all of the basic Pulumi programming model abstractions,
and pulumi.runtime, which contains the implementation of
resource registration, RPC interfacing with the engine, and so on.
* Add logic in our test framework to conditionalize on the language
type and react accordingly. This will allow us to skip Yarn for
Python projects and eventually run Pip if there's a requirements.txt.
* Created the basic project structure, including all of the usual
Make targets for installing into the proper places.
* Building also runs Pylint and we are clean.
There are a few other minor things in here:
* Add an "empty" test for both Node.js and Python. These pass.
* Fix an existing bug in plugin shutdown logic. At some point, we
started waiting for stderr/stdout to flush before shutting down
the plugin; but if certain failures happen "early" during the
plugin launch process, these channels will never get initialized
and so waiting for them deadlocks.
* Recently we seem to have added logic to delete test temp
directories if a failure happened during initialization of said
temp directories. This is unfortunate, because you often need to
look at the temp directory to see what failed. We already clean
them up elsewhere after the full test completes successfully, so
I don't think we need to be doing this, and I've removed it.
Still many loose ends (config, resources, etc), but it's a start!
This change adds a basic Python langhost RPC server. It's fairly
barebones and merely acts as a jumping off point for the Pulumi engine
to spawn a Python program. The host is written in Go, in contrast to
implementing the host in Python, and more closely resembles how I
expect the Node.js language host to work once pulumi/pulumi#331 is done.
1. Output different-colored edges for parent-child resource
relationships
2. Allow the changing of edge colors via command-line parameters
3. Allow the skipping of the parent-child graph or the
dependency graph when calculating all edges
This modifies the Graph interface slightly to allow an edge to specify
what color should be used when drawing it.
1. Various idiomatic Go and TypeScript fixes
2. Add an integration test that end-to-end roundtrips dependency
information for a simple Pulumi program
3. Add an additional test assert that tests that dependency information
comes from the language host as expected
Some file systems do not record BithTimes and BirthTime panics in
these cases. We use HasBirthTimes to guard against this and print n/a
when we do not have a BirthTime.
This change makes the engine backwards compatible with older
language host binaries, by simply ignoring GetRequiredPlugins
calls when the RPC server has not yet implemented it. This
is benign, since we will eventually fault plugins in on demand,
although it does mean that commands like `pulumi plugin install`
will become no-ops (which, thankfully, is what we want).
This commit does two things:
1. All dependencies of a resource, both implicit and explicit, are
communicated directly to the engine when registering a resource. The
engine keeps track of these dependencies and ultimately serializes
them out to the checkpoint file upon successful deployment.
2. Once a successful deployment is done, the new `pulumi stack
graph` command reads the checkpoint file and outputs the dependency
information within in the DOT format.
Keeping track of dependency information within the checkpoint file is
desirable for a number of reasons, most notably delete-before-create,
where we want to delete resources before we have created their
replacement when performing an update.
Previously, the checkpoint manifest contained the full path to a plugin
binary, in places of its friendly name. Now that we must move to a model
where we install plugins in the PPC based on the manifest contents, we
actually need to store the name, in addition to the version (which is
already there). We still also capture the path for debugging purposes.
I was reminded of this yesterday with unprintable characters as I
debugged some things on Windows. Inspired by Yarn, this change adds
a new flag --emoji (-e for short) that can be used to control whether
we show ASCII-only characters or not in the console. On Mac, it
defaults to true, and on Windows and Linux, it defaults to false.
This also brings back the retro ASCII-friendly progress spinner
when --emoji is disabled.
Backup copies of local stack checkpoints are now saved to the
user's home directory (`~/.pulumi/backups`) by default.
This enables users to recover after accidentally deleting their
local `.pulumi` directory (e.g. via `git clean`).
The behavior can be disabled by setting the
PULUMI_DISABLE_CHECKPOINT_BACKUPS environment variable, which
we use to disable backups when running all tests other than the
test for this functionality.
This change refactors the way we do ${VERSION} substitution in both
the Node.js SDK's version.js and package.json, so that it can work on
Windows. This is required now that we are actually parsing semvers.
For some tools we used, like `dep` and `gometalinter` we were just
calling `go install` which caused us to pick up whatever was in `HEAD`
at that time. Now, we move to a model where we install fixed versions,
which will change per milestone.
While doing this, I changed the way our .travis.yml file runs
everything to move as much as possible out into scripts and do so in a
way that allows us to share as much common logic across our
repositories.
On Windows, when we launch the language host, it will end up with
a ".exe" file extension at the end of os.Args[0]. This leads us to
produce a garbage filename for the -exec script -- namely,
pulumi-language-nodejs.exe-exec -- which, of course fails. We simply
need to trim off the ".exe" bit before producing the script name.
We have had a long-standing bug in here where we waiting on a
stdout channel that never got populated, when the language plugin
fails to load entirely. This would lead to hung processes. The
fix is simple: only wait for stdout/stderr channels to drain that
have actually been wired up to enjoy the requisite signaling.
This adds support for two things:
* Installing all plugins that a project requires with a single command:
$ pulumi plugin install
* Listing the plugins that this project requires:
$ pulumi plugin ls --project
$ pulumi plugin ls -p
Prior to this change, we had a flat list of files in the
~/.pulumi/plugins directory. This was simple but unfortunately
too naive, since we in fact have multi-file plugins already.
Dumping them in the same directory increases the risk of a
collision. Instead, let's put them in their own directories.
This means, for example, you'll see things like
~/.pulumi/plugins/
resource-aws-v0.11.0-dev-8-g57a0d62/
README.txt
pulumi-resource-aws
Notice that the binary name stays the same -- e.g., in this
case pulumi-resource-aws -- and does not include the version.
This makes it simple to add it to your $PATH in the usual ways
and have it loaded as a preferred location.
This change renames prune to rm, to match what we use for other
similar commands. Someday perhaps we will add a prune that uses
some smarts to prune old plugins, etc.
Also tidy up some minor things about the command. For example,
we now require --all if you want to truly clear the entire plugin
cache. We also print more detail, like the full list of plugins
to be removed, in the confirmation prompt.
The API/REST logic auto-prepended "/api", which we don't want
for the release downloads. This change just alters callsites
to specify the full path (which I prefer being explicit anyway).