This helper method is only really used for testing, but we should not
allow it to create a Key who's namespace has a colon (as ParseKey
would not build something like this).
This API was introduced to aid the refactoring, but it isn't something
we want to support long term. Remove it and for a few places, push
passing config.Key around more, instead of converting to the old type
eagerly.
When serializing config.Key's we now write them as <package>:<name>
instead of <package>:config:<name>. We continue to support reading the
older format for compatability with older files.
config.Key has become a pair of namespace and name. Because the whole
world has not changed yet, there continues to be a way to convert
between a tokens.ModuleMember and config.Key, however now sometime the
conversion from tokens.ModuleMember can fail (when the module member
is not of the form `<package>:config:<name>`).
I'll be changing the structure of the representation of config.Key, so
let's write some tests first to ensure we can continue to treat
everything as JSON and YAML.
Right now, config.Key is a type alias for tokens.ModuleMember. I did a
pass over the codebase such that we use config.Key everywhere it
looked like the value did not leak to some external process (e.g a
resource provider or a langhost).
Doing this makes it a little clearer (hopefully) where code is
depending on a module member structure (e.g. <package>:config:<value>)
instead of just an opaque type.
The change to refactor out where we store configuration data broke our
old strategy, which we discovered when we tried to take this payload
into pulumi-aws.
As it stands, we only configure those providers for which configuration
is present. This can lead to surprising failure modes if those providers
are then used to create resources. These changes ensure that all
resource providers that are not configured during plan initialization
are configured upon first load.
Fixes#758.
By using untyped deployment structures via `json.RawMessage`, we can
support round-tripping between old CLI clients and newer servers, without
dropping possibly-important information on the floor. I hadn't realized
this design goal with the original system, and after talking to @pgavlin,
I better realized the intent and that we want to preserve this.
The filenames we used to store history data locally only had second
level precision. On my machine, the test history test is able to run
multiple `pulumi update` commands in the same second, which causes a
newer history file to overwrite an older one.
This change moves to using a nanosecond precision timestamp when
writing config. In addition, the CLI was trying to sort the updates
that came back from the backend (instead of just trusting them to be
in newest first order, as we documented) so I removed that code as
well.
Make many fixes to closure serialization
Primary things that i've done as part of this change:
Added support for cyclic objects.
Properly serialize objects that are shared across different function. previously you would get multiple copies, now you properly reference the same copy.
Remove the usages of 'hashes' for functions. Because we track identity of objects, we no longer need them.
Serialize properties of functions (if they have any).
Handle Objects/Functions with different __proto__s than normal. i.e. classes/constructors. but also anything the user may have done themselves to the object.
Handle generator functions.
Handle functions with 'computed' names.
Handle functions with 'symbol' names.
Handle serializing Promises as Promises.
Removed the dual Closure/AsyncClosure tree. One existed solely so we could have a tree without promises (for use in testing maybe?). Because this all exists in a part of our codebase that is entirely async, it's fine to have promises in the tree, and to await them when serializing the Closure to a string.
Handle serializing class-constructors and methods. Including properly handling 'super' calls.
A hold-over from a previous experiment (LumiIDL) which we don't use
anymore. If we decide to bring that back, we can easily restore these
types, but for now, let's just remove this dead code.
Most of the errors in this package are holdovers from our previous
syetem where we had our own custom compiler and evaluator and are no
longer needed. The few we still use during plan applicaton (via the
diagnostics system, which is another component from the old system
that we still use) have been promoted into the diag package. Doing so,
allows us to not have to import "github.com/pkg/errors" as "goerr" in
some parts of the engine, a nice cleaup.
Migrate configuration from the old model to the new model. The
strategy here is that when we first run `pulumi` we enumerate all of
the stacks from all of the backends we know about and for each stack
get the configuration values from the project and workspace and
promote them into the new file. As we do this, we remove stack
specific config from the workspace and Pulumi.yaml file.
If we are able to upgrade all the stacks we know about, we delete all
global configuration data in the workspace and in Pulumi.yaml as well.
We have a test that ensures upgrades continue to work.
This change updates our configuration model to make it simpler to
understand by removing some features and changing how things are
persisted in files.
Notable changes:
- We've removed the notion of "workspace" vs "project"
config. Now, configuration is always stored in a file next to
`Pulumi.yaml` named `Pulumi.<stack-name>.yaml` (the same file we'd
use for an other stack specific information we would need to persist
in the future).
- We've removed the notion of project wide configuration. Every new
stack gets a completely empty set of configuration and there's no
way to share common values across stacks, instead the common value
has to be set on each stack.
We retain some of the old code for the configuration system so we can
support upgrading a project in place. That will happen with the next
change.
This change fixes some issues and allows us to close some
others (since they are no longer possible).
Fixes#866Closes#872Closes#731
We are going to be changing the configuration model. To begin, let's
take most of the existing stuff and mark it as "deprecated" so we can
keep the existing behavior (to help transition newer code forward)
while making it clear what APIs should not be called in the
implementation of `pulumi` itself.
Despite our good progress moving towards having an apitype package,
where our exchange types live and can be shared among the engine and
our services, there were a few major types that were still duplciated.
Resource was the biggest example -- and indeed, the apitype varirant
was missing the new Dependencies property -- but there were others,
like Manfiest, PluginInfo, etc. These too had semi-random omissions.
This change merges all of these types into the apitype package. This
not only cleans up the redundancy and missing properties, but will
"force the issue" with respect to keeping them in sync and properly
versioning the information in a backwards compatible way.
The resource/stack package still exists as a simple marshaling layer
to and from the engine's core data types.
Finally, I've made the controversial change to share the actual
Deployment data structure at the apitype layer also. This will force
us to confront differences in that data structure similarly, and will
allow us to leverage the strong typing throughout to catch issues.
Previously, we would prefer a plugin on the $PATH which is more or
less always the case for people hacking on `pulumi`. Later, when we
went to check the loaded plugin version matched the one we requested,
we fail.
Now, if we have a version, we'll first consult the local plugin
cache. If that fails, we'll fall back to the $PATH as we used to.
When we are loading a plugin without a version, we continue to use the
one on the $PATH (without testing the cache) on the assumption it is
newer.
In addition, we've turned the "plugin versions are mis-matched" from
an error into a warning. We expect that we'll only ever see this
warning when something strange is going on (since in the normal case,
we'll have found the exact version in the cache) but having it not
hard fail does help in development cases.
Fixes#977
The spinner code used \b, but didn't overwrite with spaces, so part
of the message could get left behind when other writes to stdout/err
occurred. This change simply overwrites characters with spaces.
This change gets enough of the Python SDK up and running that the
empty Python program will work. Mostly just scaffolding, but the
basic structure is now in place. The primary remaining work is to
wire up resource creation to the gRPC interfaces.
In summary:
* The basic structure is as follows:
- Everything goes into sdk/python/.
- sdk/python/cmd/pulumi-langhost-python is a Go language host
that simply knows how to spawn Python processes to run out
entrypoint in response to requests by the engine.
- sdk/python/cmd/pulumi-langhost-python-exec is a little Python
shim that is invoked by the language host to run Python programs,
and is responsible for setting up the minimal goo before we can
do so (RPC connections and the like).
- sdk/python/lib/ contains a Python Pip package suitable for PyPi.
- In there, we have two packages: the root pulumi package that
contains all of the basic Pulumi programming model abstractions,
and pulumi.runtime, which contains the implementation of
resource registration, RPC interfacing with the engine, and so on.
* Add logic in our test framework to conditionalize on the language
type and react accordingly. This will allow us to skip Yarn for
Python projects and eventually run Pip if there's a requirements.txt.
* Created the basic project structure, including all of the usual
Make targets for installing into the proper places.
* Building also runs Pylint and we are clean.
There are a few other minor things in here:
* Add an "empty" test for both Node.js and Python. These pass.
* Fix an existing bug in plugin shutdown logic. At some point, we
started waiting for stderr/stdout to flush before shutting down
the plugin; but if certain failures happen "early" during the
plugin launch process, these channels will never get initialized
and so waiting for them deadlocks.
* Recently we seem to have added logic to delete test temp
directories if a failure happened during initialization of said
temp directories. This is unfortunate, because you often need to
look at the temp directory to see what failed. We already clean
them up elsewhere after the full test completes successfully, so
I don't think we need to be doing this, and I've removed it.
Still many loose ends (config, resources, etc), but it's a start!
This change adds a basic Python langhost RPC server. It's fairly
barebones and merely acts as a jumping off point for the Pulumi engine
to spawn a Python program. The host is written in Go, in contrast to
implementing the host in Python, and more closely resembles how I
expect the Node.js language host to work once pulumi/pulumi#331 is done.
1. Output different-colored edges for parent-child resource
relationships
2. Allow the changing of edge colors via command-line parameters
3. Allow the skipping of the parent-child graph or the
dependency graph when calculating all edges
This modifies the Graph interface slightly to allow an edge to specify
what color should be used when drawing it.
1. Various idiomatic Go and TypeScript fixes
2. Add an integration test that end-to-end roundtrips dependency
information for a simple Pulumi program
3. Add an additional test assert that tests that dependency information
comes from the language host as expected
Some file systems do not record BithTimes and BirthTime panics in
these cases. We use HasBirthTimes to guard against this and print n/a
when we do not have a BirthTime.
This change makes the engine backwards compatible with older
language host binaries, by simply ignoring GetRequiredPlugins
calls when the RPC server has not yet implemented it. This
is benign, since we will eventually fault plugins in on demand,
although it does mean that commands like `pulumi plugin install`
will become no-ops (which, thankfully, is what we want).
This commit does two things:
1. All dependencies of a resource, both implicit and explicit, are
communicated directly to the engine when registering a resource. The
engine keeps track of these dependencies and ultimately serializes
them out to the checkpoint file upon successful deployment.
2. Once a successful deployment is done, the new `pulumi stack
graph` command reads the checkpoint file and outputs the dependency
information within in the DOT format.
Keeping track of dependency information within the checkpoint file is
desirable for a number of reasons, most notably delete-before-create,
where we want to delete resources before we have created their
replacement when performing an update.
Previously, the checkpoint manifest contained the full path to a plugin
binary, in places of its friendly name. Now that we must move to a model
where we install plugins in the PPC based on the manifest contents, we
actually need to store the name, in addition to the version (which is
already there). We still also capture the path for debugging purposes.
I was reminded of this yesterday with unprintable characters as I
debugged some things on Windows. Inspired by Yarn, this change adds
a new flag --emoji (-e for short) that can be used to control whether
we show ASCII-only characters or not in the console. On Mac, it
defaults to true, and on Windows and Linux, it defaults to false.
This also brings back the retro ASCII-friendly progress spinner
when --emoji is disabled.
Backup copies of local stack checkpoints are now saved to the
user's home directory (`~/.pulumi/backups`) by default.
This enables users to recover after accidentally deleting their
local `.pulumi` directory (e.g. via `git clean`).
The behavior can be disabled by setting the
PULUMI_DISABLE_CHECKPOINT_BACKUPS environment variable, which
we use to disable backups when running all tests other than the
test for this functionality.
We have had a long-standing bug in here where we waiting on a
stdout channel that never got populated, when the language plugin
fails to load entirely. This would lead to hung processes. The
fix is simple: only wait for stdout/stderr channels to drain that
have actually been wired up to enjoy the requisite signaling.
This adds support for two things:
* Installing all plugins that a project requires with a single command:
$ pulumi plugin install
* Listing the plugins that this project requires:
$ pulumi plugin ls --project
$ pulumi plugin ls -p
Prior to this change, we had a flat list of files in the
~/.pulumi/plugins directory. This was simple but unfortunately
too naive, since we in fact have multi-file plugins already.
Dumping them in the same directory increases the risk of a
collision. Instead, let's put them in their own directories.
This means, for example, you'll see things like
~/.pulumi/plugins/
resource-aws-v0.11.0-dev-8-g57a0d62/
README.txt
pulumi-resource-aws
Notice that the binary name stays the same -- e.g., in this
case pulumi-resource-aws -- and does not include the version.
This makes it simple to add it to your $PATH in the usual ways
and have it loaded as a preferred location.
The API/REST logic auto-prepended "/api", which we don't want
for the release downloads. This change just alters callsites
to specify the full path (which I prefer being explicit anyway).
