This change includes a bunch of refactorings I made in prep for
doing refresh (first, the command, see pulumi/pulumi#1081):
* The primary change is to change the way the engine's core update
functionality works with respect to deploy.Source. This is the
way we can plug in new sources of resource information during
planning (and, soon, diffing). The way I intend to model refresh
is by having a new kind of source, deploy.RefreshSource, which
will let us do virtually everything about an update/diff the same
way with refreshes, which avoid otherwise duplicative effort.
This includes changing the planOptions (nee deployOptions) to
take a new SourceFunc callback, which is responsible for creating
a source specific to the kind of plan being requested.
Preview, Update, and Destroy now are primarily differentiated by
the kind of deploy.Source that they return, rather than sprinkling
things like `if Destroying` throughout. This tidies up some logic
and, more importantly, gives us precisely the refresh hook we need.
* Originally, we used the deploy.NullSource for Destroy operations.
This simply returns nothing, which is how Destroy works. For some
reason, we were no longer doing this, and instead had some
`if Destroying` cases sprinkled throughout the deploy.EvalSource.
I think this is a vestige of some old way we did configuration, at
least judging by a comment, which is apparently no longer relevant.
* Move diff and diff-printing logic within the engine into its own
pkg/engine/diff.go file, to prepare for upcoming work.
* I keep noticing benign diffs anytime I regenerate protobufs. I
suspect this is because we're also on different versions. I changed
generate.sh to also dump the version into grpc_version.txt. At
least we can understand where the diffs are coming from, decide
whether to take them (i.e., a newer version), and ensure that as
a team we are monotonically increasing, and not going backwards.
* I also tidied up some tiny things I noticed while in there, like
comments, incorrect types, lint suppressions, and so on.
The semantic versions we were using for pre-release PyPI packages wasn't
quite right. We had been using local version identifiers, a la +, rather
than proper pre-release tags, which means that version specifications like
`pulumi==0.11.0` will match `0.11.0+dev.1521506136.g7f043fd.dirty`, in
addition to just `0.11.0`. This is clearly not what we want.
This change moves us over to proper alpha and release candidate versions,
as specified in https://www.python.org/dev/peps/pep-0440/#pre-releases.
Namely, any `x.y.z-dev-123456789-abcdef` version will get translated into
an alpha `x.y.za12345679-abcdef`, and any `x.y.z-rc1` will get translated
into a proper release candidate `x.y.zrc1` version number.
Implement Python invoke RPC
This change implements the invoke function for resource provider
RPCs. This is required to support a customer scenario.
There are a few other minor updates:
* Rename pulumi.export to pulumi.output.
* Change register_resource to, like invoke, return the resulting
object/dictionary, instead of the set_outputs function.
* Initialize the monitor/engine RPC connections to None when not
attached to the engine, thus ensuring good error messages.
* Fix Python project/stack metadata
Our previous strategy of just using `git describe --tags --dirty` to
compute a version caused issues. The major one was that since version
sort lexigrapically, git's strategy of having a commit count without
leading zeros lead to cases where 0.11.0-dev-9 was "newer than"
0.11.0-dev-10 which is not what you want at all.
With this change, we compute a version by first seeing if the commit
is tagged, and if so, we use that tag. Otherwise, we take the closest
tag and to it append the unix timestamp of the commit and then append
a git hash.
Because we use the commit timestamp, things will sort correctly again.
Part of pulumi/home#174
We need to support the current version of the nodejs language host
running programs that use older version of @pulumi/pulumi where the
runtime expected config keys to look like
`<package>:config:<name>`. In the language host we actually did the
transformation from the new format to the old one, for compatability
reasons but we then droped the transfomed value on the floor.
The four concerns are:
parsing a v8 function string so we can figure out captured variables.
walkgin the function/object graph producing the graph we will serialize.
rewriting constructors and methods so that 'super' works.
serializing graph to text.
Also, rename/cleanup a bunch of serialization code.
Also, generate better environment names in the serialized closure code. Thsi code should be much easier to make sense of as hte names will better track to the original names in the user code.
Also, dedupe simple non-capturing functions. This helps ensure we don't spit out N copies of __awaiter (one per file it is declared in).
This change uses virtualenv to insulate us from platform differences
in our building of the Python SDK, and to create an isolated Python 2
environment. This includes meaning we don't need to worry about the
specific location and behavior of Pylint. I *think* this will work
no matter whether it's Mac, Ubuntu, ArchLinux, Windows, and so on.
We do install to the --user directory in the install target using
`pip install -e`, however, which enables the machine-wide symlinking
that we need to support various workflows.
This fixespulumi/pulumi#1007.
This special error kind should be used by all Pulumi components as the error type for user input validation errors. Although it can already be referenced via `@pulumi/pulumi/errors`, also explicitly export it directly on `@pulumi/pulumi`.
This API was introduced to aid the refactoring, but it isn't something
we want to support long term. Remove it and for a few places, push
passing config.Key around more, instead of converting to the old type
eagerly.
We now unify new Config("package") and new Config("package:config"),
printing a warning when the new Config("package:config") form is
used and pointing consumers towards just new Config("package")
I've updated our examples to use the newer syntax, but I've added a
test ot the langhost to ensure both forms work.
Fixes#923
This change temporarily disables Pylint. Assuming it is on the path,
and furthermore that the one on the path runs under 2.7, simply won't
work. See pulumi/pulumi#1007 for details; it also tracks reenabling.
This change includes a few things:
1) Prefer python2 and pip2 when on the PATH, over the undecorated
names python and pip. This is the standard convention for package
managers like Pip, etc., to support Python2 and Python3 side-by-side.
2) Fail-fast if neither can be found on the PATH.
3) Check the reported version number for python, pip, and pylint, and
fail-fast if it doesn't report back 2.7, just to safeguard against
undecorated binaries with unsupported versions.
Also, we had not listed wheel as a dependency in the requirements.txt
file. This needs to be there to support building bdist_wheels. Fixed.
Make many fixes to closure serialization
Primary things that i've done as part of this change:
Added support for cyclic objects.
Properly serialize objects that are shared across different function. previously you would get multiple copies, now you properly reference the same copy.
Remove the usages of 'hashes' for functions. Because we track identity of objects, we no longer need them.
Serialize properties of functions (if they have any).
Handle Objects/Functions with different __proto__s than normal. i.e. classes/constructors. but also anything the user may have done themselves to the object.
Handle generator functions.
Handle functions with 'computed' names.
Handle functions with 'symbol' names.
Handle serializing Promises as Promises.
Removed the dual Closure/AsyncClosure tree. One existed solely so we could have a tree without promises (for use in testing maybe?). Because this all exists in a part of our codebase that is entirely async, it's fine to have promises in the tree, and to await them when serializing the Closure to a string.
Handle serializing class-constructors and methods. Including properly handling 'super' calls.
We now publish the Pulumi Python SDK package to our private PyPI
server at the same time we also publish the NPM package. For now,
we use the test Pulumi.com service, and will switch to staging as
soon as it becomes available.
* Produce better error messages when the main module is not found
If we fail to load a program's main module, inspect the program's
package.json and attempt to diagnose why the main module load failed.
* Code review feedback: entrypoint -> entry point, call out npm build explicitly, simplify control flow
* Code review feedback: add a little more levity to the unknown exception error message
This change passes --user to pip install, so that it installs packages
underneath the home directory. This is required because, except for the
"python" image in Travis, all Python and Pip-related directories are
root-owned. The --user approach avoids needing to sudo all over the place.
This change includes a lot more functionality. Enough to actually
run the webserver-py example through previews, updates, and destroys!
* Actually wire up the gRPC connections to the engine/monitor.
* Move the Node.js and Python generated Protobuf/gRPC files underneath
the actual SDK directories to simplify this generally. No more
copying during `make` and, in fact, this was required to give a smoother
experience with good packages/modules for the Python's SDK development.
* Build the Python egg during `make build`.
* Add support for program stacks. Just like with the Node.js runtime,
we will auto-parent any resources without explicit parents to a single
top-level resource component.
* Add support for component resource output properties.
* Add get_project() and get_stack() functions for retrieving the current
project and stack names.
* Properly use UNKNOWN sentinels.
* Add a set_outputs() function on Resource. This is defined by the
code-generator and allows custom logic for output property setting.
This is cleaner than the way we do this in Node.js, and gives us a
way to ensure that output properties are "real" properties, complete
with member documentation. This also gives us a hook to perform
name demangling, which the code-generator typically controls anyway.
* Add package dependencies to setuptools.py and requirements.txt.
This change gets enough of the Python SDK up and running that the
empty Python program will work. Mostly just scaffolding, but the
basic structure is now in place. The primary remaining work is to
wire up resource creation to the gRPC interfaces.
In summary:
* The basic structure is as follows:
- Everything goes into sdk/python/.
- sdk/python/cmd/pulumi-langhost-python is a Go language host
that simply knows how to spawn Python processes to run out
entrypoint in response to requests by the engine.
- sdk/python/cmd/pulumi-langhost-python-exec is a little Python
shim that is invoked by the language host to run Python programs,
and is responsible for setting up the minimal goo before we can
do so (RPC connections and the like).
- sdk/python/lib/ contains a Python Pip package suitable for PyPi.
- In there, we have two packages: the root pulumi package that
contains all of the basic Pulumi programming model abstractions,
and pulumi.runtime, which contains the implementation of
resource registration, RPC interfacing with the engine, and so on.
* Add logic in our test framework to conditionalize on the language
type and react accordingly. This will allow us to skip Yarn for
Python projects and eventually run Pip if there's a requirements.txt.
* Created the basic project structure, including all of the usual
Make targets for installing into the proper places.
* Building also runs Pylint and we are clean.
There are a few other minor things in here:
* Add an "empty" test for both Node.js and Python. These pass.
* Fix an existing bug in plugin shutdown logic. At some point, we
started waiting for stderr/stdout to flush before shutting down
the plugin; but if certain failures happen "early" during the
plugin launch process, these channels will never get initialized
and so waiting for them deadlocks.
* Recently we seem to have added logic to delete test temp
directories if a failure happened during initialization of said
temp directories. This is unfortunate, because you often need to
look at the temp directory to see what failed. We already clean
them up elsewhere after the full test completes successfully, so
I don't think we need to be doing this, and I've removed it.
Still many loose ends (config, resources, etc), but it's a start!
This change adds a basic Python langhost RPC server. It's fairly
barebones and merely acts as a jumping off point for the Pulumi engine
to spawn a Python program. The host is written in Go, in contrast to
implementing the host in Python, and more closely resembles how I
expect the Node.js language host to work once pulumi/pulumi#331 is done.
1. Various idiomatic Go and TypeScript fixes
2. Add an integration test that end-to-end roundtrips dependency
information for a simple Pulumi program
3. Add an additional test assert that tests that dependency information
comes from the language host as expected
This commit does two things:
1. All dependencies of a resource, both implicit and explicit, are
communicated directly to the engine when registering a resource. The
engine keeps track of these dependencies and ultimately serializes
them out to the checkpoint file upon successful deployment.
2. Once a successful deployment is done, the new `pulumi stack
graph` command reads the checkpoint file and outputs the dependency
information within in the DOT format.
Keeping track of dependency information within the checkpoint file is
desirable for a number of reasons, most notably delete-before-create,
where we want to delete resources before we have created their
replacement when performing an update.
This change refactors the way we do ${VERSION} substitution in both
the Node.js SDK's version.js and package.json, so that it can work on
Windows. This is required now that we are actually parsing semvers.
On Windows, when we launch the language host, it will end up with
a ".exe" file extension at the end of os.Args[0]. This leads us to
produce a garbage filename for the -exec script -- namely,
pulumi-language-nodejs.exe-exec -- which, of course fails. We simply
need to trim off the ".exe" bit before producing the script name.
We have had a long-standing bug in here where we waiting on a
stdout channel that never got populated, when the language plugin
fails to load entirely. This would lead to hung processes. The
fix is simple: only wait for stdout/stderr channels to drain that
have actually been wired up to enjoy the requisite signaling.
This adds support for two things:
* Installing all plugins that a project requires with a single command:
$ pulumi plugin install
* Listing the plugins that this project requires:
$ pulumi plugin ls --project
$ pulumi plugin ls -p
This brings back the Node.js language plugin's GetRequiredPlugins
function, reimplemented in Go now that the language host has been
rewritten from JavaScript. Fairly rote translation, along with
some random fixes required to get tests passing again.
This change implements the Node.js language host's GetRequiredPlugins
function. This merely scans all node_modules/*/package.json files in
the program directory, looking for those that have associated plugins.
It returns a list of any found along with their version numbers.
This change adds a GetRequiredPlugins RPC method to the language
host, enabling us to query it for its list of plugin requirements.
This is language-specific because it requires looking at the set
of dependencies (e.g., package.json files).
It also adds a call up front during any update/preview operation
to compute the set of plugins and require that they are present.
These plugins are populated in the cache and will be used for all
subsequent plugin-related operations during the engine's activity.
We now cache the language plugins, so that we may load them
eagerly too, which we never did previously due to the fact that
we needed to pass the monitor address at load time. This was a
bit bizarre anyhow, since it's really the Run RPC function that
needs this information. So, to enable caching and eager loading
-- which we need in order to invoke GetRequiredPlugins -- the
"phone home" monitor RPC address is passed at Run time.
In a subsequent change, we will switch to faulting in the plugins
that are missing -- rather than erroring -- in addition to
supporting the `pulumi plugin install` CLI command.
This change introduces a workspace.GetPluginPath function that probes
the central workspace cache of plugins for a matching plugin binary that
matches the desired kind, name, and, optionally, version. It also permits
overriding this with $PATH for developer scenarios.
The analyzer, language, and resource plugin logic now uses this function
for deciding which binary path to load at runtime.
Fixes#356. Instead of downloading a node binary with our closure
serialization code linked-in, this PR instead publishes the
`nativeruntime.node` produced by the NodeJS SDK build as part of the SDK.
This has a number of advantages. First, it is vastly more easy to
develop closure.cc in this configuration. Second, we have the ability
to ship different `nativeruntime.node`s side-by-side, paving the way
for enabling future versions of Node. Third, we don't have to stay
in the business of shipping custom builds of Node, although we do still
need to ship a version of Node with minor modifications in order for
Windows to still work.
The windows build was still on the old plan from way back when where
we had binaries littered in the build tree and you had to add parts of
your build-tree to the `%PATH%` for the integration tests to work.
This cleans that up and moves all of our scripts that invoke
javascript to be on the same plan. They invoke our specially named
node with a relative path to the JS code we want to run.
We no longer have a node_modules folder in the SDK (since all
packages now come from NPM) so we need to adjust the shell script we
use to launch our runner to use a relative path.
