Running `pulumi query` over state resources will require the ability to
filter resources by type.
This commit begins the process of making this possible simply, using the
TypeScript user-defined type guards feature. This commit changes the
`CustomResource` constructor to record the `t` argument in the
`__pulumiType` field, which acts as a "sentinel" value in the resource
base, which deriving classes need only check to ensure they are of that
type. For example:
aws.s3.Bucket.isInstance(someOb)
would check the `aws.s3.Bucket.__pulumiType` field, and make sure that
the type field lines up.
Adds a new resource option `aliases` which can be used to rename a resource. When making a breaking change to the name or type of a resource or component, the old name can be added to the list of `aliases` for a resource to ensure that existing resources will be migrated to the new name instead of being deleted and replaced with the new named resource.
There are two key places this change is implemented.
The first is the step generator in the engine. When computing whether there is an old version of a registered resource, we now take into account the aliases specified on the registered resource. That is, we first look up the resource by its new URN in the old state, and then by any aliases provided (in order). This can allow the resource to be matched as a (potential) update to an existing resource with a different URN.
The second is the core `Resource` constructor in the JavaScript (and soon Python) SDKs. This change ensures that when a parent resource is aliased, that all children implicitly inherit corresponding aliases. It is similar to how many other resource options are "inherited" implicitly from the parent.
Four specific scenarios are explicitly tested as part of this PR:
1. Renaming a resource
2. Adopting a resource into a component (as the owner of both component and consumption codebases)
3. Renaming a component instance (as the owner of the consumption codebase without changes to the component)
4. Changing the type of a component (as the owner of the component codebase without changes to the consumption codebase)
4. Combining (1) and (3) to make both changes to a resource at the same time
This commit implements read_resource functionality for Python in a
manner identical to the NodeJS implementation. If an "id" option is
passed to a resource via ResourceOptions on construction, that resource
will be read and not created.
Since we don't support nesting secrets (as they are modeled as
Outputs), as we deserialize, we push the secretness up to top level,
where we will correctly use it to mark the output as secret.
This fixes an issue where if you created a StackReference resource,
with a mix of secret and non secret properties, you would see the
"wire form" of the secrets as values on the `outputs` map of the
StackReference resource.
In our system, we model secrets as outputs with an additional bit of
metadata that says they are secret. For Read and Register resource
calls, our RPC interface says if the client side of the interface can
handle secrets being returned (i.e. the language SDK knows how to
sniff for the special signiture and resolve the output with the
special bit set).
For Invoke, we have no such model. Instead, we return a `Promise<T>`
where T's shape has just regular property fields. There's no place
for us to tack the secretness onto, since there are no Outputs.
So, for now, don't even return secret values back across the invoke
channel. We can still take them as arguments (which is good) but we
can't even return secrets as part of invoke calls. This is not ideal,
but given the way we model these sources, there's no way around
this. Fortunately, the result of these invoke calls are not stored in
the checkpoint and since the type is not Output<T> it will be clear
that the underlying value is just present in plaintext. A user that
wants to pass the result of an invoke into a resource can turn an
existing property into a secret via `pulumi.secret`.
We changed the `pulumi update` command to be `pulumi up` a while back
(`update` is an alias of `up`). This change just makes it so we refer to
the actual command, `pulumi up`, instead of the older `pulumi update`.
`pulumi query` requires that language plugins know about "query mode" so
that they don't do things like try to register the default stack
resource.
To communicate that a language host should boot into query mode, we
augment the language plugin protocol to include this information.
* NodeJS: allow callers to override provider version
* Python: allow callers to override provider version
* NodeJS: add version for invoke
* Python: add version to invoke
* NodeJS: add tests for ReadResource
* Post-merge cleanup
* update doc comments
Fixes#2277.
Adds a new ignoreChanges resource option that allows specifying a list of property names whose values will be ignored during updates. The property values will be used for Create, but will be ignored for purposes of updates, and as a result also cannot trigger replacements.
This is a feature of the Pulumi engine, not of the resource providers, so no new logic is needed in providers to support this feature. Instead, the engine simply replaces the values of input properties in the goal state with old inputs for properties marked as ignoreChanges.
Currently, only top level properties may be specified in ignoreChanges. In the future, this could be extended to support paths to nested properties (including into array elements) with a JSONPath/JMESPath syntax.
* Enable unit testing for Pulumi programs
This change enables rudimentary unit testing of your Pulumi programs, by introducing a `PULUMI_TEST_MODE` envvar that, when set, allows programs to run without a CLI. That includes
* Just being able to import your Pulumi modules, and test ordinary functions -- which otherwise would have often accidentally triggered the "Not Running in a CLI" error message
* Being able to verify a subset of resource properties and shapes, with the caveat that outputs are not included, due to the fact that this is a perpetual "dry run" without any engine operations occurring
In principle, this also means you can attach a debugger and step through your code.
* Finish the unit testing features
This change
1) Incorporates CR feedback, namely requiring that test mode be
explicitly enabled for any of this to work.
2) Implements Python support for the same capabilities.
3) Includes tests for both JavaScript and Python SDKs.
* Add a note on unit testing to the CHANGELOG
* Use Node 8 friendly assert API
* Embellish the CHANGELOG entry a bit
In pursuit of pulumi/pulumi#2389, this commit adds the necessary changes
to the resource monitor protocol so that language hosts can communicate
exactly what version of a provider should be used when servicing an
Invoke, ReadResource, or RegisterResource. The expectation here is that,
if a language host provides a version, the engine MUST use EXACTLY that
version of a provider plugin in order to service the request.
This makes the Python SDK's Config constructor name argument
optional, similar to what we do in the Node.js SDK. If not supplied,
the current project name is used as the default.
These changes take advantage of the newly-added support for returning
inputs from Read to update a resource's inputs as part of a refresh.
As a consequence, the Pulumi engine will now properly detect drift
between the actual state of a resource and the desired state described
by the program and generate appropriate update or replace steps.
As part of these changes, a resource's old inputs are now passed to the
provider when performing a refresh. The provider can take advantage of
this to maintain the accuracy of any additional data or metadata in the
resource's inputs that may need to be updated during the refresh.
This is required for the complete implementation of
https://github.com/pulumi/pulumi-terraform/pull/349. Without access to
the old inputs for a resource, TF-based providers would lose all
information about default population during a refresh.
These changes add two new methods to the provider interface and extend
the results of three others.
The new methods are `CheckConfig` and `DiffConfig`, which fill out the
set of methods required for a complete implementation of the
first-class provider design. Though these methods are optional for
backwards compatibility, they should be implemented by all future
providers for the best possible user experience.
The adjusted result types are `DiffResponse`, `ReadResponse`, and
`ErrorResourceInitFailed`. The first has been updated to include a list
of the properties that changed (if any). The latter two now include
an estimated set of inputs for the resource as well as the resource's
state. Together, these three changes enable the engine to determine the
set of inputs that should be specified by a user in order to match those
that describe the resource's current state.
This contributes to #2453, #1662, #1635, and #1718.
* Fix an issue with empty ID for CustomResource
The Python runtime was checking the ID field it receives from the engine
against None, assuming that the engine would not set the ID field if one
was not present. However, it does set the ID field; it is set to the
empty string when an ID is not known.
This commit fixes an issue that can cause certain IDs to be erroneously
considered to be known during previews, which can cause problems during
the Check phase of resources that directly reference IDs of other
resources.
* Add CHANGELOG
We do all our work in a virtualenv and across repositories we now
consume dependencies via pypi (also installed in a virtualenv) so best
case install into the user site packages wastes time and maybe prints
some warnings and worse case it clobbers something they've installed
for real.
These changes add a new flag to the various `ResourceOptions` types that
indicates that a resource should be deleted before it is replaced, even
if the provider does not require this behavior. The usual
delete-before-replace cascade semantics apply.
Fixes#1620.
* Clean up documentation for Python SDK
The Python SDK currently does a couple of bad things that make it
difficult to generate documentation:
1. It "wildcard-imports" submodules without each module specifying an
__all__ member
2. Documentation strings don't have a consistent format
3. Documentation strings are in Markdown and not reStructuredText
To remedy this, this commit addresses 1 by explicitly specifying which
members are being exported from submodules, so that we can see in one
place exactly what the public surface area of the pulumi package is. For
2 and 3, this commit fixes a large number of documentation strings to
contain metadata tags that Sphinx is capable of reading. This allows us
to generate high-quality documentation directly from the source without
having to manually parse docstrings.
* Please pylint
- Add support for per-property dependencies to the Go SDK
- Add tests for first-class secret rejection in the checkpoint and RPC
layers and language SDKs
This implements the new algorithm for deciding which resources must be
deleted due to a delete-before-replace operation.
We need to compute the set of resources that may be replaced by a
change to the resource under consideration. We do this by taking the
complete set of transitive dependents on the resource under
consideration and removing any resources that would not be replaced by
changes to their dependencies. We determine whether or not a resource
may be replaced by substituting unknowns for input properties that may
change due to deletion of the resources their value depends on and
calling the resource provider's Diff method.
This is perhaps clearer when described by example. Consider the
following dependency graph:
A
__|__
B C
| _|_
D E F
In this graph, all of B, C, D, E, and F transitively depend on A. It may
be the case, however, that changes to the specific properties of any of
those resources R that would occur if a resource on the path to A were
deleted and recreated may not cause R to be replaced. For example, the
edge from B to A may be a simple dependsOn edge such that a change to
B does not actually influence any of B's input properties. In that case,
neither B nor D would need to be deleted before A could be deleted.
In order to make the above algorithm a reality, the resource monitor
interface has been updated to include a map that associates an input
property key with the list of resources that input property depends on.
Older clients of the resource monitor will leave this map empty, in
which case all input properties will be treated as depending on all
dependencies of the resource. This is probably overly conservative, but
it is less conservative than what we currently implement, and is
certainly correct.
On startup, when we were populating the Settings object, we failed to
coerce the dry_run parameter from a string to a boolean, which resulted
in is_dry_run always believing that it was a preview. This PR fixes this
oversight by explicitly coercing to a boolean prior to sending the value
to Settings.
When is resource is waiting for its dependencies to resolve, it first
informs the RPC_MANAGER that it intends to do an RPC - this is to
prevent premature termination of the program while RPCs are still in
flight or queued to execute.
However, this is a problem whenever a resource fails to create for
whatever reason. The most common ways for this to happen are for invokes
to fail, resource creation itself to fail, or throwing in an apply.
Today, this causes a deadlock, since all consumers of the failed
resources block forever while never decrementing the RPC count.
This commit addresses the issue by deliberately (abnormally) resolving
all futures that are created in the process of preparing a resource.
This solves the problem by immediately terminating all resources that
are waiting for the failed resource's outputs to resolve - they resolve
immediately, and exceptionally.
The end result is now that, instead of deadlocking, a doomed program now
terminates as expected with a single exception message.
When previewing a real first-class provider, it is often the case that
the provider's ID is unknown during a preview. This commit fixes a bug
where we did not translate an unknown ID into the rpc layer's sentinel
UNKNOWN value where we should have, which caused the engine to fail to
resolve the provider reference.
For some reason, writing to stderr using sys.stderr.write doesn't work,
but printing to stderr using the print function does. I don't know why
this is, but I do know that this makes the printed messages show up as
messages on the Pulumi CLI.
Invoke in Node.js allows users to optionally pass a parent or a provider
to the invoke, which dictates either explicitly or implicitly which
provider to use when performing an invoke. If a provider is specified
explicitly, that provider is used to perform the invoke. If a parent is
specified, that parent's provider is used to perform the invoke.
* Implement first-class providers for Python
First-class providers are an explicit projection of providers themselves
into Pulumi programs. For the most post, providers are just regular
resources, but the addition of providers to the fray (and the ability of
resources to be constructed by providers in the same program) requires
some changes to the Python resource model.
A summary of the changes:
1. Added ProviderResource, a custom resource that is the base class of
all providers.
2. ResourceOptions now has 'provider' and 'providers' fields.
'provider', when passed to a custom resource, allows users to override
the provider that is used to construct a resource to an instance of a
ProviderResource. 'providers', when passed to a component resource,
allows users to override providers used to construct children of the
component resource.
3. 'protect', 'providers', and 'provider' are all now inherited from
a resource's parent if they aren't specified in the child.
This commit adds the requisite code for the above changes and, in
addition, adds a number of new tests that exercise them and related code
paths.
* Rebase against master
* Add 'Output.all' combinator for Python
Output.all is a useful combinator that we already have in Node that
allows the composition of a list of outputs into an output of a list.
This is very useful when authoring components and its lack of presence
in Python was an oversight.
This commit adds 'Output.all' and 'Output.from_input', adding tests and
documentation for each.
* start unwrap
* Add functionality and test for nested inputs
The langhost shares its standard out and standard error with the
language executor that it is used (python/nodejs), so we must be sure to
flush our stdout and stderr before reporting a Run failure to the
engine.
'python' is not usually symlinked to 'python3' on most distros unless
you are already running in a virtual environment. Launching 'python3'
explicitly ensures that we will either launch the program successfully
or immediately fail, instead of launching the program with Python 2 and
failing with syntax errors at runtime.
This commit also emits an error message asking users to install Python
3.6 or later if we failed to find the 'python3' executable.
* Fix, formalize and add tests for property rewrites
The Python SDK provides two hooks for resources to override how their
properties are communicated to and from the engine. The code that
performs this transformation is subtle and, before this commit, subtly
incorrect.
This commit adds a test that verifies that the SDK correctly transforms
properties recursively according to the two transformation hooks, while
also fixing a smattering of test issues encountered when adding the new
test.
* CR feedback
We run the same suite of changes that we did on gometalinter. This
ended up catching a few new issues, some of which were addressed and
some of which were baselined.
- Remove the forked copy of the toolset
- Stop installing `pipenv` in sdk/python/Makefile
After this, we'll require that you already have `pipenv` present
before building.
* Implement RegisterResourceOutputs for Python 3
RegisterResourceOutputs allows Python 3 programs to export stack outputs
and export outputs off of component resources (which, under the hood,
are the same thing).
Adds a new integration test for stack outputs for Python programs, as
well as add a langhost test for register resource outputs.
Fixespulumi/pulumi#2163
* CR: Rename stack_output -> export
Fix integration tests that hardcoded paths to stack_outputs
* Fix one more reference to stack_outputs
future_input tests that it's possible to use coroutines as inputs to
Pulumi resources. resource_thens tests that it's possible to use outputs
to chain resource inputs and outputs together and that the SDK reports
correct dependencies to the engine.
This PR also fixes two bugs exposed by the new tests: first, coroutines
needed to be scheduled before awaiting (otherwise we'd deadlock) and
Nones in maps needed to be ignored when serializing and deserializing.
* Several gardening tasks for Python
1. Update pipenv to 2018.7.1, which is the most recent release that
isn't broken on Python 2
2. Update our pylint dependency to 1.9, the most recently released
version
3. Re-enable pylint for the Pulumi package
* Back out of pipenv upgrade
It's apparently broken in our CI. Also upgrade pylint to 2.1, which is
apparently the "actual" most recently release according to PyPI.
* Fix a bad merge
* Provide a Py hook for providers to rename props
This commit adds input and output hooks that can be overridden by
providers if they would like to change the names of dictionary fields
when projecting resources into Python.
* Add syntax sugar for dict outputs
Properly recurse when rewriting input dictionaries
* Implement RPC for Python 3
* Try not setting PYTHONPATH
* Remove PYTHONPATH line
* Implement Invoke for Python 3
* Implement register resource
* progress
* Rewrite the whole thing
* Fix a few bugs
* All tests pass
* Fix an abnormal shutdown bug
* CR feedback
* Provide a hook for resources to rename properties
As dictionaries and other classes come from the engine, the
translate_property hook can be used to intercept them and rename
properties if desired.
* Fix variable names and comments
* Disable Python integration tests for now
This commit introduces a 'next' package which we can use as a staging
ground for incrementally adopting new Python 3 code. The next package is
initially populated with the non-runtime portions of the Python SDK,
which is enough to pass all tests when running on Python 3. Future
commits will reach further into the runtime.
With this commit, the functions in 'pulumi.log' can be used to send
diagnostic messages to the Pulumi CLI. The Pulumi SDK bootstrap script
now also uses this feature to send diagnostic information on unhandled
exceptions to the Pulumi CLI.
Instead of looping forever, due to some recent improvements in engine
error handling it's sufficient for a language host to exit cleanly with
a zero exit code when the resource monitor is shutting down.
This introduces a Dockerfile for the Pulumi CLI. This makes it
easier to develop and test the engine in a self-contained environment,
in addition to being suitable for running the actual CLI itself.
For instance,
$ docker run pulumi/pulumi -e "PULUMI_ACCESS_TOKEN=x" up
will run the Pulumi program mounted under the /app volume. This will
be used in some upcoming CI/CD scenarios.
This uses multi-stage builds, and Debian Stretch as the base, for
relatively fast and lean build times and resulting images. We are
intentional about restoring dep packages independent of the actual
source code so that we don't end up needlessly re-depping, which can
consume quite a bit of time. After fixing
https://github.com/pulumi/pulumi/issues/1986, we should explore an
Alpine base image option.
I made the decision to keep this image scoped to just the Go builds.
Therefore, none of the actual SDK packages themselves are built, just
the engine, CLI, and language plugins for Node.js, Python, and Go.
It's possible to create a mega-container that has all of these full
environments so that we can rebuild them too, but for now I figured
it was better to rely on package management for them.
Another alternative would have been to install released binaries,
rather than building them. To keep the useful flow for development,
however, I decided to go the build route for now. If we build at the
same hashes, the resulting binaries "should" be ~identical anyhow.
I've created a pulumi/pulumi Docker Hub repo that we can publish this
into. For now, there is no CI publishing of the image.
This fixespulumi/pulumi#1991.
* Protobuf changes
* Move management of root resource state to engine
This commit fixes a persistent side-by-side issue in the NodeJS SDK by
moving the management of root resource state to the engine. Doing so
adds two new endpoints to the Engine gRPC service: 1) GetRootResource
and 2) SetRootResource, which get and set the root resource
respectively.
* Rebase against master, regenerate proto
* Use nightly protoc gRPC plugin for Node
Newer versions of the Node gRPC plugin accept the 'minimum_node_version'
flag, which we can use to instruct protoc to not support Node versions
earlier than Node 6. This allows the compiler to use 'Buffer.from'
instead of the deprecated 'Buffer' constructor, which fixes a
deprecation warning on Node 10.
* Protobuf changes
This commit will introduce a field, `IsStatus` to `LogRequest`. A
"status" logging event will be displayed in the `Info` column of the
main display, but will not be printed out at the end, when resource
operations complete.
For example, for complex resource initialization, we'd like to display a
series of intermediate results: `[1/4] Service object created`, for
example. We'd like these to appear in the `Info` column, but not at the
end, where they are not helpful to the user.
* Added dist target for make, will help with Homebrew
* Try to install go dependencies before building
* Make sure dep ensure is called before trying to build SDKs
* Removed dep ensure from dist initial step
### First-Class Providers
These changes implement support for first-class providers. First-class
providers are provider plugins that are exposed as resources via the
Pulumi programming model so that they may be explicitly and multiply
instantiated. Each instance of a provider resource may be configured
differently, and configuration parameters may be source from the
outputs of other resources.
### Provider Plugin Changes
In order to accommodate the need to verify and diff provider
configuration and configure providers without complete configuration
information, these changes adjust the high-level provider plugin
interface. Two new methods for validating a provider's configuration
and diffing changes to the same have been added (`CheckConfig` and
`DiffConfig`, respectively), and the type of the configuration bag
accepted by `Configure` has been changed to a `PropertyMap`.
These changes have not yet been reflected in the provider plugin gRPC
interface. We will do this in a set of follow-up changes. Until then,
these methods are implemented by adapters:
- `CheckConfig` validates that all configuration parameters are string
or unknown properties. This is necessary because existing plugins
only accept string-typed configuration values.
- `DiffConfig` either returns "never replace" if all configuration
values are known or "must replace" if any configuration value is
unknown. The justification for this behavior is given
[here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106)
- `Configure` converts the config bag to a legacy config map and
configures the provider plugin if all config values are known. If any
config value is unknown, the underlying plugin is not configured and
the provider may only perform `Check`, `Read`, and `Invoke`, all of
which return empty results. We justify this behavior becuase it is
only possible during a preview and provides the best experience we
can manage with the existing gRPC interface.
### Resource Model Changes
Providers are now exposed as resources that participate in a stack's
dependency graph. Like other resources, they are explicitly created,
may have multiple instances, and may have dependencies on other
resources. Providers are referred to using provider references, which
are a combination of the provider's URN and its ID. This design
addresses the need during a preview to refer to providers that have not
yet been physically created and therefore have no ID.
All custom resources that are not themselves providers must specify a
single provider via a provider reference. The named provider will be
used to manage that resource's CRUD operations. If a resource's
provider reference changes, the resource must be replaced. Though its
URN is not present in the resource's dependency list, the provider
should be treated as a dependency of the resource when topologically
sorting the dependency graph.
Finally, `Invoke` operations must now specify a provider to use for the
invocation via a provider reference.
### Engine Changes
First-class providers support requires a few changes to the engine:
- The engine must have some way to map from provider references to
provider plugins. It must be possible to add providers from a stack's
checkpoint to this map and to register new/updated providers during
the execution of a plan in response to CRUD operations on provider
resources.
- In order to support updating existing stacks using existing Pulumi
programs that may not explicitly instantiate providers, the engine
must be able to manage the "default" providers for each package
referenced by a checkpoint or Pulumi program. The configuration for
a "default" provider is taken from the stack's configuration data.
The former need is addressed by adding a provider registry type that is
responsible for managing all of the plugins required by a plan. In
addition to loading plugins froma checkpoint and providing the ability
to map from a provider reference to a provider plugin, this type serves
as the provider plugin for providers themselves (i.e. it is the
"provider provider").
The latter need is solved via two relatively self-contained changes to
plan setup and the eval source.
During plan setup, the old checkpoint is scanned for custom resources
that do not have a provider reference in order to compute the set of
packages that require a default provider. Once this set has been
computed, the required default provider definitions are conjured and
prepended to the checkpoint's resource list. Each resource that
requires a default provider is then updated to refer to the default
provider for its package.
While an eval source is running, each custom resource registration,
resource read, and invoke that does not name a provider is trapped
before being returned by the source iterator. If no default provider
for the appropriate package has been registered, the eval source
synthesizes an appropriate registration, waits for it to complete, and
records the registered provider's reference. This reference is injected
into the original request, which is then processed as usual. If a
default provider was already registered, the recorded reference is
used and no new registration occurs.
### SDK Changes
These changes only expose first-class providers from the Node.JS SDK.
- A new abstract class, `ProviderResource`, can be subclassed and used
to instantiate first-class providers.
- A new field in `ResourceOptions`, `provider`, can be used to supply
a particular provider instance to manage a `CustomResource`'s CRUD
operations.
- A new type, `InvokeOptions`, can be used to specify options that
control the behavior of a call to `pulumi.runtime.invoke`. This type
includes a `provider` field that is analogous to
`ResourceOptions.provider`.
When this argument is not provided, we'll default to the value of
pulumi.getProject(). This is what you want for application level code
anyway and it matches the CLI behavior where if you don't qualify a
key with a package we use the name of the current project.
Fixes#1581
* Protobuf changes to record dependencies for read resources
* Add a number of tests for read resources, especially around replacement
* Place read resources in the snapshot with "external" bit set
Fixespulumi/pulumi#1521. This commit introduces two new step ops: Read
and ReadReplacement. The engine generates Read and ReadReplacement steps
when servicing ReadResource RPC calls from the language host.
* Fix an omission of OpReadReplace from the step list
* Rebase against master
* Transition to use V2 Resources by default
* Add a semantic "relinquish" operation to the engine
If the engine observes that a resource is read and also that the
resource exists in the snapshot as a non-external resource, it will not
delete the resource if the IDs of the old and new resources match.
* Typo fix
* CR: add missing comments, DeserializeDeployment -> DeserializeDeploymentV2, ID check
A critical part of the partial update protocol is to return a structured
error when a resource is successfully created, but fails to initialize.
This structured error contains the properties of the
partially-initialized resource, and instructs the engine to halt.
Most languages implement this by attaching "details" to the error, i.e.,
an arbitrary proto message attached to the error. The JavaScript
implementation is not mature enough to include all the facilities
required to use this, so here we must add a `Status` message, which
protobuf requires as part of its structure for returning details.
* Test the Python language host end-to-end
This commit introduces an end-to-end language host testing framework for
the Python SDK, similar to what already exists for the Node SDK. The
real language host is used to run Pulumi programs written in Python
while mocking out the resource monitor.
* Add new tests
* Print out better diagnostics when the langhost fails to launch
* Use the in-tree executor for testing
* CR: Place tests and code being tested in the same directory for ease of understanding, add a README
* Turns out I misunderstood the semantics of resource registration - fix two tests so that they pass now and fix a few bugs in the test harness
This change includes the Python and Golang language hosts in the Windows
SDK. As part of this change, I had to adjust how we launched the second
stage of the language host, since we can't depend on the shebang, so now
we invoke `python` passing the executor and then the arguments.
Fixes#1509
* Fix a few issues with the Python language host
1. Fix an issue where the UNKNOWN sentinel was leaking into user
programs
2. Fix an issue where Protobuf types were leaking into user programs
In fixing this issues I also added a framework for writing tests against
the Python SDK.
* License headers, and adopt a more idiomatic testing pattern
* Additional idiomatic Python
* That's what I get for trying to be fancy (Travis CI python version is very old and does not respect this form)
* CR feedback: use more comprehensions, typo fix
* Break a circular dependency between resource, runtime.resource, and runtime.rpc
* Don't check in .vscode
* CR: sort inputs, rename global variable, add a test for CustomResource serialization
* Remove accidental code duplication
* Graceful RPC shutdown: CLI side
* Handle unavailable resource monitor in language hosts
* Fix a comment
* Don't commit package-lock.json
* fix mangled pylint pragma
* Rebase against master and fix Gopkg.lock
* Code review feedback
* Fix a race between closing the callerEventsOpt channel and terminating a goroutine that writes to it
* glog -> logging
These changes enable tracing of Pulumi API calls.
The span with which to associate an API call is passed via a
`context.Context` parameter. This required plumbing a
`context.Context` parameter through a rather large number of APIs,
especially in the backend.
In general, all API calls are associated with a new root span that
exists for essentially the entire lifetime of an invocation of the
Pulumi CLI. There were a few places where the plumbing got a bit hairier
than I was willing to address with these changes; I've used
`context.Background()` in these instances. API calls that receive this
context will create new root spans, but will still be traced.
* Improve the error message when npm/yarn install hasn't been run
* Same thing, but for Python
* Use PULUMI_RUN in batch script
* Use -e, -f doesn't work for symlinked paths (e.g. yarn link)
The RPC provider interface needs a way to convey back to the engine
that a resource being read no longer exists. To do this, we'll return
the ID property that was read back. If it is empty, it means the
resource is gone. If it is non-empty, we expect it to match the input.
This change wires up the new Read RPC method in such a manner that
Pulumi programs can invoke it. This is technically not required for
refreshing state programmatically (as in pulumi/pulumi#1081), however
it's a feature we had eons ago and have wanted since (see
pulumi/pulumi#83), and will allow us to write code like
let vm = aws.ec2.Instance.get("my-vm", "i-07043cd97bd2c9cfc");
// use any property from here on out ...
The way this works is simply by bridging the Pulumi program via its
existing RPC connection to the engine, much like Invoke and
RegisterResource RPC requests already do, and then invoking the proper
resource provider in order to read the state. Note that some resources
cannot be uniquely identified by their ID alone, and so an extra
resource state bag may be provided with just those properties required.
This came almost for free (okay, not exactly) and will come in handy as
we start gaining experience with reading live state from resources.
This commit changes two things about our resource model:
* Stop performing Pulumi Engine-side diffing of resource state.
Instead, we defer to the resource plugins themselves to determine
whether a change was made and, if so, the extent of it. This
manifests as a simple change to the Diff function; it is done in
a backwards compatible way so that we continue with legacy diffing
for existing resource provider plugins.
* Add a Read RPC method for resource providers. It simply takes a
resource's ID and URN, plus an optional bag of further qualifying
state, and it returns the current property state as read back from
the actual live environment. Note that the optional bag of state
must at least include enough additional properties for resources
wherein the ID is insufficient for the provider to perform a lookup.
It may, however, include the full bag of prior state, for instance
in the case of a refresh operation.
This is part of pulumi/pulumi#1108.
* Improve the error message arising from missing required configs for
resource providers
If the resource provider that we are speaking to is new enough, it will send
across a list of keys and their descriptions alongside an error
indicating that the provider we are configuring is missing required
config. This commit packages up the list of missing keys into an error
that can be presented nicely to the user.
* Code review feedback: renaming simplification and correcting errors in comments
This change actually makes our Python version numbers conformant
to PEP440. Previously we were including the Git commit hash in the
alpha "version number" part, which is incorrect. This simply led to
warnings upon publication and installation, but that warning very
clearly states that support for invalid versions will stop at some
point. This change puts any "informative" parts, like the Git hash,
inside of a local version tag, where such things are permitted.
Also move away from the inline sed silliness so that we can more
easily share this logic across all of our repos.
It appears 15.0.2 generates a bad reference to the pip._install module.
I've tested that 15.2.0 does not so, although I don't really understand
why and when this changed (my current guess is the Travis base image
changed), this should fixpulumi/pulumi#1103.
* Send structured errors across RPC boundaries
This brings us closer to gRPC best practices where we send structured
errors with error codes across RPC endpoints. The new "rpcerrors"
package can wrap errors from RPC endpoints, so RPC servers can attach
some additional context as to why a request failed.
* Code review feedback:
1. Rename rpcerrors -> rpcerror, better package name
2. Rename RPCError -> Error, RPCErrorCause -> ErrorCause, names
suggested by gometalinter to improve their package-qualified names
3. Fix import organization in rpcerror.go
The semantic versions we were using for pre-release PyPI packages wasn't
quite right. We had been using local version identifiers, a la +, rather
than proper pre-release tags, which means that version specifications like
`pulumi==0.11.0` will match `0.11.0+dev.1521506136.g7f043fd.dirty`, in
addition to just `0.11.0`. This is clearly not what we want.
This change moves us over to proper alpha and release candidate versions,
as specified in https://www.python.org/dev/peps/pep-0440/#pre-releases.
Namely, any `x.y.z-dev-123456789-abcdef` version will get translated into
an alpha `x.y.za12345679-abcdef`, and any `x.y.z-rc1` will get translated
into a proper release candidate `x.y.zrc1` version number.
Implement Python invoke RPC
This change implements the invoke function for resource provider
RPCs. This is required to support a customer scenario.
There are a few other minor updates:
* Rename pulumi.export to pulumi.output.
* Change register_resource to, like invoke, return the resulting
object/dictionary, instead of the set_outputs function.
* Initialize the monitor/engine RPC connections to None when not
attached to the engine, thus ensuring good error messages.
* Fix Python project/stack metadata
Our previous strategy of just using `git describe --tags --dirty` to
compute a version caused issues. The major one was that since version
sort lexigrapically, git's strategy of having a commit count without
leading zeros lead to cases where 0.11.0-dev-9 was "newer than"
0.11.0-dev-10 which is not what you want at all.
With this change, we compute a version by first seeing if the commit
is tagged, and if so, we use that tag. Otherwise, we take the closest
tag and to it append the unix timestamp of the commit and then append
a git hash.
Because we use the commit timestamp, things will sort correctly again.
Part of pulumi/home#174
This change uses virtualenv to insulate us from platform differences
in our building of the Python SDK, and to create an isolated Python 2
environment. This includes meaning we don't need to worry about the
specific location and behavior of Pylint. I *think* this will work
no matter whether it's Mac, Ubuntu, ArchLinux, Windows, and so on.
We do install to the --user directory in the install target using
`pip install -e`, however, which enables the machine-wide symlinking
that we need to support various workflows.
This fixespulumi/pulumi#1007.
This change temporarily disables Pylint. Assuming it is on the path,
and furthermore that the one on the path runs under 2.7, simply won't
work. See pulumi/pulumi#1007 for details; it also tracks reenabling.
This change includes a few things:
1) Prefer python2 and pip2 when on the PATH, over the undecorated
names python and pip. This is the standard convention for package
managers like Pip, etc., to support Python2 and Python3 side-by-side.
2) Fail-fast if neither can be found on the PATH.
3) Check the reported version number for python, pip, and pylint, and
fail-fast if it doesn't report back 2.7, just to safeguard against
undecorated binaries with unsupported versions.
Also, we had not listed wheel as a dependency in the requirements.txt
file. This needs to be there to support building bdist_wheels. Fixed.
We now publish the Pulumi Python SDK package to our private PyPI
server at the same time we also publish the NPM package. For now,
we use the test Pulumi.com service, and will switch to staging as
soon as it becomes available.
This change passes --user to pip install, so that it installs packages
underneath the home directory. This is required because, except for the
"python" image in Travis, all Python and Pip-related directories are
root-owned. The --user approach avoids needing to sudo all over the place.
This change includes a lot more functionality. Enough to actually
run the webserver-py example through previews, updates, and destroys!
* Actually wire up the gRPC connections to the engine/monitor.
* Move the Node.js and Python generated Protobuf/gRPC files underneath
the actual SDK directories to simplify this generally. No more
copying during `make` and, in fact, this was required to give a smoother
experience with good packages/modules for the Python's SDK development.
* Build the Python egg during `make build`.
* Add support for program stacks. Just like with the Node.js runtime,
we will auto-parent any resources without explicit parents to a single
top-level resource component.
* Add support for component resource output properties.
* Add get_project() and get_stack() functions for retrieving the current
project and stack names.
* Properly use UNKNOWN sentinels.
* Add a set_outputs() function on Resource. This is defined by the
code-generator and allows custom logic for output property setting.
This is cleaner than the way we do this in Node.js, and gives us a
way to ensure that output properties are "real" properties, complete
with member documentation. This also gives us a hook to perform
name demangling, which the code-generator typically controls anyway.
* Add package dependencies to setuptools.py and requirements.txt.
This change gets enough of the Python SDK up and running that the
empty Python program will work. Mostly just scaffolding, but the
basic structure is now in place. The primary remaining work is to
wire up resource creation to the gRPC interfaces.
In summary:
* The basic structure is as follows:
- Everything goes into sdk/python/.
- sdk/python/cmd/pulumi-langhost-python is a Go language host
that simply knows how to spawn Python processes to run out
entrypoint in response to requests by the engine.
- sdk/python/cmd/pulumi-langhost-python-exec is a little Python
shim that is invoked by the language host to run Python programs,
and is responsible for setting up the minimal goo before we can
do so (RPC connections and the like).
- sdk/python/lib/ contains a Python Pip package suitable for PyPi.
- In there, we have two packages: the root pulumi package that
contains all of the basic Pulumi programming model abstractions,
and pulumi.runtime, which contains the implementation of
resource registration, RPC interfacing with the engine, and so on.
* Add logic in our test framework to conditionalize on the language
type and react accordingly. This will allow us to skip Yarn for
Python projects and eventually run Pip if there's a requirements.txt.
* Created the basic project structure, including all of the usual
Make targets for installing into the proper places.
* Building also runs Pylint and we are clean.
There are a few other minor things in here:
* Add an "empty" test for both Node.js and Python. These pass.
* Fix an existing bug in plugin shutdown logic. At some point, we
started waiting for stderr/stdout to flush before shutting down
the plugin; but if certain failures happen "early" during the
plugin launch process, these channels will never get initialized
and so waiting for them deadlocks.
* Recently we seem to have added logic to delete test temp
directories if a failure happened during initialization of said
temp directories. This is unfortunate, because you often need to
look at the temp directory to see what failed. We already clean
them up elsewhere after the full test completes successfully, so
I don't think we need to be doing this, and I've removed it.
Still many loose ends (config, resources, etc), but it's a start!
This change adds a basic Python langhost RPC server. It's fairly
barebones and merely acts as a jumping off point for the Pulumi engine
to spawn a Python program. The host is written in Go, in contrast to
implementing the host in Python, and more closely resembles how I
expect the Node.js language host to work once pulumi/pulumi#331 is done.
