On startup, when we were populating the Settings object, we failed to
coerce the dry_run parameter from a string to a boolean, which resulted
in is_dry_run always believing that it was a preview. This PR fixes this
oversight by explicitly coercing to a boolean prior to sending the value
to Settings.
When is resource is waiting for its dependencies to resolve, it first
informs the RPC_MANAGER that it intends to do an RPC - this is to
prevent premature termination of the program while RPCs are still in
flight or queued to execute.
However, this is a problem whenever a resource fails to create for
whatever reason. The most common ways for this to happen are for invokes
to fail, resource creation itself to fail, or throwing in an apply.
Today, this causes a deadlock, since all consumers of the failed
resources block forever while never decrementing the RPC count.
This commit addresses the issue by deliberately (abnormally) resolving
all futures that are created in the process of preparing a resource.
This solves the problem by immediately terminating all resources that
are waiting for the failed resource's outputs to resolve - they resolve
immediately, and exceptionally.
The end result is now that, instead of deadlocking, a doomed program now
terminates as expected with a single exception message.
When previewing a real first-class provider, it is often the case that
the provider's ID is unknown during a preview. This commit fixes a bug
where we did not translate an unknown ID into the rpc layer's sentinel
UNKNOWN value where we should have, which caused the engine to fail to
resolve the provider reference.
For some reason, writing to stderr using sys.stderr.write doesn't work,
but printing to stderr using the print function does. I don't know why
this is, but I do know that this makes the printed messages show up as
messages on the Pulumi CLI.
Invoke in Node.js allows users to optionally pass a parent or a provider
to the invoke, which dictates either explicitly or implicitly which
provider to use when performing an invoke. If a provider is specified
explicitly, that provider is used to perform the invoke. If a parent is
specified, that parent's provider is used to perform the invoke.
* Implement first-class providers for Python
First-class providers are an explicit projection of providers themselves
into Pulumi programs. For the most post, providers are just regular
resources, but the addition of providers to the fray (and the ability of
resources to be constructed by providers in the same program) requires
some changes to the Python resource model.
A summary of the changes:
1. Added ProviderResource, a custom resource that is the base class of
all providers.
2. ResourceOptions now has 'provider' and 'providers' fields.
'provider', when passed to a custom resource, allows users to override
the provider that is used to construct a resource to an instance of a
ProviderResource. 'providers', when passed to a component resource,
allows users to override providers used to construct children of the
component resource.
3. 'protect', 'providers', and 'provider' are all now inherited from
a resource's parent if they aren't specified in the child.
This commit adds the requisite code for the above changes and, in
addition, adds a number of new tests that exercise them and related code
paths.
* Rebase against master
* Add 'Output.all' combinator for Python
Output.all is a useful combinator that we already have in Node that
allows the composition of a list of outputs into an output of a list.
This is very useful when authoring components and its lack of presence
in Python was an oversight.
This commit adds 'Output.all' and 'Output.from_input', adding tests and
documentation for each.
* start unwrap
* Add functionality and test for nested inputs
The langhost shares its standard out and standard error with the
language executor that it is used (python/nodejs), so we must be sure to
flush our stdout and stderr before reporting a Run failure to the
engine.
'python' is not usually symlinked to 'python3' on most distros unless
you are already running in a virtual environment. Launching 'python3'
explicitly ensures that we will either launch the program successfully
or immediately fail, instead of launching the program with Python 2 and
failing with syntax errors at runtime.
This commit also emits an error message asking users to install Python
3.6 or later if we failed to find the 'python3' executable.
* Fix, formalize and add tests for property rewrites
The Python SDK provides two hooks for resources to override how their
properties are communicated to and from the engine. The code that
performs this transformation is subtle and, before this commit, subtly
incorrect.
This commit adds a test that verifies that the SDK correctly transforms
properties recursively according to the two transformation hooks, while
also fixing a smattering of test issues encountered when adding the new
test.
* CR feedback
We run the same suite of changes that we did on gometalinter. This
ended up catching a few new issues, some of which were addressed and
some of which were baselined.
- Remove the forked copy of the toolset
- Stop installing `pipenv` in sdk/python/Makefile
After this, we'll require that you already have `pipenv` present
before building.
* Implement RegisterResourceOutputs for Python 3
RegisterResourceOutputs allows Python 3 programs to export stack outputs
and export outputs off of component resources (which, under the hood,
are the same thing).
Adds a new integration test for stack outputs for Python programs, as
well as add a langhost test for register resource outputs.
Fixespulumi/pulumi#2163
* CR: Rename stack_output -> export
Fix integration tests that hardcoded paths to stack_outputs
* Fix one more reference to stack_outputs
future_input tests that it's possible to use coroutines as inputs to
Pulumi resources. resource_thens tests that it's possible to use outputs
to chain resource inputs and outputs together and that the SDK reports
correct dependencies to the engine.
This PR also fixes two bugs exposed by the new tests: first, coroutines
needed to be scheduled before awaiting (otherwise we'd deadlock) and
Nones in maps needed to be ignored when serializing and deserializing.
* Several gardening tasks for Python
1. Update pipenv to 2018.7.1, which is the most recent release that
isn't broken on Python 2
2. Update our pylint dependency to 1.9, the most recently released
version
3. Re-enable pylint for the Pulumi package
* Back out of pipenv upgrade
It's apparently broken in our CI. Also upgrade pylint to 2.1, which is
apparently the "actual" most recently release according to PyPI.
* Fix a bad merge
* Provide a Py hook for providers to rename props
This commit adds input and output hooks that can be overridden by
providers if they would like to change the names of dictionary fields
when projecting resources into Python.
* Add syntax sugar for dict outputs
Properly recurse when rewriting input dictionaries
* Implement RPC for Python 3
* Try not setting PYTHONPATH
* Remove PYTHONPATH line
* Implement Invoke for Python 3
* Implement register resource
* progress
* Rewrite the whole thing
* Fix a few bugs
* All tests pass
* Fix an abnormal shutdown bug
* CR feedback
* Provide a hook for resources to rename properties
As dictionaries and other classes come from the engine, the
translate_property hook can be used to intercept them and rename
properties if desired.
* Fix variable names and comments
* Disable Python integration tests for now
This commit introduces a 'next' package which we can use as a staging
ground for incrementally adopting new Python 3 code. The next package is
initially populated with the non-runtime portions of the Python SDK,
which is enough to pass all tests when running on Python 3. Future
commits will reach further into the runtime.
With this commit, the functions in 'pulumi.log' can be used to send
diagnostic messages to the Pulumi CLI. The Pulumi SDK bootstrap script
now also uses this feature to send diagnostic information on unhandled
exceptions to the Pulumi CLI.
Instead of looping forever, due to some recent improvements in engine
error handling it's sufficient for a language host to exit cleanly with
a zero exit code when the resource monitor is shutting down.
This introduces a Dockerfile for the Pulumi CLI. This makes it
easier to develop and test the engine in a self-contained environment,
in addition to being suitable for running the actual CLI itself.
For instance,
$ docker run pulumi/pulumi -e "PULUMI_ACCESS_TOKEN=x" up
will run the Pulumi program mounted under the /app volume. This will
be used in some upcoming CI/CD scenarios.
This uses multi-stage builds, and Debian Stretch as the base, for
relatively fast and lean build times and resulting images. We are
intentional about restoring dep packages independent of the actual
source code so that we don't end up needlessly re-depping, which can
consume quite a bit of time. After fixing
https://github.com/pulumi/pulumi/issues/1986, we should explore an
Alpine base image option.
I made the decision to keep this image scoped to just the Go builds.
Therefore, none of the actual SDK packages themselves are built, just
the engine, CLI, and language plugins for Node.js, Python, and Go.
It's possible to create a mega-container that has all of these full
environments so that we can rebuild them too, but for now I figured
it was better to rely on package management for them.
Another alternative would have been to install released binaries,
rather than building them. To keep the useful flow for development,
however, I decided to go the build route for now. If we build at the
same hashes, the resulting binaries "should" be ~identical anyhow.
I've created a pulumi/pulumi Docker Hub repo that we can publish this
into. For now, there is no CI publishing of the image.
This fixespulumi/pulumi#1991.
* Protobuf changes
* Move management of root resource state to engine
This commit fixes a persistent side-by-side issue in the NodeJS SDK by
moving the management of root resource state to the engine. Doing so
adds two new endpoints to the Engine gRPC service: 1) GetRootResource
and 2) SetRootResource, which get and set the root resource
respectively.
* Rebase against master, regenerate proto
* Use nightly protoc gRPC plugin for Node
Newer versions of the Node gRPC plugin accept the 'minimum_node_version'
flag, which we can use to instruct protoc to not support Node versions
earlier than Node 6. This allows the compiler to use 'Buffer.from'
instead of the deprecated 'Buffer' constructor, which fixes a
deprecation warning on Node 10.
* Protobuf changes
This commit will introduce a field, `IsStatus` to `LogRequest`. A
"status" logging event will be displayed in the `Info` column of the
main display, but will not be printed out at the end, when resource
operations complete.
For example, for complex resource initialization, we'd like to display a
series of intermediate results: `[1/4] Service object created`, for
example. We'd like these to appear in the `Info` column, but not at the
end, where they are not helpful to the user.
* Added dist target for make, will help with Homebrew
* Try to install go dependencies before building
* Make sure dep ensure is called before trying to build SDKs
* Removed dep ensure from dist initial step
### First-Class Providers
These changes implement support for first-class providers. First-class
providers are provider plugins that are exposed as resources via the
Pulumi programming model so that they may be explicitly and multiply
instantiated. Each instance of a provider resource may be configured
differently, and configuration parameters may be source from the
outputs of other resources.
### Provider Plugin Changes
In order to accommodate the need to verify and diff provider
configuration and configure providers without complete configuration
information, these changes adjust the high-level provider plugin
interface. Two new methods for validating a provider's configuration
and diffing changes to the same have been added (`CheckConfig` and
`DiffConfig`, respectively), and the type of the configuration bag
accepted by `Configure` has been changed to a `PropertyMap`.
These changes have not yet been reflected in the provider plugin gRPC
interface. We will do this in a set of follow-up changes. Until then,
these methods are implemented by adapters:
- `CheckConfig` validates that all configuration parameters are string
or unknown properties. This is necessary because existing plugins
only accept string-typed configuration values.
- `DiffConfig` either returns "never replace" if all configuration
values are known or "must replace" if any configuration value is
unknown. The justification for this behavior is given
[here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106)
- `Configure` converts the config bag to a legacy config map and
configures the provider plugin if all config values are known. If any
config value is unknown, the underlying plugin is not configured and
the provider may only perform `Check`, `Read`, and `Invoke`, all of
which return empty results. We justify this behavior becuase it is
only possible during a preview and provides the best experience we
can manage with the existing gRPC interface.
### Resource Model Changes
Providers are now exposed as resources that participate in a stack's
dependency graph. Like other resources, they are explicitly created,
may have multiple instances, and may have dependencies on other
resources. Providers are referred to using provider references, which
are a combination of the provider's URN and its ID. This design
addresses the need during a preview to refer to providers that have not
yet been physically created and therefore have no ID.
All custom resources that are not themselves providers must specify a
single provider via a provider reference. The named provider will be
used to manage that resource's CRUD operations. If a resource's
provider reference changes, the resource must be replaced. Though its
URN is not present in the resource's dependency list, the provider
should be treated as a dependency of the resource when topologically
sorting the dependency graph.
Finally, `Invoke` operations must now specify a provider to use for the
invocation via a provider reference.
### Engine Changes
First-class providers support requires a few changes to the engine:
- The engine must have some way to map from provider references to
provider plugins. It must be possible to add providers from a stack's
checkpoint to this map and to register new/updated providers during
the execution of a plan in response to CRUD operations on provider
resources.
- In order to support updating existing stacks using existing Pulumi
programs that may not explicitly instantiate providers, the engine
must be able to manage the "default" providers for each package
referenced by a checkpoint or Pulumi program. The configuration for
a "default" provider is taken from the stack's configuration data.
The former need is addressed by adding a provider registry type that is
responsible for managing all of the plugins required by a plan. In
addition to loading plugins froma checkpoint and providing the ability
to map from a provider reference to a provider plugin, this type serves
as the provider plugin for providers themselves (i.e. it is the
"provider provider").
The latter need is solved via two relatively self-contained changes to
plan setup and the eval source.
During plan setup, the old checkpoint is scanned for custom resources
that do not have a provider reference in order to compute the set of
packages that require a default provider. Once this set has been
computed, the required default provider definitions are conjured and
prepended to the checkpoint's resource list. Each resource that
requires a default provider is then updated to refer to the default
provider for its package.
While an eval source is running, each custom resource registration,
resource read, and invoke that does not name a provider is trapped
before being returned by the source iterator. If no default provider
for the appropriate package has been registered, the eval source
synthesizes an appropriate registration, waits for it to complete, and
records the registered provider's reference. This reference is injected
into the original request, which is then processed as usual. If a
default provider was already registered, the recorded reference is
used and no new registration occurs.
### SDK Changes
These changes only expose first-class providers from the Node.JS SDK.
- A new abstract class, `ProviderResource`, can be subclassed and used
to instantiate first-class providers.
- A new field in `ResourceOptions`, `provider`, can be used to supply
a particular provider instance to manage a `CustomResource`'s CRUD
operations.
- A new type, `InvokeOptions`, can be used to specify options that
control the behavior of a call to `pulumi.runtime.invoke`. This type
includes a `provider` field that is analogous to
`ResourceOptions.provider`.
When this argument is not provided, we'll default to the value of
pulumi.getProject(). This is what you want for application level code
anyway and it matches the CLI behavior where if you don't qualify a
key with a package we use the name of the current project.
Fixes#1581
* Protobuf changes to record dependencies for read resources
* Add a number of tests for read resources, especially around replacement
* Place read resources in the snapshot with "external" bit set
Fixespulumi/pulumi#1521. This commit introduces two new step ops: Read
and ReadReplacement. The engine generates Read and ReadReplacement steps
when servicing ReadResource RPC calls from the language host.
* Fix an omission of OpReadReplace from the step list
* Rebase against master
* Transition to use V2 Resources by default
* Add a semantic "relinquish" operation to the engine
If the engine observes that a resource is read and also that the
resource exists in the snapshot as a non-external resource, it will not
delete the resource if the IDs of the old and new resources match.
* Typo fix
* CR: add missing comments, DeserializeDeployment -> DeserializeDeploymentV2, ID check
