* Use nightly protoc gRPC plugin for Node
Newer versions of the Node gRPC plugin accept the 'minimum_node_version'
flag, which we can use to instruct protoc to not support Node versions
earlier than Node 6. This allows the compiler to use 'Buffer.from'
instead of the deprecated 'Buffer' constructor, which fixes a
deprecation warning on Node 10.
* Protobuf changes
- Do not require replacement of dynamic resources due to provider
changes. This is not necessary, and is almost certainly the wrong
thing to do if the dynamic provider is managing a physical resource.
- Return all inputs by default from a dynamic provider's check method.
Currently a dynamic provider that does not implement check will end up
receiving no inputs. This is confusing, and is not the correct default.
* Combine two gRPC servers into one for testing
For some reason, our current gRPC test setup has become flaky now that
we are spinning up two gRPC servers. Hopefully merging them into one
helps clarify what's going on.
* Add back error logging for CI
* Validate type tokens before using them
When registering or reading a resource, we take the type token given to
us from the language host and assume that it's valid, which resulted in
assertion failures in various places in the engine. This commit
validates the format of type tokens given to us from the language host
and issues an appropriate error if it's not valid.
Along the way, this commit also improves the way that fatal exceptions
are rendered in the Node language host.
* Pre-allocate an exception for ReadResource
* Fix integration test
* CR Feedback
This commit is a lower-impact change that fixes the bugs associated with
invalid types on component resources and only checks that a type is
valid on custom resources.
* CR Take 2: Fix up IsProviderType instead of fixing call sites
* Please gometalinter
Instead of trying to probe for the normal path ourselves, just use
node's `require.resolve` statement to find `@pulumi/pulumi/cmd/run`.
This allows run to be found in cases where either yarn workspaces are
used, or the module has been installed globally.
Part of #1868
`opts.providers` is currently only read by the `Resource` constructor if
either `opts.parent` or `getRootResource` is not `undefined`. In
scnearios where exactly one copy of `@pulumi/pulumi` is loaded, one of
these conditions will always be true. In SxS scenarios, however, it is
possible for neither of these conditions to be true, and the created
resource will end up without a `providers` map. These changes fix that
by always copying the contents of `opts.providers` if it is defined.
Node calls 'exit' event callbacks when a process is preparing to exit,
via process.exit or otherwise, but it does not execute the next callback
in the chain if a callback calls process.exit.
* Use Promise.resolve.
* Use `Inputs | Promise<Inputs> | Output<Inputs>` rather than
`Input<Inputs>`, which looks supremely bizarre.
* Update ComponentResource.registerOutputs also.
This change partly addresses pulumi/pulumi#1611, by permitting you
to export a promise at the top-level, and have it be recognized as
a stack output. In other words, you can now say things like
async function main() {
...
return {
a: "x",
...,
z: 42,
};
}
module.exports = main();
and your Pulumi program will record distinct outputs as you'd hope:
---outputs:---
a: "x"
...
z: 42
This is arguably just a bug in the way we implemented stack outputs.
The remainder of the requests in #1611 will remain open for future
design and discussion, as they have more subtle ramifications.
If a `tsconfig.json` file is not present at the root of the Pulumi
project, ts-node will look up the directory tree to see if there is
one. If there is, it will treat that as the root of the project. While
reasonable for some cases, this isn't the behavior we want for our use
of ts-node. We actually set compiler options such that in the common
case you don't even need a `tsconfig.json` and for pure JavaScript
projects, there wouldn't be a `tsconfig.json` file.
In both of these cases, there's a big foot-gun waiting. For example in
pulumi/pulumi#1772 we ran into a case where there was a tsconfig.json
file in $HOME, causing the entirety of $HOME to be analyzed by
TypeScript which made it look like Pulumi hung.
To address this, tell ts-node to not use a project in cases where
there is not a `tsconfig.json` at the root of the project.
Fixes#1772
This commit will introduce a field, `IsStatus` to `LogRequest`. A
"status" logging event will be displayed in the `Info` column of the
main display, but will not be printed out at the end, when resource
operations complete.
For example, for complex resource initialization, we'd like to display a
series of intermediate results: `[1/4] Service object created`, for
example. We'd like these to appear in the `Info` column, but not at the
end, where they are not helpful to the user.
This change adopts the code review suggestion to use a bag of options
for config constraints rather than having overloaded function names.
This is a much cleaner approach, lets us use more descriptive names,
and is far more future proof in case we decide to add more capabilities.
Everytime I convert a CloudFormation template to Pulumi, I inevitably
run into the fact that CloudFormation has advanced "schema" capabilities
for input variables, like min/max for numbers and string lengths, enums,
and regex pattern matching. This is always cumbersome to convert.
In this change, I've added a number of config helpers for these cases:
For string enums:
getEnum(key: string, values: string[]): string | undefined;
requireEnum(key: string: values: string[]): string;
For min/max strlen:
getMinMax(key: string, min: number, max: number): string | undefined;
requireMinMax(key: string, min: number, max: number): string;
For regex patterns:
getPattern(key: string, regexp: string | RegExp): string | undefined;
requirePattern(key: string, regexp: string | RegExp): string;
For min/max strlen _and_ regex patterns:
getMinMaxPattern(key: string, min: number, max: number,
regexp: string | RegExp): string | undefined;
requireMinMaxPattern(key: string, min: number, max: number,
regexp: string | RegExp): string;
For min/max numbers:
getNumberMinMax(key: string, min: number, max: number): number | undefined;
requireNumberMinMax(key: string, min: number, max: number): number;
Each function throws a detailed RunError-derived exception type if the
configuration value doesn't meet the constraint.
This fixespulumi/pulumi#1671.
* Fix an issue with NodeJS host logging
Related to pulumi/pulumi#1694. This issue prevented the language host
from being aware that an engine (logging endpoint) was available and
thus no log messages were sent to the engine. By default, the language
host wrote them to standard out instead, which resulted in a pretty bad
error experience.
This commit fixes the PR and adds machinery to the NodeJS langhost tests
for testing the engine RPC endpoint. It is now possible to give a "log"
function to tests which will be hooked up to the "log" RPC endpoint
normally provided by the Pulumi engine.
* Remove accidental console.log
Replace the Source-based implementation of refresh with a phase that
runs as the first part of plan execution and rewrites the snapshot in-memory.
In order to fit neatly within the existing framework for resource operations,
these changes introduce a new kind of step, RefreshStep, to represent
refreshes. RefreshSteps operate similar to ReadSteps but do not imply that
the resource being read is not managed by Pulumi.
In addition to the refresh reimplementation, these changes incorporate those
from #1394 to run refresh in the integration test framework.
Fixes#1598.
Fixespulumi/pulumi-terraform#165.
Contributes to #1449.
The plugin host can ask the language host to provide a list of
resource plugins that it thinks will be nessecary for use at
deployment time, so they can be eagerly loaded.
In NodeJS (the only language host that implements this RPC) This works
by walking the directory tree rooted at the CWD of the project,
looking for package.json files, parsing them and seeing it they have
some marker property set. If they do, we add information about them
which we return at the end of our walk.
If there is *any* error, the entire operation fails. We've seen a
bunch of cases where this happens:
- Broken symlinks written by some editors as part of autosave.
- Access denied errors when part of the tree is unwalkable (Eric ran
into this on Windows when he had a Pulumi program at the root of his
file system.
- Recusive symlinks leading to errors when trying to walk down the
infinite chain. (See #1634 for one such example).
The very frustrating thing about this is that when you hit an error
its not clear what is going on and fixing it can be non-trivial. Even
worse, in the normal case, all of these plugins are already installed
and could be loaded by the host (in the common case, plugins are
installed as a post install step when you run `npm install`) so if we
simply didn't do this check at all, things would work great.
This change does two things:
1. It does not stop at the first error we hit when discovering
plugins, instead we just record the error and continue.
2. Does not fail the overall operation if there was an error. Instead,
we return to the host what we have, which may be an incomplete view
of the world. We glog the errors we did discover for diagnostics if
we ever need them.
I believe that long term most of this code gets deleted anyway. I
expect we will move to a model long term where the engine faults in
the plugin (downloading it if needed) when a request for the plugin
arrives. But for now, we shouldn't block normal operations just
because we couldn't answer a question with full fidelity.
Fixes#1478
If a resource's options bag does not specify `protect` or `provider`,
pull a default value from the resource's parent.
In order to allow a parent resource to specify providers for multiple
resource types, component resources now accept an optional map from
package name to provider instance. When a custom resource needs a
default provider from its parent, it checks its parent provider bag for
an entry under its package. If a component resource does not have a
provider bag, its pulls a default from its parent.
These changes also add a `parent` field to `InvokeOptions` s.t. calls to
invoke can use the same behavior as resource creation w.r.t. providers.
Fixes#1735, #1736.
* Added dist target for make, will help with Homebrew
* Try to install go dependencies before building
* Make sure dep ensure is called before trying to build SDKs
* Removed dep ensure from dist initial step
### First-Class Providers
These changes implement support for first-class providers. First-class
providers are provider plugins that are exposed as resources via the
Pulumi programming model so that they may be explicitly and multiply
instantiated. Each instance of a provider resource may be configured
differently, and configuration parameters may be source from the
outputs of other resources.
### Provider Plugin Changes
In order to accommodate the need to verify and diff provider
configuration and configure providers without complete configuration
information, these changes adjust the high-level provider plugin
interface. Two new methods for validating a provider's configuration
and diffing changes to the same have been added (`CheckConfig` and
`DiffConfig`, respectively), and the type of the configuration bag
accepted by `Configure` has been changed to a `PropertyMap`.
These changes have not yet been reflected in the provider plugin gRPC
interface. We will do this in a set of follow-up changes. Until then,
these methods are implemented by adapters:
- `CheckConfig` validates that all configuration parameters are string
or unknown properties. This is necessary because existing plugins
only accept string-typed configuration values.
- `DiffConfig` either returns "never replace" if all configuration
values are known or "must replace" if any configuration value is
unknown. The justification for this behavior is given
[here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106)
- `Configure` converts the config bag to a legacy config map and
configures the provider plugin if all config values are known. If any
config value is unknown, the underlying plugin is not configured and
the provider may only perform `Check`, `Read`, and `Invoke`, all of
which return empty results. We justify this behavior becuase it is
only possible during a preview and provides the best experience we
can manage with the existing gRPC interface.
### Resource Model Changes
Providers are now exposed as resources that participate in a stack's
dependency graph. Like other resources, they are explicitly created,
may have multiple instances, and may have dependencies on other
resources. Providers are referred to using provider references, which
are a combination of the provider's URN and its ID. This design
addresses the need during a preview to refer to providers that have not
yet been physically created and therefore have no ID.
All custom resources that are not themselves providers must specify a
single provider via a provider reference. The named provider will be
used to manage that resource's CRUD operations. If a resource's
provider reference changes, the resource must be replaced. Though its
URN is not present in the resource's dependency list, the provider
should be treated as a dependency of the resource when topologically
sorting the dependency graph.
Finally, `Invoke` operations must now specify a provider to use for the
invocation via a provider reference.
### Engine Changes
First-class providers support requires a few changes to the engine:
- The engine must have some way to map from provider references to
provider plugins. It must be possible to add providers from a stack's
checkpoint to this map and to register new/updated providers during
the execution of a plan in response to CRUD operations on provider
resources.
- In order to support updating existing stacks using existing Pulumi
programs that may not explicitly instantiate providers, the engine
must be able to manage the "default" providers for each package
referenced by a checkpoint or Pulumi program. The configuration for
a "default" provider is taken from the stack's configuration data.
The former need is addressed by adding a provider registry type that is
responsible for managing all of the plugins required by a plan. In
addition to loading plugins froma checkpoint and providing the ability
to map from a provider reference to a provider plugin, this type serves
as the provider plugin for providers themselves (i.e. it is the
"provider provider").
The latter need is solved via two relatively self-contained changes to
plan setup and the eval source.
During plan setup, the old checkpoint is scanned for custom resources
that do not have a provider reference in order to compute the set of
packages that require a default provider. Once this set has been
computed, the required default provider definitions are conjured and
prepended to the checkpoint's resource list. Each resource that
requires a default provider is then updated to refer to the default
provider for its package.
While an eval source is running, each custom resource registration,
resource read, and invoke that does not name a provider is trapped
before being returned by the source iterator. If no default provider
for the appropriate package has been registered, the eval source
synthesizes an appropriate registration, waits for it to complete, and
records the registered provider's reference. This reference is injected
into the original request, which is then processed as usual. If a
default provider was already registered, the recorded reference is
used and no new registration occurs.
### SDK Changes
These changes only expose first-class providers from the Node.JS SDK.
- A new abstract class, `ProviderResource`, can be subclassed and used
to instantiate first-class providers.
- A new field in `ResourceOptions`, `provider`, can be used to supply
a particular provider instance to manage a `CustomResource`'s CRUD
operations.
- A new type, `InvokeOptions`, can be used to specify options that
control the behavior of a call to `pulumi.runtime.invoke`. This type
includes a `provider` field that is analogous to
`ResourceOptions.provider`.
