* Revert "Make toString and toJSON internal (#2489)"
This reverts commit 7579b84f73.
* Revert "Update error message to point at docs. (#2488)"
This reverts commit 9156c26a2e.
* Revert "Throw on Output.toString and toJSON (#2486)"
This reverts commit c33b4505c0.
These changes add a new flag to the various `ResourceOptions` types that
indicates that a resource should be deleted before it is replaced, even
if the provider does not require this behavior. The usual
delete-before-replace cascade semantics apply.
Fixes#1620.
* Work around commonjs protoc bug
When compiling with the commonjs target, the protoc compiler still emits
references to Closure Compiler-isms that whack global state onto the
global object. This is particularly bad for us since we expect to be
able to make backwards-compatible changes to our Protobuf definitions
without breaking things, and this bug makes it impossible to do so.
To remedy the bug, this commit hacks the output of protoc (again) with
sed in order to avoid ever touching the global object. Everything still
works fine because the commonjs target (correctly) exports the protobuf
message types via the module system - it's just not writing to global
anymore.
* Fix status.proto
* Don't hack status.proto
- Add support for per-property dependencies to the Go SDK
- Add tests for first-class secret rejection in the checkpoint and RPC
layers and language SDKs
This implements the new algorithm for deciding which resources must be
deleted due to a delete-before-replace operation.
We need to compute the set of resources that may be replaced by a
change to the resource under consideration. We do this by taking the
complete set of transitive dependents on the resource under
consideration and removing any resources that would not be replaced by
changes to their dependencies. We determine whether or not a resource
may be replaced by substituting unknowns for input properties that may
change due to deletion of the resources their value depends on and
calling the resource provider's Diff method.
This is perhaps clearer when described by example. Consider the
following dependency graph:
A
__|__
B C
| _|_
D E F
In this graph, all of B, C, D, E, and F transitively depend on A. It may
be the case, however, that changes to the specific properties of any of
those resources R that would occur if a resource on the path to A were
deleted and recreated may not cause R to be replaced. For example, the
edge from B to A may be a simple dependsOn edge such that a change to
B does not actually influence any of B's input properties. In that case,
neither B nor D would need to be deleted before A could be deleted.
In order to make the above algorithm a reality, the resource monitor
interface has been updated to include a map that associates an input
property key with the list of resources that input property depends on.
Older clients of the resource monitor will leave this map empty, in
which case all input properties will be treated as depending on all
dependencies of the resource. This is probably overly conservative, but
it is less conservative than what we currently implement, and is
certainly correct.
It is possible for the sub-process responsible for running a NodeJS
Pulumi program to exit with a success code before the user's program has
run if the process of loading the runtime generates an unhandled promise
rejection. These changes fix this by registering the unhandled exception
and rejection handlers that are responsible for ensuring a non-zero exit
code in these cases before any other action is taken.
Note that this issue is really only possible because the Node language
host (like the Python language host) is composed of two processes: one
that serves the language host gRPC service and one that loads and runs
the user's program. The former launches the latter in response to a call
to its `Run` gRPC endpoint. The lifetime of the user's program is
considered to be bounded by the lifetime of the `Run` invocation. The
NodeJS process maintains its own connection to the engine over which
resource registrations are communicated. It is tempting to add a message
to the resource monitor RPC interface that signals that no further
registrations are performed, but this is complicated due to the
three-party topology and the possibility that such an RPC may never be
sent (e.g. due to a crash or a downlevel version of the Pulumi Node
runtime).
Fixes#2316.
This is something of a quick hack to work around the limitations that
are at the root of #2310. In short, `StackReference` resources do not
change during a update because their inputs have not changed and we do
not refresh resources as part of the update by default. We want
`StackReference` to remain a resource for myriad reasons (not the least
of which is to avoid a breaking change), but it does seem correct for it
to refresh its state during each update. Because there is no actual CRUD
operation associated with a `StackReferece`, we can obtain this behavior
by changing the implementation of its ctor in the various SDKs s.t. its
options bag always contains an appropriate `id`, thus indicating a
`Read`.
Fixes#2310.
Previously, we assumed that the dynamic provider was located in
`./node_modules/@pulumi/pulumi/../` which is correct in the majority
of cases. However, tools like lerna or yarn workspaces (or custom
workflows) allow the node_modules folder to be located elsewhere on
disk, and node will still find it because of its algorithm for module
resolution.
So, do what we do in the language host itself, first launch node and
ask it to tell us where it resolves a require statement to on disk and
then launch node against that script.
Fixes#2261
If a custom resource has explicitly specified a provider, add that
provider to the resource's provider map under the resource's package.
This allows children of the custom resource to inherit the resource's
provider.
Fixes#2262.
The langhost shares its standard out and standard error with the
language executor that it is used (python/nodejs), so we must be sure to
flush our stdout and stderr before reporting a Run failure to the
engine.
This changes the input type for dependsOn from simply
`Resource[] | Resource` to `Input<Input<Resource>[]> | Input<Resource>`.
This permits `Output<Resource>`s, etc in addition to
`Promise<Resource>`s. The logic for dynamically unpicking the right
types and recursing through the data structures isn't straightforward,
but I've written a test for all of the interesting permutations.
This fixespulumi/pulumi#991.
These changes add a new resource to the Pulumi SDK,
`pulumi.StackReference`, that represents a reference to another stack.
This resource has an output property, `outputs`, that contains the
complete set of outputs for the referenced stack. The Pulumi account
performing the deployment that creates a `StackReference` must have
access to the referenced stack or the call will fail.
This resource is implemented by a builtin provider managed by the engine.
This provider will be used for any custom resources and invokes inside
the `pulumi:pulumi` module. Currently this provider supports only the
`pulumi:pulumi:StackReference` resource.
Fixes#109.
We run the same suite of changes that we did on gometalinter. This
ended up catching a few new issues, some of which were addressed and
some of which were baselined.
* Make v8 primitives async as there is no way to avoid async in node11.
* Simplify API.
* Move processing of well-known globals into the v8 layer.
We'll need this so that we can map from RemoteObjectIds back to these well known values.
* Remove unnecesssary check.
* Cleanup comments and extract helper.
* Introduce helper bridge method for the simple case of making an entry for a string.
* Make functions async. They'll need to be async once we move to the Inspector api.
* Make functions async. They'll need to be async once we move to the Inspector api.
* Make functions async. They'll need to be async once we move to the Inspector api.
* Move property access behind helpers so they can move to the Inspector API in the future.
* Only call function when we know we have a Function. Remove redundant null check.
* Properly serialize certain special JavaScript number values that JSON serialization cannot handle.
* Only marshall across the 'source' and 'flags' for a RegExp when serializing.
* Add a simple test to validate a regex without flags.
* Extract functionality into helper method.
* Add test with complex output scenarios.
* Output serialization needs to avoid recursively trying to serialize a serialized value.
* Introduce indirection for introspecting properties of an object.
* Use our own introspection API for examining an Array.
* Hide direct property access through API indirection.
* Produce values like the v8 Inspector does.
* Compute the module map asynchronously. Will need that when mapping mirrors instead.
* Cleanup a little code in closure creation.
* Get serialization working on Node11 (except function locations).
* Run tests in the same order on <v11 and >=v11
* Make tests run on multiple versions of node.
* Rename file to make PR simpler to review.
* Cleanup.
* Be more careful with global state.
* Remove commented line.
* Only allow getting a session when on Node11 or above.
* Promisify methods.
Rather than placing these combinators directly on the Output class,
which feels odd because they are special purpose to iterables, and deal
with not only Outputs but also Inputs, we will place them on a
separate and dedicated iterable module for these utility helpers.
This change adds some new constructors for output properties:
1) We alias `Output.create` to `output`, more like Promise's various
construction methods. This reads better and is more discoverable.
2) A new `Output.createMap` function will accept an array of inputs,
along with a selector function for key/value pairs, and produces
an output map with said keys and values inside of it.
3) A new `Output.createGroupByMap` functon will similarly accept an
array of inputs and a key/value selector, however it creates an
output map with said keys, but where values are arrays of values,
and all duplicate keys will lead to appending to said arrays.
Tests to come in a subsequent checkin.
* Fail closure serialization in Node 11
Node 11 changed many of the intrinsics that we depend upon for closure
serialization, so until we fix the underlying issues this commit lazily
fails if a closure is serialized when running on Node 11.
* CR feedback
Some providers (namely Kubernetes) require unbounded parallelism in
order to function correctly. This commit enables the engine to operate
in a mode with unbounded parallelism and switches to that mode by
default.
Suppose you have `pulumi.output(o).apply(foo)`, with `o` being type `O`
and `foo` taking type `O` as an argument. If `O` is a type with methods,
this will fail to type check.
The reason is that `UnwrappedObject<T>` (as well as the other
`Unwrapped*` types) will recursively wrap the types of field values
whose type was `Function`. Since `UnwrappedObject<Function>` is not the
same as `Function`, we fail to type check. Note that this does not
result in an actual "boxed" object -- this is purely at the type level.
This commit resolves this by considering `Function` a primitive type,
which will cause us to not wrap the types of field values, instead
leaving them as `Function`.
Instead of looping forever, due to some recent improvements in engine
error handling it's sufficient for a language host to exit cleanly with
a zero exit code when the resource monitor is shutting down.
If you forget to implement check on a dynamic provider, all your
inputs mysteriously disappear. It's doubly odd because many providers
don't need to perform any checking on transformation of their inputs.
This change simply propagates the new inputs as-is by default when
a user-supplied check method isn't provided. This would have saved
me 20 minutes just now ... :-)
This introduces a Dockerfile for the Pulumi CLI. This makes it
easier to develop and test the engine in a self-contained environment,
in addition to being suitable for running the actual CLI itself.
For instance,
$ docker run pulumi/pulumi -e "PULUMI_ACCESS_TOKEN=x" up
will run the Pulumi program mounted under the /app volume. This will
be used in some upcoming CI/CD scenarios.
This uses multi-stage builds, and Debian Stretch as the base, for
relatively fast and lean build times and resulting images. We are
intentional about restoring dep packages independent of the actual
source code so that we don't end up needlessly re-depping, which can
consume quite a bit of time. After fixing
https://github.com/pulumi/pulumi/issues/1986, we should explore an
Alpine base image option.
I made the decision to keep this image scoped to just the Go builds.
Therefore, none of the actual SDK packages themselves are built, just
the engine, CLI, and language plugins for Node.js, Python, and Go.
It's possible to create a mega-container that has all of these full
environments so that we can rebuild them too, but for now I figured
it was better to rely on package management for them.
Another alternative would have been to install released binaries,
rather than building them. To keep the useful flow for development,
however, I decided to go the build route for now. If we build at the
same hashes, the resulting binaries "should" be ~identical anyhow.
I've created a pulumi/pulumi Docker Hub repo that we can publish this
into. For now, there is no CI publishing of the image.
This fixespulumi/pulumi#1991.
* Revert RunError behavior. Introduce new ResourceError for errors associated with a resource.
* Fix docs.
* Use resource error.
* Use ResourceError in more places.
* Use ResourceError in a few more places.
* Throw a resource error.
* Make required.
* Revert this.
* Lint.
* Only report errors once.
* Better comment.
* Protobuf changes
* Move management of root resource state to engine
This commit fixes a persistent side-by-side issue in the NodeJS SDK by
moving the management of root resource state to the engine. Doing so
adds two new endpoints to the Engine gRPC service: 1) GetRootResource
and 2) SetRootResource, which get and set the root resource
respectively.
* Rebase against master, regenerate proto
* Use nightly protoc gRPC plugin for Node
Newer versions of the Node gRPC plugin accept the 'minimum_node_version'
flag, which we can use to instruct protoc to not support Node versions
earlier than Node 6. This allows the compiler to use 'Buffer.from'
instead of the deprecated 'Buffer' constructor, which fixes a
deprecation warning on Node 10.
* Protobuf changes
- Do not require replacement of dynamic resources due to provider
changes. This is not necessary, and is almost certainly the wrong
thing to do if the dynamic provider is managing a physical resource.
- Return all inputs by default from a dynamic provider's check method.
Currently a dynamic provider that does not implement check will end up
receiving no inputs. This is confusing, and is not the correct default.
* Combine two gRPC servers into one for testing
For some reason, our current gRPC test setup has become flaky now that
we are spinning up two gRPC servers. Hopefully merging them into one
helps clarify what's going on.
* Add back error logging for CI
* Validate type tokens before using them
When registering or reading a resource, we take the type token given to
us from the language host and assume that it's valid, which resulted in
assertion failures in various places in the engine. This commit
validates the format of type tokens given to us from the language host
and issues an appropriate error if it's not valid.
Along the way, this commit also improves the way that fatal exceptions
are rendered in the Node language host.
* Pre-allocate an exception for ReadResource
* Fix integration test
* CR Feedback
This commit is a lower-impact change that fixes the bugs associated with
invalid types on component resources and only checks that a type is
valid on custom resources.
* CR Take 2: Fix up IsProviderType instead of fixing call sites
* Please gometalinter
Instead of trying to probe for the normal path ourselves, just use
node's `require.resolve` statement to find `@pulumi/pulumi/cmd/run`.
This allows run to be found in cases where either yarn workspaces are
used, or the module has been installed globally.
Part of #1868
`opts.providers` is currently only read by the `Resource` constructor if
either `opts.parent` or `getRootResource` is not `undefined`. In
scnearios where exactly one copy of `@pulumi/pulumi` is loaded, one of
these conditions will always be true. In SxS scenarios, however, it is
possible for neither of these conditions to be true, and the created
resource will end up without a `providers` map. These changes fix that
by always copying the contents of `opts.providers` if it is defined.
Node calls 'exit' event callbacks when a process is preparing to exit,
via process.exit or otherwise, but it does not execute the next callback
in the chain if a callback calls process.exit.
* Use Promise.resolve.
* Use `Inputs | Promise<Inputs> | Output<Inputs>` rather than
`Input<Inputs>`, which looks supremely bizarre.
* Update ComponentResource.registerOutputs also.
This change partly addresses pulumi/pulumi#1611, by permitting you
to export a promise at the top-level, and have it be recognized as
a stack output. In other words, you can now say things like
async function main() {
...
return {
a: "x",
...,
z: 42,
};
}
module.exports = main();
and your Pulumi program will record distinct outputs as you'd hope:
---outputs:---
a: "x"
...
z: 42
This is arguably just a bug in the way we implemented stack outputs.
The remainder of the requests in #1611 will remain open for future
design and discussion, as they have more subtle ramifications.
If a `tsconfig.json` file is not present at the root of the Pulumi
project, ts-node will look up the directory tree to see if there is
one. If there is, it will treat that as the root of the project. While
reasonable for some cases, this isn't the behavior we want for our use
of ts-node. We actually set compiler options such that in the common
case you don't even need a `tsconfig.json` and for pure JavaScript
projects, there wouldn't be a `tsconfig.json` file.
In both of these cases, there's a big foot-gun waiting. For example in
pulumi/pulumi#1772 we ran into a case where there was a tsconfig.json
file in $HOME, causing the entirety of $HOME to be analyzed by
TypeScript which made it look like Pulumi hung.
To address this, tell ts-node to not use a project in cases where
there is not a `tsconfig.json` at the root of the project.
Fixes#1772
This commit will introduce a field, `IsStatus` to `LogRequest`. A
"status" logging event will be displayed in the `Info` column of the
main display, but will not be printed out at the end, when resource
operations complete.
For example, for complex resource initialization, we'd like to display a
series of intermediate results: `[1/4] Service object created`, for
example. We'd like these to appear in the `Info` column, but not at the
end, where they are not helpful to the user.
This change adopts the code review suggestion to use a bag of options
for config constraints rather than having overloaded function names.
This is a much cleaner approach, lets us use more descriptive names,
and is far more future proof in case we decide to add more capabilities.
Everytime I convert a CloudFormation template to Pulumi, I inevitably
run into the fact that CloudFormation has advanced "schema" capabilities
for input variables, like min/max for numbers and string lengths, enums,
and regex pattern matching. This is always cumbersome to convert.
In this change, I've added a number of config helpers for these cases:
For string enums:
getEnum(key: string, values: string[]): string | undefined;
requireEnum(key: string: values: string[]): string;
For min/max strlen:
getMinMax(key: string, min: number, max: number): string | undefined;
requireMinMax(key: string, min: number, max: number): string;
For regex patterns:
getPattern(key: string, regexp: string | RegExp): string | undefined;
requirePattern(key: string, regexp: string | RegExp): string;
For min/max strlen _and_ regex patterns:
getMinMaxPattern(key: string, min: number, max: number,
regexp: string | RegExp): string | undefined;
requireMinMaxPattern(key: string, min: number, max: number,
regexp: string | RegExp): string;
For min/max numbers:
getNumberMinMax(key: string, min: number, max: number): number | undefined;
requireNumberMinMax(key: string, min: number, max: number): number;
Each function throws a detailed RunError-derived exception type if the
configuration value doesn't meet the constraint.
This fixespulumi/pulumi#1671.
* Fix an issue with NodeJS host logging
Related to pulumi/pulumi#1694. This issue prevented the language host
from being aware that an engine (logging endpoint) was available and
thus no log messages were sent to the engine. By default, the language
host wrote them to standard out instead, which resulted in a pretty bad
error experience.
This commit fixes the PR and adds machinery to the NodeJS langhost tests
for testing the engine RPC endpoint. It is now possible to give a "log"
function to tests which will be hooked up to the "log" RPC endpoint
normally provided by the Pulumi engine.
* Remove accidental console.log
Replace the Source-based implementation of refresh with a phase that
runs as the first part of plan execution and rewrites the snapshot in-memory.
In order to fit neatly within the existing framework for resource operations,
these changes introduce a new kind of step, RefreshStep, to represent
refreshes. RefreshSteps operate similar to ReadSteps but do not imply that
the resource being read is not managed by Pulumi.
In addition to the refresh reimplementation, these changes incorporate those
from #1394 to run refresh in the integration test framework.
Fixes#1598.
Fixespulumi/pulumi-terraform#165.
Contributes to #1449.
The plugin host can ask the language host to provide a list of
resource plugins that it thinks will be nessecary for use at
deployment time, so they can be eagerly loaded.
In NodeJS (the only language host that implements this RPC) This works
by walking the directory tree rooted at the CWD of the project,
looking for package.json files, parsing them and seeing it they have
some marker property set. If they do, we add information about them
which we return at the end of our walk.
If there is *any* error, the entire operation fails. We've seen a
bunch of cases where this happens:
- Broken symlinks written by some editors as part of autosave.
- Access denied errors when part of the tree is unwalkable (Eric ran
into this on Windows when he had a Pulumi program at the root of his
file system.
- Recusive symlinks leading to errors when trying to walk down the
infinite chain. (See #1634 for one such example).
The very frustrating thing about this is that when you hit an error
its not clear what is going on and fixing it can be non-trivial. Even
worse, in the normal case, all of these plugins are already installed
and could be loaded by the host (in the common case, plugins are
installed as a post install step when you run `npm install`) so if we
simply didn't do this check at all, things would work great.
This change does two things:
1. It does not stop at the first error we hit when discovering
plugins, instead we just record the error and continue.
2. Does not fail the overall operation if there was an error. Instead,
we return to the host what we have, which may be an incomplete view
of the world. We glog the errors we did discover for diagnostics if
we ever need them.
I believe that long term most of this code gets deleted anyway. I
expect we will move to a model long term where the engine faults in
the plugin (downloading it if needed) when a request for the plugin
arrives. But for now, we shouldn't block normal operations just
because we couldn't answer a question with full fidelity.
Fixes#1478
If a resource's options bag does not specify `protect` or `provider`,
pull a default value from the resource's parent.
In order to allow a parent resource to specify providers for multiple
resource types, component resources now accept an optional map from
package name to provider instance. When a custom resource needs a
default provider from its parent, it checks its parent provider bag for
an entry under its package. If a component resource does not have a
provider bag, its pulls a default from its parent.
These changes also add a `parent` field to `InvokeOptions` s.t. calls to
invoke can use the same behavior as resource creation w.r.t. providers.
Fixes#1735, #1736.
* Added dist target for make, will help with Homebrew
* Try to install go dependencies before building
* Make sure dep ensure is called before trying to build SDKs
* Removed dep ensure from dist initial step
### First-Class Providers
These changes implement support for first-class providers. First-class
providers are provider plugins that are exposed as resources via the
Pulumi programming model so that they may be explicitly and multiply
instantiated. Each instance of a provider resource may be configured
differently, and configuration parameters may be source from the
outputs of other resources.
### Provider Plugin Changes
In order to accommodate the need to verify and diff provider
configuration and configure providers without complete configuration
information, these changes adjust the high-level provider plugin
interface. Two new methods for validating a provider's configuration
and diffing changes to the same have been added (`CheckConfig` and
`DiffConfig`, respectively), and the type of the configuration bag
accepted by `Configure` has been changed to a `PropertyMap`.
These changes have not yet been reflected in the provider plugin gRPC
interface. We will do this in a set of follow-up changes. Until then,
these methods are implemented by adapters:
- `CheckConfig` validates that all configuration parameters are string
or unknown properties. This is necessary because existing plugins
only accept string-typed configuration values.
- `DiffConfig` either returns "never replace" if all configuration
values are known or "must replace" if any configuration value is
unknown. The justification for this behavior is given
[here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106)
- `Configure` converts the config bag to a legacy config map and
configures the provider plugin if all config values are known. If any
config value is unknown, the underlying plugin is not configured and
the provider may only perform `Check`, `Read`, and `Invoke`, all of
which return empty results. We justify this behavior becuase it is
only possible during a preview and provides the best experience we
can manage with the existing gRPC interface.
### Resource Model Changes
Providers are now exposed as resources that participate in a stack's
dependency graph. Like other resources, they are explicitly created,
may have multiple instances, and may have dependencies on other
resources. Providers are referred to using provider references, which
are a combination of the provider's URN and its ID. This design
addresses the need during a preview to refer to providers that have not
yet been physically created and therefore have no ID.
All custom resources that are not themselves providers must specify a
single provider via a provider reference. The named provider will be
used to manage that resource's CRUD operations. If a resource's
provider reference changes, the resource must be replaced. Though its
URN is not present in the resource's dependency list, the provider
should be treated as a dependency of the resource when topologically
sorting the dependency graph.
Finally, `Invoke` operations must now specify a provider to use for the
invocation via a provider reference.
### Engine Changes
First-class providers support requires a few changes to the engine:
- The engine must have some way to map from provider references to
provider plugins. It must be possible to add providers from a stack's
checkpoint to this map and to register new/updated providers during
the execution of a plan in response to CRUD operations on provider
resources.
- In order to support updating existing stacks using existing Pulumi
programs that may not explicitly instantiate providers, the engine
must be able to manage the "default" providers for each package
referenced by a checkpoint or Pulumi program. The configuration for
a "default" provider is taken from the stack's configuration data.
The former need is addressed by adding a provider registry type that is
responsible for managing all of the plugins required by a plan. In
addition to loading plugins froma checkpoint and providing the ability
to map from a provider reference to a provider plugin, this type serves
as the provider plugin for providers themselves (i.e. it is the
"provider provider").
The latter need is solved via two relatively self-contained changes to
plan setup and the eval source.
During plan setup, the old checkpoint is scanned for custom resources
that do not have a provider reference in order to compute the set of
packages that require a default provider. Once this set has been
computed, the required default provider definitions are conjured and
prepended to the checkpoint's resource list. Each resource that
requires a default provider is then updated to refer to the default
provider for its package.
While an eval source is running, each custom resource registration,
resource read, and invoke that does not name a provider is trapped
before being returned by the source iterator. If no default provider
for the appropriate package has been registered, the eval source
synthesizes an appropriate registration, waits for it to complete, and
records the registered provider's reference. This reference is injected
into the original request, which is then processed as usual. If a
default provider was already registered, the recorded reference is
used and no new registration occurs.
### SDK Changes
These changes only expose first-class providers from the Node.JS SDK.
- A new abstract class, `ProviderResource`, can be subclassed and used
to instantiate first-class providers.
- A new field in `ResourceOptions`, `provider`, can be used to supply
a particular provider instance to manage a `CustomResource`'s CRUD
operations.
- A new type, `InvokeOptions`, can be used to specify options that
control the behavior of a call to `pulumi.runtime.invoke`. This type
includes a `provider` field that is analogous to
`ResourceOptions.provider`.
When this argument is not provided, we'll default to the value of
pulumi.getProject(). This is what you want for application level code
anyway and it matches the CLI behavior where if you don't qualify a
key with a package we use the name of the current project.
Fixes#1581
The pulumi runtime used to lazily load and parse both config and
settings data set by the language host. The initial reason for this
design was that we wanted the runtime to be usable in a normal node
environment, but we have moved away from supporting that.
In addition, while we claimed we loaded these value "lazily", we
actually forced their loading quite eagerly when we started
up. However, when capturing config (or settings, as we now do), we
would capture all the logic about loading these values from the
environment.
Even worse, in the case where you had two copies of @pulumi/pulumi
loaded, it would be possible to capture a config object which was not
initialized and then at runtime the initialization logic would try to
read PULUMI_CONFIG from the process environment and fail.
So we adopt a new model where configuration and settings are parsed as
we load their containing modules. In addition, to support SxS
scinerios, we continue to use `process.env` as a way to control both
configuration and settings. This means that `run.ts` must now ensure
that these values are present in the environment before either the
config or runtime modules have been loaded.
Set the following compiler defaults:
```
"target": "es6",
"module": "commonjs",
"moduleResolution": "node",
"sourceMap": true,
```
Which allows us to not even include a tsconfig.json file. If one is
present, `ts-node` will use its options, but the above settings will
override any settings in a local tsconfig.json file. This means if you
want full control over the target, you'll need to go back to the raw
tsc workflow where you explicitly build ahead of time.
This change lets us set runtime specific options in Pulumi.yaml, which
will flow as arguments to the language hosts. We then teach the nodejs
host that when the `typescript` is set to `true` that it should load
ts-node before calling into user code. This allows using typescript
natively without an explicit compile step outside of Pulumi.
This works even when a tsconfig.json file is not present in the
application and should provide a nicer inner loop for folks writing
typescript (I'm pretty sure everyone has run into the "but I fixed
that bug! Why isn't it getting picked up? Oh, I forgot to run tsc"
problem.
Fixes#958
* Protobuf changes to record dependencies for read resources
* Add a number of tests for read resources, especially around replacement
* Place read resources in the snapshot with "external" bit set
Fixespulumi/pulumi#1521. This commit introduces two new step ops: Read
and ReadReplacement. The engine generates Read and ReadReplacement steps
when servicing ReadResource RPC calls from the language host.
* Fix an omission of OpReadReplace from the step list
* Rebase against master
* Transition to use V2 Resources by default
* Add a semantic "relinquish" operation to the engine
If the engine observes that a resource is read and also that the
resource exists in the snapshot as a non-external resource, it will not
delete the resource if the IDs of the old and new resources match.
* Typo fix
* CR: add missing comments, DeserializeDeployment -> DeserializeDeploymentV2, ID check
When a resource fails to initialize (i.e., it is successfully created,
but fails to transition to a fully-initialized state), and a user
subsequently runs `pulumi update` without changing that resource, our
CLI will fail to warn the user that this resource is not initialized.
This commit begins the process of allowing our CLI to report this by
storing a list of initialization errors in the checkpoint.
* Revert "Parallelize much more of resource creation in the JS language provider SDK (#1618)"
This reverts commit 4edd244a26.
* Revert "Process our async-work-queue in parallel. (#1619)"
This reverts commit b8c1cb9574.
This commit adds CLI support for resource providers to provide partial
state upon failure. For resource providers that model resource
operations across multiple API calls, the Provider RPC interface can now
accomodate saving bags of state for resource operations that failed.
This is a common pattern for Terraform-backed providers that try to do
post-creation steps on resource as part of Create or Update resource
operations.
A critical part of the partial update protocol is to return a structured
error when a resource is successfully created, but fails to initialize.
This structured error contains the properties of the
partially-initialized resource, and instructs the engine to halt.
Most languages implement this by attaching "details" to the error, i.e.,
an arbitrary proto message attached to the error. The JavaScript
implementation is not mature enough to include all the facilities
required to use this, so here we must add a `Status` message, which
protobuf requires as part of its structure for returning details.
We retained these modules to support using v0.11.X and earlier
versions of @pulumi/pulumi, which required a native module to do
closure serialization. 0.12.X does not need this, so lets stop
including it.
Closure serialization now keeps track of the `require`d packages it sees in the function bodies that are serialized during a call to `serializeFunction`.
Also, replaces `serializeFunctionAsync` with `serializeFunction` which accepts richer parameters and return type, deprecating the former API (but leaving it available for now to avoid a breaking change).
Before the changes in #1414, all output properties were guaranteed to
have values after deserialization. After #1414, any properties with no
value were no longer resolved, which was treated as an error. These
changes resolve all missing proprties to `undefined`. If a property is
missing during an update, its `undefined` value is marked as known.
These changes add support for distinguishing an output property with
an unknown value from an output property with a known value that is
undefined.
In a broad sense, the Pulumi property type system is just JSON with the
addition of unknown values. Notably absent, however, are undefined
values. As it stands, our marshalers between JavaScript and Pulumi
property values treat all undefined JavaScript values as unknown Pulumi
values. Unfortunately, this conflates two very different concepts:
unknown Pulumi values are intended to represent values of output
properties that are unknown at time of preview, _not_ values that are
known but undefined. This results in difficulty reasoning about when
transforms are run on output properties as well as confusing output in
the `diff` view of Pulumi preview (user-specifed undefined values are
rendered as unknown values).
As it turns out, we already have a way to decide whether or not an
Output value is known or not: Output.performApply. These changes rename
this property to `isKnown`, clarify its meaning, and take advantage of
the result to decide whether or not an Output value should marshal as
an unknown Pulumi value.
This also allowed these changes to improve the serialization of
undefined object keys and array elements s.t. we better match JavaScript
to JSON serialization behavior (undefined object keys are omitted;
undefined array elements are marshaled as `null`).
Fixes https://github.com/pulumi/pulumi-cloud/issues/483.
* Delete Before Create
This commit implements the full semantics of delete before create. If a
resource is replaced and requires deletion before creation, the engine
will use the dependency graph saved in the snapshot to delete all
resources that depend on the resource being replaced prior to the
deletion of the resource to be replaced.
* Rebase against master
* CR: Simplify the control flow in makeRegisterResourceSteps
* Run Check on new inputs when re-creating a resource
* Fix an issue where the planner emitted benign but incorrect deletes of DBR-deleted resources
* CR: produce the list of dependent resources in dependency order and iterate over the list in reverse
* CR: deps->dependents, fix an issue with DependingOn where duplicate nodes could be added to the dependent set
* CR: Fix an issue where we were considering old defaults and new inputs
inappropriately when re-creating a deleted resource
* CR: save 'iter.deletes[urn]' as a local, iterate starting at cursorIndex + 1 for dependency graph
This changes two primary things about dynamic providers:
1) Always echo back the __provider upon read, even if there is a
missing read function on the dynamic provider. In fact, return
the full input state in that case.
2) Store the __provider in the output state of the dynamic resource,
in addition to the input state. My recollection of the "model"
discussion we had weeks ago was that the output properties are
mean to capture the state of a resource in its entirety; not having
this meant that refresh would marshal the outputs only, and find
on the other side of the RPC boundary that __provider was missing.
Note that an alternative to the latter fix would be to use some hybrid
of input and output state, as we used to do, by merging property maps.
* Fix a bug in promise leak detection that leaked promises when errors occur
* Add an opt-in to the super-verbose debug error message on promise leaks
* Fix a bad merge
* was/were grammar improvement in error message
* Fail the deployment if a debuggable promise leaks
* Graceful RPC shutdown: CLI side
* Handle unavailable resource monitor in language hosts
* Fix a comment
* Don't commit package-lock.json
* fix mangled pylint pragma
* Rebase against master and fix Gopkg.lock
* Code review feedback
* Fix a race between closing the callerEventsOpt channel and terminating a goroutine that writes to it
* glog -> logging
These changes enable tracing of Pulumi API calls.
The span with which to associate an API call is passed via a
`context.Context` parameter. This required plumbing a
`context.Context` parameter through a rather large number of APIs,
especially in the backend.
In general, all API calls are associated with a new root span that
exists for essentially the entire lifetime of an invocation of the
Pulumi CLI. There were a few places where the plumbing got a bit hairier
than I was willing to address with these changes; I've used
`context.Background()` in these instances. API calls that receive this
context will create new root spans, but will still be traced.
Instead of using a shell script to jump from the language host into
node, just invoke node directly. This makes our start-up path a little
simpler to understand and indirectly fixespulumi/home#156, where we
would fail on Windows if the `-exec` script was in a folder that had
spaces in it (due to a subtle interaction between how go launches cmd
files and how cmd.exe parses arguments).
Rather than filtering out the `id` and `urn` properties when serializing
the inputs to an invoke, pass these properties along. This enables the
use of invoke endpoints that accepts these as inputs (e.g. the endpoint
that backs `aws.ec2.getSubnet`).
* Improve the error message when npm/yarn install hasn't been run
* Same thing, but for Python
* Use PULUMI_RUN in batch script
* Use -e, -f doesn't work for symlinked paths (e.g. yarn link)
This change adopts `x is T` style of RTTI inquiry, which fits much
more nicely with TypeScript's typechecking flow.
Thanks to @lukehoban for teaching me a new trick today! :-)
This change moves us away from using JavaScript RTTI, by way of
`instanceof`, for built-in Pulumi types. If we use `instanceof`,
then the same logical type loaded from separate copies of the
SDK package -- as will happen in SxS scenarios -- are considered
different. This isn't actually what we want. The solution is
simple: implement our own quasi-RTTI solution, using __pulumi*
properties and manual as* and is* functions. Note that we could
have skipped the as* and is* functions, but I found that they led
to slightly easier to read code.
There is one strange thing in here, which I spoke to
@CyrusNajmabadi about: SerializedOutput<T>, because it implements
Output<T> as an _interface_, did not previously masquerade as an
actual Output<T>. In other words, `instanceof` would have returned
false, and indeed a few important properties (like promise) are
missing. This change preserves that behavior, although I'll admit
that this is slightly odd. I suspect we'll want to revisit this as
part of https://github.com/pulumi/pulumi/issues/1074.
Fixes https://github.com/pulumi/pulumi/issues/1203.
* Introduce a simple repetition operator to match expected error messages against actual ones
* Convert required and optional objects to use a Map (node v9 compat), improve the error formatting for failed tests
* Test node v6, v8, and v9 in CI
* Get rid of PULUMI_API env in .travis.yml, it's set from the Travis console now
While we no longer use the native runtime module, older versions of
@pulumi/pulumi still require it. Let's continue to have the launcher
put the native module location on the `$PATH`. And we'll include them
in the SDK for a while longer.
Fixes#1177
The RPC provider interface needs a way to convey back to the engine
that a resource being read no longer exists. To do this, we'll return
the ID property that was read back. If it is empty, it means the
resource is gone. If it is non-empty, we expect it to match the input.
* Implement closure scope chain analysis in pure TypeScript
This change makes use of four V8 intrinsics to avoid having to use a
native module to inspect the scope chains of live Function objects. This
unfortunately leads to the limitation of not allowing captures of 'this'
in arrow functions, but that is something we are willing to live with
for now.
* Remove native module build and restore from the Makefile
* CR feedback: Be a little more efficient when scanning the scope chain
* Nuke everything related to custom Node versions and the native Node module
* CR feedback: rename native.ts -> v8.ts, document some interfaces in v8.ts
As I began to write code using the ability to perform resource
lookups, especially in the code-generators, I realized the way it
was surfaced as an argument to the Resource base constructor would
lead to overload explosion. Instead of doing that, let's pass it
in the ResourceOptions bag.
Prior to this change, if you ended up with multiple Pulumi SDK
packages loaded side-by-side, we would fail in obscure ways. The
reason for this was that we initialize and store important state
in static variables. In the case that you load the same library
twice, however, you end up with separate copies of said statics,
which means we would be missing engine RPC addresses and so on.
This change adds the ability to recover from this situation by
mirroring the initialized state in process-wide environment
variables. By doing this, we can safely recover simply by reading
them back when we detect that they are missing. I think we can
eventually go even further here, and eliminate the entry point
launcher shim altogether by simply having the engine launch the
Node program with the right environment variables. This would
be a nice simplification to the system (fewer moving pieces).
There is still a risk that the separate copy is incompatible.
Presumably the reason for loading multiple copies is that the
NPM/Yarn version solver couldn't resolve to a shared version.
This may yield obscure failure modes should RPC interfaces change.
Figuring out what to do here is part of pulumi/pulumi#957.
This fixespulumi/pulumi#777 and pulumi/pulumi#1017.
This change skips unknown IDs during read operations. This can happen
when a read is performed using the output property of another resource
during planning. This is intentionally supported via ID being an
Input<ID> and all we need to do for this to work correctly is skip the
actual provider RPC and the runtime will propagate unknown outputs as
usual.
This change wires up the new Read RPC method in such a manner that
Pulumi programs can invoke it. This is technically not required for
refreshing state programmatically (as in pulumi/pulumi#1081), however
it's a feature we had eons ago and have wanted since (see
pulumi/pulumi#83), and will allow us to write code like
let vm = aws.ec2.Instance.get("my-vm", "i-07043cd97bd2c9cfc");
// use any property from here on out ...
The way this works is simply by bridging the Pulumi program via its
existing RPC connection to the engine, much like Invoke and
RegisterResource RPC requests already do, and then invoking the proper
resource provider in order to read the state. Note that some resources
cannot be uniquely identified by their ID alone, and so an extra
resource state bag may be provided with just those properties required.
This came almost for free (okay, not exactly) and will come in handy as
we start gaining experience with reading live state from resources.
This commit changes two things about our resource model:
* Stop performing Pulumi Engine-side diffing of resource state.
Instead, we defer to the resource plugins themselves to determine
whether a change was made and, if so, the extent of it. This
manifests as a simple change to the Diff function; it is done in
a backwards compatible way so that we continue with legacy diffing
for existing resource provider plugins.
* Add a Read RPC method for resource providers. It simply takes a
resource's ID and URN, plus an optional bag of further qualifying
state, and it returns the current property state as read back from
the actual live environment. Note that the optional bag of state
must at least include enough additional properties for resources
wherein the ID is insufficient for the provider to perform a lookup.
It may, however, include the full bag of prior state, for instance
in the case of a refresh operation.
This is part of pulumi/pulumi#1108.
* Improve the error message arising from missing required configs for
resource providers
If the resource provider that we are speaking to is new enough, it will send
across a list of keys and their descriptions alongside an error
indicating that the provider we are configuring is missing required
config. This commit packages up the list of missing keys into an error
that can be presented nicely to the user.
* Code review feedback: renaming simplification and correcting errors in comments
* Send structured errors across RPC boundaries
This brings us closer to gRPC best practices where we send structured
errors with error codes across RPC endpoints. The new "rpcerrors"
package can wrap errors from RPC endpoints, so RPC servers can attach
some additional context as to why a request failed.
* Code review feedback:
1. Rename rpcerrors -> rpcerror, better package name
2. Rename RPCError -> Error, RPCErrorCause -> ErrorCause, names
suggested by gometalinter to improve their package-qualified names
3. Fix import organization in rpcerror.go
Our previous strategy of just using `git describe --tags --dirty` to
compute a version caused issues. The major one was that since version
sort lexigrapically, git's strategy of having a commit count without
leading zeros lead to cases where 0.11.0-dev-9 was "newer than"
0.11.0-dev-10 which is not what you want at all.
With this change, we compute a version by first seeing if the commit
is tagged, and if so, we use that tag. Otherwise, we take the closest
tag and to it append the unix timestamp of the commit and then append
a git hash.
Because we use the commit timestamp, things will sort correctly again.
Part of pulumi/home#174
We need to support the current version of the nodejs language host
running programs that use older version of @pulumi/pulumi where the
runtime expected config keys to look like
`<package>:config:<name>`. In the language host we actually did the
transformation from the new format to the old one, for compatability
reasons but we then droped the transfomed value on the floor.
The four concerns are:
parsing a v8 function string so we can figure out captured variables.
walkgin the function/object graph producing the graph we will serialize.
rewriting constructors and methods so that 'super' works.
serializing graph to text.
Also, rename/cleanup a bunch of serialization code.
Also, generate better environment names in the serialized closure code. Thsi code should be much easier to make sense of as hte names will better track to the original names in the user code.
Also, dedupe simple non-capturing functions. This helps ensure we don't spit out N copies of __awaiter (one per file it is declared in).
This special error kind should be used by all Pulumi components as the error type for user input validation errors. Although it can already be referenced via `@pulumi/pulumi/errors`, also explicitly export it directly on `@pulumi/pulumi`.
This API was introduced to aid the refactoring, but it isn't something
we want to support long term. Remove it and for a few places, push
passing config.Key around more, instead of converting to the old type
eagerly.
We now unify new Config("package") and new Config("package:config"),
printing a warning when the new Config("package:config") form is
used and pointing consumers towards just new Config("package")
I've updated our examples to use the newer syntax, but I've added a
test ot the langhost to ensure both forms work.
Fixes#923
Make many fixes to closure serialization
Primary things that i've done as part of this change:
Added support for cyclic objects.
Properly serialize objects that are shared across different function. previously you would get multiple copies, now you properly reference the same copy.
Remove the usages of 'hashes' for functions. Because we track identity of objects, we no longer need them.
Serialize properties of functions (if they have any).
Handle Objects/Functions with different __proto__s than normal. i.e. classes/constructors. but also anything the user may have done themselves to the object.
Handle generator functions.
Handle functions with 'computed' names.
Handle functions with 'symbol' names.
Handle serializing Promises as Promises.
Removed the dual Closure/AsyncClosure tree. One existed solely so we could have a tree without promises (for use in testing maybe?). Because this all exists in a part of our codebase that is entirely async, it's fine to have promises in the tree, and to await them when serializing the Closure to a string.
Handle serializing class-constructors and methods. Including properly handling 'super' calls.
* Produce better error messages when the main module is not found
If we fail to load a program's main module, inspect the program's
package.json and attempt to diagnose why the main module load failed.
* Code review feedback: entrypoint -> entry point, call out npm build explicitly, simplify control flow
* Code review feedback: add a little more levity to the unknown exception error message
This change includes a lot more functionality. Enough to actually
run the webserver-py example through previews, updates, and destroys!
* Actually wire up the gRPC connections to the engine/monitor.
* Move the Node.js and Python generated Protobuf/gRPC files underneath
the actual SDK directories to simplify this generally. No more
copying during `make` and, in fact, this was required to give a smoother
experience with good packages/modules for the Python's SDK development.
* Build the Python egg during `make build`.
* Add support for program stacks. Just like with the Node.js runtime,
we will auto-parent any resources without explicit parents to a single
top-level resource component.
* Add support for component resource output properties.
* Add get_project() and get_stack() functions for retrieving the current
project and stack names.
* Properly use UNKNOWN sentinels.
* Add a set_outputs() function on Resource. This is defined by the
code-generator and allows custom logic for output property setting.
This is cleaner than the way we do this in Node.js, and gives us a
way to ensure that output properties are "real" properties, complete
with member documentation. This also gives us a hook to perform
name demangling, which the code-generator typically controls anyway.
* Add package dependencies to setuptools.py and requirements.txt.
1. Various idiomatic Go and TypeScript fixes
2. Add an integration test that end-to-end roundtrips dependency
information for a simple Pulumi program
3. Add an additional test assert that tests that dependency information
comes from the language host as expected
This commit does two things:
1. All dependencies of a resource, both implicit and explicit, are
communicated directly to the engine when registering a resource. The
engine keeps track of these dependencies and ultimately serializes
them out to the checkpoint file upon successful deployment.
2. Once a successful deployment is done, the new `pulumi stack
graph` command reads the checkpoint file and outputs the dependency
information within in the DOT format.
Keeping track of dependency information within the checkpoint file is
desirable for a number of reasons, most notably delete-before-create,
where we want to delete resources before we have created their
replacement when performing an update.
This change refactors the way we do ${VERSION} substitution in both
the Node.js SDK's version.js and package.json, so that it can work on
Windows. This is required now that we are actually parsing semvers.
On Windows, when we launch the language host, it will end up with
a ".exe" file extension at the end of os.Args[0]. This leads us to
produce a garbage filename for the -exec script -- namely,
pulumi-language-nodejs.exe-exec -- which, of course fails. We simply
need to trim off the ".exe" bit before producing the script name.
We have had a long-standing bug in here where we waiting on a
stdout channel that never got populated, when the language plugin
fails to load entirely. This would lead to hung processes. The
fix is simple: only wait for stdout/stderr channels to drain that
have actually been wired up to enjoy the requisite signaling.
This adds support for two things:
* Installing all plugins that a project requires with a single command:
$ pulumi plugin install
* Listing the plugins that this project requires:
$ pulumi plugin ls --project
$ pulumi plugin ls -p
This brings back the Node.js language plugin's GetRequiredPlugins
function, reimplemented in Go now that the language host has been
rewritten from JavaScript. Fairly rote translation, along with
some random fixes required to get tests passing again.
This change implements the Node.js language host's GetRequiredPlugins
function. This merely scans all node_modules/*/package.json files in
the program directory, looking for those that have associated plugins.
It returns a list of any found along with their version numbers.
This change adds a GetRequiredPlugins RPC method to the language
host, enabling us to query it for its list of plugin requirements.
This is language-specific because it requires looking at the set
of dependencies (e.g., package.json files).
It also adds a call up front during any update/preview operation
to compute the set of plugins and require that they are present.
These plugins are populated in the cache and will be used for all
subsequent plugin-related operations during the engine's activity.
We now cache the language plugins, so that we may load them
eagerly too, which we never did previously due to the fact that
we needed to pass the monitor address at load time. This was a
bit bizarre anyhow, since it's really the Run RPC function that
needs this information. So, to enable caching and eager loading
-- which we need in order to invoke GetRequiredPlugins -- the
"phone home" monitor RPC address is passed at Run time.
In a subsequent change, we will switch to faulting in the plugins
that are missing -- rather than erroring -- in addition to
supporting the `pulumi plugin install` CLI command.
This change introduces a workspace.GetPluginPath function that probes
the central workspace cache of plugins for a matching plugin binary that
matches the desired kind, name, and, optionally, version. It also permits
overriding this with $PATH for developer scenarios.
The analyzer, language, and resource plugin logic now uses this function
for deciding which binary path to load at runtime.
Fixes#356. Instead of downloading a node binary with our closure
serialization code linked-in, this PR instead publishes the
`nativeruntime.node` produced by the NodeJS SDK build as part of the SDK.
This has a number of advantages. First, it is vastly more easy to
develop closure.cc in this configuration. Second, we have the ability
to ship different `nativeruntime.node`s side-by-side, paving the way
for enabling future versions of Node. Third, we don't have to stay
in the business of shipping custom builds of Node, although we do still
need to ship a version of Node with minor modifications in order for
Windows to still work.
The windows build was still on the old plan from way back when where
we had binaries littered in the build tree and you had to add parts of
your build-tree to the `%PATH%` for the integration tests to work.
This cleans that up and moves all of our scripts that invoke
javascript to be on the same plan. They invoke our specially named
node with a relative path to the JS code we want to run.
We no longer have a node_modules folder in the SDK (since all
packages now come from NPM) so we need to adjust the shell script we
use to launch our runner to use a relative path.
This improves the failure messages in two circumstances:
1) If the resource monitor RPC connection is missing. This can happen
two ways: either you run a Pulumi program using vanilla Node.js, instead
of the CLI, or you've accidentally loaded the Pulumi SDK more than once.
2) Failure to load the custom Pulumi SDK Node.js extension. This is a new
addition and would happen if you tried running a Pulumi program using a
vanilla Node.js, rather than using the Pulumi CLI.
