Rather than placing these combinators directly on the Output class,
which feels odd because they are special purpose to iterables, and deal
with not only Outputs but also Inputs, we will place them on a
separate and dedicated iterable module for these utility helpers.
This change adds some new constructors for output properties:
1) We alias `Output.create` to `output`, more like Promise's various
construction methods. This reads better and is more discoverable.
2) A new `Output.createMap` function will accept an array of inputs,
along with a selector function for key/value pairs, and produces
an output map with said keys and values inside of it.
3) A new `Output.createGroupByMap` functon will similarly accept an
array of inputs and a key/value selector, however it creates an
output map with said keys, but where values are arrays of values,
and all duplicate keys will lead to appending to said arrays.
Tests to come in a subsequent checkin.
This commit introduces a 'next' package which we can use as a staging
ground for incrementally adopting new Python 3 code. The next package is
initially populated with the non-runtime portions of the Python SDK,
which is enough to pass all tests when running on Python 3. Future
commits will reach further into the runtime.
* Fail closure serialization in Node 11
Node 11 changed many of the intrinsics that we depend upon for closure
serialization, so until we fix the underlying issues this commit lazily
fails if a closure is serialized when running on Node 11.
* CR feedback
Some providers (namely Kubernetes) require unbounded parallelism in
order to function correctly. This commit enables the engine to operate
in a mode with unbounded parallelism and switches to that mode by
default.
Suppose you have `pulumi.output(o).apply(foo)`, with `o` being type `O`
and `foo` taking type `O` as an argument. If `O` is a type with methods,
this will fail to type check.
The reason is that `UnwrappedObject<T>` (as well as the other
`Unwrapped*` types) will recursively wrap the types of field values
whose type was `Function`. Since `UnwrappedObject<Function>` is not the
same as `Function`, we fail to type check. Note that this does not
result in an actual "boxed" object -- this is purely at the type level.
This commit resolves this by considering `Function` a primitive type,
which will cause us to not wrap the types of field values, instead
leaving them as `Function`.
With this commit, the functions in 'pulumi.log' can be used to send
diagnostic messages to the Pulumi CLI. The Pulumi SDK bootstrap script
now also uses this feature to send diagnostic information on unhandled
exceptions to the Pulumi CLI.
Instead of looping forever, due to some recent improvements in engine
error handling it's sufficient for a language host to exit cleanly with
a zero exit code when the resource monitor is shutting down.
If you forget to implement check on a dynamic provider, all your
inputs mysteriously disappear. It's doubly odd because many providers
don't need to perform any checking on transformation of their inputs.
This change simply propagates the new inputs as-is by default when
a user-supplied check method isn't provided. This would have saved
me 20 minutes just now ... :-)
This introduces a Dockerfile for the Pulumi CLI. This makes it
easier to develop and test the engine in a self-contained environment,
in addition to being suitable for running the actual CLI itself.
For instance,
$ docker run pulumi/pulumi -e "PULUMI_ACCESS_TOKEN=x" up
will run the Pulumi program mounted under the /app volume. This will
be used in some upcoming CI/CD scenarios.
This uses multi-stage builds, and Debian Stretch as the base, for
relatively fast and lean build times and resulting images. We are
intentional about restoring dep packages independent of the actual
source code so that we don't end up needlessly re-depping, which can
consume quite a bit of time. After fixing
https://github.com/pulumi/pulumi/issues/1986, we should explore an
Alpine base image option.
I made the decision to keep this image scoped to just the Go builds.
Therefore, none of the actual SDK packages themselves are built, just
the engine, CLI, and language plugins for Node.js, Python, and Go.
It's possible to create a mega-container that has all of these full
environments so that we can rebuild them too, but for now I figured
it was better to rely on package management for them.
Another alternative would have been to install released binaries,
rather than building them. To keep the useful flow for development,
however, I decided to go the build route for now. If we build at the
same hashes, the resulting binaries "should" be ~identical anyhow.
I've created a pulumi/pulumi Docker Hub repo that we can publish this
into. For now, there is no CI publishing of the image.
This fixespulumi/pulumi#1991.
* Revert RunError behavior. Introduce new ResourceError for errors associated with a resource.
* Fix docs.
* Use resource error.
* Use ResourceError in more places.
* Use ResourceError in a few more places.
* Throw a resource error.
* Make required.
* Revert this.
* Lint.
* Only report errors once.
* Better comment.
* Search for Go project executables in more places than just $PATH
This searches the following in preferred order:
1. Local directory
2. $GOPATH/bin
3. In $PATH
* Check if program is not a directory before executing
* Protobuf changes
* Move management of root resource state to engine
This commit fixes a persistent side-by-side issue in the NodeJS SDK by
moving the management of root resource state to the engine. Doing so
adds two new endpoints to the Engine gRPC service: 1) GetRootResource
and 2) SetRootResource, which get and set the root resource
respectively.
* Rebase against master, regenerate proto
* Use nightly protoc gRPC plugin for Node
Newer versions of the Node gRPC plugin accept the 'minimum_node_version'
flag, which we can use to instruct protoc to not support Node versions
earlier than Node 6. This allows the compiler to use 'Buffer.from'
instead of the deprecated 'Buffer' constructor, which fixes a
deprecation warning on Node 10.
* Protobuf changes
- Do not require replacement of dynamic resources due to provider
changes. This is not necessary, and is almost certainly the wrong
thing to do if the dynamic provider is managing a physical resource.
- Return all inputs by default from a dynamic provider's check method.
Currently a dynamic provider that does not implement check will end up
receiving no inputs. This is confusing, and is not the correct default.
* Combine two gRPC servers into one for testing
For some reason, our current gRPC test setup has become flaky now that
we are spinning up two gRPC servers. Hopefully merging them into one
helps clarify what's going on.
* Add back error logging for CI
* Validate type tokens before using them
When registering or reading a resource, we take the type token given to
us from the language host and assume that it's valid, which resulted in
assertion failures in various places in the engine. This commit
validates the format of type tokens given to us from the language host
and issues an appropriate error if it's not valid.
Along the way, this commit also improves the way that fatal exceptions
are rendered in the Node language host.
* Pre-allocate an exception for ReadResource
* Fix integration test
* CR Feedback
This commit is a lower-impact change that fixes the bugs associated with
invalid types on component resources and only checks that a type is
valid on custom resources.
* CR Take 2: Fix up IsProviderType instead of fixing call sites
* Please gometalinter
Instead of trying to probe for the normal path ourselves, just use
node's `require.resolve` statement to find `@pulumi/pulumi/cmd/run`.
This allows run to be found in cases where either yarn workspaces are
used, or the module has been installed globally.
Part of #1868
`opts.providers` is currently only read by the `Resource` constructor if
either `opts.parent` or `getRootResource` is not `undefined`. In
scnearios where exactly one copy of `@pulumi/pulumi` is loaded, one of
these conditions will always be true. In SxS scenarios, however, it is
possible for neither of these conditions to be true, and the created
resource will end up without a `providers` map. These changes fix that
by always copying the contents of `opts.providers` if it is defined.
Node calls 'exit' event callbacks when a process is preparing to exit,
via process.exit or otherwise, but it does not execute the next callback
in the chain if a callback calls process.exit.
* Use Promise.resolve.
* Use `Inputs | Promise<Inputs> | Output<Inputs>` rather than
`Input<Inputs>`, which looks supremely bizarre.
* Update ComponentResource.registerOutputs also.
This change partly addresses pulumi/pulumi#1611, by permitting you
to export a promise at the top-level, and have it be recognized as
a stack output. In other words, you can now say things like
async function main() {
...
return {
a: "x",
...,
z: 42,
};
}
module.exports = main();
and your Pulumi program will record distinct outputs as you'd hope:
---outputs:---
a: "x"
...
z: 42
This is arguably just a bug in the way we implemented stack outputs.
The remainder of the requests in #1611 will remain open for future
design and discussion, as they have more subtle ramifications.
If a `tsconfig.json` file is not present at the root of the Pulumi
project, ts-node will look up the directory tree to see if there is
one. If there is, it will treat that as the root of the project. While
reasonable for some cases, this isn't the behavior we want for our use
of ts-node. We actually set compiler options such that in the common
case you don't even need a `tsconfig.json` and for pure JavaScript
projects, there wouldn't be a `tsconfig.json` file.
In both of these cases, there's a big foot-gun waiting. For example in
pulumi/pulumi#1772 we ran into a case where there was a tsconfig.json
file in $HOME, causing the entirety of $HOME to be analyzed by
TypeScript which made it look like Pulumi hung.
To address this, tell ts-node to not use a project in cases where
there is not a `tsconfig.json` at the root of the project.
Fixes#1772
This commit will introduce a field, `IsStatus` to `LogRequest`. A
"status" logging event will be displayed in the `Info` column of the
main display, but will not be printed out at the end, when resource
operations complete.
For example, for complex resource initialization, we'd like to display a
series of intermediate results: `[1/4] Service object created`, for
example. We'd like these to appear in the `Info` column, but not at the
end, where they are not helpful to the user.
This change adopts the code review suggestion to use a bag of options
for config constraints rather than having overloaded function names.
This is a much cleaner approach, lets us use more descriptive names,
and is far more future proof in case we decide to add more capabilities.
Everytime I convert a CloudFormation template to Pulumi, I inevitably
run into the fact that CloudFormation has advanced "schema" capabilities
for input variables, like min/max for numbers and string lengths, enums,
and regex pattern matching. This is always cumbersome to convert.
In this change, I've added a number of config helpers for these cases:
For string enums:
getEnum(key: string, values: string[]): string | undefined;
requireEnum(key: string: values: string[]): string;
For min/max strlen:
getMinMax(key: string, min: number, max: number): string | undefined;
requireMinMax(key: string, min: number, max: number): string;
For regex patterns:
getPattern(key: string, regexp: string | RegExp): string | undefined;
requirePattern(key: string, regexp: string | RegExp): string;
For min/max strlen _and_ regex patterns:
getMinMaxPattern(key: string, min: number, max: number,
regexp: string | RegExp): string | undefined;
requireMinMaxPattern(key: string, min: number, max: number,
regexp: string | RegExp): string;
For min/max numbers:
getNumberMinMax(key: string, min: number, max: number): number | undefined;
requireNumberMinMax(key: string, min: number, max: number): number;
Each function throws a detailed RunError-derived exception type if the
configuration value doesn't meet the constraint.
This fixespulumi/pulumi#1671.
* Fix an issue with NodeJS host logging
Related to pulumi/pulumi#1694. This issue prevented the language host
from being aware that an engine (logging endpoint) was available and
thus no log messages were sent to the engine. By default, the language
host wrote them to standard out instead, which resulted in a pretty bad
error experience.
This commit fixes the PR and adds machinery to the NodeJS langhost tests
for testing the engine RPC endpoint. It is now possible to give a "log"
function to tests which will be hooked up to the "log" RPC endpoint
normally provided by the Pulumi engine.
* Remove accidental console.log
Replace the Source-based implementation of refresh with a phase that
runs as the first part of plan execution and rewrites the snapshot in-memory.
In order to fit neatly within the existing framework for resource operations,
these changes introduce a new kind of step, RefreshStep, to represent
refreshes. RefreshSteps operate similar to ReadSteps but do not imply that
the resource being read is not managed by Pulumi.
In addition to the refresh reimplementation, these changes incorporate those
from #1394 to run refresh in the integration test framework.
Fixes#1598.
Fixespulumi/pulumi-terraform#165.
Contributes to #1449.
The plugin host can ask the language host to provide a list of
resource plugins that it thinks will be nessecary for use at
deployment time, so they can be eagerly loaded.
In NodeJS (the only language host that implements this RPC) This works
by walking the directory tree rooted at the CWD of the project,
looking for package.json files, parsing them and seeing it they have
some marker property set. If they do, we add information about them
which we return at the end of our walk.
If there is *any* error, the entire operation fails. We've seen a
bunch of cases where this happens:
- Broken symlinks written by some editors as part of autosave.
- Access denied errors when part of the tree is unwalkable (Eric ran
into this on Windows when he had a Pulumi program at the root of his
file system.
- Recusive symlinks leading to errors when trying to walk down the
infinite chain. (See #1634 for one such example).
The very frustrating thing about this is that when you hit an error
its not clear what is going on and fixing it can be non-trivial. Even
worse, in the normal case, all of these plugins are already installed
and could be loaded by the host (in the common case, plugins are
installed as a post install step when you run `npm install`) so if we
simply didn't do this check at all, things would work great.
This change does two things:
1. It does not stop at the first error we hit when discovering
plugins, instead we just record the error and continue.
2. Does not fail the overall operation if there was an error. Instead,
we return to the host what we have, which may be an incomplete view
of the world. We glog the errors we did discover for diagnostics if
we ever need them.
I believe that long term most of this code gets deleted anyway. I
expect we will move to a model long term where the engine faults in
the plugin (downloading it if needed) when a request for the plugin
arrives. But for now, we shouldn't block normal operations just
because we couldn't answer a question with full fidelity.
Fixes#1478
If a resource's options bag does not specify `protect` or `provider`,
pull a default value from the resource's parent.
In order to allow a parent resource to specify providers for multiple
resource types, component resources now accept an optional map from
package name to provider instance. When a custom resource needs a
default provider from its parent, it checks its parent provider bag for
an entry under its package. If a component resource does not have a
provider bag, its pulls a default from its parent.
These changes also add a `parent` field to `InvokeOptions` s.t. calls to
invoke can use the same behavior as resource creation w.r.t. providers.
Fixes#1735, #1736.
* Added dist target for make, will help with Homebrew
* Try to install go dependencies before building
* Make sure dep ensure is called before trying to build SDKs
* Removed dep ensure from dist initial step
### First-Class Providers
These changes implement support for first-class providers. First-class
providers are provider plugins that are exposed as resources via the
Pulumi programming model so that they may be explicitly and multiply
instantiated. Each instance of a provider resource may be configured
differently, and configuration parameters may be source from the
outputs of other resources.
### Provider Plugin Changes
In order to accommodate the need to verify and diff provider
configuration and configure providers without complete configuration
information, these changes adjust the high-level provider plugin
interface. Two new methods for validating a provider's configuration
and diffing changes to the same have been added (`CheckConfig` and
`DiffConfig`, respectively), and the type of the configuration bag
accepted by `Configure` has been changed to a `PropertyMap`.
These changes have not yet been reflected in the provider plugin gRPC
interface. We will do this in a set of follow-up changes. Until then,
these methods are implemented by adapters:
- `CheckConfig` validates that all configuration parameters are string
or unknown properties. This is necessary because existing plugins
only accept string-typed configuration values.
- `DiffConfig` either returns "never replace" if all configuration
values are known or "must replace" if any configuration value is
unknown. The justification for this behavior is given
[here](https://github.com/pulumi/pulumi/pull/1695/files#diff-a6cd5c7f337665f5bb22e92ca5f07537R106)
- `Configure` converts the config bag to a legacy config map and
configures the provider plugin if all config values are known. If any
config value is unknown, the underlying plugin is not configured and
the provider may only perform `Check`, `Read`, and `Invoke`, all of
which return empty results. We justify this behavior becuase it is
only possible during a preview and provides the best experience we
can manage with the existing gRPC interface.
### Resource Model Changes
Providers are now exposed as resources that participate in a stack's
dependency graph. Like other resources, they are explicitly created,
may have multiple instances, and may have dependencies on other
resources. Providers are referred to using provider references, which
are a combination of the provider's URN and its ID. This design
addresses the need during a preview to refer to providers that have not
yet been physically created and therefore have no ID.
All custom resources that are not themselves providers must specify a
single provider via a provider reference. The named provider will be
used to manage that resource's CRUD operations. If a resource's
provider reference changes, the resource must be replaced. Though its
URN is not present in the resource's dependency list, the provider
should be treated as a dependency of the resource when topologically
sorting the dependency graph.
Finally, `Invoke` operations must now specify a provider to use for the
invocation via a provider reference.
### Engine Changes
First-class providers support requires a few changes to the engine:
- The engine must have some way to map from provider references to
provider plugins. It must be possible to add providers from a stack's
checkpoint to this map and to register new/updated providers during
the execution of a plan in response to CRUD operations on provider
resources.
- In order to support updating existing stacks using existing Pulumi
programs that may not explicitly instantiate providers, the engine
must be able to manage the "default" providers for each package
referenced by a checkpoint or Pulumi program. The configuration for
a "default" provider is taken from the stack's configuration data.
The former need is addressed by adding a provider registry type that is
responsible for managing all of the plugins required by a plan. In
addition to loading plugins froma checkpoint and providing the ability
to map from a provider reference to a provider plugin, this type serves
as the provider plugin for providers themselves (i.e. it is the
"provider provider").
The latter need is solved via two relatively self-contained changes to
plan setup and the eval source.
During plan setup, the old checkpoint is scanned for custom resources
that do not have a provider reference in order to compute the set of
packages that require a default provider. Once this set has been
computed, the required default provider definitions are conjured and
prepended to the checkpoint's resource list. Each resource that
requires a default provider is then updated to refer to the default
provider for its package.
While an eval source is running, each custom resource registration,
resource read, and invoke that does not name a provider is trapped
before being returned by the source iterator. If no default provider
for the appropriate package has been registered, the eval source
synthesizes an appropriate registration, waits for it to complete, and
records the registered provider's reference. This reference is injected
into the original request, which is then processed as usual. If a
default provider was already registered, the recorded reference is
used and no new registration occurs.
### SDK Changes
These changes only expose first-class providers from the Node.JS SDK.
- A new abstract class, `ProviderResource`, can be subclassed and used
to instantiate first-class providers.
- A new field in `ResourceOptions`, `provider`, can be used to supply
a particular provider instance to manage a `CustomResource`'s CRUD
operations.
- A new type, `InvokeOptions`, can be used to specify options that
control the behavior of a call to `pulumi.runtime.invoke`. This type
includes a `provider` field that is analogous to
`ResourceOptions.provider`.
When this argument is not provided, we'll default to the value of
pulumi.getProject(). This is what you want for application level code
anyway and it matches the CLI behavior where if you don't qualify a
key with a package we use the name of the current project.
Fixes#1581
The pulumi runtime used to lazily load and parse both config and
settings data set by the language host. The initial reason for this
design was that we wanted the runtime to be usable in a normal node
environment, but we have moved away from supporting that.
In addition, while we claimed we loaded these value "lazily", we
actually forced their loading quite eagerly when we started
up. However, when capturing config (or settings, as we now do), we
would capture all the logic about loading these values from the
environment.
Even worse, in the case where you had two copies of @pulumi/pulumi
loaded, it would be possible to capture a config object which was not
initialized and then at runtime the initialization logic would try to
read PULUMI_CONFIG from the process environment and fail.
So we adopt a new model where configuration and settings are parsed as
we load their containing modules. In addition, to support SxS
scinerios, we continue to use `process.env` as a way to control both
configuration and settings. This means that `run.ts` must now ensure
that these values are present in the environment before either the
config or runtime modules have been loaded.
Set the following compiler defaults:
```
"target": "es6",
"module": "commonjs",
"moduleResolution": "node",
"sourceMap": true,
```
Which allows us to not even include a tsconfig.json file. If one is
present, `ts-node` will use its options, but the above settings will
override any settings in a local tsconfig.json file. This means if you
want full control over the target, you'll need to go back to the raw
tsc workflow where you explicitly build ahead of time.
This change lets us set runtime specific options in Pulumi.yaml, which
will flow as arguments to the language hosts. We then teach the nodejs
host that when the `typescript` is set to `true` that it should load
ts-node before calling into user code. This allows using typescript
natively without an explicit compile step outside of Pulumi.
This works even when a tsconfig.json file is not present in the
application and should provide a nicer inner loop for folks writing
typescript (I'm pretty sure everyone has run into the "but I fixed
that bug! Why isn't it getting picked up? Oh, I forgot to run tsc"
problem.
Fixes#958
I tried to re-generate the Protobuf/gRPC files, but generate.sh seems to
have assumed I manually built the Dockerfile ahead-of-time (unless I'm
missing something -- very possible). Unfortunately, when I went to do
that, I ended up with a handful of errors, most of them due to
differences in versions. (We probably want to think about pinning them.)
To remedy both issues, I've fixed up the Dockerfile so that it works for
me at least, and added a build step to the front of generate.sh.
* Protobuf changes to record dependencies for read resources
* Add a number of tests for read resources, especially around replacement
* Place read resources in the snapshot with "external" bit set
Fixespulumi/pulumi#1521. This commit introduces two new step ops: Read
and ReadReplacement. The engine generates Read and ReadReplacement steps
when servicing ReadResource RPC calls from the language host.
* Fix an omission of OpReadReplace from the step list
* Rebase against master
* Transition to use V2 Resources by default
* Add a semantic "relinquish" operation to the engine
If the engine observes that a resource is read and also that the
resource exists in the snapshot as a non-external resource, it will not
delete the resource if the IDs of the old and new resources match.
* Typo fix
* CR: add missing comments, DeserializeDeployment -> DeserializeDeploymentV2, ID check
When a resource fails to initialize (i.e., it is successfully created,
but fails to transition to a fully-initialized state), and a user
subsequently runs `pulumi update` without changing that resource, our
CLI will fail to warn the user that this resource is not initialized.
This commit begins the process of allowing our CLI to report this by
storing a list of initialization errors in the checkpoint.
