* Fix some tracing issues.
- Add endpoints for `startUpdate` and `postEngineEventsBatch` so that
spans for these invocations have proper names
- Inject a tracing span when walking a plan so that resource operations
are properly parented
- When handling gRPC calls, inject a tracing span into the call's
metadata if no span is already present so that resource monitor and
engine spans are properly parented
- Do not trace client gRPC invocations of the empty method so that these
calls (which are used to determine server availability) do not muddy
the trace. Note that I tried parenting these spans appropriately, but
doing so broke the trace entirely.
With these changes, the only unparented span in a typical Pulumi
invocation is a single call to `getUser`. This span is unparented
because that call does not have a context available. Plumbing a context
into that particular call is surprisingly tricky, as it is often called
by other context-less functions.
* Make tracing support more flexible.
- Add support for writing trace data to a local file using Appdash
- Add support for viewing Appdash traces via the CLI
This package's flags conflict with those in google/glog. Replace all
references to this package with references to
pulumi/pulumi/pkg/util/logging, and change that package to explicitly
call `flag.CommandLine.Parse` with an empty slice.
This should make it much easier to consume these packages in downstream
repos that have direct or indirect dependencies on google/glog.
This command will cause `pulumi policy publish` to behave in much the
same way `pulumi up` does -- if the policy program is in TypeScript, we
will use ts-node to attempt to compile in-process before executing, and
fall back to plain-old node.
We accomplish this by moving `cmd/run/run.ts` into a generic helper
package, `runtime/run.ts`, which slightly generalizes the use cases
supported (notably, allowing us to exec some program outside of the
context of a Pulumi stack).
This new package is then called by both `cmd/run/index.ts` and
`cmd/run-policy-pack/index.ts`.
@keen99 pointed out that newer versions of golangci-lint were failing
due to some spelling errors. This change fixes them up. We have also
now have a work item to track moving to a newer golangci-lint tool in
the future.
Fixes#2841
In 3621c01f4b, we implemented
CheckConfig/DiffConfig incorrectly. We should have explicilty added
the handlers (to supress the warnings we were getting) but returned an
error saying the RPC was not implemented. Instead, we just returned
success but passed back bogus data. This was "fine" at the time
because nothing called these methods.
Now that we are actually calling them, returning incorrect values
leads to errors in grpc. To deal with this we do two things:
1. Adjust the implementations in the dynamic provider to correctly
return not implemented. This allows us to pick up the default engine
behavior going forward.
2. Add some code in CheckConfig/DiffConfig that handle the gRPC error
that is returned when calling methods on the dynamic provider and fall
back to the legacy behavior. This means updating your CLI will not
cause issues for existing resources where the SDK has not been
updated.
All existing implementations would fail if secret values were passed
to the dyanmic provider. When the provider says it does not support
secrets, the engine will do basic secrets tracking (any outputs with
the same names as secret inputs become secrets themselves).
In previous commits, we have changed the language plugin protocol to
allow the host to communicate that the plugin is meant to boot in "query
mode." In nodejs, this involves not doing things like registering the
default stack resource. This commit will implement this functionality.
When using a dynamic provider, the following messages are printed,
because we did not update the dynamic provider when we added these new
RPCs:
```
Method handler checkConfig for /pulumirpc.ResourceProvider/CheckConfig expected but not provided
Method handler diffConfig for /pulumirpc.ResourceProvider/DiffConfig expected but not provided
```
Given that we don't do anything for the `configure` RPC today, we can
also just no-op these RPCs as well.
This implements the new algorithm for deciding which resources must be
deleted due to a delete-before-replace operation.
We need to compute the set of resources that may be replaced by a
change to the resource under consideration. We do this by taking the
complete set of transitive dependents on the resource under
consideration and removing any resources that would not be replaced by
changes to their dependencies. We determine whether or not a resource
may be replaced by substituting unknowns for input properties that may
change due to deletion of the resources their value depends on and
calling the resource provider's Diff method.
This is perhaps clearer when described by example. Consider the
following dependency graph:
A
__|__
B C
| _|_
D E F
In this graph, all of B, C, D, E, and F transitively depend on A. It may
be the case, however, that changes to the specific properties of any of
those resources R that would occur if a resource on the path to A were
deleted and recreated may not cause R to be replaced. For example, the
edge from B to A may be a simple dependsOn edge such that a change to
B does not actually influence any of B's input properties. In that case,
neither B nor D would need to be deleted before A could be deleted.
In order to make the above algorithm a reality, the resource monitor
interface has been updated to include a map that associates an input
property key with the list of resources that input property depends on.
Older clients of the resource monitor will leave this map empty, in
which case all input properties will be treated as depending on all
dependencies of the resource. This is probably overly conservative, but
it is less conservative than what we currently implement, and is
certainly correct.
It is possible for the sub-process responsible for running a NodeJS
Pulumi program to exit with a success code before the user's program has
run if the process of loading the runtime generates an unhandled promise
rejection. These changes fix this by registering the unhandled exception
and rejection handlers that are responsible for ensuring a non-zero exit
code in these cases before any other action is taken.
Note that this issue is really only possible because the Node language
host (like the Python language host) is composed of two processes: one
that serves the language host gRPC service and one that loads and runs
the user's program. The former launches the latter in response to a call
to its `Run` gRPC endpoint. The lifetime of the user's program is
considered to be bounded by the lifetime of the `Run` invocation. The
NodeJS process maintains its own connection to the engine over which
resource registrations are communicated. It is tempting to add a message
to the resource monitor RPC interface that signals that no further
registrations are performed, but this is complicated due to the
three-party topology and the possibility that such an RPC may never be
sent (e.g. due to a crash or a downlevel version of the Pulumi Node
runtime).
Fixes#2316.
The langhost shares its standard out and standard error with the
language executor that it is used (python/nodejs), so we must be sure to
flush our stdout and stderr before reporting a Run failure to the
engine.
We run the same suite of changes that we did on gometalinter. This
ended up catching a few new issues, some of which were addressed and
some of which were baselined.
* Make v8 primitives async as there is no way to avoid async in node11.
* Simplify API.
* Move processing of well-known globals into the v8 layer.
We'll need this so that we can map from RemoteObjectIds back to these well known values.
* Remove unnecesssary check.
* Cleanup comments and extract helper.
* Introduce helper bridge method for the simple case of making an entry for a string.
* Make functions async. They'll need to be async once we move to the Inspector api.
* Make functions async. They'll need to be async once we move to the Inspector api.
* Make functions async. They'll need to be async once we move to the Inspector api.
* Move property access behind helpers so they can move to the Inspector API in the future.
* Only call function when we know we have a Function. Remove redundant null check.
* Properly serialize certain special JavaScript number values that JSON serialization cannot handle.
* Only marshall across the 'source' and 'flags' for a RegExp when serializing.
* Add a simple test to validate a regex without flags.
* Extract functionality into helper method.
* Add test with complex output scenarios.
* Output serialization needs to avoid recursively trying to serialize a serialized value.
* Introduce indirection for introspecting properties of an object.
* Use our own introspection API for examining an Array.
* Hide direct property access through API indirection.
* Produce values like the v8 Inspector does.
* Compute the module map asynchronously. Will need that when mapping mirrors instead.
* Cleanup a little code in closure creation.
* Get serialization working on Node11 (except function locations).
* Run tests in the same order on <v11 and >=v11
* Make tests run on multiple versions of node.
* Rename file to make PR simpler to review.
* Cleanup.
* Be more careful with global state.
* Remove commented line.
* Only allow getting a session when on Node11 or above.
* Promisify methods.
If you forget to implement check on a dynamic provider, all your
inputs mysteriously disappear. It's doubly odd because many providers
don't need to perform any checking on transformation of their inputs.
This change simply propagates the new inputs as-is by default when
a user-supplied check method isn't provided. This would have saved
me 20 minutes just now ... :-)
* Revert RunError behavior. Introduce new ResourceError for errors associated with a resource.
* Fix docs.
* Use resource error.
* Use ResourceError in more places.
* Use ResourceError in a few more places.
* Throw a resource error.
* Make required.
* Revert this.
* Lint.
* Only report errors once.
* Better comment.
- Do not require replacement of dynamic resources due to provider
changes. This is not necessary, and is almost certainly the wrong
thing to do if the dynamic provider is managing a physical resource.
- Return all inputs by default from a dynamic provider's check method.
Currently a dynamic provider that does not implement check will end up
receiving no inputs. This is confusing, and is not the correct default.
* Validate type tokens before using them
When registering or reading a resource, we take the type token given to
us from the language host and assume that it's valid, which resulted in
assertion failures in various places in the engine. This commit
validates the format of type tokens given to us from the language host
and issues an appropriate error if it's not valid.
Along the way, this commit also improves the way that fatal exceptions
are rendered in the Node language host.
* Pre-allocate an exception for ReadResource
* Fix integration test
* CR Feedback
This commit is a lower-impact change that fixes the bugs associated with
invalid types on component resources and only checks that a type is
valid on custom resources.
* CR Take 2: Fix up IsProviderType instead of fixing call sites
* Please gometalinter
Instead of trying to probe for the normal path ourselves, just use
node's `require.resolve` statement to find `@pulumi/pulumi/cmd/run`.
This allows run to be found in cases where either yarn workspaces are
used, or the module has been installed globally.
Part of #1868
Node calls 'exit' event callbacks when a process is preparing to exit,
via process.exit or otherwise, but it does not execute the next callback
in the chain if a callback calls process.exit.
If a `tsconfig.json` file is not present at the root of the Pulumi
project, ts-node will look up the directory tree to see if there is
one. If there is, it will treat that as the root of the project. While
reasonable for some cases, this isn't the behavior we want for our use
of ts-node. We actually set compiler options such that in the common
case you don't even need a `tsconfig.json` and for pure JavaScript
projects, there wouldn't be a `tsconfig.json` file.
In both of these cases, there's a big foot-gun waiting. For example in
pulumi/pulumi#1772 we ran into a case where there was a tsconfig.json
file in $HOME, causing the entirety of $HOME to be analyzed by
TypeScript which made it look like Pulumi hung.
To address this, tell ts-node to not use a project in cases where
there is not a `tsconfig.json` at the root of the project.
Fixes#1772
* Fix an issue with NodeJS host logging
Related to pulumi/pulumi#1694. This issue prevented the language host
from being aware that an engine (logging endpoint) was available and
thus no log messages were sent to the engine. By default, the language
host wrote them to standard out instead, which resulted in a pretty bad
error experience.
This commit fixes the PR and adds machinery to the NodeJS langhost tests
for testing the engine RPC endpoint. It is now possible to give a "log"
function to tests which will be hooked up to the "log" RPC endpoint
normally provided by the Pulumi engine.
* Remove accidental console.log
Replace the Source-based implementation of refresh with a phase that
runs as the first part of plan execution and rewrites the snapshot in-memory.
In order to fit neatly within the existing framework for resource operations,
these changes introduce a new kind of step, RefreshStep, to represent
refreshes. RefreshSteps operate similar to ReadSteps but do not imply that
the resource being read is not managed by Pulumi.
In addition to the refresh reimplementation, these changes incorporate those
from #1394 to run refresh in the integration test framework.
Fixes#1598.
Fixespulumi/pulumi-terraform#165.
Contributes to #1449.
The plugin host can ask the language host to provide a list of
resource plugins that it thinks will be nessecary for use at
deployment time, so they can be eagerly loaded.
In NodeJS (the only language host that implements this RPC) This works
by walking the directory tree rooted at the CWD of the project,
looking for package.json files, parsing them and seeing it they have
some marker property set. If they do, we add information about them
which we return at the end of our walk.
If there is *any* error, the entire operation fails. We've seen a
bunch of cases where this happens:
- Broken symlinks written by some editors as part of autosave.
- Access denied errors when part of the tree is unwalkable (Eric ran
into this on Windows when he had a Pulumi program at the root of his
file system.
- Recusive symlinks leading to errors when trying to walk down the
infinite chain. (See #1634 for one such example).
The very frustrating thing about this is that when you hit an error
its not clear what is going on and fixing it can be non-trivial. Even
worse, in the normal case, all of these plugins are already installed
and could be loaded by the host (in the common case, plugins are
installed as a post install step when you run `npm install`) so if we
simply didn't do this check at all, things would work great.
This change does two things:
1. It does not stop at the first error we hit when discovering
plugins, instead we just record the error and continue.
2. Does not fail the overall operation if there was an error. Instead,
we return to the host what we have, which may be an incomplete view
of the world. We glog the errors we did discover for diagnostics if
we ever need them.
I believe that long term most of this code gets deleted anyway. I
expect we will move to a model long term where the engine faults in
the plugin (downloading it if needed) when a request for the plugin
arrives. But for now, we shouldn't block normal operations just
because we couldn't answer a question with full fidelity.
Fixes#1478
The pulumi runtime used to lazily load and parse both config and
settings data set by the language host. The initial reason for this
design was that we wanted the runtime to be usable in a normal node
environment, but we have moved away from supporting that.
In addition, while we claimed we loaded these value "lazily", we
actually forced their loading quite eagerly when we started
up. However, when capturing config (or settings, as we now do), we
would capture all the logic about loading these values from the
environment.
Even worse, in the case where you had two copies of @pulumi/pulumi
loaded, it would be possible to capture a config object which was not
initialized and then at runtime the initialization logic would try to
read PULUMI_CONFIG from the process environment and fail.
So we adopt a new model where configuration and settings are parsed as
we load their containing modules. In addition, to support SxS
scinerios, we continue to use `process.env` as a way to control both
configuration and settings. This means that `run.ts` must now ensure
that these values are present in the environment before either the
config or runtime modules have been loaded.
Set the following compiler defaults:
```
"target": "es6",
"module": "commonjs",
"moduleResolution": "node",
"sourceMap": true,
```
Which allows us to not even include a tsconfig.json file. If one is
present, `ts-node` will use its options, but the above settings will
override any settings in a local tsconfig.json file. This means if you
want full control over the target, you'll need to go back to the raw
tsc workflow where you explicitly build ahead of time.
This change lets us set runtime specific options in Pulumi.yaml, which
will flow as arguments to the language hosts. We then teach the nodejs
host that when the `typescript` is set to `true` that it should load
ts-node before calling into user code. This allows using typescript
natively without an explicit compile step outside of Pulumi.
This works even when a tsconfig.json file is not present in the
application and should provide a nicer inner loop for folks writing
typescript (I'm pretty sure everyone has run into the "but I fixed
that bug! Why isn't it getting picked up? Oh, I forgot to run tsc"
problem.
Fixes#958
When a resource fails to initialize (i.e., it is successfully created,
but fails to transition to a fully-initialized state), and a user
subsequently runs `pulumi update` without changing that resource, our
CLI will fail to warn the user that this resource is not initialized.
This commit begins the process of allowing our CLI to report this by
storing a list of initialization errors in the checkpoint.
