* [sdk/go] Cache loaded configuration files
Previously, the CLI did not cache configuration files, which
required a read from disk + unmarshalling + validation each
time a consumer needed to read one of these configurations.
This change introduces global caches for each type of Pulumi
configuration file (Project, ProjectStack, PolicyPackProject, and
PluginProject). The configuration is cached after the first request
and the cached value will be used for any subsequent operations.
Important note: The global configurations are not concurrency safe,
but this same problem exists using the previous method of
reading/writing config files on disk. Synchronization
will be added in a follow up change to allow for concurrency safe config
operations.
Fixes:#6565
As part of #6460, the logic for determing the version of the build was
moved to be a dependency on pulumictl.
Unfortunately, the homebrew installs use the "make dist" command to
build + install Pulumi to the user maching and as that would have a
dependency on pulumictl and it not existing on the user machine, it
would pass an empty version to the ldflag
This then manifested to the user as:
```
▶ pulumi version
warning: A new version of Pulumi is available. To upgrade from version '0.0.0' to '2.22.0', run
$ brew upgrade pulumi
or visit https://pulumi.com/docs/reference/install/ for manual instructions and release notes.
```
We are able to mitigate this behaviour by bringing back the get-version
script and using that script as part of the make brew installation
We can see that the versions are the same between the 2 different
installation techniques
```
make dist <------- uses pulumict
DIST:
go install -ldflags "-X github.com/pulumi/pulumi/sdk/v2/go/common/version.Version=2.24.0-alpha.1616029310+787eb70a" github.com/pulumi/pulumi/sdk/v2/dotnet/cmd/pulumi-language-dotnet
DIST:
BUILD:
```
```
make brew <----- uses the legacy script
▶ make brew
BREW:
go install -ldflags "-X github.com/pulumi/pulumi/sdk/v2/go/common/version.Version=v2.24.0-alpha.1616029310+g787eb70a2" github.com/pulumi/pulumi/sdk/v2/dotnet/cmd/pulumi-language-dotnet
BREW:
```
A full post mortem will be carried out to ensure we mitigate these
types of errors going forward and that we are able to better test
these types of situations
dotnet, nodejs and python automation APIs did not specify exec-kind for
refresh or destroy operations. This is now added following the same
logic from the go automation API.
When using the filestate backend (local files and cloud buckets) there is no protection to prevent two processes from managing the same stack simultaneously.
This PR creates a locks directory in the management directory that stores lock files for a stack. Each backend implementation gets its own UUID that is joined with the stack name. The feature is currently available behind the `PULUMI_SELF_MANAGED_STATE_LOCKING=1` environment variable flag.
This change avoids `RuntimeError: There is no current event loop in thread '<thread_name>'` errors when passing a resource as an input multiple times when using mocks.
The problem is that when using mocks, we deserialize the gRPC inputs before passing them to the user's mock methods. Deserializing inputs doesn't typically require an event loop, however, during deserialization of resource references, we end up creating some instances of `Future`, which does require an event loop to be present for the current thread. If this is done multiple times for a resource, it's possible that `deserialize_properties` will be called on an asyncio thread that doesn't yet have an event loop, resulting in the error being raised.
The error does not occur when only passing the resource reference once because typically the thread (e.g. `asyncio_0`) used in that case will have already had an event loop created for it due to the use of the internal `_syncawait` when _serializing_ the source resource's properties, which ensures an event loop is set for the thread.
The fix is to ensure an event loop is created for the thread in the mocks implementation before calling `deserialize_properties`.
It is possible for the same version of the same provider SDK to be loaded multiple times in Node.js. In this case, we might legitimately get mutliple registrations of the same resource. It should not matter which we use, so we can just skip re-registering. De-serialized resources will always be instances of classes from the first registered package.
Example layout this addresses. Registrations of resources in `package3` at the same verrsion.
`node_modules`
`@pulumi/pulumi`
`package1`
`node_modules`
`package3`
`package2`
`node_modules`
`package3`
Fixes#6258.
* Init Workspace interface for C# Automation API
* fleshing out workspace interface and beginning of local workspace implementation
* initial run pulumi cmd implementation
* resolve issue with pulumi cmd cleanup wrapper task after testing
* flesh out local workspace implementation, flesh out stack implementation, cleanup run pulumi cmd implementation and make it an instance so it is mockable/testable, separate serialization in prep for custom converters
* project settings json serialization implemented
* Initial commit of language server
* Add deployment from language server
* Cleanup
* finish json serialization
* project runtime yaml serialization completed. just need stack config value yaml serialization
* Remove typed argument
* Limit concurrency
* Rename file for consistency
* final commit of a semi-working project settings & stack settings serialization so that it is in the commit history
* modify workspace API so that settings accessors aren't fully exposed since we are defering a complete serialization implementation until a later date
* yaml converters wrap any outgoing exceptions so resolve that
* getting the beginning of inline program GRPC communication set up
* stack lifecycle operations implemented, and switched to newtonsoft for JSON serialization
* change back to system.text.json with a custom object converter
* local workspace tests written, working on getting them passing
* fix the encoding on the GO files used for testing
* all tests passing except inline program, pulumi engine not available with inline
* inline program engine is now running as expecting, but inline program is not recognizing local stack config
* All tests passing, but no concurrency capability because of the singleton DeploymentInstance.
* cleanup unnecessary usings
* minor cleanup / changes after a quick review. Make sure ConfigureAwait is used where needed. Remove newtonsoft dependency from testing. Update workspace API to use existing PluginKind enum. Modify LanguageRuntimeService so that its semaphore operates process-wide.
* support for parallel execution of inline program, test included
* Update LocalWorkspaceTests.cs
remove some redundancy from the inline program parallel execution text
* flesh out some comments and make asynclocal instance readonly
* Strip out instance locking since it is no longer necessary with AsyncLocal wrapping the Deployment.Instance. Modify CreateRunner method such that we are ensuring there isn't a chance of delayed synchronous execution polluting the value of Deployment.Instance across calls to Deployment.RunAsync
* resolve conflicts with changes made to Deployment.TestAsync entrypoints
* update changelog
* write a test that fails if the CreateRunnerAndRunAsync method on Deployment is not marked async and fix test project data file ref
* make resource package state share the lifetime of the deployment so that their isn't cross deployment issues with resource packages, add support and tests for external resource packages (resource packages that aren't referenced by the executing assembly)
* enable parallel test collection execution in test suite, add some additional tests for deployment instance protection and ensuring that our first class stack exceptions are thrown when expected
* minor inline project name arg change, and re-add xunit json to build output (whoops)
* strip out concurrency changes since they are now in #6139, split automation into separate assembly, split automation tests into separate assembly
* add copyright to the top of each new file
* resolve some PR remarks
* inline program exception is now properly propagated to the caller on UpAsync and PreviewAsync
* modify PulumiFn to allow TStack and other delegate overloads without needing multiple first class delegates.
* whoops missing a copyright
* resolve getting TStack into IRunner so that outputs are registered correctly and so that there isn't 2 instances of Pulumi.Stack instantiated.
* resolve issue with propagation of TStack exceptions and add a test
* add support for a TStack PulumiFn resolved via IServiceProvider
* update automation API description
* fix comment and remove unnecessary TODOs
* disable packaging of automation api assembly
* re-name automation api documentation file appropriately
* add --limit support to dotnet automation api for stack history per #6257
* re-name XStack as WorkspaceStack
* replace --limit usage with --page-size and --page in dotnet automation api per #6292
Co-authored-by: evanboyle <evan@pulumi.com>
Co-authored-by: Josh Studt <josh.studt@figmarketing.com>
Co-authored-by: Dan Friedman <dan@thefriedmans.org>
Co-authored-by: David Ferretti <David.Ferretti@figmarketing.com>
Co-authored-by: Mikhail Shilkov <github@mikhail.io>
We previously looked for `python3` and fallback to `python` on all systems. However, our Windows CI images include a `python3.exe` symlink to `python.exe` which does not work with `venv`. So on Windows, just look for `python` first, falling back to `python3`. (The default python.org Windows installation only includes `python.exe`).
Adds a `--limit` flag to `pulumi stack history. This allows limiting to the last few entries rather than fetching the entirety of a stack's update history (which can be quite slow for stacks with lots of updates). Example: `pulumi stack history --limit 1` fetches the last history entry only.
`stack.up` and related operations in the Automation API have been updated to consume this change, drastically reducing overhead.
`Output.from_input` deeply unwraps nested output values in dicts and lists, but doesn't currently do that for the more recently added "input types" (i.e. args classes). This leads to errors when using args classes with output values with `Provider` resources, which uses `Output.from_input` on each input property and then serializes the value to JSON in an `apply`. This changes fixes `Output.from_input` to recurse into values within args classes to properly unwrap any nested outputs.
Promise leak debugging was accidentally toStringing an Output, leading to a red herring for several users trying to understand what was causing promise leaks.
Related to #6153 and #5853.
.NET's implementation of RegisterResourceOutputs was always serializing resources as resource references, regardless of the monitor's reported support. This change fixes .NET to check if the monitor supports resource references, which is consistent with all the other languages, and with serialization code elsewhere in the .NET SDK.
* Fix resource-ref-as-ID marshaling. (#6125)
This reapplies 2f0dba23ab.
* Fix malformed resource value bug
PR #6125 introduced a bug by marshaling resource
ids as PropertyValues, but not handling that case on
the unmarshaling side. The previous code assumed
that the id was a simple string value. This bug prevents
any stack update operations (preview, update, destroy,
refresh). Since this change was already
released, we must now handle both cases in the
unmarshaling code.
* Add resource ref unit tests for the Go SDK. (#6142)
This reapplies 3d505912b8.
Co-authored-by: Pat Gavlin <pat@pulumi.com>
- Add tests that deserialize known custom and component resources
- Add tests that deserialize missing custom and component resources
These changes also add support for deserializing resources with missing
modules/packages. Such resources are deserialized as generic component,
custom, or provider resources as appropriate.
Contributes to #5943.
- Add tests that deserialize known custom and component resources
- Add tests that deserialize missing custom and component resources
These changes also add support for deserializing resources with missing
modules/packages. Such resources are deserialized as generic component,
custom, or provider resources as appropriate.
Contributes to #5943.
When marshaling a resource reference as its ID (i.e. when
opts.KeepResources is false, as it will be in the case of downlevel SDKs
and resource providers), we must take care to marshal/unmarshal an empty
ID as the unknown property value.
This includes the following changes to the resource ref APIs:
- Bifurcate resource reference creation into two methods: one for
creating references to custom resources and one for creating
references to component resources.
- Store the ID in a resource reference as a PropertyValue s.t. it can be
computed.
- Add a helper method for retrieving the ID as a string + an indicator of
whether or not the reference has an ID.
Fixes#5939.
Related: #5653
This will take an existing output and then unwrap the secret, and
return a new output
```
import * as pulumi from "@pulumi/pulumi";
const x = pulumi.secret("test")
export const xVal = x;
const y = pulumi.unsecret(x);
export const yVal = y;
```
```
▶ pulumi stack output
Current stack outputs (3):
OUTPUT VALUE
xVal [secret]
yVal test
```
Also adds the ability to check if an output is as secret:
```
import * as pulumi from "@pulumi/pulumi";
const x = pulumi.secret("test")
const isSecret = x.isSecret;
export const isSecretDeets = isSecret;
```
- Add tests that serialize custom and component resources for targets
that support resource references
- Add tests that serialize custom and component resources for downlevel
targets
- Add tests that deserialize known custom and component resources
- Add tests that deserialize missing custom and component resources
These changes also fix a few bugs that were encountered during testing:
- Component resource construction was not supported
- Resources with missing packages could not be deserialized
In the latter case, a missing resource is deserialized as a generic
DependencyResource.
These changes also update the signature of IMocks.NewResourceAsync to
allow the returned ID to be null. This is technically a C# breaking change
with respect to nullability.
Contributes to #5943.
Co-authored-by: Mikhail Shilkov <github@mikhail.io>
In order to support resource construction in the Go SDK, the
engine context needs to be available in the RPC unmarshaling
code. This change adds a context parameter to the Construct and
ConstructProvider functions, and plumbs the engine context through
to the relevant calls to these functions.
When a resource reference is deserialized, it may not have a version in which case `version` will be an empty string. This change fixes `TryGetResourceType` to work correctly when an empty version is passed.
* [sdk/go] Support maps in Invoke outputs and Read inputs
These are already supported by the implementation, but were prevented by overzealous input validation in Invoke and ReadResource.
Follow-up to #4522 and #4521.
* Add CHANGELOG
* PR feedback
Adds an opt-in `allowSecrets` flag to `serializeFunction` to allow it to capture secrets. If passed, `serializeFunction` will now report back whether it captured any secrets. This information can be used by callers to wrap the resulting text in a Secret value.
Fixes#2718.
These changes are a combination of three commits, each of which
contributes to the testing and/or fixing of a problem with marshaling
unknowns in `plugin.provider.Update` when `preview` is true.
## deploytest: add support for gRPC adapters.
These changes add support for communicating with providers using the
gRPC adapters to the deploytest pacakage. This makes it easier to test
the gRPC adapters across typical lifecycle patterns.
Supporting these changes are two additions to the `resource/plugin`
package:
1. A type that bridges between the `plugin.Provider` interface and the
`pulumirpc.ResourceProviderServer`
2. A function to create a `plugin.Provider` given a
`pulumirpc.ResourceProviderClient`
The deploytest package uses these to wrap an in-process
`plugin.Provider` in a gRPC interface and connect to it without using
the default plugin host, respectively.
## pulumi_test: test provider preview over gRPC.
Add a test that runs the provider preview lifecycle, but using a
provider that communicates over gRPC.
## gRPC bridge: fix unknowns in `Update` previews
Set the `KeepUnknowns` and `RejectUnknowns` bits in the `MarshalOptions`
used when unmarshaling update results to preserve unknowns during a
preview and reject them otherwise.
These changes also set the `RejectUnknowns` bit in the `MarshalOptions`
used by `Create` if `preview` is false, and fix a bug in the array
unmarshaler that could cause out-of-bounds accesses.
Fixes https://github.com/pulumi/pulumi/issues/6004.
* Do not read TGZs into memory.
This runs a serious risk of exhausting the memory on lower-end machines
(e.g. certain CI VMs), especially given the potential size of some
plugins.
* CHANGELOG
* fixes
- Improve the existing coverage to use real resources and mocks
- Add tests for deserialization as well as serialization
- Add tests that serialize custom resources during preview
Contributes to #5943.
- Reset the runtime prior to each test
- Use the SDK's `test` decorator instead of `async_test`
- Rename a couple classes to avoid warnings from pytest
These tests cover the same scenarios that are coverted in the engine's
unit tests, but exercise the Node SDK's marshalling paths.
These changes include a few enhancements to the Node SDK's test APIs
that make it easier to more precisely control its behavior, and extend
the `Mocks` interface to allow the registration of component resources
to work properly.
Contributes to #5943.
- Add component ref coverage to the existing test
- Add coverage for a downlevel SDK communicating with an engine that
supports resource refs
- Add coverage for a downlevel engine communicating with an SDK that
supports resource refs
As part of improving coverage, these changes add a knob to explicitly
disable resource refs in the engine without the use of the environment
variable. The environment variable is now only read by the CLI, and has
been restored to its prior polarity (i.e. `PULUMI_ENABLE_RESOURCE_REFERENCES`).
* Enable resource reference feature by default
Unless the PULUMI_DISABLE_RESOURCE_REFERENCES flag
is explicitly set to a truthy value, the resource reference feature is now
enabled by default.
* Set AcceptResources in the language SDKs
This can be disabled by setting the `PULUMI_DISABLE_RESOURCE_REFERENCES` environment variable to a truthy value.
Co-authored-by: Justin Van Patten <jvp@justinvp.com>
`log.error` will call the engine's `log` gRPC endpoint (if the engine is available; otherwise it will write to `console.error`) with `LogSeverity.ERROR`, which tell the engine to stop processing further resource operations.
Without this, any uncaught errors (such as input validation errors done inside `apply`) would be written to stderr, but wouldn't actually result in an update error.
This re-applies the fix in 5857 to make credentials.json writes concurrency safe.
The original fix used `path.Dir` instead of `filepath.Dir` - which led to not placing the temp file in the same folder (and drive) as the renamed file target. This led to errors on Windows environments where the working directory was on a different drive than the `~/.pulumi` directory. The change to use `filepath.Dir` instead ensures that even on Windows, the true directory containing the credentials file is used for the temp file as well.
Fixes#3877.
* Properly resize arrays when adding
The current logic attempts to update the array but, because
append may need to allocate a new array with adequate space,
the code can currently leave dest referring to the old,
under-sized array. The solution is to use the set(dest)
logic that already exists and is used for the IsNull case.
Added a test case that would fail before this fix and now passes.
This fixespulumi/pulumi#5871.
* Add CHANGELOG entry
Implement GetRequiredPlugins for Python, which determines the plugins
required by the program.
Also, if the `virtualenv` runtime option is set, and the specified
virtual directory is missing or empty, automatically create it and
install dependencies into it.
Running `pulumi` operations in parallel could occasionally result in truncating the `~/.pulumi/credentials.json` file and reading that truncated file from another process before the content could be written.
Instead, use `os.Rename` to atomically replace the file contents.
Concurrent `pulumi` operations could still compete for who gets to write the file first, and could lead to surprising results in some extreme cases. But we should not see the corrupted file contents any longer.
Fixes#3877.
There are two significant changes in this commit: one to the way
resource packages/modules are stored and retrieved, and one to resource
ref deserialization in the face of missing resource packages/modules.
Resource packages and modules no longer require an exact version match
during deserialization. Instead, the newest compatible version of the
package or module is selected. If no version was specified, the newest
version of the package or module will be chosen. As a special case, a
package or module that has no version will always be treated as the best
version for that package or module.
If a resource package or module is not found when attempting to
deserialize a resource reference, the SDK no longer emits an error, and
instead deserializes the reference as its URN or ID (if present). This
accommodates providers that have not yet been updated to include the
appropriate factory registrations.
- Differentiate between resource references that have no ID (i.e. because
the referenced resource is not a CustomResource) and resource references
that have IDs that are not known. This is necessary for proper
backwards-compatible serialization of resource references.
- Fix the key that stores a resource reference's package version in the
.NET, NodeJS, and Python SDKs.
- Ensure that the resource monitor's marshalling/unmarshalling of inputs
and outputs to/from calls to `Construct` retain resource references as
appropriate.
- Fix serialization behavior for resources -> resource references in the
Go SDK: if a resource's ID is unknown, it should still be serialized
as a resource reference, albeit a reference with an unknown ID.
