This re-applies the fix in 5857 to make credentials.json writes concurrency safe.
The original fix used `path.Dir` instead of `filepath.Dir` - which led to not placing the temp file in the same folder (and drive) as the renamed file target. This led to errors on Windows environments where the working directory was on a different drive than the `~/.pulumi` directory. The change to use `filepath.Dir` instead ensures that even on Windows, the true directory containing the credentials file is used for the temp file as well.
Fixes#3877.
* Properly resize arrays when adding
The current logic attempts to update the array but, because
append may need to allocate a new array with adequate space,
the code can currently leave dest referring to the old,
under-sized array. The solution is to use the set(dest)
logic that already exists and is used for the IsNull case.
Added a test case that would fail before this fix and now passes.
This fixespulumi/pulumi#5871.
* Add CHANGELOG entry
Implement GetRequiredPlugins for Python, which determines the plugins
required by the program.
Also, if the `virtualenv` runtime option is set, and the specified
virtual directory is missing or empty, automatically create it and
install dependencies into it.
Running `pulumi` operations in parallel could occasionally result in truncating the `~/.pulumi/credentials.json` file and reading that truncated file from another process before the content could be written.
Instead, use `os.Rename` to atomically replace the file contents.
Concurrent `pulumi` operations could still compete for who gets to write the file first, and could lead to surprising results in some extreme cases. But we should not see the corrupted file contents any longer.
Fixes#3877.
There are two significant changes in this commit: one to the way
resource packages/modules are stored and retrieved, and one to resource
ref deserialization in the face of missing resource packages/modules.
Resource packages and modules no longer require an exact version match
during deserialization. Instead, the newest compatible version of the
package or module is selected. If no version was specified, the newest
version of the package or module will be chosen. As a special case, a
package or module that has no version will always be treated as the best
version for that package or module.
If a resource package or module is not found when attempting to
deserialize a resource reference, the SDK no longer emits an error, and
instead deserializes the reference as its URN or ID (if present). This
accommodates providers that have not yet been updated to include the
appropriate factory registrations.
- Differentiate between resource references that have no ID (i.e. because
the referenced resource is not a CustomResource) and resource references
that have IDs that are not known. This is necessary for proper
backwards-compatible serialization of resource references.
- Fix the key that stores a resource reference's package version in the
.NET, NodeJS, and Python SDKs.
- Ensure that the resource monitor's marshalling/unmarshalling of inputs
and outputs to/from calls to `Construct` retain resource references as
appropriate.
- Fix serialization behavior for resources -> resource references in the
Go SDK: if a resource's ID is unknown, it should still be serialized
as a resource reference, albeit a reference with an unknown ID.
The PULUMI_BACKEND_URL env var allows specifying the backend to use instead of deferring to the project or the ~/.pulumi/credentials.json file to decide on the "current" backend. This allows for using Pulumi without a dependence on this piece of global filesystem state, so that each `pulumi` invocation can control the exact backend it want's to operate on, without having to do stateful `pulumi login`/`pulumi logout` operations.
This is especially useful for automation scenarios like Automation API generally (and effectively solves https://github.com/pulumi/pulumi/issues/5591), or https://github.com/pulumi/pulumi-kubernetes-operator/issues/83 specifically.
This also makes things like efe7a599e6/dist/actions/entrypoint.sh (L10) less necessary, and possible to accomplish for any containerized `pulumi` execution without the need for this logic to be embedded in bash scripts wrapping the CLI.
Two improvements:
1. Don't display "[resource plugin <foo>] installing" if the plugin is already installed.
2. Close the plugin download progress bar before displaying any subsequent output, and only show output of `npm install` when there is an error.
Just what it says on the tin.
The SDK code generator will be updated to use the new `urn`
resource option inside of each module's implementation of
`ResourceModule.construct`.
Part of #2430.
Co-authored-by: Justin Van Patten <jvp@justinvp.com>
When installing a plugin, previous versions of Pulumi extracted the
plugin tarball to a temp directory and then renamed the temp directory
to the final plugin directory. This was done to prevent concurrent
installs: if a process fails to rename the temp dir because the final
dir already exists, it means another process already installed the
plugin. Unfortunately, on Windows the rename operation often fails due
to aggressive virus scanners opening files in the temp dir.
In order to provide reliable plugin installs on Windows, we now extract
the tarball directly into the final directory, and use file locks to
prevent concurrent installs from toppling over one another.
During install, a lock file is created in the plugin cache directory
with the same name as the plugin's final directory but suffixed with
`.lock`. The process that obtains the lock is responsible for extracting
the tarball. Before it does that, it cleans up any previous temp
directories of failed installs of previous versions of Pulumi. Then it
creates an empty `.partial` file next to the `.lock` file. The
`.partial` file indicates an installation is in-progress. The `.partial`
file is deleted when installation is complete, indicating the plugin was
successfully installed. If a failure occurs during installation, the
`.partial` file will remain indicating the plugin wasn't fully
installed. The next time the plugin is installed, the old installation
directory will be removed and replaced with a fresh install.
This is the same approach Go uses for installing modules in its
module cache.
Just what it says on the tin. This is implemented by changing the
`GetPackageConfig` method of `ConfigSource` to return a `PropertyMap`
and ensuring that any secret config is represented by a `Secret`.
This is necessary due to the way we've factored the libraries imported
by users into modules. The primary alternative is to ensure that each
child module imports the root module for a package and registers itself
with that package where necessary to prevent circular dependencies. This
simplifies the core SDKs slightly at the cost of greater complications
in the generated SDKs; the approach taken by these changes seems like a
more maintainable option.
Contributes to #2430.
Co-authored-by: Justin Van Patten <jvp@justinvp.com>
