#6636 inadvertently changed the minimum Go version
requirement to 1.16 since ReadFile was moved in that
version. Switch back to ioutil.ReadFile to avoid forcing
an upgrade at this time.
Some YAML parsers don't correctly handle Byte-order marks,
so automatically strip it off during load.
Related to #423
Co-authored-by: Justin Van Patten <jvp@justinvp.com>
These methods were getting promoted onto every struct that implemented
the Output interface, and are not necessary.
On a real world program, this saves 4% in binary size overall, and
15% of remaining binary size if `Apply<TypeName>` functions are
removed (https://github.com/pulumi/pulumi/issues/6592).
* [sdk/go] Cache loaded configuration files
Previously, the CLI did not cache configuration files, which
required a read from disk + unmarshalling + validation each
time a consumer needed to read one of these configurations.
This change introduces global caches for each type of Pulumi
configuration file (Project, ProjectStack, PolicyPackProject, and
PluginProject). The configuration is cached after the first request
and the cached value will be used for any subsequent operations.
Important note: The global configurations are not concurrency safe,
but this same problem exists using the previous method of
reading/writing config files on disk. Synchronization
will be added in a follow up change to allow for concurrency safe config
operations.
Fixes:#6565
As part of #6460, the logic for determing the version of the build was
moved to be a dependency on pulumictl.
Unfortunately, the homebrew installs use the "make dist" command to
build + install Pulumi to the user maching and as that would have a
dependency on pulumictl and it not existing on the user machine, it
would pass an empty version to the ldflag
This then manifested to the user as:
```
▶ pulumi version
warning: A new version of Pulumi is available. To upgrade from version '0.0.0' to '2.22.0', run
$ brew upgrade pulumi
or visit https://pulumi.com/docs/reference/install/ for manual instructions and release notes.
```
We are able to mitigate this behaviour by bringing back the get-version
script and using that script as part of the make brew installation
We can see that the versions are the same between the 2 different
installation techniques
```
make dist <------- uses pulumict
DIST:
go install -ldflags "-X github.com/pulumi/pulumi/sdk/v2/go/common/version.Version=2.24.0-alpha.1616029310+787eb70a" github.com/pulumi/pulumi/sdk/v2/dotnet/cmd/pulumi-language-dotnet
DIST:
BUILD:
```
```
make brew <----- uses the legacy script
▶ make brew
BREW:
go install -ldflags "-X github.com/pulumi/pulumi/sdk/v2/go/common/version.Version=v2.24.0-alpha.1616029310+g787eb70a2" github.com/pulumi/pulumi/sdk/v2/dotnet/cmd/pulumi-language-dotnet
BREW:
```
A full post mortem will be carried out to ensure we mitigate these
types of errors going forward and that we are able to better test
these types of situations
When using the filestate backend (local files and cloud buckets) there is no protection to prevent two processes from managing the same stack simultaneously.
This PR creates a locks directory in the management directory that stores lock files for a stack. Each backend implementation gets its own UUID that is joined with the stack name. The feature is currently available behind the `PULUMI_SELF_MANAGED_STATE_LOCKING=1` environment variable flag.
Adds a `--limit` flag to `pulumi stack history. This allows limiting to the last few entries rather than fetching the entirety of a stack's update history (which can be quite slow for stacks with lots of updates). Example: `pulumi stack history --limit 1` fetches the last history entry only.
`stack.up` and related operations in the Automation API have been updated to consume this change, drastically reducing overhead.
* Fix resource-ref-as-ID marshaling. (#6125)
This reapplies 2f0dba23ab.
* Fix malformed resource value bug
PR #6125 introduced a bug by marshaling resource
ids as PropertyValues, but not handling that case on
the unmarshaling side. The previous code assumed
that the id was a simple string value. This bug prevents
any stack update operations (preview, update, destroy,
refresh). Since this change was already
released, we must now handle both cases in the
unmarshaling code.
* Add resource ref unit tests for the Go SDK. (#6142)
This reapplies 3d505912b8.
Co-authored-by: Pat Gavlin <pat@pulumi.com>
- Add tests that deserialize known custom and component resources
- Add tests that deserialize missing custom and component resources
These changes also add support for deserializing resources with missing
modules/packages. Such resources are deserialized as generic component,
custom, or provider resources as appropriate.
Contributes to #5943.
- Add tests that deserialize known custom and component resources
- Add tests that deserialize missing custom and component resources
These changes also add support for deserializing resources with missing
modules/packages. Such resources are deserialized as generic component,
custom, or provider resources as appropriate.
Contributes to #5943.
When marshaling a resource reference as its ID (i.e. when
opts.KeepResources is false, as it will be in the case of downlevel SDKs
and resource providers), we must take care to marshal/unmarshal an empty
ID as the unknown property value.
This includes the following changes to the resource ref APIs:
- Bifurcate resource reference creation into two methods: one for
creating references to custom resources and one for creating
references to component resources.
- Store the ID in a resource reference as a PropertyValue s.t. it can be
computed.
- Add a helper method for retrieving the ID as a string + an indicator of
whether or not the reference has an ID.
Fixes#5939.
In order to support resource construction in the Go SDK, the
engine context needs to be available in the RPC unmarshaling
code. This change adds a context parameter to the Construct and
ConstructProvider functions, and plumbs the engine context through
to the relevant calls to these functions.
* [sdk/go] Support maps in Invoke outputs and Read inputs
These are already supported by the implementation, but were prevented by overzealous input validation in Invoke and ReadResource.
Follow-up to #4522 and #4521.
* Add CHANGELOG
* PR feedback
These changes are a combination of three commits, each of which
contributes to the testing and/or fixing of a problem with marshaling
unknowns in `plugin.provider.Update` when `preview` is true.
## deploytest: add support for gRPC adapters.
These changes add support for communicating with providers using the
gRPC adapters to the deploytest pacakage. This makes it easier to test
the gRPC adapters across typical lifecycle patterns.
Supporting these changes are two additions to the `resource/plugin`
package:
1. A type that bridges between the `plugin.Provider` interface and the
`pulumirpc.ResourceProviderServer`
2. A function to create a `plugin.Provider` given a
`pulumirpc.ResourceProviderClient`
The deploytest package uses these to wrap an in-process
`plugin.Provider` in a gRPC interface and connect to it without using
the default plugin host, respectively.
## pulumi_test: test provider preview over gRPC.
Add a test that runs the provider preview lifecycle, but using a
provider that communicates over gRPC.
## gRPC bridge: fix unknowns in `Update` previews
Set the `KeepUnknowns` and `RejectUnknowns` bits in the `MarshalOptions`
used when unmarshaling update results to preserve unknowns during a
preview and reject them otherwise.
These changes also set the `RejectUnknowns` bit in the `MarshalOptions`
used by `Create` if `preview` is false, and fix a bug in the array
unmarshaler that could cause out-of-bounds accesses.
Fixes https://github.com/pulumi/pulumi/issues/6004.
* Do not read TGZs into memory.
This runs a serious risk of exhausting the memory on lower-end machines
(e.g. certain CI VMs), especially given the potential size of some
plugins.
* CHANGELOG
* fixes
- Add component ref coverage to the existing test
- Add coverage for a downlevel SDK communicating with an engine that
supports resource refs
- Add coverage for a downlevel engine communicating with an SDK that
supports resource refs
As part of improving coverage, these changes add a knob to explicitly
disable resource refs in the engine without the use of the environment
variable. The environment variable is now only read by the CLI, and has
been restored to its prior polarity (i.e. `PULUMI_ENABLE_RESOURCE_REFERENCES`).
* Enable resource reference feature by default
Unless the PULUMI_DISABLE_RESOURCE_REFERENCES flag
is explicitly set to a truthy value, the resource reference feature is now
enabled by default.
* Set AcceptResources in the language SDKs
This can be disabled by setting the `PULUMI_DISABLE_RESOURCE_REFERENCES` environment variable to a truthy value.
Co-authored-by: Justin Van Patten <jvp@justinvp.com>
This re-applies the fix in 5857 to make credentials.json writes concurrency safe.
The original fix used `path.Dir` instead of `filepath.Dir` - which led to not placing the temp file in the same folder (and drive) as the renamed file target. This led to errors on Windows environments where the working directory was on a different drive than the `~/.pulumi` directory. The change to use `filepath.Dir` instead ensures that even on Windows, the true directory containing the credentials file is used for the temp file as well.
Fixes#3877.
* Properly resize arrays when adding
The current logic attempts to update the array but, because
append may need to allocate a new array with adequate space,
the code can currently leave dest referring to the old,
under-sized array. The solution is to use the set(dest)
logic that already exists and is used for the IsNull case.
Added a test case that would fail before this fix and now passes.
This fixespulumi/pulumi#5871.
* Add CHANGELOG entry
Implement GetRequiredPlugins for Python, which determines the plugins
required by the program.
Also, if the `virtualenv` runtime option is set, and the specified
virtual directory is missing or empty, automatically create it and
install dependencies into it.
Running `pulumi` operations in parallel could occasionally result in truncating the `~/.pulumi/credentials.json` file and reading that truncated file from another process before the content could be written.
Instead, use `os.Rename` to atomically replace the file contents.
Concurrent `pulumi` operations could still compete for who gets to write the file first, and could lead to surprising results in some extreme cases. But we should not see the corrupted file contents any longer.
Fixes#3877.
- Differentiate between resource references that have no ID (i.e. because
the referenced resource is not a CustomResource) and resource references
that have IDs that are not known. This is necessary for proper
backwards-compatible serialization of resource references.
- Fix the key that stores a resource reference's package version in the
.NET, NodeJS, and Python SDKs.
- Ensure that the resource monitor's marshalling/unmarshalling of inputs
and outputs to/from calls to `Construct` retain resource references as
appropriate.
- Fix serialization behavior for resources -> resource references in the
Go SDK: if a resource's ID is unknown, it should still be serialized
as a resource reference, albeit a reference with an unknown ID.
The PULUMI_BACKEND_URL env var allows specifying the backend to use instead of deferring to the project or the ~/.pulumi/credentials.json file to decide on the "current" backend. This allows for using Pulumi without a dependence on this piece of global filesystem state, so that each `pulumi` invocation can control the exact backend it want's to operate on, without having to do stateful `pulumi login`/`pulumi logout` operations.
This is especially useful for automation scenarios like Automation API generally (and effectively solves https://github.com/pulumi/pulumi/issues/5591), or https://github.com/pulumi/pulumi-kubernetes-operator/issues/83 specifically.
This also makes things like efe7a599e6/dist/actions/entrypoint.sh (L10) less necessary, and possible to accomplish for any containerized `pulumi` execution without the need for this logic to be embedded in bash scripts wrapping the CLI.
Two improvements:
1. Don't display "[resource plugin <foo>] installing" if the plugin is already installed.
2. Close the plugin download progress bar before displaying any subsequent output, and only show output of `npm install` when there is an error.
When installing a plugin, previous versions of Pulumi extracted the
plugin tarball to a temp directory and then renamed the temp directory
to the final plugin directory. This was done to prevent concurrent
installs: if a process fails to rename the temp dir because the final
dir already exists, it means another process already installed the
plugin. Unfortunately, on Windows the rename operation often fails due
to aggressive virus scanners opening files in the temp dir.
In order to provide reliable plugin installs on Windows, we now extract
the tarball directly into the final directory, and use file locks to
prevent concurrent installs from toppling over one another.
During install, a lock file is created in the plugin cache directory
with the same name as the plugin's final directory but suffixed with
`.lock`. The process that obtains the lock is responsible for extracting
the tarball. Before it does that, it cleans up any previous temp
directories of failed installs of previous versions of Pulumi. Then it
creates an empty `.partial` file next to the `.lock` file. The
`.partial` file indicates an installation is in-progress. The `.partial`
file is deleted when installation is complete, indicating the plugin was
successfully installed. If a failure occurs during installation, the
`.partial` file will remain indicating the plugin wasn't fully
installed. The next time the plugin is installed, the old installation
directory will be removed and replaced with a fresh install.
This is the same approach Go uses for installing modules in its
module cache.
Just what it says on the tin. This is implemented by changing the
`GetPackageConfig` method of `ConfigSource` to return a `PropertyMap`
and ensuring that any secret config is represented by a `Secret`.
