* Do not read TGZs into memory.
This runs a serious risk of exhausting the memory on lower-end machines
(e.g. certain CI VMs), especially given the potential size of some
plugins.
* CHANGELOG
* fixes
- Add component ref coverage to the existing test
- Add coverage for a downlevel SDK communicating with an engine that
supports resource refs
- Add coverage for a downlevel engine communicating with an SDK that
supports resource refs
As part of improving coverage, these changes add a knob to explicitly
disable resource refs in the engine without the use of the environment
variable. The environment variable is now only read by the CLI, and has
been restored to its prior polarity (i.e. `PULUMI_ENABLE_RESOURCE_REFERENCES`).
* Enable resource reference feature by default
Unless the PULUMI_DISABLE_RESOURCE_REFERENCES flag
is explicitly set to a truthy value, the resource reference feature is now
enabled by default.
* Set AcceptResources in the language SDKs
This can be disabled by setting the `PULUMI_DISABLE_RESOURCE_REFERENCES` environment variable to a truthy value.
Co-authored-by: Justin Van Patten <jvp@justinvp.com>
This re-applies the fix in 5857 to make credentials.json writes concurrency safe.
The original fix used `path.Dir` instead of `filepath.Dir` - which led to not placing the temp file in the same folder (and drive) as the renamed file target. This led to errors on Windows environments where the working directory was on a different drive than the `~/.pulumi` directory. The change to use `filepath.Dir` instead ensures that even on Windows, the true directory containing the credentials file is used for the temp file as well.
Fixes#3877.
* Properly resize arrays when adding
The current logic attempts to update the array but, because
append may need to allocate a new array with adequate space,
the code can currently leave dest referring to the old,
under-sized array. The solution is to use the set(dest)
logic that already exists and is used for the IsNull case.
Added a test case that would fail before this fix and now passes.
This fixespulumi/pulumi#5871.
* Add CHANGELOG entry
Implement GetRequiredPlugins for Python, which determines the plugins
required by the program.
Also, if the `virtualenv` runtime option is set, and the specified
virtual directory is missing or empty, automatically create it and
install dependencies into it.
Running `pulumi` operations in parallel could occasionally result in truncating the `~/.pulumi/credentials.json` file and reading that truncated file from another process before the content could be written.
Instead, use `os.Rename` to atomically replace the file contents.
Concurrent `pulumi` operations could still compete for who gets to write the file first, and could lead to surprising results in some extreme cases. But we should not see the corrupted file contents any longer.
Fixes#3877.
- Differentiate between resource references that have no ID (i.e. because
the referenced resource is not a CustomResource) and resource references
that have IDs that are not known. This is necessary for proper
backwards-compatible serialization of resource references.
- Fix the key that stores a resource reference's package version in the
.NET, NodeJS, and Python SDKs.
- Ensure that the resource monitor's marshalling/unmarshalling of inputs
and outputs to/from calls to `Construct` retain resource references as
appropriate.
- Fix serialization behavior for resources -> resource references in the
Go SDK: if a resource's ID is unknown, it should still be serialized
as a resource reference, albeit a reference with an unknown ID.
The PULUMI_BACKEND_URL env var allows specifying the backend to use instead of deferring to the project or the ~/.pulumi/credentials.json file to decide on the "current" backend. This allows for using Pulumi without a dependence on this piece of global filesystem state, so that each `pulumi` invocation can control the exact backend it want's to operate on, without having to do stateful `pulumi login`/`pulumi logout` operations.
This is especially useful for automation scenarios like Automation API generally (and effectively solves https://github.com/pulumi/pulumi/issues/5591), or https://github.com/pulumi/pulumi-kubernetes-operator/issues/83 specifically.
This also makes things like efe7a599e6/dist/actions/entrypoint.sh (L10) less necessary, and possible to accomplish for any containerized `pulumi` execution without the need for this logic to be embedded in bash scripts wrapping the CLI.
Two improvements:
1. Don't display "[resource plugin <foo>] installing" if the plugin is already installed.
2. Close the plugin download progress bar before displaying any subsequent output, and only show output of `npm install` when there is an error.
When installing a plugin, previous versions of Pulumi extracted the
plugin tarball to a temp directory and then renamed the temp directory
to the final plugin directory. This was done to prevent concurrent
installs: if a process fails to rename the temp dir because the final
dir already exists, it means another process already installed the
plugin. Unfortunately, on Windows the rename operation often fails due
to aggressive virus scanners opening files in the temp dir.
In order to provide reliable plugin installs on Windows, we now extract
the tarball directly into the final directory, and use file locks to
prevent concurrent installs from toppling over one another.
During install, a lock file is created in the plugin cache directory
with the same name as the plugin's final directory but suffixed with
`.lock`. The process that obtains the lock is responsible for extracting
the tarball. Before it does that, it cleans up any previous temp
directories of failed installs of previous versions of Pulumi. Then it
creates an empty `.partial` file next to the `.lock` file. The
`.partial` file indicates an installation is in-progress. The `.partial`
file is deleted when installation is complete, indicating the plugin was
successfully installed. If a failure occurs during installation, the
`.partial` file will remain indicating the plugin wasn't fully
installed. The next time the plugin is installed, the old installation
directory will be removed and replaced with a fresh install.
This is the same approach Go uses for installing modules in its
module cache.
Just what it says on the tin. This is implemented by changing the
`GetPackageConfig` method of `ConfigSource` to return a `PropertyMap`
and ensuring that any secret config is represented by a `Secret`.
This is necessary due to the way we've factored the libraries imported
by users into modules. The primary alternative is to ensure that each
child module imports the root module for a package and registers itself
with that package where necessary to prevent circular dependencies. This
simplifies the core SDKs slightly at the cost of greater complications
in the generated SDKs; the approach taken by these changes seems like a
more maintainable option.
Contributes to #2430.
Co-authored-by: Justin Van Patten <jvp@justinvp.com>
Resources are serialized as their URN, ID, and package version. Each
Pulumi package is expected to register itself with the SDK. The package
will be invoked to construct appropriate instances of rehydrated
resources. Packages are distinguished by their name and their version.
This is the foundation of cross-process resources.
Related to #2430.
Co-authored-by: Mikhail Shilkov <github@mikhail.io>
Co-authored-by: Luke Hoban <luke@pulumi.com>
Co-authored-by: Levi Blackstone <levi@pulumi.com>
These changes add support for provider-side previews of create and
update operations, which allows resource providers to supply output
property values for resources that are being created or updated during a
preview.
If a plugin supports provider-side preview, its create/update methods
will be invoked during previews with the `preview` property set to true.
It is the responsibility of the provider to fill in any output
properties that are known before returning. It is a best practice for
providers to only fill in property values that are guaranteed to be
identical if the preview were instead an update (i.e. only those output
properties whose values can be conclusively determined without
actually performing the create/update operation should be populated).
Providers that support previews must accept unknown values in their
create and update methods.
If a plugin does not support provider-side preview, the inputs to a
create or update operation will be propagated to the outputs as they are
today.
Fixes#4992.
The logic for validating prompted values in 'new' wasn't quite right,
leading to the possibility of creating Pulumi.yaml files with blank
project names.
This manifests in various ways and I've hit it a number of times
over the past few months because of the way we handle project/stack
name conflicts in 'new' -- which itself is a bit annoying too:
https://github.com/pulumi/pulumi/blob/master/pkg/cmd/pulumi/new.go#L206-L207
Because we substitue a default value of "", and because the prompting
logic assumed default values are always valid, we would skip validation
and therefore accept a blank Pulumi.yaml file.
This generates an invalid project which causes errors elsewhere, such as
error: failed to load Pulumi project located at ".../Pulumi.yaml":
project is missing a 'name' attribute
I hit this all the time with our getting started guide because I've
gone through it so many times and have leftover stacks from prior
run-throughs. I wouldn't be surprised if a lot of people hit this.
The solution here validates all values, including the default.
Note also that we failed to validate the value used by 'new --yes'
which meant you could bypass all validation by passing --yes, leading
to similar outcomes.
I've added a couple new tests for these cases. There is a risk we
depend on illegal default values somewhere which will now be rejected,
but that would seem strange, and assuming the tests pass, I would
assume that's not true. Let me know if that's wrong.
Fixespulumi/pulumi#3255.
* Revise host mode.
The current implementation of host mode uses a `pulumi host` command and
an ad-hoc communication protocol between the engine and client to
connect a language host after the host has begun listening. The most
significant disadvantages of this approach are the communication
protocol (which currently requires the use of stdout), the host-specific
command, and the difficulty of accommodating the typical program-bound
lifetime for an update.
These changes reimplement host mode by adding engine support for
connecting to an existing language runtime service rather than launching
a plugin. This capability is provided via an engine-specific language
runtime, `client`, which accepts the address of the existing languge
runtime service as a runtime option. The CLI exposes this runtime via
the `--client` flag to the `up` and `preview` commands, which similarly
accepts the address of an existing language runtime service as an
argument. These changes also adjust the automation API to consume the
new host mode implementation.
When installing a plugin, if it contains a `PulumiPlugin.yaml` file with a `runtime` value of `nodejs` or `python`, install dependencies for the plugin.
For Node.js, `npm install` is run (or `yarn install` if `PULUMI_PREFER_YARN` is set).
For Python, a virtual environment is created and deps installed into it.
* add initial pull-request workflow
* run SDK test all
* add SDK tests
* fixup make targets
* add dist target
* revert back to 5 updates
* disable test
* add issue for test disabling
These changes add initial support for the construction of remote
components. For now, this support is limited to the NodeJS SDK;
follow-up changes will implement support for the other SDKs.
Remote components are component resources that are constructed and
managed by plugins rather than by Pulumi programs. In this sense, they
are a bit like cloud resources, and are supported by the same
distribution and plugin loading mechanisms and described by the same
schema system.
The construction of a remote component is initiated by a
`RegisterResourceRequest` with the new `remote` field set to `true`.
When the resource monitor receives such a request, it loads the plugin
that implements the component resource and calls the `Construct`
method added to the resource provider interface as part of these
changes. This method accepts the information necessary to construct the
component and its children: the component's name, type, resource
options, inputs, and input dependencies. It is responsible for
dispatching to the appropriate component factory to create the
component, then returning its URN, resolved output properties, and
output property dependencies. The dependency information is necessary to
support features such as delete-before-replace, which rely on precise
dependency information for custom resources.
These changes also add initial support for more conveniently
implementing resource providers in NodeJS. The interface used to
implement such a provider is similar to the dynamic provider interface
(and may be unified with that interface in the future).
An example of a NodeJS program constructing a remote component resource
also implemented in NodeJS can be found in
`tests/construct_component/nodejs`.
This is the core of #2430.
We make several calls to `os/user`, which uses CGO and means
cross-compilation is not possible. This replaces `os/user` with the
`luser` package, which is a drop-in replacement which does not use `CGO`
Certain operations in `engine/diff` mutate engine events during display.
This mutation can occur concurrently with the serialization of the event
for persistence, which causes a panic in the CLI. These changes fix the
offending code and add code that copies each engine event before
persisteing it in order to guard against future issues.
This function adds a property value to another property at a given path,
creating containing properties as required. If the property cannot be
added because of a mismatch between the value types required by the path
and the values present in the destination, the add will fail. If a value
already exists at the given path, the add will succeed.
Related to #1635.
Several users reported cases where error messages would
cause a panic if they contained accented characters. I wasn't
able to reproduce this failure locally, but tracked down the
panic to logging gRPC calls. The Message field is typed as
a string, which requires all of the characters to be valid UTF-8.
This change runs each log string through the strings.ToValidUTF8
function, which will replace any invalid characters with the
"unknown" character. This should prevent the the logger from
panicking.
Automatically create a virtual environment and install dependencies in it with `pulumi new` and `pulumi policy new` for Python templates.
This will save a new `virtualenv` runtime option in `Pulumi.yaml` (`PulumiPolicy.yaml` for policy packs):
```yaml
runtime:
name: python
options:
virtualenv: venv
```
`virtualenv` is the path to a virtual environment that Pulumi will use when running `python` commands.
Existing projects are unaffected and can opt-in to using this by setting `virtualenv`, otherwise, they'll continue to work as-is.
With these changes, a resource struct may tag a field with the empty
string. If such a field is present, any resource outputs that were not
unmarshalled into other fields will be unmarshalled into this field,
which must be a `MapOutput`.
Fixes#4629.
Adds support for RegisterResource to accept map-typed implementations if Input as well as the existing struct-typed implementations. Currently these must be fully untyped - but both map[string]pulumi.Input and map[string]interface{} are allowed. In the future, it's plausible that a mode where the data itself is a map, but the ElementType implementation returns a struct could be supported, with the struct used to provide type information over the untyped map.
After importing some resources, and running a second update with the
import still applied, an unexpected replace would occur. This wouldn't
happen for the vast majority of resources, but for some it would.
It turns out that the resources that trigger this are ones that use a
different format of identifier for the import input than they do for the
ID property.
Before this change, we would trigger an import-replacement when an
existing resource's ID property didn't match the import property, which
would be the case for the small set of resources where the input
identifier is different than the ID property.
To avoid this, we now store the `importID` in the statefile, and
compare that to the import property instead of comparing the ID.
When setting structured config values using `--path`, we automatically
treat values that can be converted into an integer via `strconv.Atoi` as
an integer, rather than as a string.
However, this ends up converting values like "0123456" into the integer
123456, stripping the leading 0, which isn't desirable for values like
commit SHAs, etc., where you want to keep the 0 (and keep it a string).
This change makes it so that values starting with 0 are not implicitly
converted to an integer; instead such values will remain a string.
* Make `async:true` the default for `invoke` calls (#3750)
* Switch away from native grpc impl. (#3728)
* Remove usage of the 'deasync' library from @pulumi/pulumi. (#3752)
* Only retry as long as we get unavailable back. Anything else continues. (#3769)
* Handle all errors for now. (#3781)
* Do not assume --yes was present when using pulumi in non-interactive mode (#3793)
* Upgrade all paths for sdk and pkg to v2
* Backport C# invoke classes and other recent gen changes (#4288)
Adjust C# generation
* Replace IDeployment with a sealed class (#4318)
Replace IDeployment with a sealed class
* .NET: default to args subtype rather than Args.Empty (#4320)
* Adding system namespace for Dotnet code gen
This is required for using Obsolute attributes for deprecations
```
Iam/InstanceProfile.cs(142,10): error CS0246: The type or namespace name 'ObsoleteAttribute' could not be found (are you missing a using directive or an assembly reference?) [/Users/stack72/code/go/src/github.com/pulumi/pulumi-aws/sdk/dotnet/Pulumi.Aws.csproj]
Iam/InstanceProfile.cs(142,10): error CS0246: The type or namespace name 'Obsolete' could not be found (are you missing a using directive or an assembly reference?) [/Users/stack72/code/go/src/github.com/pulumi/pulumi-aws/sdk/dotnet/Pulumi.Aws.csproj]
```
* Fix the nullability of config type properties in C# codegen (#4379)
In order to support grpc-js gRPC servers, we need to slightly loosen
our checks on plugin readiness. This change was already in the 2.x
branch, but porting back so that 1.x CLIs can also be compatible.
Fixes#4252.
This was still in pkg and wasn't accessible from the scripts location.
This code should be in the go/common location of sdk so that it can be
accessed from within our build scripts
Without this change, we were not able to get access to the Version number
for Python and none of our Python builds have been publishing binaries
The initial config represents any config that was specified programmatically to the Policy Pack, for Policy Packs that support programmatic configuration like AWSGuard.
