When marshaling a resource reference as its ID (i.e. when
opts.KeepResources is false, as it will be in the case of downlevel SDKs
and resource providers), we must take care to marshal/unmarshal an empty
ID as the unknown property value.
This includes the following changes to the resource ref APIs:
- Bifurcate resource reference creation into two methods: one for
creating references to custom resources and one for creating
references to component resources.
- Store the ID in a resource reference as a PropertyValue s.t. it can be
computed.
- Add a helper method for retrieving the ID as a string + an indicator of
whether or not the reference has an ID.
Fixes#5939.
Although raw provider resources accept an input that allows a user to
specifiy a provider version to use, this input is not reflected in
current SDK code generation. Furthermore, we already have a method to
specify the provider version that should be used for a resource: the
"version" resource option. These changes update the code that handles
provider resource registrations to autmoatically populate the version
input from the "version" resource option if the option is present.
Fixes https://github.com/pulumi/pulumi-azure/issues/803.
Fixes: #2086
```
pulumi new --help
Create a new Pulumi project and stack from a template.
...
To create a project from a user defined template location in source control, pass the url as follows e.g.
* pulumi new https://gitlab.com/<user>/<repo>
* pulumi new https://bitbucket.org/<user>/<repo>
* pulumi new https://github.com/<user>/<repo>
To create the project from a branch of the user defined source control location, pass the url to the branch, e.g.
* pulumi new https://gitlab.com/<user>/<repo>/tree/<branch>
* pulumi new https://bitbucket.org/<user>/<repo>/tree/<branch>
* pulumi new https://github.com/<user>/<repo>/tree/<branch>
```
These changes are a combination of three commits, each of which
contributes to the testing and/or fixing of a problem with marshaling
unknowns in `plugin.provider.Update` when `preview` is true.
## deploytest: add support for gRPC adapters.
These changes add support for communicating with providers using the
gRPC adapters to the deploytest pacakage. This makes it easier to test
the gRPC adapters across typical lifecycle patterns.
Supporting these changes are two additions to the `resource/plugin`
package:
1. A type that bridges between the `plugin.Provider` interface and the
`pulumirpc.ResourceProviderServer`
2. A function to create a `plugin.Provider` given a
`pulumirpc.ResourceProviderClient`
The deploytest package uses these to wrap an in-process
`plugin.Provider` in a gRPC interface and connect to it without using
the default plugin host, respectively.
## pulumi_test: test provider preview over gRPC.
Add a test that runs the provider preview lifecycle, but using a
provider that communicates over gRPC.
## gRPC bridge: fix unknowns in `Update` previews
Set the `KeepUnknowns` and `RejectUnknowns` bits in the `MarshalOptions`
used when unmarshaling update results to preserve unknowns during a
preview and reject them otherwise.
These changes also set the `RejectUnknowns` bit in the `MarshalOptions`
used by `Create` if `preview` is false, and fix a bug in the array
unmarshaler that could cause out-of-bounds accesses.
Fixes https://github.com/pulumi/pulumi/issues/6004.
State tracking for goals was implemented using a raw map,
but this was not safe for concurrent read/write access from
multiple goroutines. Switched to using a sync.Map, which
is threadsafe.
* Do not read TGZs into memory.
This runs a serious risk of exhausting the memory on lower-end machines
(e.g. certain CI VMs), especially given the potential size of some
plugins.
* CHANGELOG
* fixes
The step generator was incorrectly tracking goal states for
old resources, which could lead to a panic if the resource
was removed in the update. This fix only generates goal
states for resources that exist in the updated program.
Now that resources are serialized as refs, ComponentResources
may try to unmarshal local resource refs before they are
initialized during the RegisterResource step.
This change avoids that issue by skipping Output marshaling
for local ComponentResources during the RegisterResource step.
These Outputs will be handled instead during the
RegisterResourceOutputs step.
Co-authored-by: Pat Gavlin <pat@pulumi.com>
The step generator applies `ignoreChanges` pre-processing for all
resources by copying old input values to the new goal for any properties
mentioned in the `ignoreChanges` list. However, this pre-processing
depends on the existence of prior inputs, which by definition does not
exist for a resource being imported prior to the issuance of the
`ImportStep`. These changes add this processing to the implementation of
`ImportStep`, using the inputs read from the provider as the prior
inputs.
- Add component ref coverage to the existing test
- Add coverage for a downlevel SDK communicating with an engine that
supports resource refs
- Add coverage for a downlevel engine communicating with an SDK that
supports resource refs
As part of improving coverage, these changes add a knob to explicitly
disable resource refs in the engine without the use of the environment
variable. The environment variable is now only read by the CLI, and has
been restored to its prior polarity (i.e. `PULUMI_ENABLE_RESOURCE_REFERENCES`).
Disable resource ref support until https://github.com/pulumi/pulumi-kubernetes/issues/1405
is fixed. This bug currently affects Python users with recent versions of the pulumi SDK
who are using the pulumi-kubernetes provider.
* Enable resource reference feature by default
Unless the PULUMI_DISABLE_RESOURCE_REFERENCES flag
is explicitly set to a truthy value, the resource reference feature is now
enabled by default.
* Set AcceptResources in the language SDKs
This can be disabled by setting the `PULUMI_DISABLE_RESOURCE_REFERENCES` environment variable to a truthy value.
Co-authored-by: Justin Van Patten <jvp@justinvp.com>
Record new resources after their operations have been run rather than
before in order to ensure that all state is available and appropriately
marked as secret.
Fixes#5803.
Fixes: #5452
When the user is requesting to change the secrets provider to a
passphrase provider, we now calculate that has been requested.
This means, we will prompt for a new passphrase for use in encrypting
the stack.
```
pulumi stack change-secrets-provider passphrase
Enter your passphrase to unlock config/secrets
(set PULUMI_CONFIG_PASSPHRASE or PULUMI_CONFIG_PASSPHRASE_FILE to remember):
Enter your new passphrase to protect config/secrets:
Re-enter your new passphrase to confirm:
Migrating old configuration and state to new secrets provider
Enter your passphrase to unlock config/secrets
(set PULUMI_CONFIG_PASSPHRASE or PULUMI_CONFIG_PASSPHRASE_FILE to remember):
```
Implement GetRequiredPlugins for Python, which determines the plugins
required by the program.
Also, if the `virtualenv` runtime option is set, and the specified
virtual directory is missing or empty, automatically create it and
install dependencies into it.
Generate ResourcePackage and ResourceModule implementations and
registrations. A ResourcePackage is generated for any module that
includes a provider resource (which should be the root module only), and
a ResourceModule is generated for any module that includes a resource.
And add version info to Python registrations.
To de-risk the Pulumi v2.15.0 release, temporarily revert 679d40950f. Use of resource references will require `PULUMI_EXPERIMENTAL_RESOURCE_REFERENCES=1` to be set until the feature has been enabled by default.
* Correctly rename stack files during a rename
This fixespulumi/pulumi#4463, by renaming a stack's configuration
file based on its stack-part, and ignoring the owner-part. Our
workspace system doesn't recognize configuration files with fully
qualified names. That, by the way, causes problems if we have
multiple stacks in different organizations that share a stack-part.
The fix here is simple: propagate the new StackReference from the
Rename operation and rely on the backend's normalization to a
simple name, and then use that the same way we are using a
StackReference to determine the path for the origin stack.
An alternative fix is to recognize fully qualified config files,
however, there's a fair bit of cleanup we will be doing as part of
https://github.com/pulumi/pulumi/issues/2522 and
https://github.com/pulumi/pulumi/issues/4605, so figured it is best
to make this work the way the system expects first, and revisit it
as part of those overall workstreams. I also suspect we may want to
consider changing the default behavior here as part of
https://github.com/pulumi/pulumi/issues/5731.
Tests TBD; need some advice on how best to test this since it
only happens with our HTTP state backend -- all integration tests
appear to use the local filestate backend at the moment.
* Add a changelog entry for bug fix
* Add some stack rename tests
* Fix a typo
* Address CR feedback
* Make some logic clearer
Use "parsedName" instead of "qn", add a comment explaining why
we're doing this, and also explicitly ignore the error rather
than implicitly doing so with _.
Fixes: #5835
when rotating a key in the Azure KeyVault secrets provider, we had
the following error:
```
error: secrets (code=InvalidArgument): keyvault.BaseClient#Decrypt: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="BadParameter" Message="The parameter is incorrect.\r\n"
```
This was because we were not regenerating the EncrytpionKey when
we were changing the secrets provider. Therefore, we now ensure
that this key is regenerated and we can successfully change the secrets provider
```
▶ pulumi stack init dev --secrets-provider="azurekeyvault://stack72kv10.vault.azure.net/keys/pulumi-secret"
Created stack 'dev'
▶ pulumi config set MyDBRootPassword Password1234! --secret
▶ pulumi config --show-secrets
KEY VALUE
MyDBRootPassword Password1234!
▶ pulumi stack change-secrets-provider "azurekeyvault://stack72kv20.vault.azure.net/keys/pulumi-secret"
▶ pulumi config --show-secrets
KEY VALUE
MyDBRootPassword Password1234!
```
Generate ResourcePackage and ResourceModule implementations and
registrations. A ResourcePackage is generated for any module that
includes a provider resource (which should be the root module only), and
a ResourceModule is generated for any module that includes a resource.
Note that version information is currently omitted. We should fix this
up before enabling resource reference deserialization end-to-end.
- Differentiate between resource references that have no ID (i.e. because
the referenced resource is not a CustomResource) and resource references
that have IDs that are not known. This is necessary for proper
backwards-compatible serialization of resource references.
- Fix the key that stores a resource reference's package version in the
.NET, NodeJS, and Python SDKs.
- Ensure that the resource monitor's marshalling/unmarshalling of inputs
and outputs to/from calls to `Construct` retain resource references as
appropriate.
- Fix serialization behavior for resources -> resource references in the
Go SDK: if a resource's ID is unknown, it should still be serialized
as a resource reference, albeit a reference with an unknown ID.
Two improvements:
1. Don't display "[resource plugin <foo>] installing" if the plugin is already installed.
2. Close the plugin download progress bar before displaying any subsequent output, and only show output of `npm install` when there is an error.
Just what it says on the tin.
The SDK code generator will be updated to use the new `urn`
resource option inside of each module's implementation of
`ResourceModule.construct`.
Part of #2430.
Co-authored-by: Justin Van Patten <jvp@justinvp.com>
These changes extend the type reference parser in the schema package to
accept references of the form "(package/version/schema.json)?#/provider".
These references refer to the package's provider type, which is
otherwise not referenceable, as it is not present in the "resources"
array.
This name better suits the semantics of the type, and aligns with the
rename of deploy.Plan to deploy.Deployment. These changes also refactor
the `update` method s.t. previews and updates are more consistent in
their behavior (e.g. duration and resource changes are now reported for
both, incl. on error paths).
Rename deploy.Plan to deploy.Deployment.
There are two benefits to this change:
1. The name "Deployment" more accurately reflects the behavior of the
type, which is responsible for previewing or executing a deployment.
2. Renaming this type frees up the name "Plan" for use when addressing
#2318.
Fixes: #5626
It used to be:
```
Policy Violations:
[advisory] aws v0.1.20200912 allowed-image-owner (demo-aws-ts-webserver-server-0: aws:ec2/instance:Instance)
Check machine image is from an approved publisher.
Publisher [137112412989] is not one of [self,099720109477].
```
Notice that it was name: type
We would rather this was type: name
```
Policy Violations:
[advisory] aws v0.1.20200912 allowed-image-owner (aws:ec2/instance:Instance: demo-aws-ts-webserver-server-0)
Check machine image is from an approved publisher.
Publisher [137112412989] is not one of [self,099720109477].
```
The way `pulumi new` installs dependencies for .NET projects is slightly different from other languages. For Node.js, Python, and Go, `pulumi new` runs the appropriate command to install project dependencies (e.g. `npm install`, `pip install`, or `go mod download`). For .NET, it calls the same routine used during `preview|up` to ensure required plugins are installed. For .NET, this ends up running `dotnet build` which implicitly installs Nuget packages, builds the project, and also attempts to determine and install the needed Pulumi plugins. When this operation runs during `preview|up`, and there are failures installing a plugin, the error is logged, but deliberately not returned, because an error will be shown for missing plugins later on during the `preview|up` operation. However, during `pulumi new`, we should show any plugin install errors.
Just what it says on the tin. This is implemented by changing the
`GetPackageConfig` method of `ConfigSource` to return a `PropertyMap`
and ensuring that any secret config is represented by a `Secret`.
The langauge SDKs will use this function to fetch the state required to
deserialize a resource reference. SDK support will be added as a follow-up
change.
Contributes to #2430.
Resources are serialized as their URN, ID, and package version. Each
Pulumi package is expected to register itself with the SDK. The package
will be invoked to construct appropriate instances of rehydrated
resources. Packages are distinguished by their name and their version.
This is the foundation of cross-process resources.
Related to #2430.
Co-authored-by: Mikhail Shilkov <github@mikhail.io>
Co-authored-by: Luke Hoban <luke@pulumi.com>
Co-authored-by: Levi Blackstone <levi@pulumi.com>
Move these tests to a new package, `lifecycletest`, that also exposes
APIs that allow consumers to implement their own lifecycle tests. This
is intended to ease the burden of testing plugin implementations and to
set the stage for cleaning up the lifecycle tests themselves.
This involves two changes to the public API, only one of which is
strictly necessary:
- The `host` field of `UpdateOptions` is now exported
- The `Journal` type has been moved from test-only code to the package
proper
The former change is necessary, as it is the mechanism by which package
consumers may inject their own plugin loaders. I was reluctant to expose
this field originally because I wanted to ensure that the behavior of
packages that embed Pulumi is consistent with that of the Pulumi CLI
with respect to plugin loading. I now believe that the risk of consumers
changing this behavior outside of test scenarios is low enough that we
can expose this field. This may also be useful for future scenarios,
e.g. statically linking providers and Pulumi programs.
The latter change is not necessary, but fleshes out the engine package
into a more complete toolkit. Downstream consumers may use the Journal
type to conveniently implement snapshotting.
Fixes: #5509
When changing from a passphrase provider to a cloud secrets provider,
the encryptionsalt is not required, so we should ensure this is removed
These changes add support for provider-side previews of create and
update operations, which allows resource providers to supply output
property values for resources that are being created or updated during a
preview.
If a plugin supports provider-side preview, its create/update methods
will be invoked during previews with the `preview` property set to true.
It is the responsibility of the provider to fill in any output
properties that are known before returning. It is a best practice for
providers to only fill in property values that are guaranteed to be
identical if the preview were instead an update (i.e. only those output
properties whose values can be conclusively determined without
actually performing the create/update operation should be populated).
Providers that support previews must accept unknown values in their
create and update methods.
If a plugin does not support provider-side preview, the inputs to a
create or update operation will be propagated to the outputs as they are
today.
Fixes#4992.
* fix pulumi bot token and env var
* use commit hash non-marketplace actions
* fix goreleaser binaries
foo
* fix goreleaser archive format
* update workflows to use repo-dispatch
* fixing tag naming
* add support for beta tags
* add prerelease workflow
* remove windows builds
* only use travis for tags
* remove prerelease steps
* fix container build indentation
* check the policy tempdir exists
Fixes: https://github.com/pulumi/docs/issues/4340
The deprecated message sits below the examples so it is not clear
to the user that the resource / datasource is deprecated
The logic for validating prompted values in 'new' wasn't quite right,
leading to the possibility of creating Pulumi.yaml files with blank
project names.
This manifests in various ways and I've hit it a number of times
over the past few months because of the way we handle project/stack
name conflicts in 'new' -- which itself is a bit annoying too:
https://github.com/pulumi/pulumi/blob/master/pkg/cmd/pulumi/new.go#L206-L207
Because we substitue a default value of "", and because the prompting
logic assumed default values are always valid, we would skip validation
and therefore accept a blank Pulumi.yaml file.
This generates an invalid project which causes errors elsewhere, such as
error: failed to load Pulumi project located at ".../Pulumi.yaml":
project is missing a 'name' attribute
I hit this all the time with our getting started guide because I've
gone through it so many times and have leftover stacks from prior
run-throughs. I wouldn't be surprised if a lot of people hit this.
The solution here validates all values, including the default.
Note also that we failed to validate the value used by 'new --yes'
which meant you could bypass all validation by passing --yes, leading
to similar outcomes.
I've added a couple new tests for these cases. There is a risk we
depend on illegal default values somewhere which will now be rejected,
but that would seem strange, and assuming the tests pass, I would
assume that's not true. Let me know if that's wrong.
Fixespulumi/pulumi#3255.
Instead of simplifying any module that ends with `/<name>`, only simplify
types where `<name>` matches the type name portion after camel-casing.
This continues to simplify tfbridge types like `aws:s3/bucket:Bucket`,
but would not simplify a type like `aws:s3:Bucket`.
We're not going to generate language-specific API docs for the Azure NextGen provider, only resource docs. This change makes it so the resource docs do not emit any links to nonexistent API docs.
