Instead of always running `npm install`, if the Policy Pack is Python, emit instructions on how to install dependencies. Also minor cleanup/refactoring.
This was still in pkg and wasn't accessible from the scripts location.
This code should be in the go/common location of sdk so that it can be
accessed from within our build scripts
Without this change, we were not able to get access to the Version number
for Python and none of our Python builds have been publishing binaries
* Decode the C# language info from the schema package and set it in the dotnet lang helper for docs. Pass the qualifier for C# property type strings based on whether it is an input or an output property.
* Don't pass a module name for Pulumi core types used in TS constructor and Function params.
* Add tests for generating doc links for nodejs types.
* Add test for confirming input doc link for C# type.
* Fix the C# type name for InvokeOptions.
The initial config represents any config that was specified programmatically to the Policy Pack, for Policy Packs that support programmatic configuration like AWSGuard.
* Add tests for the examples processing.
* Update the schema-based code generators to strip out non-relevant code examples wrapped in short-codes.
* Extract language relevant examples for Functions too.
* Simpler way of extracting the surrounding text from the description.
* Add some more new-line characters to clean-up the examples sections.
* Update the C# invoke signature in the Functions template used by resource docs generator.
* Add a new function to the DocLanguageHelper interface for resource docs generator to generate language-specific property names.
* Add a new DisplayName property to the propertyType struct for simpler display names without module names.
* Update templates to use the DisplayName property of propertyType. Strip the namespace/module name from type names for use as display names.
* Generate Go package maps at the beginning, so that the resource docs can use the appropriate package context for generating a property type string name.
* Temporarily block the resource docs from search indexing.
* Use PascalCase for nested type names in Python.
* Fix bug with Python property name casing for nested types by checking the property case maps.
* Fix rename stack message when attempting to move organizations
* Include the URL to change stack ownership
Co-authored-by: Lee-Ming Zen <lee@pulumi.com>
Rather than shelling out to the `rm` and `cp` commands,
use Go's os utils to perform these operations. This will
work on any platform rather than just Linux.
This package contains a driver for generating source code from HCL2
expressions. The driver is based on the fmt package's support for custom
formatters.
These changes implement `GetRequiredPlugins` for Go using a registry
mechanism and an alternate entry point for `pulumi.Run`. Packages that
require plugins are expected to register themselves with the Pulumi SDK.
When `pulumi.Run` is used and the `PULUMI_PLUGINS` envvar is truthy, the
program will dump a JSON-encoded description of its required plugins to
stdout. The language host then uses this description to respond to
These changes add a package for type checking and modeling HCL2
configurations. It is made up of three primary components:
1. A static type system
2. A semantic representation of HCL2 expressions and a binder from HCL2
native syntax to this representation
3. A semantic representation of HCL2 structural elements and binders
from HCL2 native syntax to this representation.
The type system is described in the "Extended Types" section of the
specification. The semantic representations of expressions and
structural elements are documented in their implementations.
* Generate the constructor params for Python along with other languages.
* Remove redundant py_function_param nested template. Declare a new type for defining property characteristics rather than using inlining formal params. Generate the Lookup functions for all languages similar to the constructor params with linking enabled.
* Fix bug with generating the input arg type name for Functions in Go.
* Add prefix for args param of a Go-based Resource Function.
* Input args for Go-based Functions use Lookup*Args and not Get*Args.
* Turns out that args for Go-based Functions use a different prefix based on whether the function is a package-level or module-level Function.
* Update the Python list and dictionary type names for the resource doc generator.
* Add a separate function for Python doc helper return a type string representing dictionaries that are simple maps and don't have a known nested element type.
* Also remove the Examples property from the resourceDocArgs type since the resource Comment will contain the multi-lang examples on their surrounded by a shortcode.
The integration test framework currently supports using `dep` for dependency management.
However, `dep` has no native ability to manage "yarn link"-style dependencies on locally available packages.
This is a necessary scenario for testing in most repos though, as (e.g.) examples in the Kubernetes repo need to test against the locally available version of `pulumi-kubernetes`.
The best we can do is a trick of (a) deleting the vendored copy of the locally available dependency (b) copying the locally available dependency into the vendor folder (c) deleting the nested vendor folder in the new copy of the locally available dependency.
* Added a new template for Functions. Implement the genFunction method for generating the docs for Functions.
* Rename type resourceArgs to resourceDocArgs. Minor updates the resource template.
* Generate nested types for Functions.
* Unexport types that don't need to be exported. Create the doc language helper objects in an init function and reuse them rather than recreating them every time. Update genNestedTypes to work with schema functions or resources.
* Fixed bug in nested type generation for Functions. Fixed bug in generating input and output doc links for nested types.
The changes in #4004 caused old provider configuration to be used even when a provider was different between inputs and outputs, in the case that the diff returned DiffUnkown.
To better handle that case, we compute a more accurate (but still conservative) DiffNone or DiffSome so that we can ensure we conservatively update to a new provider when needed, but retain the performance benefit of not creating and configuring a new provider as much as possible.
Part of https://github.com/pulumi/pulumi-aws/issues/814.
Fixes: https://github.com/pulumi/pulumi-terraform-bridge/issues/119
This allows us to specify an overlays block e.g.
```
Overlay: &tfbridge.OverlayInfo{
DestFiles: []string{
"pulumi_docker/docker.py",
"pulumi_docker/image.py",
},
},
```
The overlays files are treated differently to normal module files
as they are not generated. This structure means that we will emit
the correct entries in the __init__.py file
Without this structure (ie. pulumi_pkgname), the generator actually
copies the file (i.e. docker.py) to the root of the Python SDK. This
is because the structure of the Python SDK has a sub-folder than that
of the NodeJS SDK
I tested this using PR https://github.com/pulumi/pulumi-docker/pull/141
and this now works as expected and we can take advantage of the new
Python overlays for Docker
* Update properties.tmpl to render property comment as-is. WIP splitting out properties to lang-specific tables.
* Generate the constructor dynamically from the resource per language.
* Add doc functions in each language generator package for getting doc links for types..and later other functions too.
* Render the constructor params in the Go code and inject into the template.
* Generate nodejs types using the nodejs lang generator.
* Add a templates bundler. Added a new Make target for autogenerating a static bundle for the resource docs generator.
* Generate type links for all languages based on their schema type. Render the property type with a link if the underlying elements have a supporting type. Fix word-breaks for Python type names.
* Various changes including the introduction of an interface type under the codegen package to help with generating some language-specific information for the resource docs generator.
* Add a function to explicitly generate links for input types of nested types. Fix the resource doc link generator for Go. Don't replace the module name from the nodejs language type.
* Fix bug with C# property type html encoding.
* Fix some template formatting. Pass the state inputs for Python to generate the lookup function for it.
* Do not generate the examples section if there are none.
* Generating the property types per language.
* Formatting. Rename function for readability.
* Add comments. Update README.
* Use relative URLs for doc links within the main site
These changes implement `GetRequiredPlugins` for Go using a registry
mechanism and an alternate entry point for `pulumi.Run`. Packages that
require plugins are expected to register themselves with the Pulumi SDK.
When `pulumi.Run` is used and the `PULUMI_PLUGINS` envvar is truthy, the
program will dump a JSON-encoded description of its required plugins to
stdout. The language host then uses this description to respond to
`GetRequiredPlugins`.
* started transformations for go sdk
* added first basic test
* added second test with child
* added RegisterStackTransformation
* added a couple tests to lifecycle_test
* update CHANGELOG and test
* included TODO for #3846
In the very common case where provider configuration does not change, during preview we were calling `Configure` on the cloud provider twice - once for the "old" configuration, and once for the "new" configuration.
This is not necessary, and we can just avoid using the new provider when configuration has not changed, since we will have configured the old provider very early so if we can use that we should.
Note that this technically doesn't prevent the second call to `Configure` from being made, but it prevents us from ever waiting on it. We may want to go further and avoid even calling `Configure` on the provider in this case.
Part of #3671.
These changes add a new method to the resource provider gRPC interface,
`GetSchema`, that allows consumers of these providers to extract
JSON-serialized schema information for the provider's types, resources,
and functions.
These changes add a helper package for parsing HCL2 syntax files. The
helpers are intended to provide direct access to syntax elements that
are abstracted away by the standard HCL2 parser. The bulk of the code
deals with mapping syntax nodes to their relevant tokens in order to
avoid losing information about the comments associated with syntax
nodes.
It appears there are cases where our IsInteractive heuristics return true, but terminal.GetSize returns an error. In these cases, we should assume we do not have an interactive terminal and avoid trying to render interactive progress by default.
Fixes#3935.
* Make Python StackReference test similar to others (with two steps)
* Include new Python StackReference integration test that uses multiple stacks
* Expose various life cycle methods for ProgramTester
* Revert "Use test helper. (#1977)"
This reverts commit e498cab239.
* Avoid duplicate newlines
Ensure that we print exactly one trailing newline per log entry.
- Make go comment generation lint-clean for blank comment lines
- Fix the casing of `Provider.py` in Python to `provider.py`
- Fix a spacing issue in the NodeJS code generator
We can't correctly print simple messages for prelude events when doing progress based display in a terminal, as it would lead to resetting the display of the table rendering.
This does mean that `--show-config` no longer works in the default terminal display mode - but it's not clear it *can* work correctly (at least as currently implemented) since it doesn't cleanly participate in the table rendering.
For cases where `--show-config` is not set (the norm) -nothing would have been printed anyway, so the changes here just avoid resetting the table rendering unnecessarily.
Fixes#3469.
The provider plugin protocol is to write a port number followed by `\n`. We must guarantee we do that even on Windows, so must avoid Python `print` statements which implicitly rewrite newlines to platform specific character sequences.
Fixes#3807.
Replace the various `defaults` maps in the schema with per-property
`default` and `defaultInfo` fields. The former holds the static default
value; the latter holds the envvars and language-specific info.
Also, fix a minor bug in the Python codegen that caused diffs in
property docstrings.
We were seeing that ~all same steps were requiring checkpoint writes due to percieving a difference between `Dependencies` being `nil` and `[]URN{}` - which should be considered the same for this purpose.
The schema format is described in pkg/codegen/schema/schema.go. The code
generators are derived from the code generators contained in
https://github.com/pulumi/pulumi-terraform-bridge/pkg/tfgen, with the
exception of the Go code generator, which is net new.
The redesign is focused around providing better static typings and
improved ease-of-use for the Go SDK. Most of the redesign revolves
around three pivots:
- Strongly-typed inputs, especially for nested types
- Struct-based resource and invoke APIs
- Ease-of-use of Apply
1. Strongly-typed inputs
Input is the type of a generic input value for a Pulumi resource.
This type is used in conjunction with Output to provide polymorphism
over strongly-typed input values.
The intended pattern for nested Pulumi value types is to define an
input interface and a plain, input, and output variant of the value
type that implement the input interface.
For example, given a nested Pulumi value type with the following shape:
```
type Nested struct {
Foo int
Bar string
}
```
We would define the following:
```
var nestedType = reflect.TypeOf((*Nested)(nil)).Elem()
type NestedInput interface {
pulumi.Input
ToNestedOutput() NestedOutput
ToNestedOutputWithContext(context.Context) NestedOutput
}
type Nested struct {
Foo int `pulumi:"foo"`
Bar string `pulumi:"bar"`
}
type NestedInputValue struct {
Foo pulumi.IntInput `pulumi:"foo"`
Bar pulumi.StringInput `pulumi:"bar"`
}
func (NestedInputValue) ElementType() reflect.Type {
return nestedType
}
func (v NestedInputValue) ToNestedOutput() NestedOutput {
return pulumi.ToOutput(v).(NestedOutput)
}
func (v NestedInputValue) ToNestedOutputWithContext(ctx context.Context) NestedOutput {
return pulumi.ToOutputWithContext(ctx, v).(NestedOutput)
}
type NestedOutput struct { *pulumi.OutputState }
func (NestedOutput) ElementType() reflect.Type {
return nestedType
}
func (o NestedOutput) ToNestedOutput() NestedOutput {
return o
}
func (o NestedOutput) ToNestedOutputWithContext(ctx context.Context) NestedOutput {
return o
}
func (o NestedOutput) Foo() pulumi.IntOutput {
return o.Apply(func (v Nested) int {
return v.Foo
}).(pulumi.IntOutput)
}
func (o NestedOutput) Bar() pulumi.StringOutput {
return o.Apply(func (v Nested) string {
return v.Bar
}).(pulumi.StringOutput)
}
```
The SDK provides input and output types for primitives, arrays, and
maps.
2. Struct-based APIs
Instead of providing expected output properties in the input map passed
to {Read,Register}Resource and returning the outputs as a map, the user
now passes a pointer to a struct that implements one of the Resource
interfaces and has appropriately typed and tagged fields that represent
its output properties.
For example, given a custom resource with an int-typed output "foo" and
a string-typed output "bar", we would define the following
CustomResource type:
```
type MyResource struct {
pulumi.CustomResourceState
Foo pulumi.IntOutput `pulumi:"foo"`
Bar pulumi.StringOutput `pulumi:"bar"`
}
```
And invoke RegisterResource like so:
```
var resource MyResource
err := ctx.RegisterResource(tok, name, props, &resource, opts...)
```
Invoke arguments and results are also provided via structs, but use
plain-old Go types for their fields:
```
type MyInvokeArgs struct {
Foo int `pulumi:"foo"`
}
type MyInvokeResult struct {
Bar string `pulumi:"bar"`
}
var result MyInvokeResult
err := ctx.Invoke(tok, MyInvokeArgs{Foo: 42}, &result, opts...)
```
3. Ease-of-use of Apply
All `Apply` methods now accept an interface{} as the callback type.
The provided callback value must have one of the following signatures:
func (v T) U
func (v T) (U, error)
func (ctx context.Context, v T) U
func (ctx context.Context, v T) (U, error)
T must be assignable from the ElementType of the Output. If U is a type
that has a registered Output type, the result of the Apply will be the
corresponding Output type. Otherwise, the result of the Apply will be
AnyOutput.
Fixes https://github.com/pulumi/pulumi/issues/2149.
Fixes https://github.com/pulumi/pulumi/issues/3488.
Fixes https://github.com/pulumi/pulumi/issues/3487.
Fixes https://github.com/pulumi/pulumi-aws/issues/248.
Fixes https://github.com/pulumi/pulumi/issues/3492.
Fixes https://github.com/pulumi/pulumi/issues/3491.
Fixes https://github.com/pulumi/pulumi/issues/3562.
* Add a new metadata property for BuildNumber. Update Travis and GitLab to set both Build ID and Build Number. Add link to env vars doc for Codefresh.
* Update changelog
* Update CI vars detection test.
* Add PR number to changelog.
* Use Merge Request Instance ID instead of the Merge Request ID for GitLab CI.
* Use GitLab Pipeline Instance ID as the BuildID for GitLab CI.
* Update the changelog.
* Update the test for GitLab CI detection.
* Fix logic to determine PRNumber and BuildURL for Az Pipelines.
* Update changelog
* Set the BranchName to the PR source branch if PRNumber is not empty.
* added support for using GOOGLE_CREDENTIALS environment variable for authenticating with gs:// file state
* modified the change to fix#2791 as well
* fixed a small bug
* fixed linter error
* added code comments
* Update pkg/backend/filestate/gcpauth.go
Co-Authored-By: CyrusNajmabadi <cyrus.najmabadi@gmail.com>
* Parse provided backend url to check if scheme is gs://
* Update changelog
A regression was introduced when we added support for non-Node.js Pulumi programs to run Policy Packs. With that change, we now pass the Policy Pack's full path as the plugin's pwd (so that it would load the `@pulumi/pulumi/cmd/run-policy-pack` Node module from the Policy Pack's node_modules rather than the program's node_modules), but we also pass the path to the policy pack as well. If the path is a full rooted path, this would work fine, and that's what our tests do. However, if a relative path is specified, then it will be looking to load the Policy Pack relative to the pwd, which doesn't produce a correct path leading to failures trying to load the Policy Pack.
Since the pwd is the policy pack path, we can simply pass the path as `"."` to the analyzer plugin, and it will load the policy pack in its pwd.
This change adds support for setting `PULUMI_PREFER_YARN` to true to opt-in to preferring `yarn` over `npm` when installing Node.js dependencies (and publishing Policy Packs). If `PULUMI_PREFER_YARN` is truthy, but `yarn` cannot be found on `$PATH`, we fallback to using `npm`. If `npm` can't be found on `$PATH`, we provide a more helpful error message.
Add support for a test option that indicates that failed update steps
should be retried. Currently the maximum number of retries (3) is not
configurable.
The text "Plan applied failed: " is pretty inscrutable given our
current system. While both "plan" and "apply" are concepts inside the
the implementation of the CLI, we usually talk in terms of `preview`
and `update`. I suspect there are some cases where this prefix is not
100% technically correct, and if there's a better short way of saying
something more correct, I would love to adopt that instead, but as is,
I would really love to get rid of the "Plan apply failed" text in our
system, it pains me every time I read it.
Codepaths which could result in a hang will print a message to the console indicating the problem, along with a link to documentation on how to restructure code to best address it.
`StackReference.getOutputSync` and `requireOutputSync` have been deprecated as they may cause hangs on some combinations of Node and certain OS platforms. `StackReference.getOutput` and `requireOutput` should be used instead.
To allow Policy Packs to run against Pulumi programs written in all languages, we now look for the `@pulumi/pulumi/cmd/run-policy-pack` module in the Policy Pack's node_modules (instead of in the Pulumi program's node_modules; which doesn't exist for non-node languages). The `@pulumi/policy` library that a Policy Pack will depend on should already depend on a recent enough version of `@pulumi/pulumi`. When we can't find the module, it's more likely it's due to the dependencies for the Policy Pack not being installed. Provide a more helpful error message in this case.
- If an untargeted create would not affect the inputs of any targeted
resources, do not fail the update. Untargeted creates that are
directly dependend on by targeted resources will still cause failures
that inform the user to add the untargeted resources to the --target
list.
- Users may now pass the `--target-dependents` flag to allow targeted
destroys to automatically target dependents that must be destroyed in
order to destroy an explicitly targeted resource.
- Use a mutex + condition variable instead of a channel for
synchronizaiton in order to allow multiple calls to resolve/reject
- Properly handle outputs that are resolved to other outputs, especially
if those outputs are not of exactly type Output
- Remove the Value() methods that allowed prompt access to output values
- Add variants of `Apply` that take a context parameter
- Ensure that resource outputs properly incorporate their resource as
a dependency
- Make `Output` a plain struct. Uninitialized outputs will be treated as
resolved and unknown. This makes conversions between output
types more ergonomic.
Contributes to #3492.
These changes restore a more-correct version of the behavior that was
disabled with #3014. The original implementation of this behavior was
done in the SDKs, which do not have access to the complete inputs for a
resource (in particular, default values filled in by the provider during
`Check` are not exposed to the SDK). This lack of information meant that
the resolved output values could disagree with the typings present in
a provider SDK. Exacerbating this problem was the fact that unknown
values were dropped entirely, causing `undefined` values to appear in
unexpected places.
By doing this in the engine and allowing unknown values to be
represented in a first-class manner in the SDK, we can attack both of
these issues.
Although this behavior is not _strictly_ consistent with respect to the
resource model--in an update, a resource's output properties will come
from its provider and may differ from its input properties--this
behavior was present in the product for a fairly long time without
significant issues. In the future, we may be able to improve the
accuracy of resource outputs during a preview by allowing the provider
to dry-run CRUD operations and return partially-known values where
possible.
These changes also introduce new APIs in the Node and Python SDKs
that work with unknown values in a first-class fashion:
- A new parameter to the `apply` function that indicates that the
callback should be run even if the result of the apply contains
unknown values
- `containsUnknowns` and `isUnknown`, which return true if a value
either contains nested unknown values or is exactly an unknown value
- The `Unknown` type, which represents unknown values
The primary use case for these APIs is to allow nested, properties with
known values to be accessed via the lifted property accessor even when
the containing property is not fully know. A common example of this
pattern is the `metadata.name` property of a Kubernetes `Namespace`
object: while other properties of the `metadata` bag may be unknown,
`name` is often known. These APIs allow `ns.metadata.name` to return a
known value in this case.
In order to avoid exposing downlevel SDKs to unknown values--a change
which could break user code by exposing it to unexpected values--a
language SDK must indicate whether or not it supports first-class
unknown values as part of each `RegisterResourceRequest`.
These changes also allow us to avoid breaking user code with the new
behavior introduced by the prior commit.
Fixes#3190.
Adds a new experimental `pulumi watch` CLI command which can be used for inner loop development on a Pulumi stack. This command is only available currently via `PULUMI_EXPERIMENTAL=true` while in active development.
The `watch` command does the following:
1. Watches the workspace (the tree rooted at the `Pulumi.yaml` file) for changes
2. Triggers an `update` to the stack whenever there is a change
3. Streams output containing summaries of key update events as well as logs from any resources under management into a combined CLI output
Part of https://github.com/pulumi/pulumi/issues/3448.
The PULUMI_EXPERIMENTAL flag also makes`query` and `policy` available.
This change adds support for lists and maps in config. We now allow
lists/maps (and nested structures) in `Pulumi.<stack>.yaml` (or
`Pulumi.<stack>.json`; yes, we currently support that).
For example:
```yaml
config:
proj:blah:
- a
- b
- c
proj:hello: world
proj:outer:
inner: value
proj:servers:
- port: 80
```
While such structures could be specified in the `.yaml` file manually,
we support setting values in maps/lists from the command line.
As always, you can specify single values with:
```shell
$ pulumi config set hello world
```
Which results in the following YAML:
```yaml
proj:hello world
```
And single value secrets via:
```shell
$ pulumi config set --secret token shhh
```
Which results in the following YAML:
```yaml
proj:token:
secure: v1:VZAhuroR69FkEPTk:isKafsoZVMWA9pQayGzbWNynww==
```
Values in a list can be set from the command line using the new
`--path` flag, which indicates the config key contains a path to a
property in a map or list:
```shell
$ pulumi config set --path names[0] a
$ pulumi config set --path names[1] b
$ pulumi config set --path names[2] c
```
Which results in:
```yaml
proj:names
- a
- b
- c
```
Values can be obtained similarly:
```shell
$ pulumi config get --path names[1]
b
```
Or setting values in a map:
```shell
$ pulumi config set --path outer.inner value
```
Which results in:
```yaml
proj:outer:
inner: value
```
Of course, setting values in nested structures is supported:
```shell
$ pulumi config set --path servers[0].port 80
```
Which results in:
```yaml
proj:servers:
- port: 80
```
If you want to include a period in the name of a property, it can be
specified as:
```
$ pulumi config set --path 'nested["foo.bar"]' baz
```
Which results in:
```yaml
proj:nested:
foo.bar: baz
```
Examples of valid paths:
- root
- root.nested
- 'root["nested"]'
- root.double.nest
- 'root["double"].nest'
- 'root["double"]["nest"]'
- root.array[0]
- root.array[100]
- root.array[0].nested
- root.array[0][1].nested
- root.nested.array[0].double[1]
- 'root["key with \"escaped\" quotes"]'
- 'root["key with a ."]'
- '["root key with \"escaped\" quotes"].nested'
- '["root key with a ."][100]'
Note: paths that contain quotes can be surrounded by single quotes.
When setting values with `--path`, if the value is `"false"` or
`"true"`, it will be saved as the boolean value, and if it is
convertible to an integer, it will be saved as an integer.
Secure values are supported in lists/maps as well:
```shell
$ pulumi config set --path --secret tokens[0] shh
```
Will result in:
```yaml
proj:tokens:
- secure: v1:wpZRCe36sFg1RxwG:WzPeQrCn4n+m4Ks8ps15MxvFXg==
```
Note: maps of length 1 with a key of “secure” and string value are
reserved for storing secret values. Attempting to create such a value
manually will result in an error:
```shell
$ pulumi config set --path parent.secure foo
error: "secure" key in maps of length 1 are reserved
```
**Accessing config values from the command line with JSON**
```shell
$ pulumi config --json
```
Will output:
```json
{
"proj:hello": {
"value": "world",
"secret": false,
"object": false
},
"proj:names": {
"value": "[\"a\",\"b\",\"c\"]",
"secret": false,
"object": true,
"objectValue": [
"a",
"b",
"c"
]
},
"proj:nested": {
"value": "{\"foo.bar\":\"baz\"}",
"secret": false,
"object": true,
"objectValue": {
"foo.bar": "baz"
}
},
"proj:outer": {
"value": "{\"inner\":\"value\"}",
"secret": false,
"object": true,
"objectValue": {
"inner": "value"
}
},
"proj:servers": {
"value": "[{\"port\":80}]",
"secret": false,
"object": true,
"objectValue": [
{
"port": 80
}
]
},
"proj:token": {
"secret": true,
"object": false
},
"proj:tokens": {
"secret": true,
"object": true
}
}
```
If the value is a map or list, `"object"` will be `true`. `"value"` will
contain the object as serialized JSON and a new `"objectValue"` property
will be available containing the value of the object.
If the object contains any secret values, `"secret"` will be `true`, and
just like with scalar values, the value will not be outputted unless
`--show-secrets` is specified.
**Accessing config values from Pulumi programs**
Map/list values are available to Pulumi programs as serialized JSON, so
the existing
`getObject`/`requireObject`/`getSecretObject`/`requireSecretObject`
functions can be used to retrieve such values, e.g.:
```typescript
import * as pulumi from "@pulumi/pulumi";
interface Server {
port: number;
}
const config = new pulumi.Config();
const names = config.requireObject<string[]>("names");
for (const n of names) {
console.log(n);
}
const servers = config.requireObject<Server[]>("servers");
for (const s of servers) {
console.log(s.port);
}
```
Allow the user to specify a set of resources to replace via the
`--replace` flag on the CLI. This can be combined with `--target` to
replace a specific set of resources without changing any other
resources. `--target-replace` is shorthand for `--replace urn --target urn`.
Fixes#2643.
If a stack output includes a `Resource`, we will as of a recent change
always show the output diff, but this diff will potentially include
unknowns, leading to spurious output like:
```
+ namePrefix : output<string>
```
These changes supress these diffs by adding a special key to the POJO
we generate for resources *during preview only* that indicates that the
POJO represents a Pulumi resource, then stripping all adds of unknown
values from diffs for objects marked with that key.
Fixes#3314.
This is needed to add support for objects (maps/lists) to config. Once this has been merged, we can merge the associated change in the service, then merge the overall change in the CLI.
`engine.Query` queues up a bunch of `defer` functions that (among other
things) wait for various resources to clean themselves up.
In `query`, we have a "naive" (read: bad) `eventEmitter`, whose `done`
channel is set to `nil`. Therefore, the relevent `defer` waits for an
event on a `nil` channel which in Go will famously simply hang forever.
This commit will correct this by setting this channel appropriately, so
that it signals appropriately when it's done.
This commit will introduce the ability to load providers in `query`
mode.
Previously, `query` mode has been effectively a stand-alone execution
environment for language hosts, running without (e.g.) the
`StepExecutor` and similar engine facilities, but with some minimal
constructs hooked up, notably the ability to retrieve stack snapshots
from the backend for querying.
This commit extends this functionality somewhat by allowing `query` to
load Pulumi resource providers, and to run `Invoke` on them. This will
allow us, in the future, to "query" resource providers in the same way
we can query stack snapshots.
This method can be used to check whether or not a URN is well-formed.
This is used by the provider reference parser to avoid panicking on
malformed URNs.
- The length of the text content (i.e. the content of a colorized string
that is not control sequences) was not being correctly tracked. This
caused the "status" column of the progress display to overflow.
- Colorization was unconditionally disabled on Windows. When we were
using loreley, we had set the global colorization flag s.t.
colorization on Windows _should_ have been disabled, but we overrode
this flag each time we actually colorized anything.
Fixes#3378.
This avoids unnecessary blocking inside pre/post-step callbacks if the
reader on the other side of the event channel is slow.
We do not use a buffered channel in the event pipe because it is
empirically less likely that the goroutine reading from a buffered
channel will be scheduled when new data is placed in the channel. In the
case of our event system in which events will not be delivered to the
service and display until the copying goroutine is scheduled, this can
lead to unacceptable delay between the send of the original event and
its output.
- Cache the username and last verified time associated with each logged-in
backend
- In the HTTP backend, verify the access token explicitly at most once
per hour
This trades off a little bit of usability for improved inner-loop
latency: if a user's API token becomes invalid less than an hour after
it was last verified, the user will see 4xx errors when attempting stack
operations rather than seeing the login prompt.
The current pattern of using backend.Stack values in the CLI but
accepting backend.StackReference values in backend methods leads to
frequent transitions between these types. In the case of the HTTP
backend, the transition from a StackReference to a Stack requires an API
call. These changes refactor the backend.Backend API such that most of
its methods accept backend.Stack values in place of
backend.StackReference values, which avoids these transitions.
This removes two calls to the getStack API on the startup path of a
`pulumi preview`.
The dependency graph used to determine the set of resources that
depend on a resource being DBR'd is constructured from the list of
resource states present in the old snapshot. However, the dependencies
of resources that are present in both the old snapshot and the current
plan can be different, which in turn can cause the engine to make
incorrect decisions during DBR with respect to which resources need to
be replaced. For example, consider the following program:
```
var resA = new Resource("a", {dbr: "foo"});
var resB = new Resource("b", {dbr: resA.prop});
```
If this program is then changed to:
```
var resB = new Resource("b", {dbr: "<literal value of resA.prop>"});
var resA = new Resource("a", {dbr: "bar"});
```
The engine will first decide to make no changes to "b", as its input
property values have not changed. "b" has changed, however, such that it
no longer has a dependency on "a".
The engine will then decide to DBR "a". In the process, it will
determine that it first needs to delete "b", because the state for "b"
that is used when calculating "a"'s dependents does not reflect the
changes made during the plan.
To fix this issue, we rely on the observation that dependents can only
have been _removed_ from the base dependency graph: for a dependent to
have been added, it would have had to have been registered prior to the
root--a resource it depends on--which is not a valid operation. This
means that any resources that depend on the root must not yet have
been registered, which in turn implies that resources that have already
been registered must not depend on the root. Thus, we ignore these
resources if they are encountered while walking the old dependency graph
to determine the set of dependents.
Loreley's approach to colorization via Go templates has significant
overhead when colorization is used extensively.
Also, stop parsing colorized text with regexes and use the actual
parser.
* Fix lint warnings
* Improve error message when using PAC on old SDK
* Update pkg/resource/plugin/analyzer_plugin.go
Co-Authored-By: Justin Van Patten <jvp@justinvp.com>
Introduces `PULUMI_HOME` environment variable which points to a path to the path to `.pulumi` folder. Defaults to `<user's home dir> + ".pulumi"` if not specified.
Fixes#2966. In addition to plugins, it "moves" the credentials file, templates, workspaces.
`bin` folder is intact: to move it, we need to adjust all installation scripts to respect `PULUMI_HOME` and put executables in the proper `bin` folder.
These changes restore a more-correct version of the behavior that was
disabled with #3014. The original implementation of this behavior was
done in the SDKs, which do not have access to the complete inputs for a
resource (in particular, default values filled in by the provider during
`Check` are not exposed to the SDK). This lack of information meant that
the resolved output values could disagree with the typings present in
a provider SDK. Exacerbating this problem was the fact that unknown
values were dropped entirely, causing `undefined` values to appear in
unexpected places.
By doing this in the engine and allowing unknown values to be
represented in a first-class manner in the SDK, we can attack both of
these issues.
Although this behavior is not _strictly_ consistent with respect to the
resource model--in an update, a resource's output properties will come
from its provider and may differ from its input properties--this
behavior was present in the product for a fairly long time without
significant issues. In the future, we may be able to improve the
accuracy of resource outputs during a preview by allowing the provider
to dry-run CRUD operations and return partially-known values where
possible.
These changes also introduce new APIs in the Node and Python SDKs
that work with unknown values in a first-class fashion:
- A new parameter to the `apply` function that indicates that the
callback should be run even if the result of the apply contains
unknown values
- `containsUnknowns` and `isUnknown`, which return true if a value
either contains nested unknown values or is exactly an unknown value
- The `Unknown` type, which represents unknown values
The primary use case for these APIs is to allow nested, properties with
known values to be accessed via the lifted property accessor even when
the containing property is not fully know. A common example of this
pattern is the `metadata.name` property of a Kubernetes `Namespace`
object: while other properties of the `metadata` bag may be unknown,
`name` is often known. These APIs allow `ns.metadata.name` to return a
known value in this case.
In order to avoid exposing downlevel SDKs to unknown values--a change
which could break user code by exposing it to unexpected values--a
language SDK must indicate whether or not it supports first-class
unknown values as part of each `RegisterResourceRequest`.
These changes also allow us to avoid breaking user code with the new
behavior introduced by the prior commit.
Fixes#3190.
Adds the ability to provide `transformations` to modify the properties and resource options that will be used for any child resource of a component or stack.
This offers an "escape hatch" to modify the behaviour of a component by peeking behind it's abstraction. For example, it can be used to add a resource option (`additionalSecretOutputs`, `aliases`, `protect`, etc.) to a specific known child of a component, or to modify some input property to a child resource if the component does not (yet) expose the ability to control that input directly. It could also be used for more interesting scenarios - such as:
1. Automatically applying tags to all resources that support them in a stack (or component)
2. Injecting real dependencies between stringly-referenced resources in a Helm Chart
3. Injecting explicit names using a preferred naming convention across all resources in a stack
4. Injecting `import` onto all resources by doing a lookup into a name=>id mapping
Because this feature makes it possible to peek behind a component abstraction, it must be used with care in cases where the component is versioned independently of the use of transformations. Also, this can result in "spooky action at a distance", so should be used judiciously. That said - this can be used as an escape hatch to unblock a wide variety of common use cases without waiting on changes to be made in a component implementation.
Each transformation is passed the `resource`, `name`, `type`, `props` and `opts` that are passed into the `Resource` constructor for any resource descended from the resource that has the transformation applied. The transformation callback can optionally return alternate versions of the `props` and `opts` to be used in place of the original values provided to the resource constructor.
Fixes#2068.
This caching is enabled by wrapping the `secrets.Manager` returned by
`DefaultSecretsProvider.OfType` in an outer `secrets.Manager` that
cooperates with `stack.{Serialize,Deserialize}PropertyValue`. Ciphertext
is cached on a per-secret-instance basis (i.e. not a per-plaintext-value
basis). Cached ciphertext is only reused if the plaintext for the secret
value has not changed. Entries are inserted into the cache upon both
encryption and decryption so that values that originated from ciphertext
and that have not changed can aoid re-encryption.
Contributes to #3178.
* Fix some tracing issues.
- Add endpoints for `startUpdate` and `postEngineEventsBatch` so that
spans for these invocations have proper names
- Inject a tracing span when walking a plan so that resource operations
are properly parented
- When handling gRPC calls, inject a tracing span into the call's
metadata if no span is already present so that resource monitor and
engine spans are properly parented
- Do not trace client gRPC invocations of the empty method so that these
calls (which are used to determine server availability) do not muddy
the trace. Note that I tried parenting these spans appropriately, but
doing so broke the trace entirely.
With these changes, the only unparented span in a typical Pulumi
invocation is a single call to `getUser`. This span is unparented
because that call does not have a context available. Plumbing a context
into that particular call is surprisingly tricky, as it is often called
by other context-less functions.
* Make tracing support more flexible.
- Add support for writing trace data to a local file using Appdash
- Add support for viewing Appdash traces via the CLI
Fixes: #2319
In #2319, a user is hitting the gRPC limit on the message size the
server can receive when uploading ec2 user-data
This commit doubles the limit that can be sent from `1024*1024*4` to
`1024*1024*8`
This matches the behavior of the display logic, which does not consider
reads to be changes. This also matches the expectation of tests that
pass `--expect-no-changes` (and likely user intuition).
* Add the ability to log all engine events to a file.
The path to the file can be specified using the `--event-log` flag to
the CLI. The file will be truncated if it exists. Events are written as
a list of JSON values using the schema described by `pkg/apitype`.
* Expose update engine events to ExtraRuntimeValidation.
Just what it says on the tin. Events from previews are not exposed.
When installing plugins, we download the plugin into a temporary
folder, unpack it and then move that folder into its final
location (as an atomic operation). We do this so that if the download
fails in some way, we would not "poision" our plugin cache.
In order to ensure the move into the final location happens
atomically, we download the plugin into a folder with a `.tmpXXXXXX`
suffix inside the plugin root itself. However, we were not ignoring
this folder when enumarating plugins. This would cause some weird
behaviors:
- When a plugin was being installed `pulumi plugin ls` would show a
plugin with a version like `v1.0.0-rc.1.tmp123456`.
- For cases where the version of the plugin had applied
metadata (e.g. our alpha builds where we append the commit hash to the
version via `+gACBDEF123`) the `.tmpXXXXXX` suffix was considered part
of the build metadata. This would mean we could actually end up
selecting these plugins and trying to run them, which was not going to
work correctly (since they are in the process of being
downloaded). Because the way to hit this was using a dev release while
also trying to run a another program that used the same plugin, it was
unlikely that customers would hit this, but we would see this from
time to time in CI, where we run many pulumi programs at the same
time (the `examples` repo is one case where we'd often hit it).
Fixes: #1353
This was causing an error as follows:
```
error: could not create stack: validating stack properties: invalid stack name: a stack name may only contain alphanumeric, hyphens, underscores, or periods
```
Since we don't fully understand what was causing the assert to file,
instead of making changes to return an error when there are equal
plugins, keep the shape of the existing code and just do the work we
need to do to ensure that the assert message has actionable data for
us to look at.
If any templates are marked as `Important: true` then by default show only those templates along with an option to see additional templates.
Fixes#3094.
Present a warm welcome to users when they interactively login.
Also use this as an opportunity to present a "Tip of the Day" - which for now we will use to highlight auto-naming as this has been a common new user question.
Not all resource providers support Pulumi's Asset and Archive types. In
particular, the Kubernetes provider should reject any resource
definition that contains either of these types.
This commit will introduce two MarshalOptions that will make it easy for
the Kubernetes provider to guarantee that no properties of this type are
in a resource request, as it's deserializing the request from the
engine.
This matches what we tell our customers to do and makes some
downstream testing stuff for beta versions of our python packages
easier (since `pip` has a more straightforward package selection
algorithm than `pipenv`)
* Add support for filtering stacks by organization, tag
* Update CHANGELOG.md
* Address PR feedback
* Address even more PR feedback
* Support empty-string filters
With these changes, a user may explicitly set `deleteBeforeReplace` to
`false` in order to disable DBR behavior for a particular resource. This
is the SDK + CLI escape hatch for cases where the changes in
https://github.com/pulumi/pulumi-terraform/pull/465 cause undesirable
behavior.
The comment on QName explained that QNames match the Name regex, but
the regex written was slightly different to the actual Name regex
(missed that '.' is a valid char in each group).
This will allow us to install a latest version from PyPi or NPM
and then yarn link / pip install from local machine and test that
no changes are introduced when this occurs
If the flag is not passed, then the dependencies are installed
as before where we prefer a local package to be linked / installed
Since the filestate backend is now written using the go-cloud blob
abstraction, places where we were using functions from the `os`
package were very unlikely to be correct.
This change removes their uses in favor of APIs provided by
go-cloud (which sometimes requires more work than before).
For historical reasons, we used to need to require to load an existing
checkpoint to copy some data from it into the snapshot when saving a
new snapshot. The need for this was removed as part of the general
work in #2678, but we continued to load the checkpoint and then just
disregard the data that was returned (unless there was an error and
that error was not FileNotFound, in which case we would fail).
Our logic for checking if something was FileNotFound was correct when
we wrote it, but when we adopted go-cloud in order to have our
filestate backend also write to blob storage backends like S3, we
forgot that we had checks like `os.IsNotExists()` floating around
which were now incorrect. That meant if the file did not exist for
some reason, instead of going along as planned, we'd error out now
with an error saying something wasn't found.
When we write a checkpoint, we first "backed up" the initial version
by renaming it to include a `.bak` suffix, then we write the new file
in place. However, this can run afoul of eventual consistency models
like S3, since there will be a period of time in which a caller may
observe that the object is missing, even after a new version is
written (based on my understanding of [S3's consistency
model](https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyModel)
Since we no longer need to actually copy any information from the
previous checkpoint, we can simply remove the call entirely to load
it.
As a follow up, we need to audit places inside the filebased backend
that assume `os.*` functions are going to do what we want them to do,
since in general they will not.
Fixes#2714
Attempting to `pulumi stack rename` a stack which had been created but
never updated, when using the local backend, was broken because
code-paths were not hardened against the snapshot being `nil` (which
is the case for a stack before the initial deployment had been done).
Fixes#2654
* Allow resource IDs to change on reresh steps
This is a requirement for us to be able to move forward with
versions of the Terraform Azurerm provider. In v1.32.1, there was
a state migration that changed the ID format of the azure table
storage resource
We used to have a check in place for old ID being equal to new ID.
This has been changed now and we allow the change of ID to happen
in the RefreshStep
* Update pkg/resource/deploy/step.go
Co-Authored-By: Pat Gavlin <pat@pulumi.com>
When using StackReference, if the stack you reference contains any
secret outputs, we have to mark the entire `outputs` member as a
secret output. This is because we only track secretness on a per
`Output<T>` basis.
For `getSecret` and friends, however, we know the name of the output
you are looking up and we can be smarter about if the returned
`Output<T>` should be treated as a secret or not.
This change augments the provider for StackReference such that it also
returns a list of top level stack output names who's values contain
secrets. In the language SDKs, we use this information, when present,
to decide if we should return an `Output<T>` that is marked as a
secret or not. Since the SDK and CLI are independent components, care
is taken to ensure that when the CLI does not return this information,
we behave as we did before (i.e. if any output is a secret, we treat
every output as a secret).
Fixes#2744
If we don't process and report the stderr of `npm install`, the output
is "orphaned" during error condition, and only something like "exit code
1" is reported.
Empty `[]interface{}` values were being converted to array property
values with a `nil` element, and empty array property values were being
coverted to `nil` `[]interface{}` values. These changes fix the
converters to return empty but non-nil values in both cases.
This is part of the fix for
https://github.com/pulumi/pulumi-kubernetes/issues/693.
- Ensure that type assertions are guarded, and that incorrectly-typed
properties return errors rather than panicking
- Expand the asset/archive tests in the Node SDK to ensure that eventual
archives and assets serialize and deserialize correctly
Fixes#2836.
Fixes#3016.
Adds support for additional cloud secrets providers (AWS KMS, Azure KeyVault, Google Cloud KMS, and HashiCorp Vault) as the encryption backend for Pulumi secrets. This augments the previous choice between using the app.pulumi.com-managed secrets encryption or a fully-client-side local passphrase encryption.
This is implemented using the Go Cloud Development Kit support for pluggable secrets providers.
Like our cloud storage backend support which also uses Go Cloud Development Kit, this PR also bleeds through to users the URI scheme's that the Go CDK defines for specifying each of secrets providers - like `awskms://alias/LukeTesting?region=us-west-2` or `azurekeyvault://mykeyvaultname.vault.azure.net/keys/mykeyname`.
Also like our cloud storage backend support, this PR doesn't solve for how to configure the cloud provider client used to resolve the URIs above - the standard ambient credentials are used in both cases. Eventually, we will likely need to provide ways for both of these features to be configured independently of each other and of the providers used for resource provisioning.
A workaround for #2695
During the plugin installation, we create a temporary folder, unzip the binary, and then rename the folder to a permanent name. The rename fails 90% of the time with access denied. An immediate retry of renaming seems to always succeed.
This package's flags conflict with those in google/glog. Replace all
references to this package with references to
pulumi/pulumi/pkg/util/logging, and change that package to explicitly
call `flag.CommandLine.Parse` with an empty slice.
This should make it much easier to consume these packages in downstream
repos that have direct or indirect dependencies on google/glog.
These changes add support for passing `ignoreChanges` paths to resource
providers. This is intended to accommodate providers that perform diffs
between resource inputs and resource state (e.g. all Terraform-based
providers, the k8s provider when using API server dry-runs). These paths
are specified using the same syntax as the paths used in detailed diffs.
In addition to passing these paths to providers, the existing support
for `ignoreChanges` in inputs has been extended to accept paths rather
than top-level keys. It is an error to specify a path that is missing
one or more component in the old or new inputs.
Fixes#2936, #2663.
This removes the need for a replace directive in every downstream `go.mod`,
however it does not protect against the case of a `go.mod` being added upstream
with a different import path in future. This seems unlikely given the cadence of
work upstream, however.
Most of these options are typically left unset. In order to make it
easier to update the lifecycle test when adding new options, collect
them in a bag s.t. most callsites can go without being updated.
* Fix issue when logging out of local backend
* Put actual cloud URL into credentials.json
* Fix test on Windows; disable newly-enabled tests on Appveyor
* Keep track of the URL used to initialize local backend
If we encounter a provider with old inputs but no old outputs when reading
a checkpoint file, use the old inputs as the old outputs. This handles the
scenario where the CLI is being upgraded from a version that did not
reflect provider inputs to provider outputs, and a provider is being
upgraded from a version that did not implement `DiffConfig` to a version
that does.
Fixes https://github.com/pulumi/pulumi-kubernetes/issues/645.
Currently, `pulumi preview` fails immediately when any resource
definition in a Pulumi app is found to be in violation of a resource
policy. But, users would like `preview` to report as many policy
violations as it can before terminating with an error, so that they can
fix many of them before running `preview` again.
This commit will thus change `pulumi preview` to do this sort of
"batching" for policy violations. The engine will attempt to run the
entire preview step, validating every resource definition with the
relevant known resource policies, before finally reporting an error if
any violations are detected.
Fixespulumi/pulumi-policy#31
The current CLI update view attributes all policy violation errors to
the root Stack resource. This commit will attribute them to the resource
that violated the policy.
The reason for this mis-attribution is a simple bookkeeping error:
* Resource policies intercept and prevent RegisterResource requests for
when the resource in question violates some policy.
* The CLI "tree" view of resources "hides" rows for resources that have
not been registered. Thus, if a policy violation occurs for a
resource, it becomes "orphaned" and is attributed to the stack,
because there is no row for the resource that violates the policy.
The solution, thus, is to simply set the "hidden" flag to false when we
encounter a policy violation.
Fixespulumi/pulumi-policy#25
Fixes: #2151
This will allow us to be able to share the code that generates our
language providers. Currently there is a copy of the python code
generation in pulumi-kubernetes and also in pulumi-terraform
We want to be able to share these
Dynamic providers in Python.
This PR uses [dill](https://pypi.org/project/dill/) for code serialization, along with a customization to help ensure deterministic serialization results.
One notable limitation - which I believe is a general requirement of Python - is that any serialization of Python functions must serialize byte code, and byte code is not safely versioned across Python versions. So any resource created with Python `3.x.y` can only be updated by exactly the same version of Python. This is very constraining, but it's not clear there is any other option within the realm of what "dynamic providers" are as a feature. It is plausible that we could ensure that updates which only update the serialized provider can avoid calling the dynamic provider operations, so that version updates could still be accomplished. We can explore this separately.
```py
from pulumi import ComponentResource, export, Input, Output
from pulumi.dynamic import Resource, ResourceProvider, CreateResult, UpdateResult
from typing import Optional
from github import Github, GithubObject
auth = "<auth token>"
g = Github(auth)
class GithubLabelArgs(object):
owner: Input[str]
repo: Input[str]
name: Input[str]
color: Input[str]
description: Optional[Input[str]]
def __init__(self, owner, repo, name, color, description=None):
self.owner = owner
self.repo = repo
self.name = name
self.color = color
self.description = description
class GithubLabelProvider(ResourceProvider):
def create(self, props):
l = g.get_user(props["owner"]).get_repo(props["repo"]).create_label(
name=props["name"],
color=props["color"],
description=props.get("description", GithubObject.NotSet))
return CreateResult(l.name, {**props, **l.raw_data})
def update(self, id, _olds, props):
l = g.get_user(props["owner"]).get_repo(props["repo"]).get_label(id)
l.edit(name=props["name"],
color=props["color"],
description=props.get("description", GithubObject.NotSet))
return UpdateResult({**props, **l.raw_data})
def delete(self, id, props):
l = g.get_user(props["owner"]).get_repo(props["repo"]).get_label(id)
l.delete()
class GithubLabel(Resource):
name: Output[str]
color: Output[str]
url: Output[str]
description: Output[str]
def __init__(self, name, args: GithubLabelArgs, opts = None):
full_args = {'url':None, 'description':None, 'name':None, 'color':None, **vars(args)}
super().__init__(GithubLabelProvider(), name, full_args, opts)
label = GithubLabel("foo", GithubLabelArgs("lukehoban", "todo", "mylabel", "d94f0b"))
export("label_color", label.color)
export("label_url", label.url)
```
Fixes https://github.com/pulumi/pulumi/issues/2902.
This commit will allow the Pulumi service HTTP client to deserialize
HTTP responses that have bodies encoded as `application/octet-stream` to
be deserialized as `[]byte`.
This fixes a small bug that causes the HTTP client to fail under these
circumstances, as it expects any body to be JSON-deserializable.
This command will cause `pulumi policy publish` to behave in much the
same way `pulumi up` does -- if the policy program is in TypeScript, we
will use ts-node to attempt to compile in-process before executing, and
fall back to plain-old node.
We accomplish this by moving `cmd/run/run.ts` into a generic helper
package, `runtime/run.ts`, which slightly generalizes the use cases
supported (notably, allowing us to exec some program outside of the
context of a Pulumi stack).
This new package is then called by both `cmd/run/index.ts` and
`cmd/run-policy-pack/index.ts`.
This commit will implement the core business logic of `pulumi policy
publish` -- code to boot an analyzer, ask it for metadata about the
policies it contains, pack the code, and transmit all of this to the
Pulumi service.
When a user runs `pulumi policy publish`, we need to package up a
directory of code and send it to the service. We implemented this once
before, for PPCs, so this simply re-introduces that code as it was in
the commit that deleted it.
* Plumbing the custom timeouts from the engine to the providers
* Plumbing the CustomTimeouts through to the engine and adding test to show this
* Change the provider proto to include individual timeouts
* Plumbing the CustomTimeouts from the engine through to the Provider RPC interface
* Change how the CustomTimeouts are sent across RPC
These errors were spotted in testing. We can now see that the timeout
information is arriving in the RegisterResourceRequest
```
req=&pulumirpc.RegisterResourceRequest{
Type: "aws:s3/bucket:Bucket",
Name: "my-bucket",
Parent: "urn:pulumi:dev::aws-vpc::pulumi:pulumi:Stack::aws-vpc-dev",
Custom: true,
Object: &structpb.Struct{},
Protect: false,
Dependencies: nil,
Provider: "",
PropertyDependencies: {},
DeleteBeforeReplace: false,
Version: "",
IgnoreChanges: nil,
AcceptSecrets: true,
AdditionalSecretOutputs: nil,
Aliases: nil,
CustomTimeouts: &pulumirpc.RegisterResourceRequest_CustomTimeouts{
Create: 300,
Update: 400,
Delete: 500,
XXX_NoUnkeyedLiteral: struct {}{},
XXX_unrecognized: nil,
XXX_sizecache: 0,
},
XXX_NoUnkeyedLiteral: struct {}{},
XXX_unrecognized: nil,
XXX_sizecache: 0,
}
```
* Changing the design to use strings
* CHANGELOG entry to include the CustomTimeouts work
* Changing custom timeouts to be passed around the engine as converted value
We don't want to pass around strings - the user can provide it but we want
to make the engine aware of the timeout in seconds as a float64
A resource can be imported by setting the `import` property in the
resource options bag when instantiating a resource. In order to
successfully import a resource, its desired configuration (i.e. its
inputs) must not differ from its actual configuration (i.e. its state)
as calculated by the resource's provider.
There are a few interesting state transitions hiding here when importing
a resource:
1. No prior resource exists in the checkpoint file. In this case, the
resource is simply imported.
2. An external resource exists in the checkpoint file. In this case, the
resource is imported and the old external state is discarded.
3. A non-external resource exists in the checkpoint file and its ID is
different from the ID to import. In this case, the new resource is
imported and the old resource is deleted.
4. A non-external resource exists in the checkpoint file, but the ID is
the same as the ID to import. In this case, the import ID is ignored
and the resource is treated as it would be in all cases except for
changes that would replace the resource. In that case, the step
generator issues an error that indicates that the import ID should be
removed: were we to move forward with the replace, the new state of
the stack would fall under case (3), which is almost certainly not
what the user intends.
Fixes#1662.
Instead of simply converting a detailed diff entry that indicates an
update to an entire composite value as a simple old/new value diff,
compute the nested diff. This alllows us to render a per-element diff
for the nested object rather than simply displaying the new and the old
composite values.
This is necessary in order to improve diff rendering once
pulumi/pulumi-terraform#403 has been rolled out.
Thse changes make a subtle but critical adjustment to the process the
Pulumi engine uses to determine whether or not a difference exists
between a resource's actual and desired states, and adjusts the way this
difference is calculated and displayed accordingly.
Today, the Pulumi engine get the first chance to decide whether or not
there is a difference between a resource's actual and desired states. It
does this by comparing the current set of inputs for a resource (i.e.
the inputs from the running Pulumi program) with the last set of inputs
used to update the resource. If there is no difference between the old
and new inputs, the engine decides that no change is necessary without
consulting the resource's provider. Only if there are changes does the
engine consult the resource's provider for more information about the
difference. This can be problematic for a number of reasons:
- Not all providers do input-input comparison; some do input-state
comparison
- Not all providers are able to update the last deployed set of inputs
when performing a refresh
- Some providers--either intentionally or due to bugs--may see changes
in resources whose inputs have not changed
All of these situations are confusing at the very least, and the first
is problematic with respect to correctness. Furthermore, the display
code only renders diffs it observes rather than rendering the diffs
observed by the provider, which can obscure the actual changes detected
at runtime.
These changes address both of these issues:
- Rather than comparing the current inputs against the last inputs
before calling a resource provider's Diff function, the engine calls
the Diff function in all cases.
- Providers may now return a list of properties that differ between the
requested and actual state and the way in which they differ. This
information will then be used by the CLI to render the diff
appropriately. A provider may also indicate that a particular diff is
between old and new inputs rather than old state and new inputs.
Fixes#2453.
