* Fix logic to determine PRNumber and BuildURL for Az Pipelines.
* Update changelog
* Set the BranchName to the PR source branch if PRNumber is not empty.
* added support for using GOOGLE_CREDENTIALS environment variable for authenticating with gs:// file state
* modified the change to fix#2791 as well
* fixed a small bug
* fixed linter error
* added code comments
* Update pkg/backend/filestate/gcpauth.go
Co-Authored-By: CyrusNajmabadi <cyrus.najmabadi@gmail.com>
* Parse provided backend url to check if scheme is gs://
* Update changelog
A regression was introduced when we added support for non-Node.js Pulumi programs to run Policy Packs. With that change, we now pass the Policy Pack's full path as the plugin's pwd (so that it would load the `@pulumi/pulumi/cmd/run-policy-pack` Node module from the Policy Pack's node_modules rather than the program's node_modules), but we also pass the path to the policy pack as well. If the path is a full rooted path, this would work fine, and that's what our tests do. However, if a relative path is specified, then it will be looking to load the Policy Pack relative to the pwd, which doesn't produce a correct path leading to failures trying to load the Policy Pack.
Since the pwd is the policy pack path, we can simply pass the path as `"."` to the analyzer plugin, and it will load the policy pack in its pwd.
This change adds support for setting `PULUMI_PREFER_YARN` to true to opt-in to preferring `yarn` over `npm` when installing Node.js dependencies (and publishing Policy Packs). If `PULUMI_PREFER_YARN` is truthy, but `yarn` cannot be found on `$PATH`, we fallback to using `npm`. If `npm` can't be found on `$PATH`, we provide a more helpful error message.
Add support for a test option that indicates that failed update steps
should be retried. Currently the maximum number of retries (3) is not
configurable.
The text "Plan applied failed: " is pretty inscrutable given our
current system. While both "plan" and "apply" are concepts inside the
the implementation of the CLI, we usually talk in terms of `preview`
and `update`. I suspect there are some cases where this prefix is not
100% technically correct, and if there's a better short way of saying
something more correct, I would love to adopt that instead, but as is,
I would really love to get rid of the "Plan apply failed" text in our
system, it pains me every time I read it.
Codepaths which could result in a hang will print a message to the console indicating the problem, along with a link to documentation on how to restructure code to best address it.
`StackReference.getOutputSync` and `requireOutputSync` have been deprecated as they may cause hangs on some combinations of Node and certain OS platforms. `StackReference.getOutput` and `requireOutput` should be used instead.
To allow Policy Packs to run against Pulumi programs written in all languages, we now look for the `@pulumi/pulumi/cmd/run-policy-pack` module in the Policy Pack's node_modules (instead of in the Pulumi program's node_modules; which doesn't exist for non-node languages). The `@pulumi/policy` library that a Policy Pack will depend on should already depend on a recent enough version of `@pulumi/pulumi`. When we can't find the module, it's more likely it's due to the dependencies for the Policy Pack not being installed. Provide a more helpful error message in this case.
- If an untargeted create would not affect the inputs of any targeted
resources, do not fail the update. Untargeted creates that are
directly dependend on by targeted resources will still cause failures
that inform the user to add the untargeted resources to the --target
list.
- Users may now pass the `--target-dependents` flag to allow targeted
destroys to automatically target dependents that must be destroyed in
order to destroy an explicitly targeted resource.
- Use a mutex + condition variable instead of a channel for
synchronizaiton in order to allow multiple calls to resolve/reject
- Properly handle outputs that are resolved to other outputs, especially
if those outputs are not of exactly type Output
- Remove the Value() methods that allowed prompt access to output values
- Add variants of `Apply` that take a context parameter
- Ensure that resource outputs properly incorporate their resource as
a dependency
- Make `Output` a plain struct. Uninitialized outputs will be treated as
resolved and unknown. This makes conversions between output
types more ergonomic.
Contributes to #3492.
These changes restore a more-correct version of the behavior that was
disabled with #3014. The original implementation of this behavior was
done in the SDKs, which do not have access to the complete inputs for a
resource (in particular, default values filled in by the provider during
`Check` are not exposed to the SDK). This lack of information meant that
the resolved output values could disagree with the typings present in
a provider SDK. Exacerbating this problem was the fact that unknown
values were dropped entirely, causing `undefined` values to appear in
unexpected places.
By doing this in the engine and allowing unknown values to be
represented in a first-class manner in the SDK, we can attack both of
these issues.
Although this behavior is not _strictly_ consistent with respect to the
resource model--in an update, a resource's output properties will come
from its provider and may differ from its input properties--this
behavior was present in the product for a fairly long time without
significant issues. In the future, we may be able to improve the
accuracy of resource outputs during a preview by allowing the provider
to dry-run CRUD operations and return partially-known values where
possible.
These changes also introduce new APIs in the Node and Python SDKs
that work with unknown values in a first-class fashion:
- A new parameter to the `apply` function that indicates that the
callback should be run even if the result of the apply contains
unknown values
- `containsUnknowns` and `isUnknown`, which return true if a value
either contains nested unknown values or is exactly an unknown value
- The `Unknown` type, which represents unknown values
The primary use case for these APIs is to allow nested, properties with
known values to be accessed via the lifted property accessor even when
the containing property is not fully know. A common example of this
pattern is the `metadata.name` property of a Kubernetes `Namespace`
object: while other properties of the `metadata` bag may be unknown,
`name` is often known. These APIs allow `ns.metadata.name` to return a
known value in this case.
In order to avoid exposing downlevel SDKs to unknown values--a change
which could break user code by exposing it to unexpected values--a
language SDK must indicate whether or not it supports first-class
unknown values as part of each `RegisterResourceRequest`.
These changes also allow us to avoid breaking user code with the new
behavior introduced by the prior commit.
Fixes#3190.
Adds a new experimental `pulumi watch` CLI command which can be used for inner loop development on a Pulumi stack. This command is only available currently via `PULUMI_EXPERIMENTAL=true` while in active development.
The `watch` command does the following:
1. Watches the workspace (the tree rooted at the `Pulumi.yaml` file) for changes
2. Triggers an `update` to the stack whenever there is a change
3. Streams output containing summaries of key update events as well as logs from any resources under management into a combined CLI output
Part of https://github.com/pulumi/pulumi/issues/3448.
The PULUMI_EXPERIMENTAL flag also makes`query` and `policy` available.
This change adds support for lists and maps in config. We now allow
lists/maps (and nested structures) in `Pulumi.<stack>.yaml` (or
`Pulumi.<stack>.json`; yes, we currently support that).
For example:
```yaml
config:
proj:blah:
- a
- b
- c
proj:hello: world
proj:outer:
inner: value
proj:servers:
- port: 80
```
While such structures could be specified in the `.yaml` file manually,
we support setting values in maps/lists from the command line.
As always, you can specify single values with:
```shell
$ pulumi config set hello world
```
Which results in the following YAML:
```yaml
proj:hello world
```
And single value secrets via:
```shell
$ pulumi config set --secret token shhh
```
Which results in the following YAML:
```yaml
proj:token:
secure: v1:VZAhuroR69FkEPTk:isKafsoZVMWA9pQayGzbWNynww==
```
Values in a list can be set from the command line using the new
`--path` flag, which indicates the config key contains a path to a
property in a map or list:
```shell
$ pulumi config set --path names[0] a
$ pulumi config set --path names[1] b
$ pulumi config set --path names[2] c
```
Which results in:
```yaml
proj:names
- a
- b
- c
```
Values can be obtained similarly:
```shell
$ pulumi config get --path names[1]
b
```
Or setting values in a map:
```shell
$ pulumi config set --path outer.inner value
```
Which results in:
```yaml
proj:outer:
inner: value
```
Of course, setting values in nested structures is supported:
```shell
$ pulumi config set --path servers[0].port 80
```
Which results in:
```yaml
proj:servers:
- port: 80
```
If you want to include a period in the name of a property, it can be
specified as:
```
$ pulumi config set --path 'nested["foo.bar"]' baz
```
Which results in:
```yaml
proj:nested:
foo.bar: baz
```
Examples of valid paths:
- root
- root.nested
- 'root["nested"]'
- root.double.nest
- 'root["double"].nest'
- 'root["double"]["nest"]'
- root.array[0]
- root.array[100]
- root.array[0].nested
- root.array[0][1].nested
- root.nested.array[0].double[1]
- 'root["key with \"escaped\" quotes"]'
- 'root["key with a ."]'
- '["root key with \"escaped\" quotes"].nested'
- '["root key with a ."][100]'
Note: paths that contain quotes can be surrounded by single quotes.
When setting values with `--path`, if the value is `"false"` or
`"true"`, it will be saved as the boolean value, and if it is
convertible to an integer, it will be saved as an integer.
Secure values are supported in lists/maps as well:
```shell
$ pulumi config set --path --secret tokens[0] shh
```
Will result in:
```yaml
proj:tokens:
- secure: v1:wpZRCe36sFg1RxwG:WzPeQrCn4n+m4Ks8ps15MxvFXg==
```
Note: maps of length 1 with a key of “secure” and string value are
reserved for storing secret values. Attempting to create such a value
manually will result in an error:
```shell
$ pulumi config set --path parent.secure foo
error: "secure" key in maps of length 1 are reserved
```
**Accessing config values from the command line with JSON**
```shell
$ pulumi config --json
```
Will output:
```json
{
"proj:hello": {
"value": "world",
"secret": false,
"object": false
},
"proj:names": {
"value": "[\"a\",\"b\",\"c\"]",
"secret": false,
"object": true,
"objectValue": [
"a",
"b",
"c"
]
},
"proj:nested": {
"value": "{\"foo.bar\":\"baz\"}",
"secret": false,
"object": true,
"objectValue": {
"foo.bar": "baz"
}
},
"proj:outer": {
"value": "{\"inner\":\"value\"}",
"secret": false,
"object": true,
"objectValue": {
"inner": "value"
}
},
"proj:servers": {
"value": "[{\"port\":80}]",
"secret": false,
"object": true,
"objectValue": [
{
"port": 80
}
]
},
"proj:token": {
"secret": true,
"object": false
},
"proj:tokens": {
"secret": true,
"object": true
}
}
```
If the value is a map or list, `"object"` will be `true`. `"value"` will
contain the object as serialized JSON and a new `"objectValue"` property
will be available containing the value of the object.
If the object contains any secret values, `"secret"` will be `true`, and
just like with scalar values, the value will not be outputted unless
`--show-secrets` is specified.
**Accessing config values from Pulumi programs**
Map/list values are available to Pulumi programs as serialized JSON, so
the existing
`getObject`/`requireObject`/`getSecretObject`/`requireSecretObject`
functions can be used to retrieve such values, e.g.:
```typescript
import * as pulumi from "@pulumi/pulumi";
interface Server {
port: number;
}
const config = new pulumi.Config();
const names = config.requireObject<string[]>("names");
for (const n of names) {
console.log(n);
}
const servers = config.requireObject<Server[]>("servers");
for (const s of servers) {
console.log(s.port);
}
```
Allow the user to specify a set of resources to replace via the
`--replace` flag on the CLI. This can be combined with `--target` to
replace a specific set of resources without changing any other
resources. `--target-replace` is shorthand for `--replace urn --target urn`.
Fixes#2643.
If a stack output includes a `Resource`, we will as of a recent change
always show the output diff, but this diff will potentially include
unknowns, leading to spurious output like:
```
+ namePrefix : output<string>
```
These changes supress these diffs by adding a special key to the POJO
we generate for resources *during preview only* that indicates that the
POJO represents a Pulumi resource, then stripping all adds of unknown
values from diffs for objects marked with that key.
Fixes#3314.
This is needed to add support for objects (maps/lists) to config. Once this has been merged, we can merge the associated change in the service, then merge the overall change in the CLI.
`engine.Query` queues up a bunch of `defer` functions that (among other
things) wait for various resources to clean themselves up.
In `query`, we have a "naive" (read: bad) `eventEmitter`, whose `done`
channel is set to `nil`. Therefore, the relevent `defer` waits for an
event on a `nil` channel which in Go will famously simply hang forever.
This commit will correct this by setting this channel appropriately, so
that it signals appropriately when it's done.
This commit will introduce the ability to load providers in `query`
mode.
Previously, `query` mode has been effectively a stand-alone execution
environment for language hosts, running without (e.g.) the
`StepExecutor` and similar engine facilities, but with some minimal
constructs hooked up, notably the ability to retrieve stack snapshots
from the backend for querying.
This commit extends this functionality somewhat by allowing `query` to
load Pulumi resource providers, and to run `Invoke` on them. This will
allow us, in the future, to "query" resource providers in the same way
we can query stack snapshots.
This method can be used to check whether or not a URN is well-formed.
This is used by the provider reference parser to avoid panicking on
malformed URNs.
- The length of the text content (i.e. the content of a colorized string
that is not control sequences) was not being correctly tracked. This
caused the "status" column of the progress display to overflow.
- Colorization was unconditionally disabled on Windows. When we were
using loreley, we had set the global colorization flag s.t.
colorization on Windows _should_ have been disabled, but we overrode
this flag each time we actually colorized anything.
Fixes#3378.
This avoids unnecessary blocking inside pre/post-step callbacks if the
reader on the other side of the event channel is slow.
We do not use a buffered channel in the event pipe because it is
empirically less likely that the goroutine reading from a buffered
channel will be scheduled when new data is placed in the channel. In the
case of our event system in which events will not be delivered to the
service and display until the copying goroutine is scheduled, this can
lead to unacceptable delay between the send of the original event and
its output.
- Cache the username and last verified time associated with each logged-in
backend
- In the HTTP backend, verify the access token explicitly at most once
per hour
This trades off a little bit of usability for improved inner-loop
latency: if a user's API token becomes invalid less than an hour after
it was last verified, the user will see 4xx errors when attempting stack
operations rather than seeing the login prompt.
The current pattern of using backend.Stack values in the CLI but
accepting backend.StackReference values in backend methods leads to
frequent transitions between these types. In the case of the HTTP
backend, the transition from a StackReference to a Stack requires an API
call. These changes refactor the backend.Backend API such that most of
its methods accept backend.Stack values in place of
backend.StackReference values, which avoids these transitions.
This removes two calls to the getStack API on the startup path of a
`pulumi preview`.
The dependency graph used to determine the set of resources that
depend on a resource being DBR'd is constructured from the list of
resource states present in the old snapshot. However, the dependencies
of resources that are present in both the old snapshot and the current
plan can be different, which in turn can cause the engine to make
incorrect decisions during DBR with respect to which resources need to
be replaced. For example, consider the following program:
```
var resA = new Resource("a", {dbr: "foo"});
var resB = new Resource("b", {dbr: resA.prop});
```
If this program is then changed to:
```
var resB = new Resource("b", {dbr: "<literal value of resA.prop>"});
var resA = new Resource("a", {dbr: "bar"});
```
The engine will first decide to make no changes to "b", as its input
property values have not changed. "b" has changed, however, such that it
no longer has a dependency on "a".
The engine will then decide to DBR "a". In the process, it will
determine that it first needs to delete "b", because the state for "b"
that is used when calculating "a"'s dependents does not reflect the
changes made during the plan.
To fix this issue, we rely on the observation that dependents can only
have been _removed_ from the base dependency graph: for a dependent to
have been added, it would have had to have been registered prior to the
root--a resource it depends on--which is not a valid operation. This
means that any resources that depend on the root must not yet have
been registered, which in turn implies that resources that have already
been registered must not depend on the root. Thus, we ignore these
resources if they are encountered while walking the old dependency graph
to determine the set of dependents.
Loreley's approach to colorization via Go templates has significant
overhead when colorization is used extensively.
Also, stop parsing colorized text with regexes and use the actual
parser.
* Fix lint warnings
* Improve error message when using PAC on old SDK
* Update pkg/resource/plugin/analyzer_plugin.go
Co-Authored-By: Justin Van Patten <jvp@justinvp.com>
Introduces `PULUMI_HOME` environment variable which points to a path to the path to `.pulumi` folder. Defaults to `<user's home dir> + ".pulumi"` if not specified.
Fixes#2966. In addition to plugins, it "moves" the credentials file, templates, workspaces.
`bin` folder is intact: to move it, we need to adjust all installation scripts to respect `PULUMI_HOME` and put executables in the proper `bin` folder.
These changes restore a more-correct version of the behavior that was
disabled with #3014. The original implementation of this behavior was
done in the SDKs, which do not have access to the complete inputs for a
resource (in particular, default values filled in by the provider during
`Check` are not exposed to the SDK). This lack of information meant that
the resolved output values could disagree with the typings present in
a provider SDK. Exacerbating this problem was the fact that unknown
values were dropped entirely, causing `undefined` values to appear in
unexpected places.
By doing this in the engine and allowing unknown values to be
represented in a first-class manner in the SDK, we can attack both of
these issues.
Although this behavior is not _strictly_ consistent with respect to the
resource model--in an update, a resource's output properties will come
from its provider and may differ from its input properties--this
behavior was present in the product for a fairly long time without
significant issues. In the future, we may be able to improve the
accuracy of resource outputs during a preview by allowing the provider
to dry-run CRUD operations and return partially-known values where
possible.
These changes also introduce new APIs in the Node and Python SDKs
that work with unknown values in a first-class fashion:
- A new parameter to the `apply` function that indicates that the
callback should be run even if the result of the apply contains
unknown values
- `containsUnknowns` and `isUnknown`, which return true if a value
either contains nested unknown values or is exactly an unknown value
- The `Unknown` type, which represents unknown values
The primary use case for these APIs is to allow nested, properties with
known values to be accessed via the lifted property accessor even when
the containing property is not fully know. A common example of this
pattern is the `metadata.name` property of a Kubernetes `Namespace`
object: while other properties of the `metadata` bag may be unknown,
`name` is often known. These APIs allow `ns.metadata.name` to return a
known value in this case.
In order to avoid exposing downlevel SDKs to unknown values--a change
which could break user code by exposing it to unexpected values--a
language SDK must indicate whether or not it supports first-class
unknown values as part of each `RegisterResourceRequest`.
These changes also allow us to avoid breaking user code with the new
behavior introduced by the prior commit.
Fixes#3190.
Adds the ability to provide `transformations` to modify the properties and resource options that will be used for any child resource of a component or stack.
This offers an "escape hatch" to modify the behaviour of a component by peeking behind it's abstraction. For example, it can be used to add a resource option (`additionalSecretOutputs`, `aliases`, `protect`, etc.) to a specific known child of a component, or to modify some input property to a child resource if the component does not (yet) expose the ability to control that input directly. It could also be used for more interesting scenarios - such as:
1. Automatically applying tags to all resources that support them in a stack (or component)
2. Injecting real dependencies between stringly-referenced resources in a Helm Chart
3. Injecting explicit names using a preferred naming convention across all resources in a stack
4. Injecting `import` onto all resources by doing a lookup into a name=>id mapping
Because this feature makes it possible to peek behind a component abstraction, it must be used with care in cases where the component is versioned independently of the use of transformations. Also, this can result in "spooky action at a distance", so should be used judiciously. That said - this can be used as an escape hatch to unblock a wide variety of common use cases without waiting on changes to be made in a component implementation.
Each transformation is passed the `resource`, `name`, `type`, `props` and `opts` that are passed into the `Resource` constructor for any resource descended from the resource that has the transformation applied. The transformation callback can optionally return alternate versions of the `props` and `opts` to be used in place of the original values provided to the resource constructor.
Fixes#2068.
This caching is enabled by wrapping the `secrets.Manager` returned by
`DefaultSecretsProvider.OfType` in an outer `secrets.Manager` that
cooperates with `stack.{Serialize,Deserialize}PropertyValue`. Ciphertext
is cached on a per-secret-instance basis (i.e. not a per-plaintext-value
basis). Cached ciphertext is only reused if the plaintext for the secret
value has not changed. Entries are inserted into the cache upon both
encryption and decryption so that values that originated from ciphertext
and that have not changed can aoid re-encryption.
Contributes to #3178.
* Fix some tracing issues.
- Add endpoints for `startUpdate` and `postEngineEventsBatch` so that
spans for these invocations have proper names
- Inject a tracing span when walking a plan so that resource operations
are properly parented
- When handling gRPC calls, inject a tracing span into the call's
metadata if no span is already present so that resource monitor and
engine spans are properly parented
- Do not trace client gRPC invocations of the empty method so that these
calls (which are used to determine server availability) do not muddy
the trace. Note that I tried parenting these spans appropriately, but
doing so broke the trace entirely.
With these changes, the only unparented span in a typical Pulumi
invocation is a single call to `getUser`. This span is unparented
because that call does not have a context available. Plumbing a context
into that particular call is surprisingly tricky, as it is often called
by other context-less functions.
* Make tracing support more flexible.
- Add support for writing trace data to a local file using Appdash
- Add support for viewing Appdash traces via the CLI
