* Fix lint warnings
* Improve error message when using PAC on old SDK
* Update pkg/resource/plugin/analyzer_plugin.go
Co-Authored-By: Justin Van Patten <jvp@justinvp.com>
Introduces `PULUMI_HOME` environment variable which points to a path to the path to `.pulumi` folder. Defaults to `<user's home dir> + ".pulumi"` if not specified.
Fixes#2966. In addition to plugins, it "moves" the credentials file, templates, workspaces.
`bin` folder is intact: to move it, we need to adjust all installation scripts to respect `PULUMI_HOME` and put executables in the proper `bin` folder.
These changes restore a more-correct version of the behavior that was
disabled with #3014. The original implementation of this behavior was
done in the SDKs, which do not have access to the complete inputs for a
resource (in particular, default values filled in by the provider during
`Check` are not exposed to the SDK). This lack of information meant that
the resolved output values could disagree with the typings present in
a provider SDK. Exacerbating this problem was the fact that unknown
values were dropped entirely, causing `undefined` values to appear in
unexpected places.
By doing this in the engine and allowing unknown values to be
represented in a first-class manner in the SDK, we can attack both of
these issues.
Although this behavior is not _strictly_ consistent with respect to the
resource model--in an update, a resource's output properties will come
from its provider and may differ from its input properties--this
behavior was present in the product for a fairly long time without
significant issues. In the future, we may be able to improve the
accuracy of resource outputs during a preview by allowing the provider
to dry-run CRUD operations and return partially-known values where
possible.
These changes also introduce new APIs in the Node and Python SDKs
that work with unknown values in a first-class fashion:
- A new parameter to the `apply` function that indicates that the
callback should be run even if the result of the apply contains
unknown values
- `containsUnknowns` and `isUnknown`, which return true if a value
either contains nested unknown values or is exactly an unknown value
- The `Unknown` type, which represents unknown values
The primary use case for these APIs is to allow nested, properties with
known values to be accessed via the lifted property accessor even when
the containing property is not fully know. A common example of this
pattern is the `metadata.name` property of a Kubernetes `Namespace`
object: while other properties of the `metadata` bag may be unknown,
`name` is often known. These APIs allow `ns.metadata.name` to return a
known value in this case.
In order to avoid exposing downlevel SDKs to unknown values--a change
which could break user code by exposing it to unexpected values--a
language SDK must indicate whether or not it supports first-class
unknown values as part of each `RegisterResourceRequest`.
These changes also allow us to avoid breaking user code with the new
behavior introduced by the prior commit.
Fixes#3190.
Adds the ability to provide `transformations` to modify the properties and resource options that will be used for any child resource of a component or stack.
This offers an "escape hatch" to modify the behaviour of a component by peeking behind it's abstraction. For example, it can be used to add a resource option (`additionalSecretOutputs`, `aliases`, `protect`, etc.) to a specific known child of a component, or to modify some input property to a child resource if the component does not (yet) expose the ability to control that input directly. It could also be used for more interesting scenarios - such as:
1. Automatically applying tags to all resources that support them in a stack (or component)
2. Injecting real dependencies between stringly-referenced resources in a Helm Chart
3. Injecting explicit names using a preferred naming convention across all resources in a stack
4. Injecting `import` onto all resources by doing a lookup into a name=>id mapping
Because this feature makes it possible to peek behind a component abstraction, it must be used with care in cases where the component is versioned independently of the use of transformations. Also, this can result in "spooky action at a distance", so should be used judiciously. That said - this can be used as an escape hatch to unblock a wide variety of common use cases without waiting on changes to be made in a component implementation.
Each transformation is passed the `resource`, `name`, `type`, `props` and `opts` that are passed into the `Resource` constructor for any resource descended from the resource that has the transformation applied. The transformation callback can optionally return alternate versions of the `props` and `opts` to be used in place of the original values provided to the resource constructor.
Fixes#2068.
This caching is enabled by wrapping the `secrets.Manager` returned by
`DefaultSecretsProvider.OfType` in an outer `secrets.Manager` that
cooperates with `stack.{Serialize,Deserialize}PropertyValue`. Ciphertext
is cached on a per-secret-instance basis (i.e. not a per-plaintext-value
basis). Cached ciphertext is only reused if the plaintext for the secret
value has not changed. Entries are inserted into the cache upon both
encryption and decryption so that values that originated from ciphertext
and that have not changed can aoid re-encryption.
Contributes to #3178.
* Fix some tracing issues.
- Add endpoints for `startUpdate` and `postEngineEventsBatch` so that
spans for these invocations have proper names
- Inject a tracing span when walking a plan so that resource operations
are properly parented
- When handling gRPC calls, inject a tracing span into the call's
metadata if no span is already present so that resource monitor and
engine spans are properly parented
- Do not trace client gRPC invocations of the empty method so that these
calls (which are used to determine server availability) do not muddy
the trace. Note that I tried parenting these spans appropriately, but
doing so broke the trace entirely.
With these changes, the only unparented span in a typical Pulumi
invocation is a single call to `getUser`. This span is unparented
because that call does not have a context available. Plumbing a context
into that particular call is surprisingly tricky, as it is often called
by other context-less functions.
* Make tracing support more flexible.
- Add support for writing trace data to a local file using Appdash
- Add support for viewing Appdash traces via the CLI
Fixes: #2319
In #2319, a user is hitting the gRPC limit on the message size the
server can receive when uploading ec2 user-data
This commit doubles the limit that can be sent from `1024*1024*4` to
`1024*1024*8`
This matches the behavior of the display logic, which does not consider
reads to be changes. This also matches the expectation of tests that
pass `--expect-no-changes` (and likely user intuition).
* Add the ability to log all engine events to a file.
The path to the file can be specified using the `--event-log` flag to
the CLI. The file will be truncated if it exists. Events are written as
a list of JSON values using the schema described by `pkg/apitype`.
* Expose update engine events to ExtraRuntimeValidation.
Just what it says on the tin. Events from previews are not exposed.
When installing plugins, we download the plugin into a temporary
folder, unpack it and then move that folder into its final
location (as an atomic operation). We do this so that if the download
fails in some way, we would not "poision" our plugin cache.
In order to ensure the move into the final location happens
atomically, we download the plugin into a folder with a `.tmpXXXXXX`
suffix inside the plugin root itself. However, we were not ignoring
this folder when enumarating plugins. This would cause some weird
behaviors:
- When a plugin was being installed `pulumi plugin ls` would show a
plugin with a version like `v1.0.0-rc.1.tmp123456`.
- For cases where the version of the plugin had applied
metadata (e.g. our alpha builds where we append the commit hash to the
version via `+gACBDEF123`) the `.tmpXXXXXX` suffix was considered part
of the build metadata. This would mean we could actually end up
selecting these plugins and trying to run them, which was not going to
work correctly (since they are in the process of being
downloaded). Because the way to hit this was using a dev release while
also trying to run a another program that used the same plugin, it was
unlikely that customers would hit this, but we would see this from
time to time in CI, where we run many pulumi programs at the same
time (the `examples` repo is one case where we'd often hit it).
Fixes: #1353
This was causing an error as follows:
```
error: could not create stack: validating stack properties: invalid stack name: a stack name may only contain alphanumeric, hyphens, underscores, or periods
```
Since we don't fully understand what was causing the assert to file,
instead of making changes to return an error when there are equal
plugins, keep the shape of the existing code and just do the work we
need to do to ensure that the assert message has actionable data for
us to look at.
If any templates are marked as `Important: true` then by default show only those templates along with an option to see additional templates.
Fixes#3094.
Present a warm welcome to users when they interactively login.
Also use this as an opportunity to present a "Tip of the Day" - which for now we will use to highlight auto-naming as this has been a common new user question.
Not all resource providers support Pulumi's Asset and Archive types. In
particular, the Kubernetes provider should reject any resource
definition that contains either of these types.
This commit will introduce two MarshalOptions that will make it easy for
the Kubernetes provider to guarantee that no properties of this type are
in a resource request, as it's deserializing the request from the
engine.
This matches what we tell our customers to do and makes some
downstream testing stuff for beta versions of our python packages
easier (since `pip` has a more straightforward package selection
algorithm than `pipenv`)
* Add support for filtering stacks by organization, tag
* Update CHANGELOG.md
* Address PR feedback
* Address even more PR feedback
* Support empty-string filters
With these changes, a user may explicitly set `deleteBeforeReplace` to
`false` in order to disable DBR behavior for a particular resource. This
is the SDK + CLI escape hatch for cases where the changes in
https://github.com/pulumi/pulumi-terraform/pull/465 cause undesirable
behavior.
The comment on QName explained that QNames match the Name regex, but
the regex written was slightly different to the actual Name regex
(missed that '.' is a valid char in each group).
This will allow us to install a latest version from PyPi or NPM
and then yarn link / pip install from local machine and test that
no changes are introduced when this occurs
If the flag is not passed, then the dependencies are installed
as before where we prefer a local package to be linked / installed
Since the filestate backend is now written using the go-cloud blob
abstraction, places where we were using functions from the `os`
package were very unlikely to be correct.
This change removes their uses in favor of APIs provided by
go-cloud (which sometimes requires more work than before).
For historical reasons, we used to need to require to load an existing
checkpoint to copy some data from it into the snapshot when saving a
new snapshot. The need for this was removed as part of the general
work in #2678, but we continued to load the checkpoint and then just
disregard the data that was returned (unless there was an error and
that error was not FileNotFound, in which case we would fail).
Our logic for checking if something was FileNotFound was correct when
we wrote it, but when we adopted go-cloud in order to have our
filestate backend also write to blob storage backends like S3, we
forgot that we had checks like `os.IsNotExists()` floating around
which were now incorrect. That meant if the file did not exist for
some reason, instead of going along as planned, we'd error out now
with an error saying something wasn't found.
When we write a checkpoint, we first "backed up" the initial version
by renaming it to include a `.bak` suffix, then we write the new file
in place. However, this can run afoul of eventual consistency models
like S3, since there will be a period of time in which a caller may
observe that the object is missing, even after a new version is
written (based on my understanding of [S3's consistency
model](https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyModel)
Since we no longer need to actually copy any information from the
previous checkpoint, we can simply remove the call entirely to load
it.
As a follow up, we need to audit places inside the filebased backend
that assume `os.*` functions are going to do what we want them to do,
since in general they will not.
Fixes#2714
Attempting to `pulumi stack rename` a stack which had been created but
never updated, when using the local backend, was broken because
code-paths were not hardened against the snapshot being `nil` (which
is the case for a stack before the initial deployment had been done).
Fixes#2654
* Allow resource IDs to change on reresh steps
This is a requirement for us to be able to move forward with
versions of the Terraform Azurerm provider. In v1.32.1, there was
a state migration that changed the ID format of the azure table
storage resource
We used to have a check in place for old ID being equal to new ID.
This has been changed now and we allow the change of ID to happen
in the RefreshStep
* Update pkg/resource/deploy/step.go
Co-Authored-By: Pat Gavlin <pat@pulumi.com>
When using StackReference, if the stack you reference contains any
secret outputs, we have to mark the entire `outputs` member as a
secret output. This is because we only track secretness on a per
`Output<T>` basis.
For `getSecret` and friends, however, we know the name of the output
you are looking up and we can be smarter about if the returned
`Output<T>` should be treated as a secret or not.
This change augments the provider for StackReference such that it also
returns a list of top level stack output names who's values contain
secrets. In the language SDKs, we use this information, when present,
to decide if we should return an `Output<T>` that is marked as a
secret or not. Since the SDK and CLI are independent components, care
is taken to ensure that when the CLI does not return this information,
we behave as we did before (i.e. if any output is a secret, we treat
every output as a secret).
Fixes#2744
If we don't process and report the stderr of `npm install`, the output
is "orphaned" during error condition, and only something like "exit code
1" is reported.
Empty `[]interface{}` values were being converted to array property
values with a `nil` element, and empty array property values were being
coverted to `nil` `[]interface{}` values. These changes fix the
converters to return empty but non-nil values in both cases.
This is part of the fix for
https://github.com/pulumi/pulumi-kubernetes/issues/693.
- Ensure that type assertions are guarded, and that incorrectly-typed
properties return errors rather than panicking
- Expand the asset/archive tests in the Node SDK to ensure that eventual
archives and assets serialize and deserialize correctly
Fixes#2836.
Fixes#3016.
Adds support for additional cloud secrets providers (AWS KMS, Azure KeyVault, Google Cloud KMS, and HashiCorp Vault) as the encryption backend for Pulumi secrets. This augments the previous choice between using the app.pulumi.com-managed secrets encryption or a fully-client-side local passphrase encryption.
This is implemented using the Go Cloud Development Kit support for pluggable secrets providers.
Like our cloud storage backend support which also uses Go Cloud Development Kit, this PR also bleeds through to users the URI scheme's that the Go CDK defines for specifying each of secrets providers - like `awskms://alias/LukeTesting?region=us-west-2` or `azurekeyvault://mykeyvaultname.vault.azure.net/keys/mykeyname`.
Also like our cloud storage backend support, this PR doesn't solve for how to configure the cloud provider client used to resolve the URIs above - the standard ambient credentials are used in both cases. Eventually, we will likely need to provide ways for both of these features to be configured independently of each other and of the providers used for resource provisioning.
A workaround for #2695
During the plugin installation, we create a temporary folder, unzip the binary, and then rename the folder to a permanent name. The rename fails 90% of the time with access denied. An immediate retry of renaming seems to always succeed.
This package's flags conflict with those in google/glog. Replace all
references to this package with references to
pulumi/pulumi/pkg/util/logging, and change that package to explicitly
call `flag.CommandLine.Parse` with an empty slice.
This should make it much easier to consume these packages in downstream
repos that have direct or indirect dependencies on google/glog.
These changes add support for passing `ignoreChanges` paths to resource
providers. This is intended to accommodate providers that perform diffs
between resource inputs and resource state (e.g. all Terraform-based
providers, the k8s provider when using API server dry-runs). These paths
are specified using the same syntax as the paths used in detailed diffs.
In addition to passing these paths to providers, the existing support
for `ignoreChanges` in inputs has been extended to accept paths rather
than top-level keys. It is an error to specify a path that is missing
one or more component in the old or new inputs.
Fixes#2936, #2663.
This removes the need for a replace directive in every downstream `go.mod`,
however it does not protect against the case of a `go.mod` being added upstream
with a different import path in future. This seems unlikely given the cadence of
work upstream, however.
Most of these options are typically left unset. In order to make it
easier to update the lifecycle test when adding new options, collect
them in a bag s.t. most callsites can go without being updated.
* Fix issue when logging out of local backend
* Put actual cloud URL into credentials.json
* Fix test on Windows; disable newly-enabled tests on Appveyor
* Keep track of the URL used to initialize local backend
If we encounter a provider with old inputs but no old outputs when reading
a checkpoint file, use the old inputs as the old outputs. This handles the
scenario where the CLI is being upgraded from a version that did not
reflect provider inputs to provider outputs, and a provider is being
upgraded from a version that did not implement `DiffConfig` to a version
that does.
Fixes https://github.com/pulumi/pulumi-kubernetes/issues/645.
Currently, `pulumi preview` fails immediately when any resource
definition in a Pulumi app is found to be in violation of a resource
policy. But, users would like `preview` to report as many policy
violations as it can before terminating with an error, so that they can
fix many of them before running `preview` again.
This commit will thus change `pulumi preview` to do this sort of
"batching" for policy violations. The engine will attempt to run the
entire preview step, validating every resource definition with the
relevant known resource policies, before finally reporting an error if
any violations are detected.
Fixespulumi/pulumi-policy#31
The current CLI update view attributes all policy violation errors to
the root Stack resource. This commit will attribute them to the resource
that violated the policy.
The reason for this mis-attribution is a simple bookkeeping error:
* Resource policies intercept and prevent RegisterResource requests for
when the resource in question violates some policy.
* The CLI "tree" view of resources "hides" rows for resources that have
not been registered. Thus, if a policy violation occurs for a
resource, it becomes "orphaned" and is attributed to the stack,
because there is no row for the resource that violates the policy.
The solution, thus, is to simply set the "hidden" flag to false when we
encounter a policy violation.
Fixespulumi/pulumi-policy#25
Fixes: #2151
This will allow us to be able to share the code that generates our
language providers. Currently there is a copy of the python code
generation in pulumi-kubernetes and also in pulumi-terraform
We want to be able to share these
