* Fix issue when logging out of local backend
* Put actual cloud URL into credentials.json
* Fix test on Windows; disable newly-enabled tests on Appveyor
* Keep track of the URL used to initialize local backend
The current CLI update view attributes all policy violation errors to
the root Stack resource. This commit will attribute them to the resource
that violated the policy.
The reason for this mis-attribution is a simple bookkeeping error:
* Resource policies intercept and prevent RegisterResource requests for
when the resource in question violates some policy.
* The CLI "tree" view of resources "hides" rows for resources that have
not been registered. Thus, if a policy violation occurs for a
resource, it becomes "orphaned" and is attributed to the stack,
because there is no row for the resource that violates the policy.
The solution, thus, is to simply set the "hidden" flag to false when we
encounter a policy violation.
Fixespulumi/pulumi-policy#25
This commit will allow the Pulumi service HTTP client to deserialize
HTTP responses that have bodies encoded as `application/octet-stream` to
be deserialized as `[]byte`.
This fixes a small bug that causes the HTTP client to fail under these
circumstances, as it expects any body to be JSON-deserializable.
This commit will implement the core business logic of `pulumi policy
publish` -- code to boot an analyzer, ask it for metadata about the
policies it contains, pack the code, and transmit all of this to the
Pulumi service.
* Plumbing the custom timeouts from the engine to the providers
* Plumbing the CustomTimeouts through to the engine and adding test to show this
* Change the provider proto to include individual timeouts
* Plumbing the CustomTimeouts from the engine through to the Provider RPC interface
* Change how the CustomTimeouts are sent across RPC
These errors were spotted in testing. We can now see that the timeout
information is arriving in the RegisterResourceRequest
```
req=&pulumirpc.RegisterResourceRequest{
Type: "aws:s3/bucket:Bucket",
Name: "my-bucket",
Parent: "urn:pulumi:dev::aws-vpc::pulumi:pulumi:Stack::aws-vpc-dev",
Custom: true,
Object: &structpb.Struct{},
Protect: false,
Dependencies: nil,
Provider: "",
PropertyDependencies: {},
DeleteBeforeReplace: false,
Version: "",
IgnoreChanges: nil,
AcceptSecrets: true,
AdditionalSecretOutputs: nil,
Aliases: nil,
CustomTimeouts: &pulumirpc.RegisterResourceRequest_CustomTimeouts{
Create: 300,
Update: 400,
Delete: 500,
XXX_NoUnkeyedLiteral: struct {}{},
XXX_unrecognized: nil,
XXX_sizecache: 0,
},
XXX_NoUnkeyedLiteral: struct {}{},
XXX_unrecognized: nil,
XXX_sizecache: 0,
}
```
* Changing the design to use strings
* CHANGELOG entry to include the CustomTimeouts work
* Changing custom timeouts to be passed around the engine as converted value
We don't want to pass around strings - the user can provide it but we want
to make the engine aware of the timeout in seconds as a float64
A resource can be imported by setting the `import` property in the
resource options bag when instantiating a resource. In order to
successfully import a resource, its desired configuration (i.e. its
inputs) must not differ from its actual configuration (i.e. its state)
as calculated by the resource's provider.
There are a few interesting state transitions hiding here when importing
a resource:
1. No prior resource exists in the checkpoint file. In this case, the
resource is simply imported.
2. An external resource exists in the checkpoint file. In this case, the
resource is imported and the old external state is discarded.
3. A non-external resource exists in the checkpoint file and its ID is
different from the ID to import. In this case, the new resource is
imported and the old resource is deleted.
4. A non-external resource exists in the checkpoint file, but the ID is
the same as the ID to import. In this case, the import ID is ignored
and the resource is treated as it would be in all cases except for
changes that would replace the resource. In that case, the step
generator issues an error that indicates that the import ID should be
removed: were we to move forward with the replace, the new state of
the stack would fall under case (3), which is almost certainly not
what the user intends.
Fixes#1662.
Instead of simply converting a detailed diff entry that indicates an
update to an entire composite value as a simple old/new value diff,
compute the nested diff. This alllows us to render a per-element diff
for the nested object rather than simply displaying the new and the old
composite values.
This is necessary in order to improve diff rendering once
pulumi/pulumi-terraform#403 has been rolled out.
Thse changes make a subtle but critical adjustment to the process the
Pulumi engine uses to determine whether or not a difference exists
between a resource's actual and desired states, and adjusts the way this
difference is calculated and displayed accordingly.
Today, the Pulumi engine get the first chance to decide whether or not
there is a difference between a resource's actual and desired states. It
does this by comparing the current set of inputs for a resource (i.e.
the inputs from the running Pulumi program) with the last set of inputs
used to update the resource. If there is no difference between the old
and new inputs, the engine decides that no change is necessary without
consulting the resource's provider. Only if there are changes does the
engine consult the resource's provider for more information about the
difference. This can be problematic for a number of reasons:
- Not all providers do input-input comparison; some do input-state
comparison
- Not all providers are able to update the last deployed set of inputs
when performing a refresh
- Some providers--either intentionally or due to bugs--may see changes
in resources whose inputs have not changed
All of these situations are confusing at the very least, and the first
is problematic with respect to correctness. Furthermore, the display
code only renders diffs it observes rather than rendering the diffs
observed by the provider, which can obscure the actual changes detected
at runtime.
These changes address both of these issues:
- Rather than comparing the current inputs against the last inputs
before calling a resource provider's Diff function, the engine calls
the Diff function in all cases.
- Providers may now return a list of properties that differ between the
requested and actual state and the way in which they differ. This
information will then be used by the CLI to render the diff
appropriately. A provider may also indicate that a particular diff is
between old and new inputs rather than old state and new inputs.
Fixes#2453.
Currently if you log into s3://bucket/subdirectory, Pulumi will write
files to s3://bucket/.pulumi and not s3://bucket/subdirectory/.pulumi,
this corrects the error.
Recent changes to default provider semantics and the addition of
resource aliases allow a resource's provider reference to change even if
the resource itself is considered to have no diffs. `mustWrite` did not
expect this scenario, and indeed asserted against it. These changes
update `mustWrite` to detect such changes and require that the
checkpoint be written if and when they occur.
Fixes#2804.
Previously, when the CLI wanted to install a plugin, it used a special
method, `DownloadPlugin` on the `httpstate` backend to actually fetch
the tarball that had the plugin. The reason for this is largely tied
to history, at one point during a closed beta, we required presenting
an API key to download plugins (as a way to enforce folks outside the
beta could not download them) and because of that it was natural to
bake that functionality into the part of the code that interfaced with
the rest of the API from the Pulumi Service.
The downside here is that it means we need to host all the plugins on
`api.pulumi.com` which prevents community folks from being able to
easily write resource providers, since they have to manually manage
the process of downloading a provider to a machine and getting it on
the `$PATH` or putting it in the plugin cache.
To make this easier, we add a `--server` argument you can pass to
`pulumi plugin install` to control the URL that it attempts to fetch
the tarball from. We still have perscriptive guidence on how the
tarball must be
named (`pulumi-[<type>]-[<provider-name>]-vX.Y.Z.tar.gz`) but the base
URL can now be configured.
Folks publishing packages can use install scripts to run `pulumi
plugin install` passing a custom `--server` argument, if needed.
There are two improvements we can make to provide a nicer end to end
story here:
- We can augment the GetRequiredPlugins method on the language
provider to also return information about an optional server to use
when downloading the provider.
- We can pass information about a server to download plugins from as
part of a resource registration or creation of a first class
provider.
These help out in cases where for one reason or another where `pulumi
plugin install` doesn't get run before an update takes place and would
allow us to either do the right thing ahead of time or provide better
error messages with the correct `--server` argument. But, for now,
this unblocks a majority of the cases we care about and provides a
path forward for folks that want to develop and host their own
resource providers.
Adds a new resource option `aliases` which can be used to rename a resource. When making a breaking change to the name or type of a resource or component, the old name can be added to the list of `aliases` for a resource to ensure that existing resources will be migrated to the new name instead of being deleted and replaced with the new named resource.
There are two key places this change is implemented.
The first is the step generator in the engine. When computing whether there is an old version of a registered resource, we now take into account the aliases specified on the registered resource. That is, we first look up the resource by its new URN in the old state, and then by any aliases provided (in order). This can allow the resource to be matched as a (potential) update to an existing resource with a different URN.
The second is the core `Resource` constructor in the JavaScript (and soon Python) SDKs. This change ensures that when a parent resource is aliased, that all children implicitly inherit corresponding aliases. It is similar to how many other resource options are "inherited" implicitly from the parent.
Four specific scenarios are explicitly tested as part of this PR:
1. Renaming a resource
2. Adopting a resource into a component (as the owner of both component and consumption codebases)
3. Renaming a component instance (as the owner of the consumption codebase without changes to the component)
4. Changing the type of a component (as the owner of the component codebase without changes to the consumption codebase)
4. Combining (1) and (3) to make both changes to a resource at the same time
If --suppress-outputs is passed to `pulumi preview --json`, we
should not emit the stack outputs. This change fixespulumi/pulumi#2765.
Also adds a test case for this plus some variants of updates.
We were not actually calling our colorization routines, which lead to
printing this very confusing text:
```
<{%reset%}> --outputs:--<{%reset%}>
```
When running updates with `--diff` or when drilling into details of a
proposed operation, like a refresh.
We were dropping new and old states on the floor instead of including
them as part of the previewed operation due to a logic error (we want
to append them when there are no errors from serialization, vs when
there are errors).
When creating a new stack using the local backend, the default
checkpoint has no deployment. That means there's a nil snapshot
created, which means our strategy of using the base snapshot's secrets
manager was not going to work. Trying to do so would result in a panic
because the baseSnapshot is nil in this case.
Using the secrets manager we are going to use to persist the snapshot
is a better idea anyhow, as that's what's actually going to be burned
into the deployment when we serialize the snapshot, so let's use that
instead.
This change allows using the passphrase secrets manager when creating
a stack managed by the Pulumi service. `pulumi stack init`, `pulumi
new` and `pulumi up` all learned a new optional argument
`--secrets-provider` which can be set to "passphrase" to force the
passphrase based secrets provider to be used. When unset the default
secrets provider is used based on the backend (for local stacks this
is passphrase, for remote stacks, it is the key managed by the pulumi
service).
As part of this change, we also initialize the secrets manager when a
stack is created, instead of waiting for the first time a secret
config value is stored. We do this so that if an update is run using
`pulumi.secret` before any secret configuration values are used, we
already have the correct encryption method selected for a stack.
Logs are no longer provided by the service (this is a holdover from
the PPC days where service deployments where done in the cloud and it
handled collecting logs).
Removing this breaks another cycle that would be introduced with the
next change (in our test code)
The next change is going to do some code motion that would create some
circular imports if we did not do this. There was nothing that
required the members we were moving be in the backend package, so it
was easy enough to pull them out.
When preforming an update, require that a secrets manager is passed in
as part of the `backend.UpdateOperation` bag and use it. The CLI now
passes this in (it still uses the default base64 secrets manager, so
this is just code motion into a high layer, since the CLI will be the
one to choose what secrets manager to use based on project settings).
There are a few operations we do (stack rename, importing and edits)
where we will materialize a `deploy.Snapshot` from an existing
deployment, mutate it in somewhay, and then store it.
In these cases, we will just re-use the secrets manager that was used
to build the snapshot when we re-serialize it. This is less than ideal
in some cases, because many of these operations could run on an
"encrypted" copy of the Snapshot, where Inputs and Outputs have not
been decrypted.
Unfortunately, our system now is not set up in a great way to support
this and adding something like a `deploy.EncryptedSnapshot` would
require large scale code duplications.
So, for now, we'll take the hit of decrypting and re-encrypting, but
long term introducing a `deploy.EncryptedSnapshot` may be nice as it
would let us elide the encryption/decryption steps in some places and
would also make it clear what parts of our system have access to the
plaintext values of secrets.
We have many cases where we want to do the following:
deployment -> snapshot -> process snapshot -> deployment
We now retain information in the snapshot about the secrets manager
that was used to construct it, so in these round trip cases, we can
re-use the existing manager.
When nil, it means no information is retained in the deployment about
the manager (as there is none) and any attempt to persist secret
values fails.
This should only be used in cases where the snapshot is known to not
contain secret values.
Half of the call sites didn't care about these values and with the
secrets work the ergonmics of calling this method when it has to
return serialized ouputs isn't great. Move the serialization for this
into the CLI itself, as it was the only place that cared to do
this (so it could display things to end users).
The previous changes to remove config loading out of the backend means
that the backends no longer need to track this information, as they
never use it.
As part of the pluggable secrets work, the crypter's used for secrets
are no longer tied to a backend. To enforce this, we remove the
`backend.GetStackCrypter` function and then have the relevent logic to
construct one live inside the CLI itself.
Right now the CLI still uses the backend type to decide what Crypter
to build, but we'll change that shortly.
We require configuration to preform updates (as well as previews,
destroys and refreshes). Because of how everything evolved, loading
this configuration (and finding the coresponding decrypter) was
implemented in both the file and http backends, which wasn't great.
Refactor things such that the CLI itself builds out this information
and passes it along to the backend to preform operations. This means
less code duplicated between backends and less places the backend
assume things about the existence of `Pulumi.yaml` files and in
general makes the interface more plesent to use for others uses.
For cloud backed stacks, this was already returning nil and due to the
fact that we no longer include config in the checkpoint for local
stacks, it was nil there as well.
Removing this helps clean stuff up and is should make some future
refactorings around custom secret managers easier to land.
We can always add it back later if we miss it (and make it actually do
the right thing!)
When constructing a Deployment (which is a plaintext representation of
a Snapshot), ensure that we encrypt secret values. To do so, we
introduce a new type `secrets.Manager` which is able to encrypt and
decrypt values. In addition, it is able to reflect information about
itself that can be stored in the deployment such that we can
deserialize the deployment into a snapshot (decrypting the values in
the process) without external knowledge about how it was encrypted.
The ability to do this is import for allowing stack references to
work, since two stacks may not use the same manager (or they will use
the same type of manager, but have different state).
The state value is stored in plaintext in the deployment, so it **must
not** contain sensitive data.
A sample manager, which just base64 encodes and decodes strings is
provided, as it useful for testing. We will allow it to be varried
soon.
* Add a var for PRNumber. Add an environment metadata key for PR number.
* Move the detection of PULUMI_CI_SYSTEM into vars.DetectVars(). Set the PRNumber CI property based on respective env vars from each CI system.
* Add Azure Pipelines build variables.
* Add tests for DetectVars.
* Added changelog entry for Azure Pipelines.
* Capture the value of env var being modified for the ciutil unit test, and restore their values at the end of them.
* Simplify the DetectVars function by moving the Pulumi CI system code into the switch-case expression.
* Rename the Pulumi CI system to Generic CI. Include the GenericCI system in the test case for DetectVars.
This command exposes a new resource `Invoke` operation,
`pulumi:pulumi:readStackResourceOutputs` which retrieves all resource
outputs for some user-specified stack, not including those deleted.
Fixes#2600.
