- Add support for per-property dependencies to the Go SDK
- Add tests for first-class secret rejection in the checkpoint and RPC
layers and language SDKs
This implements the new algorithm for deciding which resources must be
deleted due to a delete-before-replace operation.
We need to compute the set of resources that may be replaced by a
change to the resource under consideration. We do this by taking the
complete set of transitive dependents on the resource under
consideration and removing any resources that would not be replaced by
changes to their dependencies. We determine whether or not a resource
may be replaced by substituting unknowns for input properties that may
change due to deletion of the resources their value depends on and
calling the resource provider's Diff method.
This is perhaps clearer when described by example. Consider the
following dependency graph:
A
__|__
B C
| _|_
D E F
In this graph, all of B, C, D, E, and F transitively depend on A. It may
be the case, however, that changes to the specific properties of any of
those resources R that would occur if a resource on the path to A were
deleted and recreated may not cause R to be replaced. For example, the
edge from B to A may be a simple dependsOn edge such that a change to
B does not actually influence any of B's input properties. In that case,
neither B nor D would need to be deleted before A could be deleted.
In order to make the above algorithm a reality, the resource monitor
interface has been updated to include a map that associates an input
property key with the list of resources that input property depends on.
Older clients of the resource monitor will leave this map empty, in
which case all input properties will be treated as depending on all
dependencies of the resource. This is probably overly conservative, but
it is less conservative than what we currently implement, and is
certainly correct.
The service also does this filtering on requests, because we'll need
to support older clients, but it would be nice if the CLI itself also
cleaned things up.
When stack names had to be unique across an entire organization, we
had a convention that stack names would be prefixed by the project
name, to prevent collisions.
Now that stack names are scoped within a project, we no longer need to
include the project name in the stack name. So when running `pulumi
new` to scaffhold a new project, just recommend the name `dev` for the
stack to create instead of `<project-name>-dev`.
Fixes#1417
This change starts to use a stack's project name as part of it's
identity when talking to the cloud backend, which the Pulumi Service
now supports.
When displaying or parsing stack names for the cloud backend, we now
support the following schemes:
`<stack-name>`
`<owner-name>/<stack-name>`
`<owner-name>/<project-name>/<stack-name>`
When the owner is not specificed, we assume the currently logged in
user (as we did before). When the project name is not specificed, we
use the current project (and fail if we can't find a `Pulumi.yaml`)
Fixes#2039
This change does two things:
- It ensures that as part of publishing the SDK, we also publish an
updated pulumi/pulumi docker image (tagged with both `latest` and
`vX.Y.Z`
- Makes this image published by this repository less perscriptive in a
workflow. The instead of having wrapper scripts that try to invoke
Pulumi based on conventions. It is now just a base image that has
the pulumi CLI installed, as well as the SDKs for the major cloud
providers. We'll use this base layer in our github actions image,
which will layer on a github actions centric workflow
Fixes#1991
Resources gain two new fields: `PropertyDependencies` and
`PendingReplacement`. The former maps an input property's name to the
dependencies that may affect the value of that property. The latter is
used to track resources that have been deleted as part of a
delete-before-replace operation but have not yet been recreated.
In addition to the new fields, resource properties may now contain
encrypted first-class secret values. These values are of type `SecretV1`,
where the `Sig` field is set to `resource.SecretSig`.
Finally, the deployment type gains a new field, `SecretsProviders`,
which contains any configuration necessary to handle secrets that may be
present in resource properties.
* Use both a in-proc and out-of-proc pipenv lock
Turns out that flock alone is not sufficient to guarantee exclusive
access to a resource within a single process. To remedy this, a few
FileMutex type wraps both an in-proc mutex and an out-of-proc
file-backed mutex to achieve the goal of exclusive access to a resource
in both in-proc and out-of-proc scenarios.
This commit also uses this lock globally in the integration test
framework in order to globally serialize invocations of pipenv install.
* Remove merge markers
Some changes to `pulumi new` to improve the experience:
- Color the default values for config differently to make them stand
out better
- Mention that `new` will also perform an initial deployment
- Add more vertical whitespace between steps in the process
- Print message indicating the "Installing dependencies" step is
complete
- After "project is ready to go", add a note about doing an initial
deployment
- Output follow-up command to run when an update fails
- Go back to showing the `npm install` output as `npm` doesn't always
return an error code when it runs into problems
We haven't shipped this yet, and it feels like returning a dictionary
where the keys are config keys and the value is an object that tells
you if it is a secret or not (and its value if it is secret and you
have passed `--show-secrets`) is going to be better overall.
This supports using `--json` to get configuration information in a
structured way.
The objects we return have the following schema:
```
{
name: string;
value: string?;
secret: bool;
}
```
In the case of `pulumi config` when --show-secrets is not passed, and
there are secret values, the `value` property of the object for that
configuration value will not be set. This differs from the normal
rendering where we show `[secret]`.
Contributes To #496
* Use a file lock for serializing pipenv installs
A in-process mutex is not sufficient for serializing pipenv installs
because the 1) go test runner occasionally will split test executions
into multiple processes and 2) each test gets an instance of a
programTester and we'd need to share the mutex globally if we wanted to
successfully serialize access to the pipenv install command.
* Please linter
When constructing an Archive based off a directory path, we would
ignore any symlinks that we saw while walking the file system
collecting files to include in the archive.
A user reported an issue where they were unable to use the
[sharp](https://www.npmjs.com/package/sharp) library from NPM with a
lambda deployed via Pulumi. The problem was that the library includes
native components, and these native components include a bunch of
`*.so` files. As is common, there's a regular file with a name like
`foo.so.1.2.3` and then symlinks to that file with the names
`foo.so.1.2`, `foo.so.1` and `foo.so`. Consumers of these SOs will
try to load the shorter names, i.e. `foo.so` and expect the symlink to
resolve to the actual library.
When these links are not present, upstack code fails to load.
This changes modifies our logic such that if we have a symlink and it
points to a regular file, we include it in the archive. At this time,
we don't try to add it to the archive as a symlink, instead we just
add it as another copy of the regular file. We could explore trying to
include these things as symlinks in archive formats that allow
them (While zip does support this, I'm less sure doing this for zip
files will be a great idea, given the set of tricks we already play to
ensure our zip files are usable across many cloud vendors serverless
offerings, it feels like throwing symlinks into the mix may end up
causing even more downstream weirdness).
We continue to ignore symlinks which point at directories. In
practice, these seem fairly uncommon and doing so lets us not worry
about trying to deal with ensuring we don't end up chasing our tail in
cases where there are circular references.
While this change is in pulumi/pulumi, the downstream resource
providers will need to update their vendored dependencies in order to
pick this up and have it work end to end.
Fixes#2077
setuptools's "develop" action is not safe to run concurrently when
targeting the same source tree. In order to work around this, this
commit explicitly serializes package installations.
- Ensure new projects have a project name in line with what we'd like
to enforce going forward
- Do more aggresive validation during the interactive prompts during
`pulumi new`
- Fix an issue where the interactive prompt rendered weridly when
there was a validation error
Contributes to #1988Fixes#1441
This reverts commit fe6567a9a7.
We've decided to change get.pulumi.com to not require this, so
reverting the change to make the already long command no longer than
needed.
We currently leave behind the template section inside Pulumi.yaml after
`pulumi new`. While we may eventually make use of it, we're not
currently using it, so it's cleaner to just remove it for now.
We upgraded get.pulumi.com to require TLS1.2. Because powershell does
not support this out of the box, we had to augment the install command
to also enable support for TLS1.2. While we did this on pulumi.io in
our documentation, we also print a similar message on Windows when the
CLI is out of date.
This change augments the message to include enabiling TLS 1.2 during
the invocation.
On startup, when we were populating the Settings object, we failed to
coerce the dry_run parameter from a string to a boolean, which resulted
in is_dry_run always believing that it was a preview. This PR fixes this
oversight by explicitly coercing to a boolean prior to sending the value
to Settings.
We continue to do so when `--debug` has been passed, similar to how
these events are elided from the local display when you are not in a
debug context.
We publish some artifacts from this repository, which are specific to
an OS. In the past, we only had a single Linux leg and so we did not
need any additional conditional logic.
However, we now have multiple Linux legs (across different node
versions) running in parallel, so if we try to publish the same
artifacts at the same time, we can run into issues because two `aws s3
cp` jobs try to write to the same file at the same time.
Follow similar logic to our skips we do around publishing NPM and PyPI
packages here, as well.
* Initial stack history command
* Adding use of color pkg, adding background colors to color pkg, and removing extra stack output
* gofmt-ed colors file
* Fixing format and removing JSON output
* Fixing nits, changing output for environment, and adding some tests
* fixing failing history test
This value was never used before, but it had a shorter name. In other
API Types we are using `projectName` which we all prefer. Since we are
going to start using this value going forward, let's adopt the good
name now when it won't break anyone.
When is resource is waiting for its dependencies to resolve, it first
informs the RPC_MANAGER that it intends to do an RPC - this is to
prevent premature termination of the program while RPCs are still in
flight or queued to execute.
However, this is a problem whenever a resource fails to create for
whatever reason. The most common ways for this to happen are for invokes
to fail, resource creation itself to fail, or throwing in an apply.
Today, this causes a deadlock, since all consumers of the failed
resources block forever while never decrementing the RPC count.
This commit addresses the issue by deliberately (abnormally) resolving
all futures that are created in the process of preparing a resource.
This solves the problem by immediately terminating all resources that
are waiting for the failed resource's outputs to resolve - they resolve
immediately, and exceptionally.
The end result is now that, instead of deadlocking, a doomed program now
terminates as expected with a single exception message.
When previewing a real first-class provider, it is often the case that
the provider's ID is unknown during a preview. This commit fixes a bug
where we did not translate an unknown ID into the rpc layer's sentinel
UNKNOWN value where we should have, which caused the engine to fail to
resolve the provider reference.
For some reason, writing to stderr using sys.stderr.write doesn't work,
but printing to stderr using the print function does. I don't know why
this is, but I do know that this makes the printed messages show up as
messages on the Pulumi CLI.
This will allow the service to include information about what project
a stack is assocated with when listing all stacks a user has access
to.
This was not previously needed because the project did not play into
the stack identity, but it will shortly.
Invoke in Node.js allows users to optionally pass a parent or a provider
to the invoke, which dictates either explicitly or implicitly which
provider to use when performing an invoke. If a provider is specified
explicitly, that provider is used to perform the invoke. If a parent is
specified, that parent's provider is used to perform the invoke.
