Today, we run 4 jobs for every push or PR build. Three Linux builds
which only differ by node version and an macOS build that also has a
different node version from the rest.
In practice this ends up generating a great deal of jobs during
development, given some of our workflows and these jobs often don't
find interesting issues.
We're going to start by relaxing things in the following ways:
- For a given PR, we'll just run a single Linux leg (as well as the
Windows leg we get from AppVeyor).
- For push jobs (which happen either when a PR is merged into master,
or a branch named `features/*` is pushed, we'll run a Linux leg and
and macOS leg (the macOS leg is run per push, so we build and
publish of the CLI for macOS builds)
- For our nightly cron job, we'll continue to run all three Linux
legs, to get coverage across different versions of nodejs.
If we end up seeing an uptick in regressions found by these jobs after
each commit, we can re-evaluate or pre checkin strategy.
In #2330 there was a case where if you didn't pass a value to the
`pulumi.Config()` constructor, things would fail in a weird manner.
This shouldn't be the case, and I'm unable to reproduce the issue. So
I'm updating the test to use the form that didn't work at one point so
we can lock in the win.
Fixes#2330
These changes add a new flag to the various `ResourceOptions` types that
indicates that a resource should be deleted before it is replaced, even
if the provider does not require this behavior. The usual
delete-before-replace cascade semantics apply.
Fixes#1620.
In general, a "delete" in Pulumi is destroying an actual physical
resource. In the case of a read resource, however, the delete is
merely removing the resource from the stack; the physical resource
is not affected. These changes attempt to clarify this situation by
using the term "discard" rather than "delete".
Fixes#2015.
* Clean up documentation for Python SDK
The Python SDK currently does a couple of bad things that make it
difficult to generate documentation:
1. It "wildcard-imports" submodules without each module specifying an
__all__ member
2. Documentation strings don't have a consistent format
3. Documentation strings are in Markdown and not reStructuredText
To remedy this, this commit addresses 1 by explicitly specifying which
members are being exported from submodules, so that we can see in one
place exactly what the public surface area of the pulumi package is. For
2 and 3, this commit fixes a large number of documentation strings to
contain metadata tags that Sphinx is capable of reading. This allows us
to generate high-quality documentation directly from the source without
having to manually parse docstrings.
* Please pylint
Because of the change to include a stack's project as part of its
identity in the service, the names passed to StackReference now
require the project name as well.
Improve the error message when they do not include them.
* Work around commonjs protoc bug
When compiling with the commonjs target, the protoc compiler still emits
references to Closure Compiler-isms that whack global state onto the
global object. This is particularly bad for us since we expect to be
able to make backwards-compatible changes to our Protobuf definitions
without breaking things, and this bug makes it impossible to do so.
To remedy the bug, this commit hacks the output of protoc (again) with
sed in order to avoid ever touching the global object. Everything still
works fine because the commonjs target (correctly) exports the protobuf
message types via the module system - it's just not writing to global
anymore.
* Fix status.proto
* Don't hack status.proto
- Add support for per-property dependencies to the Go SDK
- Add tests for first-class secret rejection in the checkpoint and RPC
layers and language SDKs
This implements the new algorithm for deciding which resources must be
deleted due to a delete-before-replace operation.
We need to compute the set of resources that may be replaced by a
change to the resource under consideration. We do this by taking the
complete set of transitive dependents on the resource under
consideration and removing any resources that would not be replaced by
changes to their dependencies. We determine whether or not a resource
may be replaced by substituting unknowns for input properties that may
change due to deletion of the resources their value depends on and
calling the resource provider's Diff method.
This is perhaps clearer when described by example. Consider the
following dependency graph:
A
__|__
B C
| _|_
D E F
In this graph, all of B, C, D, E, and F transitively depend on A. It may
be the case, however, that changes to the specific properties of any of
those resources R that would occur if a resource on the path to A were
deleted and recreated may not cause R to be replaced. For example, the
edge from B to A may be a simple dependsOn edge such that a change to
B does not actually influence any of B's input properties. In that case,
neither B nor D would need to be deleted before A could be deleted.
In order to make the above algorithm a reality, the resource monitor
interface has been updated to include a map that associates an input
property key with the list of resources that input property depends on.
Older clients of the resource monitor will leave this map empty, in
which case all input properties will be treated as depending on all
dependencies of the resource. This is probably overly conservative, but
it is less conservative than what we currently implement, and is
certainly correct.
The service also does this filtering on requests, because we'll need
to support older clients, but it would be nice if the CLI itself also
cleaned things up.
When stack names had to be unique across an entire organization, we
had a convention that stack names would be prefixed by the project
name, to prevent collisions.
Now that stack names are scoped within a project, we no longer need to
include the project name in the stack name. So when running `pulumi
new` to scaffhold a new project, just recommend the name `dev` for the
stack to create instead of `<project-name>-dev`.
Fixes#1417
This change starts to use a stack's project name as part of it's
identity when talking to the cloud backend, which the Pulumi Service
now supports.
When displaying or parsing stack names for the cloud backend, we now
support the following schemes:
`<stack-name>`
`<owner-name>/<stack-name>`
`<owner-name>/<project-name>/<stack-name>`
When the owner is not specificed, we assume the currently logged in
user (as we did before). When the project name is not specificed, we
use the current project (and fail if we can't find a `Pulumi.yaml`)
Fixes#2039
This change does two things:
- It ensures that as part of publishing the SDK, we also publish an
updated pulumi/pulumi docker image (tagged with both `latest` and
`vX.Y.Z`
- Makes this image published by this repository less perscriptive in a
workflow. The instead of having wrapper scripts that try to invoke
Pulumi based on conventions. It is now just a base image that has
the pulumi CLI installed, as well as the SDKs for the major cloud
providers. We'll use this base layer in our github actions image,
which will layer on a github actions centric workflow
Fixes#1991
Resources gain two new fields: `PropertyDependencies` and
`PendingReplacement`. The former maps an input property's name to the
dependencies that may affect the value of that property. The latter is
used to track resources that have been deleted as part of a
delete-before-replace operation but have not yet been recreated.
In addition to the new fields, resource properties may now contain
encrypted first-class secret values. These values are of type `SecretV1`,
where the `Sig` field is set to `resource.SecretSig`.
Finally, the deployment type gains a new field, `SecretsProviders`,
which contains any configuration necessary to handle secrets that may be
present in resource properties.
* Use both a in-proc and out-of-proc pipenv lock
Turns out that flock alone is not sufficient to guarantee exclusive
access to a resource within a single process. To remedy this, a few
FileMutex type wraps both an in-proc mutex and an out-of-proc
file-backed mutex to achieve the goal of exclusive access to a resource
in both in-proc and out-of-proc scenarios.
This commit also uses this lock globally in the integration test
framework in order to globally serialize invocations of pipenv install.
* Remove merge markers
Some changes to `pulumi new` to improve the experience:
- Color the default values for config differently to make them stand
out better
- Mention that `new` will also perform an initial deployment
- Add more vertical whitespace between steps in the process
- Print message indicating the "Installing dependencies" step is
complete
- After "project is ready to go", add a note about doing an initial
deployment
- Output follow-up command to run when an update fails
- Go back to showing the `npm install` output as `npm` doesn't always
return an error code when it runs into problems
We haven't shipped this yet, and it feels like returning a dictionary
where the keys are config keys and the value is an object that tells
you if it is a secret or not (and its value if it is secret and you
have passed `--show-secrets`) is going to be better overall.
This supports using `--json` to get configuration information in a
structured way.
The objects we return have the following schema:
```
{
name: string;
value: string?;
secret: bool;
}
```
In the case of `pulumi config` when --show-secrets is not passed, and
there are secret values, the `value` property of the object for that
configuration value will not be set. This differs from the normal
rendering where we show `[secret]`.
Contributes To #496
* Use a file lock for serializing pipenv installs
A in-process mutex is not sufficient for serializing pipenv installs
because the 1) go test runner occasionally will split test executions
into multiple processes and 2) each test gets an instance of a
programTester and we'd need to share the mutex globally if we wanted to
successfully serialize access to the pipenv install command.
* Please linter
When constructing an Archive based off a directory path, we would
ignore any symlinks that we saw while walking the file system
collecting files to include in the archive.
A user reported an issue where they were unable to use the
[sharp](https://www.npmjs.com/package/sharp) library from NPM with a
lambda deployed via Pulumi. The problem was that the library includes
native components, and these native components include a bunch of
`*.so` files. As is common, there's a regular file with a name like
`foo.so.1.2.3` and then symlinks to that file with the names
`foo.so.1.2`, `foo.so.1` and `foo.so`. Consumers of these SOs will
try to load the shorter names, i.e. `foo.so` and expect the symlink to
resolve to the actual library.
When these links are not present, upstack code fails to load.
This changes modifies our logic such that if we have a symlink and it
points to a regular file, we include it in the archive. At this time,
we don't try to add it to the archive as a symlink, instead we just
add it as another copy of the regular file. We could explore trying to
include these things as symlinks in archive formats that allow
them (While zip does support this, I'm less sure doing this for zip
files will be a great idea, given the set of tricks we already play to
ensure our zip files are usable across many cloud vendors serverless
offerings, it feels like throwing symlinks into the mix may end up
causing even more downstream weirdness).
We continue to ignore symlinks which point at directories. In
practice, these seem fairly uncommon and doing so lets us not worry
about trying to deal with ensuring we don't end up chasing our tail in
cases where there are circular references.
While this change is in pulumi/pulumi, the downstream resource
providers will need to update their vendored dependencies in order to
pick this up and have it work end to end.
Fixes#2077
setuptools's "develop" action is not safe to run concurrently when
targeting the same source tree. In order to work around this, this
commit explicitly serializes package installations.
- Ensure new projects have a project name in line with what we'd like
to enforce going forward
- Do more aggresive validation during the interactive prompts during
`pulumi new`
- Fix an issue where the interactive prompt rendered weridly when
there was a validation error
Contributes to #1988Fixes#1441
This reverts commit fe6567a9a7.
We've decided to change get.pulumi.com to not require this, so
reverting the change to make the already long command no longer than
needed.
We currently leave behind the template section inside Pulumi.yaml after
`pulumi new`. While we may eventually make use of it, we're not
currently using it, so it's cleaner to just remove it for now.
We upgraded get.pulumi.com to require TLS1.2. Because powershell does
not support this out of the box, we had to augment the install command
to also enable support for TLS1.2. While we did this on pulumi.io in
our documentation, we also print a similar message on Windows when the
CLI is out of date.
This change augments the message to include enabiling TLS 1.2 during
the invocation.
On startup, when we were populating the Settings object, we failed to
coerce the dry_run parameter from a string to a boolean, which resulted
in is_dry_run always believing that it was a preview. This PR fixes this
oversight by explicitly coercing to a boolean prior to sending the value
to Settings.