Pylint currently reports `E1101: Instance of 'Bucket' has no 'id' member (no-member)` on lines in Pulumi Python programs like:
```python
pulumi.export('bucket_name', bucket.id)
```
Here's a description of this message from http://pylint-messages.wikidot.com/messages:e1101:
> Used when an object (variable, function, …) is accessed for a non-existent member.
>
> False positives: This message may report object members that are created dynamically, but exist at the time they are accessed.
This appears to be a false positive case: `id` isn't set in the constructor (it's set later in `register_resource`) and Pylint isn't able to figure this out statically. `urn` has the same problem. (Oddly, Pylint doesn't complain when accessing other resource output properties).
This change refactors `register_resource` so that `id` and `urn` can be assigned in the resource's constructor, so that Pylint can see it being assigned. The change also does the same with `read_resource`.
When running `pulumi up`, after the preview, we prompt asking whether to proceed with the update. If you type to filter the options and then hit an arrow key a couple times, the CLI panics. This is a bug in the `survey` library we depend on. The issue has been fixed in the library upstream; this change updates our dependency.
Automatically create a virtual environment and install dependencies in it with `pulumi new` and `pulumi policy new` for Python templates.
This will save a new `virtualenv` runtime option in `Pulumi.yaml` (`PulumiPolicy.yaml` for policy packs):
```yaml
runtime:
name: python
options:
virtualenv: venv
```
`virtualenv` is the path to a virtual environment that Pulumi will use when running `python` commands.
Existing projects are unaffected and can opt-in to using this by setting `virtualenv`, otherwise, they'll continue to work as-is.
This class was available in the pulumi.resource module, but was not exported from the core `pulumi` module as intended for all public APIs at this level.
The previous attempt to allow this didn't actually allow it, so this is
take two. As part of the previous attempt, I thought after tweaking the
test I had observed the test failing, and then succeeding after making
the product changes, but I must have been mistaken.
It turns out that our existing mocks tests weren't running at all
because of a missing `__init__.py` file. Once the missing `__init__.py`
is added, the tests run, but other tests ("test mode" tests) fail
because the code that creates the mocks and resources will run during
test discovery, and setting the mocks modifies global state.
To address the test issue, I've moved the mocks tests into their own
`test_with_mocks` package that can be run separately from other tests.
And addressed the original issue, by creating a root Stack resource if
one isn't already present when the mocks are set.
With these changes, a resource struct may tag a field with the empty
string. If such a field is present, any resource outputs that were not
unmarshalled into other fields will be unmarshalled into this field,
which must be a `MapOutput`.
Fixes#4629.
Adds support for RegisterResource to accept map-typed implementations if Input as well as the existing struct-typed implementations. Currently these must be fully untyped - but both map[string]pulumi.Input and map[string]interface{} are allowed. In the future, it's plausible that a mode where the data itself is a map, but the ElementType implementation returns a struct could be supported, with the struct used to provide type information over the untyped map.
This change allows importing modules with calls to `pulumi.export` in unit tests. Previously, you'd have to structure the Python program in a way that avoids the `pulumi.export` from being called from unit tests.
* [sdk/python] Improve `ResoruceOptions.merge` type
The implementation correctly handles `None` inputs, so the type should allow these as well.
* Add CHANGELOG
When writing the snapshot to the filestate bucket, we can retry in the
event of an error, which helps users who are experiencing issues around
write rates to GCS
In particular, ensure that they are keyword-safe. This affects
`aws:lambda:*` in particular: prior to these changes, we were generating
code into `pulumi_aws.lamdba`, which is not referencable due to its use
of the `lamdba` keyword. With these changes, we generate code into
`pulumi_aws.lambda_`.
There is also a small fix for multi-line non-formatted strings: these
strings do not need escaped braces.
When referencing `secretOutputNames` in from another stack, spurious
diffs can often be created because the secret output slice was not
ordered.
This PR orders the slice before it's added to the propertymap, ensuring
the order always remains the same
Fixes: #4444
Before:
```
$ pulumi stack
Current stack is 47BE2956-D665-4EC3-9AE6-4D4A1C417074:
Managed by demo-mbp
No updates yet; run 'pulumi up'
Current stack resources (0):
No resources currently in this stack
Use `pulumi stack select` to change stack; `pulumi stack ls` lists known ones
```
After:
```
$ pulumi stack --show-name
47BE2956-D665-4EC3-9AE6-4D4A1C417074
```
* [codegen/go] Fix accessors on struct ptr outputs
The accesor methods on nestred struct Ptr outputs were previously not accepting pointer typed inputs as they should, and would thus always panic if used.
The "simple" fix would be to just accept the pointer type and blindly dereference it. But this doesn't seem like the right experience - it would make these accessors very unsafe to use in practice.
Instead, this PR implements the accessors on pointer-typed outputs as nil-coaslescing, always lifting the output type into a pointer type and flowing a nil value into the result type. This ensures the accessor will not nil-deref, and that user code can handle the `nil` value itself (or use `.Apply` directly to implement more specialized behaviour).
Before:
```go
// Name of your S3 bucket.
func (o BuildStorageLocationPtrOutput) Bucket() pulumi.StringOutput {
return o.ApplyT(func(v BuildStorageLocation) string { return v.Bucket }).(pulumi.StringOutput)
}
```
After:
```go
// Name of your S3 bucket.
func (o BuildStorageLocationPtrOutput) Bucket() pulumi.StringPtrOutput {
return o.ApplyT(func(v *BuildStorageLocation) *string {
if v == nil {
return nil
}
return &v.Bucket
}).(pulumi.StringPtrOutput)
}
```
However, due to the decision to have this more usable behaviour, this is a breaking change, as some/many accessors now return a pointer type when they previously did not.
Fixespulumi/pulumi-azure#530.
* Mark nested property types as requiring ptr types
* Add CHANGELOG
* More fixes
After importing some resources, and running a second update with the
import still applied, an unexpected replace would occur. This wouldn't
happen for the vast majority of resources, but for some it would.
It turns out that the resources that trigger this are ones that use a
different format of identifier for the import input than they do for the
ID property.
Before this change, we would trigger an import-replacement when an
existing resource's ID property didn't match the import property, which
would be the case for the small set of resources where the input
identifier is different than the ID property.
To avoid this, we now store the `importID` in the statefile, and
compare that to the import property instead of comparing the ID.
When setting structured config values using `--path`, we automatically
treat values that can be converted into an integer via `strconv.Atoi` as
an integer, rather than as a string.
However, this ends up converting values like "0123456" into the integer
123456, stripping the leading 0, which isn't desirable for values like
commit SHAs, etc., where you want to keep the 0 (and keep it a string).
This change makes it so that values starting with 0 are not implicitly
converted to an integer; instead such values will remain a string.
* Make `async:true` the default for `invoke` calls (#3750)
* Switch away from native grpc impl. (#3728)
* Remove usage of the 'deasync' library from @pulumi/pulumi. (#3752)
* Only retry as long as we get unavailable back. Anything else continues. (#3769)
* Handle all errors for now. (#3781)
* Do not assume --yes was present when using pulumi in non-interactive mode (#3793)
* Upgrade all paths for sdk and pkg to v2
* Backport C# invoke classes and other recent gen changes (#4288)
Adjust C# generation
* Replace IDeployment with a sealed class (#4318)
Replace IDeployment with a sealed class
* .NET: default to args subtype rather than Args.Empty (#4320)
* Adding system namespace for Dotnet code gen
This is required for using Obsolute attributes for deprecations
```
Iam/InstanceProfile.cs(142,10): error CS0246: The type or namespace name 'ObsoleteAttribute' could not be found (are you missing a using directive or an assembly reference?) [/Users/stack72/code/go/src/github.com/pulumi/pulumi-aws/sdk/dotnet/Pulumi.Aws.csproj]
Iam/InstanceProfile.cs(142,10): error CS0246: The type or namespace name 'Obsolete' could not be found (are you missing a using directive or an assembly reference?) [/Users/stack72/code/go/src/github.com/pulumi/pulumi-aws/sdk/dotnet/Pulumi.Aws.csproj]
```
* Fix the nullability of config type properties in C# codegen (#4379)
These changes implement `GetRequiredPlugins` for Go using a registry
mechanism and an alternate entry point for `pulumi.Run`. Packages that
require plugins are expected to register themselves with the Pulumi SDK.
When `pulumi.Run` is used and the `PULUMI_PLUGINS` envvar is truthy, the
program will dump a JSON-encoded description of its required plugins to
stdout. The language host then uses this description to respond to
The changes in #4004 caused old provider configuration to be used even when a provider was different between inputs and outputs, in the case that the diff returned DiffUnkown.
To better handle that case, we compute a more accurate (but still conservative) DiffNone or DiffSome so that we can ensure we conservatively update to a new provider when needed, but retain the performance benefit of not creating and configuring a new provider as much as possible.
Part of https://github.com/pulumi/pulumi-aws/issues/814.
Fixes: https://github.com/pulumi/pulumi-terraform-bridge/issues/119
This allows us to specify an overlays block e.g.
```
Overlay: &tfbridge.OverlayInfo{
DestFiles: []string{
"pulumi_docker/docker.py",
"pulumi_docker/image.py",
},
},
```
The overlays files are treated differently to normal module files
as they are not generated. This structure means that we will emit
the correct entries in the __init__.py file
Without this structure (ie. pulumi_pkgname), the generator actually
copies the file (i.e. docker.py) to the root of the Python SDK. This
is because the structure of the Python SDK has a sub-folder than that
of the NodeJS SDK
I tested this using PR https://github.com/pulumi/pulumi-docker/pull/141
and this now works as expected and we can take advantage of the new
Python overlays for Docker
These changes implement `GetRequiredPlugins` for Go using a registry
mechanism and an alternate entry point for `pulumi.Run`. Packages that
require plugins are expected to register themselves with the Pulumi SDK.
When `pulumi.Run` is used and the `PULUMI_PLUGINS` envvar is truthy, the
program will dump a JSON-encoded description of its required plugins to
stdout. The language host then uses this description to respond to
`GetRequiredPlugins`.
* started transformations for go sdk
* added first basic test
* added second test with child
* added RegisterStackTransformation
* added a couple tests to lifecycle_test
* update CHANGELOG and test
* included TODO for #3846
We upgraded to `ts-node@^8.0.0` 2.5 months ago as part of https://github.com/pulumi/pulumi/pull/3627, though it seems it wasn't really necessary to make that update for the purposes of the PR - updating the default TypeScript version.
The `8.0.0` series of `ts-node` unfortunately dropped all of it's caching support, due to what appear to be some corner-case correctness issues with the cache. We have not seen reports of those issues for Pulumi, and have much more experience with the `7.0.0` series overall (2 years vs. 2 months). The performance difference between `7.0.0` and `8.0.0` of ts-node for Pulumi is massive - it adds 4-4.5s to each of `pulumi preview` and `pulumi up` even on a trivial program.
As a result, for now we will revert back to `ts-node@^7.0.0`. In the future, we may want to look into our own caching layer or alternative to `ts-node` to ensure we get the behaviour and performance we expect.
Part of #3671.
In the very common case where provider configuration does not change, during preview we were calling `Configure` on the cloud provider twice - once for the "old" configuration, and once for the "new" configuration.
This is not necessary, and we can just avoid using the new provider when configuration has not changed, since we will have configured the old provider very early so if we can use that we should.
Note that this technically doesn't prevent the second call to `Configure` from being made, but it prevents us from ever waiting on it. We may want to go further and avoid even calling `Configure` on the provider in this case.
Part of #3671.
These changes add support for mocking the resource monitor to the NodeJS
and Python SDKs. The proposed mock interface is a simplified version of
the standard resource monitor that allows an end-user to replace the
usual implementations of ReadResource/RegisterResource and Invoke with
their own. This can be used in unit tests to allow for precise control
of resource outputs and invoke results.
It appears there are cases where our IsInteractive heuristics return true, but terminal.GetSize returns an error. In these cases, we should assume we do not have an interactive terminal and avoid trying to render interactive progress by default.
Fixes#3935.
Set an option to increase the memory limit on protobuf
parsing so that we can handle larger gRPC payloads.
Co-authored-by: Evan Boyle <EvanBoyle@users.noreply.github.com>
We can't correctly print simple messages for prelude events when doing progress based display in a terminal, as it would lead to resetting the display of the table rendering.
This does mean that `--show-config` no longer works in the default terminal display mode - but it's not clear it *can* work correctly (at least as currently implemented) since it doesn't cleanly participate in the table rendering.
For cases where `--show-config` is not set (the norm) -nothing would have been printed anyway, so the changes here just avoid resetting the table rendering unnecessarily.
Fixes#3469.
The provider plugin protocol is to write a port number followed by `\n`. We must guarantee we do that even on Windows, so must avoid Python `print` statements which implicitly rewrite newlines to platform specific character sequences.
Fixes#3807.
We were seeing that ~all same steps were requiring checkpoint writes due to percieving a difference between `Dependencies` being `nil` and `[]URN{}` - which should be considered the same for this purpose.
For example, pulumi.String also implements pulumi.StringPtr. This is
consistent with the output of the code generator, and makes optional
inputs much more ergonomic.
This assert is not correct in the case of pointer input types, in
particular `pulumi.stringPtr`. Though these types are not assignable,
they are convertible.
The redesign is focused around providing better static typings and
improved ease-of-use for the Go SDK. Most of the redesign revolves
around three pivots:
- Strongly-typed inputs, especially for nested types
- Struct-based resource and invoke APIs
- Ease-of-use of Apply
1. Strongly-typed inputs
Input is the type of a generic input value for a Pulumi resource.
This type is used in conjunction with Output to provide polymorphism
over strongly-typed input values.
The intended pattern for nested Pulumi value types is to define an
input interface and a plain, input, and output variant of the value
type that implement the input interface.
For example, given a nested Pulumi value type with the following shape:
```
type Nested struct {
Foo int
Bar string
}
```
We would define the following:
```
var nestedType = reflect.TypeOf((*Nested)(nil)).Elem()
type NestedInput interface {
pulumi.Input
ToNestedOutput() NestedOutput
ToNestedOutputWithContext(context.Context) NestedOutput
}
type Nested struct {
Foo int `pulumi:"foo"`
Bar string `pulumi:"bar"`
}
type NestedInputValue struct {
Foo pulumi.IntInput `pulumi:"foo"`
Bar pulumi.StringInput `pulumi:"bar"`
}
func (NestedInputValue) ElementType() reflect.Type {
return nestedType
}
func (v NestedInputValue) ToNestedOutput() NestedOutput {
return pulumi.ToOutput(v).(NestedOutput)
}
func (v NestedInputValue) ToNestedOutputWithContext(ctx context.Context) NestedOutput {
return pulumi.ToOutputWithContext(ctx, v).(NestedOutput)
}
type NestedOutput struct { *pulumi.OutputState }
func (NestedOutput) ElementType() reflect.Type {
return nestedType
}
func (o NestedOutput) ToNestedOutput() NestedOutput {
return o
}
func (o NestedOutput) ToNestedOutputWithContext(ctx context.Context) NestedOutput {
return o
}
func (o NestedOutput) Foo() pulumi.IntOutput {
return o.Apply(func (v Nested) int {
return v.Foo
}).(pulumi.IntOutput)
}
func (o NestedOutput) Bar() pulumi.StringOutput {
return o.Apply(func (v Nested) string {
return v.Bar
}).(pulumi.StringOutput)
}
```
The SDK provides input and output types for primitives, arrays, and
maps.
2. Struct-based APIs
Instead of providing expected output properties in the input map passed
to {Read,Register}Resource and returning the outputs as a map, the user
now passes a pointer to a struct that implements one of the Resource
interfaces and has appropriately typed and tagged fields that represent
its output properties.
For example, given a custom resource with an int-typed output "foo" and
a string-typed output "bar", we would define the following
CustomResource type:
```
type MyResource struct {
pulumi.CustomResourceState
Foo pulumi.IntOutput `pulumi:"foo"`
Bar pulumi.StringOutput `pulumi:"bar"`
}
```
And invoke RegisterResource like so:
```
var resource MyResource
err := ctx.RegisterResource(tok, name, props, &resource, opts...)
```
Invoke arguments and results are also provided via structs, but use
plain-old Go types for their fields:
```
type MyInvokeArgs struct {
Foo int `pulumi:"foo"`
}
type MyInvokeResult struct {
Bar string `pulumi:"bar"`
}
var result MyInvokeResult
err := ctx.Invoke(tok, MyInvokeArgs{Foo: 42}, &result, opts...)
```
3. Ease-of-use of Apply
All `Apply` methods now accept an interface{} as the callback type.
The provided callback value must have one of the following signatures:
func (v T) U
func (v T) (U, error)
func (ctx context.Context, v T) U
func (ctx context.Context, v T) (U, error)
T must be assignable from the ElementType of the Output. If U is a type
that has a registered Output type, the result of the Apply will be the
corresponding Output type. Otherwise, the result of the Apply will be
AnyOutput.
Fixes https://github.com/pulumi/pulumi/issues/2149.
Fixes https://github.com/pulumi/pulumi/issues/3488.
Fixes https://github.com/pulumi/pulumi/issues/3487.
Fixes https://github.com/pulumi/pulumi-aws/issues/248.
Fixes https://github.com/pulumi/pulumi/issues/3492.
Fixes https://github.com/pulumi/pulumi/issues/3491.
Fixes https://github.com/pulumi/pulumi/issues/3562.
* Add a new metadata property for BuildNumber. Update Travis and GitLab to set both Build ID and Build Number. Add link to env vars doc for Codefresh.
* Update changelog
* Update CI vars detection test.
* Add PR number to changelog.
* Use Merge Request Instance ID instead of the Merge Request ID for GitLab CI.
* Use GitLab Pipeline Instance ID as the BuildID for GitLab CI.
* Update the changelog.
* Update the test for GitLab CI detection.
* Fix logic to determine PRNumber and BuildURL for Az Pipelines.
* Update changelog
* Set the BranchName to the PR source branch if PRNumber is not empty.
* added support for using GOOGLE_CREDENTIALS environment variable for authenticating with gs:// file state
* modified the change to fix#2791 as well
* fixed a small bug
* fixed linter error
* added code comments
* Update pkg/backend/filestate/gcpauth.go
Co-Authored-By: CyrusNajmabadi <cyrus.najmabadi@gmail.com>
* Parse provided backend url to check if scheme is gs://
* Update changelog
A regression was introduced when we added support for non-Node.js Pulumi programs to run Policy Packs. With that change, we now pass the Policy Pack's full path as the plugin's pwd (so that it would load the `@pulumi/pulumi/cmd/run-policy-pack` Node module from the Policy Pack's node_modules rather than the program's node_modules), but we also pass the path to the policy pack as well. If the path is a full rooted path, this would work fine, and that's what our tests do. However, if a relative path is specified, then it will be looking to load the Policy Pack relative to the pwd, which doesn't produce a correct path leading to failures trying to load the Policy Pack.
Since the pwd is the policy pack path, we can simply pass the path as `"."` to the analyzer plugin, and it will load the policy pack in its pwd.
This change adds support for setting `PULUMI_PREFER_YARN` to true to opt-in to preferring `yarn` over `npm` when installing Node.js dependencies (and publishing Policy Packs). If `PULUMI_PREFER_YARN` is truthy, but `yarn` cannot be found on `$PATH`, we fallback to using `npm`. If `npm` can't be found on `$PATH`, we provide a more helpful error message.
Codepaths which could result in a hang will print a message to the console indicating the problem, along with a link to documentation on how to restructure code to best address it.
`StackReference.getOutputSync` and `requireOutputSync` have been deprecated as they may cause hangs on some combinations of Node and certain OS platforms. `StackReference.getOutput` and `requireOutput` should be used instead.
Fixes building grpc package with gcc8 and newer
Fixes building grpc package for node 13.x
Matches minor grpc release (1.24.x) to version used by dotnet sdk
Fixes: #3248
Before, we got a panic. in the createStack, when we had a non-default
secrets provider, we were assuming the name of the stack was correct
if we were in non-interactive mode
This commit adds a guard against this by doing a final validation of
the stack name *before* we even get into the createStack func
This means, that we get the following (and not the panic)
```
▶ pulumi stack init -s "org/" --secrets-provider="gcpkms://"
error: A stack name may only contain alphanumeric, hyphens, underscores, and periods
```
- If an untargeted create would not affect the inputs of any targeted
resources, do not fail the update. Untargeted creates that are
directly dependend on by targeted resources will still cause failures
that inform the user to add the untargeted resources to the --target
list.
- Users may now pass the `--target-dependents` flag to allow targeted
destroys to automatically target dependents that must be destroyed in
order to destroy an explicitly targeted resource.