We make several calls to `os/user`, which uses CGO and means
cross-compilation is not possible. This replaces `os/user` with the
`luser` package, which is a drop-in replacement which does not use `CGO`
Update pip, setuptools, and wheel in the virtual environment before installing dependencies as recommended by the Python documentation. This should help avoid failures when only source distributions are available for a package and pip attempts to build a wheel locally.
Certain operations in `engine/diff` mutate engine events during display.
This mutation can occur concurrently with the serialization of the event
for persistence, which causes a panic in the CLI. These changes fix the
offending code and add code that copies each engine event before
persisteing it in order to guard against future issues.
Previously, streamInvoke was only supported by
the query command. Copied the implementation
into the resource monitor, which will allow
streaming invoke commands to run during updates.
Also fixed a bug with cancellation of streaming
invokes. The check was comparing against a
hardcoded string, which did not match the actual
error string. Instead, we can rely on the error code.
This function adds a property value to another property at a given path,
creating containing properties as required. If the property cannot be
added because of a mismatch between the value types required by the path
and the values present in the destination, the add will fail. If a value
already exists at the given path, the add will succeed.
Related to #1635.
Several users reported cases where error messages would
cause a panic if they contained accented characters. I wasn't
able to reproduce this failure locally, but tracked down the
panic to logging gRPC calls. The Message field is typed as
a string, which requires all of the characters to be valid UTF-8.
This change runs each log string through the strings.ToValidUTF8
function, which will replace any invalid characters with the
"unknown" character. This should prevent the the logger from
panicking.
Pylint currently reports `E1101: Instance of 'Bucket' has no 'id' member (no-member)` on lines in Pulumi Python programs like:
```python
pulumi.export('bucket_name', bucket.id)
```
Here's a description of this message from http://pylint-messages.wikidot.com/messages:e1101:
> Used when an object (variable, function, …) is accessed for a non-existent member.
>
> False positives: This message may report object members that are created dynamically, but exist at the time they are accessed.
This appears to be a false positive case: `id` isn't set in the constructor (it's set later in `register_resource`) and Pylint isn't able to figure this out statically. `urn` has the same problem. (Oddly, Pylint doesn't complain when accessing other resource output properties).
This change refactors `register_resource` so that `id` and `urn` can be assigned in the resource's constructor, so that Pylint can see it being assigned. The change also does the same with `read_resource`.
Automatically create a virtual environment and install dependencies in it with `pulumi new` and `pulumi policy new` for Python templates.
This will save a new `virtualenv` runtime option in `Pulumi.yaml` (`PulumiPolicy.yaml` for policy packs):
```yaml
runtime:
name: python
options:
virtualenv: venv
```
`virtualenv` is the path to a virtual environment that Pulumi will use when running `python` commands.
Existing projects are unaffected and can opt-in to using this by setting `virtualenv`, otherwise, they'll continue to work as-is.
This class was available in the pulumi.resource module, but was not exported from the core `pulumi` module as intended for all public APIs at this level.
The previous attempt to allow this didn't actually allow it, so this is
take two. As part of the previous attempt, I thought after tweaking the
test I had observed the test failing, and then succeeding after making
the product changes, but I must have been mistaken.
It turns out that our existing mocks tests weren't running at all
because of a missing `__init__.py` file. Once the missing `__init__.py`
is added, the tests run, but other tests ("test mode" tests) fail
because the code that creates the mocks and resources will run during
test discovery, and setting the mocks modifies global state.
To address the test issue, I've moved the mocks tests into their own
`test_with_mocks` package that can be run separately from other tests.
And addressed the original issue, by creating a root Stack resource if
one isn't already present when the mocks are set.
With these changes, a resource struct may tag a field with the empty
string. If such a field is present, any resource outputs that were not
unmarshalled into other fields will be unmarshalled into this field,
which must be a `MapOutput`.
Fixes#4629.
This avoids the "NotSupportedException : Multiple executions of TestAsync must run serially" from calls to `Deployment.TestAsync` in the mocks test, which can happen if a stack test runs before the mock test (.NET unit tests are run in random order).
Despite having the `[assembly: CollectionBehavior(DisableTestParallelization = true)]` attribute, it appears `dotnet test` is still running tests in parallel. To address, use a configuration file to disable parallelization.
Adds support for RegisterResource to accept map-typed implementations if Input as well as the existing struct-typed implementations. Currently these must be fully untyped - but both map[string]pulumi.Input and map[string]interface{} are allowed. In the future, it's plausible that a mode where the data itself is a map, but the ElementType implementation returns a struct could be supported, with the struct used to provide type information over the untyped map.
This change allows importing modules with calls to `pulumi.export` in unit tests. Previously, you'd have to structure the Python program in a way that avoids the `pulumi.export` from being called from unit tests.
* [sdk/python] Improve `ResoruceOptions.merge` type
The implementation correctly handles `None` inputs, so the type should allow these as well.
* Add CHANGELOG
After importing some resources, and running a second update with the
import still applied, an unexpected replace would occur. This wouldn't
happen for the vast majority of resources, but for some it would.
It turns out that the resources that trigger this are ones that use a
different format of identifier for the import input than they do for the
ID property.
Before this change, we would trigger an import-replacement when an
existing resource's ID property didn't match the import property, which
would be the case for the small set of resources where the input
identifier is different than the ID property.
To avoid this, we now store the `importID` in the statefile, and
compare that to the import property instead of comparing the ID.
When setting structured config values using `--path`, we automatically
treat values that can be converted into an integer via `strconv.Atoi` as
an integer, rather than as a string.
However, this ends up converting values like "0123456" into the integer
123456, stripping the leading 0, which isn't desirable for values like
commit SHAs, etc., where you want to keep the 0 (and keep it a string).
This change makes it so that values starting with 0 are not implicitly
converted to an integer; instead such values will remain a string.
* Make `async:true` the default for `invoke` calls (#3750)
* Switch away from native grpc impl. (#3728)
* Remove usage of the 'deasync' library from @pulumi/pulumi. (#3752)
* Only retry as long as we get unavailable back. Anything else continues. (#3769)
* Handle all errors for now. (#3781)
* Do not assume --yes was present when using pulumi in non-interactive mode (#3793)
* Upgrade all paths for sdk and pkg to v2
* Backport C# invoke classes and other recent gen changes (#4288)
Adjust C# generation
* Replace IDeployment with a sealed class (#4318)
Replace IDeployment with a sealed class
* .NET: default to args subtype rather than Args.Empty (#4320)
* Adding system namespace for Dotnet code gen
This is required for using Obsolute attributes for deprecations
```
Iam/InstanceProfile.cs(142,10): error CS0246: The type or namespace name 'ObsoleteAttribute' could not be found (are you missing a using directive or an assembly reference?) [/Users/stack72/code/go/src/github.com/pulumi/pulumi-aws/sdk/dotnet/Pulumi.Aws.csproj]
Iam/InstanceProfile.cs(142,10): error CS0246: The type or namespace name 'Obsolete' could not be found (are you missing a using directive or an assembly reference?) [/Users/stack72/code/go/src/github.com/pulumi/pulumi-aws/sdk/dotnet/Pulumi.Aws.csproj]
```
* Fix the nullability of config type properties in C# codegen (#4379)
Pulumi HCL2 IR:
- Add support for invokes
- Add support for resource options, incl. ranged resources
- Allow the apply rewriter to ignore promise-typed values
- Add tests for the binder
- Add support functions for TF: entries and range
NodeJS codegen:
- Simplify for expression codegen
- Add support for invoke codegen
- Add support for entries and range functions
- Add tests
Python codegen:
- Implement codegen for most expression types
- Add support for invoke codegen
- Add tests
In order to support grpc-js gRPC servers, we need to slightly loosen
our checks on plugin readiness. This change was already in the 2.x
branch, but porting back so that 1.x CLIs can also be compatible.
Fixes#4252.
This was still in pkg and wasn't accessible from the scripts location.
This code should be in the go/common location of sdk so that it can be
accessed from within our build scripts
Without this change, we were not able to get access to the Version number
for Python and none of our Python builds have been publishing binaries
The initial config represents any config that was specified programmatically to the Policy Pack, for Policy Packs that support programmatic configuration like AWSGuard.
These changes implement `GetRequiredPlugins` for Go using a registry
mechanism and an alternate entry point for `pulumi.Run`. Packages that
require plugins are expected to register themselves with the Pulumi SDK.
When `pulumi.Run` is used and the `PULUMI_PLUGINS` envvar is truthy, the
program will dump a JSON-encoded description of its required plugins to
stdout. The language host then uses this description to respond to
It's not entirely clear why gRPC doesn't already report these cleanly as unimplemented, but for now we'll explicitly implement them to avoid any spurious warnings.
Fixes#4028.
The original version of this code caused inconsistencies in the event
loop associated with a given thread. These changes elimintate the event
loop shenanigans the mocks were trying to play by updating _sync_await
to create an event loop if none exists in the current thread.
It's possible that this will cause problems if the tests run on a
different thread than the original program, as the tests are likely to
end up waiting on outputs created by the program, which is not supported
in Python.
Also adds test coverage of the mocking/testing support in Python.
These changes implement `GetRequiredPlugins` for Go using a registry
mechanism and an alternate entry point for `pulumi.Run`. Packages that
require plugins are expected to register themselves with the Pulumi SDK.
When `pulumi.Run` is used and the `PULUMI_PLUGINS` envvar is truthy, the
program will dump a JSON-encoded description of its required plugins to
stdout. The language host then uses this description to respond to
`GetRequiredPlugins`.
* started transformations for go sdk
* added first basic test
* added second test with child
* added RegisterStackTransformation
* added a couple tests to lifecycle_test
* update CHANGELOG and test
* included TODO for #3846
We upgraded to `ts-node@^8.0.0` 2.5 months ago as part of https://github.com/pulumi/pulumi/pull/3627, though it seems it wasn't really necessary to make that update for the purposes of the PR - updating the default TypeScript version.
The `8.0.0` series of `ts-node` unfortunately dropped all of it's caching support, due to what appear to be some corner-case correctness issues with the cache. We have not seen reports of those issues for Pulumi, and have much more experience with the `7.0.0` series overall (2 years vs. 2 months). The performance difference between `7.0.0` and `8.0.0` of ts-node for Pulumi is massive - it adds 4-4.5s to each of `pulumi preview` and `pulumi up` even on a trivial program.
As a result, for now we will revert back to `ts-node@^7.0.0`. In the future, we may want to look into our own caching layer or alternative to `ts-node` to ensure we get the behaviour and performance we expect.
Part of #3671.
These changes add support for mocking the resource monitor to the NodeJS
and Python SDKs. The proposed mock interface is a simplified version of
the standard resource monitor that allows an end-user to replace the
usual implementations of ReadResource/RegisterResource and Invoke with
their own. This can be used in unit tests to allow for precise control
of resource outputs and invoke results.
This updates the protocol buffers generation script in the SDK to use
the main Pulumi Build Container instead of a custom Docker image.
Note there are some path differences owing to the manner in which the
tools are installed in the Build Container, however the actual tool
instantiations are very similar.
These changes add a new method to the resource provider gRPC interface,
`GetSchema`, that allows consumers of these providers to extract
JSON-serialized schema information for the provider's types, resources,
and functions.
In preparation for publishing a separate module of the Go SDK for Pulumi
on which providers can depend, we should reduce the dependency footprint
so as to cause end users as few issues as possible with transitive
dependency versioning.
This commit removes all use of `github.com/pkg/errors` from the Go SDK
to that end, replacing it with the standard `errors` package and `fmt`
for error formatting where appropriate. We use the new (as of Go 1.13)
"%w" syntax for wrapping errors, so this code is no longer compatible
with Go 1.12.
Set an option to increase the memory limit on protobuf
parsing so that we can handle larger gRPC payloads.
Co-authored-by: Evan Boyle <EvanBoyle@users.noreply.github.com>
The provider plugin protocol is to write a port number followed by `\n`. We must guarantee we do that even on Windows, so must avoid Python `print` statements which implicitly rewrite newlines to platform specific character sequences.
Fixes#3807.
* Do not allocate outputs for nested prompt values.
Currently, `outupt`/`all` in the NodeJS SDK work recursively, allocating
outputs for every value at every depth, then collecting the component
promises into a top-level output. In the case of prompt values, these
nested outputs are not necessary, and allocating them can create massive
amounts of garbage. This appears to be the cause of
https://github.com/pulumi/pulumi-kubernetes/issues/963.
- Do not set top-level properties that were not provided
- Check for nil after awaiting inputs
- Check for interface-typed values, which can be produced by `Any`
- Pass a settable value to `awaitInputs` in `Any`
Also:
- Cleaned up existing tags so they're consistently at the bottom of doc comments where they should be
- Cleaned up some unused imports while I was taking a pass over the files
- Marked one function `@deprecated` that should be deprecated
For example, pulumi.String also implements pulumi.StringPtr. This is
consistent with the output of the code generator, and makes optional
inputs much more ergonomic.
This assert is not correct in the case of pointer input types, in
particular `pulumi.stringPtr`. Though these types are not assignable,
they are convertible.
The redesign is focused around providing better static typings and
improved ease-of-use for the Go SDK. Most of the redesign revolves
around three pivots:
- Strongly-typed inputs, especially for nested types
- Struct-based resource and invoke APIs
- Ease-of-use of Apply
1. Strongly-typed inputs
Input is the type of a generic input value for a Pulumi resource.
This type is used in conjunction with Output to provide polymorphism
over strongly-typed input values.
The intended pattern for nested Pulumi value types is to define an
input interface and a plain, input, and output variant of the value
type that implement the input interface.
For example, given a nested Pulumi value type with the following shape:
```
type Nested struct {
Foo int
Bar string
}
```
We would define the following:
```
var nestedType = reflect.TypeOf((*Nested)(nil)).Elem()
type NestedInput interface {
pulumi.Input
ToNestedOutput() NestedOutput
ToNestedOutputWithContext(context.Context) NestedOutput
}
type Nested struct {
Foo int `pulumi:"foo"`
Bar string `pulumi:"bar"`
}
type NestedInputValue struct {
Foo pulumi.IntInput `pulumi:"foo"`
Bar pulumi.StringInput `pulumi:"bar"`
}
func (NestedInputValue) ElementType() reflect.Type {
return nestedType
}
func (v NestedInputValue) ToNestedOutput() NestedOutput {
return pulumi.ToOutput(v).(NestedOutput)
}
func (v NestedInputValue) ToNestedOutputWithContext(ctx context.Context) NestedOutput {
return pulumi.ToOutputWithContext(ctx, v).(NestedOutput)
}
type NestedOutput struct { *pulumi.OutputState }
func (NestedOutput) ElementType() reflect.Type {
return nestedType
}
func (o NestedOutput) ToNestedOutput() NestedOutput {
return o
}
func (o NestedOutput) ToNestedOutputWithContext(ctx context.Context) NestedOutput {
return o
}
func (o NestedOutput) Foo() pulumi.IntOutput {
return o.Apply(func (v Nested) int {
return v.Foo
}).(pulumi.IntOutput)
}
func (o NestedOutput) Bar() pulumi.StringOutput {
return o.Apply(func (v Nested) string {
return v.Bar
}).(pulumi.StringOutput)
}
```
The SDK provides input and output types for primitives, arrays, and
maps.
2. Struct-based APIs
Instead of providing expected output properties in the input map passed
to {Read,Register}Resource and returning the outputs as a map, the user
now passes a pointer to a struct that implements one of the Resource
interfaces and has appropriately typed and tagged fields that represent
its output properties.
For example, given a custom resource with an int-typed output "foo" and
a string-typed output "bar", we would define the following
CustomResource type:
```
type MyResource struct {
pulumi.CustomResourceState
Foo pulumi.IntOutput `pulumi:"foo"`
Bar pulumi.StringOutput `pulumi:"bar"`
}
```
And invoke RegisterResource like so:
```
var resource MyResource
err := ctx.RegisterResource(tok, name, props, &resource, opts...)
```
Invoke arguments and results are also provided via structs, but use
plain-old Go types for their fields:
```
type MyInvokeArgs struct {
Foo int `pulumi:"foo"`
}
type MyInvokeResult struct {
Bar string `pulumi:"bar"`
}
var result MyInvokeResult
err := ctx.Invoke(tok, MyInvokeArgs{Foo: 42}, &result, opts...)
```
3. Ease-of-use of Apply
All `Apply` methods now accept an interface{} as the callback type.
The provided callback value must have one of the following signatures:
func (v T) U
func (v T) (U, error)
func (ctx context.Context, v T) U
func (ctx context.Context, v T) (U, error)
T must be assignable from the ElementType of the Output. If U is a type
that has a registered Output type, the result of the Apply will be the
corresponding Output type. Otherwise, the result of the Apply will be
AnyOutput.
Fixes https://github.com/pulumi/pulumi/issues/2149.
Fixes https://github.com/pulumi/pulumi/issues/3488.
Fixes https://github.com/pulumi/pulumi/issues/3487.
Fixes https://github.com/pulumi/pulumi-aws/issues/248.
Fixes https://github.com/pulumi/pulumi/issues/3492.
Fixes https://github.com/pulumi/pulumi/issues/3491.
Fixes https://github.com/pulumi/pulumi/issues/3562.
PEP 561 specifies that packages which contain either inline type hints
or type stubs should indicate their support for type hints via
including a file named `py.typed` in the root of the package. Since
Pulumi already includes inline type hints, adding `py.typed` to the
Python SDK simply allows these hints to be used by mypy.
Fix the following warning and turn on "treat warnings as errors" for our .NET projects.
```
Deployment/InvokeOptions.cs(6,60): warning CS1574: XML comment has cref attribute 'InvokeAsync{T}(string,
```
