Python resource constructor overloads were recently added that accept a
`<Resource>Args` class for input properties, as an alternative to the
other constructor overload that accepts keyword arguments. The name of
the new args class is the name of the resource concatenated with an
`Args` suffix.
Some providers (e.g. Kubernetes, Azure Native, and Google Native) have
input types with the same name as resources in the same module, which
results in two different `<Resource>Args` classes in the same module.
When you try to use the new args class with the constructor, e.g.:
```python
pulumi_kubernetes.storage.v1.StorageClass(
resource_name='string',
args=pulumi_kubernetes.storage.v1.StorageClassArgs(...),
opts=pulumi.ResourceOptions(...),
)
```
You run into an error, because
`pulumi_kubernetes.storage.v1.StorageClassArgs` is actually referring to
the existing input type rather than the intended `StorageClassArgs`
class for the constructor arguments.
Having the duplicate classes hasn't broken existing usage of the input
type because we "export" all the input types for a module _after_ all
the resources and resource args classes are exported, so the input type
just ends up "overwriting" the duplicate resource args class.
Other languages don't have this problem because the input type is either
in it's own module/namespace (e.g. Node.js and .NET) or a different name
is used for the input type (Go). But with Python, the input types and
resources are all available in the same module.
To address this for Python, when there is an input type in the same
module with the same name as the resource, the args class for the
resource will be emitted as `<Resource>InitArgs` instead of
`<Resource>Args`.
This commit modifies the work in #7058 to permit properties which do not
pass the test of being strings directly, but which have an underlying
type of string.
When applied to `pulumi-aws`, this results in the following diff:
```
diff --git a/sdk/go/aws/provider.go b/sdk/go/aws/provider.go
index c32ad2367..8b4c9fd0a 100644
--- a/sdk/go/aws/provider.go
+++ b/sdk/go/aws/provider.go
@@ -21,6 +21,8 @@ type Provider struct {
AccessKey pulumi.StringPtrOutput `pulumi:"accessKey"`
// The profile for API operations. If not set, the default profile created with `aws configure` will be used.
Profile pulumi.StringPtrOutput `pulumi:"profile"`
+ // The region where AWS operations will take place. Examples are us-east-1, us-west-2, etc.
+ Region pulumi.StringPtrOutput `pulumi:"region"`
// The secret key for API operations. You can retrieve this from the 'Security & Credentials' section of the AWS console.
SecretKey pulumi.StringPtrOutput `pulumi:"secretKey"`
// The path to the shared credentials file. If not set this defaults to ~/.aws/credentials.
```
The primary purpose this is desirable is to expose Region from instances
of the AWS provider.
* Add get value async to output utilities
asdasd
* Update changelog
* Move public apis from unshipped to shipped
Co-authored-by: Anton Tayanovskyy <anton@pulumi.com>
Fix the generated C# code for plain properties:
- Value types should not be initialized with `= null!`
- Arrays and maps should be `List<T>` and `Dictionary<string, TValue>`
These mutually recursive functions unintentionally had exponential complexity in nesting depth of objects, arg types and most likely arrays.
Remove the exponential complexity by avoiding direct recursion of from_input on itself, and relying on mutual recursion with all alone to reduce nested substructure.
Also simplify the implementation to aid readability.
Fixes pulumi/pulumi-kubernetes#1597.
Fixes pulumi/pulumi-kubernetes#1425.
Fixes pulumi/pulumi-kubernetes#1372.
Fixes#3987.
* Send plugin install output to stderr
We currently send plugin install output to stdout. This interferes
with --json (#5747), automation API scenarios, and in general is bad
CLI hygiene. This change sends plugin output to stdout instead.
* Add a changelog entry
Temporarily disable the new config secret warning to avoid unactionable warnings from provider `config` modules. We'll re-enable the warning when we've addressed that issue.
This commit adds a fallback for the README definition in the generated
setup.py files for Python SDKs, thus allowing editable installs of
packages which not yet been built.
Co-authored-by: Luke Hoban <luke@pulumi.com>
The Pulumi Go SDK does not currently await all outstanding asynchronous
work associated with a Pulumi program. Because all relevant asynchronous
work is created via the Pulumi SDK, we can track this asynchronous work
and ensure that it has all completed prior to returning from
`Context.Run`.
This is complicated by the fact that many of the existing APIs that are
able to create `Output`s--`NewOutput`, `ToOutput`, `Any`, `ToSecret`,
and `All`--do not have a `*Context` parameter, and so have no
straightforward way to associate themselves with a `*Context`. To address
this, these changes add new versions of each of these APIs as methods on
`*Context`.
Despite these new methods, most Pulumi programs should work without
changes: the bulk of `Output`s are created by the SDK itself as part of
resource registration, and for `Any` and `All`, we can pick up the
context from any `Output`s present in the arguments. The only programs
that should require changes are those that create outputs from whole
cloth using `NewOutput`, `ToOutput`, or `ToSecret` and create unawaited
async work rooted at those outputs.
On an implementation level, these changes track asynchronous work using
a `sync.WaitGroup` associated with each `*Context`. This `WaitGroup` is
passed to each output associated with the context. The SDK increments
this `WaitGroup`'s count prior to starting any asynchronous work and
decrements it once the work (including any callbacks triggered by the
work) is complete.
This fixes the Go portion of #3991.
* Propagate workspace.Project metadata to plugin init
* Get to a working fix
* Propagate Root via plugin context
* Propagate root instead of yaml path
* Revert out unnecessary parameter propagation
* Root is now always absolute at this point; simplify code and docs
* Drop python conditional and propagate unused -root to all lang hosts
* Add tests that fail before and pass after
* Lint
* Add changelog entry
The Pulumi .NET SDK does not currently await all outstanding asynchronous
work associated with a Pulumi program. Because all relevant asynchronous
work is created via the Pulumi SDK, we can track this asynchronous work
and ensure that it has all completed prior to returning from
`Deployment.RunAsync`.
The implementation here is simpler than that in #6983, and re-uses the
existing support for tracking outstanding RPCs. If this proves to
negatively impact performance (which is a very real possibility for
programs that create many `Output` instances), we can simplify this
using a semaphore and a counter (essentially Go's `sync.WaitGroup`).
This fixes the .NET portion of #3991.
Config values that are objects are represented in memory as JSON strings. When a config map is being saved to a file, object values are first unmarshaled from JSON to `interface{}` and then the entire config map is marshaled to YAML (or JSON) and saved to disk. When an object value is unmarshaled from JSON, any numbers in the JSON string were being implicitly unmarshaled as `float64`, which resulted in some numbers in the nested objects being emitted in YAML using exponential notation (e.g. a number `12321123131` in an object value was being saved in the YAML as `1.2321123131e+10`). To address this, when unmarshaling the JSON for an object value, first try to unmarshal any numbers as `int64`, falling back to `float64`.
* add runtime stack type pulumi fn implementation
* add exception propagation test and update changelog
* slight refactor to just bring the identical service provider code into the new PulumiFn implementation.
* get tests passing, need service to be registered as transient so that the stack is instantiated each time because pulumi internal rely on instantiation
* check version command was failing because of additional white space it wasn't accounting for
Fixes: #6974
Passphrase Environment variables were set before loading the
secrets provider from state
Unfortunately, it seems that some users are using empty passphrases
and thus this newly introduced logic has broken their usecases
We now check that the environment variables are set - it doesn't
matter if they are set as empty, but the existance of an empty
environment variabe still suggests that it is an intentional
empty passphrase
Fixes: #6934
With this snippet of code:
```
func main() {
pulumi.Run(func(ctx *pulumi.Context) error {
// Create an AWS resource (S3 Bucket)
vpc, err := ec2.NewVpc(ctx, "main", &ec2.VpcArgs{
CidrBlock: pulumi.String("10.0.0.0/16"),
})
if err != nil {
return err
}
rt, err := ec2.NewRouteTable(ctx, "example", &ec2.RouteTableArgs{
VpcId: vpc.ID(),
})
// Export the name of the bucket
ctx.Export("rt", rt)
return nil
})
}
```
the CLI would panic on the diff as follows:
```
panic: fatal: An assertion has failed
goroutine 249 [running]:
github.com/pulumi/pulumi/sdk/v3/go/common/util/contract.failfast(...)
/private/tmp/pulumi-20210422-70582-1bpvlru/sdk/go/common/util/contract/failfast.go:23
github.com/pulumi/pulumi/sdk/v3/go/common/util/contract.Assert(...)
/private/tmp/pulumi-20210422-70582-1bpvlru/sdk/go/common/util/contract/assert.go:26
github.com/pulumi/pulumi/pkg/v3/engine.printPropertyValue(0xc0005d41b0, 0x57cce00, 0xc001da9050, 0x0, 0x1, 0x5932853, 0x4, 0x0)
/private/tmp/pulumi-20210422-70582-1bpvlru/pkg/engine/diff.go:511 +0x1485
```
This was due to the entire object being added to the output and
the property being a ResourceReference
On the changing of the code to use a switch statement, we can
now include the ResourceReference and ensure that we catch any
missing case statements with a panic as default
This means the same piece of code now outputs to the CLI as
follows:
```
Outputs:
rt: {
URN: "urn:pulumi:dev::testing-new-engine-diff::aws:ec2/routeTable:RouteTable::example"
ID : "rtb-09b37608ec34f3b49"
PackageVersion: ""
}
Resources:
3 unchanged
Duration: 2s
```
This commit makes it possible to opt out of loading plugins from PATH by
setting PULUMI_IGNORE_AMBIENT_PLUGINS to any non-empty value. This is
useful when automatic IDE tooling may build remote component plugins
into GOBIN unbeknownst to the user, and a resulting stale version of the
plugin is loaded in place of newer versions - even those , explicitly
installed.
Pulumi 3.0 raises an error when a dict value is passed as an input but the type annotation does not accept a dict. Unfortunately, this prevents historical cases where a dict value is allowed but the type annotation doesn't match. We need to fix the type annotations, but in the meantime, we should not raise an error in the SDK for such cases as it breaks existing programs.
Pulumi 3.0 uses type annotations for input values to determine whether dict keys should be translated from snake_case to camelCase or not. This additional inspection of types did not work correctly on Python 3.6 due to some missing functionality on that version of Python which we need to provide an implementation for. Specifically, when inspecting `Union` args to determine whether or not a value is intended to be an input class or user-defined dict. To address the issue, this change improves how we get the args for `Union` types when running on Python 3.6 to behave the same way as later versions of Python (for our purposes). Existing tests fail on Python 3.6 before this change, and pass after.
We were only looking at the current resource class's type/name metadata for camelCase <=> snake_case property name translations which prevented it from working correctly when using a subclass of a resource. This change addresses this by looking at metadata of the current class and any base classes.
Additionally, to help resolve forward references when getting type hints, we'd pass along the current resource class's globals, which doesn't work correctly when using a subclass of a resource. This change also addresses this, by using the globals of the current class and any base classes.
* Import subpackages lazily
* Tighten up lazy_import impl
* Eagerly register resources, but lazily load their impl
* Add CHANGELOG entry
* Satisfy lint
* Restore mypy behavior so the change is not breaking
* Fix golden tests
Fixes: #6286
When a user is using the passphrase provider and unsets the environment
variables that let them interact with the secrets provider, then would
get an error as follows:
```
▶ pulumi up -y -f
error: decrypting secret value: failed to decrypt: incorrect passphrase, please set PULUMI_CONFIG_PASSPHRASE to the correct passphrase
```
We are oging to change this error message to make it more obvious
to a user what the error is and how they need to fix it
```
▶ pulumi up -y -f
error: constructing secrets manager of type "passphrase": unable to find either `PULUMI_CONFIG_PASSPHRASE` nor `PULUMI_CONFIG_PASSPHRASE_FILE` when trying to access the Passphrase Secrets Manager. Please ensure one of these values are set to allow the operation to continue
```
Ideally, we would like to prompt the user for the passphrase at this
point rather than error, but the CLI could be in the middle of an
update operation as the same codepath is used for reading stackreference
values
These changes fix a regression introduced by #6686 that caused the SDK
code generators for .NET, Python, and Typescript to omit definitions for
plain object types. This regression occurred because #6686 drew a
clearer line between types used as resource arguments and types used
as function arguments, but conflated "resource arguments" with "inputty
types". This caused the code generators to generate inputty types for
any types used as resource arguments, even those that are used for
plainly-typed properties.
Fixes#6796.
Resolve providers references and include the resulting refs in the
providers field of RegisterResourceRequest that was added in
d297db3.
Co-authored-by: Justin Van Patten <jvp@justinvp.com>
See #6200 for a complete description of the issue. In short, we generate
inconsistent names for object types depending on whether or not they are
transitively reachable from resources or functions, which risks
unintentional breaking changes due to schema updates.
1. Name "input" types differently: `TArgs` for a type that is used in
resource inputs, having `Input<T>` properties, and `T` for a type
that is used in invoke inputs. The same schema type can produce both.
2. Always keep the name `T` for output types, avoid appending `Result` to
the name.
3. As needed, introduce a flag in the existing providers' schemas to avoid
breaking changes. Consider removing it on a major version bump.
Fixes#6200.
* Initial impl of import/export
* Add test
* Address PR comments
* Do not Select stack but pass it as --stack
* Add CHANGELOG entry
* Update sdk/dotnet/Pulumi.Automation/LocalWorkspace.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Update sdk/dotnet/Pulumi.Automation/LocalWorkspace.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Add helper methods to match node SDK
* Match project settings
* Update CHANGELOG_PENDING.md
Co-authored-by: Komal <komal@pulumi.com>
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
Co-authored-by: Komal <komal@pulumi.com>
* changes necessary for concurrent thread safe global deployment state
* update changelog
* backtrack resource package changes, enable test parallelization
* cleanup comment
* add copyright to new file
* resolve paralellization differences after merging automation api preview
* no longer need to null deployment instance
* Update CHANGELOG.md
* whoops - switch to CHANGELOG_PENDING
* Update CHANGELOG_PENDING.md
move note down to improvements
* attempt to exclude dynami assemblies
Co-authored-by: Anton Tayanovskyy <anton.tayanovskyy@gmail.com>
#6636 inadvertently changed the minimum Go version
requirement to 1.16 since ReadFile was moved in that
version. Switch back to ioutil.ReadFile to avoid forcing
an upgrade at this time.
This change fixes the provider implementation of `Construct` for multi-lang components written in Node.js to wait for any in-flight RPCs to finish before returning the results, s.t. all registered child resources are created.
In additional, invocations of `construct` are now serialized so that each call runs one after another, avoiding concurrent runs, since `construct` modifies global state. We'll follow-up with a more general concurrency fix to allow nested `construct` calls within the same provider.
* Add failing test
* Guard against overrding project settings accidentally
* Throw exception in case of conflct
* Update sdk/dotnet/Pulumi.Automation/DictionaryContentsComparer.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Update sdk/dotnet/Pulumi.Automation/ProjectRuntime.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Update sdk/dotnet/Pulumi.Automation/ProjectTemplateConfigValue.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Update sdk/dotnet/Pulumi.Automation/ProjectTemplate.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Update sdk/dotnet/Pulumi.Automation/ProjectRuntimeOptions.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Update sdk/dotnet/Pulumi.Automation/ProjectBackend.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Update sdk/dotnet/Pulumi.Automation/ProjectSettings.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Reduce nesting
* Make the new exception public
* Introduce a CHANGELOG entry since we add to pub API
* Stricter check before throwing
* Address PR feedback, round 1
* Use Reference.Equals check
* Move DictionaryContentsComparer out of top-level
Co-authored-by: Komal Ali <komal@pulumi.com>
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* [automation/dotnet] Start implementing structured event log support
* Add Events and their models
* Introduce new IStringToEnumConverter
This is intended to be used for converting a non-standard enum names
into corresponding enum values. The converter is utilized via
StringToEnumJsonConverter with OperationType and DiffKind.
* Add event-log support to up, refresh, destroy
* Update event-log tests
* Use lock instead of semaphore in EventLogWatcher
* Add ChangeSummary to PreviewResult
* Minor update
* Add doc comments to Events
* Use PropertyDiffModel in StepEventMetadataModel
* Set Events to public
* Set properties to public
* Fix PreludeEvent.Config doc comment
* Remove TODOs
* Update Pulumi.Automation.xml
* Set PULUMI_DEBUG_COMMANDs in Workspace.
* Split Events and their models to separate files
* Rename abbreviated Events
* Remove CancelEventModel
* Rename HandlesEvents test project and stack
* Update CHANGELOG_PENDING.md
* Rename abbreviated EngineEvent properties
* Implement custom exceptions
* Move event-log setup to LocalPulumiCmd
* Update comments
* Implement polling EventLogWatcher
This replaces the previous watcher which utilized FileSystemWatcher.
* Guard against onEvent throwing
* Dispose CancellationTokenSource
* WIP: try more direct error handling in EventLogWatcher
* A bit more tests
* Reformat
* Update sdk/dotnet/Pulumi.Automation.Tests/EventLogWatcherTests.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Update sdk/dotnet/Pulumi.Automation/Events/EventLogWatcher.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Update sdk/dotnet/Pulumi.Automation/Events/EventLogWatcher.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Update sdk/dotnet/Pulumi.Automation.Tests/EventLogWatcherTests.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Update sdk/dotnet/Pulumi.Automation/Events/EventLogWatcher.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Update sdk/dotnet/Pulumi.Automation/Events/EventLogWatcher.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Make race condition workaround more explicit
* Untabify
* Remove unnecessary using and change String to string
* Update doc comments on Events
Co-authored-by: Anton Tayanovskyy <anton@pulumi.com>
Co-authored-by: Anton Tayanovskyy <anton.tayanovskyy@gmail.com>
The current logic lets unhandled errors in the RPC invocation
unahandled in the async loop, which crashes the process due to
the way we await completion of RPCs before exiting the process
in Python. Instead of doing that, we can just marshal them back
to the synchronous awaiter as part of the calling convention,
and have that awaiter (which is called by the invoke methods)
re-raise the exception. This should fixpulumi/pulumi#3611.
Some YAML parsers don't correctly handle Byte-order marks,
so automatically strip it off during load.
Related to #423
Co-authored-by: Justin Van Patten <jvp@justinvp.com>
This commit adds a new optional parameter to the `newResource` function
of the `Mocks` interface for TypeScript. This can be useful when writing
tests which assert differing behavior between Custom and Component
resources.
Although the new parameter will always be set, the paramteter is marked
as optional in order to maintain backwards compatibility with existing
implementations of `Mocks`.
The tests are updated to verify that `custom` is set appropriately.
Co-authored-by: Luke Hoban <luke@pulumi.com>
* Avoid double-quailfying venv folder path
* Replace `path` with `filepath`
* Add a Python integration test to cover venv auto-creation
* Merged
* Fix spelling
Co-authored-by: Justin Van Patten <jvp@justinvp.com>
* Make AbsPath and RelPath test variants
* Fix issue on Windows backslash paths
* Debug windows test failure: more logging and aggressive YAML escaping
* Use filepath.IsAbs instead of path.IsAbs
Co-authored-by: Justin Van Patten <jvp@justinvp.com>
The Project and Stack save routines were erroneously
dumping the Python objects rather than the __dict__
property, which resulted in some extra annotations
in the resulting YAML files. Some parsers don't handle
these annotations correctly, and consider the resulting
YAML file to be invalid.
* [sdk/go] Cache loaded configuration files
Previously, the CLI did not cache configuration files, which
required a read from disk + unmarshalling + validation each
time a consumer needed to read one of these configurations.
This change introduces global caches for each type of Pulumi
configuration file (Project, ProjectStack, PolicyPackProject, and
PluginProject). The configuration is cached after the first request
and the cached value will be used for any subsequent operations.
Important note: The global configurations are not concurrency safe,
but this same problem exists using the previous method of
reading/writing config files on disk. Synchronization
will be added in a follow up change to allow for concurrency safe config
operations.
This change avoids `RuntimeError: There is no current event loop in thread '<thread_name>'` errors when passing a resource as an input multiple times when using mocks.
The problem is that when using mocks, we deserialize the gRPC inputs before passing them to the user's mock methods. Deserializing inputs doesn't typically require an event loop, however, during deserialization of resource references, we end up creating some instances of `Future`, which does require an event loop to be present for the current thread. If this is done multiple times for a resource, it's possible that `deserialize_properties` will be called on an asyncio thread that doesn't yet have an event loop, resulting in the error being raised.
The error does not occur when only passing the resource reference once because typically the thread (e.g. `asyncio_0`) used in that case will have already had an event loop created for it due to the use of the internal `_syncawait` when _serializing_ the source resource's properties, which ensures an event loop is set for the thread.
The fix is to ensure an event loop is created for the thread in the mocks implementation before calling `deserialize_properties`.
As part of our continued effort to make our releases more useful,
we will be adding our CHANGELOG entries to the GitHub Release.
To make this process smooth, we are going to change things a little:
1. All new changelog entries when submitting a PR for an upcoming
release will now need to get added to CHANGELOG_PENDING.md
This is the source of information for what will be delivered in the
release.
2. When a release is being made, the entries from CHANGELOG_PENDING
will be copied to a new version and dated section in CHANGLOG to
mark the release
3. The GH tags will continue as normal and Goreleaser will copy
the changelog entries to the release section in GH
