The Pulumi Go SDK does not currently await all outstanding asynchronous
work associated with a Pulumi program. Because all relevant asynchronous
work is created via the Pulumi SDK, we can track this asynchronous work
and ensure that it has all completed prior to returning from
`Context.Run`.
This is complicated by the fact that many of the existing APIs that are
able to create `Output`s--`NewOutput`, `ToOutput`, `Any`, `ToSecret`,
and `All`--do not have a `*Context` parameter, and so have no
straightforward way to associate themselves with a `*Context`. To address
this, these changes add new versions of each of these APIs as methods on
`*Context`.
Despite these new methods, most Pulumi programs should work without
changes: the bulk of `Output`s are created by the SDK itself as part of
resource registration, and for `Any` and `All`, we can pick up the
context from any `Output`s present in the arguments. The only programs
that should require changes are those that create outputs from whole
cloth using `NewOutput`, `ToOutput`, or `ToSecret` and create unawaited
async work rooted at those outputs.
On an implementation level, these changes track asynchronous work using
a `sync.WaitGroup` associated with each `*Context`. This `WaitGroup` is
passed to each output associated with the context. The SDK increments
this `WaitGroup`'s count prior to starting any asynchronous work and
decrements it once the work (including any callbacks triggered by the
work) is complete.
This fixes the Go portion of #3991.
* Propagate workspace.Project metadata to plugin init
* Get to a working fix
* Propagate Root via plugin context
* Propagate root instead of yaml path
* Revert out unnecessary parameter propagation
* Root is now always absolute at this point; simplify code and docs
* Drop python conditional and propagate unused -root to all lang hosts
* Add tests that fail before and pass after
* Lint
* Add changelog entry
The Pulumi .NET SDK does not currently await all outstanding asynchronous
work associated with a Pulumi program. Because all relevant asynchronous
work is created via the Pulumi SDK, we can track this asynchronous work
and ensure that it has all completed prior to returning from
`Deployment.RunAsync`.
The implementation here is simpler than that in #6983, and re-uses the
existing support for tracking outstanding RPCs. If this proves to
negatively impact performance (which is a very real possibility for
programs that create many `Output` instances), we can simplify this
using a semaphore and a counter (essentially Go's `sync.WaitGroup`).
This fixes the .NET portion of #3991.
Config values that are objects are represented in memory as JSON strings. When a config map is being saved to a file, object values are first unmarshaled from JSON to `interface{}` and then the entire config map is marshaled to YAML (or JSON) and saved to disk. When an object value is unmarshaled from JSON, any numbers in the JSON string were being implicitly unmarshaled as `float64`, which resulted in some numbers in the nested objects being emitted in YAML using exponential notation (e.g. a number `12321123131` in an object value was being saved in the YAML as `1.2321123131e+10`). To address this, when unmarshaling the JSON for an object value, first try to unmarshal any numbers as `int64`, falling back to `float64`.
* add runtime stack type pulumi fn implementation
* add exception propagation test and update changelog
* slight refactor to just bring the identical service provider code into the new PulumiFn implementation.
* get tests passing, need service to be registered as transient so that the stack is instantiated each time because pulumi internal rely on instantiation
* check version command was failing because of additional white space it wasn't accounting for
Fixes: #6974
Passphrase Environment variables were set before loading the
secrets provider from state
Unfortunately, it seems that some users are using empty passphrases
and thus this newly introduced logic has broken their usecases
We now check that the environment variables are set - it doesn't
matter if they are set as empty, but the existance of an empty
environment variabe still suggests that it is an intentional
empty passphrase
Fixes: #6934
With this snippet of code:
```
func main() {
pulumi.Run(func(ctx *pulumi.Context) error {
// Create an AWS resource (S3 Bucket)
vpc, err := ec2.NewVpc(ctx, "main", &ec2.VpcArgs{
CidrBlock: pulumi.String("10.0.0.0/16"),
})
if err != nil {
return err
}
rt, err := ec2.NewRouteTable(ctx, "example", &ec2.RouteTableArgs{
VpcId: vpc.ID(),
})
// Export the name of the bucket
ctx.Export("rt", rt)
return nil
})
}
```
the CLI would panic on the diff as follows:
```
panic: fatal: An assertion has failed
goroutine 249 [running]:
github.com/pulumi/pulumi/sdk/v3/go/common/util/contract.failfast(...)
/private/tmp/pulumi-20210422-70582-1bpvlru/sdk/go/common/util/contract/failfast.go:23
github.com/pulumi/pulumi/sdk/v3/go/common/util/contract.Assert(...)
/private/tmp/pulumi-20210422-70582-1bpvlru/sdk/go/common/util/contract/assert.go:26
github.com/pulumi/pulumi/pkg/v3/engine.printPropertyValue(0xc0005d41b0, 0x57cce00, 0xc001da9050, 0x0, 0x1, 0x5932853, 0x4, 0x0)
/private/tmp/pulumi-20210422-70582-1bpvlru/pkg/engine/diff.go:511 +0x1485
```
This was due to the entire object being added to the output and
the property being a ResourceReference
On the changing of the code to use a switch statement, we can
now include the ResourceReference and ensure that we catch any
missing case statements with a panic as default
This means the same piece of code now outputs to the CLI as
follows:
```
Outputs:
rt: {
URN: "urn:pulumi:dev::testing-new-engine-diff::aws:ec2/routeTable:RouteTable::example"
ID : "rtb-09b37608ec34f3b49"
PackageVersion: ""
}
Resources:
3 unchanged
Duration: 2s
```
This commit makes it possible to opt out of loading plugins from PATH by
setting PULUMI_IGNORE_AMBIENT_PLUGINS to any non-empty value. This is
useful when automatic IDE tooling may build remote component plugins
into GOBIN unbeknownst to the user, and a resulting stale version of the
plugin is loaded in place of newer versions - even those , explicitly
installed.
Pulumi 3.0 raises an error when a dict value is passed as an input but the type annotation does not accept a dict. Unfortunately, this prevents historical cases where a dict value is allowed but the type annotation doesn't match. We need to fix the type annotations, but in the meantime, we should not raise an error in the SDK for such cases as it breaks existing programs.
Pulumi 3.0 uses type annotations for input values to determine whether dict keys should be translated from snake_case to camelCase or not. This additional inspection of types did not work correctly on Python 3.6 due to some missing functionality on that version of Python which we need to provide an implementation for. Specifically, when inspecting `Union` args to determine whether or not a value is intended to be an input class or user-defined dict. To address the issue, this change improves how we get the args for `Union` types when running on Python 3.6 to behave the same way as later versions of Python (for our purposes). Existing tests fail on Python 3.6 before this change, and pass after.
We were only looking at the current resource class's type/name metadata for camelCase <=> snake_case property name translations which prevented it from working correctly when using a subclass of a resource. This change addresses this by looking at metadata of the current class and any base classes.
Additionally, to help resolve forward references when getting type hints, we'd pass along the current resource class's globals, which doesn't work correctly when using a subclass of a resource. This change also addresses this, by using the globals of the current class and any base classes.
* Import subpackages lazily
* Tighten up lazy_import impl
* Eagerly register resources, but lazily load their impl
* Add CHANGELOG entry
* Satisfy lint
* Restore mypy behavior so the change is not breaking
* Fix golden tests
Fixes: #6286
When a user is using the passphrase provider and unsets the environment
variables that let them interact with the secrets provider, then would
get an error as follows:
```
▶ pulumi up -y -f
error: decrypting secret value: failed to decrypt: incorrect passphrase, please set PULUMI_CONFIG_PASSPHRASE to the correct passphrase
```
We are oging to change this error message to make it more obvious
to a user what the error is and how they need to fix it
```
▶ pulumi up -y -f
error: constructing secrets manager of type "passphrase": unable to find either `PULUMI_CONFIG_PASSPHRASE` nor `PULUMI_CONFIG_PASSPHRASE_FILE` when trying to access the Passphrase Secrets Manager. Please ensure one of these values are set to allow the operation to continue
```
Ideally, we would like to prompt the user for the passphrase at this
point rather than error, but the CLI could be in the middle of an
update operation as the same codepath is used for reading stackreference
values
These changes fix a regression introduced by #6686 that caused the SDK
code generators for .NET, Python, and Typescript to omit definitions for
plain object types. This regression occurred because #6686 drew a
clearer line between types used as resource arguments and types used
as function arguments, but conflated "resource arguments" with "inputty
types". This caused the code generators to generate inputty types for
any types used as resource arguments, even those that are used for
plainly-typed properties.
Fixes#6796.
Resolve providers references and include the resulting refs in the
providers field of RegisterResourceRequest that was added in
d297db3.
Co-authored-by: Justin Van Patten <jvp@justinvp.com>
See #6200 for a complete description of the issue. In short, we generate
inconsistent names for object types depending on whether or not they are
transitively reachable from resources or functions, which risks
unintentional breaking changes due to schema updates.
1. Name "input" types differently: `TArgs` for a type that is used in
resource inputs, having `Input<T>` properties, and `T` for a type
that is used in invoke inputs. The same schema type can produce both.
2. Always keep the name `T` for output types, avoid appending `Result` to
the name.
3. As needed, introduce a flag in the existing providers' schemas to avoid
breaking changes. Consider removing it on a major version bump.
Fixes#6200.
* Initial impl of import/export
* Add test
* Address PR comments
* Do not Select stack but pass it as --stack
* Add CHANGELOG entry
* Update sdk/dotnet/Pulumi.Automation/LocalWorkspace.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Update sdk/dotnet/Pulumi.Automation/LocalWorkspace.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Add helper methods to match node SDK
* Match project settings
* Update CHANGELOG_PENDING.md
Co-authored-by: Komal <komal@pulumi.com>
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
Co-authored-by: Komal <komal@pulumi.com>
* changes necessary for concurrent thread safe global deployment state
* update changelog
* backtrack resource package changes, enable test parallelization
* cleanup comment
* add copyright to new file
* resolve paralellization differences after merging automation api preview
* no longer need to null deployment instance
* Update CHANGELOG.md
* whoops - switch to CHANGELOG_PENDING
* Update CHANGELOG_PENDING.md
move note down to improvements
* attempt to exclude dynami assemblies
Co-authored-by: Anton Tayanovskyy <anton.tayanovskyy@gmail.com>
#6636 inadvertently changed the minimum Go version
requirement to 1.16 since ReadFile was moved in that
version. Switch back to ioutil.ReadFile to avoid forcing
an upgrade at this time.
This change fixes the provider implementation of `Construct` for multi-lang components written in Node.js to wait for any in-flight RPCs to finish before returning the results, s.t. all registered child resources are created.
In additional, invocations of `construct` are now serialized so that each call runs one after another, avoiding concurrent runs, since `construct` modifies global state. We'll follow-up with a more general concurrency fix to allow nested `construct` calls within the same provider.
* Add failing test
* Guard against overrding project settings accidentally
* Throw exception in case of conflct
* Update sdk/dotnet/Pulumi.Automation/DictionaryContentsComparer.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Update sdk/dotnet/Pulumi.Automation/ProjectRuntime.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Update sdk/dotnet/Pulumi.Automation/ProjectTemplateConfigValue.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Update sdk/dotnet/Pulumi.Automation/ProjectTemplate.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Update sdk/dotnet/Pulumi.Automation/ProjectRuntimeOptions.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Update sdk/dotnet/Pulumi.Automation/ProjectBackend.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Update sdk/dotnet/Pulumi.Automation/ProjectSettings.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Reduce nesting
* Make the new exception public
* Introduce a CHANGELOG entry since we add to pub API
* Stricter check before throwing
* Address PR feedback, round 1
* Use Reference.Equals check
* Move DictionaryContentsComparer out of top-level
Co-authored-by: Komal Ali <komal@pulumi.com>
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* [automation/dotnet] Start implementing structured event log support
* Add Events and their models
* Introduce new IStringToEnumConverter
This is intended to be used for converting a non-standard enum names
into corresponding enum values. The converter is utilized via
StringToEnumJsonConverter with OperationType and DiffKind.
* Add event-log support to up, refresh, destroy
* Update event-log tests
* Use lock instead of semaphore in EventLogWatcher
* Add ChangeSummary to PreviewResult
* Minor update
* Add doc comments to Events
* Use PropertyDiffModel in StepEventMetadataModel
* Set Events to public
* Set properties to public
* Fix PreludeEvent.Config doc comment
* Remove TODOs
* Update Pulumi.Automation.xml
* Set PULUMI_DEBUG_COMMANDs in Workspace.
* Split Events and their models to separate files
* Rename abbreviated Events
* Remove CancelEventModel
* Rename HandlesEvents test project and stack
* Update CHANGELOG_PENDING.md
* Rename abbreviated EngineEvent properties
* Implement custom exceptions
* Move event-log setup to LocalPulumiCmd
* Update comments
* Implement polling EventLogWatcher
This replaces the previous watcher which utilized FileSystemWatcher.
* Guard against onEvent throwing
* Dispose CancellationTokenSource
* WIP: try more direct error handling in EventLogWatcher
* A bit more tests
* Reformat
* Update sdk/dotnet/Pulumi.Automation.Tests/EventLogWatcherTests.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Update sdk/dotnet/Pulumi.Automation/Events/EventLogWatcher.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Update sdk/dotnet/Pulumi.Automation/Events/EventLogWatcher.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Update sdk/dotnet/Pulumi.Automation.Tests/EventLogWatcherTests.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Update sdk/dotnet/Pulumi.Automation/Events/EventLogWatcher.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Update sdk/dotnet/Pulumi.Automation/Events/EventLogWatcher.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Make race condition workaround more explicit
* Untabify
* Remove unnecessary using and change String to string
* Update doc comments on Events
Co-authored-by: Anton Tayanovskyy <anton@pulumi.com>
Co-authored-by: Anton Tayanovskyy <anton.tayanovskyy@gmail.com>