Rotating a passphrase requires that the old passphrase is available via
one of the `PULUMI_CONFIG_PASSPHRASE` or `PULUMI_CONFIG_PASSPHRASE_FILE`
environment variables. This confuses `readPassphrase` when reading a new
passphrase, since that function checks the aforementioned environment
variables prior to reading from the console. The overall effect is that
it is impossible to rotate the passphrase for a stack using the
passphrase provider. These changes fix this by always reading from the
console when rotating a passphrase.
* resolve issue with exit code not bubbling up and inflight tasks swallowing exceptions
* update changelog
* refacter to reduce and centralize inline program exception flow complexity
* quick comments on new pulumi function return type
This commit moves from simply asserting on the assignability of types to
using `Assertf` to print the types in question. This provides more
information to a user whose code is panicking because of
non-assignability.
Ultimately it would likely be better to surface this via error messages
instead of via panic, but this at least improves the debuggability in
the meantime.
This command converts an appdash trace into a pprof file for use with
`go tool pprof`. Spans are converted into stacks by sampling each root
span at a given rate and recording the stack of subspans at each sample.
These changes also replace the conditional addition of experimental and
debug commands with conditional visibility. Experimental and debug
commands will always be available, but will be hidden unless the
appropraite environment variables are set.
Co-authored-by: Levi Blackstone <levi@pulumi.com>
Although `Output` objects can never correct support iteration, Python will see the implementation of `__getitem__` and try to iterate the object, leading to an infinite loop. To prevent this, we need to explicitly implement `__iter__` and make it return a `TypeError` to prevent iteration (and offer a useful error message).
Fixes#5028.
`log.debug` messages are part of an update and get sent to the service. Reducing the number of `log.debug` calls can significantly improve the performance of a program. This commit changes the debug logging in the Python SDK to be similar to the debug logging in the Node.js SDK.
- An `excessive_debug_output` variable has been added (that can be set to `True` during debugging), but otherwise will avoid a large number of `log.debug` calls.
- Some unhelpful/redundant `log.debug` calls have been deleted.
* Add trace proxying to fix sub-process trace collection when tracing to files
* Better func naming in test
* Avoid dealing with Windows path nightmare
* On Windows it is go.exe of course
* Rename operation to component to better align with existing trace output
* Add get value async to output utilities
asdasd
* Update changelog
* Move public apis from unshipped to shipped
Co-authored-by: Anton Tayanovskyy <anton@pulumi.com>
These mutually recursive functions unintentionally had exponential complexity in nesting depth of objects, arg types and most likely arrays.
Remove the exponential complexity by avoiding direct recursion of from_input on itself, and relying on mutual recursion with all alone to reduce nested substructure.
Also simplify the implementation to aid readability.
Fixes pulumi/pulumi-kubernetes#1597.
Fixes pulumi/pulumi-kubernetes#1425.
Fixes pulumi/pulumi-kubernetes#1372.
Fixes#3987.
This change is a simple perf optimization to speed up the process of listing plugins by excluding some metadata like size by default.
Our strategy for finding a plugin is to first look on the path, and then to iterate through all plugins in the plugin cache (a directory). We do this for each plugin that is loaded when NewProvider is called. Unfortunately, the codepath that gets all plugins is shared by pulumi plugin ls that needs to do things like display the total size of all plugins, the size of each plugin, and when the plugin was last installed/last used.
This means that any time a plugin is loaded, we are computing the size of all plugins by recursively enumerating all folder (including all of the node_modules directories of any installed node multi-lang plugins!). For my 5 gb of node plugins this translated to 10s of overhead each time a plugin was loaded.
This change is a very simple fix. pulumi plugin ls is the only code path that uses size, so we create a dedicated code path GetPluginsWithMetadata that populates that info, excluding from the result of GetPlugins by default.
* Send plugin install output to stderr
We currently send plugin install output to stdout. This interferes
with --json (#5747), automation API scenarios, and in general is bad
CLI hygiene. This change sends plugin output to stdout instead.
* Add a changelog entry
Temporarily disable the new config secret warning to avoid unactionable warnings from provider `config` modules. We'll re-enable the warning when we've addressed that issue.
The Pulumi Go SDK does not currently await all outstanding asynchronous
work associated with a Pulumi program. Because all relevant asynchronous
work is created via the Pulumi SDK, we can track this asynchronous work
and ensure that it has all completed prior to returning from
`Context.Run`.
This is complicated by the fact that many of the existing APIs that are
able to create `Output`s--`NewOutput`, `ToOutput`, `Any`, `ToSecret`,
and `All`--do not have a `*Context` parameter, and so have no
straightforward way to associate themselves with a `*Context`. To address
this, these changes add new versions of each of these APIs as methods on
`*Context`.
Despite these new methods, most Pulumi programs should work without
changes: the bulk of `Output`s are created by the SDK itself as part of
resource registration, and for `Any` and `All`, we can pick up the
context from any `Output`s present in the arguments. The only programs
that should require changes are those that create outputs from whole
cloth using `NewOutput`, `ToOutput`, or `ToSecret` and create unawaited
async work rooted at those outputs.
On an implementation level, these changes track asynchronous work using
a `sync.WaitGroup` associated with each `*Context`. This `WaitGroup` is
passed to each output associated with the context. The SDK increments
this `WaitGroup`'s count prior to starting any asynchronous work and
decrements it once the work (including any callbacks triggered by the
work) is complete.
This fixes the Go portion of #3991.
* Propagate workspace.Project metadata to plugin init
* Get to a working fix
* Propagate Root via plugin context
* Propagate root instead of yaml path
* Revert out unnecessary parameter propagation
* Root is now always absolute at this point; simplify code and docs
* Drop python conditional and propagate unused -root to all lang hosts
* Add tests that fail before and pass after
* Lint
* Add changelog entry
The Pulumi .NET SDK does not currently await all outstanding asynchronous
work associated with a Pulumi program. Because all relevant asynchronous
work is created via the Pulumi SDK, we can track this asynchronous work
and ensure that it has all completed prior to returning from
`Deployment.RunAsync`.
The implementation here is simpler than that in #6983, and re-uses the
existing support for tracking outstanding RPCs. If this proves to
negatively impact performance (which is a very real possibility for
programs that create many `Output` instances), we can simplify this
using a semaphore and a counter (essentially Go's `sync.WaitGroup`).
This fixes the .NET portion of #3991.
Config values that are objects are represented in memory as JSON strings. When a config map is being saved to a file, object values are first unmarshaled from JSON to `interface{}` and then the entire config map is marshaled to YAML (or JSON) and saved to disk. When an object value is unmarshaled from JSON, any numbers in the JSON string were being implicitly unmarshaled as `float64`, which resulted in some numbers in the nested objects being emitted in YAML using exponential notation (e.g. a number `12321123131` in an object value was being saved in the YAML as `1.2321123131e+10`). To address this, when unmarshaling the JSON for an object value, first try to unmarshal any numbers as `int64`, falling back to `float64`.
* add runtime stack type pulumi fn implementation
* add exception propagation test and update changelog
* slight refactor to just bring the identical service provider code into the new PulumiFn implementation.
* get tests passing, need service to be registered as transient so that the stack is instantiated each time because pulumi internal rely on instantiation
* check version command was failing because of additional white space it wasn't accounting for
This commit makes it possible to opt out of loading plugins from PATH by
setting PULUMI_IGNORE_AMBIENT_PLUGINS to any non-empty value. This is
useful when automatic IDE tooling may build remote component plugins
into GOBIN unbeknownst to the user, and a resulting stale version of the
plugin is loaded in place of newer versions - even those , explicitly
installed.
Pulumi 3.0 raises an error when a dict value is passed as an input but the type annotation does not accept a dict. Unfortunately, this prevents historical cases where a dict value is allowed but the type annotation doesn't match. We need to fix the type annotations, but in the meantime, we should not raise an error in the SDK for such cases as it breaks existing programs.
Pulumi 3.0 uses type annotations for input values to determine whether dict keys should be translated from snake_case to camelCase or not. This additional inspection of types did not work correctly on Python 3.6 due to some missing functionality on that version of Python which we need to provide an implementation for. Specifically, when inspecting `Union` args to determine whether or not a value is intended to be an input class or user-defined dict. To address the issue, this change improves how we get the args for `Union` types when running on Python 3.6 to behave the same way as later versions of Python (for our purposes). Existing tests fail on Python 3.6 before this change, and pass after.
We were only looking at the current resource class's type/name metadata for camelCase <=> snake_case property name translations which prevented it from working correctly when using a subclass of a resource. This change addresses this by looking at metadata of the current class and any base classes.
Additionally, to help resolve forward references when getting type hints, we'd pass along the current resource class's globals, which doesn't work correctly when using a subclass of a resource. This change also addresses this, by using the globals of the current class and any base classes.
When calling l.GetEnvVars() to evaluate whether version checking should
be skipped because of variables passed into the workspace, they have not
yet been set. This commit modifies the logic to look at the environment
variables in the options instead of in the workspace.
Since the original functionality has not yet been released, I do not
believe it warrants a CHANGELOG entry.
Resolve providers references and include the resulting refs in the
providers field of RegisterResourceRequest that was added in
d297db3.
Co-authored-by: Justin Van Patten <jvp@justinvp.com>
See #6200 for a complete description of the issue. In short, we generate
inconsistent names for object types depending on whether or not they are
transitively reachable from resources or functions, which risks
unintentional breaking changes due to schema updates.
1. Name "input" types differently: `TArgs` for a type that is used in
resource inputs, having `Input<T>` properties, and `T` for a type
that is used in invoke inputs. The same schema type can produce both.
2. Always keep the name `T` for output types, avoid appending `Result` to
the name.
3. As needed, introduce a flag in the existing providers' schemas to avoid
breaking changes. Consider removing it on a major version bump.
Fixes#6200.
This change addresses Python dictionary key translation issues. When the
type of `props` passed to the resource is decorated with `@input_type`,
the type's and resource's property name metadata will be used for dict
key translations instead of the resource's `translate_input_property`
and `translate_output_property` methods.
The generated provider SDKs will be updated to opt-in to this new
behavior:
- FIX: Keys in user-defined dicts will no longer be unintentionally
translated/modified.
- BREAKING: Dictionary keys in nested output classes are now
consistently snake_case. If accessing camelCase keys from such output
classes, move to accessing the values via the snake_case property
getters (or snake_case keys). Generated SDKs will log a warning
when accessing camelCase keys.
When serializing inputs:
- If a value is a dict and the associated type is an input type, the
dict's keys will be translated based on the input type's property
name metadata.
- If a value is a dict and the associated type is a dict (or Mapping),
the dict's keys will _not_ be translated.
When resolving outputs:
- If a value is a dict and the associated type is an output type, the
dict's keys will be translated based on the output type's property
name metadata.
- If a value is a dict and the associated type is a dict (or Mapping),
the dict's keys will _not_ be translated.
* Initial impl of import/export
* Add test
* Address PR comments
* Do not Select stack but pass it as --stack
* Add CHANGELOG entry
* Update sdk/dotnet/Pulumi.Automation/LocalWorkspace.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Update sdk/dotnet/Pulumi.Automation/LocalWorkspace.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Add helper methods to match node SDK
* Match project settings
* Update CHANGELOG_PENDING.md
Co-authored-by: Komal <komal@pulumi.com>
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
Co-authored-by: Komal <komal@pulumi.com>
* changes necessary for concurrent thread safe global deployment state
* update changelog
* backtrack resource package changes, enable test parallelization
* cleanup comment
* add copyright to new file
* resolve paralellization differences after merging automation api preview
* no longer need to null deployment instance
* Update CHANGELOG.md
* whoops - switch to CHANGELOG_PENDING
* Update CHANGELOG_PENDING.md
move note down to improvements
* attempt to exclude dynami assemblies
Co-authored-by: Anton Tayanovskyy <anton.tayanovskyy@gmail.com>
#6636 inadvertently changed the minimum Go version
requirement to 1.16 since ReadFile was moved in that
version. Switch back to ioutil.ReadFile to avoid forcing
an upgrade at this time.
This change fixes the provider implementation of `Construct` for multi-lang components written in Node.js to wait for any in-flight RPCs to finish before returning the results, s.t. all registered child resources are created.
In additional, invocations of `construct` are now serialized so that each call runs one after another, avoiding concurrent runs, since `construct` modifies global state. We'll follow-up with a more general concurrency fix to allow nested `construct` calls within the same provider.
* Add failing test
* Guard against overrding project settings accidentally
* Throw exception in case of conflct
* Update sdk/dotnet/Pulumi.Automation/DictionaryContentsComparer.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Update sdk/dotnet/Pulumi.Automation/ProjectRuntime.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Update sdk/dotnet/Pulumi.Automation/ProjectTemplateConfigValue.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Update sdk/dotnet/Pulumi.Automation/ProjectTemplate.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Update sdk/dotnet/Pulumi.Automation/ProjectRuntimeOptions.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Update sdk/dotnet/Pulumi.Automation/ProjectBackend.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Update sdk/dotnet/Pulumi.Automation/ProjectSettings.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Reduce nesting
* Make the new exception public
* Introduce a CHANGELOG entry since we add to pub API
* Stricter check before throwing
* Address PR feedback, round 1
* Use Reference.Equals check
* Move DictionaryContentsComparer out of top-level
Co-authored-by: Komal Ali <komal@pulumi.com>
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* [automation/dotnet] Start implementing structured event log support
* Add Events and their models
* Introduce new IStringToEnumConverter
This is intended to be used for converting a non-standard enum names
into corresponding enum values. The converter is utilized via
StringToEnumJsonConverter with OperationType and DiffKind.
* Add event-log support to up, refresh, destroy
* Update event-log tests
* Use lock instead of semaphore in EventLogWatcher
* Add ChangeSummary to PreviewResult
* Minor update
* Add doc comments to Events
* Use PropertyDiffModel in StepEventMetadataModel
* Set Events to public
* Set properties to public
* Fix PreludeEvent.Config doc comment
* Remove TODOs
* Update Pulumi.Automation.xml
* Set PULUMI_DEBUG_COMMANDs in Workspace.
* Split Events and their models to separate files
* Rename abbreviated Events
* Remove CancelEventModel
* Rename HandlesEvents test project and stack
* Update CHANGELOG_PENDING.md
* Rename abbreviated EngineEvent properties
* Implement custom exceptions
* Move event-log setup to LocalPulumiCmd
* Update comments
* Implement polling EventLogWatcher
This replaces the previous watcher which utilized FileSystemWatcher.
* Guard against onEvent throwing
* Dispose CancellationTokenSource
* WIP: try more direct error handling in EventLogWatcher
* A bit more tests
* Reformat
* Update sdk/dotnet/Pulumi.Automation.Tests/EventLogWatcherTests.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Update sdk/dotnet/Pulumi.Automation/Events/EventLogWatcher.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Update sdk/dotnet/Pulumi.Automation/Events/EventLogWatcher.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Update sdk/dotnet/Pulumi.Automation.Tests/EventLogWatcherTests.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Update sdk/dotnet/Pulumi.Automation/Events/EventLogWatcher.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Update sdk/dotnet/Pulumi.Automation/Events/EventLogWatcher.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Make race condition workaround more explicit
* Untabify
* Remove unnecessary using and change String to string
* Update doc comments on Events
Co-authored-by: Anton Tayanovskyy <anton@pulumi.com>
Co-authored-by: Anton Tayanovskyy <anton.tayanovskyy@gmail.com>
The current logic lets unhandled errors in the RPC invocation
unahandled in the async loop, which crashes the process due to
the way we await completion of RPCs before exiting the process
in Python. Instead of doing that, we can just marshal them back
to the synchronous awaiter as part of the calling convention,
and have that awaiter (which is called by the invoke methods)
re-raise the exception. This should fixpulumi/pulumi#3611.
Some YAML parsers don't correctly handle Byte-order marks,
so automatically strip it off during load.
Related to #423
Co-authored-by: Justin Van Patten <jvp@justinvp.com>
This commit adds a new optional parameter to the `newResource` function
of the `Mocks` interface for TypeScript. This can be useful when writing
tests which assert differing behavior between Custom and Component
resources.
Although the new parameter will always be set, the paramteter is marked
as optional in order to maintain backwards compatibility with existing
implementations of `Mocks`.
The tests are updated to verify that `custom` is set appropriately.
Co-authored-by: Luke Hoban <luke@pulumi.com>
* Avoid double-quailfying venv folder path
* Replace `path` with `filepath`
* Add a Python integration test to cover venv auto-creation
* Merged
* Fix spelling
Co-authored-by: Justin Van Patten <jvp@justinvp.com>
* Make AbsPath and RelPath test variants
* Fix issue on Windows backslash paths
* Debug windows test failure: more logging and aggressive YAML escaping
* Use filepath.IsAbs instead of path.IsAbs
Co-authored-by: Justin Van Patten <jvp@justinvp.com>
The Project and Stack save routines were erroneously
dumping the Python objects rather than the __dict__
property, which resulted in some extra annotations
in the resulting YAML files. Some parsers don't handle
these annotations correctly, and consider the resulting
YAML file to be invalid.
These methods were getting promoted onto every struct that implemented
the Output interface, and are not necessary.
On a real world program, this saves 4% in binary size overall, and
15% of remaining binary size if `Apply<TypeName>` functions are
removed (https://github.com/pulumi/pulumi/issues/6592).
* [sdk/go] Cache loaded configuration files
Previously, the CLI did not cache configuration files, which
required a read from disk + unmarshalling + validation each
time a consumer needed to read one of these configurations.
This change introduces global caches for each type of Pulumi
configuration file (Project, ProjectStack, PolicyPackProject, and
PluginProject). The configuration is cached after the first request
and the cached value will be used for any subsequent operations.
Important note: The global configurations are not concurrency safe,
but this same problem exists using the previous method of
reading/writing config files on disk. Synchronization
will be added in a follow up change to allow for concurrency safe config
operations.
Fixes:#6565
As part of #6460, the logic for determing the version of the build was
moved to be a dependency on pulumictl.
Unfortunately, the homebrew installs use the "make dist" command to
build + install Pulumi to the user maching and as that would have a
dependency on pulumictl and it not existing on the user machine, it
would pass an empty version to the ldflag
This then manifested to the user as:
```
▶ pulumi version
warning: A new version of Pulumi is available. To upgrade from version '0.0.0' to '2.22.0', run
$ brew upgrade pulumi
or visit https://pulumi.com/docs/reference/install/ for manual instructions and release notes.
```
We are able to mitigate this behaviour by bringing back the get-version
script and using that script as part of the make brew installation
We can see that the versions are the same between the 2 different
installation techniques
```
make dist <------- uses pulumict
DIST:
go install -ldflags "-X github.com/pulumi/pulumi/sdk/v2/go/common/version.Version=2.24.0-alpha.1616029310+787eb70a" github.com/pulumi/pulumi/sdk/v2/dotnet/cmd/pulumi-language-dotnet
DIST:
BUILD:
```
```
make brew <----- uses the legacy script
▶ make brew
BREW:
go install -ldflags "-X github.com/pulumi/pulumi/sdk/v2/go/common/version.Version=v2.24.0-alpha.1616029310+g787eb70a2" github.com/pulumi/pulumi/sdk/v2/dotnet/cmd/pulumi-language-dotnet
BREW:
```
A full post mortem will be carried out to ensure we mitigate these
types of errors going forward and that we are able to better test
these types of situations
dotnet, nodejs and python automation APIs did not specify exec-kind for
refresh or destroy operations. This is now added following the same
logic from the go automation API.
When using the filestate backend (local files and cloud buckets) there is no protection to prevent two processes from managing the same stack simultaneously.
This PR creates a locks directory in the management directory that stores lock files for a stack. Each backend implementation gets its own UUID that is joined with the stack name. The feature is currently available behind the `PULUMI_SELF_MANAGED_STATE_LOCKING=1` environment variable flag.
