#6636 inadvertently changed the minimum Go version
requirement to 1.16 since ReadFile was moved in that
version. Switch back to ioutil.ReadFile to avoid forcing
an upgrade at this time.
This change fixes the provider implementation of `Construct` for multi-lang components written in Node.js to wait for any in-flight RPCs to finish before returning the results, s.t. all registered child resources are created.
In additional, invocations of `construct` are now serialized so that each call runs one after another, avoiding concurrent runs, since `construct` modifies global state. We'll follow-up with a more general concurrency fix to allow nested `construct` calls within the same provider.
* Add failing test
* Guard against overrding project settings accidentally
* Throw exception in case of conflct
* Update sdk/dotnet/Pulumi.Automation/DictionaryContentsComparer.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Update sdk/dotnet/Pulumi.Automation/ProjectRuntime.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Update sdk/dotnet/Pulumi.Automation/ProjectTemplateConfigValue.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Update sdk/dotnet/Pulumi.Automation/ProjectTemplate.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Update sdk/dotnet/Pulumi.Automation/ProjectRuntimeOptions.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Update sdk/dotnet/Pulumi.Automation/ProjectBackend.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Update sdk/dotnet/Pulumi.Automation/ProjectSettings.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Reduce nesting
* Make the new exception public
* Introduce a CHANGELOG entry since we add to pub API
* Stricter check before throwing
* Address PR feedback, round 1
* Use Reference.Equals check
* Move DictionaryContentsComparer out of top-level
Co-authored-by: Komal Ali <komal@pulumi.com>
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* [automation/dotnet] Start implementing structured event log support
* Add Events and their models
* Introduce new IStringToEnumConverter
This is intended to be used for converting a non-standard enum names
into corresponding enum values. The converter is utilized via
StringToEnumJsonConverter with OperationType and DiffKind.
* Add event-log support to up, refresh, destroy
* Update event-log tests
* Use lock instead of semaphore in EventLogWatcher
* Add ChangeSummary to PreviewResult
* Minor update
* Add doc comments to Events
* Use PropertyDiffModel in StepEventMetadataModel
* Set Events to public
* Set properties to public
* Fix PreludeEvent.Config doc comment
* Remove TODOs
* Update Pulumi.Automation.xml
* Set PULUMI_DEBUG_COMMANDs in Workspace.
* Split Events and their models to separate files
* Rename abbreviated Events
* Remove CancelEventModel
* Rename HandlesEvents test project and stack
* Update CHANGELOG_PENDING.md
* Rename abbreviated EngineEvent properties
* Implement custom exceptions
* Move event-log setup to LocalPulumiCmd
* Update comments
* Implement polling EventLogWatcher
This replaces the previous watcher which utilized FileSystemWatcher.
* Guard against onEvent throwing
* Dispose CancellationTokenSource
* WIP: try more direct error handling in EventLogWatcher
* A bit more tests
* Reformat
* Update sdk/dotnet/Pulumi.Automation.Tests/EventLogWatcherTests.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Update sdk/dotnet/Pulumi.Automation/Events/EventLogWatcher.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Update sdk/dotnet/Pulumi.Automation/Events/EventLogWatcher.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Update sdk/dotnet/Pulumi.Automation.Tests/EventLogWatcherTests.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Update sdk/dotnet/Pulumi.Automation/Events/EventLogWatcher.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Update sdk/dotnet/Pulumi.Automation/Events/EventLogWatcher.cs
Co-authored-by: Ville Penttinen <villem.penttinen@gmail.com>
* Make race condition workaround more explicit
* Untabify
* Remove unnecessary using and change String to string
* Update doc comments on Events
Co-authored-by: Anton Tayanovskyy <anton@pulumi.com>
Co-authored-by: Anton Tayanovskyy <anton.tayanovskyy@gmail.com>
The current logic lets unhandled errors in the RPC invocation
unahandled in the async loop, which crashes the process due to
the way we await completion of RPCs before exiting the process
in Python. Instead of doing that, we can just marshal them back
to the synchronous awaiter as part of the calling convention,
and have that awaiter (which is called by the invoke methods)
re-raise the exception. This should fixpulumi/pulumi#3611.
Some YAML parsers don't correctly handle Byte-order marks,
so automatically strip it off during load.
Related to #423
Co-authored-by: Justin Van Patten <jvp@justinvp.com>
This commit adds a new optional parameter to the `newResource` function
of the `Mocks` interface for TypeScript. This can be useful when writing
tests which assert differing behavior between Custom and Component
resources.
Although the new parameter will always be set, the paramteter is marked
as optional in order to maintain backwards compatibility with existing
implementations of `Mocks`.
The tests are updated to verify that `custom` is set appropriately.
Co-authored-by: Luke Hoban <luke@pulumi.com>
* Avoid double-quailfying venv folder path
* Replace `path` with `filepath`
* Add a Python integration test to cover venv auto-creation
* Merged
* Fix spelling
Co-authored-by: Justin Van Patten <jvp@justinvp.com>
* Make AbsPath and RelPath test variants
* Fix issue on Windows backslash paths
* Debug windows test failure: more logging and aggressive YAML escaping
* Use filepath.IsAbs instead of path.IsAbs
Co-authored-by: Justin Van Patten <jvp@justinvp.com>
The Project and Stack save routines were erroneously
dumping the Python objects rather than the __dict__
property, which resulted in some extra annotations
in the resulting YAML files. Some parsers don't handle
these annotations correctly, and consider the resulting
YAML file to be invalid.
These methods were getting promoted onto every struct that implemented
the Output interface, and are not necessary.
On a real world program, this saves 4% in binary size overall, and
15% of remaining binary size if `Apply<TypeName>` functions are
removed (https://github.com/pulumi/pulumi/issues/6592).
* [sdk/go] Cache loaded configuration files
Previously, the CLI did not cache configuration files, which
required a read from disk + unmarshalling + validation each
time a consumer needed to read one of these configurations.
This change introduces global caches for each type of Pulumi
configuration file (Project, ProjectStack, PolicyPackProject, and
PluginProject). The configuration is cached after the first request
and the cached value will be used for any subsequent operations.
Important note: The global configurations are not concurrency safe,
but this same problem exists using the previous method of
reading/writing config files on disk. Synchronization
will be added in a follow up change to allow for concurrency safe config
operations.
Fixes:#6565
As part of #6460, the logic for determing the version of the build was
moved to be a dependency on pulumictl.
Unfortunately, the homebrew installs use the "make dist" command to
build + install Pulumi to the user maching and as that would have a
dependency on pulumictl and it not existing on the user machine, it
would pass an empty version to the ldflag
This then manifested to the user as:
```
▶ pulumi version
warning: A new version of Pulumi is available. To upgrade from version '0.0.0' to '2.22.0', run
$ brew upgrade pulumi
or visit https://pulumi.com/docs/reference/install/ for manual instructions and release notes.
```
We are able to mitigate this behaviour by bringing back the get-version
script and using that script as part of the make brew installation
We can see that the versions are the same between the 2 different
installation techniques
```
make dist <------- uses pulumict
DIST:
go install -ldflags "-X github.com/pulumi/pulumi/sdk/v2/go/common/version.Version=2.24.0-alpha.1616029310+787eb70a" github.com/pulumi/pulumi/sdk/v2/dotnet/cmd/pulumi-language-dotnet
DIST:
BUILD:
```
```
make brew <----- uses the legacy script
▶ make brew
BREW:
go install -ldflags "-X github.com/pulumi/pulumi/sdk/v2/go/common/version.Version=v2.24.0-alpha.1616029310+g787eb70a2" github.com/pulumi/pulumi/sdk/v2/dotnet/cmd/pulumi-language-dotnet
BREW:
```
A full post mortem will be carried out to ensure we mitigate these
types of errors going forward and that we are able to better test
these types of situations
dotnet, nodejs and python automation APIs did not specify exec-kind for
refresh or destroy operations. This is now added following the same
logic from the go automation API.
When using the filestate backend (local files and cloud buckets) there is no protection to prevent two processes from managing the same stack simultaneously.
This PR creates a locks directory in the management directory that stores lock files for a stack. Each backend implementation gets its own UUID that is joined with the stack name. The feature is currently available behind the `PULUMI_SELF_MANAGED_STATE_LOCKING=1` environment variable flag.
This change avoids `RuntimeError: There is no current event loop in thread '<thread_name>'` errors when passing a resource as an input multiple times when using mocks.
The problem is that when using mocks, we deserialize the gRPC inputs before passing them to the user's mock methods. Deserializing inputs doesn't typically require an event loop, however, during deserialization of resource references, we end up creating some instances of `Future`, which does require an event loop to be present for the current thread. If this is done multiple times for a resource, it's possible that `deserialize_properties` will be called on an asyncio thread that doesn't yet have an event loop, resulting in the error being raised.
The error does not occur when only passing the resource reference once because typically the thread (e.g. `asyncio_0`) used in that case will have already had an event loop created for it due to the use of the internal `_syncawait` when _serializing_ the source resource's properties, which ensures an event loop is set for the thread.
The fix is to ensure an event loop is created for the thread in the mocks implementation before calling `deserialize_properties`.
It is possible for the same version of the same provider SDK to be loaded multiple times in Node.js. In this case, we might legitimately get mutliple registrations of the same resource. It should not matter which we use, so we can just skip re-registering. De-serialized resources will always be instances of classes from the first registered package.
Example layout this addresses. Registrations of resources in `package3` at the same verrsion.
`node_modules`
`@pulumi/pulumi`
`package1`
`node_modules`
`package3`
`package2`
`node_modules`
`package3`
Fixes#6258.
* Init Workspace interface for C# Automation API
* fleshing out workspace interface and beginning of local workspace implementation
* initial run pulumi cmd implementation
* resolve issue with pulumi cmd cleanup wrapper task after testing
* flesh out local workspace implementation, flesh out stack implementation, cleanup run pulumi cmd implementation and make it an instance so it is mockable/testable, separate serialization in prep for custom converters
* project settings json serialization implemented
* Initial commit of language server
* Add deployment from language server
* Cleanup
* finish json serialization
* project runtime yaml serialization completed. just need stack config value yaml serialization
* Remove typed argument
* Limit concurrency
* Rename file for consistency
* final commit of a semi-working project settings & stack settings serialization so that it is in the commit history
* modify workspace API so that settings accessors aren't fully exposed since we are defering a complete serialization implementation until a later date
* yaml converters wrap any outgoing exceptions so resolve that
* getting the beginning of inline program GRPC communication set up
* stack lifecycle operations implemented, and switched to newtonsoft for JSON serialization
* change back to system.text.json with a custom object converter
* local workspace tests written, working on getting them passing
* fix the encoding on the GO files used for testing
* all tests passing except inline program, pulumi engine not available with inline
* inline program engine is now running as expecting, but inline program is not recognizing local stack config
* All tests passing, but no concurrency capability because of the singleton DeploymentInstance.
* cleanup unnecessary usings
* minor cleanup / changes after a quick review. Make sure ConfigureAwait is used where needed. Remove newtonsoft dependency from testing. Update workspace API to use existing PluginKind enum. Modify LanguageRuntimeService so that its semaphore operates process-wide.
* support for parallel execution of inline program, test included
* Update LocalWorkspaceTests.cs
remove some redundancy from the inline program parallel execution text
* flesh out some comments and make asynclocal instance readonly
* Strip out instance locking since it is no longer necessary with AsyncLocal wrapping the Deployment.Instance. Modify CreateRunner method such that we are ensuring there isn't a chance of delayed synchronous execution polluting the value of Deployment.Instance across calls to Deployment.RunAsync
* resolve conflicts with changes made to Deployment.TestAsync entrypoints
* update changelog
* write a test that fails if the CreateRunnerAndRunAsync method on Deployment is not marked async and fix test project data file ref
* make resource package state share the lifetime of the deployment so that their isn't cross deployment issues with resource packages, add support and tests for external resource packages (resource packages that aren't referenced by the executing assembly)
* enable parallel test collection execution in test suite, add some additional tests for deployment instance protection and ensuring that our first class stack exceptions are thrown when expected
* minor inline project name arg change, and re-add xunit json to build output (whoops)
* strip out concurrency changes since they are now in #6139, split automation into separate assembly, split automation tests into separate assembly
* add copyright to the top of each new file
* resolve some PR remarks
* inline program exception is now properly propagated to the caller on UpAsync and PreviewAsync
* modify PulumiFn to allow TStack and other delegate overloads without needing multiple first class delegates.
* whoops missing a copyright
* resolve getting TStack into IRunner so that outputs are registered correctly and so that there isn't 2 instances of Pulumi.Stack instantiated.
* resolve issue with propagation of TStack exceptions and add a test
* add support for a TStack PulumiFn resolved via IServiceProvider
* update automation API description
* fix comment and remove unnecessary TODOs
* disable packaging of automation api assembly
* re-name automation api documentation file appropriately
* add --limit support to dotnet automation api for stack history per #6257
* re-name XStack as WorkspaceStack
* replace --limit usage with --page-size and --page in dotnet automation api per #6292
Co-authored-by: evanboyle <evan@pulumi.com>
Co-authored-by: Josh Studt <josh.studt@figmarketing.com>
Co-authored-by: Dan Friedman <dan@thefriedmans.org>
Co-authored-by: David Ferretti <David.Ferretti@figmarketing.com>
Co-authored-by: Mikhail Shilkov <github@mikhail.io>
