When computing the type name for a field of an object type, we must
ensure that we do not generate invalid recursive struct types. A struct
type T contains invalid recursion if the closure of its fields and its
struct-typed fields' fields includes a field of type T. A few examples:
Directly invalid:
type T struct { Invalid T }
Indirectly invalid:
type T struct { Invalid S }
type S struct { Invalid T }
In order to avoid generating invalid struct types, we replace all
references to types involved in a cyclical definition with *T. The
examples above therefore become:
(1) type T struct { Valid *T }
(2) type T struct { Valid *S }
type S struct { Valid *T }
We do this using a rewriter that turns all fields involved in reference
cycles into optional fields.
These changes also include an enhancement to the SDK codegen test
driver in the interest of making iterating and debugging more convenient: if the -sdk.no-checks flag is passed, the driver will not run post-generation checks.
* Enable output values by default
Enable output values by default in the resource monitor and change the polarity of the envvar from `PULUMI_ENABLE_OUTPUT_VALUES` to `PULUMI_DISABLE_OUTPUT_VALUES`.
* Marshal unknown as unknown string when `!KeepOutputValues`
Marshal all unknown output values as `resource.MakeComputed(resource.NewStringProperty(""))` when not keeping output values, which is consistent with what the SDKs do.
Otherwise, when `v.OutputValue().Element` is nil, `resource.MakeComputed(v.OutputValue().Element)` will be marshaled as a null value rather than as an unknown sentinel.
* Add MarshalOptions.DontSkipOutputs and use where needed
Before we expanded the meaning of `resource.Output`, `MarshalProperties` always skipped output values:
```go
if v.IsOutput() {
logging.V(9).Infof("Skipping output property for RPC[%s]: %v", opts.Label, key)
}
```
As part of expanding the meaning of `resource.Output`, I'd adjusted `MarshalProperties` to only skip output values when the value was unknown and when not keeping output values:
```go
if v.IsOutput() && !v.OutputValue().Known && !opts.KeepOutputValues {
logging.V(9).Infof("Skipping output property for RPC[%s]: %v", opts.Label, key)
}
```
However, this doesn't work the way we want when marshaling properties that include unknown output values to a provider that does not accept outputs. In that case, `opts.KeepOutputValues` will be `false` because we want the marshaler to fall back to returning non-output-values (e.g. unknown sentinel value for unknown output values), but instead of getting the intended fallback values, the unknown output values are skipped (not what we want).
I suspect we may be able to delete the output value skipping in `MarshalProperties` altogether (it's odd that it is skipping `resource.Output` but not `resource.Computed`), but to avoid any unintended side effects of doing that, instead, this commit introduces a new `MarshalOptions.DontSkipOutputs` option that can be set to `true` to opt-in to not skipping output values when marshaling. The check in `MarshalProperties` now looks like this:
```go
if !opts.DontSkipOutputs && v.IsOutput() && !v.OutputValue().Known {
logging.V(9).Infof("Skipping output property for RPC[%s]: %v", opts.Label, key)
}
```
`opts.DontSkipOutputs` is set to `true` when marshaling properties for calls to a provider's `Construct` and `Call`.
* [sdk/nodejs] Deserialize output values
This commit adds support for deserializing output values, which is needed in some cases when serialized inputs are returned as outputs in the SDK.
* [sdk/python] Deserialize output values
This commit adds support for deserializing output values, which is needed in some cases when serialized inputs are returned as outputs in the SDK.
v3.13.0 introduces support for serializing outputs in inputs as special output value objects in the Node.js and Python SDKs when serializing inputs for remote components and method calls. This functionality is currently disabled by default in the engine (setting the `PULUMI_ENABLE_OUTPUT_VALUES` envvar to a truthy value enables it).
However, unit testing remote components with mocks results in errors being raised in v3.13.0, related to the new output value support. This is due to the mock monitor implementation saying it supports all features (which now includes output values), so the SDK serializers are serializing outputs as output values, which the mock monitor can't handle correctly.
This change addresses the issue by updating the mock monitor implementation in the Node.js and Python SDKs to indicate the specific features that are supported, excluding support for output values. New tests with mocks fail before the change and pass after.
The Pulumi Package metaschema is a JSON schema definition that describes
the format of a Pulumi Package schema. The metaschema can be used to
validate certain basic properties of a Pulumi Package schema, including
(but not limited to):
- data types (e.g. is this property a string?)
- data formats (e.g. is this string property a valid regex?)
- object shapes (e.g. is this object missing required properties?)
The schema binder has been updated to use the metaschema as its first
validation pass.
In addition to its use in the binder, the metaschema has its own page in
the developer documentation. This page is generated using a small tool,
jsonschema2md.go.
* Add pascal name case
* Add test to prevent regressions
* Update CHANGELOG_PENDING.md
* Give node/tsc more memory
* Use camelcase instead of snake case
* Add clearer comments
* Emit schema.Package.Version when possible
* Update CHANGELOG_PENDING.md
* Correctly interpret python versions (I hope)
* Update PLUGIN_VERSION to the package version
* Modify tests to conform with master merge
* Validate Name, Version and Enviroment
For the full path:
Package.Name
Package.Version
Package.Property.Default
* Update tests
* Update CHANGELOG_PENDING.md
* Add more versions to tests
* Add another "Version" field
* Even more "version" tags
* One more "version" tag added
* Update test results from codegen
* Fix py codegen tests
* Fix doc test
* Remove `version` validation
* Unformat json files
* Fail only on errors
This change adds a `RegisterInputType` function (similar to the existing `RegisterOutputType`) that is used to register an input interface's type and its associated input type, and adds registrations for the built-in input types.
This will be used when copying inputs to an args struct for multi-lang components. When a field is typed as the input interface (e.g. `StringMapInput`) and doesn't need to be an `Output`, we can use this to lookup the non-Output type that implements the interface (e.g. `StringMap`) so it can be instantiated.
A subsequent change will update the Go SDK codegen to produce input type registrations for a provider's input types.
* Thread replaceOnChanges through the Go SDK
* Add replaceOnChanges to the .NET SDK
* Update CHANGELOG_PENDING.md
* Fix null error
* Update CHANGELOG_PENDING.md
Co-authored-by: Justin Van Patten <jvp@justinvp.com>
This change expands the definition of `resource.Output` in the Go SDK with additional information about the output, i.e. dependencies and secretness, and adds support in the core Go RPC code for (un)marshaling output values.
Output values are marshaled as special objects ala archives, assets, and resource refs and are unmarshaled as `resource.Output` values.
Subsequent PRs will add:
- A monitor feature for output values, which will initially be disabled by default but available to turn on via an envvar
- Support for (un)marshaling output values in each language SDKs
- A way for providers to indicate support for receiving output values
- E2E tests
- Turn the monitor feature on by default (w/ env var to disable) (Note: the current plan is to initially scope this to only be used when marshaling inputs to a multi-language component)
Currently whenever an issue occurs in `UnmarshalProperties` and
`MarshalProperties` the offending property is hidden and very difficult
to track down.
This commit changes the behavior. For `Assets` and `Archives` the error
message now includes the URI and for other properties it includes the
key of the `PropertyMap`.
* Introduce a test that showcases the invalid generated code
* Use shared printComment function
* Check for triple quote escaping
* Accept go
* Accept dotnet
* Accept nodejs
* Move codegen exampe into an existing schema
* Add CHANGELOG entry
* Add replaceOnChange to schema
* replaceOnChange at generate time for resources
* ReplaceOnChanges sees through optional types
* Correctly deal with map,array,object,resource type
This is responding to PR clarifications from @justinvp and @lblackstone.
* Update CHANGELOG_PENDING.md
* Detect recursively defined objects
* Display recursion warning
* Check which recursive structures fail
* Add internal logic tests for replaceOnChanges
* Add tests
This reverts commit ebb0e6aaed.
The changes in #7755 introduced a regression tracked in #7795. It is not yet clear how to retain the desired behaviour introduced in #7755 while avoiding this regression, so for now we will revert those changes, and re-open #7453 to track a deeper fix. That fix may require making changes to upstream `dill` to properly support these serialization requirements.
Fixes#7795.
- Change the schema package to report semantic errors as diagnostics
rather than Go errors
- Add a `pulumi schema check` command to the CLI for static checking of
package schemas
The semantic checker can be extended in the future to add support for
target-specific checks.
* Avoid repeatedly invoking `pip show` in Python lang host
The Python language host invokes `pip show` for each Pulumi package in a
project at startup. But `pip show` is quite slow in large projects: it
takes 2+ seconds per invocation in a project at @MaterializeInc.
`pip show` is invoked to compute the installed location of each plugin
package. But it turns out `pip list` takes a `-v` flag that can supply
this information in one shot, avoiding the need to ever invoke `pip
show`.
This patch shaves about 20s off our boot time for `pulumi up`.
(There's probably a separate bug in Pip that causes `pip show` to be so
slow, because `pip list` is an order of magnitude faster and does a lot
more work, but I didn't bother tracking that down.)
* Test and fix issue with parsing non-JSON trailer returned by pip
* Fix issues found by Go lint
* CHANGELOG entry
Co-authored-by: Nikhil Benesch <nikhil.benesch@gmail.com>
* add additional plugin install options
* update changelog
* slight re-name properties to be more idiomatic
* Apply suggestions from code review
Co-authored-by: Justin Van Patten <jvp@justinvp.com>
* Add support for old install plugin overload, and move unshipped api changes to shipped
* add breaking note to changelog
Co-authored-by: Justin Van Patten <jvp@justinvp.com>
* Go support for 5758 - resurrect stale PR
* Fix listStorageAccountKeys test
* Check err so linter is satisfied
* Use all the examples
* Accept codegen results
* Regenerate with PULUMI_IGNORE_AMBIENT_PLUGINS=1
* Compile and test generated code as part of the test suite
* Add a CHANGELOG entry
* Remove temp test marker
* Shorten output type name
* Simplify code
* Add issue link
* Accept more codegen changes
* Use the suggested format for linking an issue
Change `pulumi new` to use `go mod tidy` rather than `go mod download` when installing Go project dependencies, to ensure the project is fully prepared to be used by `pulumi up`, avoiding `missing go.sum entry for module` errors.
