* Go support for 5758 - resurrect stale PR
* Fix listStorageAccountKeys test
* Check err so linter is satisfied
* Use all the examples
* Accept codegen results
* Regenerate with PULUMI_IGNORE_AMBIENT_PLUGINS=1
* Compile and test generated code as part of the test suite
* Add a CHANGELOG entry
* Remove temp test marker
* Shorten output type name
* Simplify code
* Add issue link
* Accept more codegen changes
* Use the suggested format for linking an issue
These changes take a step towards simplifying and unifying the
generation of output types in the Go SDKs, especially for pointer,
array, and map outputs. This code was previously duplicated amongst the
various specialized output type generators, which led to inconsistencies
between the various implementaitons.
This is prep work for fixing #7595.
This commit adds two new fields to the Node package info struct to
permit setting the plugin name if it differs from the package name, and
the version if it differs from the package version. This was already
supported by the loader.
Coincident with the release of Pulumi 3.0, we updated the provider SDK codegen for Python to no longer use casing tables for translating Python snake_case names to Pulumi camelCase names (and vice versa). Instead, the mapping is encoded in decorators applied on class properties.
Some of the code that was used to generate and use the casing tables has persisted. This commits removes this code, as it's no longer necessary, and will improve the quality of our generated examples.
Collection types nested inside of Input<Union<...>> types need to abide
by the usual rules for collection types nested inside of input types.
These changes replace the use of the generic SimplifyInputUnion with a
.NET-specific simplifyInputUnion that does not remove Input types
inside of a union if those Input types wrap collection types. Retaining
these Input types allows the usual logic for handling
Input<Collection<...> types in typeString to kick in.
Fixes#7569.
- Only build casing tables once per package
- Right-size buffers in name generation
These changes lead to a significant speedup in example gen for
azure-native.
* Allow non-pulumi imports for Node.js
Currently the code generator is assuming that Node.js dependencies are
following a naming scheme that is prefixed with `pulumi/`. If this is
not the case the generated import statement is incorrect.
This commit adds a map `ProviderNameToModuleName` to the language
definition that allows you to map the name of the extracted provider of
a dependency to a module name that the generator now uses to create the
import statement.
* Prepend "pulumi" to import names in Node.js SDK
It is common when writing multi-language components to have a module
name which conflicts with a provider name. This can produce unusable
code, since you cannot simultaneously import a package as `aws` and have
a namespace `aws`, for example.
This commit makes this situation much less likely, by renaming the
imported identifier for providers to `pulumiX` where it would
previously have been `x`.
This has an unfortunate side effect of making the examples in the
documentation slightly uglier, since import statements for third-party
packages are now of the form `import * as pulumiAws from "@pulumi/aws"`.
I don't see a way to discern whether code generation is for SDKs vs
examples however, and short of plumbing that through, I don't see a way
around this, so test expectations are updated accordingly.
Co-authored-by: Ben Schiborr <bschiborr@apple.com>
- Lazily produce conversion failure diagnostics. This lowers the
allocation volume and cuts down on execution time by avoiding the
conversion of source and dest types to strings.
- Add a fast path for union conversions that checks if the source type
is identical to any of the union's element types. Type equality
checks are generally much faster than type conversion checks.
These changes lead to a significant speedup in codegen time in
azure-native.
- Track which languages have been imported for a package. If a language
has already been imported, do not re-run its importers.
- Track which package contexts have been loaded in the Go code
generator, and do not reload a context that already exists.
These changes shave a profound amount of time off of codegen in
azure-native, speeding things up by a factor of 5.
When converting a `schema.InputType` to a `model.Type`, calculate the
resolved form of the type in the schema type system rather than the
model type system. The results are semantically identical, but the
number of type objects that are allocated is much smaller b/c
`model.NewOutputType` no longer allocates.
This deserves a little more explanation.
In order to prevent nested outputs and/or promises, `model.NewOutputType`
calculates the resolved form of its argument prior to allocating a new
`OutputType` value. Calculating the resolved form of the argument is a
no-op if the argument is already fully resolved. Therefore, passing in a
fully-resolved schema type prevents `model.NewOutputType` from
calulating the resolved form, and `model.NewOutputType` will only
allocate the `OutputType` itself instead of the `OutputType` and the
resolved form of any eventuals present in its argument.
This has a _very important_ knock-on benefit: the schema -> model type
translator ensures that given a `schema.Type` instance `T` it will
always return the same `model.Type` instance `U`. This termendously
speeds up type equality checks for complex types, as they will now be
referentially identical.
This change alone gives a significant speedup in azure-native code
generation.
This commit modifies the generation of `setup.py` to use Python
variables as the source for the package version and plugin version
instead of placeholder strings. This has the effect of making the
packages installable via the `-e` flag directly from their source
directory rather than requiring a build step, which is useful while
developing a plugin and examples in tandem.
This commit modifies Go program generation to prevent producing array
and slice object elements as pointers in args structures, which fails at
runtime and does not make sense in any case. For example, in the case of
a type defined like this in schema:
```json
"statements": {
"type": "array",
"items": {
"$ref": "/aws/v4.8.0/schema.json#/types/aws:iam/getPolicyDocumentStatement:getPolicyDocumentStatement"
}
},
```
The following (which fails at runtime) was produced before this change:
```go
Statements []*iam.GetPolicyDocumentStatement `pulumi:"statements"`
```
And the following is produced after after this change:
```go
Statements []iam.GetPolicyDocumentStatement `pulumi:"statements"`
```
Test expectations are updated accordingly.
This commit fixes code generation for intermediate module paths to
produce valid TypeScript identifiers.
Before this change, the following (non-compilable) import was produced
in `./jetstack/certmanager/acme`:
```
import * as jetstack/certmanager/acme/v1alpha2 from "./jetstack/certmanager/acme/v1alpha2";
```
After this change, the following import is produced:
```
import * as v1alpha2 from "./v1alpha2";
```
This example is repeated at each level of the module tree. Test
expectations are adjusted to reflect this change.
This commit modifies the Go code generator to use configured aliases for
the root package name of a Go module. This is useful when a version 2
package is present, as it prevents generation of identifiers such as
"v2" which were produced by the old logic.
These changes add support for unmarshaling and marshaling package
schemas using YAML instead of JSON. Language-specific data is
canonically JSON. Users of the `*Spec` types will need to update the
types of the the their `Language` values to use the new
`schema.RawMessage` type instead of `json.RawMessage`: the former has
support for YAML while the latter does not.
This commit adds a newline to the end of the package.json files
generated by Pulumi codegen, such that they can be installed in place
without modification.
The inputs and expected outputs for the tests are encoded using a
schema. Each property present in the schema forms a testcase; the
expected outputs for each language are stored in each property's
`Language` field with the language name "test". Expected outputs can be
regenerated using `PULUMI_ACCEPT`.
Rather than duplicating the list of tests and codegen driver across each
SDK, move its definition into `pkg/codegen/internal/test`. This has a
few notable benefits:
- All SDK code generators will be tested against each test. Though some
tests may exercise a particular code generator more than others, the
extra coverage will be generally beneficial.
- Adding a new test is simpler, as only a single file needs to be
changed.
- All SDKs now honor the `PULUMI_ACCEPT` environment variable for
updating baselines.
- Codegen tests now validate all generated files instead of only a
particular subset.
These changes support arbitrary combinations of input + plain types
within a schema. Handling plain types at the property level was not
sufficient to support such combinations. Reifying these types
required updating quite a bit of code. This is likely to have caused
some temporary complications, but should eventually lead to
substantial simplification in the SDK and program code generators.
With the new design, input and optional types are explicit in the schema
type system. Optionals will only appear at the outermost level of a type
(i.e. Input<Optional<>>, Array<Optional<>>, etc. will not occur). In
addition to explicit input types, each object type now has a "plain"
shape and an "input" shape. The former uses only plain types; the latter
uses input shapes wherever a plain type is not specified. Plain types
are indicated in the schema by setting the "plain" property of a type spec
to true.
The `bad-methods-2.json` test ensures there is an error when trying to use a function as a method twice.
The schema looks like:
```
"methods": {
"bar": "xyz:index:Foo/bar",
"baz": "xyz:index:Foo/bar"
}
```
And the expected error is:
> function xyz:index:Foo/bar for method baz is already a method
However, when the schema is unmarshalled into a map, the keys in the map are unordered (just as JSON keys in objects are unordered), so occaisonally we'd see an error mentioning method `bar` rather than `baz`:
> function xyz:index:Foo/bar for method bar is already a method
To address this, we'll fix the portion of the code that is generating the error to walk the map in a deterministic order, and we'll also return the list of methods in the same deterministic order as well.
More testing has found further issues with incorrect attribute shapes
generated by tf2pulumi. This knob will allow us to continue to generate
examples until those bugs are fixed.
Add an option to allow missing object properties. This will prevent us
from losing examples once resource typechecking is fixed (it is
currently unintentionally disabled because the resource inputs object
type has an unexpected shape).
Every time a resource is created (without an explicit provider version specified), `_utilities.get_version()` is called to determine the package version. This call is expensive and can measurably impact performance, especially for programs with many resources. This change caches the value so it is only determined once.
Python resource constructor overloads were recently added that accept a
`<Resource>Args` class for input properties, as an alternative to the
other constructor overload that accepts keyword arguments. The name of
the new args class is the name of the resource concatenated with an
`Args` suffix.
Some providers (e.g. Kubernetes, Azure Native, and Google Native) have
input types with the same name as resources in the same module, which
results in two different `<Resource>Args` classes in the same module.
When you try to use the new args class with the constructor, e.g.:
```python
pulumi_kubernetes.storage.v1.StorageClass(
resource_name='string',
args=pulumi_kubernetes.storage.v1.StorageClassArgs(...),
opts=pulumi.ResourceOptions(...),
)
```
You run into an error, because
`pulumi_kubernetes.storage.v1.StorageClassArgs` is actually referring to
the existing input type rather than the intended `StorageClassArgs`
class for the constructor arguments.
Having the duplicate classes hasn't broken existing usage of the input
type because we "export" all the input types for a module _after_ all
the resources and resource args classes are exported, so the input type
just ends up "overwriting" the duplicate resource args class.
Other languages don't have this problem because the input type is either
in it's own module/namespace (e.g. Node.js and .NET) or a different name
is used for the input type (Go). But with Python, the input types and
resources are all available in the same module.
To address this for Python, when there is an input type in the same
module with the same name as the resource, the args class for the
resource will be emitted as `<Resource>InitArgs` instead of
`<Resource>Args`.
This commit modifies the work in #7058 to permit properties which do not
pass the test of being strings directly, but which have an underlying
type of string.
When applied to `pulumi-aws`, this results in the following diff:
```
diff --git a/sdk/go/aws/provider.go b/sdk/go/aws/provider.go
index c32ad2367..8b4c9fd0a 100644
--- a/sdk/go/aws/provider.go
+++ b/sdk/go/aws/provider.go
@@ -21,6 +21,8 @@ type Provider struct {
AccessKey pulumi.StringPtrOutput `pulumi:"accessKey"`
// The profile for API operations. If not set, the default profile created with `aws configure` will be used.
Profile pulumi.StringPtrOutput `pulumi:"profile"`
+ // The region where AWS operations will take place. Examples are us-east-1, us-west-2, etc.
+ Region pulumi.StringPtrOutput `pulumi:"region"`
// The secret key for API operations. You can retrieve this from the 'Security & Credentials' section of the AWS console.
SecretKey pulumi.StringPtrOutput `pulumi:"secretKey"`
// The path to the shared credentials file. If not set this defaults to ~/.aws/credentials.
```
The primary purpose this is desirable is to expose Region from instances
of the AWS provider.
Fix the generated C# code for plain properties:
- Value types should not be initialized with `= null!`
- Arrays and maps should be `List<T>` and `Dictionary<string, TValue>`
In Go, resource types are modeled as pointers, but there were cases where the type was not being emitted as a pointer, leading to panics and marshaling errors in programs. Additionally, array and map values that are external references were being emitted as pointers, but only resources should be pointers (not types), regardless of whether the resource type is external or local.
Following pulumi/pulumi-terraform-bridge#347, properties are generated
for all provider config matching the inputs. Unfortunately this does not
work for complex values and non-string primitives generally (not only in
bridged providers) since values are JSON serialized.
While a proper solution to this is designed, it's sufficient for now to
stop generating non-string properties, which this commit does.
This commit adds a fallback for the README definition in the generated
setup.py files for Python SDKs, thus allowing editable installs of
packages which not yet been built.
Co-authored-by: Luke Hoban <luke@pulumi.com>
This commit adds a new language option for Python generation to specify
the package name instead of using `pulumi_x` where x is the name defined
in the schema.
A new test is added, and this has also been shown to produce no diff
when run against `pulumi-eks`.
* Remove leading and trailing whitespace in resource properties
* Make tests pass
* Add PULUMI_ACCEPT support to docs gen tests
* Handle a couple more places
Co-authored-by: Pat Gavlin <pat@pulumi.com>
If `rootPackageName` is set, we can look for the version in the
baseImportPath rather than at a location based on the package name -
which currently fails if every component is not named `pulumi-*`. To err
on the side of caution, this method is only used for packages where
`rootPackageName` is set, meaning existing SDKs retain their current
behavior.
The new behavior is confirmed via the test added in #6862.
* Import subpackages lazily
* Tighten up lazy_import impl
* Eagerly register resources, but lazily load their impl
* Add CHANGELOG entry
* Satisfy lint
* Restore mypy behavior so the change is not breaking
* Fix golden tests
When working in a monorepo environment, it can be desirable to generate
Go SDKs into a structure less like the upstream SDKs, and more like
this:
github.com/x/mymonorepo/sdk/go/package-name
Where `package-name` is also the root of a Go module. Since
`package-name` is not a valid package name in Go, it's also desirable to
be able to choose a replacement name and reduce the amount of nesting.
This commit adds a new Go option to the schema, `rootPackageName`, which
can be used to modify the generated root package name (e.g. to
`mypackage` instead of `package-name`, and remove the additional layer
of nesting.
Test coverage is added to ensure that the correct file structure and
package names are generated.
This commit adjusts the way that Go module versions are discovered from
packages when generating Go programs, to account for those on module
version 1. Previously, this function would panic when dereferencing a
nil instance of semver.Version.
These changes fix a regression introduced by #6686 that caused the SDK
code generators for .NET, Python, and Typescript to omit definitions for
plain object types. This regression occurred because #6686 drew a
clearer line between types used as resource arguments and types used
as function arguments, but conflated "resource arguments" with "inputty
types". This caused the code generators to generate inputty types for
any types used as resource arguments, even those that are used for
plainly-typed properties.
Fixes#6796.
Add line breaks and whitespace to avoid long horizontal scrolls for Python constructor/function arguments. Also, include the new ResourceArgs constructor overload.
See #6200 for a complete description of the issue. In short, we generate
inconsistent names for object types depending on whether or not they are
transitively reachable from resources or functions, which risks
unintentional breaking changes due to schema updates.
1. Name "input" types differently: `TArgs` for a type that is used in
resource inputs, having `Input<T>` properties, and `T` for a type
that is used in invoke inputs. The same schema type can produce both.
2. Always keep the name `T` for output types, avoid appending `Result` to
the name.
3. As needed, introduce a flag in the existing providers' schemas to avoid
breaking changes. Consider removing it on a major version bump.
Fixes#6200.
These have been deprecated for a very long time and it's a trivial change to remove them from the generated code. Let's clean this up for the 3.0-based providers.
