Unlike most languages with interpolated strings, Python's formatted
string literals do not allow the nesting of quotes. For example,
this expression is not legal Python:
f"Foo {"bar"} baz"
If an interpolation requires quotes, those quotes nust differ from the
quotes used by the enclosing literal. We can fix the previous example
by rewriting it with single quotes:
f"Foo {'bar'} baz"
However, this presents a problem if there are more than two levels of
nesting, as Python only has two kinds of quotes (four if the outermost
string uses """ or '''): in this case, the expression becomes
unspellable, and must be assigned to a local that is then used in place
of the original expression. So this:
f"Foo {bar[f'index {baz["qux"]}']} zed"
becomes this:
index = "qux"
f"Foo {bar[f'index {baz[index]}']}"
To put it bluntly, Python code generation reqiures register allocation,
but for quotes. These changes implement exactly that.
These changes also include a fix for traversals that access values that
are dictionaries rather than objects, and must use indexers rather than
attributes.
Mapping happens at package scope, not type scope. This is what causes
some of the surprising mixes of camel and snake casing in our packages.
In order to accommodate this, build the case mapping tables up front,
and refer to them when performing case mapping.
The tokens that make up the "key" portion of an index traversal
(e.g. `"foo"` in `a["foo"]`) are structured like those that make up a
block label: an open quote token, a string literal token, and a close
quote token. The token mapper did not account for that fact, and instead
recorded the key token as the open quote. These changes correct that
error, and adjust the code in `literalText` to allow for
properly-escaped and quoted strings where necessary.
If a single process is going to bind and generate multiple programs, it
is useful to be able to cache package schemas in order to avoid the
(large) overhead of deserializing schemas multiple times.
Use strings.Builder instead of raw slices. We could go further here and
collapse to a single, ~right-sized bulder at the cost of complexity in
the `stateAcronym -> stateLowerOrNumber` transition.
Also, eliminate the pyName function, as it is redundant.
* Add a new DocLangHelper interface method to generate function names per language. Use the functionNames override map in Go to correctly generate a function name.
* Add kong to the title lookup map.
* Check if item was found in map.
* Use title case for the nodejs Function args type name. Use title case for the function name in the index page.
* Also fix the result name for a Function in nodejs to use title case.
* Use the module name to lookup the package.
* Use the title-case name for the Function name in the index page. Fix the language value passed to the lang chooser in function.tmpl.
* Use title case for nested type titles.
* Add title lookup for GitHub.
Some of the apply rewriter's assumptions were broken by the richer
expressions available in HCL2. These changes fix those broken
assumptions, in particular the assumption that only scope traversal
expressions are sources of eventual values.
In particular, ensure that they are keyword-safe. This affects
`aws:lambda:*` in particular: prior to these changes, we were generating
code into `pulumi_aws.lamdba`, which is not referencable due to its use
of the `lamdba` keyword. With these changes, we generate code into
`pulumi_aws.lambda_`.
There is also a small fix for multi-line non-formatted strings: these
strings do not need escaped braces.
* Fix output property names in Python codegen.
Some property names are mapped from their `camelCase` Pulumi name to a
`snake_case` Python name. This mapping is irregular, and only occurs for
resources properties and function calls.
Note that there's still more work to do here: this only fixes names on
the output side; the input side is still broken for nested resource
proprerties and function calls.
The underlying design--annotated types in `hcl2/model`--may need some
additional work in the future, but I _believe_ it's good enough for now.
* Fix Python input property names.
- Fix input property names and forms for invokes. Previously we
generated a dict; now we generate properly-named args.
- Fix nested property names for resources.
* PR feedback
- Fix input property names and forms for invokes. Previously we
generated a dict; now we generate properly-named args.
- Fix nested property names for resources.
* [codegen/go] Fix accessors on struct ptr outputs
The accesor methods on nestred struct Ptr outputs were previously not accepting pointer typed inputs as they should, and would thus always panic if used.
The "simple" fix would be to just accept the pointer type and blindly dereference it. But this doesn't seem like the right experience - it would make these accessors very unsafe to use in practice.
Instead, this PR implements the accessors on pointer-typed outputs as nil-coaslescing, always lifting the output type into a pointer type and flowing a nil value into the result type. This ensures the accessor will not nil-deref, and that user code can handle the `nil` value itself (or use `.Apply` directly to implement more specialized behaviour).
Before:
```go
// Name of your S3 bucket.
func (o BuildStorageLocationPtrOutput) Bucket() pulumi.StringOutput {
return o.ApplyT(func(v BuildStorageLocation) string { return v.Bucket }).(pulumi.StringOutput)
}
```
After:
```go
// Name of your S3 bucket.
func (o BuildStorageLocationPtrOutput) Bucket() pulumi.StringPtrOutput {
return o.ApplyT(func(v *BuildStorageLocation) *string {
if v == nil {
return nil
}
return &v.Bucket
}).(pulumi.StringPtrOutput)
}
```
However, due to the decision to have this more usable behaviour, this is a breaking change, as some/many accessors now return a pointer type when they previously did not.
Fixespulumi/pulumi-azure#530.
* Mark nested property types as requiring ptr types
* Add CHANGELOG
* More fixes
Some property names are mapped from their `camelCase` Pulumi name to a
`snake_case` Python name. This mapping is irregular, and only occurs for
resources properties and function calls.
Note that there's still more work to do here: this only fixes names on
the output side; the input side is still broken for nested resource
proprerties and function calls.
The underlying design--annotated types in `hcl2/model`--may need some
additional work in the future, but I _believe_ it's good enough for now.
* Optimize titles and descriptions
* Move regexp to file level variable
* wip
* Add title tags and description to module and pkg index pages
* Remove block_external_search_index from index.tmpl
* Improve resource descriptions
* Add more providers to look up map
* Improve module level descriptions
Clean up
cleanup
* Update the function.tmpl with the auto-generated disclaimer. Pass title tag and meta description for the function template as well.
* Fix whitespace in the package_details template..
* Account for empty module names for package-level resources. Move logic for generating title tag and description into separate methods for Resources and Functions.
* Add a test for the generated resource and function title tag.
Co-authored-by: Praneet Loke <1466314+praneetloke@users.noreply.github.com>
* Add ids to headers so they can be linked to
* Make header ids more general
Rather than including the identifier in the name of the id, use a simpler more generic id.
* H3=>H2 for Modules/Resources/Functions/Package Details headings
This way, on individual resource/function pages, it doesn't make it look like "Package Details" is a sub heading under "Supporting Types" in the right-hand "ON THIS PAGE".
* Add auto-generated disclaimer for the resource.tmpl as well.
* Allow specifying the languages to show in the lang chooser. Hide the go language for k8s overlay resources.
* Add another exclusion for generating the C# constructor param for k8s overlay resources.
* Replace hard-coded exclusions for skipping input properties with checking for ConstValue attribute of input properties.
- Define `null` in Pulumi HCL2
- Bind Pulumi HCL2 in topological order s.t. variable types can be
properly computed
- Fix resources that range over bools and numbers
- Add element, length, lookup, readFile, and split functions
- Do not rewrite function signatures with input types during binding
- Fix splat expression binding for non-lists
- Add support for evaluating expressions
- Add support for operator precedence to code generators
- Add support for constants to the HCL2 IR
- Add support for generating ranged resources in Python
- Add support for generating conditional resource in Node and Python
- Fix various naming issues in Python
* Add a method to the DocLanguageHelper interface to get the API doc link for Pulumi types.
* Add a test for Go API doc link methods.
* Use GetDocLinkForPulumiType in gen_function as well.
* Make `async:true` the default for `invoke` calls (#3750)
* Switch away from native grpc impl. (#3728)
* Remove usage of the 'deasync' library from @pulumi/pulumi. (#3752)
* Only retry as long as we get unavailable back. Anything else continues. (#3769)
* Handle all errors for now. (#3781)
* Do not assume --yes was present when using pulumi in non-interactive mode (#3793)
* Upgrade all paths for sdk and pkg to v2
* Backport C# invoke classes and other recent gen changes (#4288)
Adjust C# generation
* Replace IDeployment with a sealed class (#4318)
Replace IDeployment with a sealed class
* .NET: default to args subtype rather than Args.Empty (#4320)
* Adding system namespace for Dotnet code gen
This is required for using Obsolute attributes for deprecations
```
Iam/InstanceProfile.cs(142,10): error CS0246: The type or namespace name 'ObsoleteAttribute' could not be found (are you missing a using directive or an assembly reference?) [/Users/stack72/code/go/src/github.com/pulumi/pulumi-aws/sdk/dotnet/Pulumi.Aws.csproj]
Iam/InstanceProfile.cs(142,10): error CS0246: The type or namespace name 'Obsolete' could not be found (are you missing a using directive or an assembly reference?) [/Users/stack72/code/go/src/github.com/pulumi/pulumi-aws/sdk/dotnet/Pulumi.Aws.csproj]
```
* Fix the nullability of config type properties in C# codegen (#4379)
- Parentheses were not handled properly
- Literals inside of template control sequences were not handled
properly
These changes also improve test coverage for the printers.
* Fix k8s input type links for constructor args.
* Skip using property case maps for k8s Python.
* Fix the namespace of C# constructor args type and resource type.
This is required for using Obsolute attributes for deprecations
```
Iam/InstanceProfile.cs(142,10): error CS0246: The type or namespace name 'ObsoleteAttribute' could not be found (are you missing a using directive or an assembly reference?) [/Users/stack72/code/go/src/github.com/pulumi/pulumi-aws/sdk/dotnet/Pulumi.Aws.csproj]
Iam/InstanceProfile.cs(142,10): error CS0246: The type or namespace name 'Obsolete' could not be found (are you missing a using directive or an assembly reference?) [/Users/stack72/code/go/src/github.com/pulumi/pulumi-aws/sdk/dotnet/Pulumi.Aws.csproj]
```
* Use the correct module names for languages.
* Add Kubernetes to the title lookup map.
* Add output properties to the map only if filtered output properties is non-zero length.
* Add a check for the Python code gen to prevent infinte recursion. Remove the custom logic for k8s for generating Python property case maps.
* Fix bug with C# links for k8s constructor resources.
* Apply a style on the deprecation message.
* Use the display name when rendering a property type that does not have a link.
* Fix the python type string generator for docs to inspect union types.
* Update cleanTypeString to account for nodejs package names.
* Add missing lookup names for packages.
Pulumi HCL2 IR:
- Add support for invokes
- Add support for resource options, incl. ranged resources
- Allow the apply rewriter to ignore promise-typed values
- Add tests for the binder
- Add support functions for TF: entries and range
NodeJS codegen:
- Simplify for expression codegen
- Add support for invoke codegen
- Add support for entries and range functions
- Add tests
Python codegen:
- Implement codegen for most expression types
- Add support for invoke codegen
- Add tests
* Add new templates for generating index files. Remove the now irrelevant code.
* Add comment for generating index files by deduping module names. Use the path format of a module name as the link.
* Add a trailing slash for module links in the index page.
* Move the categories rendered in the index file into their own template file. Handle kubernetes provider specific module name conversion while generating parent modules.
* Fix issue with the k8s provider resource being generated as a module rather than a resource on the package-level index page.
* Show the deprecation message of a resource at the top of the document.
* Move k8s specific things to its own file.
* Add title attribute to the list items. Show the last part of a module name, in case it contains path separators
* Lookup the package name from the Go language info object for k8s then use that to lookup the C# namespace.
* Move the logic for cleaning a property type string for display names to a separate function.
* Export the title function from go and dotnet code generators for use in the resource doc generator. Use the title function from the respective lang code gens to match the correct title case used there. Fix issue with the Kubernetes namespace not being stripped for C# property type strings.
* Skip including apiVersion and kind as input properties for Kubernetes until pulumi-kubernetes#1062 is merged.
* Fix an issue with stripping the module name from type doc links in nodejs.
* Remove unused code that visited object types. Update the nested types code to account for types that may reference themselves. Added glog logging statements. Save the package-level language info objects, so we can use them later.
* Check if a type token has already been added as an input or an output type when collecting nested types.
* Rename appearsIn to typeUsage.
* Scan provider resource also using the special-case function if package is k8s.
* Simplify the display names for Pulumi types by removing the Pulumi prefix.
- config variables are now specified as `config <name> <type>`
- output variables are now specified as `output <name> <type>`
- apply arguments are assigned better names by the apply rewriters
These changes add preliminary (read: incomplete) support for
representing Pulumi programs using HCL2. Language-specific code
generators can use this representation as a basis for understanding the
semantics of a Pulumi program.
Some properties have constant values, such as the
apiVersion and kind for the Kubernetes provider. Add
a Const field to the Property schema to support this.
* Decode the C# language info from the schema package and set it in the dotnet lang helper for docs. Pass the qualifier for C# property type strings based on whether it is an input or an output property.
* Don't pass a module name for Pulumi core types used in TS constructor and Function params.
* Add tests for generating doc links for nodejs types.
* Add test for confirming input doc link for C# type.
* Fix the C# type name for InvokeOptions.
* Add tests for the examples processing.
* Update the schema-based code generators to strip out non-relevant code examples wrapped in short-codes.
* Extract language relevant examples for Functions too.
* Simpler way of extracting the surrounding text from the description.
* Add some more new-line characters to clean-up the examples sections.
* Update the C# invoke signature in the Functions template used by resource docs generator.
* Add a new function to the DocLanguageHelper interface for resource docs generator to generate language-specific property names.
* Add a new DisplayName property to the propertyType struct for simpler display names without module names.
* Update templates to use the DisplayName property of propertyType. Strip the namespace/module name from type names for use as display names.
* Generate Go package maps at the beginning, so that the resource docs can use the appropriate package context for generating a property type string name.
* Temporarily block the resource docs from search indexing.
* Use PascalCase for nested type names in Python.
* Fix bug with Python property name casing for nested types by checking the property case maps.
This package contains a driver for generating source code from HCL2
expressions. The driver is based on the fmt package's support for custom
formatters.
These changes implement `GetRequiredPlugins` for Go using a registry
mechanism and an alternate entry point for `pulumi.Run`. Packages that
require plugins are expected to register themselves with the Pulumi SDK.
When `pulumi.Run` is used and the `PULUMI_PLUGINS` envvar is truthy, the
program will dump a JSON-encoded description of its required plugins to
stdout. The language host then uses this description to respond to
These changes add a package for type checking and modeling HCL2
configurations. It is made up of three primary components:
1. A static type system
2. A semantic representation of HCL2 expressions and a binder from HCL2
native syntax to this representation
3. A semantic representation of HCL2 structural elements and binders
from HCL2 native syntax to this representation.
The type system is described in the "Extended Types" section of the
specification. The semantic representations of expressions and
structural elements are documented in their implementations.
* Generate the constructor params for Python along with other languages.
* Remove redundant py_function_param nested template. Declare a new type for defining property characteristics rather than using inlining formal params. Generate the Lookup functions for all languages similar to the constructor params with linking enabled.
* Fix bug with generating the input arg type name for Functions in Go.
* Add prefix for args param of a Go-based Resource Function.
* Input args for Go-based Functions use Lookup*Args and not Get*Args.
* Turns out that args for Go-based Functions use a different prefix based on whether the function is a package-level or module-level Function.
* Update the Python list and dictionary type names for the resource doc generator.
* Add a separate function for Python doc helper return a type string representing dictionaries that are simple maps and don't have a known nested element type.
* Also remove the Examples property from the resourceDocArgs type since the resource Comment will contain the multi-lang examples on their surrounded by a shortcode.
* Added a new template for Functions. Implement the genFunction method for generating the docs for Functions.
* Rename type resourceArgs to resourceDocArgs. Minor updates the resource template.
* Generate nested types for Functions.
* Unexport types that don't need to be exported. Create the doc language helper objects in an init function and reuse them rather than recreating them every time. Update genNestedTypes to work with schema functions or resources.
* Fixed bug in nested type generation for Functions. Fixed bug in generating input and output doc links for nested types.
Fixes: https://github.com/pulumi/pulumi-terraform-bridge/issues/119
This allows us to specify an overlays block e.g.
```
Overlay: &tfbridge.OverlayInfo{
DestFiles: []string{
"pulumi_docker/docker.py",
"pulumi_docker/image.py",
},
},
```
The overlays files are treated differently to normal module files
as they are not generated. This structure means that we will emit
the correct entries in the __init__.py file
Without this structure (ie. pulumi_pkgname), the generator actually
copies the file (i.e. docker.py) to the root of the Python SDK. This
is because the structure of the Python SDK has a sub-folder than that
of the NodeJS SDK
I tested this using PR https://github.com/pulumi/pulumi-docker/pull/141
and this now works as expected and we can take advantage of the new
Python overlays for Docker
* Update properties.tmpl to render property comment as-is. WIP splitting out properties to lang-specific tables.
* Generate the constructor dynamically from the resource per language.
* Add doc functions in each language generator package for getting doc links for types..and later other functions too.
* Render the constructor params in the Go code and inject into the template.
* Generate nodejs types using the nodejs lang generator.
* Add a templates bundler. Added a new Make target for autogenerating a static bundle for the resource docs generator.
* Generate type links for all languages based on their schema type. Render the property type with a link if the underlying elements have a supporting type. Fix word-breaks for Python type names.
* Various changes including the introduction of an interface type under the codegen package to help with generating some language-specific information for the resource docs generator.
* Add a function to explicitly generate links for input types of nested types. Fix the resource doc link generator for Go. Don't replace the module name from the nodejs language type.
* Fix bug with C# property type html encoding.
* Fix some template formatting. Pass the state inputs for Python to generate the lookup function for it.
* Do not generate the examples section if there are none.
* Generating the property types per language.
* Formatting. Rename function for readability.
* Add comments. Update README.
* Use relative URLs for doc links within the main site
These changes implement `GetRequiredPlugins` for Go using a registry
mechanism and an alternate entry point for `pulumi.Run`. Packages that
require plugins are expected to register themselves with the Pulumi SDK.
When `pulumi.Run` is used and the `PULUMI_PLUGINS` envvar is truthy, the
program will dump a JSON-encoded description of its required plugins to
stdout. The language host then uses this description to respond to
`GetRequiredPlugins`.
These changes add a helper package for parsing HCL2 syntax files. The
helpers are intended to provide direct access to syntax elements that
are abstracted away by the standard HCL2 parser. The bulk of the code
deals with mapping syntax nodes to their relevant tokens in order to
avoid losing information about the comments associated with syntax
nodes.
- Make go comment generation lint-clean for blank comment lines
- Fix the casing of `Provider.py` in Python to `provider.py`
- Fix a spacing issue in the NodeJS code generator
Replace the various `defaults` maps in the schema with per-property
`default` and `defaultInfo` fields. The former holds the static default
value; the latter holds the envvars and language-specific info.
Also, fix a minor bug in the Python codegen that caused diffs in
property docstrings.
The schema format is described in pkg/codegen/schema/schema.go. The code
generators are derived from the code generators contained in
https://github.com/pulumi/pulumi-terraform-bridge/pkg/tfgen, with the
exception of the Go code generator, which is net new.
Fixes: #2151
This will allow us to be able to share the code that generates our
language providers. Currently there is a copy of the python code
generation in pulumi-kubernetes and also in pulumi-terraform
We want to be able to share these
