2017-05-18 23:51:52 +02:00
|
|
|
// Licensed to Pulumi Corporation ("Pulumi") under one or more
|
|
|
|
// contributor license agreements. See the NOTICE file distributed with
|
|
|
|
// this work for additional information regarding copyright ownership.
|
|
|
|
// Pulumi licenses this file to You under the Apache License, Version 2.0
|
|
|
|
// (the "License"); you may not use this file except in compliance with
|
|
|
|
// the License. You may obtain a copy of the License at
|
|
|
|
//
|
|
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
//
|
|
|
|
// Unless required by applicable law or agreed to in writing, software
|
|
|
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
|
|
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
|
|
// See the License for the specific language governing permissions and
|
|
|
|
// limitations under the License.
|
2017-04-24 18:50:48 +02:00
|
|
|
|
|
|
|
As we write more resource provider packages, we have begun to encounter shortcomings. Properties must currently be
|
2017-05-18 20:38:28 +02:00
|
|
|
defined both in the LumiLang type definitions and in the Go provider implementation, requiring edits to many files just
|
2017-04-24 18:50:48 +02:00
|
|
|
to add, change, or remove a property (or fix a bug). Additionally, there is considerable boilerplate in the provider
|
|
|
|
code, particularly around marshaling the same types defined in TypeScript, validation, and the like, due to the use
|
|
|
|
of an RPC boundary between the engine and dynamically loaded resource provider plugins.
|
|
|
|
|
2017-04-24 19:04:18 +02:00
|
|
|
This note describes an approach to using IDL for defining resource packages that addresses these shortcomings.
|
2017-04-24 18:50:48 +02:00
|
|
|
Resource types and their properties may be defined and managed in one place, and all of the boilerplate is automatically
|
|
|
|
generated by the system. Early results indicate that providers shrink to 1/3rd the size they are right now, letting
|
|
|
|
resource authors focus on what matters most: resource-specific logic and customizations.
|
|
|
|
|
|
|
|
## Overview
|
|
|
|
|
|
|
|
The overall idea is that every resource package contains three sub-directories:
|
|
|
|
|
|
|
|
* `idl/` - this contains *.go files, possibly organized into sub-packages, containing IDL
|
|
|
|
* `pack/` - this is where the actual package resides
|
|
|
|
* `provider/` - this is where the actual resource provider code lives
|
|
|
|
|
|
|
|
The IDL compiler is named `cidlc`. In general, the workflow will be:
|
|
|
|
|
|
|
|
* Author the IDL files in a Go subset
|
2017-05-18 20:38:28 +02:00
|
|
|
* Regenerate the Lumi type definitions: `$ cidlc idl/ --out-pack=pack/`
|
|
|
|
* Regenerate the Lumi resource provider RPC stubs: `$ cidlc idl/ --out-provider=provider/`
|
2017-04-24 18:50:48 +02:00
|
|
|
|
|
|
|
For now, just like our gRPC files, we will check in the resulting generated code.
|
|
|
|
|
2017-05-18 20:38:28 +02:00
|
|
|
The Lumi type definitions are in TypeScript for the time being (as with our current hand-coded ones). So,
|
|
|
|
`cidlc` actually generates TypeScript, not a LumiPack directly. (Eventually, `cidlc` can have a pluggable
|
|
|
|
language target.) `lumijs` is then used to compile that TypeScript in the usual ways. This allows interesting things
|
2017-04-24 18:50:48 +02:00
|
|
|
like augmenting the type definitions with functions (like `aws.getLinuxAMI`).
|
|
|
|
|
2017-05-18 20:38:28 +02:00
|
|
|
The RPC stubs are simply variations of our existing gRPC `lumirpc.ResourceProvider` base type, with some boilerplate
|
2017-04-24 18:50:48 +02:00
|
|
|
around standard marshaling and validation, so the real providers can focus on important resource-specific logic.
|
|
|
|
|
|
|
|
## IDL
|
|
|
|
|
2017-05-18 20:38:28 +02:00
|
|
|
The IDL is authored in a subset of Go. Eventually, we may choose to elevate this to full-blown LumiGo, but for now, we
|
2017-04-24 18:50:48 +02:00
|
|
|
are specifically focused on IDL-only.
|
|
|
|
|
|
|
|
The only top-level elements supported at the moment are:
|
|
|
|
|
|
|
|
* Struct definitions
|
2017-04-26 00:05:51 +02:00
|
|
|
* Type aliases (for enums)
|
2017-04-24 18:50:48 +02:00
|
|
|
* Constants
|
|
|
|
|
|
|
|
Note that, in particular, interfaces, functions, and variables, are not supported. This essentially means the IDL is
|
|
|
|
capable only of defining type shapes and values, not logic, by design.
|
|
|
|
|
2017-05-18 20:38:28 +02:00
|
|
|
The only types that can be used are limited to the "JSON-like" type system that Lumi uses:
|
2017-04-24 18:50:48 +02:00
|
|
|
|
|
|
|
* Primitives: `bool`, `float64`, `string`
|
|
|
|
* Other structs
|
|
|
|
* Pointers to any of the above (if-and-only-if an optional property, see below)
|
|
|
|
* Pointers to other resource types (capabilities)
|
|
|
|
* Arrays of the above things
|
|
|
|
* Maps with `string` keys and any of the above as values
|
|
|
|
|
2017-05-18 20:38:28 +02:00
|
|
|
A type is a resource if it embeds one of the standard Lumi IDL resource types: either `pkg/resource/idl/Resource` or
|
2017-04-24 19:04:18 +02:00
|
|
|
`pkg/resource/idl/NamedResource`, the latter having a `Name` property. For example:
|
2017-04-24 18:50:48 +02:00
|
|
|
|
|
|
|
import (
|
2017-08-02 18:25:22 +02:00
|
|
|
"github.com/pulumi/pulumi-fabric/pkg/resource/idl"
|
2017-04-24 18:50:48 +02:00
|
|
|
)
|
|
|
|
type FooResource struct {
|
|
|
|
idl.Resource
|
|
|
|
}
|
|
|
|
|
|
|
|
Because the standard Go import/export system is used, visibility is chosen based on Go's casing rules (e.g.,
|
|
|
|
`CapitalCase` is exported, while `lowerCase` is not). As a result, just as with marshaling JSON in Go, casing will
|
2017-05-18 20:38:28 +02:00
|
|
|
need to be altered during translation into a Lumi package, to achieve the desired idiomatic Lumi casing.
|
2017-04-24 18:50:48 +02:00
|
|
|
|
|
|
|
Rather than requiring explicit management of name translation -- say, using tags, as with JSON marshaling -- the IDL
|
2017-05-18 20:38:28 +02:00
|
|
|
compiler automatically turns `CapitalCase` fields into idiomatic Lumi `lowerCase` properties. For example:
|
2017-04-24 18:50:48 +02:00
|
|
|
|
|
|
|
type Foo struct {
|
|
|
|
Bar string
|
|
|
|
}
|
|
|
|
|
|
|
|
The resulting `Foo` type's property will be named `bar`, not `Bar`. This is almost always what you want, however,
|
2017-05-18 20:38:28 +02:00
|
|
|
you may explicitly specify an alternative name to use using the `lumi` tag's `name=<name>` option:
|
2017-04-24 18:50:48 +02:00
|
|
|
|
|
|
|
type Foo struct {
|
2017-05-18 20:38:28 +02:00
|
|
|
Bar string `lumi:"name=Bar"`
|
2017-04-24 18:50:48 +02:00
|
|
|
}
|
|
|
|
|
2017-05-18 20:38:28 +02:00
|
|
|
In fact, there are three other options supported by the `lumi` tag, to control code generation:
|
2017-04-24 18:50:48 +02:00
|
|
|
|
|
|
|
* `optional`: this is an optional property (required is the default)
|
|
|
|
* `out`: this property is set post-creation by the resource provider, rather than set by the client
|
2017-04-26 00:05:51 +02:00
|
|
|
* `replaces`: changing this property implies a replacement of a resource (valid only on resource types); for
|
|
|
|
conditional replacements, custom logic will be required
|
2017-04-24 18:50:48 +02:00
|
|
|
|
|
|
|
The resulting package and provider will perform client- and server-side validation automatically for each of these.
|
|
|
|
|
|
|
|
Eventually we envision additional annotations, like min/max values for numbers, regexes for strings, and so on
|
2017-08-02 18:25:22 +02:00
|
|
|
(see [pulumi/pulumi-fabric#64](https://github.com/pulumi/pulumi-fabric/issues/64) for details).
|
2017-04-24 18:50:48 +02:00
|
|
|
|
2017-04-26 00:05:51 +02:00
|
|
|
Enums are supported using semi-idiomatic Go enums, by just using a `string`-backed type alias, and by declaring the
|
|
|
|
entire set of constants that the enum may take on. The IDL compiler will treat this like an enum type:
|
2017-04-24 18:50:48 +02:00
|
|
|
|
2017-04-26 00:05:51 +02:00
|
|
|
type Baz string
|
2017-04-24 18:50:48 +02:00
|
|
|
const (
|
|
|
|
A Baz = "a"
|
|
|
|
B Baz = "b"
|
|
|
|
Z Baz = "z"
|
|
|
|
)
|
|
|
|
|
2017-04-26 00:05:51 +02:00
|
|
|
Note that all enums are `string`-based at the moment, due to the "JSON-like" type system. Furthermore, non-enum type
|
|
|
|
aliases are not supported at the moment in the IDL subset.
|
2017-04-24 18:50:48 +02:00
|
|
|
|
|
|
|
## Packages
|
|
|
|
|
|
|
|
The package output's module structure mirrors the package structure in the input IDL. For example, in the AWS resource
|
|
|
|
package's IDL, there are sub-modules like `ec2`, `iam`, `s3`, and so on.
|
|
|
|
|
|
|
|
For every input file, a like-named output file is generated in the target directory (usually `pack/`), clobbering any
|
|
|
|
files if it needs to (typically overwriting a file that is version controlled).
|
|
|
|
|
|
|
|
As a result, if the IDL directory structure is:
|
|
|
|
|
|
|
|
idl/
|
|
|
|
ec2/
|
|
|
|
instance.go
|
|
|
|
security_group.go
|
|
|
|
iam/
|
|
|
|
role.go
|
|
|
|
s3/
|
|
|
|
bucket.go
|
|
|
|
|
|
|
|
The output generated package code will look like this:
|
|
|
|
|
|
|
|
pack/
|
|
|
|
ec2/
|
|
|
|
instance.ts
|
|
|
|
security_group.ts
|
|
|
|
iam/
|
|
|
|
role.ts
|
|
|
|
s3/
|
|
|
|
bucket.ts
|
|
|
|
|
|
|
|
For every struct definition in the IDL, a corresponding interface is created with the properties with the appropriate
|
|
|
|
casing, optionality, etc., as described above.
|
|
|
|
|
2017-05-18 20:38:28 +02:00
|
|
|
For every resource in the IDL, a subclass of the appropriate lumi LumiPack resource base class is created (from the
|
|
|
|
`lumi` package). Each such class will have a constructor that accepts an arguments object containing all of its
|
2017-04-24 18:50:48 +02:00
|
|
|
properties and simply validates and self-assigns them, in the straightforward way.
|
|
|
|
|
|
|
|
At the moment, custom logic for resource constructors is not supported. Eventually, when `cidlc` is generalized to
|
2017-05-18 20:38:28 +02:00
|
|
|
supporting a more full-blown LumiGo dialect, as a first class LumiLang, this will be possible.
|
2017-04-24 18:50:48 +02:00
|
|
|
|
|
|
|
## RPC Stubs
|
|
|
|
|
|
|
|
The generated RPC code (usually underneath `provider/`), provides simple wrappers on top of the underlying gRPC
|
|
|
|
interfaces. Because the IDL types are marshaled in a way that may not match the definitions precisely -- for example,
|
|
|
|
any capabilities are translated into string URNs on the wire -- there are also distinct marshaled structs.
|
|
|
|
|
2017-05-18 20:38:28 +02:00
|
|
|
The generated directory structure, like with the LumiPack output, mirrors the IDL module structure. But the expectation
|
2017-04-24 18:50:48 +02:00
|
|
|
is that the provider will require aggressive customization, and so the files are generated with suffixes:
|
|
|
|
|
|
|
|
provider/
|
|
|
|
ec2/
|
|
|
|
instance_stub.go
|
|
|
|
security_group_stub.go
|
|
|
|
iam/
|
|
|
|
role_stub.go
|
|
|
|
s3/
|
|
|
|
bucket_stub.go
|
|
|
|
|
|
|
|
For every resource type `T`, `cidlc` will generate:
|
|
|
|
|
|
|
|
* All the marshaling structs, using lower-cased variants of their IDL names (`t`, etc).
|
2017-05-18 20:38:28 +02:00
|
|
|
* A constant `T tokens.Type` containing the full LumiIL token for the resource type (e.g., `aws:ec2:SecurityGroup`).
|
2017-04-24 18:50:48 +02:00
|
|
|
* A base provider implementation `TProvider` that performs some helpful functions:
|
|
|
|
- Unmarshals property bags into the typed marshaling structs (for operations like `Create` and `Update`).
|
|
|
|
- Marshals typed marshaling structs into the property bags (for operations like `Get`).
|
|
|
|
- For named resources, an automatic implementation of the `Name` operation that defaults to the resource name.
|
|
|
|
* A provider interface, `TProviderOps`, that contains the functions a developer needs to implement.
|
|
|
|
|
|
|
|
A developer then needs to associate the `T` constant with the `TProvider`, passing an instance of the `TProviderOps`
|
|
|
|
implementation for each resource type. The standard name is to simply call this struct `tProviderOpsImpl`.
|
|
|
|
|
|
|
|
## Example: aws/ec2/SecurityGroup
|
|
|
|
|
|
|
|
This is an example of the IDL definition for the aws/ec2/SecurityGroup resource type:
|
|
|
|
|
|
|
|
```
|
|
|
|
// Copyright 2017 Pulumi, Inc. All rights reserved.
|
|
|
|
|
|
|
|
package ec2
|
|
|
|
|
|
|
|
import (
|
2017-08-02 18:25:22 +02:00
|
|
|
"github.com/pulumi/pulumi-fabric/pkg/resource/idl"
|
2017-04-24 18:50:48 +02:00
|
|
|
)
|
|
|
|
|
|
|
|
// A SecurityGroup is an Amazon EC2 Security Group. For more information, see
|
|
|
|
// http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ec2-security-group.html.
|
|
|
|
type SecurityGroup struct {
|
|
|
|
idl.NamedResource
|
|
|
|
// A required description about the security group.
|
2017-05-18 20:38:28 +02:00
|
|
|
GroupDescription string `lumi:"replace"`
|
2017-04-24 18:50:48 +02:00
|
|
|
// The VPC in which this security group resides (or blank if the default VPC).
|
2017-05-18 20:38:28 +02:00
|
|
|
VPC *VPC `lumi:"optional,replace"`
|
2017-04-24 18:50:48 +02:00
|
|
|
// A list of Amazon EC2 security group egress rules.
|
2017-05-18 20:38:28 +02:00
|
|
|
SecurityGroupEgress *[]SecurityGroupRule `lumi:"optional"`
|
2017-04-24 18:50:48 +02:00
|
|
|
// A list of Amazon EC2 security group ingress rules.
|
2017-05-18 20:38:28 +02:00
|
|
|
SecurityGroupIngress *[]SecurityGroupRule `lumi:"optional"`
|
2017-04-24 18:50:48 +02:00
|
|
|
// The group ID of the specified security group, such as `sg-94b3a1f6`.
|
2017-05-18 20:38:28 +02:00
|
|
|
GroupID *string `lumi:"out"`
|
2017-04-24 18:50:48 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
// A SecurityGroupRule describes an EC2 security group rule embedded within a SecurityGroup.
|
|
|
|
type SecurityGroupRule struct {
|
|
|
|
// The IP name or number.
|
|
|
|
IPProtocol string
|
|
|
|
// Specifies a CIDR range.
|
2017-05-18 20:38:28 +02:00
|
|
|
CIDRIP *string `lumi:"optional"`
|
2017-04-24 18:50:48 +02:00
|
|
|
// The start of port range for the TCP and UDP protocols, or an ICMP type number. An ICMP type number of `-1`
|
|
|
|
// indicates a wildcard (i.e., any ICMP type number).
|
2017-05-18 20:38:28 +02:00
|
|
|
FromPort *float64 `lumi:"optional"`
|
2017-04-24 18:50:48 +02:00
|
|
|
// The end of port range for the TCP and UDP protocols, or an ICMP code. An ICMP code of `-1` indicates a
|
|
|
|
// wildcard (i.e., any ICMP code).
|
2017-05-18 20:38:28 +02:00
|
|
|
ToPort *float64 `lumi:"optional"`
|
2017-04-24 18:50:48 +02:00
|
|
|
}
|
|
|
|
```
|
|
|
|
|