pulumi/docs/design/idl.md

// Licensed to Pulumi Corporation ("Pulumi") under one or more // contributor license agreements. See the NOTICE file distributed with // this work for additional information regarding copyright ownership. // Pulumi licenses this file to You under the Apache License, Version 2.0 // (the "License"); you may not use this file except in compliance with // the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License.

As we write more resource provider packages, we have begun to encounter shortcomings. Properties must currently be defined both in the LumiLang type definitions and in the Go provider implementation, requiring edits to many files just to add, change, or remove a property (or fix a bug). Additionally, there is considerable boilerplate in the provider code, particularly around marshaling the same types defined in TypeScript, validation, and the like, due to the use of an RPC boundary between the engine and dynamically loaded resource provider plugins.

This note describes an approach to using IDL for defining resource packages that addresses these shortcomings. Resource types and their properties may be defined and managed in one place, and all of the boilerplate is automatically generated by the system. Early results indicate that providers shrink to 1/3rd the size they are right now, letting resource authors focus on what matters most: resource-specific logic and customizations.

Overview

The overall idea is that every resource package contains three sub-directories:

• idl/ - this contains *.go files, possibly organized into sub-packages, containing IDL
• pack/ - this is where the actual package resides
• provider/ - this is where the actual resource provider code lives

The IDL compiler is named cidlc. In general, the workflow will be:

• Author the IDL files in a Go subset
• Regenerate the Lumi type definitions: $ cidlc idl/ --out-pack=pack/
• Regenerate the Lumi resource provider RPC stubs: $ cidlc idl/ --out-provider=provider/

For now, just like our gRPC files, we will check in the resulting generated code.

The Lumi type definitions are in TypeScript for the time being (as with our current hand-coded ones). So, cidlc actually generates TypeScript, not a LumiPack directly. (Eventually, cidlc can have a pluggable language target.) lumijs is then used to compile that TypeScript in the usual ways. This allows interesting things like augmenting the type definitions with functions (like aws.getLinuxAMI).

The RPC stubs are simply variations of our existing gRPC lumirpc.ResourceProvider base type, with some boilerplate around standard marshaling and validation, so the real providers can focus on important resource-specific logic.

IDL

The IDL is authored in a subset of Go. Eventually, we may choose to elevate this to full-blown LumiGo, but for now, we are specifically focused on IDL-only.

The only top-level elements supported at the moment are:

• Struct definitions
• Type aliases (for enums)
• Constants

Note that, in particular, interfaces, functions, and variables, are not supported. This essentially means the IDL is capable only of defining type shapes and values, not logic, by design.

The only types that can be used are limited to the "JSON-like" type system that Lumi uses:

• Primitives: bool, float64, string
• Other structs
• Pointers to any of the above (if-and-only-if an optional property, see below)
• Pointers to other resource types (capabilities)
• Arrays of the above things
• Maps with string keys and any of the above as values

A type is a resource if it embeds one of the standard Lumi IDL resource types: either pkg/resource/idl/Resource or pkg/resource/idl/NamedResource, the latter having a Name property. For example:

import (
    "github.com/pulumi/pulumi-fabric/pkg/resource/idl"
)
type FooResource struct {
    idl.Resource
}

Because the standard Go import/export system is used, visibility is chosen based on Go's casing rules (e.g., CapitalCase is exported, while lowerCase is not). As a result, just as with marshaling JSON in Go, casing will need to be altered during translation into a Lumi package, to achieve the desired idiomatic Lumi casing.

Rather than requiring explicit management of name translation -- say, using tags, as with JSON marshaling -- the IDL compiler automatically turns CapitalCase fields into idiomatic Lumi lowerCase properties. For example:

type Foo struct {
    Bar string
}

The resulting Foo type's property will be named bar, not Bar. This is almost always what you want, however, you may explicitly specify an alternative name to use using the lumi tag's name=<name> option:

type Foo struct {
    Bar string `lumi:"name=Bar"`
}

In fact, there are three other options supported by the lumi tag, to control code generation:

• optional: this is an optional property (required is the default)
• out: this property is set post-creation by the resource provider, rather than set by the client
• replaces: changing this property implies a replacement of a resource (valid only on resource types); for conditional replacements, custom logic will be required

The resulting package and provider will perform client- and server-side validation automatically for each of these.

Eventually we envision additional annotations, like min/max values for numbers, regexes for strings, and so on (see pulumi/pulumi-fabric#64 for details).

Enums are supported using semi-idiomatic Go enums, by just using a string-backed type alias, and by declaring the entire set of constants that the enum may take on. The IDL compiler will treat this like an enum type:

type Baz string
const (
    A Baz = "a"
    B Baz = "b"
    Z Baz = "z"
)

Note that all enums are string-based at the moment, due to the "JSON-like" type system. Furthermore, non-enum type aliases are not supported at the moment in the IDL subset.

Packages

The package output's module structure mirrors the package structure in the input IDL. For example, in the AWS resource package's IDL, there are sub-modules like ec2, iam, s3, and so on.

For every input file, a like-named output file is generated in the target directory (usually pack/), clobbering any files if it needs to (typically overwriting a file that is version controlled).

As a result, if the IDL directory structure is:

idl/
    ec2/
        instance.go
        security_group.go
    iam/
        role.go
    s3/
        bucket.go

The output generated package code will look like this:

pack/
    ec2/
        instance.ts
        security_group.ts
    iam/
        role.ts
    s3/
        bucket.ts

For every struct definition in the IDL, a corresponding interface is created with the properties with the appropriate casing, optionality, etc., as described above.

For every resource in the IDL, a subclass of the appropriate lumi LumiPack resource base class is created (from the lumi package). Each such class will have a constructor that accepts an arguments object containing all of its properties and simply validates and self-assigns them, in the straightforward way.

At the moment, custom logic for resource constructors is not supported. Eventually, when cidlc is generalized to supporting a more full-blown LumiGo dialect, as a first class LumiLang, this will be possible.

RPC Stubs

The generated RPC code (usually underneath provider/), provides simple wrappers on top of the underlying gRPC interfaces. Because the IDL types are marshaled in a way that may not match the definitions precisely -- for example, any capabilities are translated into string URNs on the wire -- there are also distinct marshaled structs.

The generated directory structure, like with the LumiPack output, mirrors the IDL module structure. But the expectation is that the provider will require aggressive customization, and so the files are generated with suffixes:

provider/
    ec2/
        instance_stub.go
        security_group_stub.go
    iam/
        role_stub.go
    s3/
        bucket_stub.go

For every resource type T, cidlc will generate:

• All the marshaling structs, using lower-cased variants of their IDL names (t, etc).
• A constant T tokens.Type containing the full LumiIL token for the resource type (e.g., aws:ec2:SecurityGroup).
• A base provider implementation TProvider that performs some helpful functions:
  • Unmarshals property bags into the typed marshaling structs (for operations like Create and Update).
  • Marshals typed marshaling structs into the property bags (for operations like Get).
  • For named resources, an automatic implementation of the Name operation that defaults to the resource name.
• A provider interface, TProviderOps, that contains the functions a developer needs to implement.

A developer then needs to associate the T constant with the TProvider, passing an instance of the TProviderOps implementation for each resource type. The standard name is to simply call this struct tProviderOpsImpl.

Example: aws/ec2/SecurityGroup

This is an example of the IDL definition for the aws/ec2/SecurityGroup resource type:

// Copyright 2017 Pulumi, Inc. All rights reserved.

package ec2

import (
    "github.com/pulumi/pulumi-fabric/pkg/resource/idl"
)

// A SecurityGroup is an Amazon EC2 Security Group.  For more information, see
// http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ec2-security-group.html.
type SecurityGroup struct {
    idl.NamedResource
    // A required description about the security group.
    GroupDescription string `lumi:"replace"`
    // The VPC in which this security group resides (or blank if the default VPC).
    VPC *VPC `lumi:"optional,replace"`
    // A list of Amazon EC2 security group egress rules.
    SecurityGroupEgress *[]SecurityGroupRule `lumi:"optional"`
    // A list of Amazon EC2 security group ingress rules.
    SecurityGroupIngress *[]SecurityGroupRule `lumi:"optional"`
    // The group ID of the specified security group, such as `sg-94b3a1f6`.
    GroupID *string `lumi:"out"`
}

// A SecurityGroupRule describes an EC2 security group rule embedded within a SecurityGroup.
type SecurityGroupRule struct {
    // The IP name or number.
    IPProtocol string
    // Specifies a CIDR range.
    CIDRIP *string `lumi:"optional"`
    // The start of port range for the TCP and UDP protocols, or an ICMP type number.  An ICMP type number of `-1`
    // indicates a wildcard (i.e., any ICMP type number).
    FromPort *float64 `lumi:"optional"`
    // The end of port range for the TCP and UDP protocols, or an ICMP code.  An ICMP code of `-1` indicates a
    // wildcard (i.e., any ICMP code).
    ToPort *float64 `lumi:"optional"`
}