pulumi/docs/design/idl.md

// Licensed to Pulumi Corporation ("Pulumi") under one or more
// contributor license agreements.  See the NOTICE file distributed with
// this work for additional information regarding copyright ownership.
// Pulumi licenses this file to You under the Apache License, Version 2.0
// (the "License"); you may not use this file except in compliance with
// the License.  You may obtain a copy of the License at
//
//     http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

As we write more resource provider packages, we have begun to encounter shortcomings.  Properties must currently be
defined both in the LumiLang type definitions and in the Go provider implementation, requiring edits to many files just
to add, change, or remove a property (or fix a bug).  Additionally, there is considerable boilerplate in the provider
code, particularly around marshaling the same types defined in TypeScript, validation, and the like, due to the use
of an RPC boundary between the engine and dynamically loaded resource provider plugins.

This note describes an approach to using IDL for defining resource packages that addresses these shortcomings.
Resource types and their properties may be defined and managed in one place, and all of the boilerplate is automatically
generated by the system.  Early results indicate that providers shrink to 1/3rd the size they are right now, letting
resource authors focus on what matters most: resource-specific logic and customizations.

## Overview

The overall idea is that every resource package contains three sub-directories:

* `idl/` - this contains *.go files, possibly organized into sub-packages, containing IDL
* `pack/` - this is where the actual package resides
* `provider/` - this is where the actual resource provider code lives

The IDL compiler is named `cidlc`.  In general, the workflow will be:

* Author the IDL files in a Go subset
* Regenerate the Lumi type definitions: `$ cidlc idl/ --out-pack=pack/`
* Regenerate the Lumi resource provider RPC stubs: `$ cidlc idl/ --out-provider=provider/`

For now, just like our gRPC files, we will check in the resulting generated code.

The Lumi type definitions are in TypeScript for the time being (as with our current hand-coded ones).  So,
`cidlc` actually generates TypeScript, not a LumiPack directly.  (Eventually, `cidlc` can have a pluggable
language target.)  `lumijs` is then used to compile that TypeScript in the usual ways.  This allows interesting things
like augmenting the type definitions with functions (like `aws.getLinuxAMI`).

The RPC stubs are simply variations of our existing gRPC `lumirpc.ResourceProvider` base type, with some boilerplate
around standard marshaling and validation, so the real providers can focus on important resource-specific logic.

## IDL

The IDL is authored in a subset of Go.  Eventually, we may choose to elevate this to full-blown LumiGo, but for now, we
are specifically focused on IDL-only.

The only top-level elements supported at the moment are:

* Struct definitions
* Type aliases (for enums)
* Constants

Note that, in particular, interfaces, functions, and variables, are not supported.  This essentially means the IDL is
capable only of defining type shapes and values, not logic, by design.

The only types that can be used are limited to the "JSON-like" type system that Lumi uses:

* Primitives: `bool`, `float64`, `string`
* Other structs
* Pointers to any of the above (if-and-only-if an optional property, see below)
* Pointers to other resource types (capabilities)
* Arrays of the above things
* Maps with `string` keys and any of the above as values

A type is a resource if it embeds one of the standard Lumi IDL resource types: either `pkg/resource/idl/Resource` or
`pkg/resource/idl/NamedResource`, the latter having a `Name` property.  For example:

    import (
        "github.com/pulumi/pulumi-fabric/pkg/resource/idl"
    )
    type FooResource struct {
        idl.Resource
    }

Because the standard Go import/export system is used, visibility is chosen based on Go's casing rules (e.g.,
`CapitalCase` is exported, while `lowerCase` is not).  As a result, just as with marshaling JSON in Go, casing will
need to be altered during translation into a Lumi package, to achieve the desired idiomatic Lumi casing.

Rather than requiring explicit management of name translation -- say, using tags, as with JSON marshaling -- the IDL
compiler automatically turns `CapitalCase` fields into idiomatic Lumi `lowerCase` properties.  For example:

    type Foo struct {
        Bar string
    }

The resulting `Foo` type's property will be named `bar`, not `Bar`.  This is almost always what you want, however,
you may explicitly specify an alternative name to use using the `lumi` tag's `name=<name>` option:

    type Foo struct {
        Bar string `lumi:"name=Bar"`
    }

In fact, there are three other options supported by the `lumi` tag, to control code generation:

* `optional`: this is an optional property (required is the default)
* `out`: this property is set post-creation by the resource provider, rather than set by the client
* `replaces`: changing this property implies a replacement of a resource (valid only on resource types); for
    conditional replacements, custom logic will be required

The resulting package and provider will perform client- and server-side validation automatically for each of these.

Eventually we envision additional annotations, like min/max values for numbers, regexes for strings, and so on
(see [pulumi/pulumi-fabric#64](https://github.com/pulumi/pulumi-fabric/issues/64) for details).

Enums are supported using semi-idiomatic Go enums, by just using a `string`-backed type alias, and by declaring the
entire set of constants that the enum may take on.  The IDL compiler will treat this like an enum type:

    type Baz string
    const (
        A Baz = "a"
        B Baz = "b"
        Z Baz = "z"
    )

Note that all enums are `string`-based at the moment, due to the "JSON-like" type system.  Furthermore, non-enum type
aliases are not supported at the moment in the IDL subset.

## Packages

The package output's module structure mirrors the package structure in the input IDL.  For example, in the AWS resource
package's IDL, there are sub-modules like `ec2`, `iam`, `s3`, and so on.

For every input file, a like-named output file is generated in the target directory (usually `pack/`), clobbering any
files if it needs to (typically overwriting a file that is version controlled).

As a result, if the IDL directory structure is:

    idl/
        ec2/
            instance.go
            security_group.go
        iam/
            role.go
        s3/
            bucket.go

The output generated package code will look like this:

    pack/
        ec2/
            instance.ts
            security_group.ts
        iam/
            role.ts
        s3/
            bucket.ts

For every struct definition in the IDL, a corresponding interface is created with the properties with the appropriate
casing, optionality, etc., as described above.

For every resource in the IDL, a subclass of the appropriate lumi LumiPack resource base class is created (from the
`lumi` package).  Each such class will have a constructor that accepts an arguments object containing all of its
properties and simply validates and self-assigns them, in the straightforward way.

At the moment, custom logic for resource constructors is not supported.  Eventually, when `cidlc` is generalized to
supporting a more full-blown LumiGo dialect, as a first class LumiLang, this will be possible.

## RPC Stubs

The generated RPC code (usually underneath `provider/`), provides simple wrappers on top of the underlying gRPC
interfaces.  Because the IDL types are marshaled in a way that may not match the definitions precisely -- for example,
any capabilities are translated into string URNs on the wire -- there are also distinct marshaled structs.

The generated directory structure, like with the LumiPack output, mirrors the IDL module structure.  But the expectation
is that the provider will require aggressive customization, and so the files are generated with suffixes:

    provider/
        ec2/
            instance_stub.go
            security_group_stub.go
        iam/
            role_stub.go
        s3/
            bucket_stub.go

For every resource type `T`, `cidlc` will generate:

* All the marshaling structs, using lower-cased variants of their IDL names (`t`, etc).
* A constant `T tokens.Type` containing the full LumiIL token for the resource type (e.g., `aws:ec2:SecurityGroup`).
* A base provider implementation `TProvider` that performs some helpful functions:
    - Unmarshals property bags into the typed marshaling structs (for operations like `Create` and `Update`).
    - Marshals typed marshaling structs into the property bags (for operations like `Get`).
    - For named resources, an automatic implementation of the `Name` operation that defaults to the resource name.
* A provider interface, `TProviderOps`, that contains the functions a developer needs to implement.

A developer then needs to associate the `T` constant with the `TProvider`, passing an instance of the `TProviderOps`
implementation for each resource type.  The standard name is to simply call this struct `tProviderOpsImpl`.

## Example: aws/ec2/SecurityGroup

This is an example of the IDL definition for the aws/ec2/SecurityGroup resource type:

```
// Copyright 2017 Pulumi, Inc. All rights reserved.

package ec2

import (
    "github.com/pulumi/pulumi-fabric/pkg/resource/idl"
)

// A SecurityGroup is an Amazon EC2 Security Group.  For more information, see
// http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ec2-security-group.html.
type SecurityGroup struct {
    idl.NamedResource
    // A required description about the security group.
    GroupDescription string `lumi:"replace"`
    // The VPC in which this security group resides (or blank if the default VPC).
    VPC *VPC `lumi:"optional,replace"`
    // A list of Amazon EC2 security group egress rules.
    SecurityGroupEgress *[]SecurityGroupRule `lumi:"optional"`
    // A list of Amazon EC2 security group ingress rules.
    SecurityGroupIngress *[]SecurityGroupRule `lumi:"optional"`
    // The group ID of the specified security group, such as `sg-94b3a1f6`.
    GroupID *string `lumi:"out"`
}

// A SecurityGroupRule describes an EC2 security group rule embedded within a SecurityGroup.
type SecurityGroupRule struct {
    // The IP name or number.
    IPProtocol string
    // Specifies a CIDR range.
    CIDRIP *string `lumi:"optional"`
    // The start of port range for the TCP and UDP protocols, or an ICMP type number.  An ICMP type number of `-1`
    // indicates a wildcard (i.e., any ICMP type number).
    FromPort *float64 `lumi:"optional"`
    // The end of port range for the TCP and UDP protocols, or an ICMP code.  An ICMP code of `-1` indicates a
    // wildcard (i.e., any ICMP code).
    ToPort *float64 `lumi:"optional"`
}
```