pulumi/docs/language.md

Mu Language

Caution: This document is out of date. Please refer to languages for the latest thinking.

Mu Metadata is written in a "YAML-like" language, Mull (Mu's little language). (There is a "JSON-like" alternative for those who prefer a JSON style of syntax.) Mull isn't vanilla YAML, however, for two reasons:

1. It may be strongly typed to support better compile-time validation.
2. It may contain embedded quotations that generate data in a rich, semantically-aware way.

These two things are tightly integrated with one another so that the result works a bit like an ordinary programming language's type system. For example, quotations are also strongly typed, and can leverage typed metadata.

Both aspects of Mull have been heavily inspired by other systems. For typing, by TypeScript and JSON Schema. For quotations, by Hashicorp's HCL, Go templates, and Jsonnet. A key difference is that we remain true to our YAML and JSON heritage wherever possible, so that existing tools and skillsets may be leveraged. In general, if one of these formats already has an answer for a question we face -- like the syntax for number literals -- we choose it.

Strong Typing

All variables and values have types and the Mu compiler typechecks that they are used correctly. The set of types expressible remains very close at the core to YAML and JSON, but adds more structure for custom types.

Basic Types

The special type any may refer to any type.

The basic primitive types are bool, number, and string.

Any type T can be modified by appending [] to turn it into an array T[]: e.g., number[], string[], any[].

Similarly, two types can be paired up to make a map type using map[K, V], where K is the type of keys used to index into the map and V is the type of value inside: e.g., map[string, number], map[number, any], and so on. Note that only the primtive types bool, number, and string can be used as key types inside of a map.

Tuples and Custom Types

An anonymous tuple type, which is essentially a map with known properties, can be created using {}: e.g., the tuple containing two properties, a string and a number, is { string, number }. These are accessed by their ordinal position. If you prefer to name the properties, you can do so { name: string, age: number }.

For situations when an anonymous tuple becomes too verbose or repetetive, or advanced features are required, it is possible to define custom complex types.

Each custom type has the following attributes:

• A name.
• An optional "base" type.
• An optional default value.
• An optional description.
• Either of these:
  • An optional set of properties.
  • An optional set of value constraints.

If properties exist, the type must be a custom object type. If base exists, these properties extend it. Each property is a superset of the custom type, in that it can carry any or all of those attributes, with two caveats: instead of "base" it is simply called "type" and it may also carry an indicator of whether a property is required.

The value constraints only apply to custom types whose "base" is either string or number:

• For strings:
  • A maximum length in characters.
  • A minimum length in characters.
  • A regex pattern for legal values.
• For numbers:
  • A maximum value.
  • A minimum value.
• For strings and numbers:
  • An enum array of legal values.

TODO: we need to specify precisely where custom types may appear, how the system knows to bind them, etc. In the Mu case, there are actually two places: the stack's property definitions themselves (properties), and the custom type section (types). Maybe we can use some "meta-typing" to indicate this, e.g. in the Mufile schema itself.

Embedded Quotations

A quotation allows computation to be mixed into what is otherwise just markup data.

It's important to note that quotations are not templates. Although quotations are very powerful and provide functionality often offered by templating systems, the quotations are semantically understood at compile-time. They are strongly typed and are generally declarative in nature. This is in contrast to templating systems which normally perform "dumb" textual copy-and-paste, possibly leading to output that isn't even legal syntax.

All quotations are wrapped in ${} syntax -- such as ${foo} -- and can reference variables, call functions, perform conditional operations, and more. To escape a quotation, use a double dollar sign; for example, $${foo} will be rendered as the string ${foo}, instead of the variable named foo's value.

Variables

All quotations are evaluated within a scope containing any number of named variables.

To inject a variable's value into the markup stream, simply reference it by name; for example, ${foo} is replaced by the value of a variable called foo in the current scope. If foo's value is "bar", then the result is "bar".

Variable Types

Note that variables have types. So "replaced by the value" does not necessarily mean that the result is a string. This allows variables to carry complex structure that is emitted into the markup stream as-is. Note that because variable substitution works hand-in-hand with the typechecking mentioned earlier, any conflicting types that result will lead to errors as you would expect. This is one of the advantages to the quotation approach versus templates.

A function exists that will stringify any value, however: ${string(foo)}. There are also conversion functions for other types, described below in the functions section. They are unique because they typically require parsing.

Declaring New Variables

New variables may be introduced using the let syntax; a variable, once bound, is immutable and cannot be changed:

${let foo = "bar"}

A variable is unlike a property in that it is an implementation detail and not exported for public use.

A variable's type is inferred by its initialization value, although it can be specified precisely using type assertions. In general, Mull has a structural type system and permits conversions between like-structured types. However, if you would like to assert that a variable is of a given type, the type{expr} syntax will do this, where type is the desired type and expr is an arbitrary expression. A compile-time error occurs if the conversion can't take place.

This can be useful when assigning maps to variables. If the map is meant to be a custom type myType, you can say:

${let foo = myType{ a: "s", b: 42 }}

Accessing Properties

Properties of a resulting value may be accessed using dots. For example, if foo is of a complex type with multiple properties, a, b, and c, we can access them simply by saying ${foo.a}, ${foo.b}, and ${foo.c}. Those can of course also be complex types with their own properties accessed through dotting, and so on, and so forth.

Accessing Array and Map Elements

Elements of array and map values are accessed using []. For example, ${foo[0]} extracts the 0th element of an array variable foo, while ${bar["baz"]} extracts an element keyed by the string "baz" from the map variable bar.

Scopes, This, and Context

The scope can be customized and populated in any number of ways. There are always two specific special variables:

• type refers to the current type specification.
• this refers to the "current object" (whatever that means in the domain-specific context; sometimes just type).

TODO: we still don't have an elegant formalism or even a good mental "model" for the distinction between "this script" (e.g., type), and "this object" (e.g., this). In Mu, this makes sense, because the primary use case for templating is accessing the properties set by a caller instantiating the current stack represented by the file. But for many other cases, you actually just want to access enclosing properties inside of the file. I should note, if we went back to parameters and arguments, rather than properties, we'd have a distinction between the two rooted by a single object (with the unfortunate indirection of needing to say arguments.foo instead of just foo.

Much like your favorite programming language, properties on this may be accessed either by explicitly naming this, as in ${this.foo}, or simply by naming the property without a prefix, as in ${foo}.

Mu binds this to represent the current stack being constructed by a given Mufile. As a result, each property defined inside of a stack is thus available in the form of an automatically bound variable.

Additionally, Mu makes a ctx variable available that is bound to information about the current compilation unit.

TODO: describe the context.

Operators

TODO: comparisons.

TODO: math.

TODO: concatenation.

Constructors

Many values are injected simply by referencing a variable, its properties, calling a function, etc. Sometimes, however, new values are created anew out of existing components.

New array values can be created using the [] operator. For example, ${[ a, b, c ]} constructs a three-element array out of the variables a, b, and c.

New map values, on the other hand, use {}. For example, ${{ "a": a, "b": b, "c": c }} constructs a three-element map keyed by the strings "a", "b", and "c", and with those same variables as the keys' respective values.

Note that the type inference engine will try its best to get the types right. In case it gets something wrong, you may simply use a type assertion. For example, maybe instead of a map, we meant to create a custom type named myType; to say that, we simply wrap the above example in a myType{} type assertion: ${myType{ "a": a, "b": b, "c": c }}.

A Word on Nulls

Any property whose value is set to undefined is omitted from the result. This can be useful when propagating property values between different objects. For example, let's say in our Mufile the optional item property was missing, and yet we were about to propagate that property to another object. In one model, we would need to say:

${if item != null}
    item: ${item}
${done}

Instead, however, we can simply set the property and let the system omit the result if it is undefined:

item: ${item}

Functions

A function is easily recognizable by the presence of parenthesis: ${func(...)}, where ... is the optional set of comma-delimited arguments. For example, ${concat(a1, a2, a3)} concatenates three arrays, a1, a2, and a3, evaluating to this overall resulting array. Below is a complete list of the built-in functions available:

TODO: list them.

TODO: include for semantic inclusion (call it import?).

TODO: for built-ins, we probably want some combination of HCL's (https://www.terraform.io/docs/configuration/interpolation.html) and what we were getting from Sprig (https://github.com/Masterminds/sprig)

Conditionals

Mull also supports conditional code in two forms: if and for. These are slightly different than their general- purpose programming language equivalents -- in that they are "declarative" in nature -- but should feel familiar. Being more declarative helps to encourage best practices when writing deterministic and predictable Mufiles.

Both support an expression and block statement form. The expression forms look like function calls and are handy for short declarative data situations, like conditionally setting a property. The block forms are better for complex cases.

If

An if expression evaluates to a value. For example, in ${if(foo == "bar", "a", "b")}, the expression evaluates to "a" when foo == "bar" is true, and "b" otherwise. Note that the else part may also be omitted, in which case the expression yields undefined should the predicate be false, e.g. ${if(foo == "bar", "a")}.

A guarded if block, or series of them, allows more sophisticated "control flow" within a Mull file. For example:

${if foo == "bar"}
    ...
${else if bar == "baz"}
    ...
${else}
    ...
${done}

This now probably looks more like your favorite programming language except that the bodies produce values. These are evaluated in order and the first one to succeed leads to the inclusion of its body in the overall markup stream. Note that the body here is arbitrary Mull code: markup, additional quotations, or some combination thereof.

The ${done} terminates the overall cascade of guards.

TODO: switch statement?

For

Any array or map, including custom data types, may be enumerated using the for expression or statement.

In the array case, each element is available as the special item variable, whereas in the map case, each key is available as the special key variable and each value as value. Notice that the names are built-in to reduce the amount of boilerplate in Mull files; remember, they are intentionally kept simple and declarative.

In expression form, each evaluates to an array or map of values.

For example, ${for(a, item)} is the identity statement for an array variable a, while ${for(m, { key: value })} is the identity statement for a map variable m. We can do more interesting things, like square an array of numbers, ${for(a, item*item)}, or prepend a constant to the keys in a map, ${for(m, { "prefix-" + key: value }}. By default, the result of an each is an array containing the values of each iteration, in order, unless the body evaluates to a map, in which case its keys are merged to produce a single aggregate map. By default, keys are sorted before enumerating a map, in order to provide a guaranteed and deterministic execution order.

In statement form, for's body is repeated for each element in the array or map:

${for [a, b, c]}
    ...
${done}

The same special item, key, and value variables are bound in the body of a for statement block.

The contents of each body can be any combination of markup and/or quotations.

Note that the special built-in function range can be used to create a typical for loop:

${for range(0, 10)}
    ...
${done}