2016-11-04 19:23:59 +01:00
|
|
|
# Mu Metadata Specification
|
2016-11-02 21:04:27 +01:00
|
|
|
|
2016-11-04 19:23:59 +01:00
|
|
|
This document contains a formal description of Mu's metadata. For more details on how this metdata is compiled when
|
|
|
|
targeting various cloud providers, please refer to [the companion design document](targets.md).
|
2016-11-02 21:04:27 +01:00
|
|
|
|
2016-11-04 19:23:59 +01:00
|
|
|
## Overview
|
2016-11-02 21:04:27 +01:00
|
|
|
|
2016-11-03 23:12:22 +01:00
|
|
|
The essential artifact a developer uses to create Stacks and Services is something we call a Mufile. It is
|
|
|
|
conventionally named `Mu.yaml`, usually checked into the Workspace, and each single file describes a Stack. (Note that
|
|
|
|
Stacks may internally contain other Stacks, however this is purely an implementation detail of how the Stack works.)
|
2016-11-02 21:04:27 +01:00
|
|
|
|
2016-11-03 23:12:22 +01:00
|
|
|
TODO(joe): declarative specification format for Clusters.
|
2016-11-02 21:04:27 +01:00
|
|
|
|
2016-11-03 23:12:22 +01:00
|
|
|
Although all examples are in YAML, it is perfectly valid to use JSON instead if that is more desirable.
|
2016-11-02 21:04:27 +01:00
|
|
|
|
2016-11-03 23:47:52 +01:00
|
|
|
Mu preprocesses all metadata files to substitute context values not known until runtime, such as configuration,
|
|
|
|
arguments, and so on. The [Go template syntax](https://golang.org/pkg/text/template/) is used for this. Please refer
|
|
|
|
to the API documentation for the context object (TODO(joe): do this) for details on what information is available.
|
|
|
|
|
2016-11-04 19:23:59 +01:00
|
|
|
## Package Managament
|
2016-11-02 21:04:27 +01:00
|
|
|
|
|
|
|
Each Mufile begins with some standard "package manager"-like metadata, like name, version, description, and so on. As
|
2016-11-17 03:08:32 +01:00
|
|
|
with most package managers, most of these elements are optional. For example:
|
2016-11-02 21:04:27 +01:00
|
|
|
|
|
|
|
name: elk
|
|
|
|
version: 1.0.1
|
|
|
|
description: A fully functioning ELK stack (Elasticsearch, Logstash, Kibana).
|
|
|
|
author: Joe Smith <joesmith@elk.com>
|
|
|
|
website: https://github.com/joesmith/elk
|
|
|
|
|
2016-11-17 03:08:32 +01:00
|
|
|
TODO(joe): full set of attributes.
|
|
|
|
|
|
|
|
In addition to basic metadata like this, any dependency packages must also be listed explicitly.
|
|
|
|
|
2016-11-02 21:04:27 +01:00
|
|
|
TODO(joe): finish this section.
|
|
|
|
|
2016-11-04 23:43:03 +01:00
|
|
|
## Security
|
|
|
|
|
|
|
|
TODO(joe): we need the ability to override the default Role/ACLs/etc.
|
|
|
|
|
2016-11-04 19:23:59 +01:00
|
|
|
## Stacks and Subclassing
|
2016-11-03 23:52:03 +01:00
|
|
|
|
|
|
|
A Stack may subclass any other Stack, specializing aspects of it as appropriate. This facilitates reuse. For instance,
|
|
|
|
perhaps my company wishes to enforce that certain best practices and standards are adhered to, for all Stacks. Or
|
|
|
|
imagine someone in the community has published a best-in-breed Node.js application blueprint, leveraging Express,
|
|
|
|
MongoDB, and the ELK stack, and I merely want to plug in my own application logic and leverage the overall Stack.
|
|
|
|
|
|
|
|
To do this, reference another Stack's fully qualified name in the `base` property:
|
|
|
|
|
|
|
|
base: some/other/stack
|
|
|
|
|
2016-11-19 02:30:32 +01:00
|
|
|
From there, I can specify additional metadata, however we will have inherited everything from the base.
|
2016-11-03 23:52:03 +01:00
|
|
|
|
|
|
|
TODO(joe): what about mixins?
|
2016-11-04 02:29:23 +01:00
|
|
|
|
2016-11-03 23:52:03 +01:00
|
|
|
TODO(joe): get more specific about what can be overridden. Furthermore, what about "deletes"?
|
|
|
|
|
2016-11-19 02:30:32 +01:00
|
|
|
In addition to subclassing, a Stack may be marked abstract, indicating that it cannot be instantiated:
|
|
|
|
|
|
|
|
abstract: true
|
|
|
|
|
|
|
|
A non-abstract Stack must in fact have a non-zero number of Services, whereas an abstract one can omit Services
|
|
|
|
entirely; this can be used, for example, to predefine certain non-Service metadata for subclassing Stacks.
|
|
|
|
|
2016-11-04 19:23:59 +01:00
|
|
|
## APIs
|
2016-11-03 23:12:22 +01:00
|
|
|
|
|
|
|
Every Stack may choose to export one or more APIs. These APIs can be standard "unconstrained" network interfaces, such
|
|
|
|
as "HTTP over port 80", or can take on a more structured form, like leveraging OpenAPI to declare a protocol interface.
|
|
|
|
The benefits of declaring the full interfaces are that the RPC semantics are known to the system, facilitating advanced
|
|
|
|
management capabilities such as diagnostics, monitoring, fuzzing, and self-documentation, in addition to RPC code-
|
|
|
|
generation. This also adds a sort of "strong typing" to the connections between Services.
|
|
|
|
|
|
|
|
TODO(joe): articulate this section further; e.g., the metadata format, precise advantages, etc.
|
|
|
|
|
2016-11-04 19:23:59 +01:00
|
|
|
## Stack Constructors
|
2016-11-03 23:47:52 +01:00
|
|
|
|
2016-11-04 19:23:59 +01:00
|
|
|
Each Stack can declare a set of constructor parameters that callers must supply during creation:
|
2016-11-03 23:47:52 +01:00
|
|
|
|
|
|
|
parameters:
|
|
|
|
|
|
|
|
Each parameter has the following properties:
|
|
|
|
|
|
|
|
* `name`: A name unique amongst all parameters.
|
2016-11-04 02:27:21 +01:00
|
|
|
* `description`: An optional long-form description of the parameter.
|
2016-11-03 23:47:52 +01:00
|
|
|
* `type`: A parameter type, restricting the legal values.
|
|
|
|
* `default`: A default value to be supplied if missing from the caller.
|
|
|
|
* `optional`: If `true`, this parameter is optional.
|
|
|
|
|
|
|
|
The set of types a parameter may take on are "JSON-like". This includes simple primitives:
|
|
|
|
|
|
|
|
type: string
|
|
|
|
type: number
|
|
|
|
type: boolean
|
|
|
|
type: object
|
|
|
|
|
|
|
|
As well as array shapes utilizing them:
|
|
|
|
|
|
|
|
type: [ string ]
|
|
|
|
type: [ number ]
|
|
|
|
type: [ boolean ]
|
|
|
|
type: [ object ]
|
|
|
|
|
|
|
|
Complex structures can be described simply using objects with properties:
|
|
|
|
|
|
|
|
name: tag
|
|
|
|
type:
|
|
|
|
id: number
|
|
|
|
name: string
|
|
|
|
value: object
|
|
|
|
|
2016-11-04 02:27:21 +01:00
|
|
|
The most interesting feature here is the ability to request a "capability", or reference to another Service. This
|
2016-11-03 23:47:52 +01:00
|
|
|
provides a strongly typed and more formal way of expressing Service dependencies, in a way that the system can
|
|
|
|
understand and leverage in its management of the system (like ensuring Services are created in the right order). It
|
|
|
|
also eliminates some of the fragility of weakly typed and dynamic approaches, which can be prone to race conditions.
|
|
|
|
|
|
|
|
The most basic form is to use the special type `service`:
|
|
|
|
|
|
|
|
type: service
|
|
|
|
|
|
|
|
This is helpful, as it exposes a dependency to the system, but it isn't perfect. The shape of the dependency is still
|
|
|
|
opaque to the system. A step further is to express that a specific port is utilized:
|
|
|
|
|
|
|
|
type: service:80
|
|
|
|
|
|
|
|
This declaration says that we require a Service with an exposed port 80. This strong typing flows through the system,
|
|
|
|
permitting liveness detection, and even compile-time type checking that the supplied Service argument actually does
|
|
|
|
expose something on port 80 -- in the event that this ever changes, we will find out upon recompilation.
|
|
|
|
|
|
|
|
Even better still is to declare that we depend on a specific kind of Service, by specifying the fully qualified name of
|
|
|
|
a Stack. In such a case, the system ensures an instance of this Stack type, or subclass, is provided:
|
|
|
|
|
|
|
|
type: examples/keyValueStore
|
|
|
|
|
|
|
|
This hypothetical Stack defines an API that can be used as a key-value store. Presumably we would find subclasses of it
|
|
|
|
for etcd, Consul, Zookeeper, and others, which a caller is free to choose from at instantiation time.
|
|
|
|
|
|
|
|
Another example leverages the primitive `mu/volume` type to require a Service which can be mounted as a volume:
|
|
|
|
|
|
|
|
type: mu/volume
|
|
|
|
|
2016-11-04 02:27:21 +01:00
|
|
|
The simple form of expressing parameters is `name: type`:
|
|
|
|
|
|
|
|
parameters:
|
|
|
|
first: string
|
|
|
|
second: number
|
|
|
|
|
|
|
|
The long-form, should other properties be used, is to use an array:
|
|
|
|
|
|
|
|
parameters:
|
|
|
|
- name: first
|
|
|
|
type: string
|
|
|
|
...
|
|
|
|
- name: second
|
|
|
|
type: number
|
|
|
|
...
|
|
|
|
|
2016-11-03 23:47:52 +01:00
|
|
|
Finally, note that anywhere inside of this Mufile, we may access the arguments supplied at Stack instantiation time
|
|
|
|
using the Go template syntax mentioned earlier. For example, `{{.args.tag.name}}`.
|
|
|
|
|
2016-11-04 19:23:59 +01:00
|
|
|
## Configuration
|
|
|
|
|
|
|
|
TODO(joe): write this section.
|
2016-11-03 23:47:52 +01:00
|
|
|
|
2016-11-04 19:23:59 +01:00
|
|
|
## Services
|
2016-11-02 21:04:27 +01:00
|
|
|
|
|
|
|
After that comes the section that describes what Services make up this Stack:
|
|
|
|
|
|
|
|
services:
|
|
|
|
|
2016-11-03 23:12:22 +01:00
|
|
|
In this section is zero-to-many Services that are co-created with one another. Each Service has:
|
|
|
|
|
|
|
|
* A name, both for dynamic and static use.
|
|
|
|
* A type, which is just the name of a Stack to instantiate.
|
|
|
|
* A visibility governing whether consumers of this Stack have access to it or not.
|
|
|
|
* One or more named arguments, mapping to the Stack's constructor parameters.
|
|
|
|
|
|
|
|
Although these Services are co-created, they may reference one another. The references between each other forms a DAG
|
|
|
|
and the system topologically sorts that DAG in order to determine the order in which to create and destroy Services.
|
2016-11-03 23:47:52 +01:00
|
|
|
Notably there may be no cycles. By default, the system understands liveness and health (TODO(joe): how); as a result,
|
2016-11-03 23:12:22 +01:00
|
|
|
the developer need not explicitly worry about races, liveness, or retries during Service creation.
|
|
|
|
|
2016-11-04 19:23:59 +01:00
|
|
|
### Names
|
2016-11-03 23:12:22 +01:00
|
|
|
|
|
|
|
A Service's name can be set in one of two ways. The simplest is to use the "default", derived from the Stack type. For
|
|
|
|
example, in the following metadata, the single Service has type `nginx/nginx`, gets a default name of `nginx`:
|
|
|
|
|
|
|
|
services:
|
|
|
|
public:
|
|
|
|
nginx/nginx:
|
|
|
|
port: 80
|
|
|
|
|
|
|
|
Note that this is the latter part of the name; something called `elasticsearch/kibana` would get a name of `kibana`.
|
|
|
|
|
|
|
|
If we wish instead to give this an explicit name, say `www`, we can do so using the `type` property:
|
|
|
|
|
|
|
|
services:
|
|
|
|
public:
|
|
|
|
www:
|
|
|
|
type: nginx/nginx
|
|
|
|
port: 80
|
|
|
|
|
|
|
|
A Service's name is visible at runtime (e.g., in logs, diagnostics commands, and so on), in addition to controlling how
|
|
|
|
metadata cross-referenes that Service. All Services live within a Stack, which of course has a name. Inside of a
|
|
|
|
Stack, this outer name becomes its Namespace. For instance, inside of a Stack named `marapongo/mu`, a Service named `x`
|
|
|
|
has a fully qualified name (FQN) of `marapongo/mu/x`. Although we seldom need the FQN for references within a single
|
|
|
|
Stack, they are sometimes needed for inter-Stack references, in addition to management activities.
|
|
|
|
|
2016-11-04 19:23:59 +01:00
|
|
|
### Types
|
2016-11-03 23:12:22 +01:00
|
|
|
|
|
|
|
Each Service has a type, which is merely the name of another Stack. Most of the time this is the FQN, although for
|
|
|
|
references to other Stacks defined within the same Mufile (more on that later), this can just be a simple name. During
|
|
|
|
instantiation of that Service, a fresh instance of that Stack is created and bound to in place of this Service name.
|
|
|
|
|
|
|
|
Although there are obviously many opportunities for ecosystems of user-defined Stacks, and indeed a rich library offered
|
|
|
|
by the Mu platofrm itself, we eventually bottom out on a core set of "primitive" constructs.
|
|
|
|
|
2016-11-06 18:32:46 +01:00
|
|
|
The primitive types are in the `mu` namespace and include:
|
2016-11-03 23:12:22 +01:00
|
|
|
|
|
|
|
* `mu/container`: A Docker container wrapped in Mu metadata.
|
|
|
|
* `mu/gateway`: An API gateway and/or load balancer, multiplexing requests onto multiple target Services.
|
|
|
|
* `mu/func`: A single Function ordinarily used for serverless/stateless scenarios.
|
|
|
|
* `mu/event`: An Event that may be used to Trigger the execution of another Service (commonly a Function).
|
|
|
|
* `mu/volume`: A volume stores data that can be mounted by another Service.
|
|
|
|
* `mu/autoscaler`: A Service that automatically multi-instances and scales some other target Service based on policy.
|
2016-11-04 02:27:21 +01:00
|
|
|
* `mu/extension`: A logical Service that extends Mu by hooking into events, like Stack provisioning, and taking action.
|
2016-11-03 23:12:22 +01:00
|
|
|
|
|
|
|
TODO(joe): link to exhaustive details on each of these.
|
2016-11-04 20:43:31 +01:00
|
|
|
TODO(joe): consider a `mu/job` (e.g., ECS's RunTask); unclear on how this would differ from `mu/func`.
|
|
|
|
TODO(joe): consider a `mu/daemon` type, similar to Kube's DaemonSet abstraction.
|
2016-11-03 23:12:22 +01:00
|
|
|
|
|
|
|
Although these may look like "magic", each primitive Stack simply leverages an open extensibility API in the platform.
|
|
|
|
Most interesting tasks may be achieved by composing existing Stacks, however, this extensibility API may be used to
|
|
|
|
define new, custom primitive Stacks for even richer functionality. TODO(joe): more on this.
|
|
|
|
|
2016-11-06 18:32:46 +01:00
|
|
|
Finally, note that a companion namespace, `mu/x` also exists, that offers more cloud-neutral platform abstractions.
|
|
|
|
|
2016-11-04 19:23:59 +01:00
|
|
|
### Visibility
|
2016-11-03 23:12:22 +01:00
|
|
|
|
2016-11-02 21:04:27 +01:00
|
|
|
At this point, a new concept is introduced: *visibility*. Visibility works much like your favorite programming
|
|
|
|
language, in that a Stack may declare that any of its Services are `public` or `private`. This impacts the
|
|
|
|
accessibility of those Services to consumers of this Stack. A private Service is merely an implementation detail of
|
|
|
|
the Stack, whereas a public one is actually part of its outward facing interface. This facilitates encapsulation.
|
|
|
|
|
|
|
|
For instance, perhaps we are leveraging an S3 bucket to store some data in one of our Services. That obviously
|
|
|
|
shouldn't be of interest to consumers of our Stack. So, we split things accordingly:
|
|
|
|
|
|
|
|
services:
|
|
|
|
private:
|
|
|
|
aws/s3:
|
|
|
|
bucket: images
|
|
|
|
public:
|
|
|
|
nginx/nginx:
|
|
|
|
data: s3
|
|
|
|
port: 80
|
|
|
|
|
|
|
|
In this example, S3 buckets are volumes; we create a private one and mount it in our public Nginx container.
|
|
|
|
|
2016-11-04 19:23:59 +01:00
|
|
|
### Constructor Arguments
|
|
|
|
|
|
|
|
TODO(joe): describe the argument binding and verification process.
|
2016-11-02 21:04:27 +01:00
|
|
|
|
2016-11-04 19:23:59 +01:00
|
|
|
## Nested Stacks
|
2016-11-02 21:04:27 +01:00
|
|
|
|
|
|
|
Another feature that comes in handy sometimes is the ability to create nested Stacks:
|
|
|
|
|
|
|
|
stacks:
|
|
|
|
|
|
|
|
Each nested Stack is very much like the Stack defined by any given Mufile, except that it is scoped, much like a
|
2016-11-04 00:11:28 +01:00
|
|
|
nested/inner class in object-oriented languages. Doing this lets you subclass and/or multi-instance a single Stack as
|
|
|
|
multiple Services inside of the same Mufile. For example, consider a container that will be multi-instanced:
|
|
|
|
|
|
|
|
stacks:
|
|
|
|
private:
|
2016-11-17 03:08:32 +01:00
|
|
|
common:
|
2016-11-04 00:11:28 +01:00
|
|
|
type: mu/container
|
|
|
|
image: acmecorp/great
|
|
|
|
env:
|
|
|
|
NAME: {{.meta.name}}-cluster
|
|
|
|
DATA: false
|
|
|
|
MASTER: false
|
|
|
|
HTTP: false
|
2016-11-02 21:04:27 +01:00
|
|
|
|
2016-11-04 00:11:28 +01:00
|
|
|
Now that we've defined `common`, we can go ahead and create it, without needing to expose the Stack to clients:
|
2016-11-02 21:04:27 +01:00
|
|
|
|
2016-11-04 00:11:28 +01:00
|
|
|
services:
|
|
|
|
private:
|
2016-11-17 03:08:32 +01:00
|
|
|
data:
|
2016-11-04 00:11:28 +01:00
|
|
|
type: common
|
|
|
|
env:
|
|
|
|
DATA: true
|
|
|
|
public:
|
2016-11-17 03:08:32 +01:00
|
|
|
master:
|
2016-11-04 00:11:28 +01:00
|
|
|
type: common
|
|
|
|
env:
|
|
|
|
MASTER: true
|
2016-11-17 03:08:32 +01:00
|
|
|
worker:
|
2016-11-04 00:11:28 +01:00
|
|
|
type: common
|
|
|
|
env:
|
|
|
|
HTTP: true
|
|
|
|
|
|
|
|
All of these three Services -- one private and two public -- leverage the same `acmecorp/great` container image,
|
|
|
|
and each one defines the same four set of environment variables. Each instance, however, overrides a different
|
|
|
|
environment variable default value, to differentiate the roles as per the container's semantics.
|
|
|
|
|
|
|
|
Different scenarios call for subclassing versus composition, and the Mu system supports both in a first class way.
|
2016-11-02 21:04:27 +01:00
|
|
|
|
|
|
|
TODO(joe): we need to decide whether you can export public Stacks for public consumption. At this point, my stance is
|
2016-11-04 02:29:23 +01:00
|
|
|
that you must create an entirely different Stack to do that. This keeps things simple for the time being.
|
2016-11-02 21:04:27 +01:00
|
|
|
|
2016-11-04 19:23:59 +01:00
|
|
|
## Target-Specific Metadata
|
2016-11-03 23:12:22 +01:00
|
|
|
|
|
|
|
Although for the most part, metadata strives to be cloud provider-agnostic, there are two ways in which it may not be.
|
|
|
|
First, some Stack types are available only on a particular cloud, like `aws/s3/bucket` (and any that transitively
|
|
|
|
reference this). Attempting to cross-deploy Stacks referencing such things will fail at compile-time, for obvious
|
|
|
|
reasons. Second, some metadata can be cloud provider-specific. For example, even if we are creating a Service that is
|
|
|
|
logically independent from any given cloud, like a Load Balancer, we may wish to provider cloud-specific settings.
|
|
|
|
Those appear in a special metadata section and are marked in such a way that erroneous deployments fail at compile-time.
|
|
|
|
|
|
|
|
More details on target-specific Stacks and metadata settings are provided below in the relevant sections.
|
|
|
|
|