7.5 KiB
Mu Metadata
This document contains a formal description of Mu's metadata, in addition to the translation process for all current targets.
Specification
This section includes the specification for Mu's metadata plus its core platform primitives.
Platform
A Stack is comprised of many Services, instantiated from other Stacks. This of course eventually bottoms out on a core set of "primitive" constructs, offered by the Mu platform. In addition to those primitives, a rich ecosystem exists on top, so that, for the most part, end-user Stacks seldom need to interact directly even with the primitives.
The primitive types available include:
mu/container: A Docker container wrapped in Mu metadata.mu/gateway: An API gateway and/or load balancer, multiplexing requests onto multiple target Services.mu/func: A single Function ordinarily used for serverless/stateless scenarios.mu/event: An Event that may be used to Trigger the execution of another Service (commonly a Function).mu/volume: A volume stores data that can be mounted by another Service.mu/autoscaler: A Service that automatically multi-instances and scales some other target Service based on policy.mu/hook: A logical Service that has no concrete runtime manifestation other than running pre- or post-logic.
TODO(joe): link to exhaustive details on each of these.
TODO(joe): is there an extensibility model, whereby the set of primitive constructs can be extended?
Metadata
The essential artifact a developer uses to create Stacks and Services is something we call a Mufile. It is
conventionally named Mu.yaml, usually checked into the Workspace, and each single file describes a Stack. Note that
Stacks may internally contain other Stacks, however this is purely an implementation detail of how the Stack works.
Package Managament
Each Mufile begins with some standard "package manager"-like metadata, like name, version, description, and so on. As with most package managers, most of these elements are optional:
name: elk
version: 1.0.1
description: A fully functioning ELK stack (Elasticsearch, Logstash, Kibana).
author: Joe Smith <joesmith@elk.com>
website: https://github.com/joesmith/elk
TODO(joe): finish this section.
Services
After that comes the section that describes what Services make up this Stack:
services:
At this point, a new concept is introduced: visibility. Visibility works much like your favorite programming
language, in that a Stack may declare that any of its Services are public or private. This impacts the
accessibility of those Services to consumers of this Stack. A private Service is merely an implementation detail of
the Stack, whereas a public one is actually part of its outward facing interface. This facilitates encapsulation.
For instance, perhaps we are leveraging an S3 bucket to store some data in one of our Services. That obviously shouldn't be of interest to consumers of our Stack. So, we split things accordingly:
services:
private:
aws/s3:
bucket: images
public:
nginx/nginx:
data: s3
port: 80
In this example, S3 buckets are volumes; we create a private one and mount it in our public Nginx container.
TODO(joe): more examples.
TODO(joe): a section on names, scoping, etc.
Nested Stacks
Another feature that comes in handy sometimes is the ability to create nested Stacks:
stacks:
Each nested Stack is very much like the Stack defined by any given Mufile, except that it is scoped, much like a nested/inner class in object-oriented languages. There are two primary reasons you might do this:
- Multi-instancing that Stack, to create multiple Services.
- Conveniently leveraging that Stack as the "type" for one or more other Services in the file.
TODO(joe): examples of the above.
TODO(joe): we need to decide whether you can export public Stacks for public consumption. At this point, my stance is that you must create an entirely different Stack to do that. This keeps things simple for the time being.
An Illustrative Example
Before breaking down the implementation of these concepts, let's first look at an illustrative example.
TODO(joe): a more comprehensive example (like Vote50) that illustrates more of these concepts working in harmony.
Targets
This section contains a precise description of the mapping process from Mu metadata to various cloud targets. Note that there are two dimensions to this process:
-
The first dimension is the system used for hosting the cluster environment, which we will call Infrastructure-as-a-Service (IaaS). Examples of this include AWS, Google Cloud Platform (GCP), Azure, and even VM fabrics for on-premise installations, like VMWare VSphere. Note that often IaaS goes beyond simply having VMs as resources and can include hosted offerings such as blob storage, load balancers, domain name configurations, etc.
-
The second dimension is the system used for container orchestration, or what we will call, Containers-as-a-Service (CaaS). Examples of this include AWS ECS, Docker Swarm, and Kubernetes. Note that the system can handle the siituation where there is no container orchestration framework available, in which case raw VMs are utilized.
Not all combinations of IaaS and CaaS fall out naturally, although it is a goal of the system to target them orthogonally such that the incremental cost of creating new pairings is as low as possible (minimizing combinatorics). Some combinations are also clearly nonsense, such as AWS as your IaaS and GKE as your CaaS.
For reference, here is a compatibility matrix. Each cell with an X is described in this document already; each cell
with an - is planned, but not yet described; and blank entries are unsupported nonsense combinations:
| AWS | GCP | Azure | VMWare | |
|---|---|---|---|---|
| none (VMs) | X | - | - | - |
| Docker Swarm | - | - | - | - |
| Kubernetes | - | - | - | - |
| Mesos | - | - | - | - |
| ECS | X | |||
| GKE | - | |||
| ACS | - |
In all cases, the native metadata formats for the IaaS and CaaS provider in question is supported; for example, ECS on AWS will leverage CloudFormation as the target metadata. In certain cases, we also support Terraform outputs.
Refer to marapongo/mu#2 for an up-to-date prioritization of platforms.
IaaS Targets
This section describes the translation for various IaaS targets. Recall that deploying to an IaaS without any CaaS is a supported scenario, so each of these descriptions is "self-contained." In the case that a CaaS is utilized, that process -- described below -- can override certain decisions made in the IaaS translation process. For instance, rather than leveraging a VM per Docker Container, the CaaS translation will choose to target an orchestration layer.
Amazon Web Services (AWS)
Google Cloud Platform (GCP)
Microsoft Azure
VMWare
CaaS Targets
VM
Docker Swarm
Kubernetes
Mesos
AWS EC2 Container Service (ECS)
Google Container Engine (GKE)
Azure Container Service (ACS)
Terraform
TODO(joe): describe what Terraform may be used to target and how it works.