maxmustermann/pulumi
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

Refactor the Architecture doc, and start a Metadata specification

Browse source 
This just contains primarily scaffolding, however lays out a general direction
and table of contents for the metadata specification document (marapongo/mu#1).
This commit is contained in:
joeduffy 2016-11-02 13:04:27 -07:00
parent 48e178d0d2
commit 0232dd3a08
2 changed files with 180 additions and 104 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

121
docs/architecture.md
View file

@ -3,6 +3,10 @@
Mu introduces abstractions that help developers create, maintain, and reason about microservice-based architectures.
More than that, Mu facilitates sharing and reuse of these architectures between developers and teams.
This document describes the overall architecture for the system, including this translation process.
## Concepts
The three top-level architectural abstractions in Mu are:
* **Stack**: A static blueprint, or "type", that describes a topology of cloud resources.
@ -20,119 +24,28 @@ one of the toolchain's most important jobs is faithfully mapping these abstracti
abstractions, and vice versa. Much of Mu's ability to deliver on its promise of better productivity, sharing, and reuse
relies on its ability to robustly and intuitively perform these translations.
This document describes the overall architecture for the system, including this translation process.
## Toolchain
## Platform
A Stack is comprised of many Services, instantiated from other Stacks. This of course eventually bottoms out on a core
set of "primitive" constructs, offered by the Mu platform. In addition to those primitives, a rich ecosystem exists on
top, so that, for the most part, end-user Stacks seldom need to interact directly even with the primitives.
The primitive types available include:
* `mu/container`: A Docker container wrapped in Mu metadata.
* `mu/gateway`: An API gateway and/or load balancer, multiplexing requests onto multiple target Services.
* `mu/func`: A single Function ordinarily used for serverless/stateless scenarios.
* `mu/event`: An Event that may be used to Trigger the execution of another Service (commonly a Function).
* `mu/volume`: A volume stores data that can be mounted by another Service.
* `mu/autoscaler`: A Service that automatically multi-instances and scales some other target Service based on policy.
* `mu/hook`: A logical Service that has no concrete runtime manifestation other than running pre- or post-logic.
TODO(joe): link to exhaustive details on each of these.
TODO(joe): is there an extensibility model, whereby the set of primitive constructs can be extended?
## Metadata Specification
The essential artifact a developer uses to create Stacks and Services is something we call a Mufile. It is
conventionally named `Mu.yaml`, usually checked into the Workspace, and each single file describes a Stack. Note that
Stacks may internally contain other Stacks, however this is purely an implementation detail of how the Stack works.
The full Mufile format is specified in [TODO](/TODO). However, we present a brief overview here.
### Package Manager Metadata
Each Mufile begins with some standard "package manager"-like metadata, like name, version, description, and so on. As
with most package managers, most of these elements are optional:
name: elk
version: 1.0.1
description: A fully functioning ELK stack (Elasticsearch, Logstash, Kibana).
author: Joe Smith <joesmith@elk.com>
website: https://github.com/joesmith/elk
TODO(joe): finish this section.
### Services
After that comes the section that describes what Services make up this Stack:
services:
At this point, a new concept is introduced: *visibility*. Visibility works much like your favorite programming
language, in that a Stack may declare that any of its Services are `public` or `private`. This impacts the
accessibility of those Services to consumers of this Stack. A private Service is merely an implementation detail of
the Stack, whereas a public one is actually part of its outward facing interface. This facilitates encapsulation.
For instance, perhaps we are leveraging an S3 bucket to store some data in one of our Services. That obviously
shouldn't be of interest to consumers of our Stack. So, we split things accordingly:
services:
private:
aws/s3:
bucket: images
public:
nginx/nginx:
data: s3
port: 80
In this example, S3 buckets are volumes; we create a private one and mount it in our public Nginx container.
TODO(joe): more examples.
TODO(joe): a section on names, scoping, etc.
### Nested Stacks
Another feature that comes in handy sometimes is the ability to create nested Stacks:
stacks:
Each nested Stack is very much like the Stack defined by any given Mufile, except that it is scoped, much like a
nested/inner class in object-oriented languages. There are two primary reasons you might do this:
1. Multi-instancing that Stack, to create multiple Services.
2. Conveniently leveraging that Stack as the "type" for one or more other Services in the file.
TODO(joe): examples of the above.
TODO(joe): we need to decide whether you can export public Stacks for public consumption. At this point, my stance is
that you must create an entirely different Stack to do that. This keeps things simple for the time being.
## An Illustrative Example
Before breaking down the implementation of these concepts, let's first look at an illustrative example.
TODO(joe): a more comprehensive example (like Vote50) that illustrates more of these concepts working in harmony.
## Implementation
In this section, we will breakdown the primary aspects of the implementation:
1. Translation
TODO(joe): deployment, ongoing interactions, management, etc.
In this section, we will look at the toolchain that powers the overall Mu architecture.
### Translation
Now let us look at how a Mufile turns into a ready-to-run package.
Now let us look at how a Mufile turns into a ready-to-run package. This will not describe the precise translation and
set of targets (that is available in the [metadata specification document](metadata.md)), instead focusing on the
various bits of data and code involved, the plugin architecture, and overall translation process flow.
TODO(joe): write this section; just cover the overall structure and plugin model, not the details of each target.
In the future, we envision that Mufiles might be generated from code, for an even more seamless developer experience.
This is very powerful when envisioning serverless architectures, where Stacks, Services, Functions, and Triggers can be
expressed all in a single file, and managed alongside the code and libraries they depend upon. See marapongo/mu#xxx
for a work item tracking this. For now, and to simplify this doc, we will ignore this possibility.
TODO(joe): write this section; just cover the overall structure and plugin model, not the details of each target.
### Command Line Interface
TODO(joe): create a design specification that covers the detailed translation for each target; link to it.
TODO(joe): deployment, ongoing interactions, management, etc.
## Services
TODO(joe): describe package manager, artifact repository, the relationship between them, etc.

163
docs/metadata.md Normal file
View file

@ -0,0 +1,163 @@
# Mu Metadata
This document contains a formal description of Mu's metadata, in addition to the translation process for all current
targets.
## Specification
This section includes the specification for Mu's metadata plus its core platform primitives.
### Platform
A Stack is comprised of many Services, instantiated from other Stacks. This of course eventually bottoms out on a core
set of "primitive" constructs, offered by the Mu platform. In addition to those primitives, a rich ecosystem exists on
top, so that, for the most part, end-user Stacks seldom need to interact directly even with the primitives.
The primitive types available include:
* `mu/container`: A Docker container wrapped in Mu metadata.
* `mu/gateway`: An API gateway and/or load balancer, multiplexing requests onto multiple target Services.
* `mu/func`: A single Function ordinarily used for serverless/stateless scenarios.
* `mu/event`: An Event that may be used to Trigger the execution of another Service (commonly a Function).
* `mu/volume`: A volume stores data that can be mounted by another Service.
* `mu/autoscaler`: A Service that automatically multi-instances and scales some other target Service based on policy.
* `mu/hook`: A logical Service that has no concrete runtime manifestation other than running pre- or post-logic.
TODO(joe): link to exhaustive details on each of these.
TODO(joe): is there an extensibility model, whereby the set of primitive constructs can be extended?
### Metadata
The essential artifact a developer uses to create Stacks and Services is something we call a Mufile. It is
conventionally named `Mu.yaml`, usually checked into the Workspace, and each single file describes a Stack. Note that
Stacks may internally contain other Stacks, however this is purely an implementation detail of how the Stack works.
#### Package Managament
Each Mufile begins with some standard "package manager"-like metadata, like name, version, description, and so on. As
with most package managers, most of these elements are optional:
name: elk
version: 1.0.1
description: A fully functioning ELK stack (Elasticsearch, Logstash, Kibana).
author: Joe Smith <joesmith@elk.com>
website: https://github.com/joesmith/elk
TODO(joe): finish this section.
#### Services
After that comes the section that describes what Services make up this Stack:
services:
At this point, a new concept is introduced: *visibility*. Visibility works much like your favorite programming
language, in that a Stack may declare that any of its Services are `public` or `private`. This impacts the
accessibility of those Services to consumers of this Stack. A private Service is merely an implementation detail of
the Stack, whereas a public one is actually part of its outward facing interface. This facilitates encapsulation.
For instance, perhaps we are leveraging an S3 bucket to store some data in one of our Services. That obviously
shouldn't be of interest to consumers of our Stack. So, we split things accordingly:
services:
private:
aws/s3:
bucket: images
public:
nginx/nginx:
data: s3
port: 80
In this example, S3 buckets are volumes; we create a private one and mount it in our public Nginx container.
TODO(joe): more examples.
TODO(joe): a section on names, scoping, etc.
#### Nested Stacks
Another feature that comes in handy sometimes is the ability to create nested Stacks:
stacks:
Each nested Stack is very much like the Stack defined by any given Mufile, except that it is scoped, much like a
nested/inner class in object-oriented languages. There are two primary reasons you might do this:
1. Multi-instancing that Stack, to create multiple Services.
2. Conveniently leveraging that Stack as the "type" for one or more other Services in the file.
TODO(joe): examples of the above.
TODO(joe): we need to decide whether you can export public Stacks for public consumption. At this point, my stance is
that you must create an entirely different Stack to do that. This keeps things simple for the time being.
### An Illustrative Example
Before breaking down the implementation of these concepts, let's first look at an illustrative example.
TODO(joe): a more comprehensive example (like Vote50) that illustrates more of these concepts working in harmony.
## Targets
This section contains a precise description of the mapping process from Mu metadata to various cloud targets. Note
that there are two dimensions to this process:
* The first dimension is the system used for container orchestration, or what we will call, Containers-as-a-Service
(CaaS). Examples of this include AWS ECS, Docker Swarm, Kubernetes, and even plain old VM-based scheduling.
* The second dimension is the system used for hosting the VMs fueling the CaaS, which we will call
Infrastructure-as-a-Service (IaaS). Examples of this include AWS, Google Cloud Platform (GCP), Azure, and even VM
fabrics for on-premise installations, like VMWare VSphere. Note that often IaaS goes beyond simply having VMs as
resources and can include hosted offerings such as blob storage, load balancers, domain name configurations, etc.
Not all combinations of CaaS and IaaS fall out naturally, although it is a goal of the system to target them
orthogonally such that the incremental cost of creating new pairings is as low as possible (minimizing combinatorics).
Some combinations are also clearly nonsense, such as using AWS ECS as the CaaS and GCP as the IaaS.
For reference purposes, here is a compatibility matrix for what we envision supporting. Each column with an `X` is
described in this document already; each column with an `-` is planned, but not yet described; and `n/a` is nonsense:
| | AWS | GCP | Azure | VMWare |
+---------------+-----------+-----------+-----------+-----------|
| VM | X | - | - | - |
| Docker Swarm | - | - | - | - |
| Kubernetes | - | - | - | - |
| Mesos | - | - | - | - |
| ECS | X | n/a | n/a | n/a |
| GKE | n/a | - | n/a | n/a |
| ACS | n/a | n/a | - | n/a |
In all cases, the native metadata formats for the CaaS and IaaS provider in question is supported; for example, ECS on
AWS will leverage CloudFormation as the target metadata. In certain cases, we also support Terraform outputs.
### CaaS Targets
#### VM
#### Docker Swarm
#### Kubernetes
#### AWS EC2 Container Service (ECS)
#### Google Container Engine (GKE)
#### Azure Container Service (ACS)
### IaaS Targets
#### Amazon Web Services (AWS)
#### Google Cloud Platform (GCP)
#### Microsoft Azure
#### VMWare
### Specific Combinations
### Terraform
TODO(joe): describe what Terraform may be used to target and how it works.