Refactor the Architecture doc, and start a Metadata specification
This just contains primarily scaffolding, however lays out a general direction and table of contents for the metadata specification document (marapongo/mu#1).
This commit is contained in:
parent
48e178d0d2
commit
0232dd3a08
|
@ -3,6 +3,10 @@
|
|||
Mu introduces abstractions that help developers create, maintain, and reason about microservice-based architectures.
|
||||
More than that, Mu facilitates sharing and reuse of these architectures between developers and teams.
|
||||
|
||||
This document describes the overall architecture for the system, including this translation process.
|
||||
|
||||
## Concepts
|
||||
|
||||
The three top-level architectural abstractions in Mu are:
|
||||
|
||||
* **Stack**: A static blueprint, or "type", that describes a topology of cloud resources.
|
||||
|
@ -20,119 +24,28 @@ one of the toolchain's most important jobs is faithfully mapping these abstracti
|
|||
abstractions, and vice versa. Much of Mu's ability to deliver on its promise of better productivity, sharing, and reuse
|
||||
relies on its ability to robustly and intuitively perform these translations.
|
||||
|
||||
This document describes the overall architecture for the system, including this translation process.
|
||||
## Toolchain
|
||||
|
||||
## Platform
|
||||
|
||||
A Stack is comprised of many Services, instantiated from other Stacks. This of course eventually bottoms out on a core
|
||||
set of "primitive" constructs, offered by the Mu platform. In addition to those primitives, a rich ecosystem exists on
|
||||
top, so that, for the most part, end-user Stacks seldom need to interact directly even with the primitives.
|
||||
|
||||
The primitive types available include:
|
||||
|
||||
* `mu/container`: A Docker container wrapped in Mu metadata.
|
||||
* `mu/gateway`: An API gateway and/or load balancer, multiplexing requests onto multiple target Services.
|
||||
* `mu/func`: A single Function ordinarily used for serverless/stateless scenarios.
|
||||
* `mu/event`: An Event that may be used to Trigger the execution of another Service (commonly a Function).
|
||||
* `mu/volume`: A volume stores data that can be mounted by another Service.
|
||||
* `mu/autoscaler`: A Service that automatically multi-instances and scales some other target Service based on policy.
|
||||
* `mu/hook`: A logical Service that has no concrete runtime manifestation other than running pre- or post-logic.
|
||||
|
||||
TODO(joe): link to exhaustive details on each of these.
|
||||
|
||||
TODO(joe): is there an extensibility model, whereby the set of primitive constructs can be extended?
|
||||
|
||||
## Metadata Specification
|
||||
|
||||
The essential artifact a developer uses to create Stacks and Services is something we call a Mufile. It is
|
||||
conventionally named `Mu.yaml`, usually checked into the Workspace, and each single file describes a Stack. Note that
|
||||
Stacks may internally contain other Stacks, however this is purely an implementation detail of how the Stack works.
|
||||
|
||||
The full Mufile format is specified in [TODO](/TODO). However, we present a brief overview here.
|
||||
|
||||
### Package Manager Metadata
|
||||
|
||||
Each Mufile begins with some standard "package manager"-like metadata, like name, version, description, and so on. As
|
||||
with most package managers, most of these elements are optional:
|
||||
|
||||
name: elk
|
||||
version: 1.0.1
|
||||
description: A fully functioning ELK stack (Elasticsearch, Logstash, Kibana).
|
||||
author: Joe Smith <joesmith@elk.com>
|
||||
website: https://github.com/joesmith/elk
|
||||
|
||||
TODO(joe): finish this section.
|
||||
|
||||
### Services
|
||||
|
||||
After that comes the section that describes what Services make up this Stack:
|
||||
|
||||
services:
|
||||
|
||||
At this point, a new concept is introduced: *visibility*. Visibility works much like your favorite programming
|
||||
language, in that a Stack may declare that any of its Services are `public` or `private`. This impacts the
|
||||
accessibility of those Services to consumers of this Stack. A private Service is merely an implementation detail of
|
||||
the Stack, whereas a public one is actually part of its outward facing interface. This facilitates encapsulation.
|
||||
|
||||
For instance, perhaps we are leveraging an S3 bucket to store some data in one of our Services. That obviously
|
||||
shouldn't be of interest to consumers of our Stack. So, we split things accordingly:
|
||||
|
||||
services:
|
||||
private:
|
||||
aws/s3:
|
||||
bucket: images
|
||||
public:
|
||||
nginx/nginx:
|
||||
data: s3
|
||||
port: 80
|
||||
|
||||
In this example, S3 buckets are volumes; we create a private one and mount it in our public Nginx container.
|
||||
|
||||
TODO(joe): more examples.
|
||||
|
||||
TODO(joe): a section on names, scoping, etc.
|
||||
|
||||
### Nested Stacks
|
||||
|
||||
Another feature that comes in handy sometimes is the ability to create nested Stacks:
|
||||
|
||||
stacks:
|
||||
|
||||
Each nested Stack is very much like the Stack defined by any given Mufile, except that it is scoped, much like a
|
||||
nested/inner class in object-oriented languages. There are two primary reasons you might do this:
|
||||
|
||||
1. Multi-instancing that Stack, to create multiple Services.
|
||||
2. Conveniently leveraging that Stack as the "type" for one or more other Services in the file.
|
||||
|
||||
TODO(joe): examples of the above.
|
||||
|
||||
TODO(joe): we need to decide whether you can export public Stacks for public consumption. At this point, my stance is
|
||||
that you must create an entirely different Stack to do that. This keeps things simple for the time being.
|
||||
|
||||
## An Illustrative Example
|
||||
|
||||
Before breaking down the implementation of these concepts, let's first look at an illustrative example.
|
||||
|
||||
TODO(joe): a more comprehensive example (like Vote50) that illustrates more of these concepts working in harmony.
|
||||
|
||||
## Implementation
|
||||
|
||||
In this section, we will breakdown the primary aspects of the implementation:
|
||||
|
||||
1. Translation
|
||||
|
||||
TODO(joe): deployment, ongoing interactions, management, etc.
|
||||
In this section, we will look at the toolchain that powers the overall Mu architecture.
|
||||
|
||||
### Translation
|
||||
|
||||
Now let us look at how a Mufile turns into a ready-to-run package.
|
||||
Now let us look at how a Mufile turns into a ready-to-run package. This will not describe the precise translation and
|
||||
set of targets (that is available in the [metadata specification document](metadata.md)), instead focusing on the
|
||||
various bits of data and code involved, the plugin architecture, and overall translation process flow.
|
||||
|
||||
TODO(joe): write this section; just cover the overall structure and plugin model, not the details of each target.
|
||||
|
||||
In the future, we envision that Mufiles might be generated from code, for an even more seamless developer experience.
|
||||
This is very powerful when envisioning serverless architectures, where Stacks, Services, Functions, and Triggers can be
|
||||
expressed all in a single file, and managed alongside the code and libraries they depend upon. See marapongo/mu#xxx
|
||||
for a work item tracking this. For now, and to simplify this doc, we will ignore this possibility.
|
||||
|
||||
TODO(joe): write this section; just cover the overall structure and plugin model, not the details of each target.
|
||||
### Command Line Interface
|
||||
|
||||
TODO(joe): create a design specification that covers the detailed translation for each target; link to it.
|
||||
TODO(joe): deployment, ongoing interactions, management, etc.
|
||||
|
||||
## Services
|
||||
|
||||
TODO(joe): describe package manager, artifact repository, the relationship between them, etc.
|
||||
|
||||
|
|
163
docs/metadata.md
Normal file
163
docs/metadata.md
Normal file
|
@ -0,0 +1,163 @@
|
|||
# Mu Metadata
|
||||
|
||||
This document contains a formal description of Mu's metadata, in addition to the translation process for all current
|
||||
targets.
|
||||
|
||||
## Specification
|
||||
|
||||
This section includes the specification for Mu's metadata plus its core platform primitives.
|
||||
|
||||
### Platform
|
||||
|
||||
A Stack is comprised of many Services, instantiated from other Stacks. This of course eventually bottoms out on a core
|
||||
set of "primitive" constructs, offered by the Mu platform. In addition to those primitives, a rich ecosystem exists on
|
||||
top, so that, for the most part, end-user Stacks seldom need to interact directly even with the primitives.
|
||||
|
||||
The primitive types available include:
|
||||
|
||||
* `mu/container`: A Docker container wrapped in Mu metadata.
|
||||
* `mu/gateway`: An API gateway and/or load balancer, multiplexing requests onto multiple target Services.
|
||||
* `mu/func`: A single Function ordinarily used for serverless/stateless scenarios.
|
||||
* `mu/event`: An Event that may be used to Trigger the execution of another Service (commonly a Function).
|
||||
* `mu/volume`: A volume stores data that can be mounted by another Service.
|
||||
* `mu/autoscaler`: A Service that automatically multi-instances and scales some other target Service based on policy.
|
||||
* `mu/hook`: A logical Service that has no concrete runtime manifestation other than running pre- or post-logic.
|
||||
|
||||
TODO(joe): link to exhaustive details on each of these.
|
||||
|
||||
TODO(joe): is there an extensibility model, whereby the set of primitive constructs can be extended?
|
||||
|
||||
### Metadata
|
||||
|
||||
The essential artifact a developer uses to create Stacks and Services is something we call a Mufile. It is
|
||||
conventionally named `Mu.yaml`, usually checked into the Workspace, and each single file describes a Stack. Note that
|
||||
Stacks may internally contain other Stacks, however this is purely an implementation detail of how the Stack works.
|
||||
|
||||
#### Package Managament
|
||||
|
||||
Each Mufile begins with some standard "package manager"-like metadata, like name, version, description, and so on. As
|
||||
with most package managers, most of these elements are optional:
|
||||
|
||||
name: elk
|
||||
version: 1.0.1
|
||||
description: A fully functioning ELK stack (Elasticsearch, Logstash, Kibana).
|
||||
author: Joe Smith <joesmith@elk.com>
|
||||
website: https://github.com/joesmith/elk
|
||||
|
||||
TODO(joe): finish this section.
|
||||
|
||||
#### Services
|
||||
|
||||
After that comes the section that describes what Services make up this Stack:
|
||||
|
||||
services:
|
||||
|
||||
At this point, a new concept is introduced: *visibility*. Visibility works much like your favorite programming
|
||||
language, in that a Stack may declare that any of its Services are `public` or `private`. This impacts the
|
||||
accessibility of those Services to consumers of this Stack. A private Service is merely an implementation detail of
|
||||
the Stack, whereas a public one is actually part of its outward facing interface. This facilitates encapsulation.
|
||||
|
||||
For instance, perhaps we are leveraging an S3 bucket to store some data in one of our Services. That obviously
|
||||
shouldn't be of interest to consumers of our Stack. So, we split things accordingly:
|
||||
|
||||
services:
|
||||
private:
|
||||
aws/s3:
|
||||
bucket: images
|
||||
public:
|
||||
nginx/nginx:
|
||||
data: s3
|
||||
port: 80
|
||||
|
||||
In this example, S3 buckets are volumes; we create a private one and mount it in our public Nginx container.
|
||||
|
||||
TODO(joe): more examples.
|
||||
|
||||
TODO(joe): a section on names, scoping, etc.
|
||||
|
||||
#### Nested Stacks
|
||||
|
||||
Another feature that comes in handy sometimes is the ability to create nested Stacks:
|
||||
|
||||
stacks:
|
||||
|
||||
Each nested Stack is very much like the Stack defined by any given Mufile, except that it is scoped, much like a
|
||||
nested/inner class in object-oriented languages. There are two primary reasons you might do this:
|
||||
|
||||
1. Multi-instancing that Stack, to create multiple Services.
|
||||
2. Conveniently leveraging that Stack as the "type" for one or more other Services in the file.
|
||||
|
||||
TODO(joe): examples of the above.
|
||||
|
||||
TODO(joe): we need to decide whether you can export public Stacks for public consumption. At this point, my stance is
|
||||
that you must create an entirely different Stack to do that. This keeps things simple for the time being.
|
||||
|
||||
### An Illustrative Example
|
||||
|
||||
Before breaking down the implementation of these concepts, let's first look at an illustrative example.
|
||||
|
||||
TODO(joe): a more comprehensive example (like Vote50) that illustrates more of these concepts working in harmony.
|
||||
|
||||
## Targets
|
||||
|
||||
This section contains a precise description of the mapping process from Mu metadata to various cloud targets. Note
|
||||
that there are two dimensions to this process:
|
||||
|
||||
* The first dimension is the system used for container orchestration, or what we will call, Containers-as-a-Service
|
||||
(CaaS). Examples of this include AWS ECS, Docker Swarm, Kubernetes, and even plain old VM-based scheduling.
|
||||
|
||||
* The second dimension is the system used for hosting the VMs fueling the CaaS, which we will call
|
||||
Infrastructure-as-a-Service (IaaS). Examples of this include AWS, Google Cloud Platform (GCP), Azure, and even VM
|
||||
fabrics for on-premise installations, like VMWare VSphere. Note that often IaaS goes beyond simply having VMs as
|
||||
resources and can include hosted offerings such as blob storage, load balancers, domain name configurations, etc.
|
||||
|
||||
Not all combinations of CaaS and IaaS fall out naturally, although it is a goal of the system to target them
|
||||
orthogonally such that the incremental cost of creating new pairings is as low as possible (minimizing combinatorics).
|
||||
Some combinations are also clearly nonsense, such as using AWS ECS as the CaaS and GCP as the IaaS.
|
||||
|
||||
For reference purposes, here is a compatibility matrix for what we envision supporting. Each column with an `X` is
|
||||
described in this document already; each column with an `-` is planned, but not yet described; and `n/a` is nonsense:
|
||||
|
||||
| | AWS | GCP | Azure | VMWare |
|
||||
+---------------+-----------+-----------+-----------+-----------|
|
||||
| VM | X | - | - | - |
|
||||
| Docker Swarm | - | - | - | - |
|
||||
| Kubernetes | - | - | - | - |
|
||||
| Mesos | - | - | - | - |
|
||||
| ECS | X | n/a | n/a | n/a |
|
||||
| GKE | n/a | - | n/a | n/a |
|
||||
| ACS | n/a | n/a | - | n/a |
|
||||
|
||||
In all cases, the native metadata formats for the CaaS and IaaS provider in question is supported; for example, ECS on
|
||||
AWS will leverage CloudFormation as the target metadata. In certain cases, we also support Terraform outputs.
|
||||
|
||||
### CaaS Targets
|
||||
|
||||
#### VM
|
||||
|
||||
#### Docker Swarm
|
||||
|
||||
#### Kubernetes
|
||||
|
||||
#### AWS EC2 Container Service (ECS)
|
||||
|
||||
#### Google Container Engine (GKE)
|
||||
|
||||
#### Azure Container Service (ACS)
|
||||
|
||||
### IaaS Targets
|
||||
|
||||
#### Amazon Web Services (AWS)
|
||||
|
||||
#### Google Cloud Platform (GCP)
|
||||
|
||||
#### Microsoft Azure
|
||||
|
||||
#### VMWare
|
||||
|
||||
### Specific Combinations
|
||||
|
||||
### Terraform
|
||||
|
||||
TODO(joe): describe what Terraform may be used to target and how it works.
|
||||
|
Loading…
Reference in a new issue