This change eliminates the special type mu/extension in favor of extensible intrinsic types. This subsumes the previous functionality while also fixing a number of warts with the old model. In particular, the old mu/extension approach deferred property binding until very late in the compiler. In fact, too late. The backend provider for an extension simply received an untyped bag of stuff, which it then had to deal with. Unfortunately, some operations in the binder are inaccessible at this point because doing so would cause a cycle. Furthermore, some pertinent information is gone at this point, like the scopes and symtables. The canonical example where we need this is binding services names to the services themselves; e.g., the AWS CloudFormation "DependsOn" property should resolve to the actual service names, not the string values. In the limit, this requires full binding information. There were a few solutions I considered, including ones that've required less code motion, however this one feels the most elegant. Now we permit types to be marked as "intrinsic." Binding to these names is done exactly as ordinary name binding, unlike the special mu/extension provider name. In fact, just about everything except code-generation for these types is the same as ordinary types. This is perfect for the use case at hand, which is binding properties. After this change, for example, "DependsOn" is expanded to real service names precisely as we need. As part of this change, I added support for three new basic schema types: * ast.StringList ("string[]"): a list of strings. * ast.StringMap ("map[string]any"): a map of strings to anys. * ast.ServiceList ("service[]"): a list of service references. Obviously we need to revisit this and add a more complete set. This work is already tracked by marapongo/mu#9. At the end of the day, it's likely I will replace all hard-coded predefined types with intrinsic types, for similar reasons to the above.
300 lines
16 KiB
Markdown
300 lines
16 KiB
Markdown
# Mu Compilation Targets
|
|
|
|
This document describes how Mu metadata is compiled and deployed to various cloud targets. Please refer to [the
|
|
companion metadata specification](metadata.md) to understand the source input in more detail.
|
|
|
|
There are two primary dimensions to any given target:
|
|
|
|
* The first dimension is the system used for hosting the cluster environment, which we will call
|
|
Infrastructure-as-a-Service (IaaS). Examples of this include AWS, Google Cloud Platform (GCP), Azure, and even VM
|
|
fabrics for on-premise installations, like VMWare VSphere. Note that often IaaS goes beyond simply having VMs as
|
|
resources and can include hosted offerings such as blob storage, load balancers, domain name configurations, etc.
|
|
|
|
* The second dimension is the system used for container orchestration, or what we will call, Containers-as-a-Service
|
|
(CaaS). Examples of this include AWS ECS, Docker Swarm, and Kubernetes. Note that the system can handle the
|
|
siituation where there is no container orchestration framework available, in which case raw VMs are utilized.
|
|
|
|
Not all combinations of IaaS and CaaS fall out naturally, although it is a goal of the system to target them
|
|
orthogonally such that the incremental cost of creating new pairings is as low as possible (minimizing combinatorics).
|
|
Some combinations are also clearly nonsense, such as AWS as your IaaS and GKE as your CaaS.
|
|
|
|
For reference, here is a compatibility matrix. Each cell with an `X` is described in this document already; each cell
|
|
with an `-` is planned, but not yet described; and blank entries are unsupported nonsense combinations:
|
|
|
|
| | AWS | GCP | Azure | VMWare |
|
|
| ------------- | --------- | --------- | --------- | --------- |
|
|
| none (VMs) | X | - | - | - |
|
|
| Docker Swarm | - | - | - | - |
|
|
| Kubernetes | - | - | - | - |
|
|
| Mesos | - | - | - | - |
|
|
| ECS | X | | | |
|
|
| GKE | | - | | |
|
|
| ACS | | | - | |
|
|
|
|
TODO(joe): describe the "local" cases, e.g. none(?), Docker, VirtualBox, HyperV, etc.
|
|
|
|
In all cases, the native metadata formats for the IaaS and CaaS provider in question is supported; for example, ECS on
|
|
AWS will leverage CloudFormation as the target metadata. In certain cases, we also support Terraform outputs.
|
|
|
|
Refer to [marapongo/mu#2](https://github.com/marapongo/mu/issues/2) for an up-to-date prioritization of platforms.
|
|
|
|
## Clusters
|
|
|
|
A Stack is deployed to a Cluster. Any given Cluster is a fixed combination of IaaS and CaaS provider. Developers may
|
|
choose to manage Clusters and multiplex many Stacks onto any given Cluster, or they may choose to simply deploy a
|
|
Cluster per Stack. The latter is of course easier, but may potentially incur more waste than the former. Furthermore,
|
|
it will likely take more time to provision and modify entire Clusters than just the Stacks running within them.
|
|
|
|
Because creating and managing Clusters is a discrete step, the translation process will articulate them independently.
|
|
The tools make both the complex and simple workflows possible.
|
|
|
|
## Commonalities Among Targets
|
|
|
|
There are some common principles applied, no matter the target, which are worth calling out:
|
|
|
|
* DNS is the primary means of service discovery.
|
|
* TODO(joe): more...
|
|
|
|
## IaaS Targets
|
|
|
|
This section describes the translation for various IaaS targets. Recall that deploying to an IaaS *without* any CaaS is
|
|
a supported scenario, so each of these descriptions is "self-contained." In the case that a CaaS is utilized, that
|
|
process -- described below -- can override certain decisions made in the IaaS translation process. For instance, rather
|
|
than leveraging a VM per Docker Container, the CaaS translation will choose to target an orchestration layer.
|
|
|
|
### Amazon Web Services (AWS)
|
|
|
|
The output of a transformation is one or more AWS CloudFormation templates.
|
|
|
|
#### Clusters
|
|
|
|
Each Cluster is given a standard set of resources. If multiple Stacks are deployed into a shared Cluster, then those
|
|
Stacks will share all of these resources. Otherwise, each Stack is given a dedicated set of them just for itself.
|
|
|
|
TODO(joe): compare with Convox Racks: https://convox.com/docs/rack.
|
|
|
|
##### Configuration
|
|
|
|
By default, all machines are placed into the XXX region and are given a size of YYY. The choice of region may be
|
|
specified at provisioning time (TODO(joe): how), and the size may be changed as a Cluster-wide default (TODO(joe): how),
|
|
or on an individual Node basis (TODO(joe): how).
|
|
|
|
TODO(joe): multi-region.
|
|
|
|
TODO(joe): high availability.
|
|
|
|
TODO(joe): see http://kubernetes.io/docs/getting-started-guides/aws/ for reasonable defaults.
|
|
|
|
TODO(joe): see Empire for inspiration: https://s3.amazonaws.com/empirepaas/cloudformation.json, especially IAM, etc.
|
|
|
|
All Nodes in the Cluster are configured uniformly:
|
|
|
|
1. DNS for service discovery.
|
|
2. Docker volume driver for EBS-based persistence (TODO: how does this interact with Mu volumes).
|
|
|
|
TODO(joe): describe whether this is done thanks to an AMI, post-install script, or something else.
|
|
|
|
TODO(joe): CloudWatch.
|
|
|
|
TODO(joe): CloudTrail.
|
|
|
|
##### Identity, Access Management, and Keys
|
|
|
|
The AWS translation for security constructs follows the [AWS best practices for IAM and key management](
|
|
http://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html). There is a fairly direct mapping between Mu
|
|
Users, Roles, and Groups, and the IAM equivalents with the same names.
|
|
|
|
AWS does not support Group nesting or inheritance, however. Mu handles this by "template expansion"; that is, by
|
|
copying any parent Group metadata from parent to all of its ancestors.
|
|
|
|
TODO(joe): keys.
|
|
|
|
TODO(joe): auth tokens.
|
|
|
|
##### Networking
|
|
|
|
Each Cluster gets a Virtual Private Cloud (VPC) for network isolation. Along with this VPC comes the standard set of
|
|
sub-resources: a Subnet, Internet Gateway, and Route Table. By default, Ingress and Egress ports are left closed. As
|
|
Stacks are deployed, ports are managed automatically (although an administrator can lock them (TODO(joe): how)).
|
|
|
|
TODO(joe): open SSH by default?
|
|
|
|
TODO(joe): joining existing VPCs.
|
|
|
|
TODO(joe): how to override default settings.
|
|
|
|
TODO(joe): multiple Availability Zones (and a Subnet per AZ); required for ELB.
|
|
|
|
TODO(joe): HTTPS certs.
|
|
|
|
TODO(joe): describe how ports get opened or closed (e.g., top-level Stack exports).
|
|
|
|
TODO(joe): articulate how Route53 gets configured.
|
|
|
|
TODO(joe): articulate how ELBs do or do not get created for the cluster as a whole.
|
|
|
|
##### Discovery and Cluster State
|
|
|
|
Next, each Cluster gets a key/value store. By default, this is Hashicorp Consul. This is used to manage Cluster
|
|
configuration, in addition to a discovery service should a true CaaS orchestration platform be used (i.e., not VMs).
|
|
|
|
TODO(joe): it's unfortunate that we need to do this. It's a "cliff" akin to setting up a Kube cluster.
|
|
|
|
TODO(joe): ideally we would use an AWS native key/value/discovery service (or our own, leveraging e.g. DynamoDB).
|
|
|
|
TODO(joe): this should be pluggable.
|
|
|
|
TODO(joe): figure out how to handle persistence.
|
|
|
|
TODO(joe): private container registries.
|
|
|
|
TODO(joe): encrypted secret storage (a la Vault).
|
|
|
|
#### Stacks/Services
|
|
|
|
Each Mu Stack compiles into a [CloudFormation Stack](
|
|
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/stacks.html), leveraging a 1:1 mapping. The only
|
|
exceptions to this rule are resource types that map directly to a CloudFormation resource name, backed either by a
|
|
standard AWS resource -- such as `AWS::S3::Bucket` -- or a custom one -- such as one of the Mu primitive types.
|
|
|
|
We also leverage [cross-Stack references](
|
|
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/walkthrough-crossstackref.html) to wire up references.
|
|
|
|
This approach means that you can still leverage all of the same CloudFormation tooling on AWS should you need to. For
|
|
example, your IT team might have existing policies and practices in place that can be kept. Managing Stacks through the
|
|
Mu tools, however, is still ideal, as it is easier to keep your code, metadata, and live site in synch.
|
|
|
|
TODO(joe): we need a strategy for dealing with AWS limits exhaustion; e.g.
|
|
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cloudformation-limits.html.
|
|
|
|
TODO(joe): should we support "importing" or "referencing" other CloudFormation Stacks, not in the Mu system?
|
|
|
|
The most interesting question is how Mu projects the primitive concepts in the system into CloudFormation metadata. For
|
|
most Stacks, this is just "composition" that falls out from name substitution, etc.; however, the primitive concepts
|
|
introduce "abstraction" and therefore manifest as groupings of physical constructs. Let us take them in order.
|
|
|
|
TODO(joe): I'm still unsure whether each of these should be a custom CloudFormation resource type (e.g.,
|
|
`Mu::Container`, `Mu::Gateway`, etc). This could make it a bit nicer to view in the AWS tools because you'd see
|
|
our logical constructs rather than the deconstructed form. It's a little less nice, however, in that it's more
|
|
complex implementation-wise, requiring dynamic Lambda actions that I'd prefer to be static compilation actions.
|
|
|
|
`mu/container` maps to a single `AWS::EC2::Instance`. However, by default, it runs a custom AMI that uses our daemon
|
|
for container management, including configuration, image pulling policies, and more. (Note that, later on, we will see
|
|
that running a CaaS layer completely changes the shape of this particular primitive.)
|
|
|
|
`mu/gateway` maps to a `AWS::ElasticLoadBalancing::LoadBalancer` (specifically, an [Application Load Balancer](
|
|
https://aws.amazon.com/elasticloadbalancing/applicationloadbalancer/)). Numerous policies are automatically applied
|
|
to target the Services wired up to the Gateway, including routine rules and tables. In the event that a Stack is
|
|
publically exported from the Cluster, this may also entail modifications of the overall Cluster's Ingress/Egress rules.
|
|
|
|
TODO: `mu/func` and `mu/event` are more, umm, difficult.
|
|
|
|
`mu/volume` is an abstract Stack type and so has no footprint per se. However, implementations of this type exist that
|
|
do have a footprint. For example, `aws/ebs/volume` derives from `mu/volume`, enabling easy EBS-based container
|
|
persistence. Please refer to the section below on native AWS Stacks to understand how this particular one works.
|
|
|
|
`mu/autoscaler` generally maps to an `AWS::AutoScaling::AutoScalingGroup`, however, like the Gateway's mapping to the
|
|
ELB, its mapping to the AWS scaling group entails a lot of automatic policy to properly scale attached Services.
|
|
|
|
The general case is any Stack marked `intrinsic: true`. These are mapped in a cloud backend-specific manner. For
|
|
example, AWS offers an `aws/x/cf` Stack type, which merely turns around and generates CloudFormation templates.
|
|
|
|
#### AWS-Specific Metadata
|
|
|
|
#### AWS-Specific Stacks
|
|
|
|
As we saw above, AWS services are available as Stacks. Let us now look at how they are expressed in Mu metadata and,
|
|
more interestingly, how they are transformed to underlying resource concepts. It's important to remember that these
|
|
aren't "higher level" abstractions in any sense of the word; instead, they map directly onto AWS resources. (O course,
|
|
other higher level abstractions may compose these platform primitives into more interesting services.)
|
|
|
|
A simplified S3 bucket Stack, for example, looks like this:
|
|
|
|
name: bucket
|
|
properties:
|
|
accessControl: string
|
|
bucketName: string
|
|
corsConfiguration: aws/schema/corsConfiguration
|
|
lifecycleConfiguration: aws/schema/lifecycleConfiguration
|
|
loggingConfiguration: aws/schema/loggingConfiguration
|
|
notificationConfiguration: aws/schema/notificationConfiguration
|
|
replicationConfiguration: aws/schema/replicationConfiguration
|
|
tags: [ aws/schema/resourceTag ]
|
|
versioningConfiguration: aws/schema/versioningConfiguration
|
|
websiteConfiguration: aws/schema/websiteConfigurationType
|
|
services:
|
|
public:
|
|
aws/x/cf:
|
|
resource: "AWS::S3::Bucket"
|
|
|
|
The key concept at play here is `aws/x/cf`, which is an intrinsic type. This passes off lifecycle events to a
|
|
provider, in this case the AWS backend, along with some metadata, in this case a simple wrapper around the [AWS
|
|
CloudFormation S3 Bucket specification format](
|
|
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-s3-bucket.html). The provider generates
|
|
metadata and knows how to interact with AWS services required for provisioning, updating, and destroying resources.
|
|
|
|
TODO(joe): we need to specify how intrinsics work somewhere.
|
|
|
|
Mu offers all of the AWS resource type Stacks out-of-the-box, so that 3rd parties can consume them easily. For example,
|
|
to create a bucket, we simply refer to the predefined `aws/s3/bucket` Stack. Please see [the AWS documentation](
|
|
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-template-resource-type-ref.html) for an exhaustive
|
|
list of available services.
|
|
|
|
TODO(joe): should we be collapsing "single resource" stacks? Seems superfluous and wasteful otherwise.
|
|
|
|
### Google Cloud Platform (GCP)
|
|
|
|
### Microsoft Azure
|
|
|
|
### VMWare
|
|
|
|
## CaaS Targets
|
|
|
|
All of the IaaS targets above described the default behavior when deploying containers, which is to map each container
|
|
to a dedicated VM instance. This is secure, robust, and easy to reason about, but can be wasteful. A CaaS framework
|
|
like Docker Swarm, Kubernetes, Mesos, or one of the native cloud provider container services, can bring about
|
|
efficiencies by multiplexing many containers onto a smaller shared pool of physical resources. This section describes
|
|
the incremental differences brought about when targeting such a framework.
|
|
|
|
### Docker Swarm
|
|
|
|
TODO(joe): figure out how Docker InfraKit does or does not relate to all of this (maybe even beyond Swarm target).
|
|
|
|
### Kubernetes
|
|
|
|
### Mesos
|
|
|
|
### AWS EC2 Container Service (ECS)
|
|
|
|
Targeting the [ECS](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/) CaaS lets AWS's native container
|
|
service manage scheduling of containers on EC2 VMs. It is only legal when using the AWS IaaS provider.
|
|
|
|
First and foremost, every Cluster containing at least one `mu/container` in its transitive closure of Stacks gets an
|
|
associated [ECS cluster](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_GetStarted.html).
|
|
|
|
A reasonable default number of instances, of a predefined type, are chosen, but you may override them (TODO(joe): how).
|
|
All of the AWS-wide settings, such as IAM, credentials, and region, are inherited from the base AWS IaaS configuration.
|
|
|
|
The next difference is that, rather than provisioning entire VMs per `mu/container`, each one maps to an [ECS service](
|
|
http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs_services.html).
|
|
|
|
TODO(joe): describe the auto-scaling differences. In ECS, service auto-scaling is *not* the same as ordinary EC2
|
|
auto-scaling. (See [this](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-auto-scaling.html).)
|
|
This could cause some challenges around the composition of `mu/autoscaler`, particularly with encapsulation.
|
|
|
|
TODO(joe): if we do end up supporting a `mu/job` type, we would presumably map it to ECS's CreateTask construct.
|
|
|
|
### Google Container Engine (GKE)
|
|
|
|
### Azure Container Service (ACS)
|
|
|
|
## Terraform
|
|
|
|
TODO(joe): describe what Terraform may be used to target and how it works.
|
|
|
|
## Redeploying Cluster and Stack Deltas
|
|
|
|
TODO(joe): describe how we perform delta checking in `$ mu apply` and how that impacts the various target generations.
|
|
|
|
TODO(joe): look into how Convox does this https://convox.com/guide/reloading/, and others.
|
|
|