14 KiB
Mu Cross-Cloud Targeting
The Mu metadata and primitives are intentionally cloud-agnostic and have been designed to support many cloud targets. This can be used to build, share, and reuse portable abstractions.
It is easy, however, to introduce a dependency on a particular cloud provider by relying on certain stacks. For
example, mounting an aws/ebs/volume
for a database volume pins it to the AWS IaaS provider; in fact, any such
service in the transitive closure of dependencies pins the whole stack to AWS.
On one hand, this is great, because the Mu abstractions do not get in the way of leveraging the full power of your cloud provider and its latest innovations. On the other hand, it inhibits portability.
This document briefly describes how Mu enables developers to create portable cloud infrastructure abstractions.
Abstraction
The key to Mu's ability to cross-target clouds is simple: abstraction.
Mu stacks are simply ordinary classes. As a result, they can encapsulate details about how they work. If a Table
stack wants to provision a MongoDB database automatically behind the scenes, there is no need for a consumer to know.
The properties, API, and so on, can safely hide these details behind a friendly logical abstraction.
More likely, however, such an abstraction will want to leverage a database-as-a-service (DbaaS) product in the target
cloud, like AWS DynamoDB. But by doing that, you would pin the Table
abstraction to AWS, which defeats the point.
This is a problem familiar to users of platform abstractions in environments like Java, .NET, and Node.js. To create
useful low-level primitive abstractions like filesystems, process models, and timer APIs -- among other things -- they
must create an abstraction layer just above the underlying operating system. Doing so lets 99% of the users of those
abstractions forget about the gory details of targeting Linux vs. macOS vs. Windows. But those low-level developers
need to worry about #ifdef
ing their code and bridging the gap. And in Mu, we can achieve the same economics.
To do this, the developer providing a low-level abstraction must conditionalize resource usage. The context object
exposed to the MetaMu programming languages tells the program information about the target environment, including
whether the target is AWS, Google Cloud, Azure, and so on. As such, the Table
can pick DynamoDB in AWS, DocumentDB in
Azure, Bigtable in Google Cloud, and perhaps even fall back to MongoDB option as a more complex default elsewhere.
If this is done right, the 99% developer can use an elegant and simple-to-use Table
abstraction, care-free about its
details, and in the cloud provider of their choice. And the low-level developer pays some complexity as a result.
Cross-Cloud Abstractions Out-of-the-Box
To facilitate cross-cloud abstractions, Mu offers a mu/x
package containing a number of them.
mu/x
The services offered by mu/x
have been conditionalized internally and are guaranteed to run on all clouds,
including locally for development and testing. The differences between them have been abstracted and unified so that
you can configure them declaratively, using a single logical set of options, and rely on the service internally mapping
to the cloud provider's specific configuration settings.
For example, mu/x/fs/Volume
implements the mu/Volume
abstract interface, and maps to an AWS Elastic Block Store
(EBS), Azure Data Disk (DD), or GCP Persistent Disk (PD) volume, depending on the IaaS target. Although the details for
each of these differs, a standard set of options -- like capacity, filesystem type, reclaimation policy, storage class,
and so on -- and the Mu framework handles mapping these standard options to the specific underlying ones.
The goal for the mu/x
package is to facilitate a higher-level ecosystem of cloud-agnostic services and libraries.
Services
This section contains a full list of the mu/x
cloud-agnostic services:
- Apps
- Containers
- Logging
- Queueing
- Pub/sub
- RPC
- Service discovery
- Serverless / API Gateway
- Data
- Cache
- Blob Store
- Key/value Store
- SQL Database
- NoSQL Database
- NoSQL Data Warehouse
- Secret Store
- Services
- SMS
- Search
- Job Scheduling
- Workflow
- MapReduce
- Infra Services
- DNS
- Load Balancing
- CDN
- Container Registry
TODO(joe): hand wavy; flesh this out more.
TODO(joe): need to figure out the distinction between { design-time, runtime } X { dev, ops }.
Appendix A: Cloud Catalog
Here's an exhaustive list of services offered by "the big three," and an attempt to correlate them.
COMPUTE | AWS | Azure | |
---|---|---|---|
IaaS | EC2 | Virtual Machines | Compute Engine |
Container Registry | EC2 Container Registry | Container Registry | |
Container Service | EC2 Container Service | Container Service | Container Engine |
PaaS | Elastic Beanstalk | Cloud Services / Service Fabric | App Engine |
Serverless | Lambda | Functions / WebJobs | Cloud Functions |
Job Scheduling | Scheduler / Batch | Compute Engine Tasks | |
Queueing | Simple Queueing Service (SQS) | Queue Storage / ServiceBus | |
Workflow | Simple Workflow Service (SWS) | LogicApps |
STORAGE | AWS | Azure | |
---|---|---|---|
Blob Storage | S3 | Storage | Cloud Storage (Standard) |
Low-Cost Archival | Glacier | Cloud Storage (Nearline) | |
Mountable Storage | Elastic File System | File Storage | |
Data Import/Export | Snowball | Import/Export | |
On-Prem-to-Cloud | Storage Gateway | StorSimple | |
Secrets | Key Management Service (KMS) | Key Vault |
DATABASE | AWS | Azure | |
---|---|---|---|
Database (SQL) | Relational DB Service (RDS) | SQL Database | Cloud SQL |
Database (NoSQL) | DynamoDB | Document DB / Table Storage | Cloud Bigtable / Cloud Datastore |
Cache | ElastiCache | Managed Cache / Redis Cache | |
Data Warehouse (SQL) | Redshift | SQL Data Warehouse | |
Data Warehouse (NoSQL) | Data Lake Store | BigQuery | |
Data Migration | DB Migration Service (DMS) | SQL DB Migration Wizard |
NETWORKING | AWS | Azure | |
---|---|---|---|
Load Balancer | EC2 Load Balancer | Load Balancer | Cloud Load Balancing |
CDN | CloudFront | Azure CDN | Cloud CDN |
Network Mgmt | Virtual Private Cloud (VPC) | Virtual Network | Cloud Virtual Network |
VPN | Direct Connect | ExpressRoute | Cloud Interconnect |
DNS | Route 53 | DNS | Cloud DNS |
DEVELOPER TOOLS | AWS | Azure | |
---|---|---|---|
Git Hosting | CodeCommit | VSTS | Cloud Source Repositories |
C/I | C/D | CodeDeploy | VSTS |
C/I | C/D Workflow | CodePipeline |
MANAGEMENT TOOLS | AWS | Azure | |
---|---|---|---|
E2E Management | CloudWatch | Stackdriver | |
Auditing | CloudTrail | ||
Monitoring | - | VS AppInsights | Monitoring |
Logging | - | Log Analytics | Logging |
Error Reporting | - | Error Reporting | |
Perf Tracing | - | Trace | |
Debugging | - | Debugger | |
Mgmt Templates | CloudFormation | Deployment Manager | |
Governance | Config | ||
Ops | OpsWorks (Chef) | Resource Manager / Automation | |
Security Templates | Service Catalog | ||
Service Optimization | Trusted Advisor |
SECURITY & IDENTITY | AWS | Azure | |
---|---|---|---|
Identity & Access | Identity & Access Mgmt | Cloud IAM | |
LDAP / AD | Directory Service (AD) | Active Directory | Cloud Resource Manager |
Security Analysis | Inspector | Security Center | Cloud Security Scanner |
DoS/Malicious Guards | WAF | ||
SSL/TLS Cert Mgmt | Certificate Manager |
BIG DATA / ANALYTICS | AWS | Azure | |
---|---|---|---|
MapReduce | Elastic MapReduce (Hadoop) | HDInsight | Cloud Dataproc |
Data Processing | Data Pipeline | Data Factory | |
Search | Elasticsearch Service | Search | |
Streams Processing | Kinesis | Stream Analytics | Cloud Dataflow |
Data Exploration | - | PowerBI / Data Lake Analytics | Cloud Datalab |
Pub/Sub/Push Notify | Simple Notif. Service (SNS) | Notif Hub Topics / Event Hubs | Cloud Pub/Sub |
Big Science | - | Cloud Genomics |
MACHINE LEARNING | AWS | Azure | |
---|---|---|---|
ML Platform | Machine Learning | Machine Learning | Cloud ML Platform |
ML/AI Services | Cognitive Services | Vision, Speech, NL, Translate |
INTERNET OF THINGS | AWS | Azure | |
---|---|---|---|
IoT | IoT | IoT Hub | IoT |
GAME DEVELOPMENT | AWS | Azure | |
---|---|---|---|
Games | GameLift |
MOBILE SERVICES | AWS | Azure | |
---|---|---|---|
Mobile E2E | Mobile Hub | ||
Mobile Auth | Cognito | Multifactor Auth | |
Mobile Testing | Device Farm | DevTest Labs | Cloud Test Lab |
Mobile Analytics | Mobile Analytics | HockeyApp |
APP SERVICES | AWS | Azure | |
---|---|---|---|
API Gateway | API Gateway | API Management | Google Cloud Endpoints |
Remote Desktop | AppStream | RemoteApp | |
Search | CloudSearch | Search | |
Media Transcoding | Elastic Transcoder | Media Services | |
Simple Email Service (SES) |
ENTERPRISE APPS | AWS | Azure | |
---|---|---|---|
Remote Desktop | WorkSpaces | ||
Document Sharing | WorkDocs | ||
Office (Email/Calendar) | WorkMail |