This change eliminates the AWS package's cloudformation.expandTags
functionality. The primary reason is that we haven't yet implemented
string/array intrinsics (see marapongo/mu#80), and so the presence of
this routine was preventing verification of the AWS library. But, it
turns out, the expandTags routine was violating our "zero artistic
license" principle for the foundational cloud packages. Namely, it
was auto-adding the Name=<name> tag, when the name property was set,
capturing an idiomatic AWS resource pattern. Instead of doing that,
we will just project tags in their raw form, and add such conveniences
(possibly) add a higher level in the layer cake.
After a bit more thinking, we will create new SDK packages for each
of the languages we wish to support writing resource providers in.
This is where the RPC goo will live, so I have created a new sdk/
directory, moved the Protobuf/gRPC definitions underneath sdk/proto/,
and put the generated code into sdk/go/ and sdk/js/.
This change:
* Renames the Mu standard library NPM package from "mu" to "@mu/mu".
This reflects the convention that MuPackages published on NPM will,
at least initially, be published under the "@mu" scope. This
indicates that a package is intended to be consumed by Mu programs.
* Adds an NPM peerDependency on the Mu package from the AWS package.
* Adds a Mu dependency on the Mu package from the AWS MuPackage.
This nixes the old *.mu files from the AWS MuPackage (when Mull was a thing),
and the nasty YAML files (when Go templates in YAML was a thing...(?!)). The
TypeScript variants remain, however, which is what we'll use going forward.
This rearranges the library code:
* sdk/... goes away.
* What used to be sdk/javascript/ is now lib/mu/, an actual MuPackage
that provides the base abstractions for all other MuPackages to use.
* lib/aws is the @mu/aws MuPackage that exposes all AWS resources.
* lib/mux is the @mu/x MuPackage that provides cross-cloud abstractions.
A lot of what used to be in lib/mu goes here. In particular, autoscaler,
func, ..., all the "general purpose" abstractions, really.
In the old system, the core runtime/toolset understood that we are targeting
specific cloud providers at a very deep level. In fact, the whole code-generation
phase of the compiler was based on it.
In the new system, this difference is less of a "special" concern, and more of
a general one of mapping MuIL objects to resource providers, and letting *them*
gather up any configuration they need in a more general purpose way.
Therefore, most of this stuff can go. I've merged in a small amount of it to
the mu/x MuPackage, since that has to switch on cloud IaaS and CaaS providers in
order to decide what kind of resources to provision. For example, it has a
mu.x.Cluster stack type that itself provisions a lot of the barebone essential
resources, like a virtual private cloud and its associated networking components.
I suspect *some* knowledge of this will surface again as we implement more
runtime presence (discovery, etc). But for the time being, it's a distraction
getting the core model running. I've retained some of the old AWS code in the
new pkg/resource/providers/aws package, in case I want to reuse some of it when
implementing our first AWS resource providers. (Although we won't be using
CloudFormation, some of the name generation code might be useful.) So, the
ships aren't completely burned to the ground, but they are certainly on 🔥.
This change encapsulates what's required for an AWS cluster in new service,
mu/clouds/aws/Cluster, so that mu/Cluster can remain fairly simplistic (it
just switches on the names and dispatches to the appropriate sub-modules).
There's no need for quotes here and having them makes it feel somehow
"dynamically typed" (I was subconsciously influenced by Go, I think).
Removing them...
This change renames a few things; more to come, but this is at least
a self-consistent checkpoint:
* Use the "new" keyword to create service objects and not "resource".
* Use the "func" keyword to indicate functions and not "macro".
* Use "bool" instead of "boolean" for boolean types (more Go-like).
First, I meant to rename "func" to "macro". This is perhaps contentious,
however my thinking was to differentiate between functions which have runtime
representations -- like lambdas, API gateway handlers, and the like -- and
macros which are purely a metadata/compile-time construct.
Second, there was a resource naming typo.
This sketches out a minimal configuration language, inspired by a combination
of the Mull document, Hashicorp's various language efforts, Protobufs, and a
mixture of TypeScript. I'm attempting to whittle down the concepts to the bare
minimum necessary for a universal "intermediate language" for our runtime that
is (a) complete enough to serve our needs, (b) advanced enough to deliver some
of the improvements in componentization and reuse that we desire, (c) appealing
enough that humans are able to write code in the language while possibly even
enjoying it, and, yet, (d) sufficient for machine interoperability needs.
Of course, I will capture all of this in a proper specification (overwriting
the existing Mull one), however I wanted to land these as a proof of concept and
for feedback purposes. Obviously, none of this code actually compiles or works!
This is a work-in-progress. In fact, it's going to radically change
given the new approach to representing resource construction, however
I wanted the snapshot in source control as we evolve things.
This demonstrates what infra-as-code might look like. Given the difficulties
in expressing everything purely as metadata, I've decided to embark upon a little
spike to see what this would look like. At first glance, I like it.
To see the difference, compare these to the Mu.yaml files alongside them.
This projects the basic AWS::ECS::TaskDefinition CloudFormation template
type as a stack. It also maps a bunch of schema types using fictitious
syntax, since we don't yet support this (see marapongo/mu#9). On to that next...
This change leverages intrinsics in place of the predefined types.
It remains to be seen if we can reach 100% on this, however I am hopeful.
It's also nice that the system will be built "out of itself" with this
approach; in other words, each of the types is simply a Mufile that can
use conditional targeting as appropriate for the given cloud providers.
If we find that this isn't enough, we can always bring back the concept.
The route resource is a bit funny, in that it lets you jam in either
an internet gateway ID or a VPC ID as the GatewayId property, whereas
all of the other target kinds get their own property. Our stack type
offers stronger typing than this (using service references), but we
need to map the resulting ref strings correctly.
I think things have gotten a little out of hand with the way mu/x/cf
auto-maps properties. In the beginning, it looked like everything could
be trivially auto-mapped, and I wanted to avoid the verbosity of mapping
each property by hand (since you can easily fat finger a name, mess up
capitalization, forget one, etc). But then we began mapping service
references using proper CloudFormation !Refs, which meant suppressing
some of the auto-mappings, etc., etc. This led to properties, extraProperties,
skipProperties, renamedProperties, and so on... Pretty confusing IMHO.
I just took a step back and decided to eliminate auto-mapping. Instead,
you get two options: properties just lists a set of property name mappings,
and extraProperties lets you do template magic to map thing instead if you
wish to take matters into your own hands. The result isn't too verbose
and has a lot less magic going on so it's easier to understand.
This change eliminates the special type mu/extension in favor of extensible
intrinsic types. This subsumes the previous functionality while also fixing
a number of warts with the old model.
In particular, the old mu/extension approach deferred property binding until
very late in the compiler. In fact, too late. The backend provider for an
extension simply received an untyped bag of stuff, which it then had to
deal with. Unfortunately, some operations in the binder are inaccessible
at this point because doing so would cause a cycle. Furthermore, some
pertinent information is gone at this point, like the scopes and symtables.
The canonical example where we need this is binding services names to the
services themselves; e.g., the AWS CloudFormation "DependsOn" property should
resolve to the actual service names, not the string values. In the limit,
this requires full binding information.
There were a few solutions I considered, including ones that've required
less code motion, however this one feels the most elegant.
Now we permit types to be marked as "intrinsic." Binding to these names
is done exactly as ordinary name binding, unlike the special mu/extension
provider name. In fact, just about everything except code-generation for
these types is the same as ordinary types. This is perfect for the use case
at hand, which is binding properties.
After this change, for example, "DependsOn" is expanded to real service
names precisely as we need.
As part of this change, I added support for three new basic schema types:
* ast.StringList ("string[]"): a list of strings.
* ast.StringMap ("map[string]any"): a map of strings to anys.
* ast.ServiceList ("service[]"): a list of service references.
Obviously we need to revisit this and add a more complete set. This work
is already tracked by marapongo/mu#9.
At the end of the day, it's likely I will replace all hard-coded predefined
types with intrinsic types, for similar reasons to the above.
Now that we're actually type-checking property types, a number of the
AWS stacks began failing, because we haven't yet mapped the full set of
resource types. I began doing that for the missing ones, but pulling on
that thread just leads to an even larger set ... (possibly all of them!)
due to extra dependencies. For now, since they aren't required for the
basic scenario, I'll just comment these out; I've logged marapongo/mu#27
to track mapping the full set of resource types.
I wasn't sure at first how we'd distinguish between strings and cap names.
At first, I thought we'd need special syntax, e.g. `<vpn>`; however, now I
am leaning towards a "type inference" approach, where we will interpret the
string as a cap name rather than string if that's what the target property
expects. Although this isn't quite as explicitly typed, that's probably a
good thing in this case, as it eliminates verbosity and weird characters.
To avoid injecting needless whitespace -- which is unfortunately meaningful
in YAML -- we need to use the new "-" trimming operators, available in Go 1.6.
This change projects several EC2 Mu stacks that are required for Mu cluster
bootstrapping. This includes:
- aws/ec2/internetGateway
- aws/ec2/route
- aws/ec2/routeTable
- aws/ec2/securityGroup
- aws/ec2/subnet
- aws/ec2/vpc
- aws/ec2/vpcGatewayAttachment
This is still not complete, and in fact some of these reference other EC2
Stacks that do not yet exist. But we're getting a bit closer...
This change adds a super simple initial whack at a basic cluster topology
comprised of VPC, subnet, internet gateway, attachments, and route tables.
This is actually written in Mu itself, and I am committing this early, since
there are quite a few features required before we can actually make progress
getting this up and running.
