This change includes a first basic whack at implementing S3 bucket
objects. It leverages the assets infrastructure put in place in the
last commit, supporting uploads from text, files, or arbitrary URIs.
Most of the interesting object properties remain unsupported for now,
but with this we can upload and delete basic S3 objects, sufficient
for a lot of the lambda functions management we need to implement.
This change introduces the basic concept of assets. It is far from
fully featured, however, it is enough to start adding support for various
storage kinds that require access to I/O-backed data (files, etc).
The challenge is that Coconut is deterministic by design, and so you
cannot simply read a file in an ad-hoc manner and present the bytes to
a resource provider. Instead, we will model "assets" as first class
entities whose data source is described to the system in a more declarative
manner, so that the system and resource providers can manage them.
There are three ways to create an asset at the moment:
1. A constant, in-memory string.
2. A path to a file on the local filesystem.
3. A URI, whose scheme is extensible.
Eventually, we want to support byte blobs, but due to our use of a
"JSON-like" type system, this isn't easily expressible just yet.
The URI scheme is extensible in that file://, http://, and https://
are supported "out of the box", but individual providers are free to
recognize their own schemes and support them. For instance, copying
one S3 object to another will be supported simply by passing a URI
with the s3:// protocol in the usual way.
Many utility functions are yet to be written, but this is a start.
This change implements simple index-based CocoPy subscripts (and
not the more fully featured slicing ones).
Alongside this, we relax a binder-time check that all dynamic
access types must be strings. The eval code already handles
numeric (array) accesses, so we will permit these to flow through.
This permits non-nil `this` objects for dynamically loaded properties
of classes and modules, provided they are the prototype or module
object for the target function, respectively.
This change emits all CocoJS interfaces as records. This allows us to
safely construct instances of them from Python using anonymous properties,
essentially emulating object literals.
For example, given a CocoJs interface defined as such:
interface Foo {
x;
y?;
z;
}
we can easily construct fresh instances using normal JS literals:
let f = { x: 42, z: "bar" };
But, Python doesn't have the equivalent literal syntax, wedging us.
It's now possible to initialize an instance from CocoPy as follows:
let f = Foo(x=42, z="bar")
This leverages the notion of records and primary properties, as
described in our CocoPack/CocoIL design documents.
* Pushing a class scope should permit subclasses (assertion).
* Returning nothing is legal for voids *and* dynamic (the latter was missing).
* The dynamic load readonly lval check is redundant; we check at the site of
assignment and doing it here as well led to two errors per usage.
Due to Python's ... interesting ... scoping rules, we want to avoid
forcing block scopes in certain places. Instead, we will let arbitrary
statements take their place. Of course, this could be a block, but it
very well could be a multi-statement (essentially a block that doesn't
imply a new lexical scope), or anything, really.
* Move checking the validity of a derived class with no constructor
from evlauation to binding (verification). Furthermore, make it
a verification error instead of an assert, and make the checking
complete (i.e., don't just check for the existence of a base class,
also check for the existence of a ctor, recursively).
* Refactor the constructor accessor out of the evaluator and make it
a method, Ctor, on the Function abstraction.
* Add a recursive Module population case to property initialization.
* Only treat the top-most frame in a module's initializer as belonging
to the module scope. This avoids interpreting block scopes within
that initializer as belonging to the module scope. Ultimately, it's
up to the language compiler to decide where to place scopes, but this
gives it more precise control over module scoping.
* Assert against unsupported named/star/keyword args in CocoPy.
If we encounter a dynamic invocation of a prototype object, we will
interpret it as an object allocation. This corresponds to code like
import ec2 from aws
instance = ec2.Instance(...)
where the second line dynamically loads the prototype object for the
Instance class from the module object for the aws/ec2 module, and
invokes it.
This change adds support for naming imports. At the moment, this simply
makes the names dynamically accessible for languages that do dynamic loads
against module objects, versus strongly typed tokens. The basic scheme
is to keep two objects per module: one that contains the globals and its
prototype parent that contains just the exports. This ensures we can
share the same slots while attaining the desired information hiding
(e.g., when handing out an object for dynamic access, we give out this
parent objects, while when loading globals from within a module, we use
the childmost one that contains all private and exported variables).
The old model for imports was to use top-level declarations on the
enclosing module itself. This was a laudible attempt to simplify
matters, but just doesn't work.
For one, the order of initialization doesn't precisely correspond
to the imports as they appear in the source code. This could incur
some weird module initialization problems that lead to differing
behavior between a language and its Coconut variant.
But more pressing as we work on CocoPy support, it doesn't give
us an opportunity to dynamically bind names in a correct way. For
example, "import aws" now needs to actually translate into a variable
declaration and assignment of sorts. Furthermore, that variable name
should be visible in the environment block in which it occurs.
This change switches imports to act like statements. For the most
part this doesn't change much compared to the old model. The common
pattern of declaring imports at the top of a file will translate to
the imports happening at the top of the module's initializer. This
has the effect of initializing the transitive closure just as it
happened previously. But it enables alternative models, like imports
inside of functions, and -- per the above -- dynamic name binding.
Previously, it was an error to look up a local that didn't exist.
Now it is common, thanks to dynamic lookups. A few code-paths didn't
previously handle this adequately; now they do.
This rearranges the way dynamic loads work a bit. Previously, they¬
required an object, and did a dynamic lookup in the object's property¬
map. For real dynamic loads -- of the kind Python uses, obviously,¬
but also ECMAScript -- we need to search the "environment".
This change searches the environment by looking first in the lexical¬
scope in the current function. If a variable exists, we will use it.¬
If that misses, we then look in the module scope. If a variable exists¬
there, we will use it. Otherwise, if the variable is used in a non-lval
position, an dynamic error will be raised ("name not declared"). If
an lval, however, we will lazily allocate a slot for it.
Note that Python doesn't use block scoping in the same way that most
languages do. This behavior is simply achieved by Python not emitting
any lexically scoped blocks other than at the function level.
This doesn't perfectly achieve the scoping behavior, because we don't
yet bind every name in a way that they can be dynamically discovered.
The two obvious cases are class names and import names. Those will be
covered in a subsequent commit.
Also note that we are getting lucky here that class static/instance
variables aren't accessible in Python or ECMAScript "ambiently" like
they are in some languages (e.g., C#, Java); as a result, we don't need
to introduce a class scope in the dynamic lookup. Some day, when we
want to support such languages, we'll need to think about how to let
languages control the environment probe order; for instance, perhaps
the LoadDynamicExpression node can have an "environment" property.
This changes the CocoPy default load type from static to dynamic,
since we don't have enough information at compile-time to emit
fully qualified tokens. Previously, Coconut only supported dynamic
loads with object targets, however we will need to support the full
scope search (class, module, global, etc).
This change makes node optional in the lookupBasicType function, which is
necessary in cases where diagnostics information isn't available (such as
with configuration application). This eliminates an assert when you fat-
finger a configuration key. The associated message was missing apostrophes.
This change eliminates the need to constantly type in the environment
name when performing major commands like configuration, planning, and
deployment. It's probably due to my age, however, I keep fat-fingering
simple commands in front of investors and I am embarrassed!
In the new model, there is a notion of a "current environment", and
I have modeled it kinda sorta just like Git's notion of "current branch."
By default, the current environment is set when you `init` something.
Otherwise, there is the `coco env select <env>` command to change it.
(Running this command w/out a new <env> will show you the current one.)
The major commands `config`, `plan`, `deploy`, and `destroy` will prefer
to use the current environment, unless it is overridden by using the
--env flag. All of the `coco env <cmd> <env>` commands still require the
explicit passing of an environment which seems reasonable since they are,
after all, about manipulating environments.
As part of this, I've overhauled the aging workspace settings cruft,
which had fallen into disrepair since the initial prototype.
This changes the object mapper infrastructure to offer more fine-grained
reporting of errors, and control over verification, during the mapping from
an untyped payload to a typed one. As a result, we can eliminate a bit of
the explicit unmarshaling goo in the AWS providers (but not all of it; I'm
sure there is more we can, and should, be doing here...)
This change adds the ability to specify analyzers in two ways:
1) By listing them in the project file, for example:
analyzers:
- acmecorp/security
- acmecorp/gitflow
2) By explicitly listing them on the CLI, as a "one off":
$ coco deploy <env> \
--analyzer=acmecorp/security \
--analyzer=acmecorp/gitflow
This closes out pulumi/coconut#119.
This change introduces the basic requirements for analyzers, as per
pulumi/coconut#119. In particular, an analyzer can implement either,
or both, of the RPC methods, Analyze and AnalyzeResource. The former
is meant to check an overall deployment (e.g., to ensure it has been
signed off on) and the latter is to check individual resources (e.g.,
to ensure properties of them are correct, such as checking style,
security, etc. rules). These run simultaneous to overall checking.
Analyzers are loaded as plugins just like providers are. The difference
is mainly in their naming ("analyzer-" prefix, rather than "resource-"),
and the RPC methods that they support.
This isn't 100% functional since we need a way to specify at the CLI
that a particular analyzer should be run, in addition to a way of
recording which analyzers certain projects should use in their manifests.
This changes a few naming things:
* Rename "husk" to "environment" (`coco env` for short).
* Rename NutPack/NutIL to CocoPack/CocoIL.
* Rename the primary Nut.yaml/json project file to Coconut.yaml/json.
* Rename the compiled Nutpack.yaml/json file to Cocopack.yaml/json.
* Rename the package asset directory from nutpack/ to .coconut/.
This change adds a new Check RPC method on the provider interface,
permitting resource providers to perform arbitrary verification on
the values of properties. This is useful for validating things
that might be difficult to express in the type system, and it runs
before *any* modifications are run (so failures can be caight early
before it's too late). My favorite motivating example is verifying
that an AWS EC2 instance's AMI is available within the target region.
This resolvespulumi/coconut#107, although we aren't using this
in any resource providers just yet. I'll add a work item now for that...
This change is mostly just a rename of Moniker to URN. It does also
prefix resource URNs to have a standard URN namespace; in other words,
"urn🥥<name>", where <name> is the same as the prior Moniker.
This is a minor step that helps to prepare us for pulumi/coconut#109.
This change adds a new resource.NewUniqueHex API, that simply generates
a unique hex string with the given prefix, with a specific count of
random bytes, and optionally capped to a maximum length.
This is used in the AWS SecurityGroup resource provider to avoid name
collisions, which is especially important during replacements (otherwise
we cannot possibly create a new instance before deleting the old one).
This resolvespulumi/coconut#108.
This change tracks which updates triggered a replacement. This enables
better output and diagnostics. For example, we now colorize those
properties differently in the output. This makes it easier to diagnose
why an unexpected resource might be getting deleted and recreated.
This change, part of pulumi/coconut#105, rearranges support for
resource replacement. The old model didn't properly account for
the cascading updates and possible replacement of dependencies.
Namely, we need to model a replacement as a creation followed by
a deletion, inserted into the overall DAG correctly so that any
resources that must be updated are updated after the creation but
prior to the deletion. This is done by inserting *three* nodes
into the graph per replacement: a physical creation step, a
physical deletion step, and a logical replacement step. The logical
step simply makes it nicer in the output (the plan output shows
a single "replacement" rather than the fine-grained outputs, unless
they are requested with --show-replace-steps). It also makes it
easier to fold all of the edges into a single linchpin node.
As part of this, the update step no longer gets to choose whether
to recreate the resource. Instead, the engine takes care of
orchestrating the replacement through actual create and delete calls.
This change introduces a new RPC function to the provider interface;
in pseudo-code:
UpdateImpact(id ID, t Type, olds PropertyMap, news PropertyMap)
(bool, PropertyMap, error)
Essentially, during the planning phase, we will consult each provider
about the nature of a proposed update. This update includes a set of
old properties and the new ones and, if the resource provider will need
to replace the property as a result of the update, it will return true;
in general, the PropertyMap will eventually contain a list of all
properties that will be modified as a result of the operation (see below).
The planning phase reacts to this by propagating the change to dependent
resources, so that they know that the ID will change (and so that they
can recalculate their own state accordingly, possibly leading to a ripple
effect). This ensures the overall DAG / schedule is ordered correctly.
This change is most of pulumi/coconut#105. The only missing piece
is to generalize replacing the "ID" property with replacing arbitrary
properties; there are hooks in here for this, but until pulumi/coconut#90
is addressed, it doesn't make sense to make much progress on this.
For cerain update shapes, we will need to recover an ID of an already-deleted,
or soon-to-be-deleted resource; in those cases, we have a moniker but want to
serialize an ID. This change implements support for remembering/recovering them.
This change fixes a couple issues that prevented restarting a
deployment after partial failure; this was due to the fact that
unchanged resources didn't propagate IDs from old to new. This
is remedied by making unchanged a map from new to old, and making
ID propagation the first thing plan application does.
