Commit graph

31 commits

Author SHA1 Message Date
joeduffy 9698309f2b Model resource ID and URN as output properties
This change exposes ID and URN properties on resources, as appropriate,
so that they may be read and used in Lumi scripts.
2017-06-14 17:00:13 -07:00
joeduffy 2ac303f703 Fix deployment hang (pulumi/lumi#246)
The recent change to run the interpreter and planner on separate goroutines
created the need to perform rendezvous-style synchronization between them.
Although the case of an invoked function properly tore down the synchronization
by communicating the error, we seldom directly invoke functions for JavaScript
programs because the way module entrypoint code ends up in initializers.
This requires that we propagate errors correctly out of module and class
initializers, in the standard way, so that the unwind makes its way to the top.

This fixes pulumi/lumi#246.
2017-06-14 15:52:36 -07:00
Luke Hoban 282f40d3e3 Merge branch 'master' into bforsyth927-gometalinter 2017-06-13 16:28:12 -07:00
joeduffy 0d836ae0bd Recover from deployment failures 2017-06-13 07:10:13 -07:00
joeduffy 75a2f14d10 Propagate IDs/outs differently based on step kind
This change updates the ID/output propagation logic to properly handle
the case of replacements, in addition to accurately conveying the fact
that an update may change the values of output properties (but not the ID).

Also fixes a formatting issue with the replacement diffing displays.
2017-06-13 07:10:13 -07:00
joeduffy 25c52a04c5 Tidy up some loose ends
This removes some loose ends and reimplements `lumi pack eval`.
2017-06-13 07:10:13 -07:00
joeduffy dd9e6b35f4 Introduce an OpSame planning step
This change introduces an OpSame planning step.  The reason we need
this is so that we can apply the necessary output properties, including
the ID, even as we are simply walking the plan (i.e., when we aren't
actually performing a deployment).  This ensures that the object state
evolves as required to let reads of output properties propagate in the
ways necessary to reproduce past executions of the program.
2017-06-13 07:10:13 -07:00
joeduffy d1414af321 Fix a few minor things; clean stuff up
* Assert new things in new places.

* Log more interesting tidbits during evaluation.

* Invoke the OnStart hook before triggering initializers.

* Tolerate nil prev snapshots during deletion calculation.

* Handle and serialize missing resource IDs as output props.

* Return "done" flag from Rendezvous.Meet.
2017-06-13 07:10:13 -07:00
joeduffy d277dd5800 More progress on pulumi/lumi#90
This change refactors a number of aspects of the CLI's treatment of
steps, in line with the new scheme, and a number of other miscellaneous
and minor fixes.  It also regenerates all RPC code impacted by recent renames.
2017-06-13 07:10:13 -07:00
joeduffy d044720045 Make more progress on the new deployment model
This change restructures a lot more pertaining to deployments, snapshots,
environments, and the like.

The most notable change is that the notion of a deploy.Source is introduced,
which splits the responsibility between the deploy.Plan -- which simply
understands how to compute and carry out deployment plans -- and the idea
of something that can produce new objects on-demand during deployment.

The primary such implementation is evalSource, which encapsulates an
interpreter and takes a package, args, and config map, and proceeds to run
the interpreter in a distinct goroutine.  It synchronizes as needed to
poke and prod the interpreter along its path to create new resource objects.

There are two other sources, however.  First, a nullSource, which simply
refuses to create new objects.  This can be handy when writing isolated
tests but is also used to simulate the "empty" environment as necessary to
do a complete teardown of the target environment.  Second, a fixedSource,
which takes a pre-computed array of objects, and hands those, in order, to
the planning engine; this is mostly useful as a testing technique.

Boatloads of code is now changed and updated in the various CLI commands.

This further chugs along towards pulumi/lumi#90.  The end is in sight.
2017-06-13 07:10:13 -07:00
joeduffy 6b2408e086 Rewrite plans and deployments
This change guts the deployment planning and execution process, a
necessary component of pulumi/lumi#90.

The major effect of this change is that resources are actually
connected to the live objects, instead of being snapshots taken at
inopportune moments in time.
2017-06-13 07:10:13 -07:00
joeduffy c53ddeb678 Overhaul resources, planning, and environments
This change, part of pulumi/lumi#90, overhauls quite a bit of the
core resource, planning, environments, and related areas.

The biggest amount of movement comes from the splitting of pkg/resource
into multiple sub-packages.  This results in:

- pkg/resource: just the core resource data structures.

- pkg/resource/deployment: all planning and deployment logic.

- pkg/resource/environment: all environment, configuration, and
      serialized checkpoint structures and logic.

- pkg/resource/plugin: all dynamically loaded analyzer and
      provider logic, including the actual loading and RPC mechanisms.

This also splits the resource abstraction up.  We now have:

- resource.Resource: a shared interface.

- resource.Object: a resource that is connected to a live object
      that will periodically observe mutations due to ongoing
      evaluation of computations.  Snapshots of its state may be
      taken; however, this is purely a "pre-planning" abstraction.

- resource.State: a snapshot of a resource's state that is frozen.
      In other words, it is no longer connected to a live object.
      This is what will store provider outputs (ID and properties),
      and is what may be serialized into a deployment record.

The branch is in a half-baked state as of this change; more changes
are to come...
2017-06-13 07:10:13 -07:00
Britton Forsyth 3066fcda78 Implemented suggested edits 2017-06-08 11:44:16 -07:00
Britton Forsyth 7457cadf58 Fixed various additional linting issues 2017-06-08 10:21:17 -07:00
Britton Forsyth 00ade9f28a Fixed some gometalinter issues 2017-06-07 10:52:03 -07:00
joeduffy c7dc3036d7 Finish scrubbing TODOs
This is a final pass over our TODOs, and closes pulumi/lumi#212.
2017-06-06 06:05:35 -07:00
joeduffy db99092334 Implement mapper.Encode "for real"
This change implements `mapper.Encode` "for real" (that is, in a way
that isn't a complete embarrassment).  It uses the obvious reflection
trickery to encode a tagged struct and its values as a JSON-like
in-memory map and collection of keyed values.

During this, I took the opportunity to also clean up a few other things
that had been bugging me.  Namely, the presence of `mapper.Object` was
always error prone, since it isn't a true "typedef" in the sence that
it carries extra RTTI.  Instead of doing that, let's just use the real
`map[string]interface{}` "JSON-map-like" object type.  Even better, we
no longer require resource providers to deal with the mapper
infrastructure.  Instead, the `Check` function can simply return an
array of errors.  It's still best practice to return field-specific errors
to facilitate better diagnostics, but it's no longer required; and I've
added `resource.NewFieldError` to eliminate the need to import mapper.

As of this change, we can also consistently emit RPC structs with `lumi`
tags, rather than `lumi` tags on the way in and `json` on the way out.

This completes pulumi/lumi#183.
2017-06-05 17:49:00 -07:00
joeduffy 87004a124e Store both input and output properties distinctly
This changes the resource model to persist input and output properties
distinctly, so that when we diff changes, we only do so on the programmer-
specified input properties.  This eliminates problems when the outputs
differ slightly; e.g., when the provider normalizes inputs, adds its own
values, or fails to produce new values that match the inputs.

This change simultaneously makes progress on pulumi/lumi#90, by beginning
tracking the resource objects implicated in a computed property's value.

I believe this fixes both #189 and #198.
2017-06-04 19:24:48 -07:00
joeduffy 43bcbed23d Tidy up project loading for pack commands
There are a few things that annoyed me about the way our CLI works with
directories when loading packages.  For example, `lumi pack info some/pack/dir/`
never worked correctly.  This is unfortunate when scripting commands.
This change fixes the workspace detection logic to handle these cases.
2017-06-02 12:43:04 -07:00
joeduffy b07056ab10 Create a plan plugin host
This is a minor refactoring to introduce a ProviderHost interface
that is associated with the context and can be swapped in and out for
custom plugin behavior.  This is required to write tests that mock
certain aspects, like loading packages from the filesystem.

In theory, this change incurs zero behavioral changes.
2017-06-01 11:41:24 -07:00
joeduffy 0e5ba9655f Pretty print outputs during planning 2017-06-01 10:52:25 -07:00
joeduffy 7b5f9df917 Make updates work in the face of output properties
This change fixes up a few things so that updates correctly deal
with output properties.  This involves a few things:

    1) All outputs stored on the pre snapshot need to get propagated
       to the post snapshot during planning at various points.  This
       ensures that the diffing logic doesn't need to be special cased
       everywhere, including both the Lumi and the provider sides.

    2) Names are changed to "input" properties (using a new `lumi` tag
       option, `in`).  These are properties that providers are expected
       to know nothing about, which we must treat with care during diffs.

    3) We read back properties, via Get, after doing an Update just like
       we do after performing a Create.  This ensures that if an update
       has a cascading impact on other properties, it will be detected.

    4) Inspecting a change, prior to updating, must be done using the
       computed property set instead of the real one.  This is to avoid
       mutating the resource objects ahead of actually applying a plan,
       which would be wrong and misleading.
2017-06-01 10:09:52 -07:00
joeduffy ae8cefcb20 Print output properties in the CLI
This change skips printing output<T> properties as we perform a
deployment, instead showing the real values inline after the resource
has been created.  (output<T> is still shown during planning, of course.)
2017-06-01 08:37:56 -07:00
joeduffy 87ad371107 Only flow logging to plugins if --logflow
The change to flow logging to plugins is nice, however, it can be
annoying because all writes to stderr are interepreted on the Lumi
side as errors.  After this change, we will only flow if
--logflow is passed, e.g. as in

    $ lumi --logtostderr --logflow -v=9 deploy ...
2017-06-01 08:37:56 -07:00
joeduffy 47e242f9a7 Rearrange some deployment logic
This change prepares for integrating more planning and deployment logic
closer to the runtime itself.  For historical reasons, we ended up with these
in the env.go file which really has nothing to do with deployments anymore.
2017-06-01 08:36:43 -07:00
joeduffy 7f98387820 Distinguish between computed and output properties
This change introduces the notion of a computed versus an output
property on resources.  Technically, output is a subset of computed,
however it is a special kind that we want to treat differently during
the evaluation of a deployment plan.  Specifically:

* An output property is any property that is populated by the resource
  provider, not code running in the Lumi type system.  Because these
  values aren't available during planning -- since we have not yet
  performed the deployment operations -- they will be latent values in
  our runtime and generally missing at the time of a plan.  This is no
  problem and we just want to avoid marshaling them in inopportune places.

* A computed property, on the other hand, is a different beast altogehter.
  Although true one of these is missing a value -- by virtue of the fact
  that they too are latent values, bottoming out in some manner on an
  output property -- they will appear in serializable input positions.
  Not only must we treat them differently during the RPC handshake and
  in the resource providers, but we also want to guarantee they are gone
  by the time we perform any CRUD operations on a resource.  They are
  purely a planning-time-only construct.
2017-06-01 08:36:43 -07:00
joeduffy 7879032e88 Pretty-print attributes in lumi pack info command
This change pretty-prints attribute metadata in `lumi pack info`.
For example:

    package "basic/decorators" {
        dependencies []
        module "index" {
            exports []
            method ".main": ()
            class "TestDecorators" [@basic/decorators:index:classDecorate] {
                property "a" [public, @basic/decorators:index:propertyDecorate]: string
                method "m1" [public, @basic/decorators:index:methodDecorate]: (): string
            }
        }
    }

It also includes support for printing property getters/setters:

    property "p1" [public]: string {
        method "get" [public, @basic/decorators:index:methodDecorate]: (): string
        method "set" [public]: (v: string)
    }
2017-06-01 08:32:12 -07:00
joeduffy d79c41f620 Initial support for output properties (1 of 3)
This change includes approximately 1/3rd of the change necessary
to support output properties, as per pulumi/lumi#90.

In short, the runtime now has a new hidden type, Latent<T>, which
represents a "speculative" value, whose eventual type will be T,
that we can use during evaluation in various ways.  Namely,
operations against Latent<T>s generally produce new Latent<U>s.

During planning, any Latent<T>s that end up in resource properties
are transformed into "unknown" property values.  An unknown property
value is legal only during planning-time activities, such as Check,
Name, and InspectChange.  As a result, those RPC interfaces have
been updated to include lookaside maps indicating which properties
have unknown values.  My intent is to add some helper functions to
make dealing with this circumstance more correct-by-construction.

For now, using an unresolved Latent<T> in a conditional will lead
to an error.  See pulumi/lumi#67.  Speculating beyond these -- by
supporting iterative planning and application -- is something we
want to support eventually, but it makes sense to do that as an
additive change beyond this initial support.  That is a missing 1/3.

Finally, the other missing 1/3rd which will happen much sooner
than the rest is restructuing plan application so that it will
correctly observe resolution of Latent<T> values.  Right now, the
evaluation happens in one single pass, prior to the application, and
so Latent<T>s never actually get witnessed in a resolved state.
2017-06-01 08:32:12 -07:00
joeduffy ce1dc4e30b Fix an erroneous reference to lumi env deploy 2017-05-18 15:54:40 -07:00
joeduffy 4108c51549 Reclassify Lumi under the Apache 2.0 license
This is part of pulumi/lumi#147.
2017-05-18 14:51:52 -07:00
joeduffy dafeb77dff Rename Coconut to Lumi
This is part of pulumi/coconut#147.

After it has landed, I will rename the repo on GitHub.
2017-05-18 11:38:28 -07:00