Commit graph

225 commits

Author SHA1 Message Date
joeduffy
a6caef973a Make assets and archives 1st class
This change recognizes assets and archives as 1st class resource
property values.  This is necessary to support them in the new bridge
work, and lays the foundation for fixing pulumi/lumi#153.

I also took the opportunity to clean up some old cruft in the
resource properties area.
2017-07-14 12:28:43 -07:00
joeduffy
c3db70849d Rename Lumi resource properties
This renames the basemost resource properties, id and urn, to
names that are less likely to conflict with properties that real
resources will want to use, pid and upn (provider ID and Universal
Pulumi Name, respectively).

I actually ran into this with the current bridge work.  An alternative
solution would be to require derived resources to pick different names,
however this is unfortunate because usually they are more "user-facing"
than ours.  Another alternative is to not hijack the object properties
at all, but that too is problematic because we use these properties
during the evaluation of plans and deployments.

This seems like a reasonable middle ground.
2017-07-14 08:55:07 -07:00
joeduffy
a70790b81a Add key replacement functions
This continues to add some handy replacement routines for property
maps, so we can transform keys and values to and fro.
2017-07-12 13:03:18 -07:00
joeduffy
ea0461dadb Export plugin prefix constants 2017-07-06 09:46:00 -04:00
joeduffy
aeefc27a08 Add some ReadLocations tracing; and skip nulls 2017-07-06 00:02:33 -04:00
joeduffy
06ad983541 Add a ReadLocations engine-side RPC function
This adds a ReadLocations RPC function to the engine interface, alongside
the singular ReadLocation.  The plural function takes a single token that
represents a module or class and we will then return all of the module
or class (static) properties that are currently known.
2017-07-01 13:26:49 -07:00
joeduffy
5d9f7918e9 Add a PropertyMap/Value.MapReplace function
This adds a handy MapReplace function on pkg/resource's PropertyMap and
PropertyValue types.  This is just like the existing Mappable function,
except that it permits easy replacement of elements as the map transformation
occurs.  We need this to perform float64=>int transformations.
2017-07-01 12:08:55 -07:00
joeduffy
27b819dec6 Add the ability to skip nulls when un/marshaling properties 2017-07-01 11:51:52 -07:00
joeduffy
15a75c9ee4 Catch duplicate URNs during planning
We fail very late in the process of plan application, should a duplicate
URN arise.  This change fails as early in the process as possible and
ensures that it does so with good line number information.
2017-06-27 13:04:06 -07:00
joeduffy
24fd8e8f4a Add cancellation to interpreter
This properly unwinds the interpreter should something happen that
results in cancellation.  This occurs, for example, when the planning
engine encounters an error and decides that it doesn't need to proceed
further with evaluation before it simply goes ahead and exits.
2017-06-27 11:31:17 -07:00
joeduffy
19a359e65d Specialize check failure messages 2017-06-27 10:56:33 -07:00
joeduffy
0ea55328f6 Specialize the null message for config reads 2017-06-27 10:45:29 -07:00
joeduffy
2daea4c3d8 Clarify aspects of using the DCO 2017-06-26 14:46:34 -07:00
joeduffy
3c1041af49 Update license headers 2017-06-23 14:53:41 -07:00
joeduffy
d05e7ace91 Ensure we close the plugin host/context
This adds a few missing closes for the plugin host/context.  This
should fix pulumi/lumi#261.  Eventually when we have more robust
nightly test options, and want to spend the time, we should think
about doing more rigorous stress testing that kills processes at
inopportune times and guarantees we don't leak.  I've filed
pulumi/lumi#263 to do that.
2017-06-22 15:18:29 -07:00
joeduffy
8b57310854 Tidy up more lint
This change fixes a few things:

* Most importantly, we need to place a leading "." in the paths
  to Gometalinter, otherwise some sub-linters just silently skip
  the directory altogether.  errcheck is one such linter, which
  is a very important one!

* Use an explicit Gometalinter.json file to configure the various
  settings.  This flips on a few additional linters that aren't
  on by default (line line length checking).  Sadly, a few that
  I'd like to enable take waaaay too much time, so in the future
  we may consider a nightly job (this includes code similarity,
  unused parameters, unused functions, and others that generally
  require global analysis).

* Now that we're running more, however, linting takes a while!
  The core Lumi project now takes 26 seconds to lint on my laptop.
  That's not terrible, but it's long enough that we don't want to
  do the silly "run them twice" thing our Makefiles were previously
  doing.  Instead, we shall deploy some $$($${PIPESTATUS[1]}-1))-fu
  to rely on the fact that grep returns 1 on "zero lines".

* Finally, fix the many issues that this turned up.

I think(?) we are done, except, of course, for needing to drive
down some of the cyclomatic complexity issues (which I'm possibly
going to punt on; see pulumi/lumi#259 for more details).
2017-06-22 12:09:46 -07:00
Luke Hoban
a63efc42a3 Propagate errors on deployment failures
We were not propagating the error from `deployLatest` through
to the CLI error result.  Despite out recent efforts to integrate
gometalinter, there were also several additional similar cases of
ignored error results reported by `errcheck`.  Not yet clear why
these are not being reported via gometalinter.

Fixes #262.
2017-06-21 22:02:57 -07:00
joeduffy
7fe8052941 Fix some lint in our lint
After 233c5a8 landed, I noticed there are a few things to be fixed up:

    * Run gometalinter in all the right places.  We need to run both in
      lint and lint_quiet targets.  I've also cleaned up some of the logic
      around what to suppress so there's less repetition.

    * We currently @ meaningful commands, which is unfortunate, since it
      makes debugging Makefiles tough (especially when looking at CI build
      logs).  Going forward, we should only use @ for meaningless commands,
      like @echo.

    * The AWS project wasn't actually running tslint, because it needs to
      say `tslint './pack/**/*.ts' --exclude='./pack/node_modules/**'`.
      The current script of `tslint lib/aws/pack/...` wasn't actually
      running lint, hence we missed a lot of AWS lint issues.

    * Fix up the issues that these fixes uncovered.  Mostly err shadowing.
2017-06-21 13:24:35 -07:00
joeduffy
f6f166a5c3 Add a TODO for pulumi/lumi#260 2017-06-21 11:33:10 -07:00
joeduffy
97deabb9bd Finish interface for reading configuration¬
This continues the previous commit and establishes the interpreter
context so that we can use the new host interface.  In summary:

    * Instead of using the NullSource for destructions -- which
      doesn't hook up an interpreter and so any reads of configuration
      variables will fail -- we will enlighten the EvalSource to know
      how to orchestrate destruction interpretation.  The primary
      difference is that we don't actually run the code, but *we do*
      perform all of the necessary configuration and variable init.

    * Associate the active interpreter with the plugin context as
      we are executing, so that the host object can actually read the
      state from the heap as requested to do so by attached plugins.

    * Rename anything "engine" related to use the term "host"; this
      avoids introducing unnecesarily new terminology.

    * Add a new pkg/resource/provider/ package where we can begin
      consolidating helper functionality for resource providers.
      Right now, this includes a wrapper interface atop the gRPC
      machinery necessary to contact the host, in addition to a
      Main function that hides some boilerplate entrypoint code.

    * Add a rpcutil.IsBenignCloseErr routine to let us ignore
      "benign" gRPC errors that are knowingly returned at shutdown.

This commit completes pulumi/lumi#117.
2017-06-21 10:31:06 -07:00
joeduffy
e5f229aad8 Use Location.Read from plugin host
This addresses CR feedback from @lukehoban; namely, that we should
be going through the Read API for location reads in the plugin host
to ensure that getters are invoked as appropriate.

I also made Location's various fields private so that we aren't
tempted to make this mistake elsewhere, effectively "forcing" us
to go through the accessor methods.
2017-06-21 08:43:05 -07:00
joeduffy
d7093188f0 Introduce an interface to read config
This change adds an engine gRPC interface, and associated implementation,
so that plugins may do interesting things that require "phoning home".
Previously, the engine would fire up plugins and talk to them directly,
but there was no way for a plugin to ask the engine to do anything.

The motivation here is so that plugins can read evaluator state, such
as config information, but this change also allows richer logging
functionality than previously possible.  We will still auto-log any
stdout/stderr writes; however, explicit errors, warnings, informational,
and even debug messages may be written over the Log API.
2017-06-20 19:45:07 -07:00
joeduffy
26cf93f759 Implement get functions on all resources
This change implements the `get` function for resources.  Per pulumi/lumi#83,
this allows Lumi scripts to actually read from the target environment.

For example, we can now look up a SecurityGroup from its ARN:

    let group = aws.ec2.SecurityGroup.get(
        "arn:aws:ec2:us-west-2:153052954103:security-group:sg-02150d79");

The returned object is a fully functional resource object.  So, we can then
link it up with an EC2 instance, for example, in the usual ways:

    let instance = new aws.ec2.Instance(..., {
        securityGroups: [ group ],
    });

This didn't require any changes to the RPC or provider model, since we
already implement the Get function.

There are a few loose ends; two are short term:

    1) URNs are not rehydrated.
    2) Query is not yet implemented.

One is mid-term:

    3) We probably want a URN-based lookup function.  But we will likely
       wait until we tackle pulumi/lumi#109 before adding this.

And one is long term (and subtle):

    4) These amount to I/O and are not repeatable!  A change in the target
       environment may cause a script to generate a different plan
       intermittently.  Most likely we want to apply a different kind of
       deployment "policy" for such scripts.  These are inching towards the
       scripting model of pulumi/lumi#121, which is an entirely different
       beast than the repeatable immutable infrastructure deployments.

Finally, it is worth noting that with this, we have some of the fundamental
underpinnings required to finally tackle "inference" (pulumi/lumi#142).
2017-06-19 17:29:02 -07:00
joeduffy
7f710a241f Better handle errors during config apply 2017-06-16 15:37:58 -07:00
joeduffy
5f9ed13069 Simplify Check; make it tolerant of computed values
This change simplifies the generated Check interface for providers.
Instead of

    Check(ctx context.Context, obj *T) ([]error, error)

where T is the resource type, we have

    Check(ctx context.Context, obj *T, property string) error

This is done so that we can drive the calls to Check one property
at a time, allowing us to skip any that are computed.  (Otherwise,
we may fail the verification erroneously.)

This has the added advantage that the Check implementations are
simpler and can simply return a single error.  Furthermore, the
generated RPC code handles wrapping the result, so we can just do

    return errors.New("bad");

rather than the previous reflection-laden junk

    return resource.NewFieldError(
        reflect.TypeOf(obj), awsservice.AWSResource_Property,
        errors.New("bad"))
2017-06-16 13:34:11 -07:00
Luke Hoban
33a9452ece Merge pull request #256 from pulumi/examplestest
Add integration testing for examples
2017-06-16 10:17:51 -07:00
joeduffy
5e85e6d543 Improve property validation diagnostics
The change prints the value of a property that fails resource validation.
This makes it much easier to diagnose what's going on.
2017-06-16 10:04:49 -07:00
joeduffy
7d19abc2a3 Print the current environment
This change implements showing a summary of the current environment.
All you need to do is run

    $ lumi env

and the current environment's information will be printed.

This makes it convenient to grab resource information that might be
required, for instance, to correlate with logs (e.g., lambda ARNs).

Eventually, as per pulumi/lumi#184, we want to print details about
all of the resources too.
2017-06-16 09:46:09 -07:00
Luke Hoban
639a2d323d Test more examples
Tests all of our commonly used examples.

Also sets test parallelism to 10 by default
since we are I/O bound on API calls to
the resource providers.

Also avoids using larger EC2 examples in
our samples so that we can keep our test
costs lower :-).
2017-06-16 09:24:31 -07:00
joeduffy
d013501e04 Restructure Rendezvous to have a distinct Let vs. Meet
On the first turn, we want to distinguish between a coroutine
running that owns its turn, and a coroutine that knows it doesn't
own the turn and is simply awaiting its turn.  The old Meet logic
wasn't quite right; instead, we'll have the caller tell us this.
2017-06-15 18:20:12 -07:00
Luke Hoban
ae03d69645 Wire up APIs to lambdas using output properties
We now have enough output properties implementation
working to change our API gateway examples and API
wrapper to correctly wire the API routes to the ARNs of
lambdas passed in to them.

We both wire up the lambda to the route, but also create
a permission specific to each route to assign to the
corresponding lambda - providing least privelege needed
for the API definition.

Also adds `string#toUpperCase` and fixes NewUniqueHex
to match how we are using it.
2017-06-15 16:01:00 -07:00
joeduffy
085e39230b Test missing and output properties
This tests the conditions that triggered pulumi/lumi#251.
2017-06-15 13:08:06 -07:00
joeduffy
3e4e791fa4 Also overwrite output property slots
This change overwrites output property slots in runtime objects
after performing a CRUD operation, in addition to null or missing
slots, fixing #251.  The problem is that we sometimes have output
property values pre-populated in an object, and sometimes don't,
depending on various things (both are legal).  We should handle both.
2017-06-15 13:08:06 -07:00
joeduffy
9698309f2b Model resource ID and URN as output properties
This change exposes ID and URN properties on resources, as appropriate,
so that they may be read and used in Lumi scripts.
2017-06-14 17:00:13 -07:00
joeduffy
2ac303f703 Fix deployment hang (pulumi/lumi#246)
The recent change to run the interpreter and planner on separate goroutines
created the need to perform rendezvous-style synchronization between them.
Although the case of an invoked function properly tore down the synchronization
by communicating the error, we seldom directly invoke functions for JavaScript
programs because the way module entrypoint code ends up in initializers.
This requires that we propagate errors correctly out of module and class
initializers, in the standard way, so that the unwind makes its way to the top.

This fixes pulumi/lumi#246.
2017-06-14 15:52:36 -07:00
Luke Hoban
441f32d155 A few tweaks to lint fixes 2017-06-13 16:47:55 -07:00
Luke Hoban
282f40d3e3 Merge branch 'master' into bforsyth927-gometalinter 2017-06-13 16:28:12 -07:00
Britton Forsyth
01003ad48b Implemented highlighted edits 2017-06-13 11:01:23 -07:00
joeduffy
01bbc23144 Only mark immediate outputs as outputs
The primary purposes of this change is to mark only immediate ouptuts
on a resource object as "output" and categories the rest as computed.

It also contains a few minor things:

    * Rebase atop the latest in master.

    * Always marshal unknows as their default value.

    * Permit computed as the existing ID property, in addition to null.

    * Tidy up some asserts.
2017-06-13 07:10:13 -07:00
joeduffy
0d836ae0bd Recover from deployment failures 2017-06-13 07:10:13 -07:00
joeduffy
2eb78e1036 Label replacements as simply "replace" 2017-06-13 07:10:13 -07:00
joeduffy
75a2f14d10 Propagate IDs/outs differently based on step kind
This change updates the ID/output propagation logic to properly handle
the case of replacements, in addition to accurately conveying the fact
that an update may change the values of output properties (but not the ID).

Also fixes a formatting issue with the replacement diffing displays.
2017-06-13 07:10:13 -07:00
joeduffy
25c52a04c5 Tidy up some loose ends
This removes some loose ends and reimplements `lumi pack eval`.
2017-06-13 07:10:13 -07:00
joeduffy
dd9e6b35f4 Introduce an OpSame planning step
This change introduces an OpSame planning step.  The reason we need
this is so that we can apply the necessary output properties, including
the ID, even as we are simply walking the plan (i.e., when we aren't
actually performing a deployment).  This ensures that the object state
evolves as required to let reads of output properties propagate in the
ways necessary to reproduce past executions of the program.
2017-06-13 07:10:13 -07:00
joeduffy
9d73c54994 Run post-construction hooks before freezing
We need to run the post-construction hook *before* freezing an object's
readonly properties, since the hook will actually mutate the object in
the case of a deployment (it stores the output properties).  In a sense,
this hook simply becomes an extension of the object's constructor.
2017-06-13 07:10:13 -07:00
joeduffy
d1414af321 Fix a few minor things; clean stuff up
* Assert new things in new places.

* Log more interesting tidbits during evaluation.

* Invoke the OnStart hook before triggering initializers.

* Tolerate nil prev snapshots during deletion calculation.

* Handle and serialize missing resource IDs as output props.

* Return "done" flag from Rendezvous.Meet.
2017-06-13 07:10:13 -07:00
joeduffy
d277dd5800 More progress on pulumi/lumi#90
This change refactors a number of aspects of the CLI's treatment of
steps, in line with the new scheme, and a number of other miscellaneous
and minor fixes.  It also regenerates all RPC code impacted by recent renames.
2017-06-13 07:10:13 -07:00
joeduffy
d044720045 Make more progress on the new deployment model
This change restructures a lot more pertaining to deployments, snapshots,
environments, and the like.

The most notable change is that the notion of a deploy.Source is introduced,
which splits the responsibility between the deploy.Plan -- which simply
understands how to compute and carry out deployment plans -- and the idea
of something that can produce new objects on-demand during deployment.

The primary such implementation is evalSource, which encapsulates an
interpreter and takes a package, args, and config map, and proceeds to run
the interpreter in a distinct goroutine.  It synchronizes as needed to
poke and prod the interpreter along its path to create new resource objects.

There are two other sources, however.  First, a nullSource, which simply
refuses to create new objects.  This can be handy when writing isolated
tests but is also used to simulate the "empty" environment as necessary to
do a complete teardown of the target environment.  Second, a fixedSource,
which takes a pre-computed array of objects, and hands those, in order, to
the planning engine; this is mostly useful as a testing technique.

Boatloads of code is now changed and updated in the various CLI commands.

This further chugs along towards pulumi/lumi#90.  The end is in sight.
2017-06-13 07:10:13 -07:00
joeduffy
6b2408e086 Rewrite plans and deployments
This change guts the deployment planning and execution process, a
necessary component of pulumi/lumi#90.

The major effect of this change is that resources are actually
connected to the live objects, instead of being snapshots taken at
inopportune moments in time.
2017-06-13 07:10:13 -07:00
joeduffy
c53ddeb678 Overhaul resources, planning, and environments
This change, part of pulumi/lumi#90, overhauls quite a bit of the
core resource, planning, environments, and related areas.

The biggest amount of movement comes from the splitting of pkg/resource
into multiple sub-packages.  This results in:

- pkg/resource: just the core resource data structures.

- pkg/resource/deployment: all planning and deployment logic.

- pkg/resource/environment: all environment, configuration, and
      serialized checkpoint structures and logic.

- pkg/resource/plugin: all dynamically loaded analyzer and
      provider logic, including the actual loading and RPC mechanisms.

This also splits the resource abstraction up.  We now have:

- resource.Resource: a shared interface.

- resource.Object: a resource that is connected to a live object
      that will periodically observe mutations due to ongoing
      evaluation of computations.  Snapshots of its state may be
      taken; however, this is purely a "pre-planning" abstraction.

- resource.State: a snapshot of a resource's state that is frozen.
      In other words, it is no longer connected to a live object.
      This is what will store provider outputs (ID and properties),
      and is what may be serialized into a deployment record.

The branch is in a half-baked state as of this change; more changes
are to come...
2017-06-13 07:10:13 -07:00
Luke Hoban
9bd441be05 Support for nested lambdas and node_modules
LumiJS lambdas can now be serialized when they include calls to other LumiJS lambdas.  The chain of lambda dependencies is jointly serialized into the target Lambda.

Also, LumiJS lambdas now include `node_modules` automatically in the AWS Lambda, ensuring the the runtime execution environment more closely matches the deployment time environment.

An early version of the gh-cicd example supporting #134 is added which uses these capabilities, currently including a mocked GitHub resource provider.
2017-06-12 10:15:20 -07:00
Britton Forsyth
69e4834f63 Merge branch 'master' into gometalinter 2017-06-09 14:34:51 -07:00
Britton Forsyth
8c8f68d4ae Added specified changes 2017-06-09 12:51:31 -07:00
Britton Forsyth
3066fcda78 Implemented suggested edits 2017-06-08 11:44:16 -07:00
Britton Forsyth
7457cadf58 Fixed various additional linting issues 2017-06-08 10:21:17 -07:00
joeduffy
2ab2b09474 Introduce object resources
This change slightly refactors the way resources are created and
implemented.  We now have two implementations of the Resource interface:

* `resource` (in resource_value.go), which is a snapshot of a resource's
  state.  All values are resolved and there is no live reference to any
  heap state or objects.  This will be used when serializing and/or
  deserializing snapshots of deployments.

* `objectResource` (in resource_object.go), which is an implementation
  of the Resource interface that wraps an underlying, live runtime object.
  This currently introduces no functional difference, as fetching Inputs()
  amounts to taking a snapshot of the full state.  But this at least
  gives us a leg to stand on in making sure that output properties are
  read at the right times during evaluation.

This is a fundamental part of pulumi/lumi#90.
2017-06-08 09:26:06 -07:00
joeduffy
027fe0766c Rename computed/output's eventual property to element
This rename now ensures the name mirrors that in the type/symbol layer.
2017-06-08 06:48:23 -07:00
joeduffy
dce2fae4c2 Track implicated objects
This change begins to track objects that are implicated in the
creation of computed values.  This ultimately translates into the
resource URNs which are used during dependency analysis and
serialization.  This is part of pulumi/lumi#90.
2017-06-08 06:46:28 -07:00
joeduffy
fd719d64cd Fix pulumi/lumi#198
This change fixes the serialization of resource properties during
deployment checkpoints.  We erroneously serialized empty arrays and
empty maps as though they were nil; instead, we want to keep them
serialized as non-nil empty collections, since the presence of a
value might be semantically meaningful.  (We still skip nils.)

Also added some test cases.
2017-06-06 16:42:14 -07:00
joeduffy
ec2b964daa Do an initial pass over TODOs
This scrubs about 80% of our TODOs, as part of pulumi/lumi#212.
The remaining 20% will come shortly.
2017-06-05 18:11:51 -07:00
joeduffy
db99092334 Implement mapper.Encode "for real"
This change implements `mapper.Encode` "for real" (that is, in a way
that isn't a complete embarrassment).  It uses the obvious reflection
trickery to encode a tagged struct and its values as a JSON-like
in-memory map and collection of keyed values.

During this, I took the opportunity to also clean up a few other things
that had been bugging me.  Namely, the presence of `mapper.Object` was
always error prone, since it isn't a true "typedef" in the sence that
it carries extra RTTI.  Instead of doing that, let's just use the real
`map[string]interface{}` "JSON-map-like" object type.  Even better, we
no longer require resource providers to deal with the mapper
infrastructure.  Instead, the `Check` function can simply return an
array of errors.  It's still best practice to return field-specific errors
to facilitate better diagnostics, but it's no longer required; and I've
added `resource.NewFieldError` to eliminate the need to import mapper.

As of this change, we can also consistently emit RPC structs with `lumi`
tags, rather than `lumi` tags on the way in and `json` on the way out.

This completes pulumi/lumi#183.
2017-06-05 17:49:00 -07:00
joeduffy
87004a124e Store both input and output properties distinctly
This changes the resource model to persist input and output properties
distinctly, so that when we diff changes, we only do so on the programmer-
specified input properties.  This eliminates problems when the outputs
differ slightly; e.g., when the provider normalizes inputs, adds its own
values, or fails to produce new values that match the inputs.

This change simultaneously makes progress on pulumi/lumi#90, by beginning
tracking the resource objects implicated in a computed property's value.

I believe this fixes both #189 and #198.
2017-06-04 19:24:48 -07:00
joeduffy
f552832a7a Alter diag.Message to discourage format mistakes
This change alters diag.Message to not format strings and, instead,
encourages developers to use the Infof, Errorf, and Warningf varargs
functions.  It also tests that arguments are never interepreted as
format strings.
2017-06-02 18:37:28 -07:00
joeduffy
ff37f0b8f9 Fix two lint issues that crept in 2017-06-02 09:05:10 -07:00
joeduffy
e1673dfdb0 Add some basic plan tests 2017-06-02 08:53:40 -07:00
joeduffy
ce35fc78cf Add some property diff tests 2017-06-01 15:36:22 -07:00
Luke Hoban
5358080ca6 Output property improvements for AWS Function and Role
The AssumeRolePolicyDocument property returned by the AWS IAM GetRole API returns
a URL-encoded JSON string, so we need to decode this before JSON unmarshalling.

The Code property returned by AWS Lambda GetFunction provides a pre-signed S3 URL,
which changes on each call, and is of a different format to what is provided by the user.
For now, we'll not store this back into the Function object.

Add additional output properties to AWS Lambda Function that are stable values returned
from GetFunction.

Also corrects a gap where some property delete operations were not being correctly reported.
2017-06-01 15:04:37 -07:00
joeduffy
b07056ab10 Create a plan plugin host
This is a minor refactoring to introduce a ProviderHost interface
that is associated with the context and can be swapped in and out for
custom plugin behavior.  This is required to write tests that mock
certain aspects, like loading packages from the filesystem.

In theory, this change incurs zero behavioral changes.
2017-06-01 11:41:24 -07:00
joeduffy
7b5f9df917 Make updates work in the face of output properties
This change fixes up a few things so that updates correctly deal
with output properties.  This involves a few things:

    1) All outputs stored on the pre snapshot need to get propagated
       to the post snapshot during planning at various points.  This
       ensures that the diffing logic doesn't need to be special cased
       everywhere, including both the Lumi and the provider sides.

    2) Names are changed to "input" properties (using a new `lumi` tag
       option, `in`).  These are properties that providers are expected
       to know nothing about, which we must treat with care during diffs.

    3) We read back properties, via Get, after doing an Update just like
       we do after performing a Create.  This ensures that if an update
       has a cascading impact on other properties, it will be detected.

    4) Inspecting a change, prior to updating, must be done using the
       computed property set instead of the real one.  This is to avoid
       mutating the resource objects ahead of actually applying a plan,
       which would be wrong and misleading.
2017-06-01 10:09:52 -07:00
joeduffy
b5df277815 Fix a few merges when this branch hit master 2017-06-01 08:51:33 -07:00
joeduffy
23493be8af Classify output properties as adds too 2017-06-01 08:39:48 -07:00
joeduffy
e84c2d9388 Remember output properties in snapshot records
This change remembers which properties were computed as outputs,
or even just read back as default values, during a deployment.  This
information is required in the before/after comparison in order to
perform an intelligent diff that doesn't flag, for example, the absence
of "default" values in the after image as deletions (among other things).
As I was in here, I also cleaned up the way the provider interface
works, dealing with concrete resource types, making it feel a little
richer and less like we're doing in-memory RPC.
2017-06-01 08:39:48 -07:00
joeduffy
08ca40c6c6 Unmap properties on the receive side
This change makes progress on a few things with respect to properly
receiving properties on the engine side, coming from the provider side,
of the RPC boundary.  The issues here are twofold:

    1. Properties need to get unmapped using a JSON-tag-sensitive
       marshaler, so that they are cased properly, etc.  For that, we
       have a new mapper.Unmap function (which is ultra lame -- see
       pulumi/lumi#138).

    2. We have the reverse problem with respect to resource IDs: on
       the send side, we must translate from URNs (which the engine
       knows about) and provider IDs (which the provider knows about);
       similarly, then, on the receive side, we must translate from
       provider IDs back into URNs.

As a result of these getting fixed, we can now properly marshal the
resulting properties back into the resource object during the plan
execution, alongside propagating and memoizing its ID.
2017-06-01 08:39:48 -07:00
joeduffy
87ad371107 Only flow logging to plugins if --logflow
The change to flow logging to plugins is nice, however, it can be
annoying because all writes to stderr are interepreted on the Lumi
side as errors.  After this change, we will only flow if
--logflow is passed, e.g. as in

    $ lumi --logtostderr --logflow -v=9 deploy ...
2017-06-01 08:37:56 -07:00
joeduffy
0a72d5360a Modify provider creates; use get for outs
This change modifies the existing resource provider RPC interface slightly.
Instead of the Create API returning the bag of output properties, we will
rely on the Get API to do so.  As a result, this change takes an initial
whack at implementing Get on all existing AWS resources.  The Get API needs
to return a fully populated structure containing all inputs and outputs.

Believe it or not, this is actually part of pulumi/lumi#90.

This was done because just returning output properties is insufficient.
Any input properties that weren't supplied may have default values, for
example, and it is wholly reasonable to expect Lumi scripts to depend on
those values in addition to output values.

This isn't fully functional in its current form, because doing this
change turned up many other related changes required to enable output
properties.  For instance, at the moment resource properties are defined
in terms of `resource.URN`s, and yet unfortunately the provider side
knows nothing of URNs (instead preferring to deal in `resource.ID`s).
I am going to handle that in a subsequent isolated change, since it will
have far-reaching implications beyond just modifying create and get.
2017-06-01 08:36:43 -07:00
joeduffy
7f98387820 Distinguish between computed and output properties
This change introduces the notion of a computed versus an output
property on resources.  Technically, output is a subset of computed,
however it is a special kind that we want to treat differently during
the evaluation of a deployment plan.  Specifically:

* An output property is any property that is populated by the resource
  provider, not code running in the Lumi type system.  Because these
  values aren't available during planning -- since we have not yet
  performed the deployment operations -- they will be latent values in
  our runtime and generally missing at the time of a plan.  This is no
  problem and we just want to avoid marshaling them in inopportune places.

* A computed property, on the other hand, is a different beast altogehter.
  Although true one of these is missing a value -- by virtue of the fact
  that they too are latent values, bottoming out in some manner on an
  output property -- they will appear in serializable input positions.
  Not only must we treat them differently during the RPC handshake and
  in the resource providers, but we also want to guarantee they are gone
  by the time we perform any CRUD operations on a resource.  They are
  purely a planning-time-only construct.
2017-06-01 08:36:43 -07:00
joeduffy
d79c41f620 Initial support for output properties (1 of 3)
This change includes approximately 1/3rd of the change necessary
to support output properties, as per pulumi/lumi#90.

In short, the runtime now has a new hidden type, Latent<T>, which
represents a "speculative" value, whose eventual type will be T,
that we can use during evaluation in various ways.  Namely,
operations against Latent<T>s generally produce new Latent<U>s.

During planning, any Latent<T>s that end up in resource properties
are transformed into "unknown" property values.  An unknown property
value is legal only during planning-time activities, such as Check,
Name, and InspectChange.  As a result, those RPC interfaces have
been updated to include lookaside maps indicating which properties
have unknown values.  My intent is to add some helper functions to
make dealing with this circumstance more correct-by-construction.

For now, using an unresolved Latent<T> in a conditional will lead
to an error.  See pulumi/lumi#67.  Speculating beyond these -- by
supporting iterative planning and application -- is something we
want to support eventually, but it makes sense to do that as an
additive change beyond this initial support.  That is a missing 1/3.

Finally, the other missing 1/3rd which will happen much sooner
than the rest is restructuing plan application so that it will
correctly observe resolution of Latent<T> values.  Right now, the
evaluation happens in one single pass, prior to the application, and
so Latent<T>s never actually get witnessed in a resolved state.
2017-06-01 08:32:12 -07:00
Luke Hoban
9531483e19 Add tests for AWS DynamoDB Table provider 2017-05-31 17:06:16 -07:00
joeduffy
ab6e2466c7 Flow logging information to plugins
This change flows --logtostderr and -v=x settings to any dynamically
loaded plugins so that running Lumi's command line with these flags
will also result in the plugins logging at the requested levels.  I've
found this handy for debugging purposes.
2017-05-30 10:19:33 -07:00
Luke Hoban
7f8b1e59c1 Support for lambdas (#158)
Resolves #137.

This is an initial pass for supporting JavaScript lambda syntax for defining an AWS Lambda Function.

A higher level API for defining AWS Lambda Function objects `aws.lambda.FunctionX` is added which accepts a Lumi lambda as an argument, and uses that lambda to generate the AWS Lambda Function code package.

LumiJS lambdas are serialized as the JavaScript text of the lambda body, along with a serialized version of the environment that is deserialized at runtime and used as the context for the body of the lambda.

Remaining work to further improve support for lambdas is being tracked in #173, #174, #175, and #177.
2017-05-25 16:55:14 -07:00
Luke Hoban
35a41f9e4a Support Update on IAM Role and Lambda Function 2017-05-22 22:57:55 -07:00
joeduffy
3bad4fde98 Disable storing output properties
The storing of output properties won't work correctly until pulumi/lumi#90
is completed.  The update logic sees properties that weren't supplied by the
developer and thinks this means an update is required; this is easy to fix
but better to just roll into the overall pending change that will land soon.
2017-05-22 13:20:57 -07:00
Luke Hoban
0f99762e2e Add AWS Elastic Beanstalk resource providers (#154)
Includes support for:
* Application
* ApplicationVersion
* Environment
2017-05-21 21:45:28 -07:00
joeduffy
4108c51549 Reclassify Lumi under the Apache 2.0 license
This is part of pulumi/lumi#147.
2017-05-18 14:51:52 -07:00
joeduffy
dafeb77dff Rename Coconut to Lumi
This is part of pulumi/coconut#147.

After it has landed, I will rename the repo on GitHub.
2017-05-18 11:38:28 -07:00
joeduffy
eee0f3b717 Fix some golint warnings 2017-05-13 20:04:35 -04:00
joeduffy
7d8aadeb1e Implement chronological stable object keys
We need a stable object key enumeration order and we might as well leverage
ECMAScript's definition for this.  As of ES6, key ordering is specified; see
https://tc39.github.io/ecma262/#sec-ordinaryownpropertykeys.

I haven't fully implemented the "numbers come first part" (we can do this as
soon as we have support for Object.keys()), but the chronological part works.
2017-05-06 16:09:49 -07:00
joeduffy
1e67162331 Fix a couple silly mistakes 2017-05-04 09:53:52 -07:00
joeduffy
6902d7e1b2 Update AWS Lambdas to take archives, not assets 2017-05-01 09:38:23 -07:00
joeduffy
335ea01275 Implement archives
Our initial implementation of assets was intentionally naive, because
they were limited to single-file assets.  However, it turns out that for
real scenarios (like lambdas), we want to support multi-file assets.

In this change, we introduce the concept of an Archive.  An archive is
what the term classically means: a collection of files, addressed as one.
For now, we support three kinds: tarfile archives (*.tar), gzip-compressed
tarfile archives (*.tgz, *.tar), and normal zipfile archives (*.zip).

There is a fair bit of library support for manipulating Archives as a
logical collection of Assets.  I've gone to great length to avoid making
copies, however, sometimes it is unavoidable (for example, when sizes
are required in order to emit offsets).  This is also complicated by the
fact that the AWS libraries often want seekable streams, if not actual
raw contiguous []byte slices.
2017-04-30 12:37:24 -07:00
joeduffy
fe93f5e76f Strongly type resource IDs in the IDL/RPC/Providers
This change simply uses the `resource.ID` type in all the places
where it belongs, rather than using `string`-typed resource IDs.
2017-04-29 13:27:39 -07:00
joeduffy
40ac0a1d2b Support assets in the IDL 2017-04-28 12:07:49 -07:00
joeduffy
47ef3f673b Rename PreviewUpdate (again)
Unfortunately, this wasn't a great name.  The old one stunk, but the
new one was misleading at best.  The thing is, this isn't about performing
an update -- it's about NOT doing an update, depending on its return value.
Further, it's not just previewing the changes, it is actively making a
decision on what to do in response to them.  InspectUpdate seems to convey
this and I've unified the InspectUpdate and Update routines to take a
ChangeRequest, instead of UpdateRequest, to help imply the desired behavior.
2017-04-27 11:18:49 -07:00
joeduffy
507a2609a7 Add an initial implementation of CIDLC
This is an initial implementation of the Coconut IDL Compiler (CIDLC).
This is described further in
https://github.com/pulumi/coconut/blob/master/docs/design/idl.md,
and the work is tracked by coconut/pulumi#133.

I've been kicking the tires with this locally enough to checkpoint the
current version.  There are quite a few loose ends not yet implemented,
most of them minor, with the exception of the RPC stub generation which
I need to flesh out more before committing.
2017-04-25 15:05:51 -07:00
joeduffy
8c58950639 Tolerate nils in output property marshaling 2017-04-25 14:04:22 -07:00
joeduffy
1edced2d4b Add the ability to convert structs to PropertyMaps 2017-04-21 15:27:32 -07:00
joeduffy
d6abea728c Add outputs to the Create provider's return
In order to support output properties (pulumi/coconut#90), we need to
modify the Create gRPC interface for resource providers slightly.  In
addition to returning the ID, we need to also return any properties
computed by the AWS provider itself.  For instance, this includes ARNs
and IDs of various kinds.  This change simply propagates the resources
but we don't actually support reading the outputs just yet.
2017-04-21 14:15:06 -07:00
joeduffy
0b6e262b46 Rename resource provider methods
This change renames two provider methods:

    * Read becomes Get.

    * UpdateImpact becomes PreviewUpdate.

These just read a whole lot nicer than the old names.
2017-04-20 14:09:00 -07:00
joeduffy
53e7bfbb86 Rearrange the way stderr/stdout is handled for plugins
The order of operations for stderr/stdout monitoring with plugins
managed to hide some important errors.  For example, if something
was written to stderr *before* the port was parsed from stdout, a
very possible scenario if the plugin fails before it has even
properly sarted, then we would silently drop the stderr on the floor
leaving behind no indication of what went wrong.  The new ordering
ensures that stderr is never ignored with some minor improvements
in the case that part of the port is parsed from stdout but it
ultimately ends in an error (e.g., if an EOF occurs prematurely).
2017-04-19 15:01:04 -07:00
joeduffy
f429bc6a0c Use github.com/pkg/errors for errors
This change moves us over to the github.com/pkg/errors package to
encourage the addition of more context associated with failures.
2017-04-19 14:46:50 -07:00