Commit graph

838 commits

Author SHA1 Message Date
joeduffy 324d3dad85 Generate module names with file parts 2017-09-11 17:08:41 -07:00
joeduffy 9c7f6b678c Bring LUMIDL up to code
This gets LUMIDL to generate code in the new way.
2017-09-11 16:58:25 -07:00
joeduffy f9995159c6 Fix a handful of things, mostly logging
* Initialize the diganostics logger with opts.Debug when doing
  a Deploy, like we do Plan.

* Don't spew leaked promises if there were Log.errors.

* Serialize logging RPC calls so that they can't appear out of order.

* Print stack traces in more places and, in particular, remember
  the original context for any errors that may occur asynchronously,
  like resource registration and calls to mapValue.

* Include origin stack traces generally in more error messages.

* Add some more mapValue test cases.

* Only undefined-propagate mapValue values during dry-runs.
2017-09-09 13:43:51 -07:00
joeduffy f74e9e04bd Ensure we fail during deploys
If an error gets emitted during a deployment, we should fail.
This also reenables colors that seems to have gotten lost somewhere.
2017-09-09 12:42:04 -07:00
joeduffy 8aba3aae12 Upgrade gRPC to 1.6.0; use full addresses
This change upgrades gRPC to 1.6.0 to pick up a few bug fixes.

We also use the full address for gRPC endpoints, including the
interface name, as otherwise we pick the wrong interface on Linux.
2017-09-09 07:37:10 -07:00
joeduffy 67e5750742 Fix a bunch of Linux issues
There's a fair bit of clean up in here, but the meat is:

* Allocate the language runtime gRPC client connection on the
  goroutine that will use it; this eliminates race conditions.

* The biggie: there *appears* to be a bug in gRPC's implementation
  on Linux, where it doesn't implement WaitForReady properly.  The
  behavior I'm observing is that RPC calls will not retry as they
  are supposed to, but will instead spuriously fail during the RPC
  startup.  To work around this, I've added manual retry logic in
  the shared plugin creation function so that we won't even try
  to use the client connection until it is in a well-known state.
  pulumi/pulumi-fabric#337 tracks getting to the bottom of this and,
  ideally, removing the work around.

The other minor things are:

* Separate run.js into its own module, so it doesn't include
  index.js and do a bunch of random stuff it shouldn't be doing.

* Allow run.js to be invoked without a --monitor.  This makes
  testing just the run part of invocation easier (including
  config, which turned out to be super useful as I was debugging).

* Tidy up some messages.
2017-09-08 15:11:09 -07:00
joeduffy ebc3bf1dd0 Reap the language host process 2017-09-07 15:03:48 -07:00
joeduffy 0d2f84c194 Fix github.com/pkg/errors reference 2017-09-07 07:25:08 -07:00
joeduffy f0389799d8 Convey errors if any error messages occurred 2017-09-06 09:35:35 -07:00
joeduffy 3164572b6e Fix some free variable capture logic
* Use `global.hasOwnProperty(ident)`, rather than `global[ident] !== undefined`,
  to avoid classifying references to globals as free variables.  Surprise(!!),
  the prior logic wouldn't work for `undefined` itself... 😒

* Expand this check to include the built-in Node.js module variables, namely
  `__dirname`, `__filename`, `exports`, `module`, and `require`, so that
  references to them don't get classified as serializable free variables either.

* Place catch variables in scope, so that `catch (err) { ... }` won't yield
  free variables for references to `err` within `...`.

* Place recursive function definitions into the top-level `var`-like scope of
  variables so that we don't consider references to them free.

* Harden all error pathways in the native C++ add-on so that we terminate
  anytime an exception is in-flight, rather than limping along and making
  things worse...
2017-09-05 15:21:14 -07:00
joeduffy f2d53459eb Add the notion of stable states
If a resource's planning operation is to do nothing, we can safely
assume that all of its properties are stable.  This can be used during
planning to avoid cascading updates that we know will never happen.
2017-09-05 10:01:00 -07:00
joeduffy f3cf73d790 Change plugin prefixes to "pulumi-" 2017-09-04 11:35:21 -07:00
joeduffy fe66a0eba7 Use the new URN during creates 2017-09-04 11:35:21 -07:00
joeduffy 77bbf443bc Synchronize with the resource channel properly 2017-09-04 11:35:21 -07:00
joeduffy f718ab6501 Add a runtime.Log class
This change adds the ability to perform runtime logging, including
debug logging, that wires up to the Pulumi Fabric engine in the usual
ways.  Most stdout/stderr will automatically go to the right place,
but this lets us add some debug tracing in the implementation of the
runtime itself (and should come in handy in other places, like perhaps
the Pulumi Framework and even low-level end-user code).
2017-09-04 11:35:21 -07:00
joeduffy a13a83b067 Pass the monitor address correctly to language plugins 2017-09-04 11:35:21 -07:00
joeduffy 9f160a7f91 Configure providers at well-defined points
As explained in pulumi/pulumi-fabric#293, we were a little ad-hoc in
how configuration was "applied" to resource providers.

In fact, config wasn't ever communicated directly to providers; instead,
the resource providers would simply ask the engine to read random heap
locations (via tokens). Now that we're on a plan where configuration gets
handed to the program at startup, and that's that, and where generally
speaking resource providers never communicate directly with the language
runtime, we need to take a different approach.

As such, the resource provider interface now offers a Configure RPC
method that the resource planning engine will invoke at the right
times with the right subset of configuration variables filtered to
just that provider's package.  This fixes pulumi/pulumi#293.
2017-09-04 11:35:21 -07:00
joeduffy 375fc399c3 Eliminate yarn version printing
Apparently yarn requires a TTY to print the version.  (No idea why.)
This wasn't an essential change, so I'll just nix it for now.
2017-09-04 11:35:21 -07:00
joeduffy 70d0fac1c0 Simplify resource provider RPC interface
This change simplifies the provider RPC interface slightly:

1) Eliminate Get.  We really don't need it anymore.  There are
   several possibly-interesting scenarios down the road that may
   demand it, but when we get there, we can consider how best to
   bring this back.  Furthermore, the old-style Get remains mostly
   incompatible with Terraform anyway.

2) Pass URNs, not type tokens, across the RPC boundary.  This gives
   the provider access to more interesting information: the type,
   still, but also the name (which is no longer an object property).
2017-09-04 11:35:21 -07:00
joeduffy 8f742e1cd0 Run yarn install before integration tests 2017-09-04 11:35:21 -07:00
joeduffy 590e9e539b Rename Lumi.yaml to Pulumi.yaml
And also eliminate lots of accumulated cruft around "packfiles", etc.
in the workspace code.
2017-09-04 11:35:21 -07:00
joeduffy 1df1b6d572 Get integration tests passing
This makes a few tweaks to get the integration tests passing:

* Add `runtime: nodejs` to the minimal example's `Lumi.yaml` file.

* Remove usage of `@lumi/lumirt { printf }` and just use `console.log`.

* Remove calls to `lumijs` in the integration test framework and
  the minimal example's package.json.  Instead, we just run
  `yarn run build`, which itself internally just invokes `tsc`.

* Add package validation logic and eliminate the pkg/compiler/metadata
  library, in favor of the simpler code in pkg/engine.

* Simplify the Node.js langhost plugin CLI, and simply take an
  argument rather than requiring required and optional --flags.

* Use a default path of "." if the program path isn't provided.  This
  is a legal scenario if you've passed a pwd and just want to load
  the package's default module ("./index.js" or whatever main says).

* Add an executable script, lumi-langhost-nodejs, that fires up the
  `bin/cmd/langhost/index.js` file to serve the Node.js language plugin.
2017-09-04 11:35:21 -07:00
joeduffy 9599ea2e55 Get planning engine unit tests running again
We now build and run cleanly locally (for unit tests).  The
integration tests are still on the floor at the moment.
2017-09-04 11:35:21 -07:00
joeduffy f189c40f35 Wire up Lumi to the new runtime strategy
🔥 🔥 🔥  🔥 🔥 🔥

Getting closer on #311.
2017-09-04 11:35:21 -07:00
Luke Hoban 7425c4d106 Avoid concurrent map updates in default sink
Fixes #324
2017-08-31 14:36:19 -07:00
Matt Ellis 24ac95c998 Adopt github.com/pkg/errors in a few more places 2017-08-31 10:28:20 -07:00
Matt Ellis cda0fd9bca Add godoc comments 2017-08-31 10:28:02 -07:00
Matt Ellis be13c39586 Adopt EnvironmentProvider interface in engine
The existing implementation of the interface (backed by the file
system) has moved into cmd/lumi. The deployment service will start to
provide its own version.
2017-08-30 16:47:33 -07:00
Matt Ellis 34d52cc527 Add EnvironmentProvider interface 2017-08-29 18:47:32 -07:00
Matt Ellis fa033e985e Adopt error at API boundary 2017-08-29 18:47:29 -07:00
Matt Ellis a4c97d7225 Have saveEnv always override an existing environment
`saveEnv` had a flag which would prevent an environment from being
overwritten if it already existed, which was only used by `lumi env
init`. Refactor the code so the check is done inside `lumi` instead of
against this API. We don't need this functionality for the service and
so requiring support for this at the API boundary for environments
feels like a bad idea.
2017-08-29 18:05:42 -07:00
Matt Ellis 871b8ba962 Remove ability to specify a file name when saving an environment
We'd like to abstract out environment CRUD operations and I'd prefer
not to have to bake in the conspect of a file name like thing in the
abstraction. Since we were not really using this feature many places,
let's just get rid of it.
2017-08-29 18:00:28 -07:00
Matt Ellis 3becc6a4f4 Adopt glog for a few functions
The implementation of these functions will be moving out of the engine
and into `lumi` itself, it's a little easier if we move away from
spewing stuff to the diag interface, so just use glog instead (which
`lumi` already uses for logging)
2017-08-29 17:51:58 -07:00
Luke Hoban 24393cfe7b Read path assets into memory instead of holding open file handles
This helps avoid running into file handle limits when creating archives including thousands of node_modules files.

Tracking a more complete fix through all other codepaths related to assets as part of #325.
2017-08-29 13:33:02 -07:00
joeduffy 627d97d83f Close open Blobs
This change ensures we close all Blobs in the asset/archive logic.
In particular, the archive.Read function returns a map of files to
Blobs and after we are done copying the contents we must ensure
that we invoke Close, otherwise we may leak file handles, sockets,
and so on.  This may or may not be the culprit to the "too many
files open" errors we are hitting while deploying the M5 bits.
2017-08-28 18:52:51 -07:00
Matthew Riley fc17c146fd Merge pull request #322 from pulumi/log-plan-error
Bubble up errors from Plan.Apply. Should help debug #321.
2017-08-27 10:25:09 -07:00
Matthew Riley 63df03c556 Bubble up errors from Plan.Apply 2017-08-27 00:38:17 -07:00
Matt Ellis dee8114556 Add ReplaceConfig to the engine
The deployment service will use this before a deployment to copy the
config from the database to the local files that lumi will use when
deploying.
2017-08-25 10:52:44 -07:00
Matt Ellis 971b36b21a Fix lint issues 2017-08-24 18:09:37 -07:00
Matt Ellis b7388fa99a Clean up Destroy API boundary 2017-08-24 18:09:37 -07:00
Matt Ellis 73d64dc686 Fix prompt for env name in lumi destory 2017-08-24 18:09:37 -07:00
Matt Ellis 865422567c Alow multiple instances of engine.Engine
This refactors the engine so all of the APIs on it are instance
methods on the type instead of raw methods that float around and use
data from a global engine.

A mechcanical change as we remove the global `E` and then make
anything that interacted with that in pkg/engine to be an instance
method and the dealing with the fallout.
2017-08-24 18:09:37 -07:00
Matt Ellis be586a1fbf Wire up sink to custom stdout and stderr 2017-08-24 18:00:46 -07:00
Matt Ellis 14c2474e6f Remove use of ambient cmdutil.Diag(), instead consult the engine 2017-08-24 18:00:46 -07:00
Matt Ellis 5ae02ad581 Remove use of ambient stdout/stderr streams
Instead of talking directly to stdout or stderr via methods on fmt,
indirect through an Engine type (presently a global, but soon to
change) to allow control of where the streams actually end up.
2017-08-24 18:00:46 -07:00
Matt Ellis cdacc46931 Clean up the engine API a small amount
Prevously, we would throw raw args arrays across the interface and the
engine would do some additional parsing. Clean this up so we don't do
that and all the parsing stays in `lumi`
2017-08-24 18:00:46 -07:00
Matt Ellis a6eabdc34b Move a bunch of code around
Move most of the guts of `lumi` into the newly created `engine`
package.
2017-08-24 18:00:46 -07:00
Luke Hoban c837aa6024 Fix indexing outside array length in Lumi runtime 2017-08-19 14:39:50 -07:00
Luke Hoban 575102c704 Add support for serializaing captured arrays
Previous logic had assumed arrays were treated like objects
in the runtime (like in ECMAScript), but they are a unique
value kind in current Lumi runtime, so must be handled separately.
2017-08-19 13:34:47 -07:00
Luke Hoban e5c4827975 Fix gometalinter issue 2017-08-17 22:32:25 -07:00
Luke Hoban e5405accab Ensure ExtraRuntimeValidation is optional 2017-08-17 22:24:15 -07:00
Luke Hoban 85a95f9353 Add ability to run runtime validation in test harness
Adds an `ExtraRuntimeValidation` hook to the test harness.

This runs after the test app is deployed, and can be used to test publically
exposed endpoints on the example to validate additional runtime correctness
of the Lumi app under test.
2017-08-17 22:12:58 -07:00
Luke Hoban 127519129b Suport ternary operator
Also allows relational operators to take string arguments.
2017-08-10 13:21:59 -07:00
joeduffy a626dcf6a3 Prettify the CLI in a few places
This changes a few things in the CLI, mostly just prettying it up:

    * Label all steps more clearly with the kind of step.  Also
      unify the way we present this during planning and deployment.

    * Summarize the changes that *did not* get made just as clearly
      as those that did.  In other words, stuff like this:

        info: 2 resources changed:
            +1 resource created
            -1 resource deleted
            5 resources unchanged

      and

        info: no resources required
            5 resources unchanged

    * Always print output properties when they are pertinent.
      This includes creates, replacements, and updates.

    * Show replacement creates and deletes very distinctly.  The
      create parts show up minty green and the delete parts show up
      rosey red.  These are the "physical" steps, compared to the
      "logical" step of replacement (which remains marigold).

      I still don't love where we are here.  The asymmetry between
      planning and deployment bugs me, and could be surprising.
      ("Hey, my deploy doesn't look like my plan!")  I don't know
      what developers will want to see here and I feel like in
      general we are spewing far too much into the CLI to make it
      even useful for anything but diagnosing failures afterwards.

      I propose that we should do a deep dive on this during the
      CLI epic, pulumi/pulumi-service#2.

This resolves pulumi/pulumi-fabric#305.
2017-08-06 10:05:51 -07:00
joeduffy d7273b1e9e Perform updates during integration tests
This adds the ability to apply "edits" during integration tests, so
that we can test update and replacement paths.

This fixes pulumi/pulumi-fabric#310.
2017-08-05 17:49:48 -07:00
joeduffy ed58a2cbd1 Run empty plan/deploy in integration tests
This change tests that a plan and deploy immediately following another
deploy, when no edits have taken place, correctly results in no action.

I also cleaned up a few things in the code, like using fmt.Printf rather
than fmt.Fprintf(os.Stdout, ...), to clean up error paths, giving the
package a slightly shorter name, and adding missing copyright headers.

This is part of pulumi/pulumi-fabric#310.
2017-08-05 11:37:14 -07:00
joeduffy ff0eb81944 Export urnName constants
This avoids us needing to hard-code the urnName property in
various tools, in case we ever need to change it again down the road.
2017-08-05 08:32:50 -07:00
joeduffy 007c2f216d Rename name property to urnName
At the moment, we permit resources to carry a name, which the
engine uses as part of URN creation.  Unfortunately, the property
"name" has a very high chance of conflicting with meaningful
user-authored properties.  And furthermore this sort of name,
although key to creating URNs which are core to how the overall
system performs deployments and manages resources, are seldom
used programmatically in Lumi programs.  As a result, it's a real
nuisance that we stole the good name.

This change renames that property from "name" to "urnName".  This
not only has a lower likelihood of conflicting, but it also looks
reasonable sitting alongside the "urn" property.  In fact, in some
future universe, after some of the upcoming runtime changes, we
may not even need the name property on the objects whatsoever.

I had originally toyed with the idea of eliminating the Resource
versus NamedResource distinction.  This is certainly simpler and
remains a possibility.  I didn't do that right now, however, because
the flexibility of letting resource providers name resources however
they see fit still seems possibly useful.  For example, we keep
talking about whether functions can be auto-named based on hashes.
Until we've run those conversations to ground, I'd hate to do some
work that just needs to be undone in order to enable a scenario that
has a non-trivial likelihood of us wanting to explore.
2017-08-05 08:20:57 -07:00
joeduffy c0e4bdb03b This shouldn't be here!
😬
2017-08-03 18:09:14 -07:00
joeduffy a160741931 Tolerate computed and output properties
We are now freely flowing computed and output properties across the
RPC boundary with providers.  As such, we need to tolerate them in
a few more places.  Namely, mapping to and from regular non-resource
property values, and also when copying RPC resource state back onto
live runtime objects.
2017-08-03 11:01:38 -07:00
joeduffy 7e09bd055a Fix variable shadowing 2017-08-02 15:40:07 -07:00
joeduffy 1df1792d84 Ensure all exit paths call OnDone
If certain early errors occurred, like failing to find a default module
or main entrypoint, we never properly invoked OnDone (or, sometimes,
OnStart, for that matter).  This meant that callers like the eval source
in the deployment engine could end up missing signals; in this particular
case, it led to a failure to signal a rendezvous synchronization object,
which itself led to a hang.

The fix is simple: make sure to call the On* methods in the right places.
I've added tests to probe the interesting paths, including failures.

This fixes pulumi/pulumi-fabric#281.
2017-08-02 15:28:48 -07:00
joeduffy a9939c8e6a Remove workaround for pulumi/pulumi-fabric#276 2017-08-02 13:31:38 -07:00
joeduffy 35aa6b7559 Rename pulumi/lumi to pulumi/pulumi-fabric
We are renaming Lumi to Pulumi Fabric.  This change simply renames the
pulumi/lumi repo to pulumi/pulumi-fabric, without the CLI tools and other
changes that will follow soon afterwards.
2017-08-02 09:25:22 -07:00
joeduffy 5fb014e53c Explicitly track default properties
This changes the RPC interfaces between Lumi and provider ever so
slightly, so that we can track default properties explicitly.  This
is required to perform accurate diffing between inputs provided by
the developer, inputs provided by the system, and outputs.  This is
particularly important for default values that may be indeterminite,
such as those we use in the bridge to auto-generate unique IDs.
Otherwise, we fail to reapply defaults correctly, and trick the
provider into thinking that properties changed when they did not.

This is a small step towards pulumi/lumi#306, in which we will defer
even more responsibility for diffing semantics to the providers.
2017-07-31 18:26:15 -07:00
joeduffy 300af62747 Fix erroneous assert when unmarshaling computed properties 2017-07-31 11:44:34 -07:00
Luke Hoban 6016146750 Fix up RPC generation to align with recent RPC changes
Fixes #304.
2017-07-25 16:07:10 -07:00
joeduffy d6b6cbf4ff Build/test the minimal Lumi program as a nightly integration test 2017-07-23 12:12:43 -06:00
joeduffy 00442b73b4 Alter the way unknown properties are serialized
This change serializes unknown properties anywhere in the entire
property structure, including deeply embedded inside object maps, etc.

This is now done in such a way that we can recover both the computed
nature of the serialized property, along with its expected eventual
type, on the other side of the RPC boundary.

This will let us have perfect fidelity with the new bridge's view on
computed properties, rather than special casing them on "one side".
2017-07-21 14:00:30 -07:00
joeduffy b8be7fa5e6 Add a handy Copy function to PropertyMap 2017-07-21 14:00:30 -07:00
Luke Hoban 844b2bf1ea Support for string.split
Adds the string.split intrinsic function and places
it on the String protoype object.
2017-07-21 14:00:30 -07:00
Luke Hoban 18c3caf093 Fix continue in loops
Make sure that continue runs post expression
before looping.
2017-07-21 14:00:30 -07:00
Luke Hoban 9243bd5f3f Fix tests after previous commit 2017-07-21 14:00:30 -07:00
joeduffy 40747a94bf Merge inputs with outputs
For Update and Delete operations, we provided just the input state
for a resource.  This is insufficient, because the provider may need
to depend on output state from the Create or prior Update operations.
This change merges the output atop the input during the step application.
2017-07-21 14:00:30 -07:00
joeduffy 4e02105355 Pass old state to the provider's API 2017-07-21 14:00:30 -07:00
joeduffy ae92e68902 Return state as part of Create and Update¬
As part of the bridge bringup, I've discoverd that the property state
returned from Creates does *not* always equal the state that is then
read from calls to Get.  (I suspect this is a bug and that they should
be equivalent, but I doubt it's fruitfal to try and track down all
occurrences of this; I bet it's widespread).  To cope with this, we will
return state from Create and Update, instead of issuing a call to Get.
This was a design we considered to start with and frankly didn't have
a super strong reason to do it the current way, other than that it seemed
elegant to place all of the Get logic in one place.

Note that providers may choose to return nil, in which case we will read
state from the provider in the usual Get style.
2017-07-21 14:00:29 -07:00
joeduffy ceb32b9e2b Add a sig field to Asset/Archive
This change mirrors the dynamic marshaling structure on the static
definition of the Asset and Archive types.  This ensures that they
marshal correctly even when deeply embedded inside other structures.
2017-07-18 09:30:31 -07:00
joeduffy 5bf43743ab Add a rudimentary toString runtime helper
This just works around some limitations that will exist until
we do the runtime work next sprint, and achieve pulumi/lumi#169.
2017-07-17 13:43:59 -07:00
joeduffy 91c90cf2a9 Add diffing logic for assets/archives 2017-07-17 12:11:15 -07:00
joeduffy 002618e605 Add some more asset serialization round-tripping tests 2017-07-17 11:30:10 -07:00
joeduffy 4d708c8567 Fix asset diffing
This change brings the same typed serialization we use for RPC
to the serialization of deployments.  This ensures that we get
repeatable diffs from one deployment to the next.
2017-07-17 10:38:57 -07:00
joeduffy ee1acd365c Add a defaultIfComputed intrinsic
Until we support output-conditional code (pulumi/lumi#170), we
run into cases where we want to make a decision based on a computed
property if it is available, but can't possible know the value.  In
such cases, we get an error ("Unexpected computed value").

In fact, our generated code currently includes client side validation
of properties, since it leads to a(n admittedly only slightly) better
developer experience.  But that triggers the error for required
properties that are computed, basically hosing our ability to plan.

This change introduces a defaultIfComputed intrinsic to the lumirt
library that can be used to work around this.  It takes two arguments,
obj and def, and returns the actual property value obj if it is NOT
computed; if it is computed, def will be substituted in its stead.

The generated code changes to use this.  It's possible it will miss
validation, of course, if a computed property turns out not to match
the precondition.  But this is strictly better than the alternative
of not trying to validate any of them to begin with.
2017-07-16 15:16:48 -07:00
joeduffy bda607abd8 Permit -1 for randlen and maxlen
This allows -1 for randlen and maxlen to use defaults.  The default
behavior is that randlen uses sha1.Size and maxlen is "no max".
2017-07-15 09:59:44 -07:00
joeduffy c61dcb5206 Revert "Rename Lumi resource properties"
This reverts commit c3db70849d.

I've opted to take a new strategy to ensure the bridge properties
don't conflict (with manual renames), similar to the name property.
2017-07-15 09:33:23 -07:00
joeduffy 0f18bb40b2 Tolerate some nils in important places 2017-07-14 18:08:33 -07:00
joeduffy a6caef973a Make assets and archives 1st class
This change recognizes assets and archives as 1st class resource
property values.  This is necessary to support them in the new bridge
work, and lays the foundation for fixing pulumi/lumi#153.

I also took the opportunity to clean up some old cruft in the
resource properties area.
2017-07-14 12:28:43 -07:00
joeduffy c3db70849d Rename Lumi resource properties
This renames the basemost resource properties, id and urn, to
names that are less likely to conflict with properties that real
resources will want to use, pid and upn (provider ID and Universal
Pulumi Name, respectively).

I actually ran into this with the current bridge work.  An alternative
solution would be to require derived resources to pick different names,
however this is unfortunate because usually they are more "user-facing"
than ours.  Another alternative is to not hijack the object properties
at all, but that too is problematic because we use these properties
during the evaluation of plans and deployments.

This seems like a reasonable middle ground.
2017-07-14 08:55:07 -07:00
Luke Hoban 3efcf91933 Addres PR feedback on #288
Move ByName into the shared tokens package.
2017-07-13 23:03:28 -07:00
Luke Hoban e9f6fbeed0 Merge pull request #288 from pulumi/closure-fixes
Closure fixes
2017-07-13 23:00:10 -07:00
joeduffy 539ccc8f04 Add a --debug option to plan, deploy, and destroy
This change introduces a --debug option to the plan, deploy, and
destroy commands.  Unlike --logtostderr, which merely hooks into the
copious Glogging that we perform (and is therefore meant for developers
of the tools themselves and not end users), --debug hooks into the
user-facing debug stream.  This now includes any debug messages coming
from the resource providers as they perform their tasks.
2017-07-13 17:13:19 -07:00
Luke Hoban 31c62ec8e6 Function update improvements
Remove duplicative call to UpdateFunctionConfiguration.

Also ensure that free variables returns stable variable order
to avoid unnecessary lambda updates.
2017-07-13 17:01:11 -07:00
Luke Hoban c6ef2f6c66 Fix golint errors in previous commit 2017-07-13 12:40:10 -07:00
Luke Hoban 92418cf687 Refactor integration testing framework
Generalizes Lumi program validation so that it can be applied
to integration testing for other packages (such as the
pulumi/lumi-platform package examples).
2017-07-13 12:20:57 -07:00
Luke Hoban 0b24ab7070 Serialize lambdas nested in captured objects
Adds support for serializing lambdas which are
properties of captured objects.

For example:

```
let o = {
  f: () => 12
};
let func = new aws.serverless.Function("func", {policies: [aws.iam.AWSLambdaFullAccess]}, (ev, ctx, cb) => {
    console.log(o.f());
    cb(null, null);
});
```
2017-07-12 17:43:17 -07:00
Luke Hoban ac21b5bd82 Add test for serializeClosure 2017-07-12 15:19:20 -07:00
joeduffy a70790b81a Add key replacement functions
This continues to add some handy replacement routines for property
maps, so we can transform keys and values to and fro.
2017-07-12 13:03:18 -07:00
Luke Hoban 15be6850a2 Support cycles and avoid name collisions in closure serialization
Generate names for serialized functions as hashes of the function text, and
bind these to the actual referenced names locally to each serialized closure.
This avoids potential conflicts between different functions which are referenced
with the same name.

For example:
```
function __8c176d768f95567f8c3acd08e486a1da7299fbc1() {
  with({ f: __e61452ceb9d53f3611ed543f3822a85ad29345dd}) {
    return (() => {

return function (cb) { cb(null, f(7)); };

    })().apply(this, arguments);
  }
}
```

Also fixes the stack overflows when there are cycles in the Closure graph by
adding caches of serialized functions.

Fixes #238.
2017-07-12 12:49:04 -07:00
Luke Hoban a2d6e67deb Fix a comment typo 2017-07-08 14:57:11 -07:00
Luke Hoban 87a18e6ad2 Resolve to dynamic instead of error type on TypeNotfound
This unblocks some cases with generics without having to
implement full generics support in the type LumiRT type system
(which we directionally will be removing anyway).
2017-07-07 16:24:52 -07:00
Luke Hoban 829b977bcf Support try/catch in Lumi and async/await in Node.js
We would like to allow developers to use async/await
on the inside (Node.js) of Lumi programs.

We now support (don't error on) usage of async/await
inside runtime callbacks in Lumi programs.  If await is
used during deployment, it will trigger an error.

Also adds support for try/catch in LumiJS, as these are
used more heavily in async/await code.

Since we target Node.js environments without native support
for async/await, we also emit runtime helpers to support TS
transpilation of async/await for Node.js pre-7.6.
2017-07-07 12:47:27 -07:00
joeduffy ea0461dadb Export plugin prefix constants 2017-07-06 09:46:00 -04:00
joeduffy aeefc27a08 Add some ReadLocations tracing; and skip nulls 2017-07-06 00:02:33 -04:00
joeduffy 9babb4b993 Refactor EnsurePath to be clear about dir vs. file 2017-07-04 20:52:25 -04:00
joeduffy 303fd6322b Refactor code generator goo
This change rearranges some of the code generator logic in LumIDL
in such a way that we can use it from other code generators (like
the ongoing bridge work).
2017-07-01 14:00:06 -07:00
joeduffy 06ad983541 Add a ReadLocations engine-side RPC function
This adds a ReadLocations RPC function to the engine interface, alongside
the singular ReadLocation.  The plural function takes a single token that
represents a module or class and we will then return all of the module
or class (static) properties that are currently known.
2017-07-01 13:26:49 -07:00
joeduffy 5d9f7918e9 Add a PropertyMap/Value.MapReplace function
This adds a handy MapReplace function on pkg/resource's PropertyMap and
PropertyValue types.  This is just like the existing Mappable function,
except that it permits easy replacement of elements as the map transformation
occurs.  We need this to perform float64=>int transformations.
2017-07-01 12:08:55 -07:00
joeduffy 27b819dec6 Add the ability to skip nulls when un/marshaling properties 2017-07-01 11:51:52 -07:00
joeduffy 15a75c9ee4 Catch duplicate URNs during planning
We fail very late in the process of plan application, should a duplicate
URN arise.  This change fails as early in the process as possible and
ensures that it does so with good line number information.
2017-06-27 13:04:06 -07:00
joeduffy 24fd8e8f4a Add cancellation to interpreter
This properly unwinds the interpreter should something happen that
results in cancellation.  This occurs, for example, when the planning
engine encounters an error and decides that it doesn't need to proceed
further with evaluation before it simply goes ahead and exits.
2017-06-27 11:31:17 -07:00
joeduffy 23045c5792 Simply panic for failfast
The old contract library tried to be glog-friendly in its failfast behavior.
It turns out glog seldom does the right thing when goroutines are involved
(which, as of last sprint, they now are).  We already had issues with stacks
not getting printed when --logtostderr was turned on, and the code tried
to work around this; but this still didn't work for the goroutines case.

All of this seems like way too much cleverness.  Let's just use Go panics.
2017-06-27 11:12:06 -07:00
joeduffy 19a359e65d Specialize check failure messages 2017-06-27 10:56:33 -07:00
joeduffy 41719683e6 Run check with "" property
This change starts running Check with a "" property for cases where
global validation must take place (such as ensuring that required
configuration variables were set).  It may be safely ignored if per-
property validation is preferred by a given resource.
2017-06-27 10:46:20 -07:00
joeduffy 0ea55328f6 Specialize the null message for config reads 2017-06-27 10:45:29 -07:00
joeduffy 2daea4c3d8 Clarify aspects of using the DCO 2017-06-26 14:46:34 -07:00
Luke Hoban aaa00f5a29 Fix for lambda variable capture
Fixes a bug in free variables analysis when local variable
declarations are inside MultiStatements.
2017-06-25 13:25:17 -07:00
joeduffy 3c1041af49 Update license headers 2017-06-23 14:53:41 -07:00
Luke Hoban 201a3d1a51 Additional nightly test fixes
Address several issues with running the Beanstalk
example in newer AWS regions with different requirements.

Ensures S3 bucket names adhere to required naming patterns
outside of us-east-1.

Also add InstanceProfile and ServiceRole configuration to the
beanstalk example as required in newer regions.
2017-06-22 16:59:16 -07:00
joeduffy d05e7ace91 Ensure we close the plugin host/context
This adds a few missing closes for the plugin host/context.  This
should fix pulumi/lumi#261.  Eventually when we have more robust
nightly test options, and want to spend the time, we should think
about doing more rigorous stress testing that kills processes at
inopportune times and guarantees we don't leak.  I've filed
pulumi/lumi#263 to do that.
2017-06-22 15:18:29 -07:00
joeduffy 8b57310854 Tidy up more lint
This change fixes a few things:

* Most importantly, we need to place a leading "." in the paths
  to Gometalinter, otherwise some sub-linters just silently skip
  the directory altogether.  errcheck is one such linter, which
  is a very important one!

* Use an explicit Gometalinter.json file to configure the various
  settings.  This flips on a few additional linters that aren't
  on by default (line line length checking).  Sadly, a few that
  I'd like to enable take waaaay too much time, so in the future
  we may consider a nightly job (this includes code similarity,
  unused parameters, unused functions, and others that generally
  require global analysis).

* Now that we're running more, however, linting takes a while!
  The core Lumi project now takes 26 seconds to lint on my laptop.
  That's not terrible, but it's long enough that we don't want to
  do the silly "run them twice" thing our Makefiles were previously
  doing.  Instead, we shall deploy some $$($${PIPESTATUS[1]}-1))-fu
  to rely on the fact that grep returns 1 on "zero lines".

* Finally, fix the many issues that this turned up.

I think(?) we are done, except, of course, for needing to drive
down some of the cyclomatic complexity issues (which I'm possibly
going to punt on; see pulumi/lumi#259 for more details).
2017-06-22 12:09:46 -07:00
Luke Hoban a63efc42a3 Propagate errors on deployment failures
We were not propagating the error from `deployLatest` through
to the CLI error result.  Despite out recent efforts to integrate
gometalinter, there were also several additional similar cases of
ignored error results reported by `errcheck`.  Not yet clear why
these are not being reported via gometalinter.

Fixes #262.
2017-06-21 22:02:57 -07:00
joeduffy d1182dc54f Improve pkg/compiler/ast/ cyclomatic complexity
Part of pulumi/lumi#259.
2017-06-21 14:24:10 -07:00
joeduffy 7fe8052941 Fix some lint in our lint
After 233c5a8 landed, I noticed there are a few things to be fixed up:

    * Run gometalinter in all the right places.  We need to run both in
      lint and lint_quiet targets.  I've also cleaned up some of the logic
      around what to suppress so there's less repetition.

    * We currently @ meaningful commands, which is unfortunate, since it
      makes debugging Makefiles tough (especially when looking at CI build
      logs).  Going forward, we should only use @ for meaningless commands,
      like @echo.

    * The AWS project wasn't actually running tslint, because it needs to
      say `tslint './pack/**/*.ts' --exclude='./pack/node_modules/**'`.
      The current script of `tslint lib/aws/pack/...` wasn't actually
      running lint, hence we missed a lot of AWS lint issues.

    * Fix up the issues that these fixes uncovered.  Mostly err shadowing.
2017-06-21 13:24:35 -07:00
joeduffy f6f166a5c3 Add a TODO for pulumi/lumi#260 2017-06-21 11:33:10 -07:00
joeduffy 97deabb9bd Finish interface for reading configuration¬
This continues the previous commit and establishes the interpreter
context so that we can use the new host interface.  In summary:

    * Instead of using the NullSource for destructions -- which
      doesn't hook up an interpreter and so any reads of configuration
      variables will fail -- we will enlighten the EvalSource to know
      how to orchestrate destruction interpretation.  The primary
      difference is that we don't actually run the code, but *we do*
      perform all of the necessary configuration and variable init.

    * Associate the active interpreter with the plugin context as
      we are executing, so that the host object can actually read the
      state from the heap as requested to do so by attached plugins.

    * Rename anything "engine" related to use the term "host"; this
      avoids introducing unnecesarily new terminology.

    * Add a new pkg/resource/provider/ package where we can begin
      consolidating helper functionality for resource providers.
      Right now, this includes a wrapper interface atop the gRPC
      machinery necessary to contact the host, in addition to a
      Main function that hides some boilerplate entrypoint code.

    * Add a rpcutil.IsBenignCloseErr routine to let us ignore
      "benign" gRPC errors that are knowingly returned at shutdown.

This commit completes pulumi/lumi#117.
2017-06-21 10:31:06 -07:00
joeduffy e5f229aad8 Use Location.Read from plugin host
This addresses CR feedback from @lukehoban; namely, that we should
be going through the Read API for location reads in the plugin host
to ensure that getters are invoked as appropriate.

I also made Location's various fields private so that we aren't
tempted to make this mistake elsewhere, effectively "forcing" us
to go through the accessor methods.
2017-06-21 08:43:05 -07:00
joeduffy d7093188f0 Introduce an interface to read config
This change adds an engine gRPC interface, and associated implementation,
so that plugins may do interesting things that require "phoning home".
Previously, the engine would fire up plugins and talk to them directly,
but there was no way for a plugin to ask the engine to do anything.

The motivation here is so that plugins can read evaluator state, such
as config information, but this change also allows richer logging
functionality than previously possible.  We will still auto-log any
stdout/stderr writes; however, explicit errors, warnings, informational,
and even debug messages may be written over the Log API.
2017-06-20 19:45:07 -07:00
joeduffy c39280a545 Synchronize access to type caches
We run tests in parallel and recently we began hitting a high enough
degree of parallelism that we've begun seeing "unsynchronized access
to map" errors in our test passes (intermittently).  The root cause
is that access to the type symbol caches aren't synchronized.  It would
be ideal if we actually rewired these to be cached in the compiler
context -- rather than being global -- but this fix is sufficient for
now.  We will simply synchronize access using a Mutex.
2017-06-20 18:16:53 -07:00
Luke Hoban 6b1fbe9ba0 Fix intrinsics tets
The previous change broke the intrinsics tests in the
case that the `lumirt` package was not installed on
the system (which is the case in Travis given our build
order).

Reverting to the previous pattern of creating a fake
`lumirt` package to run the tests in to avoid the
dependency on an externally installed `lumirt`.
2017-06-20 11:07:13 -07:00
Luke Hoban 6b7616ce1a Merge branch 'master' of https://github.com/pulumi/lumi 2017-06-20 10:30:27 -07:00
Luke Hoban 8d80871712 Fix string quoting in jsonStringify
Implements correct QuoteJSONString behaviour per ECMAScript
spec.

Also refactors intrinsic test harness to make it easier to add new
intrinsics tests going forward.
2017-06-20 10:29:53 -07:00
joeduffy 26cf93f759 Implement get functions on all resources
This change implements the `get` function for resources.  Per pulumi/lumi#83,
this allows Lumi scripts to actually read from the target environment.

For example, we can now look up a SecurityGroup from its ARN:

    let group = aws.ec2.SecurityGroup.get(
        "arn:aws:ec2:us-west-2:153052954103:security-group:sg-02150d79");

The returned object is a fully functional resource object.  So, we can then
link it up with an EC2 instance, for example, in the usual ways:

    let instance = new aws.ec2.Instance(..., {
        securityGroups: [ group ],
    });

This didn't require any changes to the RPC or provider model, since we
already implement the Get function.

There are a few loose ends; two are short term:

    1) URNs are not rehydrated.
    2) Query is not yet implemented.

One is mid-term:

    3) We probably want a URN-based lookup function.  But we will likely
       wait until we tackle pulumi/lumi#109 before adding this.

And one is long term (and subtle):

    4) These amount to I/O and are not repeatable!  A change in the target
       environment may cause a script to generate a different plan
       intermittently.  Most likely we want to apply a different kind of
       deployment "policy" for such scripts.  These are inching towards the
       scripting model of pulumi/lumi#121, which is an entirely different
       beast than the repeatable immutable infrastructure deployments.

Finally, it is worth noting that with this, we have some of the fundamental
underpinnings required to finally tackle "inference" (pulumi/lumi#142).
2017-06-19 17:29:02 -07:00
joeduffy 7f710a241f Better handle errors during config apply 2017-06-16 15:37:58 -07:00
joeduffy 5f9ed13069 Simplify Check; make it tolerant of computed values
This change simplifies the generated Check interface for providers.
Instead of

    Check(ctx context.Context, obj *T) ([]error, error)

where T is the resource type, we have

    Check(ctx context.Context, obj *T, property string) error

This is done so that we can drive the calls to Check one property
at a time, allowing us to skip any that are computed.  (Otherwise,
we may fail the verification erroneously.)

This has the added advantage that the Check implementations are
simpler and can simply return a single error.  Furthermore, the
generated RPC code handles wrapping the result, so we can just do

    return errors.New("bad");

rather than the previous reflection-laden junk

    return resource.NewFieldError(
        reflect.TypeOf(obj), awsservice.AWSResource_Property,
        errors.New("bad"))
2017-06-16 13:34:11 -07:00
Luke Hoban 33a9452ece Merge pull request #256 from pulumi/examplestest
Add integration testing for examples
2017-06-16 10:17:51 -07:00
joeduffy 5e85e6d543 Improve property validation diagnostics
The change prints the value of a property that fails resource validation.
This makes it much easier to diagnose what's going on.
2017-06-16 10:04:49 -07:00
joeduffy 7d19abc2a3 Print the current environment
This change implements showing a summary of the current environment.
All you need to do is run

    $ lumi env

and the current environment's information will be printed.

This makes it convenient to grab resource information that might be
required, for instance, to correlate with logs (e.g., lambda ARNs).

Eventually, as per pulumi/lumi#184, we want to print details about
all of the resources too.
2017-06-16 09:46:09 -07:00
Luke Hoban 6840dba051 Move examples test to seperate package
We need to run examples tests only after building and
installing all Lumi commands.
2017-06-16 09:24:31 -07:00
Luke Hoban 639a2d323d Test more examples
Tests all of our commonly used examples.

Also sets test parallelism to 10 by default
since we are I/O bound on API calls to
the resource providers.

Also avoids using larger EC2 examples in
our samples so that we can keep our test
costs lower :-).
2017-06-16 09:24:31 -07:00
Luke Hoban 8d8eba5c65 Add integration testing for examples
Adds an integration test that runs the following commands on the
AWS webserver example, failing if any command returns an error
code:
* lumijs
* lumi env init
* lumi config
* lumi plan
* lumi deploy
* lumi destroy
* lumi env rm

Also ensures that plan and deploy failures propagate errors through
to error codes at the CLI.
2017-06-16 09:24:31 -07:00
joeduffy d013501e04 Restructure Rendezvous to have a distinct Let vs. Meet
On the first turn, we want to distinguish between a coroutine
running that owns its turn, and a coroutine that knows it doesn't
own the turn and is simply awaiting its turn.  The old Meet logic
wasn't quite right; instead, we'll have the caller tell us this.
2017-06-15 18:20:12 -07:00
joeduffy 34ea2e8e43 Rewrite some binop code to share more logic
Also add some logging to help diagnose some issues I was running into.
2017-06-15 17:29:51 -07:00
joeduffy e26f934aec Propagate computed values in assignments
This change fixes an issue with the way we deal with computed values in
assignments.  Specifically, the assignment expression should resolve to
the computed value itself, but it must actually perform the assignment!
Previously, we evaluated to the right thing, but skipped the assignment.
2017-06-15 17:04:16 -07:00
Luke Hoban ae03d69645 Wire up APIs to lambdas using output properties
We now have enough output properties implementation
working to change our API gateway examples and API
wrapper to correctly wire the API routes to the ARNs of
lambdas passed in to them.

We both wire up the lambda to the route, but also create
a permission specific to each route to assign to the
corresponding lambda - providing least privelege needed
for the API definition.

Also adds `string#toUpperCase` and fixes NewUniqueHex
to match how we are using it.
2017-06-15 16:01:00 -07:00
joeduffy b7a3b28734 Implement array push and pop
This implements array push and pop as intrinsics.

Also:

* Tighten up some assertions while I'm in here.

* Default initialize pointer slots to Null, if not done explicitly.
2017-06-15 14:51:57 -07:00
joeduffy 085e39230b Test missing and output properties
This tests the conditions that triggered pulumi/lumi#251.
2017-06-15 13:08:06 -07:00
joeduffy 3e4e791fa4 Also overwrite output property slots
This change overwrites output property slots in runtime objects
after performing a CRUD operation, in addition to null or missing
slots, fixing #251.  The problem is that we sometimes have output
property values pre-populated in an object, and sometimes don't,
depending on various things (both are legal).  We should handle both.
2017-06-15 13:08:06 -07:00
joeduffy 9698309f2b Model resource ID and URN as output properties
This change exposes ID and URN properties on resources, as appropriate,
so that they may be read and used in Lumi scripts.
2017-06-14 17:00:13 -07:00
joeduffy 2ac303f703 Fix deployment hang (pulumi/lumi#246)
The recent change to run the interpreter and planner on separate goroutines
created the need to perform rendezvous-style synchronization between them.
Although the case of an invoked function properly tore down the synchronization
by communicating the error, we seldom directly invoke functions for JavaScript
programs because the way module entrypoint code ends up in initializers.
This requires that we propagate errors correctly out of module and class
initializers, in the standard way, so that the unwind makes its way to the top.

This fixes pulumi/lumi#246.
2017-06-14 15:52:36 -07:00
joeduffy 3a899b304e Fix empty body issues
We recently changed the Resource base type to have no constructor,
rather than a manual empty constructor.  This ought to work just fine.
The LumiJS compiler indeed generates a constructor, however, it is
missing a body and when the interpreter tries to invoke it, we crash
with a nil reference panic.  The runtime actually tolerates missing
constructors entirely, although the way LumiJS binds super calls
doesn't tolerate the missing base constructor.  This change simply
generates such constructors in LumiJS with empty bodies.

In addition, I've added an error that will catch the empty body
problem during binding, since technically speaking, all functions
must have bodies.  (Our runtime happens to support the notion of
"abstract", however, so we only fire the error on concrete functions.)
2017-06-14 10:30:46 -07:00
joeduffy 792490e814 Fix the polarity of some asserts 2017-06-14 09:44:58 -07:00