Commit graph

295 commits

Author SHA1 Message Date
Joe Duffy
69f7f51375 Many asset improvements
This improves a few things about assets:

* Compute and store hashes as input properties, so that changes on
  disk are recognized and trigger updates (pulumi/pulumi#153).

* Issue explicit and prompt diagnostics when an asset is missing or
  of an unexpected kind, rather than failing late (pulumi/pulumi#156).

* Permit raw directories to be passed as archives, in addition to
  archive formats like tar, zip, etc. (pulumi/pulumi#240).

* Permit not only assets as elements of an archive's member list, but
  also other archives themselves (pulumi/pulumi#280).
2017-10-22 13:39:21 -07:00
Matt Ellis
a749ac1102 Use go-yaml directly
Instead of doing the logic to see if a type has YAML tags and then
dispatching based on that to use either the direct go-yaml marshaller
or the one that works in terms of JSON tags, let's just say that we
always add YAML tags as well, and use go-yaml directly.
2017-10-20 14:01:37 -07:00
pat@pulumi.com
20e71fa5c4 Set old.Delete when previewing a CreateReplace step.
This is required to prevent an assertion when skipping a `Delete` step.
2017-10-19 13:08:17 -07:00
Pat Gavlin
9895e8006f Merge pull request #434 from pulumi/PendingDeletes
Track resources that are pending deletion in checkpoints.
2017-10-19 10:57:00 -07:00
pat@pulumi.com
23864b9459 PR feedback. 2017-10-19 10:34:23 -07:00
joeduffy
599ca8ea43 Add accessors to fetch the Pulumi project and stack names
This change adds functions, `pulumi.getProject()` and `pulumi.getStack()`,
to fetch the names of the project and stack, respectively.  These can be
handy in generating names, specializing areas of the code, etc.

This fixes pulumi/pulumi#429.
2017-10-19 08:26:57 -07:00
pat@pulumi.com
6b66437fae Track resources that are pending deletion in checkpoints.
During the course of a `pulumi update`, it is possible for a resource to
become slated for deletion. In the case that this deletion is part of a
replacement, another resource with the same URN as the to-be-deleted
resource will have been created earlier. If the `update` fails after the
replacement resource is created but before the original resource has been
deleted, the snapshot must capture that the original resource still exists
and should be deleted in a future update without losing track of the order
in which the deletion must occur relative to other deletes. Currently, we
are unable to track this information because the our checkpoints require
that no two resources have the same URN.

To fix this, these changes introduce to the update engine the notion of a
resource that is pending deletion and change checkpoint serialization to
use an array of resources rather than a map. The meaning of the former is
straightforward: a resource that is pending deletion should be deleted
during the next update.

This is a fairly major breaking change to our checkpoint files, as the
map of resources is no more. Happily, though, it makes our checkpoint
files a bit more "obvious" to any tooling that might want to grovel
or rewrite them.

Fixes #432, #387.
2017-10-18 17:09:00 -07:00
Pat Gavlin
dacff3db48 Merge pull request #425 from pulumi/DrainStreamsPluginLoadFailure
Drain std{out,err} when a plugin fails to load.
2017-10-16 23:32:02 -07:00
pat@pulumi.com
9453f86c2e Implement dynamic resources.
A dynamic resource is a resource whose provider is implemented alongside
the resource itself. This provider may close over and use orther
resources in the implementation of its CRUD operations. The provider
itself must be stateless, as each CRUD operation for a particular
dynamic resource type may use an independent instance of the provider.
Changes to the definition of a resource's provider result in replacement
of the resource itself (rather than a simple update), as this allows the
old provider definition to delete the old resource and the new provider
definition to create an appropriate replacement.
2017-10-16 23:06:53 -07:00
Pat Gavlin
546612a354 Drain std{out,err} when a plugin fails to load.
If a plugin fails to load after we've set up the goroutines that copy
from its std{out,err} streams, then those goroutines can end up writing
to a closed event channel. This change ensures that we properly drain
those streams in this case.
2017-10-16 21:38:11 -07:00
Matt Ellis
996ac8872d Merge pull request #420 from pulumi/rename-env-to-stack
Use `Stack` over `Environment` to describe a deployment target
2017-10-16 14:24:11 -07:00
pat@pulumi.com
bdbb1b59d4 Ensure a plugin's std{out,err} streams are drained in Close().
Not doing so can cause panics, as the goroutines we use to copy these
streams can end up writing to a closed channel.
2017-10-16 13:44:37 -07:00
Matt Ellis
22c9e0471c Use Stack over Environment to describe a deployment target
Previously we used the word "Environment" as the term for a deployment
target, but since then we've started to use the term Stack. Adopt this
across the CLI.

From a user's point of view, there are a few changes:

1. The `env` verb has been renamed to `stack`
2. The `-e` and `--env` options to commands which operate on an
environment now take `-s` or `--stack` instead.
3. Becase of (2), the commands that used `-s` to display a summary now
only support passing the full option name (`--summary`).

On the local file system, we still store checkpoint data in the `env`
sub-folder under `.pulumi` (so we can reuse existing checkpoint files
that were written to the old folder)
2017-10-16 13:04:20 -07:00
joeduffy
301739c6b5 Add auto-parenting
This changes a few things about "components":

* Rename what was previously ExternalResource to CustomResource,
  and all of the related fields and parameters that this implies.
  This just seems like a much nicer and expected name for what
  these represent.  I realize I am stealing a name we had thought
  about using elsewhere, but this seems like an appropriate use.

* Introduce ComponentResource, to make initializing resources
  that merely aggregate other resources easier to do correctly.

* Add a withParent and parentScope concept to Resource, to make
  allocating children less error-prone.  Now there's no need to
  explicitly adopt children as they are allocated; instead, any
  children allocated as part of the withParent callback will
  auto-parent to the resource provided.  This is used by
  ComponentResource's initialization function to make initialization
  easier, including the distinction between inputs and outputs.
2017-10-15 04:38:26 -07:00
joeduffy
fbfca58a3f Implement components
This change implements core support for "components" in the Pulumi
Fabric.  This work is described further in pulumi/pulumi#340, where
we are still discussing some of the finer points.

In a nutshell, resources no longer imply external providers.  It's
entirely possible to have a resource that logically represents
something but without having a physical manifestation that needs to
be tracked and managed by our typical CRUD operations.

For example, the aws/serverless/Function helper is one such type.
It aggregates Lambda-related resources and exposes a nice interface.
All of the Pulumi Cloud Framework resources are also examples.

To indicate that a resource does participate in the usual CRUD resource
provider, it simply derives from ExternalResource instead of Resource.

All resources now have the ability to adopt children.  This is purely
a metadata/tagging thing, and will help us roll up displays, provide
attribution to the developer, and even hide aspects of the resource
graph as appropriate (e.g., when they are implementation details).

Our use of this capability is ultra limited right now; in fact, the
only place we display children is in the CLI output.  For instance:

    + aws:serverless:Function: (create)
      [urn=urn:pulumi:demo::serverless::aws:serverless:Function::mylambda]
      => urn:pulumi:demo::serverless::aws:iam/role:Role::mylambda-iamrole
      => urn:pulumi:demo::serverless::aws:iam/rolePolicyAttachment:RolePolicyAttachment::mylambda-iampolicy-0
      => urn:pulumi:demo::serverless::aws:lambda/function:Function::mylambda

The bit indicating whether a resource is external or not is tracked
in the resulting checkpoint file, along with any of its children.
2017-10-14 18:30:59 -07:00
Matt Ellis
e7e4e75af3 Don't examine the Checkpoint in the CLI
The checkpoint is an implementation detail of the storage of an
environment. Instead of interacting with it, make sure that all the
data we need from it either hangs off the Snapshot or Target
objects (which you can get from a Checkpoint) and then start consuming
that data.
2017-10-09 18:21:55 -07:00
joeduffy
7e30dde8f4 Update plan test to new interface 2017-10-04 08:30:50 -04:00
joeduffy
b7576b9b14 Add a notion of stable properties
This change adds the capability for a resource provider to indicate
that, where an action carried out in response to a diff, a certain set
of properties would be "stable"; that is to say, they are guaranteed
not to change.  As a result, properties may be resolved to their final
values during previewing, avoiding erroneous cascading impacts.

This avoids the ever-annoying situation I keep running into when demoing:
when adding or removing an ingress rule to a security group, we ripple
the impact through the instance, and claim it must be replaced, because
that instance depends on the security group via its name.  Well, the name
is a great example of a stable property, in that it will never change, and
so this is truly unfortunate and always adds uncertainty into the demos.
Particularly since the actual update doesn't need to perform replacements.

This resolves pulumi/pulumi#330.
2017-10-04 08:22:21 -04:00
pat@pulumi.com
252fd8e6bb Remove deploy.Step.Pre.
As per @joeduffy, this is an artifact of the prior runtime model.
2017-10-03 10:03:05 -07:00
Pat Gavlin
ff2a3fa242 Replace Plan.Apply with planResult.Walk. (#383)
`deploy.Plan.Apply` was only consumed by the engine, and seemed to be in
the wrong place given the API exported by the rest of `Plan` (i.e.
`Plan.Start` + `PlanIterator`). Furthermore, we were missing a reasonable
opportunity to share code between `update` and `preview`, both of which
need to walk the plan. These changes move the plan walk into `package engine`
as `planResult.Walk` and replace the `Progress` interface with a new interface,
`StepActions`, which subsumes the functionality of the former and adds support
for implementation-specific step execution. `planResult.Walk` is then
consumed by both `Engine.Deploy` and `Engine.PrintPlan`.
2017-10-02 14:26:51 -07:00
joeduffy
ac2dbc80fa Add an Invoke RPC method on ResourceProvider
This change enables us to make progress on exposing data sources
(see pulumi/pulumi-terraform#29).  The idea is to have an Invoke
function that simply takes a function token and arguments, performs
the function lookup and invocation, and then returns a return value.
2017-09-30 14:53:27 -04:00
pat@pulumi.com
28d02e232b Fix #372.
Print "modified" rather than "modifyd". This introduces a new method,
`resource.StepOp.PastTense()`, which returns the past tense description
of the operation.
2017-09-28 09:28:36 -07:00
Joe Duffy
b4646db39b Merge branch 'master' into RenameVerbs 2017-09-23 11:31:29 -07:00
joeduffy
141a112950 Improve output formatting
This change improves our output formatting by generally adding
fewer prefixes.  As shown in pulumi/pulumi#359, we were being
excessively verbose in many places, including prefixing every
console.out with "langhost[nodejs].stdout: ", displaying full
stack traces for simple errors like missing configuration, etc.

Overall, this change includes the following:

* Don't prefix stdout and stderr output from the program, other
  than the standard "info:" prefix.  I experimented with various
  schemes here, but they all felt gratuitous.  Simply emitting
  the output seems fine, especially as it's closer to what would
  happen if you just ran the program under node.

* Do NOT make writes to stderr fail the plan/deploy.  Previously
  we assumed that any console.errors, for instance, meant that
  the overall program should fail.  This simply isn't how stderr
  is treated generally and meant you couldn't use certain
  logging techniques and libraries, among other things.

* Do make sure that stderr writes in the program end up going to
  stderr in the Pulumi CLI output, however, so that redirection
  works as it should.  This required a new Infoerr log level.

* Make a small fix to the planning logic so we don't attempt to
  print the summary if an error occurs.

* Finally, add a new error type, RunError, that when thrown and
  uncaught does not result in a full stack trace being printed.
  Anyone can use this, however, we currently use it for config
  errors so that we can terminate with a pretty error message,
  rather than the monstrosity shown in pulumi/pulumi#359.
2017-09-23 05:20:11 -07:00
pat@pulumi.com
69341fa7c8 push is dead; long live update.
After discussion with Joe and Luke, we've decided to use `update` instead
of `push` as it more intuitively fits the operation being performed.
2017-09-22 17:23:40 -07:00
pat@pulumi.com
597db186ec Renames: plan -> preview, deploy -> push.
Part of #353.

These changes also remove all command aliases from the `pulumi` command.
2017-09-22 15:28:03 -07:00
Joe Duffy
f6e694c72b Rename pulumi-fabric to pulumi
This includes a few changes:

* The repo name -- and hence the Go modules -- changes from pulumi-fabric to pulumi.

* The Node.js SDK package changes from @pulumi/pulumi-fabric to just pulumi.

* The CLI is renamed from lumi to pulumi.
2017-09-21 19:18:21 -07:00
Matt Ellis
25ae463915 Listen only on 127.0.0.1
Instead of binding on 0.0.0.0 (which will listen on every interface)
let's only listen on localhost. On windows, this both makes the
connection Just Work and also prevents the Windows Firewall from
blocking the listen (and displaying UI saying it has blocked an
application and asking if the user should allow it)
2017-09-21 10:56:45 -07:00
joeduffy
22387d24cd Switch to a --parallel=P flag
This change flips the polarity on parallelism: rather than having a
--serialize flag, we will have a --parallel=P flag, and by default
we will shut off parallelism.  We aren't benefiting from it at the
moment (until we implement pulumi/pulumi-fabric#106), and there are
more hidden dependencies in places like AWS Lambdas and Permissions
than I had realized.  We may revisit the default, but this allows
us to bite off the messiness of dependsOn only when we benefit from
it.  And in any case, the --parallel=P capability will be useful.
2017-09-17 08:10:46 -07:00
joeduffy
087deb7643 Add optional dependsOn to Resource constructors
This change adds an optiona dependsOn parameter to Resource constructors,
to "force" a fake dependency between resources.  We have an extremely strong
desire to resort to using this only in unusual cases -- and instead rely
on the natural dependency DAG based on properties -- but experience in other
resource provisioning frameworks tells us that we're likely to need this in
the general case.  Indeed, we've already encountered the need in AWS's
API Gateway resources... and I suspect we'll run into more especially as we
tackle non-serverless resources like EC2 Instances, where "ambient"
dependencies are far more commonplace.

This also makes parallelism the default mode of operation, and we have a
new --serialize flag that can be used to suppress this default behavior.
Full disclosure: I expect this to become more Make-like, i.e. -j 8, where
you can specify the precise width of parallelism, when we tackle
pulumi/pulumi-fabric#106.  I also think there's a good chance we will flip
the default, so that serial execution is the default, so that developers
who don't benefit from the parallelism don't need to worry about dependsOn
in awkward ways.  This tends to be the way most tools (like Make) operate.

This fixes pulumi/pulumi-fabric#335.
2017-09-15 16:38:52 -07:00
joeduffy
016adec9f7 Allow unknowns in defaults
The "defaults" may include computed values, so we should pass
AllowUnknowns as true during unmarshaling of Check calls.
2017-09-15 09:56:42 -07:00
joeduffy
5b779798c4 Fix (don't torch) LUMIDL
This change finishes the conversion of LUMIDL over to the new
runtime model, with the appropriate code generation changes.

It turns out the old model actually had a flaw in it anyway that we
simply didn't hit because we hadn't been stressing output properties
nearly as much as the new model does.  This resulted in needing to
plumb the rejection (or allowance) of computed properties more
deeply into the resource property marshaling/unmarshaling logic.

As of these changes, I can run the GitHub provider again locally.

This change fixes pulumi/pulumi-fabric#332.
2017-09-14 16:40:44 -07:00
joeduffy
9c7f6b678c Bring LUMIDL up to code
This gets LUMIDL to generate code in the new way.
2017-09-11 16:58:25 -07:00
joeduffy
8aba3aae12 Upgrade gRPC to 1.6.0; use full addresses
This change upgrades gRPC to 1.6.0 to pick up a few bug fixes.

We also use the full address for gRPC endpoints, including the
interface name, as otherwise we pick the wrong interface on Linux.
2017-09-09 07:37:10 -07:00
joeduffy
67e5750742 Fix a bunch of Linux issues
There's a fair bit of clean up in here, but the meat is:

* Allocate the language runtime gRPC client connection on the
  goroutine that will use it; this eliminates race conditions.

* The biggie: there *appears* to be a bug in gRPC's implementation
  on Linux, where it doesn't implement WaitForReady properly.  The
  behavior I'm observing is that RPC calls will not retry as they
  are supposed to, but will instead spuriously fail during the RPC
  startup.  To work around this, I've added manual retry logic in
  the shared plugin creation function so that we won't even try
  to use the client connection until it is in a well-known state.
  pulumi/pulumi-fabric#337 tracks getting to the bottom of this and,
  ideally, removing the work around.

The other minor things are:

* Separate run.js into its own module, so it doesn't include
  index.js and do a bunch of random stuff it shouldn't be doing.

* Allow run.js to be invoked without a --monitor.  This makes
  testing just the run part of invocation easier (including
  config, which turned out to be super useful as I was debugging).

* Tidy up some messages.
2017-09-08 15:11:09 -07:00
joeduffy
ebc3bf1dd0 Reap the language host process 2017-09-07 15:03:48 -07:00
joeduffy
f2d53459eb Add the notion of stable states
If a resource's planning operation is to do nothing, we can safely
assume that all of its properties are stable.  This can be used during
planning to avoid cascading updates that we know will never happen.
2017-09-05 10:01:00 -07:00
joeduffy
f3cf73d790 Change plugin prefixes to "pulumi-" 2017-09-04 11:35:21 -07:00
joeduffy
fe66a0eba7 Use the new URN during creates 2017-09-04 11:35:21 -07:00
joeduffy
77bbf443bc Synchronize with the resource channel properly 2017-09-04 11:35:21 -07:00
joeduffy
f718ab6501 Add a runtime.Log class
This change adds the ability to perform runtime logging, including
debug logging, that wires up to the Pulumi Fabric engine in the usual
ways.  Most stdout/stderr will automatically go to the right place,
but this lets us add some debug tracing in the implementation of the
runtime itself (and should come in handy in other places, like perhaps
the Pulumi Framework and even low-level end-user code).
2017-09-04 11:35:21 -07:00
joeduffy
a13a83b067 Pass the monitor address correctly to language plugins 2017-09-04 11:35:21 -07:00
joeduffy
9f160a7f91 Configure providers at well-defined points
As explained in pulumi/pulumi-fabric#293, we were a little ad-hoc in
how configuration was "applied" to resource providers.

In fact, config wasn't ever communicated directly to providers; instead,
the resource providers would simply ask the engine to read random heap
locations (via tokens). Now that we're on a plan where configuration gets
handed to the program at startup, and that's that, and where generally
speaking resource providers never communicate directly with the language
runtime, we need to take a different approach.

As such, the resource provider interface now offers a Configure RPC
method that the resource planning engine will invoke at the right
times with the right subset of configuration variables filtered to
just that provider's package.  This fixes pulumi/pulumi#293.
2017-09-04 11:35:21 -07:00
joeduffy
70d0fac1c0 Simplify resource provider RPC interface
This change simplifies the provider RPC interface slightly:

1) Eliminate Get.  We really don't need it anymore.  There are
   several possibly-interesting scenarios down the road that may
   demand it, but when we get there, we can consider how best to
   bring this back.  Furthermore, the old-style Get remains mostly
   incompatible with Terraform anyway.

2) Pass URNs, not type tokens, across the RPC boundary.  This gives
   the provider access to more interesting information: the type,
   still, but also the name (which is no longer an object property).
2017-09-04 11:35:21 -07:00
joeduffy
590e9e539b Rename Lumi.yaml to Pulumi.yaml
And also eliminate lots of accumulated cruft around "packfiles", etc.
in the workspace code.
2017-09-04 11:35:21 -07:00
joeduffy
9599ea2e55 Get planning engine unit tests running again
We now build and run cleanly locally (for unit tests).  The
integration tests are still on the floor at the moment.
2017-09-04 11:35:21 -07:00
joeduffy
f189c40f35 Wire up Lumi to the new runtime strategy
🔥 🔥 🔥  🔥 🔥 🔥

Getting closer on #311.
2017-09-04 11:35:21 -07:00
Luke Hoban
24393cfe7b Read path assets into memory instead of holding open file handles
This helps avoid running into file handle limits when creating archives including thousands of node_modules files.

Tracking a more complete fix through all other codepaths related to assets as part of #325.
2017-08-29 13:33:02 -07:00
joeduffy
627d97d83f Close open Blobs
This change ensures we close all Blobs in the asset/archive logic.
In particular, the archive.Read function returns a map of files to
Blobs and after we are done copying the contents we must ensure
that we invoke Close, otherwise we may leak file handles, sockets,
and so on.  This may or may not be the culprit to the "too many
files open" errors we are hitting while deploying the M5 bits.
2017-08-28 18:52:51 -07:00
joeduffy
a626dcf6a3 Prettify the CLI in a few places
This changes a few things in the CLI, mostly just prettying it up:

    * Label all steps more clearly with the kind of step.  Also
      unify the way we present this during planning and deployment.

    * Summarize the changes that *did not* get made just as clearly
      as those that did.  In other words, stuff like this:

        info: 2 resources changed:
            +1 resource created
            -1 resource deleted
            5 resources unchanged

      and

        info: no resources required
            5 resources unchanged

    * Always print output properties when they are pertinent.
      This includes creates, replacements, and updates.

    * Show replacement creates and deletes very distinctly.  The
      create parts show up minty green and the delete parts show up
      rosey red.  These are the "physical" steps, compared to the
      "logical" step of replacement (which remains marigold).

      I still don't love where we are here.  The asymmetry between
      planning and deployment bugs me, and could be surprising.
      ("Hey, my deploy doesn't look like my plan!")  I don't know
      what developers will want to see here and I feel like in
      general we are spewing far too much into the CLI to make it
      even useful for anything but diagnosing failures afterwards.

      I propose that we should do a deep dive on this during the
      CLI epic, pulumi/pulumi-service#2.

This resolves pulumi/pulumi-fabric#305.
2017-08-06 10:05:51 -07:00
joeduffy
ff0eb81944 Export urnName constants
This avoids us needing to hard-code the urnName property in
various tools, in case we ever need to change it again down the road.
2017-08-05 08:32:50 -07:00
joeduffy
007c2f216d Rename name property to urnName
At the moment, we permit resources to carry a name, which the
engine uses as part of URN creation.  Unfortunately, the property
"name" has a very high chance of conflicting with meaningful
user-authored properties.  And furthermore this sort of name,
although key to creating URNs which are core to how the overall
system performs deployments and manages resources, are seldom
used programmatically in Lumi programs.  As a result, it's a real
nuisance that we stole the good name.

This change renames that property from "name" to "urnName".  This
not only has a lower likelihood of conflicting, but it also looks
reasonable sitting alongside the "urn" property.  In fact, in some
future universe, after some of the upcoming runtime changes, we
may not even need the name property on the objects whatsoever.

I had originally toyed with the idea of eliminating the Resource
versus NamedResource distinction.  This is certainly simpler and
remains a possibility.  I didn't do that right now, however, because
the flexibility of letting resource providers name resources however
they see fit still seems possibly useful.  For example, we keep
talking about whether functions can be auto-named based on hashes.
Until we've run those conversations to ground, I'd hate to do some
work that just needs to be undone in order to enable a scenario that
has a non-trivial likelihood of us wanting to explore.
2017-08-05 08:20:57 -07:00
joeduffy
c0e4bdb03b This shouldn't be here!
😬
2017-08-03 18:09:14 -07:00
joeduffy
a160741931 Tolerate computed and output properties
We are now freely flowing computed and output properties across the
RPC boundary with providers.  As such, we need to tolerate them in
a few more places.  Namely, mapping to and from regular non-resource
property values, and also when copying RPC resource state back onto
live runtime objects.
2017-08-03 11:01:38 -07:00
joeduffy
35aa6b7559 Rename pulumi/lumi to pulumi/pulumi-fabric
We are renaming Lumi to Pulumi Fabric.  This change simply renames the
pulumi/lumi repo to pulumi/pulumi-fabric, without the CLI tools and other
changes that will follow soon afterwards.
2017-08-02 09:25:22 -07:00
joeduffy
5fb014e53c Explicitly track default properties
This changes the RPC interfaces between Lumi and provider ever so
slightly, so that we can track default properties explicitly.  This
is required to perform accurate diffing between inputs provided by
the developer, inputs provided by the system, and outputs.  This is
particularly important for default values that may be indeterminite,
such as those we use in the bridge to auto-generate unique IDs.
Otherwise, we fail to reapply defaults correctly, and trick the
provider into thinking that properties changed when they did not.

This is a small step towards pulumi/lumi#306, in which we will defer
even more responsibility for diffing semantics to the providers.
2017-07-31 18:26:15 -07:00
joeduffy
300af62747 Fix erroneous assert when unmarshaling computed properties 2017-07-31 11:44:34 -07:00
joeduffy
00442b73b4 Alter the way unknown properties are serialized
This change serializes unknown properties anywhere in the entire
property structure, including deeply embedded inside object maps, etc.

This is now done in such a way that we can recover both the computed
nature of the serialized property, along with its expected eventual
type, on the other side of the RPC boundary.

This will let us have perfect fidelity with the new bridge's view on
computed properties, rather than special casing them on "one side".
2017-07-21 14:00:30 -07:00
joeduffy
b8be7fa5e6 Add a handy Copy function to PropertyMap 2017-07-21 14:00:30 -07:00
Luke Hoban
9243bd5f3f Fix tests after previous commit 2017-07-21 14:00:30 -07:00
joeduffy
40747a94bf Merge inputs with outputs
For Update and Delete operations, we provided just the input state
for a resource.  This is insufficient, because the provider may need
to depend on output state from the Create or prior Update operations.
This change merges the output atop the input during the step application.
2017-07-21 14:00:30 -07:00
joeduffy
4e02105355 Pass old state to the provider's API 2017-07-21 14:00:30 -07:00
joeduffy
ae92e68902 Return state as part of Create and Update¬
As part of the bridge bringup, I've discoverd that the property state
returned from Creates does *not* always equal the state that is then
read from calls to Get.  (I suspect this is a bug and that they should
be equivalent, but I doubt it's fruitfal to try and track down all
occurrences of this; I bet it's widespread).  To cope with this, we will
return state from Create and Update, instead of issuing a call to Get.
This was a design we considered to start with and frankly didn't have
a super strong reason to do it the current way, other than that it seemed
elegant to place all of the Get logic in one place.

Note that providers may choose to return nil, in which case we will read
state from the provider in the usual Get style.
2017-07-21 14:00:29 -07:00
joeduffy
ceb32b9e2b Add a sig field to Asset/Archive
This change mirrors the dynamic marshaling structure on the static
definition of the Asset and Archive types.  This ensures that they
marshal correctly even when deeply embedded inside other structures.
2017-07-18 09:30:31 -07:00
joeduffy
91c90cf2a9 Add diffing logic for assets/archives 2017-07-17 12:11:15 -07:00
joeduffy
002618e605 Add some more asset serialization round-tripping tests 2017-07-17 11:30:10 -07:00
joeduffy
4d708c8567 Fix asset diffing
This change brings the same typed serialization we use for RPC
to the serialization of deployments.  This ensures that we get
repeatable diffs from one deployment to the next.
2017-07-17 10:38:57 -07:00
joeduffy
bda607abd8 Permit -1 for randlen and maxlen
This allows -1 for randlen and maxlen to use defaults.  The default
behavior is that randlen uses sha1.Size and maxlen is "no max".
2017-07-15 09:59:44 -07:00
joeduffy
c61dcb5206 Revert "Rename Lumi resource properties"
This reverts commit c3db70849d.

I've opted to take a new strategy to ensure the bridge properties
don't conflict (with manual renames), similar to the name property.
2017-07-15 09:33:23 -07:00
joeduffy
0f18bb40b2 Tolerate some nils in important places 2017-07-14 18:08:33 -07:00
joeduffy
a6caef973a Make assets and archives 1st class
This change recognizes assets and archives as 1st class resource
property values.  This is necessary to support them in the new bridge
work, and lays the foundation for fixing pulumi/lumi#153.

I also took the opportunity to clean up some old cruft in the
resource properties area.
2017-07-14 12:28:43 -07:00
joeduffy
c3db70849d Rename Lumi resource properties
This renames the basemost resource properties, id and urn, to
names that are less likely to conflict with properties that real
resources will want to use, pid and upn (provider ID and Universal
Pulumi Name, respectively).

I actually ran into this with the current bridge work.  An alternative
solution would be to require derived resources to pick different names,
however this is unfortunate because usually they are more "user-facing"
than ours.  Another alternative is to not hijack the object properties
at all, but that too is problematic because we use these properties
during the evaluation of plans and deployments.

This seems like a reasonable middle ground.
2017-07-14 08:55:07 -07:00
joeduffy
a70790b81a Add key replacement functions
This continues to add some handy replacement routines for property
maps, so we can transform keys and values to and fro.
2017-07-12 13:03:18 -07:00
joeduffy
ea0461dadb Export plugin prefix constants 2017-07-06 09:46:00 -04:00
joeduffy
aeefc27a08 Add some ReadLocations tracing; and skip nulls 2017-07-06 00:02:33 -04:00
joeduffy
06ad983541 Add a ReadLocations engine-side RPC function
This adds a ReadLocations RPC function to the engine interface, alongside
the singular ReadLocation.  The plural function takes a single token that
represents a module or class and we will then return all of the module
or class (static) properties that are currently known.
2017-07-01 13:26:49 -07:00
joeduffy
5d9f7918e9 Add a PropertyMap/Value.MapReplace function
This adds a handy MapReplace function on pkg/resource's PropertyMap and
PropertyValue types.  This is just like the existing Mappable function,
except that it permits easy replacement of elements as the map transformation
occurs.  We need this to perform float64=>int transformations.
2017-07-01 12:08:55 -07:00
joeduffy
27b819dec6 Add the ability to skip nulls when un/marshaling properties 2017-07-01 11:51:52 -07:00
joeduffy
15a75c9ee4 Catch duplicate URNs during planning
We fail very late in the process of plan application, should a duplicate
URN arise.  This change fails as early in the process as possible and
ensures that it does so with good line number information.
2017-06-27 13:04:06 -07:00
joeduffy
24fd8e8f4a Add cancellation to interpreter
This properly unwinds the interpreter should something happen that
results in cancellation.  This occurs, for example, when the planning
engine encounters an error and decides that it doesn't need to proceed
further with evaluation before it simply goes ahead and exits.
2017-06-27 11:31:17 -07:00
joeduffy
19a359e65d Specialize check failure messages 2017-06-27 10:56:33 -07:00
joeduffy
0ea55328f6 Specialize the null message for config reads 2017-06-27 10:45:29 -07:00
joeduffy
2daea4c3d8 Clarify aspects of using the DCO 2017-06-26 14:46:34 -07:00
joeduffy
3c1041af49 Update license headers 2017-06-23 14:53:41 -07:00
joeduffy
d05e7ace91 Ensure we close the plugin host/context
This adds a few missing closes for the plugin host/context.  This
should fix pulumi/lumi#261.  Eventually when we have more robust
nightly test options, and want to spend the time, we should think
about doing more rigorous stress testing that kills processes at
inopportune times and guarantees we don't leak.  I've filed
pulumi/lumi#263 to do that.
2017-06-22 15:18:29 -07:00
joeduffy
8b57310854 Tidy up more lint
This change fixes a few things:

* Most importantly, we need to place a leading "." in the paths
  to Gometalinter, otherwise some sub-linters just silently skip
  the directory altogether.  errcheck is one such linter, which
  is a very important one!

* Use an explicit Gometalinter.json file to configure the various
  settings.  This flips on a few additional linters that aren't
  on by default (line line length checking).  Sadly, a few that
  I'd like to enable take waaaay too much time, so in the future
  we may consider a nightly job (this includes code similarity,
  unused parameters, unused functions, and others that generally
  require global analysis).

* Now that we're running more, however, linting takes a while!
  The core Lumi project now takes 26 seconds to lint on my laptop.
  That's not terrible, but it's long enough that we don't want to
  do the silly "run them twice" thing our Makefiles were previously
  doing.  Instead, we shall deploy some $$($${PIPESTATUS[1]}-1))-fu
  to rely on the fact that grep returns 1 on "zero lines".

* Finally, fix the many issues that this turned up.

I think(?) we are done, except, of course, for needing to drive
down some of the cyclomatic complexity issues (which I'm possibly
going to punt on; see pulumi/lumi#259 for more details).
2017-06-22 12:09:46 -07:00
Luke Hoban
a63efc42a3 Propagate errors on deployment failures
We were not propagating the error from `deployLatest` through
to the CLI error result.  Despite out recent efforts to integrate
gometalinter, there were also several additional similar cases of
ignored error results reported by `errcheck`.  Not yet clear why
these are not being reported via gometalinter.

Fixes #262.
2017-06-21 22:02:57 -07:00
joeduffy
7fe8052941 Fix some lint in our lint
After 233c5a8 landed, I noticed there are a few things to be fixed up:

    * Run gometalinter in all the right places.  We need to run both in
      lint and lint_quiet targets.  I've also cleaned up some of the logic
      around what to suppress so there's less repetition.

    * We currently @ meaningful commands, which is unfortunate, since it
      makes debugging Makefiles tough (especially when looking at CI build
      logs).  Going forward, we should only use @ for meaningless commands,
      like @echo.

    * The AWS project wasn't actually running tslint, because it needs to
      say `tslint './pack/**/*.ts' --exclude='./pack/node_modules/**'`.
      The current script of `tslint lib/aws/pack/...` wasn't actually
      running lint, hence we missed a lot of AWS lint issues.

    * Fix up the issues that these fixes uncovered.  Mostly err shadowing.
2017-06-21 13:24:35 -07:00
joeduffy
f6f166a5c3 Add a TODO for pulumi/lumi#260 2017-06-21 11:33:10 -07:00
joeduffy
97deabb9bd Finish interface for reading configuration¬
This continues the previous commit and establishes the interpreter
context so that we can use the new host interface.  In summary:

    * Instead of using the NullSource for destructions -- which
      doesn't hook up an interpreter and so any reads of configuration
      variables will fail -- we will enlighten the EvalSource to know
      how to orchestrate destruction interpretation.  The primary
      difference is that we don't actually run the code, but *we do*
      perform all of the necessary configuration and variable init.

    * Associate the active interpreter with the plugin context as
      we are executing, so that the host object can actually read the
      state from the heap as requested to do so by attached plugins.

    * Rename anything "engine" related to use the term "host"; this
      avoids introducing unnecesarily new terminology.

    * Add a new pkg/resource/provider/ package where we can begin
      consolidating helper functionality for resource providers.
      Right now, this includes a wrapper interface atop the gRPC
      machinery necessary to contact the host, in addition to a
      Main function that hides some boilerplate entrypoint code.

    * Add a rpcutil.IsBenignCloseErr routine to let us ignore
      "benign" gRPC errors that are knowingly returned at shutdown.

This commit completes pulumi/lumi#117.
2017-06-21 10:31:06 -07:00
joeduffy
e5f229aad8 Use Location.Read from plugin host
This addresses CR feedback from @lukehoban; namely, that we should
be going through the Read API for location reads in the plugin host
to ensure that getters are invoked as appropriate.

I also made Location's various fields private so that we aren't
tempted to make this mistake elsewhere, effectively "forcing" us
to go through the accessor methods.
2017-06-21 08:43:05 -07:00
joeduffy
d7093188f0 Introduce an interface to read config
This change adds an engine gRPC interface, and associated implementation,
so that plugins may do interesting things that require "phoning home".
Previously, the engine would fire up plugins and talk to them directly,
but there was no way for a plugin to ask the engine to do anything.

The motivation here is so that plugins can read evaluator state, such
as config information, but this change also allows richer logging
functionality than previously possible.  We will still auto-log any
stdout/stderr writes; however, explicit errors, warnings, informational,
and even debug messages may be written over the Log API.
2017-06-20 19:45:07 -07:00
joeduffy
26cf93f759 Implement get functions on all resources
This change implements the `get` function for resources.  Per pulumi/lumi#83,
this allows Lumi scripts to actually read from the target environment.

For example, we can now look up a SecurityGroup from its ARN:

    let group = aws.ec2.SecurityGroup.get(
        "arn:aws:ec2:us-west-2:153052954103:security-group:sg-02150d79");

The returned object is a fully functional resource object.  So, we can then
link it up with an EC2 instance, for example, in the usual ways:

    let instance = new aws.ec2.Instance(..., {
        securityGroups: [ group ],
    });

This didn't require any changes to the RPC or provider model, since we
already implement the Get function.

There are a few loose ends; two are short term:

    1) URNs are not rehydrated.
    2) Query is not yet implemented.

One is mid-term:

    3) We probably want a URN-based lookup function.  But we will likely
       wait until we tackle pulumi/lumi#109 before adding this.

And one is long term (and subtle):

    4) These amount to I/O and are not repeatable!  A change in the target
       environment may cause a script to generate a different plan
       intermittently.  Most likely we want to apply a different kind of
       deployment "policy" for such scripts.  These are inching towards the
       scripting model of pulumi/lumi#121, which is an entirely different
       beast than the repeatable immutable infrastructure deployments.

Finally, it is worth noting that with this, we have some of the fundamental
underpinnings required to finally tackle "inference" (pulumi/lumi#142).
2017-06-19 17:29:02 -07:00
joeduffy
7f710a241f Better handle errors during config apply 2017-06-16 15:37:58 -07:00
joeduffy
5f9ed13069 Simplify Check; make it tolerant of computed values
This change simplifies the generated Check interface for providers.
Instead of

    Check(ctx context.Context, obj *T) ([]error, error)

where T is the resource type, we have

    Check(ctx context.Context, obj *T, property string) error

This is done so that we can drive the calls to Check one property
at a time, allowing us to skip any that are computed.  (Otherwise,
we may fail the verification erroneously.)

This has the added advantage that the Check implementations are
simpler and can simply return a single error.  Furthermore, the
generated RPC code handles wrapping the result, so we can just do

    return errors.New("bad");

rather than the previous reflection-laden junk

    return resource.NewFieldError(
        reflect.TypeOf(obj), awsservice.AWSResource_Property,
        errors.New("bad"))
2017-06-16 13:34:11 -07:00
Luke Hoban
33a9452ece Merge pull request #256 from pulumi/examplestest
Add integration testing for examples
2017-06-16 10:17:51 -07:00
joeduffy
5e85e6d543 Improve property validation diagnostics
The change prints the value of a property that fails resource validation.
This makes it much easier to diagnose what's going on.
2017-06-16 10:04:49 -07:00
joeduffy
7d19abc2a3 Print the current environment
This change implements showing a summary of the current environment.
All you need to do is run

    $ lumi env

and the current environment's information will be printed.

This makes it convenient to grab resource information that might be
required, for instance, to correlate with logs (e.g., lambda ARNs).

Eventually, as per pulumi/lumi#184, we want to print details about
all of the resources too.
2017-06-16 09:46:09 -07:00
Luke Hoban
639a2d323d Test more examples
Tests all of our commonly used examples.

Also sets test parallelism to 10 by default
since we are I/O bound on API calls to
the resource providers.

Also avoids using larger EC2 examples in
our samples so that we can keep our test
costs lower :-).
2017-06-16 09:24:31 -07:00
joeduffy
d013501e04 Restructure Rendezvous to have a distinct Let vs. Meet
On the first turn, we want to distinguish between a coroutine
running that owns its turn, and a coroutine that knows it doesn't
own the turn and is simply awaiting its turn.  The old Meet logic
wasn't quite right; instead, we'll have the caller tell us this.
2017-06-15 18:20:12 -07:00