This change upgrades gRPC to 1.6.0 to pick up a few bug fixes.
We also use the full address for gRPC endpoints, including the
interface name, as otherwise we pick the wrong interface on Linux.
There's a fair bit of clean up in here, but the meat is:
* Allocate the language runtime gRPC client connection on the
goroutine that will use it; this eliminates race conditions.
* The biggie: there *appears* to be a bug in gRPC's implementation
on Linux, where it doesn't implement WaitForReady properly. The
behavior I'm observing is that RPC calls will not retry as they
are supposed to, but will instead spuriously fail during the RPC
startup. To work around this, I've added manual retry logic in
the shared plugin creation function so that we won't even try
to use the client connection until it is in a well-known state.
pulumi/pulumi-fabric#337 tracks getting to the bottom of this and,
ideally, removing the work around.
The other minor things are:
* Separate run.js into its own module, so it doesn't include
index.js and do a bunch of random stuff it shouldn't be doing.
* Allow run.js to be invoked without a --monitor. This makes
testing just the run part of invocation easier (including
config, which turned out to be super useful as I was debugging).
* Tidy up some messages.
If a resource's planning operation is to do nothing, we can safely
assume that all of its properties are stable. This can be used during
planning to avoid cascading updates that we know will never happen.
This change adds the ability to perform runtime logging, including
debug logging, that wires up to the Pulumi Fabric engine in the usual
ways. Most stdout/stderr will automatically go to the right place,
but this lets us add some debug tracing in the implementation of the
runtime itself (and should come in handy in other places, like perhaps
the Pulumi Framework and even low-level end-user code).
As explained in pulumi/pulumi-fabric#293, we were a little ad-hoc in
how configuration was "applied" to resource providers.
In fact, config wasn't ever communicated directly to providers; instead,
the resource providers would simply ask the engine to read random heap
locations (via tokens). Now that we're on a plan where configuration gets
handed to the program at startup, and that's that, and where generally
speaking resource providers never communicate directly with the language
runtime, we need to take a different approach.
As such, the resource provider interface now offers a Configure RPC
method that the resource planning engine will invoke at the right
times with the right subset of configuration variables filtered to
just that provider's package. This fixespulumi/pulumi#293.
This change simplifies the provider RPC interface slightly:
1) Eliminate Get. We really don't need it anymore. There are
several possibly-interesting scenarios down the road that may
demand it, but when we get there, we can consider how best to
bring this back. Furthermore, the old-style Get remains mostly
incompatible with Terraform anyway.
2) Pass URNs, not type tokens, across the RPC boundary. This gives
the provider access to more interesting information: the type,
still, but also the name (which is no longer an object property).
This helps avoid running into file handle limits when creating archives including thousands of node_modules files.
Tracking a more complete fix through all other codepaths related to assets as part of #325.
This change ensures we close all Blobs in the asset/archive logic.
In particular, the archive.Read function returns a map of files to
Blobs and after we are done copying the contents we must ensure
that we invoke Close, otherwise we may leak file handles, sockets,
and so on. This may or may not be the culprit to the "too many
files open" errors we are hitting while deploying the M5 bits.
This changes a few things in the CLI, mostly just prettying it up:
* Label all steps more clearly with the kind of step. Also
unify the way we present this during planning and deployment.
* Summarize the changes that *did not* get made just as clearly
as those that did. In other words, stuff like this:
info: 2 resources changed:
+1 resource created
-1 resource deleted
5 resources unchanged
and
info: no resources required
5 resources unchanged
* Always print output properties when they are pertinent.
This includes creates, replacements, and updates.
* Show replacement creates and deletes very distinctly. The
create parts show up minty green and the delete parts show up
rosey red. These are the "physical" steps, compared to the
"logical" step of replacement (which remains marigold).
I still don't love where we are here. The asymmetry between
planning and deployment bugs me, and could be surprising.
("Hey, my deploy doesn't look like my plan!") I don't know
what developers will want to see here and I feel like in
general we are spewing far too much into the CLI to make it
even useful for anything but diagnosing failures afterwards.
I propose that we should do a deep dive on this during the
CLI epic, pulumi/pulumi-service#2.
This resolvespulumi/pulumi-fabric#305.
At the moment, we permit resources to carry a name, which the
engine uses as part of URN creation. Unfortunately, the property
"name" has a very high chance of conflicting with meaningful
user-authored properties. And furthermore this sort of name,
although key to creating URNs which are core to how the overall
system performs deployments and manages resources, are seldom
used programmatically in Lumi programs. As a result, it's a real
nuisance that we stole the good name.
This change renames that property from "name" to "urnName". This
not only has a lower likelihood of conflicting, but it also looks
reasonable sitting alongside the "urn" property. In fact, in some
future universe, after some of the upcoming runtime changes, we
may not even need the name property on the objects whatsoever.
I had originally toyed with the idea of eliminating the Resource
versus NamedResource distinction. This is certainly simpler and
remains a possibility. I didn't do that right now, however, because
the flexibility of letting resource providers name resources however
they see fit still seems possibly useful. For example, we keep
talking about whether functions can be auto-named based on hashes.
Until we've run those conversations to ground, I'd hate to do some
work that just needs to be undone in order to enable a scenario that
has a non-trivial likelihood of us wanting to explore.
We are now freely flowing computed and output properties across the
RPC boundary with providers. As such, we need to tolerate them in
a few more places. Namely, mapping to and from regular non-resource
property values, and also when copying RPC resource state back onto
live runtime objects.
We are renaming Lumi to Pulumi Fabric. This change simply renames the
pulumi/lumi repo to pulumi/pulumi-fabric, without the CLI tools and other
changes that will follow soon afterwards.
This changes the RPC interfaces between Lumi and provider ever so
slightly, so that we can track default properties explicitly. This
is required to perform accurate diffing between inputs provided by
the developer, inputs provided by the system, and outputs. This is
particularly important for default values that may be indeterminite,
such as those we use in the bridge to auto-generate unique IDs.
Otherwise, we fail to reapply defaults correctly, and trick the
provider into thinking that properties changed when they did not.
This is a small step towards pulumi/lumi#306, in which we will defer
even more responsibility for diffing semantics to the providers.
This change serializes unknown properties anywhere in the entire
property structure, including deeply embedded inside object maps, etc.
This is now done in such a way that we can recover both the computed
nature of the serialized property, along with its expected eventual
type, on the other side of the RPC boundary.
This will let us have perfect fidelity with the new bridge's view on
computed properties, rather than special casing them on "one side".
For Update and Delete operations, we provided just the input state
for a resource. This is insufficient, because the provider may need
to depend on output state from the Create or prior Update operations.
This change merges the output atop the input during the step application.
As part of the bridge bringup, I've discoverd that the property state
returned from Creates does *not* always equal the state that is then
read from calls to Get. (I suspect this is a bug and that they should
be equivalent, but I doubt it's fruitfal to try and track down all
occurrences of this; I bet it's widespread). To cope with this, we will
return state from Create and Update, instead of issuing a call to Get.
This was a design we considered to start with and frankly didn't have
a super strong reason to do it the current way, other than that it seemed
elegant to place all of the Get logic in one place.
Note that providers may choose to return nil, in which case we will read
state from the provider in the usual Get style.
This change mirrors the dynamic marshaling structure on the static
definition of the Asset and Archive types. This ensures that they
marshal correctly even when deeply embedded inside other structures.
This change brings the same typed serialization we use for RPC
to the serialization of deployments. This ensures that we get
repeatable diffs from one deployment to the next.
This reverts commit c3db70849d.
I've opted to take a new strategy to ensure the bridge properties
don't conflict (with manual renames), similar to the name property.
This change recognizes assets and archives as 1st class resource
property values. This is necessary to support them in the new bridge
work, and lays the foundation for fixing pulumi/lumi#153.
I also took the opportunity to clean up some old cruft in the
resource properties area.
This renames the basemost resource properties, id and urn, to
names that are less likely to conflict with properties that real
resources will want to use, pid and upn (provider ID and Universal
Pulumi Name, respectively).
I actually ran into this with the current bridge work. An alternative
solution would be to require derived resources to pick different names,
however this is unfortunate because usually they are more "user-facing"
than ours. Another alternative is to not hijack the object properties
at all, but that too is problematic because we use these properties
during the evaluation of plans and deployments.
This seems like a reasonable middle ground.
This adds a ReadLocations RPC function to the engine interface, alongside
the singular ReadLocation. The plural function takes a single token that
represents a module or class and we will then return all of the module
or class (static) properties that are currently known.
This adds a handy MapReplace function on pkg/resource's PropertyMap and
PropertyValue types. This is just like the existing Mappable function,
except that it permits easy replacement of elements as the map transformation
occurs. We need this to perform float64=>int transformations.
We fail very late in the process of plan application, should a duplicate
URN arise. This change fails as early in the process as possible and
ensures that it does so with good line number information.
This properly unwinds the interpreter should something happen that
results in cancellation. This occurs, for example, when the planning
engine encounters an error and decides that it doesn't need to proceed
further with evaluation before it simply goes ahead and exits.
