* Update properties.tmpl to render property comment as-is. WIP splitting out properties to lang-specific tables.
* Generate the constructor dynamically from the resource per language.
* Add doc functions in each language generator package for getting doc links for types..and later other functions too.
* Render the constructor params in the Go code and inject into the template.
* Generate nodejs types using the nodejs lang generator.
* Add a templates bundler. Added a new Make target for autogenerating a static bundle for the resource docs generator.
* Generate type links for all languages based on their schema type. Render the property type with a link if the underlying elements have a supporting type. Fix word-breaks for Python type names.
* Various changes including the introduction of an interface type under the codegen package to help with generating some language-specific information for the resource docs generator.
* Add a function to explicitly generate links for input types of nested types. Fix the resource doc link generator for Go. Don't replace the module name from the nodejs language type.
* Fix bug with C# property type html encoding.
* Fix some template formatting. Pass the state inputs for Python to generate the lookup function for it.
* Do not generate the examples section if there are none.
* Generating the property types per language.
* Formatting. Rename function for readability.
* Add comments. Update README.
* Use relative URLs for doc links within the main site
We run the same suite of changes that we did on gometalinter. This
ended up catching a few new issues, some of which were addressed and
some of which were baselined.
We end up doing a lot of duplicate work by invoking Gometalinter three
times when we do `make lint`. On my machine this decreases the wall
clock time of `make lint` from ~90 sec to ~15 sec, which is a pretty
drastic speedup.
* Added dist target for make, will help with Homebrew
* Try to install go dependencies before building
* Make sure dep ensure is called before trying to build SDKs
* Removed dep ensure from dist initial step
After making my prior change, I discovered that there's a build target
for this (the missing step). Since cached builds are very quick, it
seems fine to just do this as part of generate.sh, so I'm removing.
This change eliminates our dependencies on the SDK repo. Now that
SDKs are comprised solely of pulumi/pulumi artifacts, a separate repo
isn't required. This allows us to simplify some of the distribution.
The install.sh script is modified slightly, to permit overriding the
default install location using $PULUMI_INSTALL_PATH.
Our previous strategy of just using `git describe --tags --dirty` to
compute a version caused issues. The major one was that since version
sort lexigrapically, git's strategy of having a commit count without
leading zeros lead to cases where 0.11.0-dev-9 was "newer than"
0.11.0-dev-10 which is not what you want at all.
With this change, we compute a version by first seeing if the commit
is tagged, and if so, we use that tag. Otherwise, we take the closest
tag and to it append the unix timestamp of the commit and then append
a git hash.
Because we use the commit timestamp, things will sort correctly again.
Part of pulumi/home#174
While it's safe to publish the tgz that we use internally for other
repositories that are on "the link plan" after the build completes, we
shouldn't publish packages to NPM and PyPi at that point. There are
two reasons for doing this:
1. Publishing packages before they are tested, which means we could
end up publishing packages that don't work.
2. NPM prevents publishing the same package more than once, so if we
had to re-run the job (due to tests failing for transient issues), the
publish step will start failing, preventing us from running the tests
at all.
This change gets enough of the Python SDK up and running that the
empty Python program will work. Mostly just scaffolding, but the
basic structure is now in place. The primary remaining work is to
wire up resource creation to the gRPC interfaces.
In summary:
* The basic structure is as follows:
- Everything goes into sdk/python/.
- sdk/python/cmd/pulumi-langhost-python is a Go language host
that simply knows how to spawn Python processes to run out
entrypoint in response to requests by the engine.
- sdk/python/cmd/pulumi-langhost-python-exec is a little Python
shim that is invoked by the language host to run Python programs,
and is responsible for setting up the minimal goo before we can
do so (RPC connections and the like).
- sdk/python/lib/ contains a Python Pip package suitable for PyPi.
- In there, we have two packages: the root pulumi package that
contains all of the basic Pulumi programming model abstractions,
and pulumi.runtime, which contains the implementation of
resource registration, RPC interfacing with the engine, and so on.
* Add logic in our test framework to conditionalize on the language
type and react accordingly. This will allow us to skip Yarn for
Python projects and eventually run Pip if there's a requirements.txt.
* Created the basic project structure, including all of the usual
Make targets for installing into the proper places.
* Building also runs Pylint and we are clean.
There are a few other minor things in here:
* Add an "empty" test for both Node.js and Python. These pass.
* Fix an existing bug in plugin shutdown logic. At some point, we
started waiting for stderr/stdout to flush before shutting down
the plugin; but if certain failures happen "early" during the
plugin launch process, these channels will never get initialized
and so waiting for them deadlocks.
* Recently we seem to have added logic to delete test temp
directories if a failure happened during initialization of said
temp directories. This is unfortunate, because you often need to
look at the temp directory to see what failed. We already clean
them up elsewhere after the full test completes successfully, so
I don't think we need to be doing this, and I've removed it.
Still many loose ends (config, resources, etc), but it's a start!
For some tools we used, like `dep` and `gometalinter` we were just
calling `go install` which caused us to pick up whatever was in `HEAD`
at that time. Now, we move to a model where we install fixed versions,
which will change per milestone.
While doing this, I changed the way our .travis.yml file runs
everything to move as much as possible out into scripts and do so in a
way that allows us to share as much common logic across our
repositories.
This change adds a new manifest section to the checkpoint files.
The existing time moves into it, and we add to it the version of
the Pulumi CLI that created it, along with the names, types, and
versions of all plugins used to generate the file. There is a
magic cookie that we also use during verification.
This is to help keep us sane when debugging problems "in the wild,"
and I'm sure we will add more to it over time (checksum, etc).
For example, after an up, you can now see this in `pulumi stack`:
```
Current stack is demo:
Last updated at 2017-12-01 13:48:49.815740523 -0800 PST
Pulumi version v0.8.3-79-g1ab99ad
Plugin pulumi-provider-aws [resource] version v0.8.3-22-g4363e77
Plugin pulumi-langhost-nodejs [language] version v0.8.3-79-g77bb6b6
Checkpoint file is /Users/joeduffy/dev/code/src/github.com/pulumi/pulumi-aws/.pulumi/stacks/webserver/demo.json
```
This addresses pulumi/pulumi#628.
We have some lint debt (around exported functions being
underdocumented) so we have a custom target that ignores some lint
warnings. However, that target (which is the only lint target that
needs to be clean to check in) is called `lint_quiet` instead of
`lint` which is our normal linting target.
Rename the target so your fingers can learn that `make lint` is always
the right way to run linting before checkin.
This PR adds integration tests for exercising `pulumi init` and the `pulumi stack *` commands. The only functional change is merging in https://github.com/pulumi/pulumi/pull/492 , which I found while writing the tests and (of course 😁 ) wrote a regression for.
To do this I introduce a new test driver called `PulumiProgram`. This is different from the one found in the `testing/integration`package in that it doesn't try to prescribe a workflow. It really just deals in executing commands, and confirming strings are in the output.
While it doesn't hurt to have more tests for `pulumi`, my motivation here was so that I could reuse these to ensure I keep the same behavior for my pending PR that implements Cloud-enabled variants of some of these commands.
During the course of a `pulumi update`, it is possible for a resource to
become slated for deletion. In the case that this deletion is part of a
replacement, another resource with the same URN as the to-be-deleted
resource will have been created earlier. If the `update` fails after the
replacement resource is created but before the original resource has been
deleted, the snapshot must capture that the original resource still exists
and should be deleted in a future update without losing track of the order
in which the deletion must occur relative to other deletes. Currently, we
are unable to track this information because the our checkpoints require
that no two resources have the same URN.
To fix this, these changes introduce to the update engine the notion of a
resource that is pending deletion and change checkpoint serialization to
use an array of resources rather than a map. The meaning of the former is
straightforward: a resource that is pending deletion should be deleted
during the next update.
This is a fairly major breaking change to our checkpoint files, as the
map of resources is no more. Happily, though, it makes our checkpoint
files a bit more "obvious" to any tooling that might want to grovel
or rewrite them.
Fixes#432, #387.
We use `git describe --tags` to construct a version number based on
the current version tag.
The properties VERSION (when using make) and Version (when using
MSBuild) can be explicitly set to use a fixed value instead.
Fixes#13
Now that we no longer recursively call yarn via yarn run and scripts,
we can use a `--mutex network` directive in a .yarnrc file at the root
of the repository to prevent multiple copies of yarn running
concurrently.
Defer build and publish logic into the makefiles instead of trying to
use Travis to do so. This let's us sidestep Travis's not great error
handling (see pulumi/home#21 for more details)
This includes a few changes:
* The repo name -- and hence the Go modules -- changes from pulumi-fabric to pulumi.
* The Node.js SDK package changes from @pulumi/pulumi-fabric to just pulumi.
* The CLI is renamed from lumi to pulumi.
This provides a friendlier experience if the nodejs language service is
not on the path. Also, print a helpful message at the end of
`make configure` if PATH has not yet been setup.
This change adds a `make configure` target, which handles preparing
the environment for building the project. This includes existing
steps, like dep ensure and yarn installing the Node.js SDK NPM
dependencies, and also includes downloading the right Node.js/V8
includes, putting them in the right place, and then generating the
appropriate node-gyp project files that reference those includes.
This change runs the examples integration tests for every test
run. They used to be split out because the AWS tests take so long,
but now those are in their own separate package. Running the
integration tests here more frequently will prevent breaking the
most basic Lumi CLI commands and capabilities.
We are renaming Lumi to Pulumi Fabric. This change simply renames the
pulumi/lumi repo to pulumi/pulumi-fabric, without the CLI tools and other
changes that will follow soon afterwards.
This change creates a scripts/ folder, moves our existing shell
script, gocover.sh, underneath it, and factors the publish logic
out of the Makefile and into the publish.sh file.
The syntax for sed differs between Mac OS X and GNU versions of
the tool, which is rather annoying. This fixes it by leveraging
the fact that `sed -i.bak ...` works, although we now need to
clean up the *.bak file left behind. (No big deal and way better
than maintaining OS-dependent logic.)
This adds a new make target, publish, that will create a release
package and upload it to an S3 bucket. This package simply contains
the built CLI plus the core library packages, lumi, lumirt, and lumijs.
This target is invoked automatically at the end of a successful
Travis run against a push to master.
Adds a make task to generate code coverage for all Go sources.
That make task re-runs the tests, and can be fairly expensive,
so it is enabled only in the nightly tests for now.
Part of #206.
This should fix the issues with escaping in Travis (fingers crossed).
Also specifies the -e flag for echo, since switching the shell to
bash led to ANSI escape codes being uninterpreted by default.
This change fixes a few things:
* Most importantly, we need to place a leading "." in the paths
to Gometalinter, otherwise some sub-linters just silently skip
the directory altogether. errcheck is one such linter, which
is a very important one!
* Use an explicit Gometalinter.json file to configure the various
settings. This flips on a few additional linters that aren't
on by default (line line length checking). Sadly, a few that
I'd like to enable take waaaay too much time, so in the future
we may consider a nightly job (this includes code similarity,
unused parameters, unused functions, and others that generally
require global analysis).
* Now that we're running more, however, linting takes a while!
The core Lumi project now takes 26 seconds to lint on my laptop.
That's not terrible, but it's long enough that we don't want to
do the silly "run them twice" thing our Makefiles were previously
doing. Instead, we shall deploy some $$($${PIPESTATUS[1]}-1))-fu
to rely on the fact that grep returns 1 on "zero lines".
* Finally, fix the many issues that this turned up.
I think(?) we are done, except, of course, for needing to drive
down some of the cyclomatic complexity issues (which I'm possibly
going to punt on; see pulumi/lumi#259 for more details).
This disables gotype as a linter, since it depends on object files
in an annoying way (and doesn't add much beyond go build anyway).
From alecthomas/gometalinter#206, it sounds like the current plan
is to remove gotype entirely from the set of linters GML runs.
After 233c5a8 landed, I noticed there are a few things to be fixed up:
* Run gometalinter in all the right places. We need to run both in
lint and lint_quiet targets. I've also cleaned up some of the logic
around what to suppress so there's less repetition.
* We currently @ meaningful commands, which is unfortunate, since it
makes debugging Makefiles tough (especially when looking at CI build
logs). Going forward, we should only use @ for meaningless commands,
like @echo.
* The AWS project wasn't actually running tslint, because it needs to
say `tslint './pack/**/*.ts' --exclude='./pack/node_modules/**'`.
The current script of `tslint lib/aws/pack/...` wasn't actually
running lint, hence we missed a lot of AWS lint issues.
* Fix up the issues that these fixes uncovered. Mostly err shadowing.
Make nightly tests more verbose to avoid 10 minute
timeout in Travis when we have no test output.
Run aws provider tests by default again on full builds.
Adds a nightly build target that runs the long running provider
and examples tests. Enables travis to run this on cron jobs,
which we will configure to trigger nightly.
Tests all of our commonly used examples.
Also sets test parallelism to 10 by default
since we are I/O bound on API calls to
the resource providers.
Also avoids using larger EC2 examples in
our samples so that we can keep our test
costs lower :-).
This change splits the core Lumi library -- which is meant to be a pure
LumiJS library without any special status -- from the runtime library --
which is really meant to be the underpinnings of "special" functionality
that integrates with the runtime in sophisticated ways.
After this change, LumiRT is at the very bottom, and, despite it using
a subset of LumiJS, it must not trigger any functionality that would
mandate the use of the LumiJS runtime library. Atop that, the LumiJS
library is layered. And finally, above that, Lumi depends on LumiJS.
In the places we run `go build`, we should use
`go build -i` to save the `.a` files generated
during the build. This ensures the artifacts
are availble for other Go tools (linters, IDEs), and
should also improve build speeds.
This change eliminates the use of nonstandard tools in our build:
* `go test` automatically uses `GOMAXPROCS` for its parallelism
setting. In modern Go, this is now set to the number of processors.
So, there is no need to set it explicitly using `nproc`.
* We can avoid `realpath` in the `lumijs` executable by making it
a Node.js file and using a relative require import of main.
* We can avoid `realpath` in our Makefiles by just using `pwd`.
* Makeify more; add a "full build" target
This change uses make for more of our tree. Namely, the AWS provider
and LumiJS compilers each now use make to build and/or install them.
Not only does this bring about some consistency to how we build and
test things, but also made it easy to add a full build target:
$ make all
This target will build, test, and install the core Go tools, the LumiJS
compiler, and the AWS provider, in that order.
Each can be made in isolation, however, which ensures that the inner
loop for those is fast and so that, when it comes to finishing
pulumi/lumi#147, we can easily split them out and make from the top.
The AWS tests take a while to run and it's easy to forget. Further,
they are going to start taking quite a bit longer very soon.
So, this change introduces a `make full` target, which is what our
CI tests run. But the ordinary `make` skips the long tests for faster
inner loop verification. Over time, I'm sure we'll get far more
sophisticated with the split between inner vs. outer loop testing,
especially for performance, stress, and so on.
Golint by default doesn't return a non-zero exit code when lint errors
are present. To address this, we need to pass -set_exit_status.
But it gets more complicated: because we are suppressing some lint
warnings, we cannot use -set_exit_status in the lint_quiet target.
For that, we use test -z to fail if any output passes the filter.
Eventually when we do pulumi/lumi#191, we can remove this hack.
This change enables parallelism for our tests.
It also introdues a `test_core` Makefile target to just run the
core engine tests, and not the providers, since they take a long time.
This is intended only as part of the inner developer loop.
Due to https://github.com/golang/go/issues/12294, our Travis jobs are
failing with
pkg/compiler/binder/stmtexpr.go:160: no formatting directive in Errorf call
etc, etc. Perplexingly, this is supposedly fixed as of Go 1.7.5, and we are
running Go 1.8.2 in Travis. However, vet is a separate binary tool, so it seems
either (1) Go's Linux distro doesn't contain the right version or (2) Travis's
image doesn't contain the right version. I have to imagine (2) is more likely,
except that I can't actually look at the machine (and go vet doesn't have a version)!
For now, we will just skip the -printf checks.
