Commit graph

1447 commits

Author SHA1 Message Date
joeduffy 3c73b32391 Fix ineffective break lint warning
It appears Gometalinter updated to a more recent version of staticcheck,
which is more aggressive in some of its checking.  This turned up a break
warning (which was a false positive but I find the code easier to follow
this new way anyhow).
2017-06-27 11:09:46 -07:00
joeduffy 19a359e65d Specialize check failure messages 2017-06-27 10:56:33 -07:00
joeduffy 41719683e6 Run check with "" property
This change starts running Check with a "" property for cases where
global validation must take place (such as ensuring that required
configuration variables were set).  It may be safely ignored if per-
property validation is preferred by a given resource.
2017-06-27 10:46:20 -07:00
joeduffy 0ea55328f6 Specialize the null message for config reads 2017-06-27 10:45:29 -07:00
joeduffy b96538e5f5 Tolerate nil snapshots 2017-06-27 10:08:42 -07:00
joeduffy daaadd8c07 Update test baselines after source changes 2017-06-26 14:55:38 -07:00
joeduffy 2daea4c3d8 Clarify aspects of using the DCO 2017-06-26 14:46:34 -07:00
Luke Hoban d16fa87923 Address code review feedback
Use existing conversion utilities for type conversions.
2017-06-26 12:05:18 -07:00
Luke Hoban 53743da4b9 Merge pull request #269 from pulumi/nightly
Add Go code coverage reports to nightly tests
2017-06-26 09:09:29 -07:00
Luke Hoban 7164de2338 Additional gometalinter fix 2017-06-25 13:59:26 -07:00
Luke Hoban 971b12c441 Add missing error check in aws.s3.Object tests
Fix gometalinter error.
2017-06-25 13:39:14 -07:00
Luke Hoban aaa00f5a29 Fix for lambda variable capture
Fixes a bug in free variables analysis when local variable
declarations are inside MultiStatements.
2017-06-25 13:25:17 -07:00
Luke Hoban ee2c165a17 Add content properties for aws.s3.Object
Adds the `Content*` properties on S3 Objects to the
Object resource.  Allow update of Objects with new
sources and/or content properties.

Also adds AWS provider test support for validating
resource output properties.

Contributes to #218.
2017-06-25 13:24:29 -07:00
joeduffy 5362536396 Remove some obsolete names 2017-06-24 11:55:16 -07:00
Luke Hoban 1d266e38aa Add Go code coverage reports to nightly tests
Adds a make task to generate code coverage for all Go sources.

That make task re-runs the tests, and can be fairly expensive,
so it is enabled only in the nightly tests for now.

Part of #206.
2017-06-23 16:07:05 -07:00
Luke Hoban a11396f58f Merge pull request #266 from pulumi/testfix
Make concurrent AWS test runs safer
2017-06-23 16:06:09 -07:00
Luke Hoban e663b90ce7 Make concurrent AWS test runs safer
Instead of cleaning up all resources that might have been
created by the test suite, we now only cleanup resources
created by this specific run of the tests.  That makes
concurrent test execution safer, and the tests more self-
contained.  However, it does increase the chances of
accidentially leaving behind resources when tests fail.

Fixes #265.
2017-06-23 15:50:58 -07:00
joeduffy 3c1041af49 Update license headers 2017-06-23 14:53:41 -07:00
joeduffy 06aa828487 Update CONTRIBUTING.md and LICENSE
These changes reflect the decision to use the DCO instead of CLA.
2017-06-23 13:49:53 -07:00
Joe Duffy b4e8af9f32 Enable VM-based builds in Travis (#264)
This change enables VM-based builds in Travis, versus the default of
container-based builds.  This will give us more memory (7.5GB rather
than 4GB max), and more compute (~2, bursted rather than 2).  This
may help to fix some of the build/test speed time issues we are seeing.
2017-06-22 17:56:45 -07:00
Luke Hoban 4f3c616a45 Fix lint errors
Fix lint errors triggered by the previous commit.
2017-06-22 17:29:34 -07:00
Luke Hoban 201a3d1a51 Additional nightly test fixes
Address several issues with running the Beanstalk
example in newer AWS regions with different requirements.

Ensures S3 bucket names adhere to required naming patterns
outside of us-east-1.

Also add InstanceProfile and ServiceRole configuration to the
beanstalk example as required in newer regions.
2017-06-22 16:59:16 -07:00
joeduffy d05e7ace91 Ensure we close the plugin host/context
This adds a few missing closes for the plugin host/context.  This
should fix pulumi/lumi#261.  Eventually when we have more robust
nightly test options, and want to spend the time, we should think
about doing more rigorous stress testing that kills processes at
inopportune times and guarantees we don't leak.  I've filed
pulumi/lumi#263 to do that.
2017-06-22 15:18:29 -07:00
joeduffy a019880ad5 Switch to /bin/bash as the shell for our Makefiles
This should fix the issues with escaping in Travis (fingers crossed).
Also specifies the -e flag for echo, since switching the shell to
bash led to ANSI escape codes being uninterpreted by default.
2017-06-22 14:57:09 -07:00
joeduffy 7a3dfb41be Increase Gometalinter deadline to 5m
Apparently because Travis is using containers with a small number
of processors, 2m isn't long enough a deadline.  (I have never seen
us exceed this on my own machine).  Increasing to 5m.
2017-06-22 12:32:50 -07:00
joeduffy 8b57310854 Tidy up more lint
This change fixes a few things:

* Most importantly, we need to place a leading "." in the paths
  to Gometalinter, otherwise some sub-linters just silently skip
  the directory altogether.  errcheck is one such linter, which
  is a very important one!

* Use an explicit Gometalinter.json file to configure the various
  settings.  This flips on a few additional linters that aren't
  on by default (line line length checking).  Sadly, a few that
  I'd like to enable take waaaay too much time, so in the future
  we may consider a nightly job (this includes code similarity,
  unused parameters, unused functions, and others that generally
  require global analysis).

* Now that we're running more, however, linting takes a while!
  The core Lumi project now takes 26 seconds to lint on my laptop.
  That's not terrible, but it's long enough that we don't want to
  do the silly "run them twice" thing our Makefiles were previously
  doing.  Instead, we shall deploy some $$($${PIPESTATUS[1]}-1))-fu
  to rely on the fact that grep returns 1 on "zero lines".

* Finally, fix the many issues that this turned up.

I think(?) we are done, except, of course, for needing to drive
down some of the cyclomatic complexity issues (which I'm possibly
going to punt on; see pulumi/lumi#259 for more details).
2017-06-22 12:09:46 -07:00
Luke Hoban 9f5baea84c Fix minor typo in README 2017-06-22 11:36:09 -07:00
Luke Hoban a63efc42a3 Propagate errors on deployment failures
We were not propagating the error from `deployLatest` through
to the CLI error result.  Despite out recent efforts to integrate
gometalinter, there were also several additional similar cases of
ignored error results reported by `errcheck`.  Not yet clear why
these are not being reported via gometalinter.

Fixes #262.
2017-06-21 22:02:57 -07:00
Luke Hoban 72bb2b2309 Further improvements for nightly tests
Make RestAPI more robust to TooManyRequestsException.

Fix imports in minimal example.

Make printing for examples test more explicit to help with diagnostics during parallel test execution.
2017-06-21 17:33:47 -07:00
joeduffy 9fe8ae1c7e Disable gotype linter
This disables gotype as a linter, since it depends on object files
in an annoying way (and doesn't add much beyond go build anyway).
From alecthomas/gometalinter#206, it sounds like the current plan
is to remove gotype entirely from the set of linters GML runs.
2017-06-21 14:46:03 -07:00
joeduffy 3fdfc9c7bc Run lint after build/test 2017-06-21 14:38:52 -07:00
joeduffy d1182dc54f Improve pkg/compiler/ast/ cyclomatic complexity
Part of pulumi/lumi#259.
2017-06-21 14:24:10 -07:00
joeduffy 9179d58eb1 Install GoMetaLinter in Travis
And also rejigger the Makefiles to properly error if it's missing.
2017-06-21 14:23:48 -07:00
joeduffy 7fe8052941 Fix some lint in our lint
After 233c5a8 landed, I noticed there are a few things to be fixed up:

    * Run gometalinter in all the right places.  We need to run both in
      lint and lint_quiet targets.  I've also cleaned up some of the logic
      around what to suppress so there's less repetition.

    * We currently @ meaningful commands, which is unfortunate, since it
      makes debugging Makefiles tough (especially when looking at CI build
      logs).  Going forward, we should only use @ for meaningless commands,
      like @echo.

    * The AWS project wasn't actually running tslint, because it needs to
      say `tslint './pack/**/*.ts' --exclude='./pack/node_modules/**'`.
      The current script of `tslint lib/aws/pack/...` wasn't actually
      running lint, hence we missed a lot of AWS lint issues.

    * Fix up the issues that these fixes uncovered.  Mostly err shadowing.
2017-06-21 13:24:35 -07:00
Britton Forsyth 233c5a8c52 [#245, #226, #185] Finished linting (#257)
* Added typescript linting to Makefiles
* Added tslint to CI and ran on examples
* Fixed tslint calling in Makefiles
* Deleted unnecessary files
2017-06-21 11:58:22 -07:00
joeduffy f6f166a5c3 Add a TODO for pulumi/lumi#260 2017-06-21 11:33:10 -07:00
joeduffy 986f6cbe28 Specify nil for the host in AWS tests 2017-06-21 11:27:08 -07:00
joeduffy dae6736772 Read real config for the AWS region
This change uses the machinery added to pulumi/lumi#117 to read the
live AWS region config state, and prefer it to `AWS_REGION`.  We
still respect the region environment variable -- since this is the
common way to do configuration like this with AWS tools -- but this
at least avoids the issue of AWS_REGION being different and having
your Lumi scripts compute differing regions and failing impressively.
2017-06-21 10:58:23 -07:00
joeduffy 97deabb9bd Finish interface for reading configuration¬
This continues the previous commit and establishes the interpreter
context so that we can use the new host interface.  In summary:

    * Instead of using the NullSource for destructions -- which
      doesn't hook up an interpreter and so any reads of configuration
      variables will fail -- we will enlighten the EvalSource to know
      how to orchestrate destruction interpretation.  The primary
      difference is that we don't actually run the code, but *we do*
      perform all of the necessary configuration and variable init.

    * Associate the active interpreter with the plugin context as
      we are executing, so that the host object can actually read the
      state from the heap as requested to do so by attached plugins.

    * Rename anything "engine" related to use the term "host"; this
      avoids introducing unnecesarily new terminology.

    * Add a new pkg/resource/provider/ package where we can begin
      consolidating helper functionality for resource providers.
      Right now, this includes a wrapper interface atop the gRPC
      machinery necessary to contact the host, in addition to a
      Main function that hides some boilerplate entrypoint code.

    * Add a rpcutil.IsBenignCloseErr routine to let us ignore
      "benign" gRPC errors that are knowingly returned at shutdown.

This commit completes pulumi/lumi#117.
2017-06-21 10:31:06 -07:00
joeduffy e5f229aad8 Use Location.Read from plugin host
This addresses CR feedback from @lukehoban; namely, that we should
be going through the Read API for location reads in the plugin host
to ensure that getters are invoked as appropriate.

I also made Location's various fields private so that we aren't
tempted to make this mistake elsewhere, effectively "forcing" us
to go through the accessor methods.
2017-06-21 08:43:05 -07:00
joeduffy d7093188f0 Introduce an interface to read config
This change adds an engine gRPC interface, and associated implementation,
so that plugins may do interesting things that require "phoning home".
Previously, the engine would fire up plugins and talk to them directly,
but there was no way for a plugin to ask the engine to do anything.

The motivation here is so that plugins can read evaluator state, such
as config information, but this change also allows richer logging
functionality than previously possible.  We will still auto-log any
stdout/stderr writes; however, explicit errors, warnings, informational,
and even debug messages may be written over the Log API.
2017-06-20 19:45:07 -07:00
joeduffy c39280a545 Synchronize access to type caches
We run tests in parallel and recently we began hitting a high enough
degree of parallelism that we've begun seeing "unsynchronized access
to map" errors in our test passes (intermittently).  The root cause
is that access to the type symbol caches aren't synchronized.  It would
be ideal if we actually rewired these to be cached in the compiler
context -- rather than being global -- but this fix is sufficient for
now.  We will simply synchronize access using a Mutex.
2017-06-20 18:16:53 -07:00
Luke Hoban bcc37d78a8 Nightly test fixes
Fix a couple of issues that have been preventing nightlies
from running cleanly.
2017-06-20 12:29:04 -07:00
Luke Hoban 6b1fbe9ba0 Fix intrinsics tets
The previous change broke the intrinsics tests in the
case that the `lumirt` package was not installed on
the system (which is the case in Travis given our build
order).

Reverting to the previous pattern of creating a fake
`lumirt` package to run the tests in to avoid the
dependency on an externally installed `lumirt`.
2017-06-20 11:07:13 -07:00
Luke Hoban 6b7616ce1a Merge branch 'master' of https://github.com/pulumi/lumi 2017-06-20 10:30:27 -07:00
Luke Hoban 8d80871712 Fix string quoting in jsonStringify
Implements correct QuoteJSONString behaviour per ECMAScript
spec.

Also refactors intrinsic test harness to make it easier to add new
intrinsics tests going forward.
2017-06-20 10:29:53 -07:00
joeduffy 26cf93f759 Implement get functions on all resources
This change implements the `get` function for resources.  Per pulumi/lumi#83,
this allows Lumi scripts to actually read from the target environment.

For example, we can now look up a SecurityGroup from its ARN:

    let group = aws.ec2.SecurityGroup.get(
        "arn:aws:ec2:us-west-2:153052954103:security-group:sg-02150d79");

The returned object is a fully functional resource object.  So, we can then
link it up with an EC2 instance, for example, in the usual ways:

    let instance = new aws.ec2.Instance(..., {
        securityGroups: [ group ],
    });

This didn't require any changes to the RPC or provider model, since we
already implement the Get function.

There are a few loose ends; two are short term:

    1) URNs are not rehydrated.
    2) Query is not yet implemented.

One is mid-term:

    3) We probably want a URN-based lookup function.  But we will likely
       wait until we tackle pulumi/lumi#109 before adding this.

And one is long term (and subtle):

    4) These amount to I/O and are not repeatable!  A change in the target
       environment may cause a script to generate a different plan
       intermittently.  Most likely we want to apply a different kind of
       deployment "policy" for such scripts.  These are inching towards the
       scripting model of pulumi/lumi#121, which is an entirely different
       beast than the repeatable immutable infrastructure deployments.

Finally, it is worth noting that with this, we have some of the fundamental
underpinnings required to finally tackle "inference" (pulumi/lumi#142).
2017-06-19 17:29:02 -07:00
Luke Hoban ac4a56bb6d Additional tslint cleanup in examples 2017-06-19 17:09:55 -07:00
Luke Hoban c265620f28 Makefile updates
Make nightly tests more verbose to avoid 10 minute
timeout in Travis when we have no test output.

Run aws provider tests by default again on full builds.
2017-06-17 15:43:50 -07:00
Britton Forsyth 2d639abcf1 Added typescript linting to Makefiles (#248)
Added necessary tslint.json files to Makefile locations and enabled tslint at these under the lib folder to lint as part of the build.
2017-06-16 18:01:36 -07:00