Commit graph

2 commits

Author SHA1 Message Date
Nathan Shively-Sanders fce7f9f87b Move .test-cost.json
Also include parameter name in test output so that people will know what
to do to change the percent chance of failure.
2019-06-19 08:40:06 -07:00
Nathan Shively-Sanders a852f2feea Skip costly tests
1. Add a script to generate a sorted list of most costly tests. A tests'
cost is roughly `runtime% / number of edits`. A slow test that's only
been updated once is much less valuable than a slow test that has
been updated 20 times: the latter test is catching more changes in the
type system.

2. Check in the results of running this script. I want to make the
skipping behaviour deterministic and the same for everybody, even though
you may get slightly better performance by examining only *your* test
changes.

3. Add code to skip tests until it reaches a 5% chance of missing an
edit. Right now this provides a 38% speedup.

Still not done:
4. Make this value configurable.
5. Make the CI configuration specify a 0% chance of missing an edit.
2019-06-14 13:35:41 -07:00