The tests in master are currently failing regularly because our current browser tests are serious memory hogs. Investigation reveals that nearly every test is retaining all of the memory it causes to be allocated. We have made some progress to being able to diagnose the problems, but we expect that problem to take some serious work to fix. We need a short-term solution though, and this is it.
Rather than modify the bundling process, we will shard the top-level test suites by name. For now, we've created 4 shards, but adding new shards is trivial if we need to.
Sharding is accomplished by creating a murmur3 hash of the top level suite names, then bucketing based on the hash output. If a test suite resolves to shard2, but we are running shard1, we simply never pass the function to `mocha.describe()`. Rather than redefine every describe statement, we have shimmed the global `window.describe()` function to accomplish this.
Production builds should never be published directly from a local
machine. Instead, the release command will now publish to a
commit-specific staging URL, so you use it to publish a release
candidate, and then when those builds have been verified, you need to
copy the RC builds from on the staging location on s3 to the production
folder.
- Add 'test:visualRegression' grunt task.
- Run 'test:visualRegression' as part of npm script 'test'.
- Add 'clean:screenshots task'.
- Clean screenshots/session when funtional tests are run.
The default behavior of the build task is to now apply the -snapshot
suffix dynamically rather than us manually hardcoding and managing it
within the source code itself. The `--release` flag will drop the
-snapshot suffix on a build, which should be used for any release
candidate.
The default behavior of the build task has also changed to create
rpm/deb packages as well. Since we've only confirmed that this works on
linux, you can override that behavior by passing `skip-os-packages`.
If you do not want to create any zip or tar.gz archives, you can pass
`--skip-archives`.
- Remove test/output and added test/screenshots (requires a Jenkins change).
- Add test/screenshots/baseline images. These document the expected state of the UI.
- Add dependency on image-diff package.
- Add utilities/compareScreenshots.js, which can be run via 'npm run compareScreenshots'.
