[SIEM] Run Cypress Tests Against Elastic Cloud & Cypress Command Line / Reporting (#42804)

# [SIEM] Run Cypress Tests Against Elastic Cloud & Cypress Command Line / Reporting (#42804) This PR contains enhancements to the `Cypress` end-to-end (e2e) testing support in the SIEM app, as discussed with @spalger Note: this PR depends on test refactorings in https://github.com/elastic/kibana/pull/42766 ## New: Run `Cypress` Tests Against Elastic Cloud Cypress tests may now be run against a remote Elastic Cloud instance (override `baseUrl`), interactively or via the command line. Credentials are specified via `kibna.dev.yml` or environment variables. ## New: Run `Cypress` Tests On the Command Line Run tests on the command line (override `baseUrl` and pass credentials via the `CYPRESS_ELASTICSEARCH_USERNAME` and `CYPRESS_ELASTICSEARCH_PASSWORD` environment variables), via command line. ## New: Reporting Reports are configured to include: - An HTML report, suitable for e-mail notifications - Screenshots - Junit reports (for integration with Kibana CI) - Videos (optional) Reports generated when `Cypress` tests are run on the command line are output to the `target` directory, which follows specific conventions used by the Kibana CI process: * Any directory under `target` that begins with `kibana-`, e.g. `kibana-siem` will be uploaded as an artifact to a bucket * Junit reports are picked up from the `target/junit` directory The artifacts generated by running `Cypress` tests on the command line conform to the above conventions. ### HTML Reports An HTML report (e.g. for email notifications) is output to: ``` target/kibana-siem/cypress/results/output.html ``` ### Screenshots Screenshots of failed tests are output to: ``` target/kibana-siem/cypress/screenshots ``` ### `junit` Reports The Kibana CI process reports `junit` test results from the `target/junit` directory. Cypress `junit` reports are generated in `target/kibana-siem/cypress/results` and copied to the `target/junit` directory. ### Videos (optional) Videos are disabled by default, but can optionally be enabled by setting the `CYPRESS_video=true` environment variable: ``` CYPRESS_video=true yarn cypress:run ``` Videos are (optionally) output to: ``` target/kibana-siem/cypress/videos ``` # Updated `README` The full details and instructions for running Cypress tests are in the `siem/cypress/README.md`, which was updated to reflect the changes in this in this PR, but in a nutshell, you may run tests interactively with the following (new) environment variables: ```sh cd x-pack/legacy/plugins/siem CYPRESS_baseUrl=http://localhost:5601 CYPRESS_ELASTICSEARCH_USERNAME=elastic CYPRESS_ELASTICSEARCH_PASSWORD=<password> yarn cypress:open ``` Running the command line version of the tests, which will output the reports described above, is a similar experence: ```sh cd x-pack/legacy/plugins/siem CYPRESS_baseUrl=http://localhost:5601 CYPRESS_ELASTICSEARCH_USERNAME=elastic CYPRESS_ELASTICSEARCH_PASSWORD=<password> yarn cypress:run ``` https://github.com/elastic/siem-team/issues/435 https://github.com/elastic/siem-team/issues/437
2019-08-14 13:38:23 -06:00 · 2019-08-14 13:38:23 -06:00 · 017c860a3f
parent 8c134f6c58
commit 017c860a3f
8 changed files with 431 additions and 77 deletions
--- a/x-pack/legacy/plugins/siem/cypress.json
+++ b/x-pack/legacy/plugins/siem/cypress.json
@ -1,3 +1,7 @@
 {
-  "baseUrl": "http://localhost:5601"
+  "baseUrl": "http://localhost:5601",
+  "screenshotsFolder": "../../../../target/kibana-siem/cypress/screenshots",
+  "trashAssetsBeforeRuns": false,
+  "video": false,
+  "videosFolder": "../../../../target/kibana-siem/cypress/videos"
 }
--- a/x-pack/legacy/plugins/siem/cypress/README.md
+++ b/x-pack/legacy/plugins/siem/cypress/README.md
@ -1,70 +1,242 @@
 # Cypress Tests

-The `siem/cypress` directory contains end to end tests (specific to the `SIEM` app) that execute via [Cypress](https://www.cypress.io/).
+The `siem/cypress` directory contains end to end tests, (plus a few tests
+that rely on mocked API calls), that execute via [Cypress](https://www.cypress.io/).

-At present, these tests are only executed in a local development environment; they are **not** integrated in the Kibana CI infrastructure, and therefore do **not** run automatically when you submit a PR.
+Cypress tests may be run against:

-See the `Server and Authentication Requirements` section below for additional details.
+- A local Kibana instance, interactively or via the command line. Credentials
+are specified via `kibna.dev.yml` or environment variables.
+- A remote Elastic Cloud instance (override `baseUrl`), interactively or via
+the command line. Again, credentials are specified via `kibna.dev.yml` or
+environment variables.
+- As part of CI (override `baseUrl` and pass credentials via the
+`CYPRESS_ELASTICSEARCH_USERNAME` and `CYPRESS_ELASTICSEARCH_PASSWORD`
+environment variables), via command line.

-## Organizing Tests and (Mock) Data
+At present, Cypress tests are only executed manually. They are **not** yet
+integrated in the Kibana CI infrastructure, and therefore do **not** run
+automatically when you submit a PR.

- Code and CSS selectors that may be re-used across tests should be added to `siem/cypress/integration/lib`, as described below
- Smoke Tests are located in `siem/cypress/integration/smoke_tests`
- Mocked responses from the server are located in `siem/cypress/fixtures`
+## Smoke Tests

-### `cypress/integration/lib`
+Smoke Tests are located in `siem/cypress/integration/smoke_tests`

-The `cypress/integration/lib` folder contains code intended to be re-used across many different tests.
+## Test Helpers

- Files named `helpers.ts` (e.g. `siem/cypress/integration/lib/login/helpers.ts`) contain functions (e.g. `login`) that may be imported and invoked from multiple tests.
+_Test helpers_ are functions that may be re-used across tests.

- Files named `selectors.ts` export CSS selectors for re-use. For example, `siem/cypress/integration/lib/login/selectors.ts` exports the following selector that matches the Username text area in the Kibana login page:
+- Reusable code and CSS selectors should be added to
+`siem/cypress/integration/lib`, as described below.

-```
+### Reusable Test Helper Functions and CSS Selectors
+
+The `cypress/integration/lib` directory contains code intended to be re-used
+across many different tests. Add reusable test helper functions and CSS
+selectors to directories under `cypress/integration/lib`.
+
+- Files named `helpers.ts` (e.g. `siem/cypress/integration/lib/login/helpers.ts`)
+contain functions (e.g. `login`) that may be imported and invoked from multiple tests.
+
+- Files named `selectors.ts` export CSS selectors for re-use. For example,
+`siem/cypress/integration/lib/login/selectors.ts` exports the following selector
+that matches the Username text area in the Kibana login page:
+
+```sh
 export const USERNAME = '[data-test-subj="loginUsername"]';
 ```

-## Server and Authentication Requirements
+## Mock Data

-The current version of the Smoke Tests require running a local Kibana server that connects to an instance of `elasticsearch`. A file named `config/kibana.dev.yml` like the example below is required to run the tests:
+We prefer not to mock API responses in most of our smoke tests, but sometimes
+it's necessary because a test must assert that a specific value is rendered,
+and it's not possible to derive that value based on the data in the
+envrionment where tests are running.
+
+Mocked responses API from the server are located in `siem/cypress/fixtures`.
+
+## Authentication
+
+When running tests, there are two ways to specify the credentials used to
+authenticate with Kibana:
+
+- Via `kibana.dev.yml` (recommended for developers)
+- Via the `CYPRESS_ELASTICSEARCH_USERNAME` and `CYPRESS_ELASTICSEARCH_PASSWORD`
+environment variables (recommended for CI), or when testing a remote Kibana
+instance, e.g. in Elastic Cloud.
+
+Note: Tests that use the `login()` test helper function for authentication will
+automatically use the `CYPRESS_ELASTICSEARCH_USERNAME` and `CYPRESS_ELASTICSEARCH_PASSWORD`
+environment variables when they are defined, and fall back to the values in
+`config/kibana.dev.yml` when they are unset.
+
+### Content Security Policy (CSP) Settings
+
+Your local or cloud Kibana server must have the `csp.strict: false` setting
+configured in `kibana.dev.yml`, or `kibana.yml`, as shown in the example below:

 ```yaml
+csp.strict: false
+```
+
+The above setting is required to prevent the _Please upgrade
+your browser_ / _This Kibana installation has strict security requirements
+enabled that your current browser does not meet._ warning that's displayed for
+unsupported user agents, like the one reported by Cypress when running tests.
+
+### Example `kibana.dev.yml`
+
+If you're a developer running tests interactively or on the command line, the
+easiset way to specify the credentials used for authentication is to update
+ `kibana.dev.yml` per the following example:
+
+```yaml
+csp.strict: false
 elasticsearch:
  username: 'elastic'
  password: '<password>'
  hosts: ['https://<server>:9200']
 ```

-The `username` and `password` from `config/kibana.dev.yml` will be read by the `login` test helper function when tests authenticate with Kibana.
-
-See the `Running Tests Interactively` section for details.
-
-### Content Security Policy (CSP) Settings
-
- Your local or cloud Kibana server must have the `csp.strict: false` setting
-configured in `kibana.dev.yml`, or `kibana.yml`, as shown in the example below:
-
- ```yaml
-csp.strict: false
-```
-
 ## Running Tests Interactively

-To run tests in interactively via the Cypress test runner:
+Use the Cypress interactive test runner to develop and debug specific tests
+by adding a `.only` to the test you're developing, or click on a specific
+spec in the interactive test runner to run just the tests in that spec.

-1. Create and configure a `config/kibana.dev.yml`, as described in the `Server and Authentication Requirements` section above.
+To run and debug tests in interactively via the Cypress test runner:

-2. Start a local instance of the Kibana development server:
+1. Disable CSP on the local or remote Kibana instance, as described in the
+_Content Security Policy (CSP) Settings_ section above.

-```
+2. To specify the credentials required for authentication, configure
+`config/kibana.dev.yml`, as described in the _Server and Authentication
+Requirements_ section above, or specify them via environment variables
+as described later in this section.
+
+3. Start a local instance of the Kibana development server (only if testing against a
+local host):
+
+```sh
 yarn start --no-base-path
 ```

-3. Launch the Cypress interactive test runner:
+4. Launch the Cypress interactive test runner via one of the following options:
+
+- To run tests interactively against the default (local) host specified by
+`baseUrl`, as configured in `plugins/siem/cypress.json`:

 ```sh
 cd x-pack/legacy/plugins/siem
 yarn cypress:open
 ```

-4. Click the `Run all specs` button in the Cypress test runner
+- To (optionally) run tests interactively against a different host, pass the
+`CYPRESS_baseUrl` environment variable on the command line when launching the
+test runner, as shown in the following example:
+
+```sh
+cd x-pack/legacy/plugins/siem
+CYPRESS_baseUrl=http://localhost:5601 yarn cypress:open
+```
+
+- To (optionally) override username and password via environment variables when
+running tests interactively:
+
+```sh
+cd x-pack/legacy/plugins/siem
+CYPRESS_baseUrl=http://localhost:5601 CYPRESS_ELASTICSEARCH_USERNAME=elastic CYPRESS_ELASTICSEARCH_PASSWORD=<password> yarn cypress:open
+```
+
+5. Click the `Run all specs` button in the Cypress test runner (after adding
+a `.only` to an `it` or `describe` block).
+
+## Running (Headless) Tests on the Command Line
+
+To run (headless) tests on the command line:
+
+1. Disable CSP on the local or remote Kibana instance, as described in the
+_Content Security Policy (CSP) Settings_ section above.
+
+2. To specify the credentials required for authentication, configure
+`config/kibana.dev.yml`, as described in the _Server and Authentication
+Requirements_ section above, or specify them via environment variables
+as described later in this section.
+
+3. Start a local instance of the Kibana development server (only if testing against a
+local host):
+
+```sh
+yarn start --no-base-path
+```
+
+4. Launch the Cypress command line test runner via one of the following options:
+
+- To run tests on the command line against the default (local) host specified by
+`baseUrl`, as configured in `plugins/siem/cypress.json`:
+
+```sh
+cd x-pack/legacy/plugins/siem
+yarn cypress:run
+```
+
+- To (optionally) run tests on the command line against a different host, pass
+`CYPRESS_baseUrl` as an environment variable on the command line, as shown in
+the following example:
+
+```sh
+cd x-pack/legacy/plugins/siem
+CYPRESS_baseUrl=http://localhost:5601 yarn cypress:run
+```
+
+- To (optionally) override username and password via environment variables when
+running via the command line:
+
+```sh
+cd x-pack/legacy/plugins/siem
+CYPRESS_baseUrl=http://localhost:5601 CYPRESS_ELASTICSEARCH_USERNAME=elastic CYPRESS_ELASTICSEARCH_PASSWORD=<password> yarn cypress:run
+```
+
+## Reporting
+
+When Cypress tests are run on the command line via `yarn cypress:run`,
+reporting artifacts are generated under the `target` directory in the root
+of the Kibana, as detailed for each artifact type in the sections bleow.
+
+### HTML Reports
+
+An HTML report (e.g. for email notifications) is output to:
+
+```
+target/kibana-siem/cypress/results/output.html
+```
+
+### Screenshots
+
+Screenshots of failed tests are output to:
+
+```
+target/kibana-siem/cypress/screenshots
+```
+
+### `junit` Reports
+
+The Kibana CI process reports `junit` test results from the `target/junit` directory.
+
+Cypress `junit` reports are generated in `target/kibana-siem/cypress/results`
+and copied to the `target/junit` directory.
+
+### Videos (optional)
+
+Videos are disabled by default, but can optionally be enabled by setting the
+`CYPRESS_video=true` environment variable:
+
+```
+CYPRESS_video=true yarn cypress:run
+```
+
+Videos are (optionally) output to:
+
+```
+target/kibana-siem/cypress/videos
+```
--- a/x-pack/legacy/plugins/siem/cypress/integration/smoke_tests/fields_browser/fields_browser.spec.ts
+++ b/x-pack/legacy/plugins/siem/cypress/integration/smoke_tests/fields_browser/fields_browser.spec.ts
@ -21,7 +21,7 @@ import {
 } from '../../lib/fields_browser/selectors';
 import { logout } from '../../lib/logout';
 import { HOSTS_PAGE } from '../../lib/urls';
-import { loginAndWaitForPage } from '../../lib/util/helpers';
+import { loginAndWaitForPage, DEFAULT_TIMEOUT } from '../../lib/util/helpers';

 const defaultHeaders = [
  { id: '@timestamp' },
@ -200,9 +200,9 @@ describe('Fields Browser', () => {

    cy.get(`[data-test-subj="headers-group"]`).then(headersDropArea => drop(headersDropArea));

-    clickOutsideFieldsBrowser();
-
-    cy.get(`[data-test-subj="header-text-${toggleField}"]`).should('exist');
+    cy.get(`[data-test-subj="header-text-${toggleField}"]`, { timeout: DEFAULT_TIMEOUT }).should(
+      'exist'
+    );
  });

  it('resets all fields in the timeline when `Reset Fields` is clicked', () => {
--- a/x-pack/legacy/plugins/siem/cypress/integration/smoke_tests/timeline/toggle_column.spec.ts
+++ b/x-pack/legacy/plugins/siem/cypress/integration/smoke_tests/timeline/toggle_column.spec.ts
@ -9,7 +9,7 @@ import { populateTimeline } from '../../lib/fields_browser/helpers';
 import { logout } from '../../lib/logout';
 import { toggleFirstEventDetails } from '../../lib/timeline/helpers';
 import { HOSTS_PAGE } from '../../lib/urls';
-import { loginAndWaitForPage } from '../../lib/util/helpers';
+import { loginAndWaitForPage, DEFAULT_TIMEOUT } from '../../lib/util/helpers';

 describe('toggle column in timeline', () => {
  beforeEach(() => {
@ -72,6 +72,8 @@ describe('toggle column in timeline', () => {

    cy.get(`[data-test-subj="headers-group"]`).then(headersDropArea => drop(headersDropArea));

-    cy.get(`[data-test-subj="header-text-${idField}"]`).should('exist');
+    cy.get(`[data-test-subj="header-text-${idField}"]`, { timeout: DEFAULT_TIMEOUT }).should(
+      'exist'
+    );
  });
 });
--- a/x-pack/legacy/plugins/siem/package.json
+++ b/x-pack/legacy/plugins/siem/package.json
@ -6,20 +6,37 @@
  "license": "Elastic-License",
  "scripts": {
    "build-graphql-types": "node scripts/generate_types_from_graphql.js",
-    "cypress:open": "cypress open"
+    "cypress:open": "cypress open",
+    "cypress:run": "cypress run --spec ./cypress/integration/**/*.spec.ts --reporter mocha-multi-reporters --reporter-options configFile=./reporter_config.json; mochawesome-merge --reportDir ../../../../target/kibana-siem/cypress/results > ../../../../target/kibana-siem/cypress/results/output.json; marge ../../../../target/kibana-siem/cypress/results/output.json --reportDir ../../../../target/kibana-siem/cypress/results; mkdir -p ../../../../target/junit && cp ../../../../target/kibana-siem/cypress/results/*.xml ../../../../target/junit/"
  },
  "devDependencies": {
    "@cypress/webpack-preprocessor": "^4.1.0",
    "@types/js-yaml": "^3.12.1",
    "@types/lodash": "^4.14.110",
    "@types/react-beautiful-dnd": "^10.0.1",
-    "cypress": "^3.3.1",
+    "cypress": "^3.4.1",
    "js-yaml": "^3.13.1",
+    "mocha-junit-reporter": "^1.23.1",
+    "mocha-multi-reporters": "^1.1.7",
+    "mochawesome": "^4.0.1",
+    "mochawesome-merge": "^2.0.1",
+    "mochawesome-report-generator": "^4.0.1",
    "ts-loader": "^6.0.4"
  },
  "dependencies": {
    "lodash": "^4.17.13",
    "react-beautiful-dnd": "^10.0.1",
    "react-markdown": "^4.0.6"
+  },
+  "workspaces": {
+    "packages": [
+      "packages/*"
+    ],
+    "nohoist": [
+      "**/mochawesome",
+      "**/mochawesome/**",
+      "**/mocha-multi-reporters",
+      "**/mocha-multi-reporters/**"
+    ]
  }
-}
+}
--- a/x-pack/legacy/plugins/siem/reporter_config.json
+++ b/x-pack/legacy/plugins/siem/reporter_config.json
@ -0,0 +1,10 @@
+{
+  "reporterEnabled": "mochawesome, mocha-junit-reporter",
+  "reporterOptions": {
+    "html": false,
+    "json": true,
+    "mochaFile": "../../../../target/kibana-siem/cypress/results/results-[hash].xml",
+    "overwrite": false,
+    "reportDir": "../../../../target/kibana-siem/cypress/results"
+  }
+}
--- a/x-pack/package.json
+++ b/x-pack/package.json
@ -119,6 +119,7 @@
    "cheerio": "0.22.0",
    "commander": "3.0.0",
    "copy-webpack-plugin": "^5.0.0",
+    "cypress": "^3.4.1",
    "del": "^4.0.0",
    "dotenv": "2.0.0",
    "enzyme": "^3.10.0",
@ -140,8 +141,14 @@
    "jest-cli": "^24.8.0",
    "jest-styled-components": "^6.2.2",
    "jsdom": "^12.0.0",
+    "js-yaml": "^3.13.1",
    "madge": "3.4.4",
    "mocha": "3.5.3",
+    "mocha-junit-reporter": "^1.23.1",
+    "mocha-multi-reporters": "^1.1.7",
+    "mochawesome": "^4.0.1",
+    "mochawesome-merge": "^2.0.1",
+    "mochawesome-report-generator": "^4.0.1",
    "mustache": "^2.3.0",
    "mutation-observer": "^1.0.3",
    "node-fetch": "^2.1.2",
@ -165,6 +172,7 @@
    "supertest-as-promised": "^4.0.2",
    "tmp": "0.1.0",
    "tree-kill": "^1.1.0",
+    "ts-loader": "^6.0.4",
    "typescript": "3.5.3",
    "vinyl-fs": "^3.0.2",
    "xml-crypto": "^0.10.1",
@ -360,5 +368,13 @@
  },
  "engines": {
    "yarn": "^1.10.1"
+  },
+  "workspaces": {
+    "nohoist": [
+      "**/mochawesome",
+      "**/mochawesome/**",
+      "**/mocha-multi-reporters",
+      "**/mocha-multi-reporters/**"
+    ]
  }
-}
+}
--- a/yarn.lock
+++ b/yarn.lock
@ -7900,6 +7900,11 @@ chardet@^0.7.0:
  resolved "https://registry.yarnpkg.com/chardet/-/chardet-0.7.0.tgz#90094849f0937f2eedc2425d0d28a9e5f0cbad9e"
  integrity sha512-mT8iDcrh03qDGRRmoA2hmBJnxpllMR+0/0qlzjqZES6NdiWDcZkCNAk4rPFZ9Q85r27unkiNNg8ZOiwZXBHwcA==

+charenc@~0.0.1:
+  version "0.0.2"
+  resolved "https://registry.yarnpkg.com/charenc/-/charenc-0.0.2.tgz#c0a1d2f3a7092e03774bfa83f14c0fc5790a8667"
+  integrity sha1-wKHS86cJLgN3S/qD8UwPxXkKhmc=
+
 check-disk-space@^2.1.0:
  version "2.1.0"
  resolved "https://registry.yarnpkg.com/check-disk-space/-/check-disk-space-2.1.0.tgz#2e77fe62f30d9676dc37a524ea2008f40c780295"
@ -9248,6 +9253,11 @@ cross-spawn@^5.0.1, cross-spawn@^5.1.0:
    shebang-command "^1.2.0"
    which "^1.2.9"

+crypt@~0.0.1:
+  version "0.0.2"
+  resolved "https://registry.yarnpkg.com/crypt/-/crypt-0.0.2.tgz#88d7ff7ec0dfb86f713dc87bbb42d044d3e6c41b"
+  integrity sha1-iNf/fsDfuG9xPch7u0LQRNPmxBs=
+
 cryptiles@2.x.x:
  version "2.0.5"
  resolved "https://registry.yarnpkg.com/cryptiles/-/cryptiles-2.0.5.tgz#3bdfecdc608147c1c67202fa291e7dca59eaa3b8"
@ -9509,10 +9519,10 @@ cyclist@~0.2.2:
  resolved "https://registry.yarnpkg.com/cyclist/-/cyclist-0.2.2.tgz#1b33792e11e914a2fd6d6ed6447464444e5fa640"
  integrity sha1-GzN5LhHpFKL9bW7WRHRkRE5fpkA=

-cypress@^3.3.1:
-  version "3.3.1"
-  resolved "https://registry.yarnpkg.com/cypress/-/cypress-3.3.1.tgz#8a127b1d9fa74bff21f111705abfef58d595fdef"
-  integrity sha512-JIo47ZD9P3jAw7oaK7YKUoODzszJbNw41JmBrlMMiupHOlhmXvZz75htuo7mfRFPC9/1MDQktO4lX/V2+a6lGQ==
+cypress@^3.4.1:
+  version "3.4.1"
+  resolved "https://registry.yarnpkg.com/cypress/-/cypress-3.4.1.tgz#ca2e4e9864679da686c6a6189603efd409664c30"
+  integrity sha512-1HBS7t9XXzkt6QHbwfirWYty8vzxNMawGj1yI+Fu6C3/VZJ8UtUngMW6layqwYZzLTZV8tiDpdCNBypn78V4Dg==
  dependencies:
    "@cypress/listr-verbose-renderer" "0.4.1"
    "@cypress/xvfb" "1.2.4"
@ -9527,20 +9537,19 @@ cypress@^3.3.1:
    execa "0.10.0"
    executable "4.1.1"
    extract-zip "1.6.7"
-    fs-extra "4.0.1"
+    fs-extra "5.0.0"
    getos "3.1.1"
-    glob "7.1.3"
    is-ci "1.2.1"
    is-installed-globally "0.1.0"
    lazy-ass "1.6.0"
    listr "0.12.0"
-    lodash "4.17.11"
+    lodash "4.17.15"
    log-symbols "2.2.0"
    minimist "1.2.0"
    moment "2.24.0"
    ramda "0.24.1"
    request "2.88.0"
-    request-progress "0.4.0"
+    request-progress "3.0.0"
    supports-color "5.5.0"
    tmp "0.1.0"
    url "0.11.0"
@ -9862,6 +9871,11 @@ dateformat@^2.0.0:
  resolved "https://registry.yarnpkg.com/dateformat/-/dateformat-2.2.0.tgz#4065e2013cf9fb916ddfd82efb506ad4c6769062"
  integrity sha1-QGXiATz5+5Ft39gu+1Bq1MZ2kGI=

+dateformat@^3.0.2:
+  version "3.0.3"
+  resolved "https://registry.yarnpkg.com/dateformat/-/dateformat-3.0.3.tgz#a6e37499a4d9a9cf85ef5872044d62901c9889ae"
+  integrity sha512-jyCETtSl3VMZMWeRo7iY1FL19ges1t55hMo5yaam4Jrsm5EPL89UQkoQRyiI+Yf4k8r2ZpdngkV8hr1lIdjb3Q==
+
 debug-fabulous@1.X:
  version "1.1.0"
  resolved "https://registry.yarnpkg.com/debug-fabulous/-/debug-fabulous-1.1.0.tgz#af8a08632465224ef4174a9f06308c3c2a1ebc8e"
@ -10439,6 +10453,11 @@ diff@^3.5.0:
  resolved "https://registry.yarnpkg.com/diff/-/diff-3.5.0.tgz#800c0dd1e0a8bfbc95835c202ad220fe317e5a12"
  integrity sha512-A46qtFgd+g7pDZinpnwiRJtxbC1hpgf0uzP3iG89scHk0AUC7A1TGxf5OiiOUv/JMZR8GOt8hL900hV0bOy5xA==

+diff@^4.0.1:
+  version "4.0.1"
+  resolved "https://registry.yarnpkg.com/diff/-/diff-4.0.1.tgz#0c667cb467ebbb5cea7f14f135cc2dba7780a8ff"
+  integrity sha512-s2+XdvhPCOF01LRQBC8hf4vhbVmI2CGS5aZnxLJlT5FtdhPCDFq80q++zK2KlrVorVDdL5BOGZ/VfLrVtYNF+Q==
+
 diffie-hellman@^5.0.0:
  version "5.0.2"
  resolved "https://registry.yarnpkg.com/diffie-hellman/-/diffie-hellman-5.0.2.tgz#b5835739270cfe26acf632099fded2a07f209e5e"
@ -13047,13 +13066,13 @@ fs-exists-sync@^0.1.0:
  resolved "https://registry.yarnpkg.com/fs-exists-sync/-/fs-exists-sync-0.1.0.tgz#982d6893af918e72d08dec9e8673ff2b5a8d6add"
  integrity sha1-mC1ok6+RjnLQjeyehnP/K1qNat0=

-fs-extra@4.0.1:
-  version "4.0.1"
-  resolved "https://registry.yarnpkg.com/fs-extra/-/fs-extra-4.0.1.tgz#7fc0c6c8957f983f57f306a24e5b9ddd8d0dd880"
-  integrity sha1-f8DGyJV/mD9X8waiTlud3Y0N2IA=
+fs-extra@5.0.0:
+  version "5.0.0"
+  resolved "https://registry.yarnpkg.com/fs-extra/-/fs-extra-5.0.0.tgz#414d0110cdd06705734d055652c5411260c31abd"
+  integrity sha512-66Pm4RYbjzdyeuqudYqhFiNBbCIuI9kgRqLPSHIlXHidW8NIQtVdkM1yeZ4lXwuhbTETv3EUGMNHAAw6hiundQ==
  dependencies:
    graceful-fs "^4.1.2"
-    jsonfile "^3.0.0"
+    jsonfile "^4.0.0"
    universalify "^0.1.0"

 fs-extra@^0.30.0:
@ -13162,6 +13181,11 @@ fstream@^1.0.12:
    mkdirp ">=0.5 0"
    rimraf "2"

+fsu@^1.0.2:
+  version "1.1.1"
+  resolved "https://registry.yarnpkg.com/fsu/-/fsu-1.1.1.tgz#bd36d3579907c59d85b257a75b836aa9e0c31834"
+  integrity sha512-xQVsnjJ/5pQtcKh+KjUoZGzVWn4uNkchxTF6Lwjr4Gf7nQr8fmUfhKJ62zE77+xQg9xnxi5KUps7XGs+VC986A==
+
 fullname@^3.2.0:
  version "3.3.0"
  resolved "https://registry.yarnpkg.com/fullname/-/fullname-3.3.0.tgz#a08747d6921229610b8178b7614fce10cb185f5a"
@ -15857,7 +15881,7 @@ is-boolean-object@^1.0.0:
  resolved "https://registry.yarnpkg.com/is-boolean-object/-/is-boolean-object-1.0.0.tgz#98f8b28030684219a95f375cfbd88ce3405dff93"
  integrity sha1-mPiygDBoQhmpXzdc+9iM40Bd/5M=

-is-buffer@^1.0.2, is-buffer@^1.1.4, is-buffer@^1.1.5:
+is-buffer@^1.0.2, is-buffer@^1.1.4, is-buffer@^1.1.5, is-buffer@~1.1.1:
  version "1.1.6"
  resolved "https://registry.yarnpkg.com/is-buffer/-/is-buffer-1.1.6.tgz#efaa2ea9daa0d7ab2ea13a97b2b8ad51fefbe8be"
  integrity sha512-NcdALwpXkTm5Zvvbk7owOUSvVvBKDgKP5/ewfXEznmQFfs4ZRmanOeKBTjRVjka3QFoN6XJ+9F3USqfHqTaU5w==
@ -18403,6 +18427,11 @@ lodash.isequal@^4.0.0, lodash.isequal@^4.1.1, lodash.isequal@^4.2.0, lodash.iseq
  resolved "https://registry.yarnpkg.com/lodash.isequal/-/lodash.isequal-4.5.0.tgz#415c4478f2bcc30120c22ce10ed3226f7d3e18e0"
  integrity sha1-QVxEePK8wwEgwizhDtMib30+GOA=

+lodash.isfunction@^3.0.8, lodash.isfunction@^3.0.9:
+  version "3.0.9"
+  resolved "https://registry.yarnpkg.com/lodash.isfunction/-/lodash.isfunction-3.0.9.tgz#06de25df4db327ac931981d1bdb067e5af68d051"
+  integrity sha512-AirXNj15uRIMMPihnkInB4i3NHeb4iBtNg9WRWuK2o31S+ePwwNmDPaTL3o7dTJ+VXNZim7rFs4rxN4YU1oUJw==
+
 lodash.isinteger@^4.0.4:
  version "4.0.4"
  resolved "https://registry.yarnpkg.com/lodash.isinteger/-/lodash.isinteger-4.0.4.tgz#619c0af3d03f8b04c31f5882840b77b11cd68343"
@ -18413,6 +18442,11 @@ lodash.isnumber@^3.0.3:
  resolved "https://registry.yarnpkg.com/lodash.isnumber/-/lodash.isnumber-3.0.3.tgz#3ce76810c5928d03352301ac287317f11c0b1ffc"
  integrity sha1-POdoEMWSjQM1IwGsKHMX8RwLH/w=

+lodash.isobject@^3.0.2:
+  version "3.0.2"
+  resolved "https://registry.yarnpkg.com/lodash.isobject/-/lodash.isobject-3.0.2.tgz#3c8fb8d5b5bf4bf90ae06e14f2a530a4ed935e1d"
+  integrity sha1-PI+41bW/S/kK4G4U8qUwpO2TXh0=
+
 lodash.isplainobject@^4.0.6:
  version "4.0.6"
  resolved "https://registry.yarnpkg.com/lodash.isplainobject/-/lodash.isplainobject-4.0.6.tgz#7c526a52d89b45c45cc690b88163be0497f550cb"
@ -18635,7 +18669,7 @@ lodash.uniqby@^4.7.0:
  resolved "https://registry.yarnpkg.com/lodash.uniqby/-/lodash.uniqby-4.7.0.tgz#d99c07a669e9e6d24e1362dfe266c67616af1302"
  integrity sha1-2ZwHpmnp5tJOE2Lf4mbGdhavEwI=

-lodash@4.17.11, lodash@4.17.13, lodash@>4.17.4, lodash@^4.0.0, lodash@^4.0.1, lodash@^4.11.1, lodash@^4.13.1, lodash@^4.14.0, lodash@^4.15.0, lodash@^4.17.0, lodash@^4.17.10, lodash@^4.17.11, lodash@^4.17.13, lodash@^4.17.2, lodash@^4.17.4, lodash@^4.17.5, lodash@^4.2.0, lodash@^4.2.1, lodash@^4.3.0, lodash@^4.5.0, lodash@^4.6.1, lodash@~4.17.10, lodash@~4.17.5:
+lodash@4.17.13, lodash@4.17.15, lodash@>4.17.4, lodash@^4.0.0, lodash@^4.0.1, lodash@^4.11.1, lodash@^4.13.1, lodash@^4.14.0, lodash@^4.15.0, lodash@^4.17.0, lodash@^4.17.10, lodash@^4.17.11, lodash@^4.17.13, lodash@^4.17.2, lodash@^4.17.4, lodash@^4.17.5, lodash@^4.2.0, lodash@^4.2.1, lodash@^4.3.0, lodash@^4.5.0, lodash@^4.6.1, lodash@~4.17.10, lodash@~4.17.5:
  version "4.17.13"
  resolved "https://registry.yarnpkg.com/lodash/-/lodash-4.17.13.tgz#0bdc3a6adc873d2f4e0c4bac285df91b64fc7b93"
  integrity sha512-vm3/XWXfWtRua0FkUyEHBZy8kCPjErNBT9fJx8Zvs+U6zjqPbTUOpkaoum3O5uiA8sm+yNMHXfYkTUHFoMxFNA==
@ -18645,6 +18679,11 @@ lodash@^3.10.1, lodash@^3.3.1, lodash@~3.10.1:
  resolved "https://registry.yarnpkg.com/lodash/-/lodash-3.10.1.tgz#5bf45e8e49ba4189e17d482789dfd15bd140b7b6"
  integrity sha1-W/Rejkm6QYnhfUgnid/RW9FAt7Y=

+lodash@^4.16.4:
+  version "4.17.15"
+  resolved "https://registry.yarnpkg.com/lodash/-/lodash-4.17.15.tgz#b447f6670a0455bbfeedd11392eff330ea097548"
+  integrity sha512-8xOcRHvCjnocdS5cpwXQXVzmmh5e5+saE2QGoeQmbKmRS6J3VQppPOIt0MnmE+4xlZoumy0GPG0D0MVIQbNA1A==
+
 lodash@^4.17.12:
  version "4.17.14"
  resolved "https://registry.yarnpkg.com/lodash/-/lodash-4.17.14.tgz#9ce487ae66c96254fe20b599f21b6816028078ba"
@ -19072,6 +19111,15 @@ md5.js@^1.3.4:
    hash-base "^3.0.0"
    inherits "^2.0.1"

+md5@^2.1.0:
+  version "2.2.1"
+  resolved "https://registry.yarnpkg.com/md5/-/md5-2.2.1.tgz#53ab38d5fe3c8891ba465329ea23fac0540126f9"
+  integrity sha1-U6s41f48iJG6RlMp6iP6wFQBJvk=
+  dependencies:
+    charenc "~0.0.1"
+    crypt "~0.0.1"
+    is-buffer "~1.1.1"
+
 mdast-add-list-metadata@1.0.1:
  version "1.0.1"
  resolved "https://registry.yarnpkg.com/mdast-add-list-metadata/-/mdast-add-list-metadata-1.0.1.tgz#95e73640ce2fc1fa2dcb7ec443d09e2bfe7db4cf"
@ -19619,6 +19667,25 @@ mobx@^4.9.2:
  resolved "https://registry.yarnpkg.com/mobx/-/mobx-4.9.4.tgz#bb37a0e4e05f0b02be89ced9d23445cad73377ad"
  integrity sha512-RaEpydw7D1ebp1pdFHrEMZcLk4nALAZyHAroCPQpqLzuIXIxJpLmMIe5PUZwYHqvlcWL6DVqDYCANZpPOi9iXA==

+mocha-junit-reporter@^1.23.1:
+  version "1.23.1"
+  resolved "https://registry.yarnpkg.com/mocha-junit-reporter/-/mocha-junit-reporter-1.23.1.tgz#ba11519c0b967f404e4123dd69bc4ba022ab0f12"
+  integrity sha512-qeDvKlZyAH2YJE1vhryvjUQ06t2hcnwwu4k5Ddwn0GQINhgEYFhlGM0DwYCVUHq5cuo32qAW6HDsTHt7zz99Ng==
+  dependencies:
+    debug "^2.2.0"
+    md5 "^2.1.0"
+    mkdirp "~0.5.1"
+    strip-ansi "^4.0.0"
+    xml "^1.0.0"
+
+mocha-multi-reporters@^1.1.7:
+  version "1.1.7"
+  resolved "https://registry.yarnpkg.com/mocha-multi-reporters/-/mocha-multi-reporters-1.1.7.tgz#cc7f3f4d32f478520941d852abb64d9988587d82"
+  integrity sha1-zH8/TTL0eFIJQdhSq7ZNmYhYfYI=
+  dependencies:
+    debug "^3.1.0"
+    lodash "^4.16.4"
+
 mocha@3.5.3:
  version "3.5.3"
  resolved "https://registry.yarnpkg.com/mocha/-/mocha-3.5.3.tgz#1e0480fe36d2da5858d1eb6acc38418b26eaa20d"
@ -19653,6 +19720,51 @@ mocha@^2.0.1, mocha@^2.3.4:
    supports-color "1.2.0"
    to-iso-string "0.0.2"

+mochawesome-merge@^2.0.1:
+  version "2.0.1"
+  resolved "https://registry.yarnpkg.com/mochawesome-merge/-/mochawesome-merge-2.0.1.tgz#c690433acc78fd769effe4db1a107508351e2dc5"
+  integrity sha512-QRYok/9y9MJ4zlWGajC/OV6BxjUGyv1AYX3DBOPSbpzk09p2dFBWV1QYSN/dHu7bo/q44ZGmOBHO8ZnAyI+Yug==
+  dependencies:
+    fs-extra "^7.0.1"
+    minimatch "^3.0.4"
+    uuid "^3.3.2"
+    yargs "^12.0.5"
+
+mochawesome-report-generator@^4.0.0, mochawesome-report-generator@^4.0.1:
+  version "4.0.1"
+  resolved "https://registry.yarnpkg.com/mochawesome-report-generator/-/mochawesome-report-generator-4.0.1.tgz#0a010d1ecf379eb26ba05300feb59e2665076080"
+  integrity sha512-hQbmQt8/yCT68GjrQFat+Diqeuka3haNllexYfja1+y0hpwi3yCJwFpQCdWK9ezzcXL3Nu80f2I6SZeyspwsqg==
+  dependencies:
+    chalk "^2.4.2"
+    dateformat "^3.0.2"
+    fs-extra "^7.0.0"
+    fsu "^1.0.2"
+    lodash.isfunction "^3.0.8"
+    opener "^1.4.2"
+    prop-types "^15.7.2"
+    react "^16.8.5"
+    react-dom "^16.8.5"
+    tcomb "^3.2.17"
+    tcomb-validation "^3.3.0"
+    validator "^10.11.0"
+    yargs "^13.2.2"
+
+mochawesome@^4.0.1:
+  version "4.0.1"
+  resolved "https://registry.yarnpkg.com/mochawesome/-/mochawesome-4.0.1.tgz#351af69c8904468e75a71f8704ed0b5767795ccc"
+  integrity sha512-F/hVmiwWCvwBiW/UPhs4/lfgf8mBJBr89W/9fDu+hb+rQ9gFxWh9N/BU7RtEH+dMfBF4o8XIdYHrEcwxJhzqsw==
+  dependencies:
+    chalk "^2.4.1"
+    diff "^4.0.1"
+    json-stringify-safe "^5.0.1"
+    lodash.isempty "^4.4.0"
+    lodash.isfunction "^3.0.9"
+    lodash.isobject "^3.0.2"
+    lodash.isstring "^4.0.1"
+    mochawesome-report-generator "^4.0.0"
+    strip-ansi "^5.0.0"
+    uuid "^3.3.2"
+
 module-definition@^3.0.0, module-definition@^3.1.0:
  version "3.2.0"
  resolved "https://registry.yarnpkg.com/module-definition/-/module-definition-3.2.0.tgz#a1741d5ddf60d76c60d5b1f41ba8744ba08d3ef4"
@ -20046,11 +20158,6 @@ node-ensure@^0.0.0:
  resolved "https://registry.yarnpkg.com/node-ensure/-/node-ensure-0.0.0.tgz#ecae764150de99861ec5c810fd5d096b183932a7"
  integrity sha1-7K52QVDemYYexcgQ/V0Jaxg5Mqc=

-node-eta@^0.1.1:
-  version "0.1.1"
-  resolved "https://registry.yarnpkg.com/node-eta/-/node-eta-0.1.1.tgz#4066109b39371c761c72b7ebda9a9ea0a5de121f"
-  integrity sha1-QGYQmzk3HHYccrfr2pqeoKXeEh8=
-
 node-fetch@1.7.3, node-fetch@^1.0.1:
  version "1.7.3"
  resolved "https://registry.yarnpkg.com/node-fetch/-/node-fetch-1.7.3.tgz#980f6f72d85211a5347c6b2bc18c5b84c3eb47ef"
@ -20720,6 +20827,11 @@ onetime@^2.0.0:
  dependencies:
    mimic-fn "^1.0.0"

+opener@^1.4.2:
+  version "1.5.1"
+  resolved "https://registry.yarnpkg.com/opener/-/opener-1.5.1.tgz#6d2f0e77f1a0af0032aca716c2c1fbb8e7e8abed"
+  integrity sha512-goYSy5c2UXE4Ra1xixabeVh1guIX/ZV/YokJksb6q2lubWu6UbvPQ20p542/sFIll1nl8JnCyK9oBaOcCWXwvA==
+
 opentracing@^0.13.0:
  version "0.13.0"
  resolved "https://registry.yarnpkg.com/opentracing/-/opentracing-0.13.0.tgz#6a341442f09d7d866bc11ed03de1e3828e3d6aab"
@ -22910,7 +23022,7 @@ react-dom@^16.8.1:
    prop-types "^15.6.2"
    scheduler "^0.13.5"

-react-dom@^16.8.3:
+react-dom@^16.8.3, react-dom@^16.8.5:
  version "16.8.6"
  resolved "https://registry.yarnpkg.com/react-dom/-/react-dom-16.8.6.tgz#71d6303f631e8b0097f56165ef608f051ff6e10f"
  integrity sha512-1nL7PIq9LTL3fthPqwkvr2zY7phIPjYrT0jp4HjyEQrEROnw4dG41VVwi/wfoCneoleqrNX7iAD+pXebJZwrwA==
@ -23526,7 +23638,7 @@ react@^16.8.1:
    prop-types "^15.6.2"
    scheduler "^0.13.5"

-react@^16.8.3:
+react@^16.8.3, react@^16.8.5:
  version "16.8.6"
  resolved "https://registry.yarnpkg.com/react/-/react-16.8.6.tgz#ad6c3a9614fd3a4e9ef51117f54d888da01f2bbe"
  integrity sha512-pC0uMkhLaHm11ZSJULfOBqV4tIZkx87ZLvbbQYunNixAAvjnC+snJCg0XQXn9VIsttVsbZP/H/ewzgsd5fxKXw==
@ -24241,13 +24353,12 @@ replace-ext@1.0.0, replace-ext@^1.0.0:
  resolved "https://registry.yarnpkg.com/replace-ext/-/replace-ext-1.0.0.tgz#de63128373fcbf7c3ccfa4de5a480c45a67958eb"
  integrity sha1-3mMSg3P8v3w8z6TeWkgMRaZ5WOs=

-request-progress@0.4.0:
-  version "0.4.0"
-  resolved "https://registry.yarnpkg.com/request-progress/-/request-progress-0.4.0.tgz#c1954e39086aa85269c5660bcee0142a6a70d7e7"
-  integrity sha1-wZVOOQhqqFJpxWYLzuAUKmpw1+c=
+request-progress@3.0.0:
+  version "3.0.0"
+  resolved "https://registry.yarnpkg.com/request-progress/-/request-progress-3.0.0.tgz#4ca754081c7fec63f505e4faa825aa06cd669dbe"
+  integrity sha1-TKdUCBx/7GP1BeT6qCWqBs1mnb4=
  dependencies:
-    node-eta "^0.1.1"
-    throttleit "^0.0.2"
+    throttleit "^1.0.0"

 request-promise-core@1.1.1:
  version "1.1.1"
@ -26831,6 +26942,18 @@ tar@^4:
    safe-buffer "^5.1.2"
    yallist "^3.0.2"

+tcomb-validation@^3.3.0:
+  version "3.4.1"
+  resolved "https://registry.yarnpkg.com/tcomb-validation/-/tcomb-validation-3.4.1.tgz#a7696ec176ce56a081d9e019f8b732a5a8894b65"
+  integrity sha512-urVVMQOma4RXwiVCa2nM2eqrAomHROHvWPuj6UkDGz/eb5kcy0x6P0dVt6kzpUZtYMNoAqJLWmz1BPtxrtjtrA==
+  dependencies:
+    tcomb "^3.0.0"
+
+tcomb@^3.0.0, tcomb@^3.2.17:
+  version "3.2.29"
+  resolved "https://registry.yarnpkg.com/tcomb/-/tcomb-3.2.29.tgz#32404fe9456d90c2cf4798682d37439f1ccc386c"
+  integrity sha512-di2Hd1DB2Zfw6StGv861JoAF5h/uQVu/QJp2g8KVbtfKnoHdBQl5M32YWq6mnSYBQ1vFFrns5B1haWJL7rKaOQ==
+
 tcp-port-used@^1.0.1:
  version "1.0.1"
  resolved "https://registry.yarnpkg.com/tcp-port-used/-/tcp-port-used-1.0.1.tgz#46061078e2d38c73979a2c2c12b5a674e6689d70"
@ -26984,10 +27107,10 @@ throat@^4.0.0:
  resolved "https://registry.yarnpkg.com/throat/-/throat-4.1.0.tgz#89037cbc92c56ab18926e6ba4cbb200e15672a6a"
  integrity sha1-iQN8vJLFarGJJua6TLsgDhVnKmo=

-throttleit@^0.0.2:
-  version "0.0.2"
-  resolved "https://registry.yarnpkg.com/throttleit/-/throttleit-0.0.2.tgz#cfedf88e60c00dd9697b61fdd2a8343a9b680eaf"
-  integrity sha1-z+34jmDADdlpe2H90qg0OptoDq8=
+throttleit@^1.0.0:
+  version "1.0.0"
+  resolved "https://registry.yarnpkg.com/throttleit/-/throttleit-1.0.0.tgz#9e785836daf46743145a5984b6268d828528ac6c"
+  integrity sha1-nnhYNtr0Z0MUWlmEtiaNgoUorGw=

 through2-filter@^2.0.0:
  version "2.0.0"
@ -28653,6 +28776,11 @@ validate-npm-package-name@2.2.2:
  dependencies:
    builtins "0.0.7"

+validator@^10.11.0:
+  version "10.11.0"
+  resolved "https://registry.yarnpkg.com/validator/-/validator-10.11.0.tgz#003108ea6e9a9874d31ccc9e5006856ccd76b228"
+  integrity sha512-X/p3UZerAIsbBfN/IwahhYaBbY68EN/UQBWHtsbXGT5bfrH/p4NQzUCG1kF/rtKaNpnJ7jAu6NGTdSNtyNIXMw==
+
 validator@^8.0.0:
  version "8.2.0"
  resolved "https://registry.yarnpkg.com/validator/-/validator-8.2.0.tgz#3c1237290e37092355344fef78c231249dab77b9"
@ -29900,6 +30028,11 @@ xml2js@^0.4.19, xml2js@^0.4.5:
    sax ">=0.6.0"
    xmlbuilder "~9.0.1"

+xml@^1.0.0:
+  version "1.0.1"
+  resolved "https://registry.yarnpkg.com/xml/-/xml-1.0.1.tgz#78ba72020029c5bc87b8a81a3cfcd74b4a2fc1e5"
+  integrity sha1-eLpyAgApxbyHuKgaPPzXS0ovweU=
+
 xmlbuilder@8.2.2:
  version "8.2.2"
  resolved "https://registry.yarnpkg.com/xmlbuilder/-/xmlbuilder-8.2.2.tgz#69248673410b4ba42e1a6136551d2922335aa773"
@ -30108,7 +30241,7 @@ yargs@^12.0.2:
    y18n "^3.2.1 || ^4.0.0"
    yargs-parser "^10.1.0"

-yargs@^13.3.0:
+yargs@^13.2.2, yargs@^13.3.0:
  version "13.3.0"
  resolved "https://registry.yarnpkg.com/yargs/-/yargs-13.3.0.tgz#4c657a55e07e5f2cf947f8a366567c04a0dedc83"
  integrity sha512-2eehun/8ALW8TLoIl7MVaRUrg+yCnenu8B4kBlRxj3GJGDKU1Og7sMXPNm1BYyM1DOJmTZ4YeN/Nwxv+8XJsUA==