kibana/x-pack/test
Frank Hassanabad d8e9327db4
[SIEM][Detection Engine] Fixes skipped tests (#71347)
## Summary

* https://github.com/elastic/kibana/issues/69632
* Adds a retry loop in case of a network outage/issue which should increase the chances of success
* If there is still an issue after the 20th try, then it moves on and there is a high likelihood the tests will continue without issues.
* Adds console logging statements so we know if this flakiness happens again a bit more insight into why the network is behaving the way it is.
* Helps prevent the other tests from being skipped in the future due to bad networking issues. 

The errors that were coming back from the failed tests are in the `afterEach` and look to be network related or another test interfering:

```ts
1) detection engine api security and spaces enabled
01:59:54         find_statuses
01:59:54           "after each" hook for "should return a single rule status when a single rule is loaded from a find status with defaults added":
01:59:54       ResponseError: Response Error
01:59:54        at IncomingMessage.response.on (/dev/shm/workspace/kibana/node_modules/@elastic/elasticsearch/lib/Transport.js:287:25)
01:59:54        at endReadableNT (_stream_readable.js:1145:12)
01:59:54        at process._tickCallback (internal/process/next_tick.js:63:19)
01:59:54  
01:59:54               └- ✖ fail: "detection engine api security and spaces enabled find_statuses "after each" hook for "should return a single rule status when a single rule is loaded from a find status with defaults added""
01:59:54               │
01:59:54               └-> "after all" hook
01:59:54             └-> "after all" hook
01:59:54         │
01:59:54         │42 passing (2.0m)
01:59:54         │1 failing
```

So this should fix it to where the afterEach calls try up to 20 times before giving up and then on giving up they move on with the hope a different test doesn't fail.


### Checklist
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
2020-07-09 20:36:51 -06:00
..
accessibility Removing flaky axe rule (#69418) 2020-06-17 13:00:52 -05:00
alerting_api_integration [SIEM][CASE] IBM Resilient Connector (#66385) 2020-07-10 02:23:20 +03:00
api_integration New Enterprise Search Kibana plugin (#66922) 2020-07-09 13:10:31 -07:00
apm_api_integration [RUM Dashboard] New rum services api to replace usage of get services API (#70746) 2020-07-08 11:26:34 +02:00
case_api_integration Register privileges in Kibana Platform Security plugin and remove legacy getUser API. (#65472) 2020-06-05 08:11:58 +02:00
common [SECURITY SOLUTION][INGEST] Task/endpoint list tests (#69419) 2020-06-25 11:10:39 -04:00
detection_engine_api_integration [SIEM][Detection Engine] Fixes skipped tests (#71347) 2020-07-09 20:36:51 -06:00
encrypted_saved_objects_api_integration [Encrypted Saved Objects] Adds support for migrations in ESO (#69513) 2020-06-25 17:23:31 +01:00
endpoint_api_integration_no_ingest Skip failing endgame tests (#70548) 2020-07-01 23:01:21 -07:00
functional Remove IE support in functional tests (#71285) 2020-07-09 22:12:52 +02:00
functional_embedded Rename HttpServerInfo.host to hostname (#70792) 2020-07-07 16:54:39 +02:00
functional_enterprise_search New Enterprise Search Kibana plugin (#66922) 2020-07-09 13:10:31 -07:00
functional_with_es_ssl Fixed adding an extra space character on selecting alert variable in action text fields (#70028) 2020-07-02 18:08:14 -07:00
ingest_manager_api_integration [Ingest Manager] Fix limited packages incorrect response (#71292) 2020-07-09 14:02:55 -07:00
kerberos_api_integration Re-enable mistakenly skipped tests. (#69123) 2020-06-15 11:38:04 +02:00
licensing_plugin Type safe browser.executeAsync (#69018) 2020-06-15 15:32:08 +02:00
login_selector_api_integration Support deep links inside of RelayState for SAML IdP initiated login. (#69401) 2020-06-24 08:05:02 +02:00
observability_api_integration Register privileges in Kibana Platform Security plugin and remove legacy getUser API. (#65472) 2020-06-05 08:11:58 +02:00
oidc_api_integration Register privileges in Kibana Platform Security plugin and remove legacy getUser API. (#65472) 2020-06-05 08:11:58 +02:00
page_load_metrics Report page load asset size (#66224) 2020-06-05 16:42:55 +02:00
pki_api_integration Register privileges in Kibana Platform Security plugin and remove legacy getUser API. (#65472) 2020-06-05 08:11:58 +02:00
plugin_api_integration Bump TypeScript to v3.9 (#67666) 2020-06-11 10:04:09 +02:00
plugin_api_perf chore(NA): upgrade to lodash@4 (#69868) 2020-07-03 01:30:13 +01:00
plugin_functional [Audit Logging] Add AuditTrail service (#69278) 2020-07-07 21:16:39 +02:00
reporting_api_integration [Reporting] remove async execution from csv_from_savedobject (#71031) 2020-07-09 09:58:45 -07:00
saml_api_integration Rename HttpServerInfo.host to hostname (#70792) 2020-07-07 16:54:39 +02:00
saved_object_api_integration Register privileges in Kibana Platform Security plugin and remove legacy getUser API. (#65472) 2020-06-05 08:11:58 +02:00
saved_objects_field_count CI Reporter for saved objects field count (#70580) 2020-07-07 16:53:47 +02:00
security_solution_cypress updates consumer to siem (#71117) 2020-07-09 07:45:10 +02:00
security_solution_endpoint [SECURITY SOLUTION] Rename to hosts and administration (#70913) 2020-07-08 17:27:52 -04:00
spaces_api_integration Register privileges in Kibana Platform Security plugin and remove legacy getUser API. (#65472) 2020-06-05 08:11:58 +02:00
stack_functional_integration Remove IE support in functional tests (#71285) 2020-07-09 22:12:52 +02:00
token_api_integration Register privileges in Kibana Platform Security plugin and remove legacy getUser API. (#65472) 2020-06-05 08:11:58 +02:00
ui_capabilities New Enterprise Search Kibana plugin (#66922) 2020-07-09 13:10:31 -07:00
upgrade_assistant_integration Add new elasticsearch client (#69905) 2020-07-08 17:21:00 +02:00
visual_regression
mocha_decorations.d.ts
tsconfig.json