## Summary
Fixes flake tests of:
https://github.com/elastic/kibana/issues/115918https://github.com/elastic/kibana/issues/103273https://github.com/elastic/kibana/issues/108640https://github.com/elastic/kibana/issues/109447https://github.com/elastic/kibana/issues/100630https://github.com/elastic/kibana/issues/94535https://github.com/elastic/kibana/issues/104260
Security solution has been using `bsearch` and has encountered flake in various forms. Different developers have been fixing the flake in a few odd ways (myself included) which aren't 100%. This PR introduces a once-in-for-all REST API retry service called `bsearch` which will query `bsearch` and if `bsearch` is not completed because of async occurring due to slower CI runtimes it will continuously call into the `bsearch` with the correct API to ensure it gets a complete response before returning.
## Usage
Anyone can use this service like so:
```ts
const bsearch = getService('bsearch');
const response = await bsearch.send<MyType>({
supertest,
options: {
defaultIndex: ['large_volume_dns_data'],
}
strategy: 'securitySolutionSearchStrategy',
});
```
If you're using a custom auth then you can set that beforehand like so:
```ts
const bsearch = getService('bsearch');
const supertestWithoutAuth = getService('supertestWithoutAuth');
const supertest supertestWithoutAuth.auth(username, password);
const response = await bsearch.send<MyType>({
supertest,
options: {
defaultIndex: ['large_volume_dns_data'],
}
strategy: 'securitySolutionSearchStrategy',
});
```
## Misconceptions in the tests leading to flake
* Can you just call the bsearch REST API and it will always return data first time? Not always true, as when CI slows down or data increases `bsearch` will give you back an async reference and then your test will blow up.
* Can we wrap the REST API in `retry` to fix the flake? Not always but mostly true, as when CI slows down or data increases `bsearch` could return the async version continuously which could then fail your test. It's also tedious to tell everyone in code reviews to wrap everything in `retry` instead of just fixing it with a service as well as inform new people why we are constantly wrapping these tests in `retry`.
* Can we manually parse the `bsearch` if it has `async` for each test? This is true but is error prone and I did this for one test and it's ugly and I had issues as I have to wrap 2 things in `retry` and test several conditions. Also it's harder for people to read the tests rather than just reading there is a service call. Also people in code reviews missed where I had bugs with it. Also lots of boiler plate.
* Can we just increase the timeout with `wait_for_completion_timeout` and the tests will pass for sure then? Not true today but maybe true later, as this hasn't been added as plumbing yet. See this [open ticket](https://github.com/elastic/kibana/issues/107241). Even if it is and we increase the timeout to a very large number bsearch might return with an `async` or you might want to test the `async` path. Either way, if/when we add the ability we can increase it within 1 spot which is this service for everyone rather than going to each individual test to add it. If/when it's added if people don't use the bsearch service we can remove it later if we find this is deterministic enough and no one wants to test bsearch features with their strategies down the road.
## Manual test of bsearch service
If you want to manually watch the bsearch operate as if the CI system is running slow or to cause an `async` manually you manually modify this setting here:
https://github.com/elastic/kibana/blob/master/src/plugins/data/server/search/strategies/ese_search/request_utils.ts#L61
To be of a lower number such as `1ms` and then you will see it enter the `async` code within `bsearch` consistently
## Reference PRs
We cannot set the wait_for_complete just yet
https://github.com/elastic/kibana/issues/107241 so we decided this was the best way to reduce flake for testing for now.
### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
* bump to a pre-8.0 version
* export KibanaClient from /lib sub-folder
* workaround the problem of the absence of estypes
* update es client usage in pacakges
* export estypes from another path
* import errors from root
* import errors from root 2
* update transport import
* update import path for /api/types
* update import path for /api/types
* import errors from top export
* use TransportResult instead if ApiResponse
* fix errors in client_config
* fix src/core/server/saved_objects/migrationsv2/actions/integration_tests/actions.test.ts
* use KibanaClient in mock. we dont export the original Client
* fix client mocks
* fix errors on SO
* fix remaining core errors
* update estype import path
* fix errors in data plugin
* fix data_views
* fix es_ui_shared
* fix errors in interactive_setup
* fix errors in ./test folder
* add @elastic/transport to the runtime deps
* fix errors in packages
* fix erros in src/core
* fix errors in test/
* fix an error in actions plugin
* woraround and fix errors in APM plugin
* fix errors in canvas
* fix errors in event_log
* fix errors in fleet
* fix errors in ILM
* fix errors in infra
* fix errors in ingest_pipeline
* fix errors in lens
* fix errors in license_management
* fix errors in licensing
* fix errors in logstash
* fix errors in ml
* fix errors in monitoring
* fix errors in observability
* fix errors in rule_registry
* fix errors in reporting
* fix errors in rule_registry
* fix errors in security
* fix errors in security_solution
* fix errors in snapshot_restore
* fix errors in transform
* fix errors in UA
* fix errors in uptime
* fix errors in x-pack/test
* fix eslint errors
* fix new errors
* use default HTTP Connection. Undici does not support agent config options keepAlive and maxSockets
* create does not accept require_alias option
* update deps
* use transport types exported from ES client package
* fix ErrorCause | string errors
* do not use enum
* fix errors in data plugin
* update x-pack code
* fix transport
* fix apm search request
* do not crash on reporting
* fix kbn-test build
* mute reporting error to start
* fix ftr build
* another attempt
* update import path
* address or mute new errors
* REMOVE me. pin transport version temporarily.
* remove deep imports from transport package
* fix jest crash
* fix product check tests
* remove unnecessary ts-expect-error
* fix a few failed unit tests
* bump to canary 24
* remove unnecessary ts-expect-error
* remove dependency on transport
* fix types in tests
* mute errors in xpack tests
* product check doesn;t spam in logs anymore
* filterPath --> filter_path
* ignoreUnavailable --> ignore_unavailable
* ignoreUnavailable --> ignore_unavailable
* trackScores --> track_scores
* trackTotalHits --> track_total_hits
* fix es-arcives
* fix data plugin crashes
* fix watcher test utils
* rollback unnecessary changes
* fix another problem in es-archiver
* fix scroll. for whatever reason scroll fails when request scroll_id in body
* add meta: true in kbn-securitysolution-es-utils
* bump client to canary 25
* fix errors in accordance with the es client spec
* update securityscolution-es-utils
* unify scroll api in reporting and fix tests
* fix unit tests in watcher
* refactor APM to abort request with AbortController API
* fix missing es client calls in tests
* fix missing meta in detection engine FTR tests
* fix another bunch of errors in js tests
* fix wrong coercion
* remove test-grep pattern
* fix apm unit test
* rename terminateAfter to terminate_after in infra plugin
* rename terminateAfter to terminate_after in uptime plugin
* rename terminateAfter to terminate_after in apm plugin
* fix security roles FTR tests
* fix reference
* fix post_privilidges test
* fix post_privilidges
* bump client to 26
* add meta for index_management test helpers
* remove ts-expect-error caused by bad type in reason
* bump client to 27
* REMOVE me. workaround until fixed in the es client
* fix incorrect type casting
* swtich from camelCase params
* use `HttpConnection` for FTR-related clients
* bump client to 29
* Revert "REMOVE me. workaround until fixed in the es client"
This reverts commit c038850c09.
* fix new util
* revert repository changes
* do not crash if cannot store event_loop data
* fix new estypes imports
* fix more types
* fix security test types and add ts-ignore for custom ES client
* fix more estypes imports
* yet more ts violations
* line by line fixing is hard
* adapt `evaluateAlert` from infra as it's also used from FTR tests
* use convertToKibanaClient in FTR test instead of meta:true in plugin code
* migrate from deprecated API in fleet
* fix intergration tests
* fix fleet tests
* fix another fleet test
* fix more tests
* let's call it a day
* Removes custom header check on 404 responses, includes es client ProductNotSupportedError in EsUnavailableError conditional (#116029)
* Removes custom header check on 404 responses, includes es client ProductNotSupportedError in EsUnavailableError conditional
* Updates proxy response integration test
* disable APM until compatible with client v8
* skip async_search FTR test
* use kbnClient in integration tests
* bump version to 29
* bump to 30
* have configureClient return a KibanaClient instead of Client, remove resolved violations.
* bump to 31
* bump to 31
* Revert "bump to 31"
This reverts commit 5ac713e640.
* trigger stop to unusubscribe
* update generated docs
* remove obsolete test
* put "as" back
* cleanup
* skip test
* remove new type errors in apm package
* remove ErrorCause casting
* update a comment
* bump version to 32
* remove unnecessary ts-expect-error in apm code
* update comments
* update to client v33
* remove outdated type definition
* bump to 34 without params mutation
* unskip the test that should not fail anymore
* remove unnecessary ts-expect-error comments
* update to v35. body can be string
* move `sort` to body and use body friendly syntax
* fix a failing test. maps register the same SO that has been already registered by home
Co-authored-by: pgayvallet <pierre.gayvallet@gmail.com>
Co-authored-by: Christiane (Tina) Heiligers <christiane.heiligers@elastic.co>
* remove kbn-legacy-logging package
* remove legacy service
* remove legacy appender
* remove LegacyObjectToConfigAdapter
* gix types
* remove @hapi/good / @hapi/good-squeeze / @hapi/podium
* remove `default` appender validation for `root` logger
* remove old config key from kibana-docker
* fix FTR config
* fix dev server
* remove reference from readme
* fix unit test
* clean CLI args and remove quiet option
* fix type
* fix status test config
* remove from test config
* fix snapshot
* use another regexp
* update generated doc
* fix createRootWithSettings
* fix some integration tests
* another IT fix
* yet another IT fix
* (will be reverted) add assertion for CI failure
* Revert "(will be reverted) add assertion for CI failure"
This reverts commit 78d5560f9e.
* switch back to json layout for test
* remove legacy logging config deprecations
* address some review comments
* update documentation
* update kibana.yml config examples
* add config example for `metrics.ops`
Co-authored-by: Tyler Smalley <tyler.smalley@elastic.co>
* remove unnecessary ts-ignore
* add context propagation to x-opaque-id header tests
* run tests on CI
* simplify logging. the action purpose follows from the context name
* extend tests with the assertion against execution_context from the Kibana logs
* split JSON log records only
* apply suggestions proposed by Spencer
* make owner attribute required
* Add owner properties in more places
* add test for owner attribute
* add error check too in the test
* Fix tests
* fix tests and update docs
* wip
* More test fixes
* Fix All The Errorz
* Adding more owner attributes
* Update x-pack/test/saved_object_api_integration/common/fixtures/saved_object_test_plugin/kibana.json
Co-authored-by: Larry Gregory <lgregorydev@gmail.com>
* Update x-pack/test/ui_capabilities/common/fixtures/plugins/foo_plugin/kibana.json
Co-authored-by: Larry Gregory <lgregorydev@gmail.com>
* commeeeooonnnn
* Update docs
* soooo many kibanajsons
* adjust plugin generator to add an owner
* Add owner to the plugin generator scripts
* update snapshot
* Fix snapshot
* review updates
Co-authored-by: Larry Gregory <lgregorydev@gmail.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* [kbnArchiver] convert archive names to root-relative paths
* ensure that newly multiline hooks are explicitly async
* missed a newly multiline hook
* fix exists check
* avoid extra lines by wrapping arrow body in {}
* one block more
* fix errant `name` variable
Co-authored-by: spalger <spalger@users.noreply.github.com>
* bump @elastic/elasticsearch to canary.7
* address errors in core
* address errors in data plugin
* address errors in Alerting team plugins
* remove outdated messages in Lens
* remove unnecessary comments in ML
* address errors in Observability plugin
* address errors in reporting plugin
* address errors in Rule registry plugin
* fix errors in Security plugins
* fix errors in ES-UI plugin
* remove unnecessary union.
* update core tests
* fix kbn-es-archiver
* update to canary 8
* bump to v9
* use new typings
* fix new errors in core
* fix errors in core typeings
* fix type errors in data plugin
* fix type errors in telemetray plugin
* fix data plugin tests
* fix search examples type error
* fix errors in discover plugin
* fix errors in index_pattern_management
* fix type errors in vis_type_*
* fix errors in typings/elasticsearch
* fix type errors in actions plugin
* fix type errors in alerting and apm plugins
* fix type errors in canvas and cases
* fix errors in event_log
* fix type errors in ILM and ingest_pipelines
* fix errors in lens plugin
* fix errors in lists plugin
* fix errors in logstash
* fix errors in metrics_entities
* fix errors in o11y
* fix errors in watcher
* fix errors in uptime
* fix errors in upgrade_assistant
* fix errors in task_manager
* fix errors in stack_alerts
* fix errors in security_solution
* fix errors in rule_registry
* fix errors in snapshot_restore
* fix remaining errors
* fix search intergration tests
* adjust assetion
* bump version to canary.10
* adapt code to new naming schema
* use mapping types provided by the client library
* Revert "adjust assetion"
This reverts commit 19b8fe0464.
* fix so intergration tests
* fix http integration tests
* bump version to canary 11
* fix login test
* fix http integration test
* fix apm test
* update docs
* fixing some ml types
* fix new errors in data plugin
* fix new errors in alerting plugin
* fix new errors in lists plugin
* fix new errors in reporting
* fix or mute errors in rule_registry plugin
* more ML type fixes
* bump to canary 12
* fix errors after merge conflict
* additional ML fixes
* bump to canary 13
* fix errors in apm plugin
* fix errors in fleet plugin
* fix errors in infra plugin
* fix errors in monitoring plugin
* fix errors in osquery plugin
* fix errors in security solution plugins
* fix errors in transform plugin
* Update type imports for ES
* fix errors in x-pack plugins
* fix errors in tests
* update docs
* fix errors in x-pack/test
* update error description
* fix errors after master merge
* update comment in infra plugin
* fix new errors on xpack tests/
Co-authored-by: James Gowdy <jgowdy@elastic.co>
Co-authored-by: Dario Gieselaar <dario.gieselaar@elastic.co>
* Use client from branch
* Get type checking working in core
* Fix types in other plugins
* Update client types + remove type errors from core
* migrate Task Manager Elasticsearch typing from legacy library to client library
* use SortOrder instead o string in alerts
* Update client types + fix core type issues
* fix maps ts errors
* Update Lens types
* Convert Search Profiler body from a string to an object to conform to SearchRequest type.
* Fix SOT types
* Fix/mute Security/Spaces plugins type errors.
* Fix bootstrap types
* Fix painless_lab
* corrected es typing in Event Log
* Use new types from client for inferred search responses
* Latest type defs
* Integrate latest type defs for APM/UX
* fix core errors
* fix telemetry errors
* fix canvas errors
* fix data_enhanced errors
* fix event_log errors
* mute lens errors
* fix or mute maps errors
* fix reporting errors
* fix security errors
* mute errors in task_manager
* fix errors in telemetry_collection_xpack
* fix errors in data plugins
* fix errors in alerts
* mute errors in index_management
* fix task_manager errors
* mute or fix lens errors
* fix upgrade_assistant errors
* fix or mute errors in index_lifecycle_management
* fix discover errors
* fix core tests
* ML changes
* fix core type errors
* mute error in kbn-es-archiver
* fix error in data plugin
* fix error in telemetry plugin
* fix error in discover
* fix discover errors
* fix errors in task_manager
* fix security errors
* fix wrong conflict resolution
* address errors with upstream code
* update deps to the last commit
* remove outdated comments
* fix core errors
* fix errors after update
* adding more expect errors to ML
* pull the lastest changes
* fix core errors
* fix errors in infra plugin
* fix errors in uptime plugin
* fix errors in ml
* fix errors in xpack telemetry
* fix or mute errors in transform
* fix errors in upgrade assistant
* fix or mute fleet errors
* start fixing apm errors
* fix errors in osquery
* fix telemetry tests
* core cleanup
* fix asMutableArray imports
* cleanup
* data_enhanced cleanup
* cleanup events_log
* cleaup
* fix error in kbn-es-archiver
* fix errors in kbn-es-archiver
* fix errors in kbn-es-archiver
* fix ES typings for Hit
* fix SO
* fix actions plugin
* fix fleet
* fix maps
* fix stack_alerts
* fix eslint problems
* fix event_log unit tests
* fix failures in data_enhanced tests
* fix test failure in kbn-es-archiver
* fix test failures in index_pattern_management
* fixing ML test
* remove outdated comment in kbn-es-archiver
* fix error type in ml
* fix eslint errors in osquery plugin
* fix runtime error in infra plugin
* revert changes to event_log cluser exist check
* fix eslint error in osquery
* fixing ML endpoint argument types
* fx types
* Update api-extractor docs
* attempt fix for ese test
* Fix lint error
* Fix types for ts refs
* Fix data_enhanced unit test
* fix lens types
* generate docs
* Fix a number of type issues in monitoring and ml
* fix triggers_actions_ui
* Fix ILM functional test
* Put search.d.ts typings back
* fix data plugin
* Update typings in typings/elasticsearch
* Update snapshots
* mute errors in task_manager
* mute fleet errors
* lens. remove unnecessary ts-expect-errors
* fix errors in stack_alerts
* mute errors in osquery
* fix errors in security_solution
* fix errors in lists
* fix errors in cases
* mute errors in search_examples
* use KibanaClient to enforce promise-based API
* fix errors in test/ folder
* update comment
* fix errors in x-pack/test folder
* fix errors in ml plugin
* fix optional fields in ml api_integartoon tests
* fix another casting problem in ml tests
* fix another ml test failure
* fix fleet problem after conflict resolution
* rollback changes in security_solution. trying to fix test
* Update type for discover rows
* uncomment runtime_mappings as its outdated
* address comments from Wylie
* remove eslint error due to any
* mute error due to incompatibility
* Apply suggestions from code review
Co-authored-by: John Schulz <github.com@jfsiii.org>
* fix type error in lens tests
* Update x-pack/plugins/upgrade_assistant/server/lib/reindexing/reindex_service.ts
Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com>
* Update x-pack/plugins/upgrade_assistant/server/lib/reindexing/reindex_service.test.ts
Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com>
* update deps
* fix errors in core types
* fix errors for the new elastic/elasticsearch version
* remove unused type
* remove unnecessary manual type cast and put optional chaining back
* ML: mute Datafeed is missing indices_options
* Apply suggestions from code review
Co-authored-by: Josh Dover <1813008+joshdover@users.noreply.github.com>
* use canary pacakge instead of git commit
Co-authored-by: Josh Dover <me@joshdover.com>
Co-authored-by: Josh Dover <1813008+joshdover@users.noreply.github.com>
Co-authored-by: Gidi Meir Morris <github@gidi.io>
Co-authored-by: Nathan Reese <reese.nathan@gmail.com>
Co-authored-by: Wylie Conlon <wylieconlon@gmail.com>
Co-authored-by: CJ Cenizal <cj@cenizal.com>
Co-authored-by: Aleh Zasypkin <aleh.zasypkin@gmail.com>
Co-authored-by: Dario Gieselaar <dario.gieselaar@elastic.co>
Co-authored-by: restrry <restrry@gmail.com>
Co-authored-by: James Gowdy <jgowdy@elastic.co>
Co-authored-by: John Schulz <github.com@jfsiii.org>
Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com>
* Revert "Revert "Migrations v2: don't auto-create indices + FTR/esArchiver support (#85778)""
This reverts commit f97958043f.
* Fix flaky saved objects management test #89953
* If a clone target exists, wait for yellow, not green, index status
* Fix test after master merge
* Fix types
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Migrations V2 on by default
* esArchiver delete migrations v2 indices
* Fix saved_objects_management api_integration tests
* Try to fix v2 migrations for pre-release builds
* esArchiver delete auto-created v2 migration indices like .kibana_8.0.0
* Try to fix v2 migrations for pre-release builds
* Use require_alias to prevent auto-created saved objects index
* Wrap SO routes until core logs all internal errors
* Fix api_integration tests requiring an empty kibana index
* Delete corrupt saved object from lens archives
* Update docs
* Fix ui_settings tests
* Fix core jest tests
* Fix type errors
* Fix accessibility tests
* Fix plugin functional tests
* Fix api_integration tests after merging in master
* Fix plugin functional tests #2
* EsArchiver: Don't reset ui settings after the .kibana index was deleted
* Fix functional management/visualize tests
* Fix oss security functional tests
* EsArchiver clean task manager indices to fix alerting api integration tests
* migrationsv2 correctly handle unknown saved object type mappings
* Revert "Try to fix v2 migrations for pre-release builds"
This reverts commit a1a1567501.
* Revert "Try to fix v2 migrations for pre-release builds"
This reverts commit a9a935558c.
* Re-enable v2 migrations in tests after merging in master
* Try to fix async dashboard functional test
* Restore UiSettings defaults after emptyKibanaIndex()
* Review feedback: rename test to match behaviour
* Enable v2 so migrations, disable in FTR tests
* Disable v2 migrations for ui_settings integration tests
* Disable v2 migrations for reporting without serucity api integration test
* remove unused "paths" from tsconfig
* remove unused test utils, import get-url from kbn/test
* dont pollute globals with jest when importing getUrl
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
