* Simplify our kibana mocks
* Simpler mock factory that returns an object instead of a thunk
* We can use mockReturnValue instead of mockImplementation to
accomplish the same
* Allows us to replace createStartServices mock
* Uses unknown instead of any for mocks
* Clean up our manual use of kibana mocks in tests
* Since our useKibana mock returns a consistent mock, we can modify its
return value instead of re-mocking the entire thing
* Removes unnecessary uses of clearing/resetting mocks
* If your mocks are configured at the beginning of each test this is
usually unnecessary.
* I left one case of clearAllMocks in all_cases/index.test since it
defined several mock functions that were persistent across tests, and
it was easier than moving their definitions to a beforeEach
* Removes some unnecessary overrides that seemed due to storage
previously not being mocked
* Rename some old occurrences of SIEM
* Cross-reference similar hooks via JSDoc
There's a good chance that the consumer might want the OTHER hook, so
let's make that discoverable.
* Adds jest tests for our useListsConfig hook
* adds mocks for the hooks upon which it depends
* Add a mock for our useListsConfig hook
Leverages this mock factory in our manual mock for this hook.
* Remove unneeded eslint exception
* Move kibana_react mocks into their own .mock file
We're trying to consolidate mocks to this pattern so that they're easier
to find and reuse.
* Remove intermediate mock factory
This was only being consumed by our general createStartServicesMock.
* Replace security_solution's alias for a core mock
This is just noise/tech debt, we should use the core mock directly when
we can.
* Remove unnecessary wrapper around core mocks
Instead let's just reference the core mocks themselves.
* Remove outdated references from upstream
* More accurate mock
Throw an error of the same type if an unexpected key is used.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* migrate ui settings to core
* add basic test on service
* add unit tests
* adapt buildNum schema
* use any for buildNum...
* move i18n keys to core prefix
* translate added validation messages
* using number for schema for buildNum
* move state:storeInSessionStorage setting to core
* remove overrides config validation
* remove defaultRoute from config schema
## Summary
On the `security_solution` project from different customers we have been getting reports of scaling issues and excessive NodeJS event blocking times. After in-depth tracing through some of the index and field capabilities calls we identified two of the "hot paths" running through `field_capabilities` to where it is using double looped arrays rather than hashes. By switching these two hot spots out for hashes we are now able to reduce the event loop block times by an order of magnitude.
Before this PR you can see event loop block times as high as:
```ts
field_cap: 575.131ms
```
And after this PR you will see event loop block times drop by an order of magnitude to:
```ts
field_cap: 31.783ms
```
when you're calling into indexes as large as `filebeat-*`. This number can be higher if you're concatenating several large indexes together trying to get capabilities from each one all at once. We already only call `getFieldCapabilities` with one index at a time to spread out event block times.
The fix is to use a hash within two key areas within these two files:
```ts
src/plugins/data/server/index_patterns/fetcher/lib/field_capabilities/field_capabilities.ts
src/plugins/data/server/index_patterns/fetcher/lib/field_capabilities/field_caps_response.ts
```
This effect happens during the query of `SourceQuery`/`IndexFields` within `security_solution` but you should be able to trigger it with any application code who calls into those code paths with large index sizes such as `filebeat-*` anywhere in Kibana.
An explanation of how to see the block times for before and after
---
Add, `console.time('field_cap');` and `console.timeEnd('field_cap');` to where the synchronize code is for testing the optimizations of before and after.
For example around lines 45 with the original code:
```ts
const esFieldCaps: FieldCapsResponse = await callFieldCapsApi(callCluster, indices);
console.time('field_cap'); // <--- start timer
const fieldsFromFieldCapsByName = keyBy(readFieldCapsResponse(esFieldCaps), 'name');
const allFieldsUnsorted = Object.keys(fieldsFromFieldCapsByName)
.filter((name) => !name.startsWith('_'))
.concat(metaFields)
.reduce(concatIfUniq, [] as string[])
.map<FieldDescriptor>((name) =>
defaults({}, fieldsFromFieldCapsByName[name], {
name,
type: 'string',
searchable: false,
aggregatable: false,
readFromDocValues: false,
})
)
.map(mergeOverrides);
const sorted = sortBy(allFieldsUnsorted, 'name');
console.timeEnd('field_cap'); // <--- outputs the end timer
return sorted;
```
And around lines 45 with this pull request:
```ts
const esFieldCaps: FieldCapsResponse = await callFieldCapsApi(callCluster, indices);
console.time('field_cap'); // <--- start timer
const fieldsFromFieldCapsByName = keyBy(readFieldCapsResponse(esFieldCaps), 'name');
const allFieldsUnsorted = Object.keys(fieldsFromFieldCapsByName)
.filter((name) => !name.startsWith('_'))
.concat(metaFields)
.reduce<{ names: string[]; hash: Record<string, string> }>(
(agg, value) => {
// This is intentionally using a "hash" and a "push" to be highly optimized with very large indexes
if (agg.hash[value] != null) {
return agg;
} else {
agg.hash[value] = value;
agg.names.push(value);
return agg;
}
},
{ names: [], hash: {} }
)
.names.map<FieldDescriptor>((name) =>
defaults({}, fieldsFromFieldCapsByName[name], {
name,
type: 'string',
searchable: false,
aggregatable: false,
readFromDocValues: false,
})
)
.map(mergeOverrides);
const sorted = sortBy(allFieldsUnsorted, 'name');
console.timeEnd('field_cap'); // <--- outputs the end timer
return sorted;
```
And then reload the security solutions application web page or generically anything that is going to call filebeat-* index or another large index or you could concatenate several indexes together as well to test out the performance. For security solutions we can just visit any page such as this one below which has a filebeat-* index:
```
http://localhost:5601/app/security/timelines/
```
Be sure to load it _twice_ for testing as NodeJS will sometimes report better numbers the second time as it does optimizations after the first time it encounters some code paths.
You should begin to see numbers similar to this in the before:
```ts
field_cap: 575.131ms
```
This indicates that it is blocking the event loop for around half a second before this fix. If an application adds additional indexes on-top of `filebeat`, or if it tries to execute other code after this (which we do in security solutions) then the block times will climb even higher.
However, after this fix, the m^n are changed to use hashing so this only climb by some constant * n where n is your fields and for filebeat-* it will should very low around:
```ts
field_cap: 31.783ms
```
### Checklist
Unit tests already present, so this shouldn't break anything 🤞 .
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
* add asciidoc support for generated plugin list
Try level offset "=+2" instead of "=+1" to stop the inlining of the includes.
remove +2 back to +1
* Remove asciidoc, switch to regex. Rearrange dev guide to avoid nesting limit.
* Add tests for regex
* add a description to not throw off the table. Remove the heading from the paragraph snippet.
* Fix more READMEs so table renders correctly
* Update plugin list
* Remove code-exploration file, moved to plugin-list
* fix typo
* Add link to developer examples
* Update plugin list
* fix typo
* Add support for decorating 429 errors in the saved objects client
* Update the docs
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* check for packages stuck installing and reinstall
* update mock endpoint package
* diferentiate between reinstall and reupdate type of install, remove isUpdate, add integration test
* create new EpmPackageInstallStatus type instead of using InstallStatus
* fix merge conflict
* change EpmPackageInstallStatus to a union type
* change time to install to 1 minute
* used saved object find
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* fixes 'sets and reads the url state for timeline by id' timeline ttest
* makes test more reliable
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* improve test stability
* Replace SearchRequest = any with Record<string, any>
* Remove SearchResponse = any from data plugin
* docs
* logs
* Revert "Replace SearchRequest = any with Record<string, any>"
This reverts commit 9914ab5a01.
* code review
* list control
* null check
* null null null
* Jest fix
* add security solution search strategy on server side
* get security solution search strategy in the public app for all host
* fix types
* fix Check core API changes
* thank you cypress test
* Remove any by the right type IESearchRequest
Co-authored-by: Lukas Olson <olson.lukas@gmail.com>
* add translation and filter error when we abort the query
* pr review
* fix translation
* review II
* fix merge issue
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Lukas Olson <olson.lukas@gmail.com>
* Add Index Management README and quick testing steps for data streams.
* Surface data stream health in Data Streams tab (table and detail panel).
- Extract out DataHealth component for use in both Data Streams and Indices tabs.
- Refactor detail panel to use data structure & algo to build component.
- Refactor detail panel to use i18n.translate instead of FormattedMessage.
* Render index template name and index lifecycle policy name in the detail panel.
* Render storage size and max timestamp information in table and detail panel.
- Add 'Include stats' switch.
- Add humanizeTimeStamp service, localized to data streams.
