## Summary
Before this PR you can see event loop block times of:
```ts
formatIndexFields: 7986.884ms
```
After this PR you will see event loop block times of:
```ts
formatIndexFields: 85.012ms
```
within the file:
```ts
x-pack/plugins/security_solution/server/lib/index_fields/elasticsearch_adapter.ts
```
For the GraphQL query of `SourceQuery`/`IndexFields`
This also fixes the issue of `unknown` being returned to the front end by removing code that is no longer functioning as it was intended. Ensure during testing of this PR that blank/default and non exist indexes within `securitySolution:defaultIndex` still work as expected.
Before, notice the `unknown` instead of the `filebeat-*`:
<img width="733" alt="Screen Shot 2020-08-20 at 4 55 52 PM" src="https://user-images.githubusercontent.com/1151048/90949129-f5047900-e402-11ea-9278-b4c7bf5cd16d.png">
After:
<img width="830" alt="Screen Shot 2020-08-20 at 4 56 03 PM" src="https://user-images.githubusercontent.com/1151048/90949133-02b9fe80-e403-11ea-8504-f5bbe043048a.png">
An explanation of how to see the block times for before and after
---
For perf testing you first add timed testing to the file:
```ts
x-pack/plugins/security_solution/server/lib/index_fields/elasticsearch_adapter.ts
```
Before this PR, around lines 42:
```ts
console.time('formatIndexFields'); // <--- start timer
const fields = formatIndexFields(
responsesIndexFields,
Object.keys(indexesAliasIndices) as IndexAlias[]
);
console.timeEnd('formatIndexFields'); // <--- outputs the end timer
return fields;
```
After this PR, around lines 42:
```ts
console.time('formatIndexFields'); // <--- start timer
const fields = await formatIndexFields(responsesIndexFields, indices);
console.timeEnd('formatIndexFields'); // <--- outputs the end timer
return fields;
```
And then reload the security solutions application web page here:
```
http://localhost:5601/app/security/timelines/default
```
Be sure to load it _twice_ for testing as NodeJS will sometimes report better numbers the second time as it does optimizations after the first time it encounters some code paths.
You will begin to see numbers similar to this before this PR:
```ts
formatIndexFields: 2553.279ms
```
This indicates that it is blocking the event loop for ~2.5 seconds befofe this fix. If you add additional indexes to your `securitySolution:defaultIndex` indexes that have additional fields then this amount will increase exponentially. For developers using our test servers I created two other indexes called delme-1 and delme-2 with additional mappings you can add like below
```ts
apm-*-transaction*, auditbeat-*, endgame-*, filebeat-*, logs-*, packetbeat-*, winlogbeat-*, delme-1, delme-2
```
<img width="980" alt="Screen Shot 2020-08-21 at 8 21 50 PM" src="https://user-images.githubusercontent.com/1151048/90949142-211ffa00-e403-11ea-8ab2-f66de977dce3.png">
Then you are going to see times approaching 8 seconds of blocking the event loop like so:
```ts
formatIndexFields: 7986.884ms
```
After this fix on the first pass unoptimized it will report
```ts
formatIndexFields: 373.082ms
```
Then after it optimizes the code paths on a second page load it will report
```ts
formatIndexFields: 84.304ms
```
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
Restore the resolverTest plugin. This will allow us to run the test plugin and try out Resolver using our mock data access layers. Eventually this could be expanded to support multiple different data access layers. It could even be expanded to allow us to control the data access layer via the browser. Another option: we could export the APIs from the server and use those in this test plugin.
We eventually expect other plugins to use Resolver. This test plugin could allow us to test Resolver via the FTR (separately of the Security Solution.)
This would also be useful for writing tests than use the FTR but which are essentially unit tests. For example: taking screenshots, using the mouse to zoom/pan.
Start using: `yarn start --plugin-path x-pack/test/plugin_functional/plugins/resolver_test/`
## Summary
Assert unreachable was created through advice given by both the Typescript community and through the techniques that TyepScript is trying to achieve type safety with switch statements.
This fixes recent bugs by:
* Re-adding the never type
* Reduces the two different types by putting the helper within the common section so there's not duplication
* Fixes on type that looks like it was a regular string rather than a one of the enum types
The reasoning for exhaustive checks within switch statements and techniques can be seen in numerous areas such as here:
https://stackoverflow.com/questions/39419170/how-do-i-check-that-a-switch-block-is-exhaustive-in-typescript
You can do it either way with TypeScript as long as you ensure you have a explicit return type and you do early return statements you can actually avoid having to call into the assertUnreachable.
If introduced and used correctly it is there to help out like this error it is telling us that this string type is not exhaustive:
<img width="921" alt="Screen Shot 2020-08-24 at 10 39 42 AM" src="https://user-images.githubusercontent.com/1151048/91075618-9b1ad380-e5fb-11ea-9200-1c355faf5dca.png">
You can notice that for this pull request I actually remove the assertion like so if someone accidentally removes one of the switch statements:
<img width="1014" alt="Screen Shot 2020-08-24 at 10 42 08 AM" src="https://user-images.githubusercontent.com/1151048/91075662-a968ef80-e5fb-11ea-8d74-a92eedd63892.png">
And since the function has an explicit return type it is not needed. You will see that TypeScript improved its never types behind the scenes where it actually will tell you that it will never reach the `assertUnreachable` and want to remove it as an auto-refactor. That is ok as long as we have explicit return types and what I did with one line of code here.
<img width="536" alt="Screen Shot 2020-08-24 at 11 21 05 AM" src="https://user-images.githubusercontent.com/1151048/91075861-efbe4e80-e5fb-11ea-9991-dda111a04f1d.png">
Without this fix, and having the never type become an unknown it introduces less safety where any code that is utilizing the assertUnknown without explicit return types will be prone to having run time errors being thrown when something new is added to their switch enum types.
* changing transaction type filter to radio group
* fixing unit test
* changing transaction type filter to radio group
* adding onclick to the badge component
* adding onclick to the badge component
* adding i18n to aria
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
resolves https://github.com/elastic/kibana/issues/68209
Since routing key figures fairly prominently throughout PagerDuty APIs,
and ours, it seems like it make sense to include it in the single validation
message we have for it, as well as using the term we use for it in the product:
"integration key".
See the referenced issue for more background.
* Fixed alerting_api_integration/security_and_spaces tests failing if actions proxy set on for parallel process running using commands 'scripts/functional_tests_server' and 'scripts/functional_test_runner'
* -
* Fixed get port from range for Slack and webhook simulators, removed some test warnings
* Added check for listening proxy server
* changed logger to debug removed not useful error
* -
* changed proxy to dynamic target in a single place
* test retry
* -
* -
* -
* -
* test with no cleanup
* -
* -
* -
* -
* Added environment variable ALERTING_PROXY_PORT
* fixed type checks
* fixed clean up proxy server port
* Rename Whitelist to AllowList in Actions and Alerting
* revert not related change
* Fixed due to comments and tests failing
* Fixed failing tests
* Fixed due to comments
The release manager is currently expecting a Docker asset image with
the format of `kibana-8.0.0-SNAPSHOT-docker-image.tar.gz`. If this
target is not found, it will re-export the image. Making this change to
produce the expected filename will remove that duplicated effort.
Additionally, the release manager plans to remove this fallback in the
future anyways.
Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* [Setup] Change error connecting status code to 502
- For clearer error handling
* Set up new HttpProvider/Logic Kea store & listeners
- This allows us to:
- connect() directly to HttpLogic in other Kea logic files that need to make http calls, instead of passing in http manually via args
- Set http interceptors & remove them interceptors on unmount within Kea
- Share state derived from http (e.g. errorConnecting, readOnlyMode) between both AS & WS (but allow each app to handle that state differently if needed)
+ Refactors necessary for these changes:
- Kea types - add events key, clarify that mount returns an unmount function, fix reducer state type
- ReactDOM unmount - remove resetContext({}), was preventing logic from unmounting properly
* Update AS & WS to show error connecting component at app level
* [WS] Remove errorConnecting logic & http arg from Overview
- Since main app is now handling errorConnecting
- http can now be connected directly from HttpLogic Kea store, so no need to pass it
+ minor cleanup in logic_overview.test.ts - remove unneeded unmount(), act(), switch to HttpLogic mock
* [AS] Add top-level ErrorConnecting component & remove error logic from EngineOverview
* [AS] Clean up/move EngineOverview child components into subfolder
- delete old ErrorState component
- move LoadingState, EmptyState, and EngineOverviewHeader into subfolders in engine_overview
* PR feedback: Update test assertions 404 copy
* Simplify our kibana mocks
* Simpler mock factory that returns an object instead of a thunk
* We can use mockReturnValue instead of mockImplementation to
accomplish the same
* Allows us to replace createStartServices mock
* Uses unknown instead of any for mocks
* Clean up our manual use of kibana mocks in tests
* Since our useKibana mock returns a consistent mock, we can modify its
return value instead of re-mocking the entire thing
* Removes unnecessary uses of clearing/resetting mocks
* If your mocks are configured at the beginning of each test this is
usually unnecessary.
* I left one case of clearAllMocks in all_cases/index.test since it
defined several mock functions that were persistent across tests, and
it was easier than moving their definitions to a beforeEach
* Removes some unnecessary overrides that seemed due to storage
previously not being mocked
* Rename some old occurrences of SIEM
* Cross-reference similar hooks via JSDoc
There's a good chance that the consumer might want the OTHER hook, so
let's make that discoverable.
* Adds jest tests for our useListsConfig hook
* adds mocks for the hooks upon which it depends
* Add a mock for our useListsConfig hook
Leverages this mock factory in our manual mock for this hook.
* Remove unneeded eslint exception
* Move kibana_react mocks into their own .mock file
We're trying to consolidate mocks to this pattern so that they're easier
to find and reuse.
* Remove intermediate mock factory
This was only being consumed by our general createStartServicesMock.
* Replace security_solution's alias for a core mock
This is just noise/tech debt, we should use the core mock directly when
we can.
* Remove unnecessary wrapper around core mocks
Instead let's just reference the core mocks themselves.
* Remove outdated references from upstream
* More accurate mock
Throw an error of the same type if an unexpected key is used.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* migrate ui settings to core
* add basic test on service
* add unit tests
* adapt buildNum schema
* use any for buildNum...
* move i18n keys to core prefix
* translate added validation messages
* using number for schema for buildNum
* move state:storeInSessionStorage setting to core
* remove overrides config validation
* remove defaultRoute from config schema
## Summary
On the `security_solution` project from different customers we have been getting reports of scaling issues and excessive NodeJS event blocking times. After in-depth tracing through some of the index and field capabilities calls we identified two of the "hot paths" running through `field_capabilities` to where it is using double looped arrays rather than hashes. By switching these two hot spots out for hashes we are now able to reduce the event loop block times by an order of magnitude.
Before this PR you can see event loop block times as high as:
```ts
field_cap: 575.131ms
```
And after this PR you will see event loop block times drop by an order of magnitude to:
```ts
field_cap: 31.783ms
```
when you're calling into indexes as large as `filebeat-*`. This number can be higher if you're concatenating several large indexes together trying to get capabilities from each one all at once. We already only call `getFieldCapabilities` with one index at a time to spread out event block times.
The fix is to use a hash within two key areas within these two files:
```ts
src/plugins/data/server/index_patterns/fetcher/lib/field_capabilities/field_capabilities.ts
src/plugins/data/server/index_patterns/fetcher/lib/field_capabilities/field_caps_response.ts
```
This effect happens during the query of `SourceQuery`/`IndexFields` within `security_solution` but you should be able to trigger it with any application code who calls into those code paths with large index sizes such as `filebeat-*` anywhere in Kibana.
An explanation of how to see the block times for before and after
---
Add, `console.time('field_cap');` and `console.timeEnd('field_cap');` to where the synchronize code is for testing the optimizations of before and after.
For example around lines 45 with the original code:
```ts
const esFieldCaps: FieldCapsResponse = await callFieldCapsApi(callCluster, indices);
console.time('field_cap'); // <--- start timer
const fieldsFromFieldCapsByName = keyBy(readFieldCapsResponse(esFieldCaps), 'name');
const allFieldsUnsorted = Object.keys(fieldsFromFieldCapsByName)
.filter((name) => !name.startsWith('_'))
.concat(metaFields)
.reduce(concatIfUniq, [] as string[])
.map<FieldDescriptor>((name) =>
defaults({}, fieldsFromFieldCapsByName[name], {
name,
type: 'string',
searchable: false,
aggregatable: false,
readFromDocValues: false,
})
)
.map(mergeOverrides);
const sorted = sortBy(allFieldsUnsorted, 'name');
console.timeEnd('field_cap'); // <--- outputs the end timer
return sorted;
```
And around lines 45 with this pull request:
```ts
const esFieldCaps: FieldCapsResponse = await callFieldCapsApi(callCluster, indices);
console.time('field_cap'); // <--- start timer
const fieldsFromFieldCapsByName = keyBy(readFieldCapsResponse(esFieldCaps), 'name');
const allFieldsUnsorted = Object.keys(fieldsFromFieldCapsByName)
.filter((name) => !name.startsWith('_'))
.concat(metaFields)
.reduce<{ names: string[]; hash: Record<string, string> }>(
(agg, value) => {
// This is intentionally using a "hash" and a "push" to be highly optimized with very large indexes
if (agg.hash[value] != null) {
return agg;
} else {
agg.hash[value] = value;
agg.names.push(value);
return agg;
}
},
{ names: [], hash: {} }
)
.names.map<FieldDescriptor>((name) =>
defaults({}, fieldsFromFieldCapsByName[name], {
name,
type: 'string',
searchable: false,
aggregatable: false,
readFromDocValues: false,
})
)
.map(mergeOverrides);
const sorted = sortBy(allFieldsUnsorted, 'name');
console.timeEnd('field_cap'); // <--- outputs the end timer
return sorted;
```
And then reload the security solutions application web page or generically anything that is going to call filebeat-* index or another large index or you could concatenate several indexes together as well to test out the performance. For security solutions we can just visit any page such as this one below which has a filebeat-* index:
```
http://localhost:5601/app/security/timelines/
```
Be sure to load it _twice_ for testing as NodeJS will sometimes report better numbers the second time as it does optimizations after the first time it encounters some code paths.
You should begin to see numbers similar to this in the before:
```ts
field_cap: 575.131ms
```
This indicates that it is blocking the event loop for around half a second before this fix. If an application adds additional indexes on-top of `filebeat`, or if it tries to execute other code after this (which we do in security solutions) then the block times will climb even higher.
However, after this fix, the m^n are changed to use hashing so this only climb by some constant * n where n is your fields and for filebeat-* it will should very low around:
```ts
field_cap: 31.783ms
```
### Checklist
Unit tests already present, so this shouldn't break anything 🤞 .
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
