* adding alerts and connectors to import/export test
* removing beforeEach
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* expression_reveal_image skeleton.
* expression_functions added.
* expression_renderers added.
* Backup of daily work.
* Fixed errors.
* Added legacy support. Added button for legacy.
* Added storybook.
* Removed revealImage from canvas.
* setState while rendering error fixed.
* tsconfig.json added.
* jest.config.js added.
* Demo doc added.
* Types fixed.
* added limits.
* Removed not used imports.
* i18n namespaces fixed.
* Fixed test suite error.
* Some errors fixed.
* Fixed eslint error.
* Removed more unused translations.
* Moved UI and elements, related to expressionRevealImage from canvas.
* Fixed unused translations errors.
* Moved type of element to types.
* Fixed types and added service for representing elements, ui and supported renderers to canvas.
* Added expression registration to canvas.
* Fixed
* Fixed mutiple call of the function.
* Removed support of a legacy lib for revealImage chart.
* Removed legacy presentation_utils plugin import.
* Doc error fixed.
* Removed useless translations and tried to fix error.
* One more fix.
* Small imports fix.
* Fixed translations.
* Made fixes based on nits.
* Removed useless params.
* fix.
* Fixed errors, related to jest and __mocks__.
* Removed useless type definition.
* Replaced RendererHandlers with IInterpreterRendererHandlers.
* fixed supported_shareable.
* Moved elements back to canvas.
* Moved views to canvas, removed expression service and imported renderer to canvas.
* Fixed translations.
* Types fix.
* Moved libs to presentation utils.
* Fixed one mistake.
* removed dataurl lib.
* Fixed jest files.
* elasticLogo removed.
* Removed elastic_outline.
* removed httpurl.
* Removed missing_asset.
* removed url.
* replaced mostly all tests.
* Fixed types.
* Fixed types and removed function_wrapper.ts
* Fixed types of test helpers.
* Changed limits of presentationUtil plugin.
* Fixed imports.
* One more fix.
* Fixed huge size of bundle.
* Reduced allow limit for presentationUtil
* Updated limits for presentationUtil.
* Fixed public API.
* fixed type errors.
* Moved css to component.
* Fixed spaces at element.
* Changed order of requiredPlugins.
* Updated limits.
* Removed unused plugin.
* Added rule for allowing import from __stories__ directory.
* removed useless comment.
* Changed readme.md
* Fixed docs error.
* A possible of smoke test.
* onResize changed to useResizeObserver.
* Remove useless events and `useEffect` block.
* Changed from passing handlers to separate functions.
* `function` moved to `server`.
* Fixed eslint error.
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Should reduce flake in two of our Cypress tests.
* Removed skip on a test recently skipped
* Removes a wait() that doesn't seem to have been reducing flake added by a EUI team member
* Adds a `.click()` to give focus to a component in order to improve our chances of typing in the input box
* Adds some `.should('exists')` which will cause Cypress to ensure something exists and a better chance for click handlers to be added
* Adds a pipe as suggested by @yctercero in the flake test
### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
* Added initial work for date index processor CITs.
* Fixed the tests and added the remaining coverage.
* Fixed message for date rounding error and updated tests to use GMT since that timezone actually works with the API.
* Update Date Index Name processor test name.
Co-authored-by: Yulia Čech <6585477+yuliacech@users.noreply.github.com>
Co-authored-by: Yulia Čech <6585477+yuliacech@users.noreply.github.com>
## Summary
We had `unHandledPromise` rejections within some of our unit tests which still pass on CI but technically those tests are not running correctly and will not catch bugs.
We were seeing them showing up like so:
```ts
PASS x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/import_rules_route.test.ts (10.502 s)
(node:21059) UnhandledPromiseRejectionWarning: [object Object]
at emitUnhandledRejectionWarning (internal/process/promises.js:170:15)
at processPromiseRejections (internal/process/promises.js:247:11)
at processTicksAndRejections (internal/process/task_queues.js:96:32)
(node:21059) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 3)
(node:21059) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
at emitDeprecationWarning (internal/process/promises.js:180:11)
at processPromiseRejections (internal/process/promises.js:249:13)
at processTicksAndRejections (internal/process/task_queues.js:96:32)
PASS x-pack/plugins/security_solution/server/lib/detection_engine/signals/signal_rule_alert_type.test.ts
PASS x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/add_prepackaged_rules_route.test.ts
PASS x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/update_rules_route.test.ts
(node:21059) UnhandledPromiseRejectionWarning: Error: bulk failed
at emitUnhandledRejectionWarning (internal/process/promises.js:170:15)
at processPromiseRejections (internal/process/promises.js:247:11)
at processTicksAndRejections (internal/process/task_queues.js:96:32)
(node:21059) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 7)
````
You can narrow down `unHandledPromise` rejections and fix tests one by one by running the following command:
```ts
node --trace-warnings --unhandled-rejections=strict scripts/jest.js --runInBand x-pack/plugins/security_solution
```
You can manually test if I fixed them by running that command and ensuring all tests run without errors and that the process exits with a 0 for detections only by running:
```ts
node --trace-warnings --unhandled-rejections=strict scripts/jest.js --runInBand x-pack/plugins/security_solution/public/detections
```
and
```ts
node --trace-warnings --unhandled-rejections=strict scripts/jest.js --runInBand x-pack/plugins/security_solution/server/lib/detection_engine
```
### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or
* Refactor: extract agent status to endpoint host status to reusable utiltiy
* Show Fleet Agent status + isolation status
* Refactor EndpoinAgentStatus component to use `<AgentStatus>` common component
* Move actions service to `endpoint/services` directory
* Add pending actions to the search strategy for endpoint data
## Summary
This is a follow up considered critical addition to:
https://github.com/elastic/kibana/pull/102280
This adds a key of `xpack.securitySolution.alertMergeStrategy` to `kibana.yml` which allows users to change their merge strategy between their raw events and the signals/alerts that are generated. This also adds additional security keys to the docker container that were overlooked in the past from security solutions.
The values you can use and add to to `xpack.securitySolution.alertMergeStrategy` are:
* missingFields (The default)
* allFields
* noFields
## missingFields
The default merge strategy we are using starting with 7.14 which will merge any primitive data types from the [fields API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-fields.html#search-fields-param) into the resulting signal/alert. This will copy over fields such as `constant_keyword`, `copy_to`, `runtime fields`, `field aliases` which previously were not copied over as long as they are primitive data types such as `keyword`, `text`, `numeric` and are not found in your original `_source` document. This will not copy copy `geo points`, `nested objects`, and in some cases if your `_source` contains arrays or top level objects or conflicts/ambiguities it will not merge them. This will _not_ merge existing values between `_source` and `fields` for `runtime fields` as well. It only merges missing primitive data types.
## allFields
A very aggressive merge strategy which should be considered experimental. It will do everything `missingFields` does but in addition to that it will merge existing values between `_source` and `fields` which means if you change values or override values with `runtime fields` this strategy will attempt to merge those values. This will also merge in most instances your nested fields but it will not merge `geo` data types due to ambiguities. If you have multi-fields this will choose your default field and merge that into `_source`. This can change a lot your data between your original `_source` and `fields` when the data is copied into an alert/signal which is why it is considered an aggressive merge strategy.
Both these strategies attempts to unbox single array elements when it makes sense and assumes you only want values in an array when it sees them in `_source` or if it sees multiple elements within an array.
## noFields
The behavior before https://github.com/elastic/kibana/pull/102280 was introduced and is a do nothing strategy. This should only be used if you are seeing problems with alerts/signals being inserted due to conflicts and/or bugs for some reason with `missingFields`. We are not anticipating this, but if you are setting `noFields` please reach out to our [forums](https://discuss.elastic.co/c/security/83) and let us know we have a bug so we can fix it. If you are encountering undesired merge behaviors or have other strategies you want us to implement let us know on the forums as well.
The missing keys added for docker are:
* xpack.securitySolution.alertMergeStrategy
* xpack.securitySolution.alertResultListDefaultDateRange
* xpack.securitySolution.endpointResultListDefaultFirstPageIndex
* xpack.securitySolution.endpointResultListDefaultPageSize
* xpack.securitySolution.maxRuleImportExportSize
* xpack.securitySolution.maxRuleImportPayloadBytes
* xpack.securitySolution.maxTimelineImportExportSize
* xpack.securitySolution.maxTimelineImportPayloadBytes
* xpack.securitySolution.packagerTaskInterval
* xpack.securitySolution.validateArtifactDownloads
I intentionally skipped adding the other `kibana.yml` keys which are considered either experimental flags or are for internal developers and are not documented and not supported in production by us.
## Manual testing of the different strategies
First add this mapping and document in the dev tools for basic tests
```json
# Mapping with two constant_keywords and a runtime field
DELETE frank-test-delme-17
PUT frank-test-delme-17
{
"mappings": {
"dynamic": "strict",
"runtime": {
"host.name": {
"type": "keyword",
"script": {
"source": "emit('changed_hostname')"
}
}
},
"properties": {
"@timestamp": {
"type": "date"
},
"host": {
"properties": {
"name": {
"type": "keyword"
}
}
},
"data_stream": {
"properties": {
"dataset": {
"type": "constant_keyword",
"value": "datastream_dataset_name_1"
},
"module": {
"type": "constant_keyword",
"value": "datastream_module_name_1"
}
}
},
"event": {
"properties": {
"dataset": {
"type": "constant_keyword",
"value": "event_dataset_name_1"
},
"module": {
"type": "constant_keyword",
"value": "event_module_name_1"
}
}
}
}
}
}
# Document without an existing host.name
PUT frank-test-delme-17/_doc/1
{
"@timestamp": "2021-06-30T15:46:31.800Z"
}
# Document with an existing host.name
PUT frank-test-delme-17/_doc/2
{
"@timestamp": "2021-06-30T15:46:31.800Z",
"host": {
"name": "host_name"
}
}
# Query it to ensure the fields is returned with data that does not exist in _soruce
GET frank-test-delme-17/_search
{
"fields": [
{
"field": "*"
}
]
}
```
For all the different key combinations do the following:
Run a single detection rule against the index:
<img width="1139" alt="Screen Shot 2021-06-30 at 9 49 12 AM" src="https://user-images.githubusercontent.com/1151048/123997522-b8dc6600-d98d-11eb-9407-5480d5b2cc8a.png">
Ensure two signals are created:
<img width="1376" alt="Screen Shot 2021-06-30 at 10 26 03 AM" src="https://user-images.githubusercontent.com/1151048/123997739-f17c3f80-d98d-11eb-9eb9-90e9410f0cde.png">
If your `kibana.yml` or `kibana.dev.yml` you set this key (or omit it as it is the default):
```yml
xpack.securitySolution.alertMergeStrategy: 'missingFields'
```
When you click on each signal you should see that `event.module` and `event.dataset` were copied over as well as `data_stream.dataset` and `data_stream.module` since they're `constant_keyword`:
<img width="877" alt="Screen Shot 2021-06-30 at 10 20 44 AM" src="https://user-images.githubusercontent.com/1151048/123997961-31432700-d98e-11eb-96ee-06524f21e2d6.png">
However since this only merges missing fields, you should see that in the first record the `host.name` is the runtime field defined since `host.name` does not exist in `_source` and that in the second record it still shows up as `host_name` since we do not override merges right now:
First:
<img width="887" alt="Screen Shot 2021-06-30 at 10 03 31 AM" src="https://user-images.githubusercontent.com/1151048/123998398-b2022300-d98e-11eb-87be-aa5a153a91bc.png">
Second:
<img width="838" alt="Screen Shot 2021-06-30 at 10 03 44 AM" src="https://user-images.githubusercontent.com/1151048/123998413-b4fd1380-d98e-11eb-9821-d6189190918f.png">
When you set in your `kibana.yml` or `kibana.dev.yml` this key:
```yml
xpack.securitySolution.alertMergeStrategy: 'noFields'
```
Expect that your `event.module`, `event.dataset`, `data_stream.module`, `data_stream.dataset` are all non-existent since we do not copy anything over from `fields` at all and only use things within `_source`:
<img width="804" alt="Screen Shot 2021-06-30 at 9 58 25 AM" src="https://user-images.githubusercontent.com/1151048/123998694-f8578200-d98e-11eb-8d71-a0858d3ed3e7.png">
Expect that `host.name` is missing in the first record and has the default value in the second:
First:
<img width="797" alt="Screen Shot 2021-06-30 at 9 58 37 AM" src="https://user-images.githubusercontent.com/1151048/123998797-10c79c80-d98f-11eb-81b6-5174d8ef14f2.png">
Second:
<img width="806" alt="Screen Shot 2021-06-30 at 9 58 52 AM" src="https://user-images.githubusercontent.com/1151048/123998816-158c5080-d98f-11eb-87a0-0ac2f58793b3.png">
When you set in your `kibana.yml` or `kibana.dev.yml` this key:
```yml
xpack.securitySolution.alertMergeStrategy: 'allFields'
```
Expect that `event.module` and `event.dataset` were copied over as well as `data_stream.dataset` and `data_stream.module` since they're `constant_keyword`:
<img width="864" alt="Screen Shot 2021-06-30 at 10 03 15 AM" src="https://user-images.githubusercontent.com/1151048/123999000-48364900-d98f-11eb-9803-05349744ac10.png">
Expect that both the first and second records contain the runtime field since we merge both of them:
<img width="887" alt="Screen Shot 2021-06-30 at 10 03 31 AM" src="https://user-images.githubusercontent.com/1151048/123999078-58e6bf00-d98f-11eb-83bd-dda6b50fabcd.png">
### Checklist
Delete any items that are not applicable to this PR.
- [x] If a plugin configuration key changed, check if it needs to be allowlisted in the [cloud](https://github.com/elastic/cloud) and added to the [docker list](c29adfef29/src/dev/build/tasks/os_packages/docker_generator/resources/bin/kibana-docker)
* add not found page
* fix url state
* fix url state
* revert cypress test case
* add tests for new links
* rename detections to alerts
* move function to helper
* add cypress tests
* clean up routes
* clean up routes
* styling for not found page
* clean up rules routes
* rm unused i18n
* add cypress tests
* add cypress tests
* rm unused i18n
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
