When setting the environment variable KBN_PATH_CONF in Windows, paths
containing special characters will be expanded and interpreted, instead
of being treated as a string. This wraps `%KBN_PATH_CONF%` in quotes to
prevent this.
Testing:
1) set "KBN_PATH_CONF=C:\With space\special()%$characters\kibana\config"
2) bin\kibana*.bat should run
* Add partial result flag to the execution result
* Update expressions plugin run method to return observable
* Update data getter in the execution contract to return observable
* Update the expression loader to take into account the partial results flag
* expression_reveal_image skeleton.
* expression_functions added.
* expression_renderers added.
* Backup of daily work.
* Fixed errors.
* Added legacy support. Added button for legacy.
* Added storybook.
* Removed revealImage from canvas.
* setState while rendering error fixed.
* tsconfig.json added.
* jest.config.js added.
* Demo doc added.
* Types fixed.
* added limits.
* Removed not used imports.
* i18n namespaces fixed.
* Fixed test suite error.
* Some errors fixed.
* Fixed eslint error.
* Removed more unused translations.
* Moved UI and elements, related to expressionRevealImage from canvas.
* Fixed unused translations errors.
* Moved type of element to types.
* Fixed types and added service for representing elements, ui and supported renderers to canvas.
* Added expression registration to canvas.
* Fixed
* Fixed mutiple call of the function.
* Removed support of a legacy lib for revealImage chart.
* Removed legacy presentation_utils plugin import.
* Doc error fixed.
* Removed useless translations and tried to fix error.
* One more fix.
* Small imports fix.
* Fixed translations.
* Made fixes based on nits.
* Removed useless params.
* fix.
* Fixed errors, related to jest and __mocks__.
* Removed useless type definition.
* Replaced RendererHandlers with IInterpreterRendererHandlers.
* fixed supported_shareable.
* Moved elements back to canvas.
* Moved views to canvas, removed expression service and imported renderer to canvas.
* Fixed translations.
* Types fix.
* Moved libs to presentation utils.
* Fixed one mistake.
* removed dataurl lib.
* Fixed jest files.
* elasticLogo removed.
* Removed elastic_outline.
* removed httpurl.
* Removed missing_asset.
* removed url.
* replaced mostly all tests.
* Fixed types.
* Fixed types and removed function_wrapper.ts
* Fixed types of test helpers.
* Changed limits of presentationUtil plugin.
* Fixed imports.
* One more fix.
* Fixed huge size of bundle.
* Reduced allow limit for presentationUtil
* Updated limits for presentationUtil.
* Fixed public API.
* fixed type errors.
* Moved css to component.
* Fixed spaces at element.
* Changed order of requiredPlugins.
* Updated limits.
* Removed unused plugin.
* Added rule for allowing import from __stories__ directory.
* removed useless comment.
* Changed readme.md
* Fixed docs error.
* A possible of smoke test.
* onResize changed to useResizeObserver.
* Remove useless events and `useEffect` block.
* Changed from passing handlers to separate functions.
* `function` moved to `server`.
* Fixed eslint error.
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This is a follow up considered critical addition to:
https://github.com/elastic/kibana/pull/102280
This adds a key of `xpack.securitySolution.alertMergeStrategy` to `kibana.yml` which allows users to change their merge strategy between their raw events and the signals/alerts that are generated. This also adds additional security keys to the docker container that were overlooked in the past from security solutions.
The values you can use and add to to `xpack.securitySolution.alertMergeStrategy` are:
* missingFields (The default)
* allFields
* noFields
## missingFields
The default merge strategy we are using starting with 7.14 which will merge any primitive data types from the [fields API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-fields.html#search-fields-param) into the resulting signal/alert. This will copy over fields such as `constant_keyword`, `copy_to`, `runtime fields`, `field aliases` which previously were not copied over as long as they are primitive data types such as `keyword`, `text`, `numeric` and are not found in your original `_source` document. This will not copy copy `geo points`, `nested objects`, and in some cases if your `_source` contains arrays or top level objects or conflicts/ambiguities it will not merge them. This will _not_ merge existing values between `_source` and `fields` for `runtime fields` as well. It only merges missing primitive data types.
## allFields
A very aggressive merge strategy which should be considered experimental. It will do everything `missingFields` does but in addition to that it will merge existing values between `_source` and `fields` which means if you change values or override values with `runtime fields` this strategy will attempt to merge those values. This will also merge in most instances your nested fields but it will not merge `geo` data types due to ambiguities. If you have multi-fields this will choose your default field and merge that into `_source`. This can change a lot your data between your original `_source` and `fields` when the data is copied into an alert/signal which is why it is considered an aggressive merge strategy.
Both these strategies attempts to unbox single array elements when it makes sense and assumes you only want values in an array when it sees them in `_source` or if it sees multiple elements within an array.
## noFields
The behavior before https://github.com/elastic/kibana/pull/102280 was introduced and is a do nothing strategy. This should only be used if you are seeing problems with alerts/signals being inserted due to conflicts and/or bugs for some reason with `missingFields`. We are not anticipating this, but if you are setting `noFields` please reach out to our [forums](https://discuss.elastic.co/c/security/83) and let us know we have a bug so we can fix it. If you are encountering undesired merge behaviors or have other strategies you want us to implement let us know on the forums as well.
The missing keys added for docker are:
* xpack.securitySolution.alertMergeStrategy
* xpack.securitySolution.alertResultListDefaultDateRange
* xpack.securitySolution.endpointResultListDefaultFirstPageIndex
* xpack.securitySolution.endpointResultListDefaultPageSize
* xpack.securitySolution.maxRuleImportExportSize
* xpack.securitySolution.maxRuleImportPayloadBytes
* xpack.securitySolution.maxTimelineImportExportSize
* xpack.securitySolution.maxTimelineImportPayloadBytes
* xpack.securitySolution.packagerTaskInterval
* xpack.securitySolution.validateArtifactDownloads
I intentionally skipped adding the other `kibana.yml` keys which are considered either experimental flags or are for internal developers and are not documented and not supported in production by us.
## Manual testing of the different strategies
First add this mapping and document in the dev tools for basic tests
```json
# Mapping with two constant_keywords and a runtime field
DELETE frank-test-delme-17
PUT frank-test-delme-17
{
"mappings": {
"dynamic": "strict",
"runtime": {
"host.name": {
"type": "keyword",
"script": {
"source": "emit('changed_hostname')"
}
}
},
"properties": {
"@timestamp": {
"type": "date"
},
"host": {
"properties": {
"name": {
"type": "keyword"
}
}
},
"data_stream": {
"properties": {
"dataset": {
"type": "constant_keyword",
"value": "datastream_dataset_name_1"
},
"module": {
"type": "constant_keyword",
"value": "datastream_module_name_1"
}
}
},
"event": {
"properties": {
"dataset": {
"type": "constant_keyword",
"value": "event_dataset_name_1"
},
"module": {
"type": "constant_keyword",
"value": "event_module_name_1"
}
}
}
}
}
}
# Document without an existing host.name
PUT frank-test-delme-17/_doc/1
{
"@timestamp": "2021-06-30T15:46:31.800Z"
}
# Document with an existing host.name
PUT frank-test-delme-17/_doc/2
{
"@timestamp": "2021-06-30T15:46:31.800Z",
"host": {
"name": "host_name"
}
}
# Query it to ensure the fields is returned with data that does not exist in _soruce
GET frank-test-delme-17/_search
{
"fields": [
{
"field": "*"
}
]
}
```
For all the different key combinations do the following:
Run a single detection rule against the index:
<img width="1139" alt="Screen Shot 2021-06-30 at 9 49 12 AM" src="https://user-images.githubusercontent.com/1151048/123997522-b8dc6600-d98d-11eb-9407-5480d5b2cc8a.png">
Ensure two signals are created:
<img width="1376" alt="Screen Shot 2021-06-30 at 10 26 03 AM" src="https://user-images.githubusercontent.com/1151048/123997739-f17c3f80-d98d-11eb-9eb9-90e9410f0cde.png">
If your `kibana.yml` or `kibana.dev.yml` you set this key (or omit it as it is the default):
```yml
xpack.securitySolution.alertMergeStrategy: 'missingFields'
```
When you click on each signal you should see that `event.module` and `event.dataset` were copied over as well as `data_stream.dataset` and `data_stream.module` since they're `constant_keyword`:
<img width="877" alt="Screen Shot 2021-06-30 at 10 20 44 AM" src="https://user-images.githubusercontent.com/1151048/123997961-31432700-d98e-11eb-96ee-06524f21e2d6.png">
However since this only merges missing fields, you should see that in the first record the `host.name` is the runtime field defined since `host.name` does not exist in `_source` and that in the second record it still shows up as `host_name` since we do not override merges right now:
First:
<img width="887" alt="Screen Shot 2021-06-30 at 10 03 31 AM" src="https://user-images.githubusercontent.com/1151048/123998398-b2022300-d98e-11eb-87be-aa5a153a91bc.png">
Second:
<img width="838" alt="Screen Shot 2021-06-30 at 10 03 44 AM" src="https://user-images.githubusercontent.com/1151048/123998413-b4fd1380-d98e-11eb-9821-d6189190918f.png">
When you set in your `kibana.yml` or `kibana.dev.yml` this key:
```yml
xpack.securitySolution.alertMergeStrategy: 'noFields'
```
Expect that your `event.module`, `event.dataset`, `data_stream.module`, `data_stream.dataset` are all non-existent since we do not copy anything over from `fields` at all and only use things within `_source`:
<img width="804" alt="Screen Shot 2021-06-30 at 9 58 25 AM" src="https://user-images.githubusercontent.com/1151048/123998694-f8578200-d98e-11eb-8d71-a0858d3ed3e7.png">
Expect that `host.name` is missing in the first record and has the default value in the second:
First:
<img width="797" alt="Screen Shot 2021-06-30 at 9 58 37 AM" src="https://user-images.githubusercontent.com/1151048/123998797-10c79c80-d98f-11eb-81b6-5174d8ef14f2.png">
Second:
<img width="806" alt="Screen Shot 2021-06-30 at 9 58 52 AM" src="https://user-images.githubusercontent.com/1151048/123998816-158c5080-d98f-11eb-87a0-0ac2f58793b3.png">
When you set in your `kibana.yml` or `kibana.dev.yml` this key:
```yml
xpack.securitySolution.alertMergeStrategy: 'allFields'
```
Expect that `event.module` and `event.dataset` were copied over as well as `data_stream.dataset` and `data_stream.module` since they're `constant_keyword`:
<img width="864" alt="Screen Shot 2021-06-30 at 10 03 15 AM" src="https://user-images.githubusercontent.com/1151048/123999000-48364900-d98f-11eb-9803-05349744ac10.png">
Expect that both the first and second records contain the runtime field since we merge both of them:
<img width="887" alt="Screen Shot 2021-06-30 at 10 03 31 AM" src="https://user-images.githubusercontent.com/1151048/123999078-58e6bf00-d98f-11eb-83bd-dda6b50fabcd.png">
### Checklist
Delete any items that are not applicable to this PR.
- [x] If a plugin configuration key changed, check if it needs to be allowlisted in the [cloud](https://github.com/elastic/cloud) and added to the [docker list](c29adfef29/src/dev/build/tasks/os_packages/docker_generator/resources/bin/kibana-docker)
* [TSVB] Metric count is depicted as - instead of 0
* Rename extractData to mapEmptyToZero and remove unnecessary intervalString undefined assignment in get_bucket_size.js
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* [TSVB] Fix TSVB is not reporting all categories of Elasticsearch error
Closes: #94182
* move validation to server side
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* [labs] Update Labs Status
* Fix translations
* Supply IntersectionObserver mock
* Set defer fold project to not enabled by default
* Update copy for labs flyout
* [Solution Toolbar] Fixing button border on non-text color versions
* [Alerts] Removed extra wrappers and use EuiPageHeader
* [Logstash] Basic conversion to template
* [Reporting] Adding bottomBorder to page header
* [ML] Fix display of main navigation tabs
* [Stack Management] Fix side nav not updating when going back to landing page
* [Tags] Add spacing after page header
* [License Management] Full width on file uploader
* [Page Template] Fixed `emptyState` default template for pages with side nav
* [Infra] Removing some page header displays in empty states
* [Enterprise Search] Fix some error layouts
* [Index Patterns] Quick fix for empty state
* snaps
* [Page Template] Remove forced padding when `centeredBody`
* small hack for tab padding for ml
* scroll ML page to fix test
* fix test method type signature
Co-authored-by: Dave Snider <dave.snider@gmail.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Michail Yasonik <michail.yasonik@elastic.co>
This has been enabled by default, and there isn't any remaining reason to turn it off.
Mark it as deprecated and remove conditionals that check for it.
* Migrate kibana.autocomplete config to data plugin
* Fix CI
* Fix tests
* Use new terms enum API for autocomplete value suggestions
* Add tiers to config
* Re-introduce terms agg and add config/tests for swapping algorithms
* Add data_content and data_cold tiers by default
* Fix types
* Fix maps test
* Update tests
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Added dashboard no items message to dashboard_strings, wrote new message for read only empty state. Moved visualize library dashboard callout under title
Adds the ability to collapse the sidenav. This should work in all solutions. It also adds breakpoints that turn it into a flyout at lower screen widths.
* initial draft
* fix some tests
* fix additional unit tests
* move all the things
* create error generation fn
* add correct error message
* add unknown types to log message
* fix types
* fix existing test suites
* add IT test
* review comments
* add tests + use unknown instead of undefined for empty types
* update RFC with new step
* [Discover] replace legacy table with euiInMemoryTable
* [Discover] update styles, add badge
* fix font in badge and adjust line height
* add tooltip
* [Discover] update unit tests, return actions column to left side
* [Discover] update field name test snapshot
* [Discover] update wording
* [Discover] handle pagination, return formatting value styles
* [Discover] fix failing stylelint error
* [Discover] return responsive prop, update classes
* [Discover] update test and meet formatting rules
* improve table view on medium
* remove extra file
* [Discover] fix unit test
* [Discover] align top vertically field name and action cells, disable table responsive design
* [Discover] adjust styles for cross browser compatibility
* [Discover] remove pagination optimize styles, update test
* [Discover] fix eslint
* [Discover] clean up styles
* [Discover] fix single doc view
* [Discover] add check lack of multifieldBadge
Co-authored-by: Andrea Del Rio <delrio.andre@gmail.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* [SecuritySolutions] [Navigation] Prepare new routing and migrate overview (#101733)
* prepare new routing and migrate overview
* test fix and todo comments identified
* telemetry using app views
* navigation groups implemented
* cleaning
* export subplugin routes as route props array
* [Security Solution][Navigation] Migrate Security Solutions 'explore' tab group to deep link navigation (#102306)
* Update navigateToApp and getUrlForApp to provide the deepLinkId
* Update Hosts and Network routes to start from /hosts and /network
* Add Hosts and Network to side nav menu under "Explore" menu group
* Delete Hosts and Network old menu code
* Fix broken tests
* [SecuritySolution] Add detections subplugin to deeplink (#101791)
* prepare new routing and migrate overview
* init nav deeplink
* split detections into rules and alerts
* init exception link
* init detections
* link to rules creation page
* link to rules creation page
* rename detections to alerts
* fix unit tests
* fix rules creation page
* remove console
* fix lint error
* fix unit tests
* fix unit tests
* isolating rules and exceptions page
* replace history push with navigateToApp
* fix unit test
* temporary fix for createCoreStartMock
* update cypress
* skip failing cypress
* skip failing cypress
Co-authored-by: semd <sergi.massaneda@elastic.co>
* Migrate "Investigate" tab group to new side navigation (#102705)
* Migrate "Investigate" tab group to new side navigation
It includes:
* Timelines
* Cases
* Quick fix useFormatUrl and HeaderPage navigation
* [Security Solutions] Management navigation (#102685)
* prepare new routing and migrate overview
* test fix and todo comments identified
* telemetry using app views
* navigation groups implemented
* cleaning
* export subplugin routes as route props array
* breadcrumbs changes and sidenav generation improvements
* jest tests for breadcrumbs and navigation changes
* retrocompatibility for sections that are not yet migrated to deepLinks
* management deepLinks and plugin refactoring
* home navigation changes
* management navigation migrated to deeplinks
* jest tests fixed
* header page back link improved and tests fixed
* type errors fixes
* improve home navigation encapsulation
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Fix type checking
* export header page
* fix padding
* add redirect routes
* unskip detection cypress
* fix i18n
* fix create your own rules btn
* fix cancel button on rules creation page
* test fixes
* fix breadcrumbs for rules pages
* unit test fixes
* additional fixes
* [Security Solutions] Navigation usage tracker and general changes (#103271)
* [Security Solutions] use of currentAppId$ migrated. and some small fixes
* unused constants removed
* remove unused constant
* test fix and types
* fix cypress
* fix cypress tests
* Fix case navTab permission and tests
* Revert 'timeline.isOpen' breadcrumb code that was deleted during merge
* Fix useInsertTimeline test by removing '/'
* change global navigation visible deeplinks
* fix /admininstration top level redirect to
* fix global search icon, nav order and overview hosts link
* update start a new case link
* fix rules link in exception list table
* unskip cypress tests
* update rules link
* fix full screen timeline
* fixing broken links and administration telemetry split
* remove unused comments
* remove timeline z-index and cleanup global header component
* some minor fixes
* add unit tests for detections breadcrumbs
* remove case to global/search nav when cases is none
* rename test scenario
* fix side_panel flyout
* fix cases use cases between search/gobal nav
* timeline snapshot regenerated and cypres test fixed
* rollback management tracking split as it causes unexpected errors on the telemetry component
Co-authored-by: Pablo Machado <pablo.nevesmachado@elastic.co>
Co-authored-by: Angela Chuang <6295984+angorayc@users.noreply.github.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Michael Olorunnisola <michael.olorunnisola@elastic.co>
Co-authored-by: Angela Chuang <yi-chun.chuang@elastic.co>
Co-authored-by: Xavier Mouligneau <189600+XavierM@users.noreply.github.com>
* fix up CCR centered sates in tabs content
* update snapshots list
* fix lint errors
* Change tab states for all pages in snapshot+restore
* Remove unnecessary variables
* Seems we dont need the class wrapper
* fix broken type
* Fix bug in ILM table when filtering it down
* center align search box
* fix linter errors
* fix prettier warnings
* revert content var refactor and just focus on ux
* add breakword class to paragraph so we avoid text overflowing
* fix prettier errors
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Removing feature flag changes
* Adding isExportable flag to rule type definition
* Adding isExportable flag to rule type definition
* Adding isExportable flag to rule type definition
* Filtering rule on export by rule type isExportable flag
* Fixing types
* Adding docs
* Fix condition when exportCount is 0
* Unit test for fix condition when exportCount is 0
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Remove es sql strategy from behind Labs project, remove legacy essql code, remove last spot of legacy elasticsearch client from canvas
* clean up test
* fix es field test
* remove comment
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
