## Summary
This adds utilities and two strategies for merging using the [fields API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-fields.html) and the `_source` document during signal generation. This gives us the ability to support `constant_keyword`, field alias value support, some runtime fields support, and `copy_to` support. Previously we did not copy any of these values and only generated signals based on the `_source` record values. This changes the behavior to allow us to copy some of the mentioned values above.
The folder of `source_fields_merging` contains a `strategy` folder and a `utils` folder which contains both the strategies and the utilities for this implementation. The two strategies are `merge_all_fields_with_source` and `merge_missing_fields_with_source`. The defaulted choice for this PR is we use `merge_missing_fields_with_source` and not the `merge_all_fields_with_source`. The reasoning is that this is much lower risk and lower behavior changes to the signals detection engine.
The main driving force behind this PR is that ECS has introduced `constant_keyword` and that field has the possibility of only showing up in the fields section of a document and not `_source` when index authors do not push the `constant_keyword` into the `_source` section. The secondary driving forces behind this behavioral change is that some users have been expecting their runtime fields, `copy_to` fields, and field alias values of their indexes to be copied into the signals index.
Both strategies of `merge_missing_fields_with_source` and `merge_all_fields_with_source` are considered Best Effort meaning that both strategies will not always merge as expected when they encounter ambiguous use cases as outlined in the `README.md` text at the top of `source_fields_merging` in detail.
The default used strategy of `merge_missing_fields_with_source` which has the simplest behavior will work in most common use cases. This is simply if the `_source` document is missing a value that is present in the `fields`, and the `fields` value is a primitive concrete value such as a `string` or `number` or `boolean` and the `_source` document does not contain an existing object or ambiguous array, then the value will be merged into `_source` and a new reference is returned. If you call the strategy twice it should be idempotent meaning that the second call will detect a value is now present in `_source` and not re-merge a second time.
* 301 unit tests were added
* Extensive README.md docs are added
* e2e tests are updated to test scenarios and ambiguity and conflicts from previously to support this effort.
* Other e2e tests were updated
* One bug with EQL and fields was found with a workaround implemented. See https://github.com/elastic/elasticsearch/issues/74582
* SearchTypes adjusted to use recursive TypeScript types
* Changed deprecated for `@deprecated` in a few spots
* Removed some `ts-expect-error` in favor of `??` in a few areas
* Added a new handling of epoch strings and tests to `detection_engine/signals/utils.ts` since fields returns `epoch_millis` as a string instead of as a number.
* Uses lodash safer set to reduce changes of prototype pollution
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
### Risk Matrix
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Prototype pollution | Low | High | Used lodash safer set |
| Users which have existing rules that work, upgrade and now we do not generate signals due to bad merging of fields and _source | Mid | High | We use the safer strategy method, `merge_missing_fields_with_source `, that is lighter weight to start with. We might add a follow up PR which enables a key in Kibana to turn off merging of fields with source. We added extensive unit tests and e2e tests. However, unexpected unknowns and behaviors from runtime fields and fields API such as geo-points looking like nested fields or `epoch_milliseconds` being a string value or runtime fields allowing invalid values were uncovered and tests and utilities around that have been added which makes this PR risky |
| Found a bug with using fields and EQL which caused EQL rules to not run. | Low | High | Implemented workaround for tests to pass and created an Elastic ticket and communicated the bug to EQL developers. |
* Update Enterprise Search request handler to send back an error connecting header
- vs only distinguishing error connecting issues by 502 status
+ clarify comment where this.handleConnectionError is called - for the most part, auth issues should already be caught by 401s in logic above
* Update HttpLogic to set errorConnecting state based on header
+ update tests etc to match read-only-mode state
* [Tech debt] Gracefully handle invalid HTTP responses
I've noticed this error a few times after Kibana gets shut down (http.response is undefined) so figured I would catch it here
* Fix missing try/catch/flashAPIErrors on engines overview
- This is the only http call I found missing a try/catch across our codebase, so we should be set for all views correctly flashing an API error that receive a 5xx response from ent-search
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Set up new BetaNotification component
* Update shared page template to append new beta notification item to side nav
NOTE: I'm mutating the array because:
- returning a new instance leads to a lot of really annoying type errors
- the side nav's we're getting are entirely static with predictable items & and always come from us anyway
- this is eventually going to get removed, and I'm optimizing for easy-to-remove code
* Add beta notification to error connecting state
- to help users/SDH cases where users cannot connect at all
* Fix type error
- sideNav itself can be undefined but not `sideNav.items`
Default to either the installed version of an integration, or the latest
available version based on installation status when a version is not
included in the integration details URL.
Closes#93393
* move content from modal to flyout with message table below chart
* update file name from modal to flyout
* update messages endpoint for range to use with chart range
* add show in chart action for messages table
* add job messages title and make flyout smaller
* [Alerting][Docs] Support enablement documentation.
* additional docs
* fixed links
* Apply suggestions from code review
Co-authored-by: gchaps <33642766+gchaps@users.noreply.github.com>
* fixed common issues
* Apply suggestions from code review
Co-authored-by: gchaps <33642766+gchaps@users.noreply.github.com>
* fixed due to comments
* fixed TM health api page
* fixed TM health api page 2
* Apply suggestions from code review
Co-authored-by: ymao1 <ying.mao@elastic.co>
Co-authored-by: Mike Côté <mikecote@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Mike Côté <mikecote@users.noreply.github.com>
Co-authored-by: ymao1 <ying.mao@elastic.co>
* fixed due to the comments
* fixed due to the comments
* fixed experimental flag
* fixed due to the comments
* Apply suggestions from code review
Co-authored-by: ymao1 <ying.mao@elastic.co>
* Update docs/user/alerting/alerting-troubleshooting.asciidoc
Co-authored-by: ymao1 <ying.mao@elastic.co>
* fixed due to the comments
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: gchaps <33642766+gchaps@users.noreply.github.com>
Co-authored-by: ymao1 <ying.mao@elastic.co>
Co-authored-by: Mike Côté <mikecote@users.noreply.github.com>
* Precision is now a required param for search settings
* Made RelevanceTuningLogic aware of precision param
* New PrecisionSlider component
* Add PrecisionSlider to RelevanceTuning
* Fix types in test
* Fix imports for PrecisionSlider
* Slight panel and text adjustments.
* Comment out docs link
* Add commented out test for docs
* Can we just all agree not to talk about this commit
* Restore docs link
* Fix docs link again
* Clean-up step description logic
* Test for documentation link
* Moving the spacer to align titles.
* Missing test for updatePrecision
* Fix CSS for step description
* Remove containing EuiPanel
* Improve screen reader experience
Co-authored-by: Davey Holler <daveyholler@hey.com>
* Remove es sql strategy from behind Labs project, remove legacy essql code, remove last spot of legacy elasticsearch client from canvas
* clean up test
* fix es field test
* remove comment
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Closes#103061. We use polished.tint() in several places to have a subdued highlight for an element. With the polished upgrade to 3.x from several weeks ago came a bug fix for tint() that applied the tint fraction in a different way. The fix for us to invert those fractions (eg 0.1 becomes 0.9).
* Set up new BetaNotification component
* Update shared page template to append new beta notification item to side nav
NOTE: I'm mutating the array because:
- returning a new instance leads to a lot of really annoying type errors
- the side nav's we're getting are entirely static with predictable items & and always come from us anyway
- this is eventually going to get removed, and I'm optimizing for easy-to-remove code
* Add beta notification to error connecting state
- to help users/SDH cases where users cannot connect at all
* [TSVB] Importing a dashboard with only TSVB viz on another space, breaks the dashboard
Closes: #103059
* move index-pattern to constant
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* When type endpoint gets os type from os name instead of os family
* Allow users add event filters only for endpoint events
* Fixes error with wrong map function
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Triggers search even if the query hasn't changed
* Add await on async function call
* refactor, use forceSearch store flag instead of new actin type to force search
* fix error when refreshing trusted apps
* Fix ts-error by adding ts-ignore '{ type: "LoadingResourceState"; previousState: ImmutableObject<UninitialisedResourceState> | ImmutableObject<LoadingResourceState<EventFiltersListPageData, ServerApiError>> | ImmutableObject<...> | ImmutableObject<...>; }' is not assignable to type 'ImmutableObject<{ forceRefresh: boolean; }>'.
Object literal may only specify known properties, and 'type' does not exist in type 'ImmutableObject<{ forceRefresh: boolean; }>'
