* Add new runtime types for parsing on client/server.
* Add more runtime types.
* Remove dead code.
* Mark parameter as unused.
* Improve typing for failed journey request function.
* Add new API functions, improve typing in a few others.
* Modify API calls to work with new screenshot_ref data.
* Fix untested refactor error.
* Add required fields to runtime type.
* Update typing in failed steps component.
* Adapt client to work with old screenshots as well as new screenshot_ref.
* Refactor composite code to reusable hook.
* Implement screenshot blocks endpoint.
* Define runtime types for full-size screenshots.
* Delete dedicated screenshot and ref queries.
* Optimize screenshot endpoint by combining queries.
* Handle parsing error.
* Clean up screenshot/ref typings.
* Remove dead types. DRY a type out.
* Simplify types.
* Improve typing in step screenshot components.
* Prefer PNG to JPG for canvas composite op.
* Simplify and clean up some code.
* Remove reliance on `Ping` type, clean up types.
* Add a comment.
* Add a comment.
* Fix typing for `FailedStep` component.
* Standardize loading spinner sizes.
* Add comments to composite code.
* Remove unnecessary optional chaining.
* Reformat error string.
* Remove unneeded key from request return object.
* Add a comment to a return object explaining very large cache value.
* Make type annotation more accurate.
* Resolve some type and test errors.
* Clean up remaining type errors.
* Move type definitions to simplify imports.
* Simplify `PingTimestamp` interface.
* Refactor failing unit test to use RTL and actually test things.
* Add tests for new helper functions.
* Add a comment.
* Test `PingTimestamp` for screenshot ref data.
* Test `StepImageCaption` for ref data.
* Improve typing for step list column definitions.
* Harden a test.
* Extract code to avoid repeated declarations.
* Create centralized mock for `useCompositeImage`.
* Add test for ref to `StepScreenshotDisplay`.
* Add tests for `getJourneyDetails`.
* Extract search results wrapper to helper lib.
* Add tests for `getJourneyFailedSteps`.
* Add support for aggs to result helper wrapper.
* Write tests for `getJourneyScreenshot` and simplify type checking.
* Write tests for `getJourneyScreenshotBlocks`.
* Simplify prop types for `FailedStep`.
* Remove unused type.
* Fix regression in step navigating for new style screenshots.
* Implement PR feedback.
* Implement PR feedback.
* Implement PR feedback.
* Reduce limit of screenshot block queries from 10k to 1k.
* Remove redundant field selection from ES query.
* Implement PR feedback.
* Fix regression that caused "Last successful step" to not show an image.
* Delete unused props from `Ping` runtime type.
* More precise naming.
* Naming improvements. Add `useCallback` to prevent callback re-declaration.
* Prefer explicit props to `{...spread}` syntax.
* Remove redundant type checking.
* Delete obsolete unit tests.
* Fix a regression.
* Add effect to `useEffect`.
* fix up CCR centered sates in tabs content
* update snapshots list
* fix lint errors
* Change tab states for all pages in snapshot+restore
* Remove unnecessary variables
* Seems we dont need the class wrapper
* fix broken type
* Fix bug in ILM table when filtering it down
* center align search box
* fix linter errors
* fix prettier warnings
* revert content var refactor and just focus on ux
* add breakword class to paragraph so we avoid text overflowing
* fix prettier errors
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Removing feature flag changes
* Adding isExportable flag to rule type definition
* Adding isExportable flag to rule type definition
* Adding isExportable flag to rule type definition
* Filtering rule on export by rule type isExportable flag
* Fixing types
* Adding docs
* Fix condition when exportCount is 0
* Unit test for fix condition when exportCount is 0
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Remove role region on flash messages
- just `aria-live` is enough for screen readers to read it out, and `role` was causing "Flash messages" to get read out loud repeatedly between page navigation even when empty which was annoying and not good
* Further a11y attribute recommendations from @myasonik
This PR fixes an edge case where a race condition mught cause the total_results from a federated content source to come back null from the server. This PR tells the server to expect null in those edge cases to prevent browser errors
## Summary
This adds utilities and two strategies for merging using the [fields API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-fields.html) and the `_source` document during signal generation. This gives us the ability to support `constant_keyword`, field alias value support, some runtime fields support, and `copy_to` support. Previously we did not copy any of these values and only generated signals based on the `_source` record values. This changes the behavior to allow us to copy some of the mentioned values above.
The folder of `source_fields_merging` contains a `strategy` folder and a `utils` folder which contains both the strategies and the utilities for this implementation. The two strategies are `merge_all_fields_with_source` and `merge_missing_fields_with_source`. The defaulted choice for this PR is we use `merge_missing_fields_with_source` and not the `merge_all_fields_with_source`. The reasoning is that this is much lower risk and lower behavior changes to the signals detection engine.
The main driving force behind this PR is that ECS has introduced `constant_keyword` and that field has the possibility of only showing up in the fields section of a document and not `_source` when index authors do not push the `constant_keyword` into the `_source` section. The secondary driving forces behind this behavioral change is that some users have been expecting their runtime fields, `copy_to` fields, and field alias values of their indexes to be copied into the signals index.
Both strategies of `merge_missing_fields_with_source` and `merge_all_fields_with_source` are considered Best Effort meaning that both strategies will not always merge as expected when they encounter ambiguous use cases as outlined in the `README.md` text at the top of `source_fields_merging` in detail.
The default used strategy of `merge_missing_fields_with_source` which has the simplest behavior will work in most common use cases. This is simply if the `_source` document is missing a value that is present in the `fields`, and the `fields` value is a primitive concrete value such as a `string` or `number` or `boolean` and the `_source` document does not contain an existing object or ambiguous array, then the value will be merged into `_source` and a new reference is returned. If you call the strategy twice it should be idempotent meaning that the second call will detect a value is now present in `_source` and not re-merge a second time.
* 301 unit tests were added
* Extensive README.md docs are added
* e2e tests are updated to test scenarios and ambiguity and conflicts from previously to support this effort.
* Other e2e tests were updated
* One bug with EQL and fields was found with a workaround implemented. See https://github.com/elastic/elasticsearch/issues/74582
* SearchTypes adjusted to use recursive TypeScript types
* Changed deprecated for `@deprecated` in a few spots
* Removed some `ts-expect-error` in favor of `??` in a few areas
* Added a new handling of epoch strings and tests to `detection_engine/signals/utils.ts` since fields returns `epoch_millis` as a string instead of as a number.
* Uses lodash safer set to reduce changes of prototype pollution
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
### Risk Matrix
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Prototype pollution | Low | High | Used lodash safer set |
| Users which have existing rules that work, upgrade and now we do not generate signals due to bad merging of fields and _source | Mid | High | We use the safer strategy method, `merge_missing_fields_with_source `, that is lighter weight to start with. We might add a follow up PR which enables a key in Kibana to turn off merging of fields with source. We added extensive unit tests and e2e tests. However, unexpected unknowns and behaviors from runtime fields and fields API such as geo-points looking like nested fields or `epoch_milliseconds` being a string value or runtime fields allowing invalid values were uncovered and tests and utilities around that have been added which makes this PR risky |
| Found a bug with using fields and EQL which caused EQL rules to not run. | Low | High | Implemented workaround for tests to pass and created an Elastic ticket and communicated the bug to EQL developers. |
* Update Enterprise Search request handler to send back an error connecting header
- vs only distinguishing error connecting issues by 502 status
+ clarify comment where this.handleConnectionError is called - for the most part, auth issues should already be caught by 401s in logic above
* Update HttpLogic to set errorConnecting state based on header
+ update tests etc to match read-only-mode state
* [Tech debt] Gracefully handle invalid HTTP responses
I've noticed this error a few times after Kibana gets shut down (http.response is undefined) so figured I would catch it here
* Fix missing try/catch/flashAPIErrors on engines overview
- This is the only http call I found missing a try/catch across our codebase, so we should be set for all views correctly flashing an API error that receive a 5xx response from ent-search
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Set up new BetaNotification component
* Update shared page template to append new beta notification item to side nav
NOTE: I'm mutating the array because:
- returning a new instance leads to a lot of really annoying type errors
- the side nav's we're getting are entirely static with predictable items & and always come from us anyway
- this is eventually going to get removed, and I'm optimizing for easy-to-remove code
* Add beta notification to error connecting state
- to help users/SDH cases where users cannot connect at all
* Fix type error
- sideNav itself can be undefined but not `sideNav.items`
Default to either the installed version of an integration, or the latest
available version based on installation status when a version is not
included in the integration details URL.
Closes#93393
* move content from modal to flyout with message table below chart
* update file name from modal to flyout
* update messages endpoint for range to use with chart range
* add show in chart action for messages table
* add job messages title and make flyout smaller
* [Alerting][Docs] Support enablement documentation.
* additional docs
* fixed links
* Apply suggestions from code review
Co-authored-by: gchaps <33642766+gchaps@users.noreply.github.com>
* fixed common issues
* Apply suggestions from code review
Co-authored-by: gchaps <33642766+gchaps@users.noreply.github.com>
* fixed due to comments
* fixed TM health api page
* fixed TM health api page 2
* Apply suggestions from code review
Co-authored-by: ymao1 <ying.mao@elastic.co>
Co-authored-by: Mike Côté <mikecote@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Mike Côté <mikecote@users.noreply.github.com>
Co-authored-by: ymao1 <ying.mao@elastic.co>
* fixed due to the comments
* fixed due to the comments
* fixed experimental flag
* fixed due to the comments
* Apply suggestions from code review
Co-authored-by: ymao1 <ying.mao@elastic.co>
* Update docs/user/alerting/alerting-troubleshooting.asciidoc
Co-authored-by: ymao1 <ying.mao@elastic.co>
* fixed due to the comments
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: gchaps <33642766+gchaps@users.noreply.github.com>
Co-authored-by: ymao1 <ying.mao@elastic.co>
Co-authored-by: Mike Côté <mikecote@users.noreply.github.com>
* Precision is now a required param for search settings
* Made RelevanceTuningLogic aware of precision param
* New PrecisionSlider component
* Add PrecisionSlider to RelevanceTuning
* Fix types in test
* Fix imports for PrecisionSlider
* Slight panel and text adjustments.
* Comment out docs link
* Add commented out test for docs
* Can we just all agree not to talk about this commit
* Restore docs link
* Fix docs link again
* Clean-up step description logic
* Test for documentation link
* Moving the spacer to align titles.
* Missing test for updatePrecision
* Fix CSS for step description
* Remove containing EuiPanel
* Improve screen reader experience
Co-authored-by: Davey Holler <daveyholler@hey.com>
* Remove es sql strategy from behind Labs project, remove legacy essql code, remove last spot of legacy elasticsearch client from canvas
* clean up test
* fix es field test
* remove comment
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
