* [APM] Adds migration to fleet-managed APM server in APM UI Settings (#100657)
* adds useStrictParams option to apm server routes to allow unknown record type in param.bopy
* Adds checks for required roles, policies, and config before allowing user to initiate migration
* refactored and cleaned up server-side code
* i18n and link to Fleet
* fixes linting issues and unit tests
* updates the apm package policy to 0.3.0 and adds some missing config mappings
* PR feedback
* Handles case where the cloud policy doesn't exist
* Reverts the addition of the useStrictParams option since strictKeysRt now supports records
* fixes default input var values and uses correct published package version
* displays reasons the switch to data streams is disabled
* Store apm-server schema with the internal saved objects client
* Fix 'Detection' / 'Investigation Guide' UI broken when it contains long words
* Fix investigation guide is not formatted under Alert details flyout
* Add LineClamp to investigation guide field
* It enhances LineClamp to support a react node instead of only text
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* [Discover] replace legacy table with euiInMemoryTable
* [Discover] update styles, add badge
* fix font in badge and adjust line height
* add tooltip
* [Discover] update unit tests, return actions column to left side
* [Discover] update field name test snapshot
* [Discover] update wording
* [Discover] handle pagination, return formatting value styles
* [Discover] fix failing stylelint error
* [Discover] return responsive prop, update classes
* [Discover] update test and meet formatting rules
* improve table view on medium
* remove extra file
* [Discover] fix unit test
* [Discover] align top vertically field name and action cells, disable table responsive design
* [Discover] adjust styles for cross browser compatibility
* [Discover] remove pagination optimize styles, update test
* [Discover] fix eslint
* [Discover] clean up styles
* [Discover] fix single doc view
* [Discover] add check lack of multifieldBadge
Co-authored-by: Andrea Del Rio <delrio.andre@gmail.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* [SecuritySolutions] [Navigation] Prepare new routing and migrate overview (#101733)
* prepare new routing and migrate overview
* test fix and todo comments identified
* telemetry using app views
* navigation groups implemented
* cleaning
* export subplugin routes as route props array
* [Security Solution][Navigation] Migrate Security Solutions 'explore' tab group to deep link navigation (#102306)
* Update navigateToApp and getUrlForApp to provide the deepLinkId
* Update Hosts and Network routes to start from /hosts and /network
* Add Hosts and Network to side nav menu under "Explore" menu group
* Delete Hosts and Network old menu code
* Fix broken tests
* [SecuritySolution] Add detections subplugin to deeplink (#101791)
* prepare new routing and migrate overview
* init nav deeplink
* split detections into rules and alerts
* init exception link
* init detections
* link to rules creation page
* link to rules creation page
* rename detections to alerts
* fix unit tests
* fix rules creation page
* remove console
* fix lint error
* fix unit tests
* fix unit tests
* isolating rules and exceptions page
* replace history push with navigateToApp
* fix unit test
* temporary fix for createCoreStartMock
* update cypress
* skip failing cypress
* skip failing cypress
Co-authored-by: semd <sergi.massaneda@elastic.co>
* Migrate "Investigate" tab group to new side navigation (#102705)
* Migrate "Investigate" tab group to new side navigation
It includes:
* Timelines
* Cases
* Quick fix useFormatUrl and HeaderPage navigation
* [Security Solutions] Management navigation (#102685)
* prepare new routing and migrate overview
* test fix and todo comments identified
* telemetry using app views
* navigation groups implemented
* cleaning
* export subplugin routes as route props array
* breadcrumbs changes and sidenav generation improvements
* jest tests for breadcrumbs and navigation changes
* retrocompatibility for sections that are not yet migrated to deepLinks
* management deepLinks and plugin refactoring
* home navigation changes
* management navigation migrated to deeplinks
* jest tests fixed
* header page back link improved and tests fixed
* type errors fixes
* improve home navigation encapsulation
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Fix type checking
* export header page
* fix padding
* add redirect routes
* unskip detection cypress
* fix i18n
* fix create your own rules btn
* fix cancel button on rules creation page
* test fixes
* fix breadcrumbs for rules pages
* unit test fixes
* additional fixes
* [Security Solutions] Navigation usage tracker and general changes (#103271)
* [Security Solutions] use of currentAppId$ migrated. and some small fixes
* unused constants removed
* remove unused constant
* test fix and types
* fix cypress
* fix cypress tests
* Fix case navTab permission and tests
* Revert 'timeline.isOpen' breadcrumb code that was deleted during merge
* Fix useInsertTimeline test by removing '/'
* change global navigation visible deeplinks
* fix /admininstration top level redirect to
* fix global search icon, nav order and overview hosts link
* update start a new case link
* fix rules link in exception list table
* unskip cypress tests
* update rules link
* fix full screen timeline
* fixing broken links and administration telemetry split
* remove unused comments
* remove timeline z-index and cleanup global header component
* some minor fixes
* add unit tests for detections breadcrumbs
* remove case to global/search nav when cases is none
* rename test scenario
* fix side_panel flyout
* fix cases use cases between search/gobal nav
* timeline snapshot regenerated and cypres test fixed
* rollback management tracking split as it causes unexpected errors on the telemetry component
Co-authored-by: Pablo Machado <pablo.nevesmachado@elastic.co>
Co-authored-by: Angela Chuang <6295984+angorayc@users.noreply.github.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Michael Olorunnisola <michael.olorunnisola@elastic.co>
Co-authored-by: Angela Chuang <yi-chun.chuang@elastic.co>
Co-authored-by: Xavier Mouligneau <189600+XavierM@users.noreply.github.com>
* Add new runtime types for parsing on client/server.
* Add more runtime types.
* Remove dead code.
* Mark parameter as unused.
* Improve typing for failed journey request function.
* Add new API functions, improve typing in a few others.
* Modify API calls to work with new screenshot_ref data.
* Fix untested refactor error.
* Add required fields to runtime type.
* Update typing in failed steps component.
* Adapt client to work with old screenshots as well as new screenshot_ref.
* Refactor composite code to reusable hook.
* Implement screenshot blocks endpoint.
* Define runtime types for full-size screenshots.
* Delete dedicated screenshot and ref queries.
* Optimize screenshot endpoint by combining queries.
* Handle parsing error.
* Clean up screenshot/ref typings.
* Remove dead types. DRY a type out.
* Simplify types.
* Improve typing in step screenshot components.
* Prefer PNG to JPG for canvas composite op.
* Simplify and clean up some code.
* Remove reliance on `Ping` type, clean up types.
* Add a comment.
* Add a comment.
* Fix typing for `FailedStep` component.
* Standardize loading spinner sizes.
* Add comments to composite code.
* Remove unnecessary optional chaining.
* Reformat error string.
* Remove unneeded key from request return object.
* Add a comment to a return object explaining very large cache value.
* Make type annotation more accurate.
* Resolve some type and test errors.
* Clean up remaining type errors.
* Move type definitions to simplify imports.
* Simplify `PingTimestamp` interface.
* Refactor failing unit test to use RTL and actually test things.
* Add tests for new helper functions.
* Add a comment.
* Test `PingTimestamp` for screenshot ref data.
* Test `StepImageCaption` for ref data.
* Improve typing for step list column definitions.
* Harden a test.
* Extract code to avoid repeated declarations.
* Create centralized mock for `useCompositeImage`.
* Add test for ref to `StepScreenshotDisplay`.
* Add tests for `getJourneyDetails`.
* Extract search results wrapper to helper lib.
* Add tests for `getJourneyFailedSteps`.
* Add support for aggs to result helper wrapper.
* Write tests for `getJourneyScreenshot` and simplify type checking.
* Write tests for `getJourneyScreenshotBlocks`.
* Simplify prop types for `FailedStep`.
* Remove unused type.
* Fix regression in step navigating for new style screenshots.
* Implement PR feedback.
* Implement PR feedback.
* Implement PR feedback.
* Reduce limit of screenshot block queries from 10k to 1k.
* Remove redundant field selection from ES query.
* Implement PR feedback.
* Fix regression that caused "Last successful step" to not show an image.
* Delete unused props from `Ping` runtime type.
* More precise naming.
* Naming improvements. Add `useCallback` to prevent callback re-declaration.
* Prefer explicit props to `{...spread}` syntax.
* Remove redundant type checking.
* Delete obsolete unit tests.
* Fix a regression.
* Add effect to `useEffect`.
* fix up CCR centered sates in tabs content
* update snapshots list
* fix lint errors
* Change tab states for all pages in snapshot+restore
* Remove unnecessary variables
* Seems we dont need the class wrapper
* fix broken type
* Fix bug in ILM table when filtering it down
* center align search box
* fix linter errors
* fix prettier warnings
* revert content var refactor and just focus on ux
* add breakword class to paragraph so we avoid text overflowing
* fix prettier errors
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Removing feature flag changes
* Adding isExportable flag to rule type definition
* Adding isExportable flag to rule type definition
* Adding isExportable flag to rule type definition
* Filtering rule on export by rule type isExportable flag
* Fixing types
* Adding docs
* Fix condition when exportCount is 0
* Unit test for fix condition when exportCount is 0
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Remove role region on flash messages
- just `aria-live` is enough for screen readers to read it out, and `role` was causing "Flash messages" to get read out loud repeatedly between page navigation even when empty which was annoying and not good
* Further a11y attribute recommendations from @myasonik
This PR fixes an edge case where a race condition mught cause the total_results from a federated content source to come back null from the server. This PR tells the server to expect null in those edge cases to prevent browser errors
## Summary
This adds utilities and two strategies for merging using the [fields API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-fields.html) and the `_source` document during signal generation. This gives us the ability to support `constant_keyword`, field alias value support, some runtime fields support, and `copy_to` support. Previously we did not copy any of these values and only generated signals based on the `_source` record values. This changes the behavior to allow us to copy some of the mentioned values above.
The folder of `source_fields_merging` contains a `strategy` folder and a `utils` folder which contains both the strategies and the utilities for this implementation. The two strategies are `merge_all_fields_with_source` and `merge_missing_fields_with_source`. The defaulted choice for this PR is we use `merge_missing_fields_with_source` and not the `merge_all_fields_with_source`. The reasoning is that this is much lower risk and lower behavior changes to the signals detection engine.
The main driving force behind this PR is that ECS has introduced `constant_keyword` and that field has the possibility of only showing up in the fields section of a document and not `_source` when index authors do not push the `constant_keyword` into the `_source` section. The secondary driving forces behind this behavioral change is that some users have been expecting their runtime fields, `copy_to` fields, and field alias values of their indexes to be copied into the signals index.
Both strategies of `merge_missing_fields_with_source` and `merge_all_fields_with_source` are considered Best Effort meaning that both strategies will not always merge as expected when they encounter ambiguous use cases as outlined in the `README.md` text at the top of `source_fields_merging` in detail.
The default used strategy of `merge_missing_fields_with_source` which has the simplest behavior will work in most common use cases. This is simply if the `_source` document is missing a value that is present in the `fields`, and the `fields` value is a primitive concrete value such as a `string` or `number` or `boolean` and the `_source` document does not contain an existing object or ambiguous array, then the value will be merged into `_source` and a new reference is returned. If you call the strategy twice it should be idempotent meaning that the second call will detect a value is now present in `_source` and not re-merge a second time.
* 301 unit tests were added
* Extensive README.md docs are added
* e2e tests are updated to test scenarios and ambiguity and conflicts from previously to support this effort.
* Other e2e tests were updated
* One bug with EQL and fields was found with a workaround implemented. See https://github.com/elastic/elasticsearch/issues/74582
* SearchTypes adjusted to use recursive TypeScript types
* Changed deprecated for `@deprecated` in a few spots
* Removed some `ts-expect-error` in favor of `??` in a few areas
* Added a new handling of epoch strings and tests to `detection_engine/signals/utils.ts` since fields returns `epoch_millis` as a string instead of as a number.
* Uses lodash safer set to reduce changes of prototype pollution
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
### Risk Matrix
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Prototype pollution | Low | High | Used lodash safer set |
| Users which have existing rules that work, upgrade and now we do not generate signals due to bad merging of fields and _source | Mid | High | We use the safer strategy method, `merge_missing_fields_with_source `, that is lighter weight to start with. We might add a follow up PR which enables a key in Kibana to turn off merging of fields with source. We added extensive unit tests and e2e tests. However, unexpected unknowns and behaviors from runtime fields and fields API such as geo-points looking like nested fields or `epoch_milliseconds` being a string value or runtime fields allowing invalid values were uncovered and tests and utilities around that have been added which makes this PR risky |
| Found a bug with using fields and EQL which caused EQL rules to not run. | Low | High | Implemented workaround for tests to pass and created an Elastic ticket and communicated the bug to EQL developers. |
