* [ML] Adding runtime_mappings to job wizards
* fixing test
* adding runtime fields to fields list in datafeed preview
* fixing cardinality count for runtime field
* fixing cardinality check in data visualizer
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Pass filters from SIEM to resolver
* Fix test type errors
* Revert loading state change, update snapshots
* Make correct check in nodeData selector
* Fix inverted logic in nodeData selector
* Adding request id to node data
* Working time range filters and node data stale selector
* Fixing random name from generator and panel icons
* Fixing unused import
* Remove hardcoded node id
* Ensure that the actions array is reset before each test
Co-authored-by: Kevin Qualters <kevin.qualters@elastic.co>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
# [Security Solution][Timeline] Timeline and Draggable Accessibility (a11y) Improvements
This PR improves the accessibility of Timeline and the draggables used throughout the Security Solution.
- ⌨️ Keyboard support for all draggables
- 🖱 Click on any draggable with the mouse to give it keyboard focus
- 🕵️♀️ Press `Enter` to display the draggable's action / options menu (e.g. Filter value, Investigate in Timeline, Show Top N). The menu also displays hyperlinks to allow for keyboard navigation from draggables containing links.
- 🚀 Press `Space` to start dragging via keyboard, and press `Space` again to drop
- 🏓 Timeline, + all Timeline-based tables (i.e. Detections, Host / Network Events, External alerts) support keyboard navigation
- ⬆️⬇️⬅️➡️ arrow key navigation per the `Keyboard Support` section of [w3.org grid examples](https://www.w3.org/TR/wai-aria-practices-1.1/examples/grid/dataGrids.html)
- 📉📈 `page down` and `page up` support per the `Keyboard interactions` section of the [MDN ARIA grid role documentation](<https://developer.mozilla.org/en-US/docs/Web/Accessibility/ARIA/Roles/Grid_Role>) (and also the [w3](https://www.w3.org/TR/wai-aria-practices-1.1/examples/grid/dataGrids.html))
- 🏠🔚 `home` and `end` moves focus to the first and last cell in the row, per [w3.org](https://www.w3.org/TR/wai-aria-practices-1.1/examples/grid/dataGrids.html) & [MDN](<https://developer.mozilla.org/en-US/docs/Web/Accessibility/ARIA/Roles/Grid_Role>)
- 🕹🏠 🕹🔚 `ctrl + home` and `ctrl + end` moves focus to the first and last cell in the row, per [w3.org](https://www.w3.org/TR/wai-aria-practices-1.1/examples/grid/dataGrids.html) & [MDN](<https://developer.mozilla.org/en-US/docs/Web/Accessibility/ARIA/Roles/Grid_Role>)
- 🎉 Screenreader improvements for Timeline, + all Timeline-based tables, and draggables (tested with Voiceover for `macOS` version `10.15`)
- 👂 Draggables implement custom screenreader announcements. For example, when a draggable field is selected, the screenreader reads a custom message that reads the _field name_ in addition to the value, to provide additional context. For example, selecting a field named `network_flow` may be read phonetically by the screen reader as:
> event.action network underscore flow Press enter for actions, or press space to begin dragging
- 📅 Timeline, + all Timeline-based tables provide an enhanced screenreader experience via the use of `EuiScreenReaderOnly` and [w3 grid roles](https://www.w3.org/TR/wai-aria-practices-1.1/examples/grid/dataGrids.html) to provide context. For example, after clicking an arrow key to select a cell, the screenreader reads:
> You are in a table cell. row: 2, column: 5, event.module system Press enter for options, or press space to begin dragging.
- 📰 Timeline's table supports Voiceover screenreader shortcuts that read all cells across a row, or all rows down a column. See <https://github.com/elastic/kibana/issues/83364> for details
- 🍿 popovers and tooltips were enhanced via `EuiScreenReaderOnly` to provide additional context to screen readers
- 🪓 The [axe - Web Accessibility Testing Chrome extension](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd) was run interactively throughout the Security Solution to find and fix the highest-severity issues found by `axe`
- 🧹 Other keyboard-related improvements include
- The `Customize Columns` popover traps keyboard focus, and allows navigation of content via arrow keys & home / end
- The `Event details` view allows navigation of content via arrow keys & home / end
- Improvements to keyboard-focus trapping in popovers throughout the Security Solution
This PR addresses the following issues:
- [X] [[Security Solution] Add support for Voiceover announcements In Timeline's table #83364](https://github.com/elastic/kibana/issues/83364)
- [X] [[Security app] Detections events list 'overflow' ... button and expand > button do not have hover over names #74140](https://github.com/elastic/kibana/issues/74140)
- [X] [SIEM app accessibility issues #64596](https://github.com/elastic/kibana/issues/64596)
- After merging with the _Timeline Evolution_ PR, the drop zone at the bottom of the screen stopped accepting draggables after the first draggable is dropped. The current workaround is to press `Enter` to open the draggables action menu, and select the `Investigate in timeline` action. When Timeline is open, the drag area at the top of the screen accepts additional draggables via keyboard. EDIT: I paired with @XavierM on this, and he has a fix for this issue (duplicate draggable IDs) in his PR.
## Summary
Resolves https://github.com/elastic/kibana/issues/77324, https://github.com/elastic/kibana/issues/77325, resolves https://github.com/elastic/kibana/issues/77325, and resolves https://github.com/elastic/kibana/issues/81302
This PR addresses referential integrity issues when deleting value lists. Previously when deleting value lists, any references in Exception Lists/Items would be left behind. This PR introduces a new confirmation modal when deleting value lists that are referenced in either space aware (`simple`) or space `agnostic` exception lists.
Also includes:
* Fixed Lists plugin `quick_start.sh` as it was using endpoint exception list + value lists (unsupported)
* Adds `quick_start_value_list_references.sh` to create exception lists/items, value lists, and references to easily test
* Add support to `findExceptionList` for searching for both `simple` and `agnostic` list types
* Two new query params have been added to the `deleteListRoute`
* `ignoreReferences` (default:false) when true, maintains pre-7.11 behavior of deleting value list without performing any additional checks.
* NOTE: As written, this becomes an API breaking change as existing existing calls to the same API will `409` conflict if references exist. cc @jmikell821 @DonNateR
* `deleteReferences` (default:false) to perform dry run and identify referenced exception lists/items
## Testing
To test, run `quick_start_value_list_references.sh` and it will create all the necessary resources/references to easily exercise the above functionality. The below diagram details the resources created and how the references are wired up.
> Creates three different exception lists and value lists, and associates as
> below to test referential integrity functionality.
>
> NOTE: Endpoint lists don't support value lists, and are not tested here
>
> EL: Exception list
> ELI Exception list Item
> VL: Value list
>
> EL1 EL2 (Agnostic) EL3
> | | |
> ELI1 ELI2 ELI3
> |\ /| |
> | \ / | |
> | \ / | |
> | \ / | |
> | \/ | |
> | /\ | |
> | / \ | |
> | / \ | |
> | / \ | |
> |/ \| |
> VL1 VL2 VL3 VL4
> ips.txt ip_range.txt text.txt hosts.txt
>
Corner cases to be aware of:
* An exception item may have multiple value list entries -- only referenced value list entries should be removed
* There is no API for removing individual entries. If all entries are references the entire item is deleted. If only some entries are references, the item is updated via a `PUT` (no `PATCH` support for exception items)
* It's not possible via the UI to create a space agnostic list that has value list exception items (only agnostic endpoint exception lists can be created and they do not support value lists). Please use above script to exercise this behavior.
Additional notes:
* Once the Exception List table is introduced (https://github.com/elastic/kibana/pull/85465), we can add an enhancement for deeplinking to exception lists from the reference error modal.
* The `deleteListRoute` response has been updated to include the responses from the reference checks to provide maximum flexibility
* There is no bulk API for deleting exception list items, and so they are iterated over via the `deleteExceptionListItem` API.
##### Reference error modal
<p align="center">
<img width="500" src="https://user-images.githubusercontent.com/2946766/102199153-813e1e80-3e80-11eb-8a9b-af116ca13df9.gif" />
</p>
##### Overflow example
<p align="center">
<img width="500" src="https://user-images.githubusercontent.com/2946766/102199032-5784f780-3e80-11eb-81c7-17283d002ce4.gif" />
</p>
### Checklist
Delete any items that are not applicable to this PR.
- [X] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)
- [X] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
- [X] Any UI touched in this PR is usable by keyboard only (learn more about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
### For maintainers
- [X] This was checked for breaking API changes and was [labeled appropriately](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
* [Maps] unify legend for percentiles, interpolate, and custom ordinal color breaks (#85343)
* [Maps] unify legend for percentiles, interpolate, and custom ordinal color breaks
* unify mapbox style expression
* snapshot updates
* update jest expects
* use less than and great than symbol for first and last stop
* tslint
* do not show legend if customColorRamp is not provided
* update functional test expects
* update functional test expect
* update mapbox expect expressions in join functional tests
* update mvt expects
* revert mapbox style expression changes for interpole color ramp
* add greater then and equal to symbol for last stop
* disable stop input for first ordinal color stop
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* fix crossmerge
Co-authored-by: Nathan Reese <reese.nathan@gmail.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
**Addresses:** https://github.com/elastic/kibana/issues/76587
## Summary
In this PR I'm doing basically 2 things:
1. Making readonly callouts we have in Detections `primary` instead of `warning` and thus persistable in local storage (if a callout is dismissed, we remember it there)
2. Creating a reusable implementation for that.
TODO:
- [x] Adjust all callouts used in Detections to be of type `primary`
- [x] Implement the local storage logic (dumb)
- [x] Implement the local storage logic (reusable)
- [x] Add a new user role: "Reader" (read-only user)
- [x] Add Cypress tests
Out of scope:
- Add unit tests. I'll probably address this in a follow-up PR. Do you think it's worth it or better to wait until the rework (see the next point)?
- Rework callouts to standardise them across Detections, Cases and other parts of the Security app? See my comment below in this PR.
## Screen recordings
Quick demo of how this implementation works:
- [primary callouts](https://drive.google.com/file/d/1SYQd_ihKPvzlVUxELI8qNEqLBOkL18Gd/view?usp=sharing)
- [warning, danger](https://drive.google.com/file/d/1lrAFPyXNjOYSiEsUXxY_fjXsvmyDcdWY/view?usp=sharing) (callout types here were manually adjusted)
## Additional notes
Cypress tests are based on the work done in https://github.com/elastic/kibana/pull/81866.
* [Maps] unify legend for percentiles, interpolate, and custom ordinal color breaks
* unify mapbox style expression
* snapshot updates
* update jest expects
* use less than and great than symbol for first and last stop
* tslint
* do not show legend if customColorRamp is not provided
* update functional test expects
* update functional test expect
* update mapbox expect expressions in join functional tests
* update mvt expects
* revert mapbox style expression changes for interpole color ramp
* add greater then and equal to symbol for last stop
* disable stop input for first ordinal color stop
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* [Setup] Add EuiCardTo helper
* Set up very basic DocumentCreationLogic file modal management
* Create buttons and modal components
* Update EngineOverviewEmpty to use only DocumentCreationButtons
- Remove API section/example/constants for now: per design discussion, we'll be moving this section into the document creation modal
- Move preceding DOCUMENT_CREATION_DESCRIPTION paragraph directly into DocumentCreationButtons, since the Documents view will just use this verbatim
* Update Documents view Index Documents button to show DocumentsCreationModal
- with some extra logic to add a new creationStep view
+ some minor test cleanup / clarification for openDocumentCreation
* Update Crawler button with conditional check
- Forgot to port this over initially
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
resolves https://github.com/elastic/kibana/issues/75601
Previously, if a context variable that is an object is referenced in a
mustache template used as an action parameter, the resulting variable
expansion will be `[Object object]`. In this PR, we change this so that
the expansion is a JSON representation of the object.
This is primarily for diagnostic purposes, so that customers can see
all the context variables available, and their values, while testing
testing their alerting actions.
* Initial structure of migration state-action machine
* Fix type import
* Retries with exponential back off
* Use discriminated union for state type
* Either type for actions
* Test exponential retries
* TaskEither types for actions
* Fetch indices instead of aliases so we can collect all index state in one request
* Log document id if transform fails
* WIP: Legacy pre-migrations
* UPDATE_TARGET_MAPPINGS
* WIP OUTDATED_DOCUMENTS_TRANSFORM
* Narrow res types depending on control state
* OUTDATED_DOCUMENTS_TRANSFORM
* Use .kibana instead of .kibana_current
* rename control states TARGET_DOCUMENTS* -> OUTDATED_DOCUMENTS*
* WIP MARK_VERSION_INDEX_READY
* Fix and expand INIT -> * transition tests
* Add alias/index name helper functions
* Add feature flag for enabling v2 migrations
* split state_action_machine, reindex legacy indices
* Don't use a scroll search for migrating outdated documents
* model: test control state progressions
* Action integration tests
* Fix existing tests and type errors
* snapshot_in_progress_exception can only happen when closing/deleting an index
* Retry steps up to 10 times
* Update api.md documentation files
* Further actions integration tests
* Action unit tests
* Fix actions integration tests
* Rename actions to be more domain-specific
* Apply suggestions from code review
Co-authored-by: Josh Dover <me@joshdover.com>
* Review feedback: polish and flesh out inline comments
* Fix unhandled rejections in actions unit tests
* model: only delay retryable_es_client_error, reset for other left responses
* Actions unit tests
* More inline comments
* Actions: Group index settings under 'index' key
* bulkIndex -> bulkOverwriteTransformedDocuments to be more domain specific
* state_action_machine tests, fix and add additional tests
* Action integration tests: updateAndPickupMappings, searchForOutdatedDocuments
* oops: uncomment commented out code
* actions integration tests: rejection for createIndex
* update state properties: clearer names, mark all as readonly
* add state properties currentAlias, versionAlias, legacyIndex and test for invalid version scheme in index names
* Use CONSTANTS for constants :D
* Actions: Clarify behaviour and impact of acknowledged: false responses
* Use consistent vocabulary for action responses
* KibanaMigrator test for migrationsV2
* KibanaMigrator test for FATAL state and action exceptions in v2 migrations
* Fix ts error in test
* Refactor: split index file up into a file per model, next, types
* next: use partial application so we don't generate a nextActionMap on every call
* move logic from index.ts to migrations_state_action_machine.ts and test
* add test
* use `Root` to allow specifying oss mode
* Add fix and todo tests for reindexing with preMigrationScript
* Dump execution log of state transitions and responses if we hit FATAL
* add 7.3 xpack tests
* add 100k test data
* Reindex instead of cloning for migrations
* Skip 100k x-pack integration test
* MARK_VERSION_INDEX_READY_CONFLICT for dealing with different versions migrating in parallel
* Track elapsed time
* Fix tests
* Model: make exhaustiveness checks more explicit
* actions integration tests: add additional tests from CR
* migrations_state_action_machine fix flaky test
* Fix flaky integration test
* Reserve FATAL termination only for situations which we never can recover from such as later version already migrated the index
* Handle incompatible_mapping_exception caused by another instance
* Cleanup logging
* Fix/stabilize integration tests
* Add REINDEX_SOURCE_TO_TARGET_VERIFY step
* Strip tests archives of */.DS_Store and __MAC_OSX
* Task manager migrations: remove invalid kibana property when converting legacy indices
* Add disabled mappings for removed field in map saved object type
* verifyReindex action: use count API
* REINDEX_BLOCK_* to prevent lost deletes (needs tests)
* Split out 100k docs integration test so that it has it's own kibana process
* REINDEX_BLOCK_* action tests
* REINDEX_BLOCK_* model tests
* Include original error message when migration_state_machine throws
* Address some CR nits
* Fix TS errors
* Fix bugs
* Reindex then clone to prevent lost deletes
* Fix tests
Co-authored-by: Josh Dover <me@joshdover.com>
Co-authored-by: pgayvallet <pierre.gayvallet@elastic.co>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Copy SourceLogic to AddSourceLogic
This commit merely copies SourceLogic and renames it to AddSourceLogic with the name of the component and interfaces changed. This will make reviewing the changes easier later
* Delete duplicated logic data
* Update component paths
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* [Maps] provide function to create basemap layer descriptor
* update typing
* update APM map to use createBasemapLayerDescriptor
* fix useLayerList jest test
* tslint fixes
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
