* adding alerts and connectors to import/export test
* removing beforeEach
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* converting Maps es_archiver to kbn_archiver
* delete the esArchiver .kibana reference directory
* fix the path of the json file
* use the delete API to delete the missing references populated in the data.json
* fix the path
* kbn_archiver_maps.json
* added the missing ref
* restoring it to use esArchiver
* replace esArchiver to use kbnArchiver
* moved the data.json directly under kbnArchiver
Please enter the commit message for your changes. Lines starting
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Add pure fn and consuming hook to fetch event enrichment
It's not being invoked yet, but I've added a placeholder where it's
going.
* Move existing enrichment tests to new spec file
This is a rough copy/paste, I'll clean up as I flesh out the new tests.
* Move test constants into tests that use them
* style: declare FC function as an FC
* Extract some inline parsing logic into a helper function
And test it!
* Solidifying enrichment types on the backend
* Declares an enum for our types
* Sets type during indicator match rule enrichment
* Sets type during investigation-time enrichment
* WIP: Enrichment rows are rendered on the alerts summary
There are lots of TODOs here, but this implements the following:
* Fetching investigation-time enrichments from the backend
* Parsing existing enrichments from timeline data
* Merging the two enrichment types together, and rendering them in rows
as specified
Much of the data-fetching is hardcoded, and this broke the existing
pattern with SummaryView/SummaryRow so that got a little messy; I may
end up just using my own EuiTable but we'll see.
Threat Intel tab is currently broken; that's up next.
* Updates ThreatDetailsView to accept an array of enrichments
The investigation-time enrichments are a little messy because they
contain all the non-ECS fields that indicators contain; other than that,
this is looking good.
Still need to add the new header, and potentially sort the fields.
* Sort our details fields
This promotes sanity for the user.
* Add "view threat intel data" button
This simply opens the threat intel tab.
* Implement header for threat details sections
* Add a basic jest "unit" test around ThreatSummaryView
* Fix remaining tests for components we modified
This also addresses a bug where we were not properly sorting new
enrichments by first_seen; this is covered under the tests that were
fixed.
* Filter out duplicate investigation-time enrichments
Because the enrichment endpoint is dumb and doesn't know about the
existing event or its enrichments, we need to merge these together on
the client to reduce noise and redundant data.
* Add inspect button to investigation enrichments
* Massages the response into the format that the inspect component uses
* Moves stateful fetching of query and persisting in redux to new, more
specialized hook
* Moves existing enrichment hook to a more suitable location in
containers/
* Fix failing unit tests
* indicator match rule now specifies `matched.type` as coming from the
rule
* Inspecting the enrichment query requires use of the redux store, which
was not previously mocked
* Fix existing CTI cypress tests
This covers the basics of the Alert Summary and Threat Intel tabs; the
investigation-time enrichment functionality is up next.
* Adds a cypress test exercising investigation time enrichment
* Loads more indicators (filebeat data, `threat_indicator2` archive)
AFTER the rule has executed
* Asserts that those indicators are also found on the alert summary.
* Populate event enrichment call with actual alert fields
This was previously hardcoded during development.
* Add a new field to our suspicious event to trigger enrichment
The existing myhash field will generate an alert due to the way the rule
is written, but the alert had no other fields that would match the
investigation time enrichment. This gives it a source.ip, and updates
the indicator to match.
* Only fetch enrichments data if there are valid event fields
If none of the alert's fields would be relevant to the enrichment query,
then we don't make the request at all.
* Update enrichments matched.typed in integration tests
This field was updated to reflect the source of the match, in this case:
indicator match rules.
* Ensure draggable fields are unique in a multi-match scenario
If a given field matched multiple indicators, then the previous
contextId was not unique as it was based on field/value that matched.
Adding provider to the mix would fix it, except that we're not
guaranteed to have a provider.
I've added both provider (if present) and an index value to the key to
ensure that it's unique.
* Simplify types
This field can never be null, as we always set it in our response.
* Move helper functioons out of shared location and into consuming component
These are unlikely to be used elsewhere.
* Clean up data parsing logic using reduce
This obviates the need for our filter/guard function and the extra loop
that it entails. We have to specify the return value of our reduce fn,
however, but that's mostly equivalent to our type guard.
* Move our general function into a general location
* Extract the concept of "enrichment identifiers"
This was already partially codified with 'buildEnrichmentId,' which is
used to dedup enrichments; this extends the idea to all fields that
could uniquely identify a given indicator.
* Use existing constant as the source of our enrichments query
This is now used by both the overview card and the enrichment query.
* Codify our default enrichment lookback as constants
* Remove unnecessary flexbox
The generic SummaryView component previously had to deal with
multi-valued CTI fields, representing the multiple values coming from
the multiple nested objects with that field.
However, with the new UI we no longer have that constraint, and so the
default columnar style, and the corresponding overriding styles, are no
longer necessary.
* Filter out partial responses in the event enrichment observable
The UI does not currently handle these. We need to test the behavior of
long-running queries with this filter, but this should simplify the
behavior to complete/error until we handle partial responses.
* Display placeholders while event enrichment is loading
Displays a loading spinner in the Threat Intel tab title, and some
loading lines where the enrichments summary is.
* Update our indicator data to be within the last 30 days
This fixes our cypress test, but it's going to start failing again in 30
days. However, by that time I'll have implemented the absolute data
picker, which will allow for a more comprehensive test in addition to us
sidestepping this issue.
* Fix type error with our details tabs
The name prop on a Tab will be rendered as a node, so both strings and
elements are acceptable. This relaxes the types to inherit from the
component itself.
* Fix failing jest tests
The addition of our filtering of the search observable broke this test,
since we now need to implement the search observable.
Rather than do that, we'll instead mock our local hook as that's more
likely to change.
* [Solution Toolbar] Fixing button border on non-text color versions
* [Alerts] Removed extra wrappers and use EuiPageHeader
* [Logstash] Basic conversion to template
* [Reporting] Adding bottomBorder to page header
* [ML] Fix display of main navigation tabs
* [Stack Management] Fix side nav not updating when going back to landing page
* [Tags] Add spacing after page header
* [License Management] Full width on file uploader
* [Page Template] Fixed `emptyState` default template for pages with side nav
* [Infra] Removing some page header displays in empty states
* [Enterprise Search] Fix some error layouts
* [Index Patterns] Quick fix for empty state
* snaps
* [Page Template] Remove forced padding when `centeredBody`
* small hack for tab padding for ml
* scroll ML page to fix test
* fix test method type signature
Co-authored-by: Dave Snider <dave.snider@gmail.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Michail Yasonik <michail.yasonik@elastic.co>
* Migrate kibana.autocomplete config to data plugin
* Fix CI
* Fix tests
* Use new terms enum API for autocomplete value suggestions
* Add tiers to config
* Re-introduce terms agg and add config/tests for swapping algorithms
* Add data_content and data_cold tiers by default
* Fix types
* Fix maps test
* Update tests
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Adds the ability to collapse the sidenav. This should work in all solutions. It also adds breakpoints that turn it into a flyout at lower screen widths.
* create alert per node instead of per cluster
* add comment
* fix test, replace alert state with empty array with no node is firing
* update cpu usage action messaging
* fix internationalization
* update disk usage rule action messaging
* update memory usage rule action messaging
* update other action messaging
* update missing monitoring data alert action messaging
* remove comment
* fix bug where threadpool alerts were not firing
* fix bug with threadpool rejections and update alert action messaging to be per node
* update comments
* unit test for thread pool write rejections alert
* update messaging for CCR read rejection
* fix cluster level alerts to use the cluster id when its not node level
* add more tests to nodes changed alert
* update default message
* update alert messaging for large shard size
* update default messaging
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Removing feature flag changes
* Adding isExportable flag to rule type definition
* Adding isExportable flag to rule type definition
* Adding isExportable flag to rule type definition
* Filtering rule on export by rule type isExportable flag
* Fixing types
* Adding docs
* Fix condition when exportCount is 0
* Unit test for fix condition when exportCount is 0
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
This adds utilities and two strategies for merging using the [fields API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-fields.html) and the `_source` document during signal generation. This gives us the ability to support `constant_keyword`, field alias value support, some runtime fields support, and `copy_to` support. Previously we did not copy any of these values and only generated signals based on the `_source` record values. This changes the behavior to allow us to copy some of the mentioned values above.
The folder of `source_fields_merging` contains a `strategy` folder and a `utils` folder which contains both the strategies and the utilities for this implementation. The two strategies are `merge_all_fields_with_source` and `merge_missing_fields_with_source`. The defaulted choice for this PR is we use `merge_missing_fields_with_source` and not the `merge_all_fields_with_source`. The reasoning is that this is much lower risk and lower behavior changes to the signals detection engine.
The main driving force behind this PR is that ECS has introduced `constant_keyword` and that field has the possibility of only showing up in the fields section of a document and not `_source` when index authors do not push the `constant_keyword` into the `_source` section. The secondary driving forces behind this behavioral change is that some users have been expecting their runtime fields, `copy_to` fields, and field alias values of their indexes to be copied into the signals index.
Both strategies of `merge_missing_fields_with_source` and `merge_all_fields_with_source` are considered Best Effort meaning that both strategies will not always merge as expected when they encounter ambiguous use cases as outlined in the `README.md` text at the top of `source_fields_merging` in detail.
The default used strategy of `merge_missing_fields_with_source` which has the simplest behavior will work in most common use cases. This is simply if the `_source` document is missing a value that is present in the `fields`, and the `fields` value is a primitive concrete value such as a `string` or `number` or `boolean` and the `_source` document does not contain an existing object or ambiguous array, then the value will be merged into `_source` and a new reference is returned. If you call the strategy twice it should be idempotent meaning that the second call will detect a value is now present in `_source` and not re-merge a second time.
* 301 unit tests were added
* Extensive README.md docs are added
* e2e tests are updated to test scenarios and ambiguity and conflicts from previously to support this effort.
* Other e2e tests were updated
* One bug with EQL and fields was found with a workaround implemented. See https://github.com/elastic/elasticsearch/issues/74582
* SearchTypes adjusted to use recursive TypeScript types
* Changed deprecated for `@deprecated` in a few spots
* Removed some `ts-expect-error` in favor of `??` in a few areas
* Added a new handling of epoch strings and tests to `detection_engine/signals/utils.ts` since fields returns `epoch_millis` as a string instead of as a number.
* Uses lodash safer set to reduce changes of prototype pollution
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
### Risk Matrix
| Risk | Probability | Severity | Mitigation/Notes |
|---------------------------|-------------|----------|-------------------------|
| Prototype pollution | Low | High | Used lodash safer set |
| Users which have existing rules that work, upgrade and now we do not generate signals due to bad merging of fields and _source | Mid | High | We use the safer strategy method, `merge_missing_fields_with_source `, that is lighter weight to start with. We might add a follow up PR which enables a key in Kibana to turn off merging of fields with source. We added extensive unit tests and e2e tests. However, unexpected unknowns and behaviors from runtime fields and fields API such as geo-points looking like nested fields or `epoch_milliseconds` being a string value or runtime fields allowing invalid values were uncovered and tests and utilities around that have been added which makes this PR risky |
| Found a bug with using fields and EQL which caused EQL rules to not run. | Low | High | Implemented workaround for tests to pass and created an Elastic ticket and communicated the bug to EQL developers. |
* Automatically install and update the security_detection_engine package
* Remove security_detection_engine from required Fleet packages
* Update fleet package-registry image
* Add sha256: to the distribution package
* Use distribution from https://beats-ci.elastic.co/job/Ingest-manager/job/release-distribution/152
* Change fleet required packag
* Fix bad merge
* Update rules to 0.13.1 package
* Fix NOTICE.txt
resolves#98634
This adds a new object property to the event log kibana object named
task, with two properties to track the time the task was scheduled to
run, and the delay between when it was supposed to run and when it
actually started. This task property is only added to the appropriate
events.
task: schema.maybe(
schema.object({
scheduled: ecsDate(),
schedule_delay: ecsNumber(),
})
),
Note that these changes were previously merged to master in https://github.com/elastic/kibana/pull/102252 which had to be reverted - this PR contains the same commits, plus some additional ones to resolve the tests that were broken during the bad merge.
* cancel the previous session
* split to 3 tasks
* fixes
* cancellation
* updated tests
* split out and improve jest tests
* cleanup previous session properly
* don't fail delete and cancel if item was already cleaned up
* test
* test
* ignore resource_not_found_exception when deleting an already cleared \ expired async search
* jest
* update jest
* api int
* fix jest
* testssss
* Code review @dosant
* types
* remove any
* Fix merge
* type
* test
* jest
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Reporting: Check for pending jobs scheduled with ESQueue
* Update x-pack/plugins/reporting/server/lib/tasks/execute_report.ts
Co-authored-by: Vadim Dalecky <streamich@gmail.com>
* update test assertions, use more explicit types
* update comment
* Update x-pack/plugins/reporting/server/lib/store/store.ts
Co-authored-by: Vadim Dalecky <streamich@gmail.com>
* fix field mapping
* Update x-pack/plugins/reporting/server/lib/store/store.ts
Co-authored-by: Jean-Louis Leysens <jloleysens@gmail.com>
* Report also implements ReportDocumentHead
* the actual ID of the task is prefixed with `task:`
* remove pointless update to the report instance after failing
* comment clarification
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Vadim Dalecky <streamich@gmail.com>
Co-authored-by: Jean-Louis Leysens <jloleysens@gmail.com>
