* Set up Observability rule APIs
* Populate alerts table with data from API
* Move field map types/utils to common
* Format reason/link in alert type
* Format reason/link in alert type
* Fix issues with tsconfigs
* Storybook cleanup for example alerts
* Use `MemoryRouter` in the stories and `useHistory` in the component to get the history
* Replace examples with ones from "real" data
* Use `() => {}` instead of `jest.fn()` in mock registry data
* Store/display evaluations, add active/recovered badge
* Some more story fixes
* Decode rule data with type from owning registry
* Use transaction type/environment in link to app
* Fix type issues
* Fix API tests
* Undo changes in task_runner.ts
* Remove Mutable<> wrappers for field map
* Remove logger.debug calls in alerting es client
* Add API test for recovery of alerts
* Revert changes to src/core/server/http/router
* Use type imports where possible
* Update limits
* Set limit to 100kb
Co-authored-by: Nathan L Smith <smith@nlsmith.com>
Co-authored-by: Nathan L Smith <smith@nlsmith.com>
* Intercept installation errors and add meta info.
* Adjust mock.
* Catch errors in all steps of install/upgrade.
* Adjust handler for direct package upload.
* Don't throw not-found errors on assets during rollback.
* Correctly catch errors from _installPackage()
* Propagate error from installResult in bulk install case.
* Add tests for rollback.
* Remove unused code.
* Skipping test that doesn't test what it says.
* Fix and reenable test.
* Move inspector adapter integration into search source
* docs and ts
* Move other bucket to search source
* test ts + delete unused tabilfy function
* hierarchical param in aggconfig.
ts improvements
more inspector tests
* fix jest
* separate inspect
more tests
* jest
* inspector
* Error handling and more tests
* put the fun in functional tests
* code review
* Add functional test for other bucket in search example app
* test
* test
* ts
* test
* test
* ts
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Liza Katz <lizka.k@gmail.com>
## Summary
Fixes some recent flakeyness with Cypress tests
* Adds cypress.pipe() on button clicks around the area of flakes
* Adds an alerting threshold to the utilities so we can wait for when an exact number of alerts are available on a page
* Changes the alerts to not run again with 10 seconds, because if a test takes longer than 10 seconds, the rule can run a second time which can invalidate some of the text when running checks when timeline or other components update on their button clicks.
### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
Co-authored-by: Frank Hassanabad <frank.hassanabad@elastic.co>
* step 1 to add aggs in the find function of saved object
* setp 2 - add specific unit test to aggs + fix bug found during integrations
* step 3 - add security api_integration arounds aggs
* fix types
* unit test added for aggs_utils
* add documentation
* fix docs
* review I
* doc
* try to fix test
* add the new property to the saved object globaltype
* fix types
* delete old files
* fix types + test api integration
* type fix + test
* Update src/core/server/saved_objects/types.ts
Co-authored-by: Rudolf Meijering <skaapgif@gmail.com>
* review I
* change our validation to match discussion with Pierre and Rudolph
* Validate multiple items nested filter query through KueryNode
* remove unused import
* review + put back test
* migrate added tests to new TS file
* fix documentation
* fix license header
* move stuff
* duplicating test mappings
* rename some stuff
* move ALL the things
* cast to aggregation container
* update generated doc
* add deep nested validation
* rewrite the whole validation mechanism
* some cleanup
* minor cleanup
* update generated doc
* adapt telemetry client
* fix API integ tests
* fix doc
* TOTO-less
* remove xpack tests
* list supported / unsupported aggregations
* typo fix
* extract some validation function
* fix indent
* add some unit tests
* adapt FTR assertions
* update doc
* fix doc
* doc again
* cleanup test names
* improve tsdoc on validation functions
* perf nit
Co-authored-by: Xavier Mouligneau <189600+XavierM@users.noreply.github.com>
Co-authored-by: Rudolf Meijering <skaapgif@gmail.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Xavier Mouligneau <189600+XavierM@users.noreply.github.com>
Co-authored-by: Rudolf Meijering <skaapgif@gmail.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* added threat-match timeline template
* added indicator match timeline template
* updated name to threat match
* updated name to threat match
* changed file name to threat.json
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Andrew Pease <7442091+peasead@users.noreply.github.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Adds helper to normalize legacy ML rule field to an array
This will be used on read of rules, to normalize legacy rules while
avoiding an explicit migration.
* Fix our detection-specific ML search function
Luckily this was just a translation layer to our anomaly call, and the
underlying functions already accepted an array of strings.
* WIP: Run rules against multiple ML Job IDs
We don't yet support creation of rules with multiple job ids, either on
the API or the UI, but when we do they will work.
Note: the logic was previously to generate an error if the underlying
job was not running, but to still query and generate alerts. Extending
that logic to multiple jobs: if any are not running, we generate an
error but continue querying and generating alerts.
* WIP: updating ml rule schemas to support multiple job IDs
* Simplify normalization method
We don't care about null or empty string values here; those were
holdovers from copying the logic of normalizeThreshold and don't apply
to this situation.
* Move normalized types to separate file to fix circular dependency
Our use of NonEmptyArray within common/schemas seemed to be causing the
above; this fixes it for now.
* Normalize ML job_ids param at the API layer
Previous changes to the base types already covered the majority of
routes; this updates the miscellaneous helpers that don't leverage those
shared utilities.
At the DB level, the forthcoming migration will ensure that we always
have "normalized" job IDs as an array.
* Count stopped ML Jobs as partial failure during ML Rule execution
Since we continue to query anomalies and potentially generate alerts, a
"failure" status is no longer the most accurate for this situation.
* Update 7.13 alerts migration to allow multi-job ML Rules
This ensures that we can assume string[] for this field during rule
execution.
* Display N job statuses on rule details
* WIP: converts MLJobSelect to a multiselect
Unfortunately, the SuperSelect does not allow multiselect so we need to
convert this to a combobox. Luckily we can reuse most of the code here
and remain relatively clean.
Since all combobox options must be the same (fixed) height, we're
somewhat more limited than before for displaying the rows. The
truncation appears fine, but I need to figure out a way to display the
full description as well.
* Update client-side logic to handle an array of ML job_ids
* Marginally more legible error message
* Conditionally call our normalize helper only if we have a value
This fixes a type error where TS could not infer that the return value
would not be undefined despite knowing that the argument was never
undefined. I tried some fancy conditional generic types, but that didn't
work.
This is more analogous to normalizeThresholdObject now, anyway.
* Fix remaining type error
* Clean up our ML executor tests with existing contract mocks
* Update ML Executor tests with new logic
We now record a partial failure instead of an error.
* Add and update tests for new ML normalization logic
* Add and update integration tests for ML Rules
Ensures that dealing with legacy job formats continues to work in the
API.
* Fix a type error
These params can no longer be strings.
* Update ML cypress test to create a rule with 2 ML jobs
If we can create a rule with 2 jobs, we should also be able to create a
rule with 1 job.
* Remove unused constant
* Persist a partial failure message written by a rule executor
We added the result.warning field as a way to indicate that a partial
failure was written to the rule, but neglected to account for that in the
main rule execution code, which caused a success status to immediately
overwrite the partial failure if the rule execution did not otherwise
fail/short-circuit.
Co-authored-by: Ryland Herrick <ryalnd@gmail.com>
* Move alert-specific mocks to more declarative mock file
* Add placeholder interface for ECS threat fields
* Test and implement CTI row renderer
The display details are not yet implemented, but those will be fleshed
out in the ThreatMatchRow component.
* Pass full fields data to our row renderers
This data is not used by any existing row renderers and so this commit
is mostly just plumbing that data through.
This is necessary, however, for our new threat match row renderer as it
requires nested fields, which cannot be retrieved through the mechanism
that retrieves the existing row renderer data. However, these nested
fields are available, if requested, through this other data structure,
hence this plumbing.
For now to minimize changes I'm marking this as an optional field;
however in reality a value will always be present.
* Rewrite existing row renderer in terms of flattened data
Updates logic, tests and mocks accordingly.
* Moving logic into discrete files
* helpers
* explicit fields file, which will hopefully be part of the renderer API
at some point
* parent component to split data into "rows" as defined by our renderer
* row component for stateless presentation of a single match
* Register threat match row rendere
Adds tentative copy, example row, and accompanying mock data.
* WIP: Rendering draggable fields but hit the data loss issue with nested fields being flattened
* WIP: implementing row renderer against new data format
I haven't yet deleted the old (new?) unused path yet. Cleanup to come.
* Updating based on new data
* Rewrites isInstance logic for new data as helper, hasThreatMatchValue
* Updating types and tests
* Adds to the previously empty ThreatEcs
* Revert "Pass full fields data to our row renderers"
This reverts commit 19c93ee0732166747b5472433cd5fc813638e21b.
We ended up extending the existing data (albeit from the fields
response!).
* Fix draggables
* adds contextId and eventId to pass to draggable
* We don't have a order-independent key for each individual
ThreatMatchRow, due to matched.id not being mapped/returned in the
fields response
* Fixes up a few things related to using the new data format
* Move indicator field strings to constants
* Fix example data for CTI row renderer
* Adds missing Threat ECS types
* Move CTI field constants to common folder
In order to use these in both the row renderer and the server request,
we need to move them to common/
* Remove redundant CTI fields from client request
These are currently hardcoded on the backend of the events/all query
(via TIMELINE_EVENTS_FIELDS); declaring them on both ends is arguably
confusing, and we're going with YAGNI for now.
* Add missing graphQL type
This was causing type errors as this enum exists both here and in
common/, and I had only updated one of them.
* Updates tests
One is still failing due to an outdated test subject, but I expect this
to change after an upcoming meeting so leaving it for now.
* Split ThreatMatchRow into subcomponents
One for displaying match details, and another for indicator details
The indicator details will be sparse, so there's going to be some
conditional rendering in there.
* Make CTI row renderer look nice
* Adds translations for copy
* Fixes most of our layout woes with more flexbox!
* Conditional rendering of indicator details based on data
* tests
* Make indicator reference field an external link
Leverages the existing FormattedFieldValue component, with one minor
tweak to add this field to the URL allowlist.
* Back to consistent horizontal spacing, here
The draggable badges are a little odd in that their full box isn't
indicated until hover, making the visual weight a little off.
* Add hr as a visual separator between each match "row" of the row renderer
* Fix tests broken due to addition of a new row renderer
These tests are all implicitly testing the list of row renderers.
* Full-width hr
At certain container widths, a half-width hr is not sufficient.
* More descriptive constant
Obviates the need for the accompanying comments.
* More realistic data
Also ensures less traffic to urlhaus ;)
* Remove useless comment
* Add threat_match row renderer type to GQL client
Gennin' beanz
* Ensure contextId is unique for each CTI subrow
We need to add the row index to our contextId to ensure that our
draggables work correctly for multiple rows, since each row will
necessarily have the same eventId and timelineId.
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Ryland Herrick <ryalnd@gmail.com>
* [ML] Data Frame Analytics: Ensure creation and results views display nested fields correctly (#96905)
* create analytics field service
* remove unnecessary field filter. update types
* create common base class for newJobCapabilites classes for AD and DFA
* fix column schema for histogram
* update endpoint to be consistent with AD job caps
* add unit test for removeNestedFieldChildren helper function
* removes obsolete const
* skip flaky suite to be consistent with master
Remove `getMockTheme` and use `EuiThemeProvider` from the kibana_react plugin.
Use the CSF-style decorators with `EuiThemeProvider` in the stories.
No functional changes, but should be less code to maintain.
Co-authored-by: Nathan L Smith <nathan.smith@elastic.co>
* Counting number of alert history connectors in use
* Telemetry for preconfigured alert history config enabled
* Updating telemetry mappings
* Updating tests
* Adding descriptions to new telemetry fields
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: ymao1 <ying.mao@elastic.co>
* Remove field type filtering from IndexPatternSelect
* remove unused import
* api doc updates
* update jest snapshots
* another fix for jest test
* review feedback
* tslint
* update jest snapshot for changes
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* network module
adds the network module with four ml jobs for the 7.13 release
* Update datafeed_high_count_network_denies.json
json formatting
* update test
added the security_network module to the list
* renames
module name change to security_network / Security: Network
* formatting
change hyphen char to underscores
* fixes and name changes
fixes to df queries, descriptions. created_by param
* update tests
tests need the security_network module added
* formatting
change hyphens to underscores
* descriptions
format descriptions
* Update datafeed_high_count_network_events.json
indentation fixes
* Update x-pack/plugins/ml/server/models/data_recognizer/modules/security_network/manifest.json
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
* Update x-pack/plugins/ml/server/models/data_recognizer/modules/security_network/ml/high_count_network_denies.json
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
* Update datafeed_high_count_network_events.json
change to a filter
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
Co-authored-by: Craig Chamberlain <randomuserid@users.noreply.github.com>
Co-authored-by: Lisa Cawley <lcawley@elastic.co>
* Adapt the anomalies table to work in overlay
* Wire up the onClose function
* Make "show in inventory" filter waffle map
* Remove unused variable
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
