* Allow ML Rules to be patched
* Test passing of params from our patch routes to our helpers
Since patchRules accepts a partial there's no way to verify this in
typescript, we need regression tests instead.
* Update lists when importing with overwrite
This was simply missed earlier.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Adds rotation transform which does the top->bottom to left->right
transformation + an extra 5 degrees which results in taxi edges
separating when rendered.
* PR feedback to reduce edge width on hover, and assure that connected
edges are highlighted when node is selected/focused
* update disabled kuery bar placeholder text for service map
resolves https://github.com/elastic/kibana/issues/60744
This is a fairly complex test, with alerts that run actions that write to
an index which we then do queries over. The tests didn't account for some
slop in all that async activity, but now should be about as flake-free as they
can be.
resolves https://github.com/elastic/kibana/issues/59889
The index threshold APIs - used by both the index threshold UI and the
alert executor - were returning errors (500's from http endpoints) when
getting errors from ES.
These have been changed so that the error is logged as a warning, and the
relevant API returns an "empty" result.
Another 500 response was found while experimenting with this. Apparently
the date_range agg requires a date format to be passed in if the date format
in ES is not an ISO date. The repro on this was to select the `.security`
alias (or it's index) within the index threshold alert UI, and then select
one of it's date fields.
* Use ingest API to get endpoint datasources
* Add `application` service to `KibanaContextProvider`
* Adjust Policy list to show data available from API
* Added ingest service + refactored middleware
* handle api failures/errors
* Removed policy list fake_data generator
* Fix typing
* Rename method + added explicit return type
* move dispatch outside of try block
* Removed unnecessary action
* Added FIXME comments with link to issue
* Skip some functional tests
* added tests for ingest service
* Policy list tests - turn it all off
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* update to eui 21.0.1
* most changes needed for search bar ts changes
* Passing types
* snapshots
* jest tests
* Removed IQuery placeholder types
* Updated functional test to only look at table headers with content
* Moved 'filters' definition around in api docs
* Update types
* update snapshot
* typo
* Move DATA_FRAME_TASK_STATE enum to its own file to fix x-pack functional test config imports
* merge public api 'changes'
Co-authored-by: patrykkopycinski <patryk.kopycinski@elastic.co>
Co-authored-by: Patryk Kopycinski <contact@patrykkopycinski.com>
Remove the getCytoscapeElements function.
On the server:
* Replace `source` with `sourceData`, `destination` with `targetData`, `source.id` with `source`, and `destination.id` with `target`.
* Return a single array as an `elements` property instead of `nodes` and `connections`
* Map all of the items data to be inside of a `data` object
* Replace SERVICE_AGENT_NAME with AGENT_NAME
* Add some missing constants
On the client:
* Remove getCytoscapeElements
* Move all presentation-specific data transformation to use the original attributes in the place where they're needed
* Remove `href` since it wasn't being used
* Move BetaBadge to its own file
* Move cytoscapeDivStyle to cytoscapeOptions
* Fix storybook to work with new data formats
* First pass
* First pass
* Add new routes
* Getting closer
* Remove legacy server code, and other fixes
* Register the plugin with xpack
* Pass a legacy client to telemetry
* Suport callWithInternalUser
* Remove this
* More NP work
* Fix some tests
* Fix broken test
* Move over new telemetry changes, and fix other issues
* Fix TODO item
* Reuse the same schema as elasticsearch module
* Use a singular config definition here
* Disable this for now
* Use the right method
* Use custom config again
* Tweak the config to make this optional
* Remove these
* Remove these unnecessary files
* Fix jest test
* Fix some linting issues
* Fix type issue
* Fix localization issues
* Use the elasticsearch config
* Remove todos
* Fix this check
* Move kibana alerting over
* PR feedback
* Use new metrics core service
* Change config for xpack_api_polling_frequency_millis
* Make sure this is disabled for now
* Disable both
* Update this to the new function
* Tighten up legacy api needs
* Check for existence
* Fix jest tests
* Cleaning up the plugin definition
* Create custom type in our plugin
* Revert this change
* Fix CI issues
* Add these tests back
* Just use a different collector type
* Handle errors better
* Use custom type
* PR feedback
* Fix type issues
* PR feedback
Enables the FT that tests the View inApp functionality.
It addresses an issue that causes a race condition on CI where the ViewInApp button was thought to be enabled when it was, in fact, still disabled.
This meant that the click on the button didn't trigger the handler which, in turn, made the test fail.
* Define minimum license required for each action type (#58668)
* Add minimum required license
* Require at least gold license as a minimum license required on third party action types
* Use strings for license references
* Ensure license type is valid
* Fix some tests
* Add servicenow to gold
* Add tests
* Set license requirements on other built in action types
* Use jest.Mocked<ActionType> instead
* Change servicenow to platinum
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Make actions config mock and license state mock use factory pattern and jest mocks (#59370)
* Add license checks to action HTTP APIs (#59153)
* Initial work
* Handle errors in update action API
* Add unit tests for APIs
* Make action executor throw when action type isn't enabled
* Add test suite for basic license
* Fix ESLint errors
* Fix failing tests
* Attempt 1 to fix CI
* ESLint fixes
* Create sendResponse function on ActionTypeDisabledError
* Make disabled action types by config return 403
* Remove switch case
* Fix ESLint
* Add license checks within alerting / actions framework (#59699)
* Initial work
* Handle errors in update action API
* Add unit tests for APIs
* Verify action type before scheduling action task
* Make actions plugin.execute throw error if action type is disabled
* Bug fixes
* Make action executor throw when action type isn't enabled
* Add test suite for basic license
* Fix ESLint errors
* Stop action task from re-running when license check fails
* Fix failing tests
* Attempt 1 to fix CI
* ESLint fixes
* Create sendResponse function on ActionTypeDisabledError
* Make disabled action types by config return 403
* Remove switch case
* Fix ESLint
* Fix confusing assertion
* Add comment explaining double mock
* Log warning when alert action isn't scheduled
* Disable action types in UI when license doesn't support it (#59819)
* Initial work
* Handle errors in update action API
* Add unit tests for APIs
* Verify action type before scheduling action task
* Make actions plugin.execute throw error if action type is disabled
* Bug fixes
* Make action executor throw when action type isn't enabled
* Add test suite for basic license
* Fix ESLint errors
* Stop action task from re-running when license check fails
* Fix failing tests
* Attempt 1 to fix CI
* ESLint fixes
* Return enabledInConfig and enabledInLicense from actions get types API
* Disable cards that have invalid license in create connector flyout
* Create sendResponse function on ActionTypeDisabledError
* Make disabled action types by config return 403
* Remove switch case
* Fix ESLint
* Disable when creating alert action
* Return minimumLicenseRequired in /types API
* Disable row in connectors when action type is disabled
* Fix failing jest test
* Some refactoring
* Card in edit alert flyout
* Sort action types by name
* Add tooltips to create connector action type selector
* Add tooltips to alert flyout action type selector
* Add get more actions link in alert flyout
* Add callout when creating a connector
* Typos
* remove float right and use flexgroup
* replace pixels with eui variables
* turn on sass lint for triggers_actions_ui dir
* trying to add padding around cards
* Add callout in edit alert screen when some actions are disabled
* improve card selection for Add Connector flyout
* Fix cards for create connector
* Add tests
* ESLint issue
* Cleanup
* Cleanup pt2
* Fix type check errors
* moving to 3-columns cards for connector selection
* Change re-enable to enable terminology
* Revert "Change re-enable to enable terminology"
This reverts commit b497dfd6b6.
* Add re-enable comment
* Remove unecessary fragment
* Add type to actionTypeNodes
* Fix EuiLink to not have opacity of 0.7 when not hovered
* design cleanup in progress
* updating classNames
* using EuiIconTip
* Remove label on icon tip
* Fix failing jest test
Co-authored-by: Andrea Del Rio <delrio.andre@gmail.com>
* Add index to .index action type test
* PR feedback
* Add isErrorThatHandlesItsOwnResponse
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Andrea Del Rio <delrio.andre@gmail.com>
* Implemented ability to clear and properly validate alert interval
* Fixed due to comments
* Fixed additional request for the last field
* Fixed failing test
* Layers dir up through sources migrated. Kibana services updated
* Create separate init method for plugin setup, leverage in embeddable factory
* Add NP timefilter, http, IndexPatternSelect
* Pull vis color utils into Maps
* Add NP dark mode and toast handling. Some fixes
* Init autocomplete and indexPattern via normal paths
* Test fixes and clean up
* Update index pattern and autocomplete refs. Make getters functions
* Fix remaining broken jest tests
* Update inspector start contract
* Clean up plugin and legacy files. Fix type issues
* Set inspector in plugin start method not external function
* Keep both injected var functions (legacy and NP). Move inspector init back to separate init function
* Add back ts-ignore on NP kibana services import
* update layout
* add utility bars
* add icon
* adding a route for exporting timeline
* organizing data
* fix types
* fix incorrect props for timeline table
* add export timeline to tables action
* fix types
* add client side unit test
* add server-side unit test
* fix title for delete timelines
* fix unit tests
* update snapshot
* fix dependency
* add table ref
* remove custom link
* remove custom links
* Update x-pack/legacy/plugins/siem/common/constants.ts
Co-Authored-By: Xavier Mouligneau <189600+XavierM@users.noreply.github.com>
* remove type ExportTimelineIds
* reduce props
* Get notes and pinned events by timeline id
* combine notes and pinned events data
* fix unit test
* fix type error
* fix type error
* fix unit tests
* fix for review
* clean up generic downloader
* review with angela
* review utils
* fix for code review
* fix for review
* fix tests
* review
* fix title of delete modal
* remove an extra bracket
Co-authored-by: Xavier Mouligneau <189600+XavierM@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Also move the hook use_ui_ace_keyboard_mode.tsx into es_ui_shared
This was defined (and used) in both Console and SearchProfiler.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* [canvas/shareable_runtime] sync sass loaders with kbn/optimizer
* limit sass options to those relevant in this context
Co-authored-by: spalger <spalger@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Fix updating of ML rules
* Add a regression test for updating ML Rules
* Allow ML Rules to be patched
And adds a regression unit test.
* Allow ML rule params to be imported when overwriting
* Add a basic regression test for creating a rule with ML params
* Prevent users from changing an existing Rule's type
* boundary test for first alert retrieval
* boundary test for first alert retrieval cleaned up
* redo merge conflict resolving for api test
* redo merge conflict resolving for api test try 2
* updating to current dataset expectations
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
- Fixes regression caused by elastic/elasticsearch#53572.
- Adjusts the TS mappings and code to reflect the newly returned API response.
- Re-enables functional tests.
* WIP trying things.
Add new alert type for Uptime.
Add defensive checks to alert executor.
Move status check code to dedicated adapter function.
Clean up code.
* Port adapter function to dedicated file.
* WIP.
* Working on parameter selection.
* Selector expressions working.
* Working on actions.
* Change anchor prop for popovers.
* Reference migrated alerting plugin.
* Clean up code for draft.
* Add button to expose flyout. Clean up some client code.
* Add test for requests function, add support for filters.
* Reorganize and clean up files.
* Add location and filter support to monitor status request function.
* Add tests for monitor status request function.
* Specify default action group id in alert registration.
* Extract repeated string value to a constant.
* Move test file to server in NP plugin.
* Update imports after NP migration.
* Fix UI bug that caused incorrect location selections in alert creation.
* Change alert expression language to clarify meaning.
* Add ability for user to select timerange units.
* Add code that fixes active item highlighting.
* Add better default value for active index selection.
* Introduce dedicated field number component.
* Add message to status check alert.
* Add tests for context message.
* Formalize alert action group definitions.
* Extract monitor id squashing from context message generator.
* Write test for monitor ID uniqueness function.
* Add alert state creator function and tests.
* Update action group id value.
* Add tests for alert factory and executor function.
* Rename alert context props to be more domain-specific.
* Clean up unnecessary type markup.
* Clean up alert ui controls file.
* Better organize new registration code.
* Simplify some logic code.
* Clean up bootstrap code.
* Add unit tests for alert type.
* Delete temporary test code from triggers_actions_ui.
* Rename a test file.
* Add some comments to annotate a file.
* Add io-ts type checking to alert create validation and alert executor.
* Add translation of plaintext content string.
* Further simplify monitor status alert validation.
* Add io-ts type checking to alert params.
* Update a comment.
* Prefer inline snapshots to more error-prone assertions.
* Clean up and comment request function.
* Rename a symbol.
* Fix broken types in reducer file and add a test.
* Fix a validation logic error and add tests.
* Delete unused import.
* Delete obsolete dependency.
* Fix function call to have correct parameters.
* Fixing some import weirdness.
* Reintroduce accidentally-deleted code.
* Delete unneeded require from legacy entry file.
* Remove unneeded connected component.
* Update flyout controls for new interface and delete connected components.
* Remove unneeded require from app index file.
* Introduce data-test-subj attributes to various components to assist with functional tests.
* Introduce functional test helpers for alert flyout.
* Add functional test arch and a test for alerting UI to ES SSL test suite.
* Add explicit exports to module index.
* Reorganize file to keep interfaces closer to their implementations.
* Move create alert button to better position.
* Clean up a file.
* Update a functional test attribute, clean up a file, rename a selector, add tests.
* Add a comment.
* Make better default alert message, translate messages, add/update tests.
* Fix broken type.
* Update obsolete snapshot.
* Introduce mock provider to tests and update snapshots.
* Reduce a strange type to `any`.
* Add alert flyout button connected component.
* Add alert flyout wrapper connected component.
* Create connected component for alert monitor status alert.
* Clean up index files.
* Update i18nrc file to cover translation in server plugin code.
* Fix broken imports.
* Update test snapshots.
* Prefer more descriptive type.
* Prefer more descriptive type.
* Prefer built-in React propType to custom.
* Prefer simpler validation.
* Add whitespace to clean up file.
* Extract function and write tests.
* Simplify validation function.
* Add navigate to alerting button.
* Move context item inside the items list.
* Clean up alert creation component.
* Update type check parsing and error messaging, and update snapshot/test assertions.
* Update broken snapshot.
* Update README for running functional tests.
* Update functional test service to reflect improved UX.
* Fix broken type that resulted from a mistake during a merge resolution.
* Add spacer between alert title and kuery bar.
* Update the id and name of our alert type because it was never changed from placeholder value.
* Rename alert keys.
* Fix broken unit tests.
* Add aria-labels to alert UI.
* Implement design feedback.
* Fix broken test snapshots.
* Add missing props to unit tests to staisfy updated types.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Clean up use of ace in autocomplete in public
Remove ace from lib/autocomplete.ts and set up hooking up of ace
in legacy_core_editor. Also remove use of ace mocks in tests.
* Added TODO in lib/kb (console public)
* Server-side cleanup
Refactored the loading of spec into a new SpecDefinitionsService.
In this way, state can be contained inside of the service as much
as possible. Also converted all JS spec to TS and updated the
Console plugin contract so that processors (which alter loaded
spec) happen at plugin "start" phase.
* Fix types
* Small refactor
- Updated naming of argument variable in registerAutocompleter
- Refactored the SpecDefinitionsService to handle binding of
it's own functions
* Public in WiP state, removed all 'ui/' imports
* First iteration of public shimmed and working
* A whole lotta WIP server side
* Server-side to using the NP router + client side changes
Updated the client code to properly encode requests to the
server. Did first E2E test.
Route tests are probably broken, need to fix them.
* Removed unused error wrapping code
* Update client Jest tests
* Add breadcrumbs service mock
* Fix server side Jest tests
* Add helper functions file for server side Jest tests
* Fix API integration tests
* Fixed boolean logic mistake in due to refactor in index mgmt ext.
Also migrated to the a more NP friendly version of index mgmt
extension.
* Remove unused import
* Clean up some cruft and refactor URL variable names
* Fix stringification of body and fix boolean server logic
* Fix mocha
Folder called __tests__ with Jest tests was breaking mocha.
* Refactor to Jest test
* Fix types issues in jest test
* Migrate to new config-schema API
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Adds Navigation APIs to Alerting.
Parts to this PR:
Adds a client side (Public) plugin to Alerting, including two APIs: registerNavigation & registerDefaultNavigation. These allow a plugin to register navigation handlers for any alerts which it is the consumer of- one for specific AlertTypes and one for a default handler for all AlertTypes created by the plugin.
The Alert Details page now uses these navigation handlers for the View In App button. If there's an AlertType specific handler it uses that, otherwise it uses a default one and if the consumer has not registered a handler - it remains disabled.
A generic Alerting Example plugin that demonstrates usage of these APIs including two AlertTypes - one that always fires, and another that checks how many people are in Outer Space and allows you to trigger based on that. 😉 To enable the plugin run yarn start --ssl --run-examples
* Add async search strategy
* Add async search
* Fix async strategy and add tests
* Move types to separate file
* Revert changes to demo search
* Update demo search strategy to use async
* Add async es search strategy
* Return response as rawResponse
* Poll after initial request
* Add cancellation to search strategies
* Add tests
* Simplify async search strategy
* Move loadingCount to search strategy
* Update abort controller library
* Bootstrap
* Abort when the request is aborted
* Add utility and update value suggestions route
* Fix bad merge conflict
* Update tests
* Move to data_enhanced plugin
* Remove bad merge
* Revert switching abort controller libraries
* Revert package.json in lib
* Move to previous abort controller
* Add support for frozen indices
* Fix test to use fake timers to run debounced handlers
* Revert changes to example plugin
* Fix loading bar not going away when cancelling
* Call getSearchStrategy instead of passing directly
* Add async demo search strategy
* Fix error with setting state
* Update how aborting works
* Fix type checks
* Add test for loading count
* Attempt to fix broken example test
* Revert changes to test
* Fix test
* Update name to camelCase
* Fix failing test
* Don't require data_enhanced in example plugin
* Actually send DELETE request
* Use waitForCompletion parameter
* Use default search params
* Add support for rollups
* Only make changes needed for frozen indices/rollups
* Only make changes needed for frozen indices/rollups
* Add back in async functionality
* Fix tests/types
* Fix issue with sending empty body in GET
* Don't include skipped in loaded/total
* Don't wait before polling the next time
* Add search interceptor for bulk managing searches
* Simplify search logic
* Fix merge error
* Review feedback
* Add service for running beyond timeout
* Refactor abort utils
* Remove unneeded changes
* Add tests
* cleanup mocks
* Update src/legacy/core_plugins/kibana/public/dashboard/np_ready/dashboard_app.html
Co-Authored-By: Lukas Olson <olson.lukas@gmail.com>
Co-authored-by: Lukas Olson <olson.lukas@gmail.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
This PR adds basic functional tests for the file data visualizer, covering a file import and error messages for non-log files. It also moves the file input path handling to a common location in order to avoid code duplication.
## Summary
* https://github.com/elastic/kibana/issues/60022
* Adds the feature flag for simple list values
* Adds the boolean filters of "and", "and not" to further filter based on simple values
* Adds unit tests and e2e tests for the values.
* Most tests can include the simple list values but some have to be skipped until we move those to more functions or just enable simple list values as a permanent feature.
* DOES NOT FILTER ON THE VALUES JUST YET (That will be a follow on PR)
## Testing:
To turn on/off the feature flag do this with an env variable (set this in your .bashrc/.zshrc):
```ts
export ELASTIC_XPACK_SIEM_LISTS_FEATURE=true
```
Expect to see this error in the console when the environment variable is set:
```ts
server log [11:41:16.245] [error][plugins][siem] You have activated the lists feature flag which is NOT currently supported for SIEM! You should turn this feature flag off immediately by un-setting the environment variable: ELASTIC_XPACK_SIEM_LISTS_FEATURE and restarting Kibana
```
Expect create and update to work when the environment variable is set and look like this:
```ts
./update_rule.sh ./rules/updates/update_list.json
{
"created_at": "2020-03-15T17:42:37.074Z",
"updated_at": "2020-03-15T17:54:22.427Z",
"created_by": "yo",
"description": "Query with a list",
"enabled": true,
"false_positives": [],
"from": "now-6m",
"id": "c602e3f6-713b-4f43-9bdd-b60fbfead1c5",
"immutable": false,
"interval": "5m",
"rule_id": "query-with-list",
"language": "kuery",
"output_index": ".siem-signals-hassanabad-frank-default",
"max_signals": 100,
"risk_score": 1,
"name": "Query with a list",
"query": "user.name: root or user.name: admin",
"references": [],
"severity": "high",
"updated_by": "yo",
"tags": [],
"to": "now",
"type": "query",
"threat": [],
"version": 6,
"lists": [
{
"field": "source.ip",
"boolean_operator": "and",
"values": [
{
"name": "127.0.0.1",
"type": "value"
}
]
},
{
"field": "host.name",
"boolean_operator": "and not",
"values": [
{
"name": "rock01",
"type": "value"
}
]
}
],
"status": "succeeded",
"status_date": "2020-03-15T17:42:40.718Z",
"last_success_at": "2020-03-15T17:42:40.718Z",
"last_success_message": "succeeded"
}
```
```ts
./post_rule.sh ./rules/queries/query_with_list.json
{
"created_at": "2020-03-15T17:42:37.074Z",
"updated_at": "2020-03-15T17:42:37.116Z",
"created_by": "yo",
"description": "Query with a list",
"enabled": true,
"false_positives": [],
"from": "now-6m",
"id": "c602e3f6-713b-4f43-9bdd-b60fbfead1c5",
"immutable": false,
"interval": "5m",
"rule_id": "query-with-list",
"language": "kuery",
"output_index": ".siem-signals-hassanabad-frank-default",
"max_signals": 100,
"risk_score": 1,
"name": "Query with a list",
"query": "user.name: root or user.name: admin",
"references": [],
"severity": "high",
"updated_by": "yo",
"tags": [],
"to": "now",
"type": "query",
"threat": [],
"version": 1,
"lists": [
{
"field": "source.ip",
"boolean_operator": "and",
"values": [
{
"name": "127.0.0.1",
"type": "value"
}
]
},
{
"field": "host.name",
"boolean_operator": "and not",
"values": [
{
"name": "rock01",
"type": "value"
},
{
"name": "mothra",
"type": "value"
}
]
}
]
}
```
```ts
./patch_rule.sh ./rules/patches/update_list.json
{
"created_at": "2020-03-15T18:02:52.434Z",
"updated_at": "2020-03-15T18:02:57.675Z",
"created_by": "yo",
"description": "Query with a list",
"enabled": true,
"false_positives": [],
"from": "now-6m",
"id": "40b7c2fb-83b4-4820-bf7c-056f3a631126",
"immutable": false,
"interval": "5m",
"rule_id": "query-with-list",
"language": "kuery",
"output_index": ".siem-signals-hassanabad-frank-default",
"max_signals": 100,
"risk_score": 1,
"name": "Query with a list",
"query": "user.name: root or user.name: admin",
"references": [],
"severity": "high",
"updated_by": "yo",
"tags": [],
"to": "now",
"type": "query",
"threat": [],
"version": 1,
"lists": [
{
"field": "source.ip",
"boolean_operator": "and",
"values": [
{
"name": "127.0.0.1",
"type": "value"
}
]
},
{
"field": "host.name",
"boolean_operator": "and not",
"values": [
{
"name": "rock01",
"type": "value"
},
{
"name": "mothra",
"type": "value"
}
]
}
],
"status": "succeeded",
"status_date": "2020-03-15T18:02:56.426Z",
"last_success_at": "2020-03-15T18:02:56.426Z",
"last_success_message": "succeeded"
}
```
```ts
./get_rule_by_rule_id.sh query-with-list
{
"created_at": "2020-03-15T18:10:07.657Z",
"updated_at": "2020-03-15T18:10:08.479Z",
"created_by": "yo",
"description": "Query with a list",
"enabled": true,
"false_positives": [],
"from": "now-6m",
"id": "9854162b-003c-47be-af59-8c3c9545aafa",
"immutable": false,
"interval": "5m",
"rule_id": "query-with-list",
"language": "kuery",
"output_index": ".siem-signals-hassanabad-frank-default",
"max_signals": 100,
"risk_score": 1,
"name": "Query with a list",
"query": "user.name: root or user.name: admin",
"references": [],
"severity": "high",
"updated_by": "yo",
"tags": [],
"to": "now",
"type": "query",
"threat": [],
"version": 1,
"lists": [
{
"field": "source.ip",
"boolean_operator": "and",
"values": [
{
"name": "127.0.0.1",
"type": "value"
}
]
},
{
"field": "host.name",
"boolean_operator": "and not",
"values": [
{
"name": "rock01",
"type": "value"
},
{
"name": "mothra",
"type": "value"
}
]
}
],
"status": "going to run",
"status_date": "2020-03-15T18:10:10.738Z"
}
```
Expect these errors when the environment variable is not set:
```ts
./post_rule.sh ./rules/queries/query_with_list.json
{
"statusCode": 400,
"error": "Bad Request",
"message": "[request body]: child \"lists\" fails because [\"lists\" is not allowed]"
}
```
```ts
./update_rule.sh ./rules/queries/query_with_list.json
{
"statusCode": 400,
"error": "Bad Request",
"message": "[request body]: child \"lists\" fails because [\"lists\" is not allowed]"
}
```
```ts
./patch_rule.sh ./rules/patches/update_list.json
{
"statusCode": 400,
"error": "Bad Request",
"message": "[request body]: child \"lists\" fails because [\"lists\" is not allowed]"
}
```
Expect that this is _backwards_ compatible with the feature flag but not necessarily _forwards_ compatible. This means:
* You can have older data that never had lists and it will show up as an empty list when you query it. (backwards compatible)
* You _might_ have lists and remove the env. variable and get back items as if the list was not there for (forwards compatible)
* You can export without lists, flip on the env flag and import with newer lists feature (backwards compatible)
* You can export lists and it will _not_ work with an older system (not forwards compatible)
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
* Initial Commit
Update to ESDocs datasource per team feedback
* Updates
Updates per Ryan's mockups
* Updates II
Updates per Poff's review
* Updates III
Update to some of the verbiage and card sizes - working on re-ordering and adding a link to the lucen query syntax
* design tweaks
* Adding lucene hyperlink
update to add hyperlink help for Lucene query syntax
* Consollidating datasources to sort
Consolidating the ESDocs datasource with the rest, so that we can order them
* updates for i18n
updates for i18n
* Updates
Updates from Gail for verbiage and integrating Ryan's change for style
* Update ui.ts
Updates for i18n
* Updates for datasource order
moving the esdocs datasource to live with the rest of the UI datasources, and sorting them accordingly.
* Update datasource_component.js
removing console log, whoops
* Update ui.ts
Update to fix i18n essql issue
* Update ui.ts
Updates to fix i18n references for the esdocs datasource move
* Update to Timelion URL
I noticed that the Timelion datasource showed "Lucene query syntax" which wasn't relevant, so I updated it to "Timelion", along with a tutorial, as the link for current Timelion docs does not provide any syntax tutorial.
* Update ui.ts
update for i18n
* Update ui.ts
update for i18n
* Update ui.ts
Update to removed unused value - the i18n check gave me latent errors, sorry for the repost
* i18n updates
Updating nomenclature to get past i18n errors
* Updates
Code review updates to remove extraneous code
* Update timelion.js
update to remove extraneous comment per code review
* More i18n updates
translation updates to accommodate the esdocs datasource move
* Update datasource_component.js
Update to toggle datasource icon in selected element mode
* Update ui.ts
hopefully last i18n fix
Co-authored-by: Ryan Keairns <contactryank@gmail.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* refactor `use_details_uri` hook and introduce `useAgentConfigLink`
* Refactor structure for datasources view
* Sync up table columns
* Added row actions to Datasources list
* Datasources table filters
* Support deleting datasource action
* Added PackageIcon to datasources list
* Remove unnecessary linter exceptions
Not sure what was causing issues here, but it's gone now.
* WIP: Simple form to test creation of ML rules
This will be integrated into the regular rule creation workflow, but for
now this simple form should allow us to exercise the full ML rule
workflow.
* WIP: Adds POST to backend, and type/payload changes necessary to make that work
* Simplify logic with Math.min
* WIP: Failed spike of making an http call
* WIP: Hacking together an ML client
The rest of this is going to be easier if I have actual data. For now
this is mostly copy/pasted and simplified ML code. I've hardcoded time
ranges to a period I know has data for a particular job.
* Threading through our new ML Rule params
It's a bummer that we normalize our rule alert params across all rule
types currently, but that's the deal.
* Retrieve our anomalies during rule execution
Next step: generate signals
* WIP: Generate ECS-compatible ML Signals
This uses as much of the existing signal-creation code as possible. I
skipped the search_after stuff for now because it would require us
recreating the anomalies query which we really shouldn't own. For now,
here's how it works:
* Adds a separate branch of the rule executor for machine_learning rules
* In that branch, we call our new bulkCreateMlSignal function
* This function first transforms the anomaly document into ECS fields
* We then pass the transformed documents to singleBulkCreate, which
does the rest
* After both branches, we update the rule's status appropriately.
We need to do some more work on the anomaly transformation, but this
works!
* Extract setting of rule failure to helper function
We were doing this identically in three places.
* Remove unused import
* Define a field for our Rule Type selection
This adds most of the markup and logic to allow an ML rule type to be
selected. We still need to add things like license-checking and
showing/hiding of fields based on type.
* Hide Query Fields when ML is selected
These are still getting set on the form. We'll need to filter these
fields before we send off the data, and not show them on the readonly
display either.
ALso, edit is majorly broken.
* Add input field for anomaly threshold
* Display numberic values in the readonly view of a step
TIL that isEmpty returns false for numbers and other non-iterable
values. I don't think it's exactly what we want here, but until I figure
out the intention this gets our anomalyThreshold showing up without a
separate logic branch here. Removes the unnecessary branch that was
redundant with the 'else' clause.
* Add field for selecting an ML job
This is not the same as the mockups and lacks some functionality, but
it'll allow us to select a job for now.
* Format our new ML Fields when sending them to the server
So that we don't get rejected due to snake case vs camelcase.
* Put back code that respects a rule's schedule
It was previously hardcoded to a time period I knew had anomalies.
* ML fields are optional in our creation step
In that we don't initialize them like we do the query (default) fields.
* Only send along type-specific Rule fields from form
This makes any query- or ML-specific fields optional on a Rule, and
performs some logic on the frontend to group and include these fieldsets
conditionally based on the user's selection. The one place we don't
handle this well is on the readonly view of a completed step in the
rules creation, but we'll address that.
* Rename anomalies query
It's no longer tabular data. If we need that, we can use the ML client.
* Remove spike page with simple form
* Remove unneeded ES option
This response isn't going to HTTP, which is where this option would
matter.
* Fix bulk create logic
I made a happy accident and flipped the logic here, which meant we
weren't capping the signals we created.
* Rename argument
Value is a little more ambiguous than data, here: this is our step data.
* Create Rule form stores all values, but filters by type for use
When sending off to the backend, or displaying on the readonly view, we
inspect which rule type we've currently selected, and filter our form
values appropriately.
* Fix editing of ML fields on Rule Create
We need to inherit the field value from our form on initial render, and
everything works as expected.
* Clear form errors when switching between rule types
Validation errors prevent us from moving to the next step, so it was
previously possible to get an error for Query fields, switch to an ML
rule, and be unable to continue because the form had Query errors.
This also adds a helper for checking whether a ruleType is ML, to
prevent having to change all these references if the type string
changes.
* Validate the selection of an ML Job
* Fix type errors on frontend
According to the types, this is essentially the opposite of formatRule,
so we need to reinflate all potential form values from the rule.
* Don't set defaults for query-specific rules
For ML rules these types should not be included.
* Return ML Fields in Rule responses
This adds these fields to our rule serialization, and then adds
conditional validation around those fields if the rule type is ML.
Conversely, we moved the 'language' and 'query' fields to be
conditionally validated if the rule is a query/saved_query rule.
* Fix editing of ML rules by changing who controls the field values
The source of truth for their state is the parent form object; these
inputs should not have local state.
* Fix type errors related to new ML fields
In adding the new ML fields, some other fields (e.g. `query` and
`index`) that were previously required but implicitly part of Query
Rules are now marked as optional.
Consequently, any downstream code that actually required these fields
started to complain. In general, the fix was to verify that those fields
exist, and throw an error otherwise as to appease the linter.
Runtime-wise, the new ML rules/signals follow a separate code path and
both branches should be unaffected by these changes; the issue is simply
that our conditional types don't work well with Typescript.
* Fix failing route tests
Error message changed.
* Fix integration tests
We were not sending required properties when creating a rule(index and
language).
* Fix non-ML Rule creation
I was accidentally dropping this parameter for our POST payload. Whoops.
* More informative logging during ML signal generation
The messaging diverged from the normal path here because we don't have
index patterns to display. However, we have the rest of the rule
context, and should report it appropriately.
* Prefer keyof for string union types
* Tidy up our new form components
* Type them as React.FCs
* Remove unnecessary use of styled-components
* Prefer destructuring to lodash's omit
* Fix mock params for helper functions
These were updated to take simpler parameters.
* Remove any type
This could have been a boolean all along, whoops
* Fix mock types
* Update outdated tests
These were added on master, but behavior has been changed on my branch.
* Add some tests around our helper function
I need to refactor it, so this is as good a time as any to pin down the
behavior.
* Remove uses of any in favor of actual types
Mainly leverages ML typings instead of our placeholder types. This
required handling a null case in our formatting of anomalies.
* Annotate our anomalies with @timestamp field
We were notably lacking this ECS field in our post-conversion anomalies,
and typescript was rightly complaining about it.
* ml_job_id -> machine_learning_job_id
* PR Feedback
* Stricter threshold type
* More robust date parsing
* More informative log/error messages
* Remove redundant runtime checks
* Cleaning up our new ML types
* Fix types on our Rest types
* Use less ambiguous machineLearningJobId over mlJobId
* Declare our ML params as required keys, and ensure we pass them around
everywhere we might need them (creating, importing, updating rules).
* Use implicit type to avoid the need for a ts-ignore
FormSchema has a very generic index signature such that our
filterRuleFieldsForType helper cannot infer that it has our necessary
rule fields (when in fact it does). By removing the FormSchema hint we
get the actual keys of our schema, and things work as expected.
All other uses of schema continue to work because they're expecting
FormSchema, which is effectively { [key: string]: any }.
* New ML params are not nullable
Rather than setting a null and then never using it, let's just make it
truly optional in terms of default values.
* Query and language are conditional based on rule type
For ML Rules, we don't use them.
* Remove defaulted parameter in API test
We don't need to specify this, and we should continue not to for
backwards compatibility.
* Use explicit types over implicit ones
The concern is that not typing our schemae as FormSchema could break our
form if there are upstream changes. For now, we simply use the
intersection of FormSchema and our generic parameter to satisfy our use
within the function.
* Add integration test for creation of ML Rule
* Add ML fields to route schemae
* threshold and job id are conditional on type
* makes query and language mutually exclusive with above
* Fix router test for creating an ML rule
We were sending invalid parameters.
* Remove null check against index for query rules
We support not having an index here, as getInputIndex will return the
current UI setting if none is specified.
* Add regression test for API compatibility
We were previously able to create a rule without an input index; we
should continue to support that, as verified by this test!
* Respect the index pattern determined at runtime when performing search_after
If a rule does not specify an input index pattern on creation, we use
the current UI default when the rule is evaluated. This ensures that any
subsequent searches use that same index.
We're not currently persisting that runtime index to the generated
signal, but we should.
* Fix type errors in our bulk create tests
We added a new argument, but didn't update the tests.
* Fixed default message for index threshold includes both threshold values even if not used
* fixed due to review comments
* Fixed validation errors with ability to clear input
* Support yaml var type:
* Change stream config model to save type and value, instead of just value
* Add code editor for configuring yaml vars
* Adjust tests
* Account for empty yaml value
* Better account for invalid yaml parsing
* Apply action types to fields
* Add information to each field
* Do not create or update comments when actionType is set to nothing
* Improve helpers tests
* Improve tests
* Refactor: Use transformers and pipes
* Better types
* Refactor tests to new changes
* Better error messages
* Improve field formatting and display
* Improve integration tests
* Make username mandatory field
* Translate transformers
* Refactor schema
* Translate appendInformationToField helper
* Improve intergration tests
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* [Maps] Blended layer that switches between documents and clusters
* change layer type when scalingType changes
* getSource
* use cluster source when count exceeds value
* ensure doc source stays in editor
* start creating cluster style
* pass all parts of style descriptor
* get toggling between sources working
* derive cluster style from document style
* remove references to METRIC_TYPE
* fix import
* start typescripting blended_vector_layer
* more typescript work
* last of the TS errors
* add migration to convert useTopTerm to scalingType
* clean up
* remove MapSavedObject work since its in a seperate PR now
* fix EsSearchSource update editor jest test
* fix map_selector jest test
* move mutable state out of BlendedVectorLayer
* one more change for removing mutable BlendedVectorLayer state
* integrate newly merged MapSavedObjectAttributes type
* review feedback
* use data request for fetching feature count
* add functional test
* fix functional test
* review feedback
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Unifying the test index name for resolver and alerts
* Endpoint isn't sending the agent field so check for it
* Update resolver to use either legacy or ecs events
* Use correct format for child events api
* Adding string or array for category and type
* Add return types to process event models
* Create a common/models.ts for common event logic
* Decrease resolver min height
* Update types to match cli tool
* Add a smoke test for resolver rendering nodes, remove unused selector
* Add common/models/event
* Internationalize some strings, address pr comments
Co-authored-by: Jonathan Buttner <jonathan.buttner@elastic.co>
* Add properties needed for index templates to Field
* Add data type handling to template generation
* Adjust tests
* Update fields test snapshots
* Remove duplicate fields from test file
* Add test cases
* Enhance processFields
* move expand stage to expandFields
* fix expandFields
* add deduplication stage dedupFields
* Use processField() to preprocess fields
* Remove alias fields with invalid path
* Remove obsolete code.
* Fix documentation.
* Add unit tests for getField()
* Don't fail on invalid input for now.
* Validate array fields.
* Guard against invalid input.
