kibana/x-pack/plugins
Frank Hassanabad 28738e6b4b
[Security Solution] Fixes CIDR, float, long, integer, array, and text based issues when using value lists in exceptions (#85191)
## Summary

Fixes different bugs/issues when using exceptions with value based lists for both the UI, the backend, and the large value based lists. See https://github.com/elastic/kibana/issues/79516, but this also fixes several other bugs found mentioned below.

For the front end UI:
* Adds the ability to specify value based lists that are IP Ranges when the source event is an IP. Before you could only match IP to IP and the IP Ranges lists could not be used. 
* Breaks down a few functions into smaller functions for unit test writing abilities.

You can now add ip ranges as list values for the UI when before it would not show up:
<img width="1035" alt="Screen Shot 2020-12-07 at 2 15 39 PM" src="https://user-images.githubusercontent.com/1151048/101406552-d6819b00-3896-11eb-9fb5-4c7c2ad93b2e.png">

For value based lists:
* Fixes text data type to use "and" between matching using `operator: 'and'` and changes it from a `terms query to a `match` query
* Adds new API for searching against types called `searchListItemByValues ` so that numeric, text, array based, and other non-stringable types can be sent and then the value based lists will push that to ElasticSearch. This shifts as many corner cases and string/numeric coercions to ElasticSearch rather than Kibana client side code.
* Adds ability to handle arrays within arrays through a `flatten` call.
* Utilizes the `named queries` from ElasticSearch for the new API so that clients can get which parts matched and then use that for their exception list logic rather than in-memory string to string checks. This fixes CIDR and ranges as well as works with arrays.

For the backend exception lists that used value based lists:
* Broke down the `filterEventsAgainstList` function into a folder called `filters` and the functions into other files for better unit based testing.
* Changed the calls from `getListItemByValues` to `searchListItemByValues` which can return exactly what it matched against and this will not break anyone using the existing REST API for `getListItemByValues` since that REST API and client side API stays the same.
* Cleaned up extra promises being used in a few spots that async/await automatically will create. 
* Removed the stringabilities and stringify in favor of just a simpler exact check using `JSON.stringify()`

For the tests:
* Adds unit tests to broken down functions
* Adds ip_array, keyword_array, text_array, FTR tests for the backend.
* Adds more CIDR and range based FTR tests for the backend.
* Unskips and fixes all the numeric tests and range tests that could not operate previously from bugs.

### Checklist

Delete any items that are not applicable to this PR.

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
- [ ] Any UI touched in this PR is usable by keyboard only (learn more about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures (run axe in browser: [FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/), [Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] This renders correctly on smaller devices using a responsive layout. (You can test this [in your browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
2020-12-10 18:07:47 -07:00
..
actions [Actions] Fix external service urls (#85556) 2020-12-10 23:24:53 +02:00
alerting_builtins Jest multi-project configuration (#77894) 2020-12-02 11:42:23 -08:00
alerts [Actions] Notify only on action group change (#82969) 2020-12-10 15:51:52 -05:00
apm [APM] add sanitize_field_names & transaction_ignore_urls vars to Node.js agent remote config (#85655) 2020-12-10 16:41:48 -08:00
audit_trail Jest multi-project configuration (#77894) 2020-12-02 11:42:23 -08:00
beats_management Jest multi-project configuration (#77894) 2020-12-02 11:42:23 -08:00
canvas [Reporting/PDF] Layout option for generating full-page Canvas reports (#84959) 2020-12-10 13:34:47 -06:00
case [Security Solution][Case] Add in-progress status to case (#84321) 2020-12-04 21:36:23 +02:00
cloud Jest multi-project configuration (#77894) 2020-12-02 11:42:23 -08:00
code Jest multi-project configuration (#77894) 2020-12-02 11:42:23 -08:00
console_extensions [ML] Update console autocomplete for ML data frame evaluate API (#83151) 2020-11-17 12:48:25 +02:00
cross_cluster_replication [Telemetry] Introduce UI Counters (#84224) 2020-12-04 17:47:04 +02:00
dashboard_enhanced Jest multi-project configuration (#77894) 2020-12-02 11:42:23 -08:00
dashboard_mode Jest multi-project configuration (#77894) 2020-12-02 11:42:23 -08:00
data_enhanced [Search] Session SO polling (#84225) 2020-12-09 14:05:01 +02:00
discover_enhanced Jest multi-project configuration (#77894) 2020-12-02 11:42:23 -08:00
drilldowns Jest multi-project configuration (#77894) 2020-12-02 11:42:23 -08:00
embeddable_enhanced Jest multi-project configuration (#77894) 2020-12-02 11:42:23 -08:00
encrypted_saved_objects ECS audit events for alerting (#84113) 2020-12-04 19:13:30 +00:00
enterprise_search [Workplace Search] Polish Workplace Search Sources & Groups UI (#85071) 2020-12-08 15:30:41 -06:00
event_log [Alerting] Introduces a ActionSubGroup which allows for more granular action group scheduling (#84751) 2020-12-10 15:16:42 +00:00
features Jest multi-project configuration (#77894) 2020-12-02 11:42:23 -08:00
file_upload Jest multi-project configuration (#77894) 2020-12-02 11:42:23 -08:00
fleet [Security Solution][Endpoint][Admin] Api guard for paid policy features (#84667) 2020-12-10 18:53:55 -05:00
global_search [GS] add tag and dashboard suggestion results (#85144) 2020-12-09 11:05:59 +01:00
global_search_bar [GS] adding tags UI to search results (#85084) 2020-12-10 11:16:21 -06:00
global_search_providers [GS] adding tags UI to search results (#85084) 2020-12-10 11:16:21 -06:00
graph [Graph] Fix graph saved object references (#85295) 2020-12-10 09:36:06 +01:00
grokdebugger [Grokdebugger] Fix simulate error handling (#83036) 2020-11-11 15:43:17 +01:00
index_lifecycle_management fix serialization of rollover (#85582) 2020-12-10 17:53:01 +01:00
index_management Integrate painless autocomplete in runtime fields editor (#84943) 2020-12-07 12:55:53 -05:00
infra [Metrics UI] Node details design feedback (#85433) 2020-12-10 13:38:17 -06:00
ingest_manager Jest multi-project configuration (#77894) 2020-12-02 11:42:23 -08:00
ingest_pipelines Added accessibility test for ingest node pipelines (#84983) 2020-12-10 15:33:58 -05:00
lens Lens save modal should conditionally save to library (#85568) 2020-12-10 09:55:52 -06:00
license_management Upgrade EUI to v30.5.1 (#84677) 2020-12-04 09:39:03 -07:00
licensing Jest multi-project configuration (#77894) 2020-12-02 11:42:23 -08:00
lists [Security Solution] Fixes CIDR, float, long, integer, array, and text based issues when using value lists in exceptions (#85191) 2020-12-10 18:07:47 -07:00
logstash Jest multi-project configuration (#77894) 2020-12-02 11:42:23 -08:00
maps [Maps] fix unlinking an embedded map by reference Causes Error (#85485) 2020-12-10 08:31:28 -07:00
maps_legacy_licensing
ml [ML] Only show _doc_count for anomaly detection job summary count field (#85444) 2020-12-10 22:30:39 +00:00
monitoring [Actions] Notify only on action group change (#82969) 2020-12-10 15:51:52 -05:00
observability [Observability][Metrics UI] Refactor Observability Overview for Performance (#84955) 2020-12-10 12:35:27 -07:00
painless_lab Jest multi-project configuration (#77894) 2020-12-02 11:42:23 -08:00
remote_clusters [Telemetry] Introduce UI Counters (#84224) 2020-12-04 17:47:04 +02:00
reporting [Reporting/PDF] Layout option for generating full-page Canvas reports (#84959) 2020-12-10 13:34:47 -06:00
rollup Remove 'minute' frequency option from SLM policy form because ES won't allow a frequency faster than every 15 minutes. (#84854) 2020-12-09 09:14:57 -08:00
runtime_fields Add help text for runtime fields source. (#85204) 2020-12-08 14:51:47 -05:00
saved_objects_tagging [GS] adding tags UI to search results (#85084) 2020-12-10 11:16:21 -06:00
searchprofiler Jest multi-project configuration (#77894) 2020-12-02 11:42:23 -08:00
security Require gold license for ECS audit logging (#85537) 2020-12-10 16:34:26 +00:00
security_solution [Security Solution] Fixes CIDR, float, long, integer, array, and text based issues when using value lists in exceptions (#85191) 2020-12-10 18:07:47 -07:00
snapshot_restore Remove 'minute' frequency option from SLM policy form because ES won't allow a frequency faster than every 15 minutes. (#84854) 2020-12-09 09:14:57 -08:00
spaces [jest] fix errors and warnings (#85291) 2020-12-09 15:04:21 +01:00
stack_alerts Add test handling for alert instantiation (#85632) 2020-12-10 16:25:55 -07:00
task_manager Jest multi-project configuration (#77894) 2020-12-02 11:42:23 -08:00
telemetry_collection_xpack Add bulk assign action to tag management (#84177) 2020-12-07 11:18:43 +01:00
transform [Transform] Replace legacy elasticsearch client (#84932) 2020-12-09 12:55:54 +01:00
translations Deangularize Dashboard (#82909) 2020-12-10 16:46:50 -05:00
triggers_actions_ui [Actions] Notify only on action group change (#82969) 2020-12-10 15:51:52 -05:00
ui_actions_enhanced [jest] fix errors and warnings (#85291) 2020-12-09 15:04:21 +01:00
upgrade_assistant Jest multi-project configuration (#77894) 2020-12-02 11:42:23 -08:00
uptime Apply terminateAfter params in index status API (#85432) 2020-12-10 21:42:49 +01:00
vis_type_timeseries_enhanced Jest multi-project configuration (#77894) 2020-12-02 11:42:23 -08:00
watcher Jest multi-project configuration (#77894) 2020-12-02 11:42:23 -08:00
xpack_legacy Jest multi-project configuration (#77894) 2020-12-02 11:42:23 -08:00