## Summary Fixes different bugs/issues when using exceptions with value based lists for both the UI, the backend, and the large value based lists. See https://github.com/elastic/kibana/issues/79516, but this also fixes several other bugs found mentioned below. For the front end UI: * Adds the ability to specify value based lists that are IP Ranges when the source event is an IP. Before you could only match IP to IP and the IP Ranges lists could not be used. * Breaks down a few functions into smaller functions for unit test writing abilities. You can now add ip ranges as list values for the UI when before it would not show up: <img width="1035" alt="Screen Shot 2020-12-07 at 2 15 39 PM" src="https://user-images.githubusercontent.com/1151048/101406552-d6819b00-3896-11eb-9fb5-4c7c2ad93b2e.png"> For value based lists: * Fixes text data type to use "and" between matching using `operator: 'and'` and changes it from a `terms query to a `match` query * Adds new API for searching against types called `searchListItemByValues ` so that numeric, text, array based, and other non-stringable types can be sent and then the value based lists will push that to ElasticSearch. This shifts as many corner cases and string/numeric coercions to ElasticSearch rather than Kibana client side code. * Adds ability to handle arrays within arrays through a `flatten` call. * Utilizes the `named queries` from ElasticSearch for the new API so that clients can get which parts matched and then use that for their exception list logic rather than in-memory string to string checks. This fixes CIDR and ranges as well as works with arrays. For the backend exception lists that used value based lists: * Broke down the `filterEventsAgainstList` function into a folder called `filters` and the functions into other files for better unit based testing. * Changed the calls from `getListItemByValues` to `searchListItemByValues` which can return exactly what it matched against and this will not break anyone using the existing REST API for `getListItemByValues` since that REST API and client side API stays the same. * Cleaned up extra promises being used in a few spots that async/await automatically will create. * Removed the stringabilities and stringify in favor of just a simpler exact check using `JSON.stringify()` For the tests: * Adds unit tests to broken down functions * Adds ip_array, keyword_array, text_array, FTR tests for the backend. * Adds more CIDR and range based FTR tests for the backend. * Unskips and fixes all the numeric tests and range tests that could not operate previously from bugs. ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] Any UI touched in this PR is usable by keyboard only (learn more about [keyboard accessibility](https://webaim.org/techniques/keyboard/)) - [ ] Any UI touched in this PR does not create any new axe failures (run axe in browser: [FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/), [Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US)) - [ ] This renders correctly on smaller devices using a responsive layout. (You can test this [in your browser](https://www.browserstack.com/guide/responsive-testing-on-local-server)) - [ ] This was checked for [cross-browser compatibility](https://www.elastic.co/support/matrix#matrix_browsers) |
||
---|---|---|
.. | ||
common | ||
public | ||
server | ||
jest.config.js | ||
kibana.json | ||
README.md |
README.md for developers working on the backend lists on how to get started using the CURL scripts in the scripts folder.
The scripts rely on CURL and jq:
Install curl and jq (mac instructions)
brew update
brew install curl
brew install jq
Open $HOME/.zshrc
or ${HOME}.bashrc
depending on your SHELL output from echo $SHELL
and add these environment variables:
export ELASTICSEARCH_USERNAME=${user}
export ELASTICSEARCH_PASSWORD=${password}
export ELASTICSEARCH_URL=https://${ip}:9200
export KIBANA_URL=http://localhost:5601
export TASK_MANAGER_INDEX=.kibana-task-manager-${your user id}
export KIBANA_INDEX=.kibana-${your user id}
source $HOME/.zshrc
or ${HOME}.bashrc
to ensure variables are set:
source ~/.zshrc
Open your kibana.dev.yml
file and add these lines with your name:
xpack.lists.listIndex: '.lists-your-name'
xpack.lists.listItemIndex: '.items-your-name'
Restart Kibana and ensure that you are using --no-base-path
as changing the base path is a feature but will
get in the way of the CURL scripts written as is.
Go to the scripts folder cd kibana/x-pack/plugins/lists/server/scripts
and run:
./hard_reset.sh
./post_list.sh
which will:
- Delete any existing lists you have
- Delete any existing list items you have
- Delete any existing exception lists you have
- Delete any existing exception list items you have
- Delete any existing mapping, policies, and templates, you might have previously had.
- Add the latest list and list item index and its mappings using your settings from
kibana.dev.yml
environment variable ofxpack.lists.listIndex
andxpack.lists.listItemIndex
. - Posts the sample list from
./lists/new/ip_list.json
Now you can run
./post_list.sh
You should see the new list created like so:
{
"id": "ip_list",
"created_at": "2020-05-28T19:15:22.344Z",
"created_by": "yo",
"description": "This list describes bad internet ip",
"name": "Simple list with an ip",
"tie_breaker_id": "c57efbc4-4977-4a32-995f-cfd296bed521",
"type": "ip",
"updated_at": "2020-05-28T19:15:22.344Z",
"updated_by": "yo"
}
You can add a list item like so:
./post_list_item.sh
You should see the new list item created and attached to the above list like so:
{
"id": "hand_inserted_item_id",
"type": "ip",
"value": "127.0.0.1",
"created_at": "2020-05-28T19:15:49.790Z",
"created_by": "yo",
"list_id": "ip_list",
"tie_breaker_id": "a881bf2e-1e17-4592-bba8-d567cb07d234",
"updated_at": "2020-05-28T19:15:49.790Z",
"updated_by": "yo"
}
If you want to post an exception list it would be like so:
./post_exception_list.sh
You should see the new exception list created like so:
{
"created_at": "2020-05-28T19:16:31.052Z",
"created_by": "yo",
"description": "This is a sample endpoint type exception",
"id": "bcb94680-a117-11ea-ad9d-c71f4820e65b",
"list_id": "endpoint_list",
"name": "Sample Endpoint Exception List",
"namespace_type": "single",
"tags": [
"user added string for a tag",
"malware"
],
"tie_breaker_id": "86e08c8c-c970-4b08-a6e2-cdba7bb4e023",
"type": "endpoint",
"updated_at": "2020-05-28T19:16:31.080Z",
"updated_by": "yo"
}
And you can attach exception list items like so:
{
"comments": [],
"created_at": "2020-05-28T19:17:21.099Z",
"created_by": "yo",
"description": "This is a sample endpoint type exception",
"entries": [
{
"field": "actingProcess.file.signer",
"operator": "included",
"type": "match",
"value": "Elastic, N.V."
},
{
"field": "event.category",
"operator": "included",
"type": "match_any",
"value": [
"process",
"malware"
]
}
],
"id": "da8d3b30-a117-11ea-ad9d-c71f4820e65b",
"item_id": "endpoint_list_item",
"list_id": "endpoint_list",
"name": "Sample Endpoint Exception List",
"namespace_type": "single",
"os_types": ["linux"],
"tags": [
"user added string for a tag",
"malware"
],
"tie_breaker_id": "21f84703-9476-4af8-a212-aad31e18dcb9",
"type": "simple",
"updated_at": "2020-05-28T19:17:21.123Z",
"updated_by": "yo"
}
You can then do find for each one like so:
./find_lists.sh
{
"cursor": "WzIwLFsiYzU3ZWZiYzQtNDk3Ny00YTMyLTk5NWYtY2ZkMjk2YmVkNTIxIl1d",
"data": [
{
"id": "ip_list",
"created_at": "2020-05-28T19:15:22.344Z",
"created_by": "yo",
"description": "This list describes bad internet ip",
"name": "Simple list with an ip",
"tie_breaker_id": "c57efbc4-4977-4a32-995f-cfd296bed521",
"type": "ip",
"updated_at": "2020-05-28T19:15:22.344Z",
"updated_by": "yo"
}
],
"page": 1,
"per_page": 20,
"total": 1
}
or for finding exception lists:
./find_exception_lists.sh
{
"data": [
{
"created_at": "2020-05-28T19:16:31.052Z",
"created_by": "yo",
"description": "This is a sample endpoint type exception",
"id": "bcb94680-a117-11ea-ad9d-c71f4820e65b",
"list_id": "endpoint_list",
"name": "Sample Endpoint Exception List",
"namespace_type": "single",
"os_types": ["linux"],
"tags": [
"user added string for a tag",
"malware"
],
"tie_breaker_id": "86e08c8c-c970-4b08-a6e2-cdba7bb4e023",
"type": "endpoint",
"updated_at": "2020-05-28T19:16:31.080Z",
"updated_by": "yo"
}
],
"page": 1,
"per_page": 20,
"total": 1
}
See the full scripts folder for all the capabilities.