## Summary
Fixes different bugs/issues when using exceptions with value based lists for both the UI, the backend, and the large value based lists. See https://github.com/elastic/kibana/issues/79516, but this also fixes several other bugs found mentioned below.
For the front end UI:
* Adds the ability to specify value based lists that are IP Ranges when the source event is an IP. Before you could only match IP to IP and the IP Ranges lists could not be used.
* Breaks down a few functions into smaller functions for unit test writing abilities.
You can now add ip ranges as list values for the UI when before it would not show up:
<img width="1035" alt="Screen Shot 2020-12-07 at 2 15 39 PM" src="https://user-images.githubusercontent.com/1151048/101406552-d6819b00-3896-11eb-9fb5-4c7c2ad93b2e.png">
For value based lists:
* Fixes text data type to use "and" between matching using `operator: 'and'` and changes it from a `terms query to a `match` query
* Adds new API for searching against types called `searchListItemByValues ` so that numeric, text, array based, and other non-stringable types can be sent and then the value based lists will push that to ElasticSearch. This shifts as many corner cases and string/numeric coercions to ElasticSearch rather than Kibana client side code.
* Adds ability to handle arrays within arrays through a `flatten` call.
* Utilizes the `named queries` from ElasticSearch for the new API so that clients can get which parts matched and then use that for their exception list logic rather than in-memory string to string checks. This fixes CIDR and ranges as well as works with arrays.
For the backend exception lists that used value based lists:
* Broke down the `filterEventsAgainstList` function into a folder called `filters` and the functions into other files for better unit based testing.
* Changed the calls from `getListItemByValues` to `searchListItemByValues` which can return exactly what it matched against and this will not break anyone using the existing REST API for `getListItemByValues` since that REST API and client side API stays the same.
* Cleaned up extra promises being used in a few spots that async/await automatically will create.
* Removed the stringabilities and stringify in favor of just a simpler exact check using `JSON.stringify()`
For the tests:
* Adds unit tests to broken down functions
* Adds ip_array, keyword_array, text_array, FTR tests for the backend.
* Adds more CIDR and range based FTR tests for the backend.
* Unskips and fixes all the numeric tests and range tests that could not operate previously from bugs.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
- [ ] Any UI touched in this PR is usable by keyboard only (learn more about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures (run axe in browser: [FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/), [Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] This renders correctly on smaller devices using a responsive layout. (You can test this [in your browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
* Separated out service layer for trusted apps.
* Improved the type structure a bit to avoid using explicit string literals and to add possibility to return OS specific parts of trusted app object in type safe manner.
* Added support for mapping of trusted app to exception item and back.
* Changed schema to support signer in the API.
* Renamed utils to mapping.
* Exported some types in lists plugin and used them in trusted apps.
* Added tests for mapping.
* Added tests for service.
* Switched deletion to use exceptions for not found case.
* Added resetting of the mocks in service layer tests.
* Added handlers tests.
* Refactored mapping tests to be more granular based on the case.
* Restored lowercasing of hash.
* Added schema tests for signer field.
* Removed the grouped tests (they were split into tests for separate concerns).
* Corrected the tests.
* Lowercased the hashes in the service test.
* Moved the lowercasing to the right location.
* Fixed the tests.
* Added test for lowercasing hash value.
* Introduced OperatingSystem enum instead of current types.
* Removed os list constant in favour of separate lists in places that use it (each place has own needs to the ordering).
* Fixed the missed OperatingSystem enum usage.
* Finish adding .lower to exceptionable fields
* Add back migrations
* .lower -> .caseless
* Add separate field for os type
* updates
* Type updates
* Switch over to osTypes
* get rid of _tags
* Add tests for schema validation
* Remove remaining references to _tags
* Another round of test fixes
* DefaultArray tests
* More test fixes
* Fix remaining test failures
* types / tests
* more test updates
* lowercase os values
* Address feedback + fix test failure
* tests
* Fix integration test
* process.executable.path -> process.executable.caseless
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Trusted Apps initial setup for route registration
* Added types for TrustedApp entries
* trusted apps list API returns results
* use methods and const from latest PR merge to lists
* a quick generator for trusted apps entries
* support cli options for trusted app data loader
* Add mocked `createTrustedAppsList()` method to `ExceptionListClientMock`
* tests fro trusted apps route handlers
* tests for trusted apps schema
* Correct name of mock method
* Fix service to ensure return value of `getExceptionList` service throws if service not available
* Fix types
* Refactor TrustedApp type + code review feedback
## Summary
Fixes a bug where the list privileges was returning the `.list` privileges twice instead of returning it once and returning the `.items` privileges second with the call. No UI has to change as the way it was written was dynamic to grab the first key found.
This also adds the functional tests to `x-pack/scripts/functional_tests.js` which was not there originally so the end to tend tests should actually run on the CI machine where it was not running on CI before.
Adds the functional tests to the code owners file as well.
Ensure that you go to the test results page from the Jenkins build:
<img width="901" alt="Screen Shot 2020-08-18 at 1 13 18 AM" src="https://user-images.githubusercontent.com/1151048/90482180-13f7c800-e0f0-11ea-92f2-b30a8fffe84e.png">
And ensure you see the tests under:
```
X-Pack Lists Integration Tests
```
Then click through it and ensure they are shown as running and passing
### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
* Route options timeout -> timeout.payload
* timeout.idleSocket can now be specified per route
* Removing nested ternary
* Fixing integration tests
* Trying to actually fix the integration tests. Existing tests are hitting
idle socket timeout, not the payload timeout
* Fixing payload post timeout integration test
* Fixing PUT and DELETE payload sending too long tests
* Fixing type-script errors
* GET routes can't specify the payload timeout, they can't accept payloads
* Removing some redundancy in the tests
* Adding 'Transfer-Encoding: chunked' to the POST test
* Fixing POST/GET/PUT quick tests
* Adding idleSocket timeout test
* Removing unnecessary `isSafeMethod` call
* Updating documentation
* Removing PUT/DELETE integration tests
* Working around the HapiJS bug
* Deleting unused type import
* The socket can be undefined...
This occurs when using @hapi/shot directly or indirectly via
Server.inject. In these scenarios, there isn't a socket. This can also
occur when a "fake request" is used by the hacky background jobs:
Reporting and Alerting...
* Update src/core/server/http/http_server.ts
Co-authored-by: Josh Dover <me@joshdover.com>
* Adding payload timeout functional tests
* Adding idle socket timeout functional tests
* Adding better comments, using ?? instead of ||
* Fixing the plugin fixture TS
* Fixing some typescript errors
* Fixing plugin fixture tsconfig.json
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Josh Dover <me@joshdover.com>
* Add separate io-ts types for endpoint exception entries
* Fix text typos
* Fix test
* address review comments
* Add extra entry validation when adding items to endpoint_list
* fix test
* fix tests again
* really actually fix the tests
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
This is the basics of end to end tests, so there could be a lot more, but this ties to cover the basics of the tests.
Test with:
```ts
node scripts/functional_tests --config x-pack/test/lists_api_integration/security_and_spaces/config.ts
```
Adds these tests for the route counter parts:
* create_exception_list_items.ts
* create_exception_lists.ts
* delete_exception_list_items.ts
* delete_exception_lists.ts
* find_exception_list_items.ts
* find_exception_lists.ts
* read_exception_list_items.ts
* read_exception_lists.ts
* update_exception_list_items.ts
* update_exception_lists.ts
Fixes a few minor strings, other tests, but no large bugs found with these tests
### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
## Summary
Adds initial set of end to end tests for lists
You can run all of these with the command from kibana root:
```ts
node scripts/functional_tests --config x-pack/test/lists_api_integration/security_and_spaces/config.ts
```
Fixes a few minor bugs found such as...
* Validation for importing lists was not checking if the indexes were created first
* Some wording for the error messages had duplicate words within them
### Checklist
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
## Summary
Addresses feedback from https://github.com/elastic/kibana/pull/72748
- Updates `plugins/lists` tests text from `should not validate` to `should FAIL validation` after feedback that previous text is a bit confusing and can be interpreted to mean that validation is not conducted
- Remove unnecessary spreads from one of my late night PRs
- Removes `siem_common_deps` in favor of `shared_imports` in `plugins/lists`
- Updates `build_exceptions_query.test.ts` to use existing mocks
## Summary
This PR focuses on addressing issues around the pagination and functionality of rules with numerous (2+) exception lists.
- Updated the `use_exception_list.ts` hook to make use of the new multi list find API
- Updated the viewer to make use of the new multi list find API
- Previously was doing a lot of the filtering and paging manually (and badly) in the UI, now the _find takes care of all that
- Added logic for showing `No results` text if user filter/search returns no items
- Previously would show the `This rule has not exceptions` text
* Validate exception list size when adding new items
* Update comment
* Extract list size validation and apply to endpoint route also
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
By default the upload time limit for payloads is 10 seconds. This is really too short and we were getting internal QA bug reports that uploads are timing out on large value list importing. This PR adds the plumbing and unit tests to make the timeout configurable for routes.
* Adds a single timeout option for routes and then normalizes that through Hapi for the socket, payload, and server timeouts.
* Adds unit tests which test the various options
* Adds integration tests which test the various options
* Adds some NOTES about where there are odd behaviors/bugs within Hapi around validations and the timeouts
* Adds a configurable 5 minute timeout to the large value lists route
**Manual testing of the feature**
You can manually test this by adding a configurable option to your chrome network throttle like so below where you throttle upload by some configurable amount. I chose to use 300 kbs/s upload
<img width="556" alt="Screen Shot 2020-07-23 at 11 26 01 AM" src="https://user-images.githubusercontent.com/1151048/88318015-5ab3f700-ccd7-11ea-9d9b-7e3649ec65de.png">
And then run an import of large value lists using a large enough file that it will exceed 5 minutes:
After 5 minutes you should see this message within your server side messages if you have configured your kibana.dev.yml to allow for these messages:
```ts
server respons [10:52:31.377] [access:lists-all] POST /api/lists/items/_import?type=keyword 408 318292ms - 9.0B
```
Note that it should show you that it is trying to return a `408` after `318292ms` the timeout period. Sometimes you will get the 408 in the browser and sometimes the browser actually will not respect the 408 and continue staying in a pending state forever. This seems to be browser side issue and not a client/user land issue. If you get the browser message it will be this error toaster
### Checklist
- [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)
- [x] [Documentation](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#writing-documentation) was added for features that require explanation or tutorials
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
### For maintainers
- [x] This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)
## Summary
This PR is somewhat of an intermediary step. Comments on exception list items are denormalized. We initially decided that we would not add `uuid` to comments, but found that it is in fact necessary. This is intermediary in the sense that what we ideally want to have is a dedicated `comments` CRUD route.
Also just note that I added a callout for when a version conflict occurs (ie: exception item was updated by someone else while a user is editing the same item).
With this PR users are able to:
- Create comments when creating exception list items
- Add new comments on exception item update
Users will currently be blocked from:
- Deleting comments
- Updating comments
- Updating exception item if version conflict is found
* adapt retryCallCluster for new ES client
* review comments
* retry on 408 ResponseError
* remove legacy retry functions
* use Migrator Es client in SO migration
* update migration tests
* improve ES typings and mocks
* migrate decorate ES errors
* add repository es client
* use new es client in so repository
* update repository tests
* fix migrator integration tests
* declare _seq_no & _primary_term on get response. _source expect to be a string
* make _sourceIncludes and refresh compatible with the client
* add test for repository_es_client
* move ApiResponse to es client mocks
* TEMP: handle wait_for as true for deleteByNamespace
* add tests for migration_es_client
* TEMP: skip test for deleteByNamespace refresh
* pass ignore as transport option in mget
* log both es client and response errors
* fix update method test failures
* update deleteByNamespace refresh settings
es doesn't support 'refresh: wait_for' for `updateByQuery` endpoint
* update repository tests. we do not allow customising wait_for
* do not delegate retry logic to es client
* fix type errors after master merged
* fix repository tests
* fix security solutions code
SO doesn't throw Error with status code anymore. Always use SO error helpers
* switch error conditions to use SO error helpers
* cleanup
* address comments about mocks
* use isResponseError helper
* address comments
* fix type errors
Co-authored-by: pgayvallet <pierre.gayvallet@elastic.co>
## Summary
Limits the lists to 9 megs upload size so we don't blow up smaller Kibana installs. Users can change/override this using the switch of `xpack.lists.maxImportPayloadBytes` like so:
```
xpack.lists.maxImportPayloadBytes: 40000000
```
That will increase the amount of bytes that can pushed through REST endpoints from 9 megs to something like 40 megs if the end users want to increase the size of their lists and have enough memory in Kibana.
Metrics and suggestions from testing looks like:
```ts
Kibana with 1 gig of memory can upload ~10 megs of a list before possible out of memory issue
Kibana with 2 gig of memory can upload ~20 megs of a list before possible out of memory issue
```
Things can vary depending on the speed of the uploads of the lists where faster connections to Kibana but slower connections from Kibana to Elastic Search can influence the numbers.
### Checklist
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
* wip - comment and sample json for exceptions
* promise.all for OR-ing exception items and quick-start script
* logging, added/updated json sample scripts, fixed missing await on filter with lists
* WIP
* bug fix where two lists when 'anded' together were not filtering down result set
* undo changes from testing
* fix changes to example json and fixes missed conflict with master
* update log message and fix type errors
* change log statement and add unit test for when exception items without a value list are passed in to the filter function
* fix failing test
* update expect on one test and adds a new test to ensure anding of value lists when appearing in different exception items
* update test after rebasing with master
* properly ands exception item entries together with proper test cases
* fix test (log statement tests - need to come up with a better way to cover these)
* cleans up json examples
* rename test and use 'every' in lieu of 'some' when determining if the filter logic should execute
## Summary
This PR updates the exception list entries schemas.
- **Prior:** `entries` could be `undefined` or empty array on `ExceptionListItemSchema`
- **Now:** `entries` is a required field that cannot be empty - there's really no use for an item without `entries`
- **Prior:** `field` and `value` could be empty string in `EntryMatch`
- **Now:** `field` and `value` can no longer be empty strings
- **Prior:** `field` could be empty string and `value` could be empty array in `EntryMatchAny`
- **Now:** `field` and `value` can no longer be empty string and array respectively
- **Prior:** `field` and `list.id` could be empty string in `EntryList`
- **Now:** `field` and `list.id` can no longer be empty strings
- **Prior:** `field` could be empty string in `EntryExists`
- **Now:** `field` can no longer be empty string
- **Prior:** `field` could be empty string in `EntryNested`
- **Now:** `field` can no longer be empty string
- **Prior:** `entries` could be empty array in `EntryNested`
- **Now:** `entries` can no longer be empty array
### Summary
The intent is to get the data structures in similar to rules so that we can have eventually immutable and versioned lists in later releases without too much hassle of upgrading the list and list item data structures.
* Adds version and immutability data structures to the exception lists and the value lists.
* Adds an optional version number to the update route of each so that you can modify the number either direction or you can omit it and it works like the detection rules where it will auto-increment the number.
* Does _not_ add a version and immutability to the exception list items and value list items.
* Does _not_ update the version number when you add a new exception list item or value list item.
**Examples:**
❯ ./post_list.sh
```json
{
"_version": "WzAsMV0=",
"id": "ip_list",
"created_at": "2020-07-21T20:31:11.679Z",
"created_by": "yo",
"description": "This list describes bad internet ip",
"immutable": false,
"name": "Simple list with an ip",
"tie_breaker_id": "d6bd7552-84d1-4f95-88c4-cc504517b4e5",
"type": "ip",
"updated_at": "2020-07-21T20:31:11.679Z",
"updated_by": "yo",
"version": 1
}
```
❯ ./post_exception_list.sh
```json
{
"_tags": [
"endpoint",
"process",
"malware",
"os:linux"
],
"_version": "WzMzOTgsMV0=",
"created_at": "2020-07-21T20:31:35.933Z",
"created_by": "yo",
"description": "This is a sample endpoint type exception",
"id": "2c24b100-cb91-11ea-a872-adfddf68361e",
"immutable": false,
"list_id": "simple_list",
"name": "Sample Endpoint Exception List",
"namespace_type": "single",
"tags": [
"user added string for a tag",
"malware"
],
"tie_breaker_id": "c11c4d53-d0be-4904-870e-d33ec7ca387f",
"type": "detection",
"updated_at": "2020-07-21T20:31:35.952Z",
"updated_by": "yo",
"version": 1
}
```
```json
❯ ./update_list.sh
{
"_version": "WzEsMV0=",
"created_at": "2020-07-21T20:31:11.679Z",
"created_by": "yo",
"description": "Some other description here for you",
"id": "ip_list",
"immutable": false,
"name": "Changed the name here to something else",
"tie_breaker_id": "d6bd7552-84d1-4f95-88c4-cc504517b4e5",
"type": "ip",
"updated_at": "2020-07-21T20:31:47.089Z",
"updated_by": "yo",
"version": 2
}
```
```json
❯ ./update_exception_list.sh
{
"_tags": [
"endpoint",
"process",
"malware",
"os:linux"
],
"_version": "WzMzOTksMV0=",
"created_at": "2020-07-21T20:31:35.933Z",
"created_by": "yo",
"description": "Different description",
"id": "2c24b100-cb91-11ea-a872-adfddf68361e",
"immutable": false,
"list_id": "simple_list",
"name": "Sample Endpoint Exception List",
"namespace_type": "single",
"tags": [
"user added string for a tag",
"malware"
],
"tie_breaker_id": "c11c4d53-d0be-4904-870e-d33ec7ca387f",
"type": "endpoint",
"updated_at": "2020-07-21T20:31:56.628Z",
"updated_by": "yo",
"version": 2
}
```
### Checklist
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
## Summary
* Adds list permissions as a feature control to SIEM.
* Separates the controls between two, one of which is `access:lists-all` and the other is `access:lists-read`
* Grants SIEM the ability to utilize both depending on which feature mode the space is in.
## Summary
* Adds conflict versioning by exposing the "_version" from the saved object system. It renames "version" to "_version" so that we can use regular "version" later for versioning things for pre-packaged lists abilities.
* Utilizes `t.OutputOf` in the requests and the data types to give us more correctly types
* Removes the `Identity` utility as that is adding confusion and can confuse vs code rather than improves things
* Removes extra types that were causing confusion which was an idiom from io-ts
* Changes the wording of `Partial` by removing that and instead focuses the request types on either client side or server side at this point.
NOTE: The UI can migrate to holding onto the `_version` and then push it back down when it wants to migrate to using the conflict resolution. If the UI does not push it down, then a value of undefined will be used which is indicating that no conflict errors are wanted.
Output example of posting an exception list:
❯ ./post_exception_list.sh
```ts
{
"_tags": [
"endpoint",
"process",
"malware",
"os:linux"
],
"_version": "Wzk4NiwxXQ==",
"created_at": "2020-07-17T18:59:22.872Z",
"created_by": "yo",
"description": "This is a sample endpoint type exception",
"id": "a08795b0-c85f-11ea-b1a6-c155df988a92",
"list_id": "simple_list",
"name": "Sample Endpoint Exception List",
"namespace_type": "single",
"tags": [
"user added string for a tag",
"malware"
],
"tie_breaker_id": "b789ec05-3e0f-4344-a156-0c0f5b6e2f9c",
"type": "detection",
"updated_at": "2020-07-17T18:59:22.891Z",
"updated_by": "yo"
}
```
Output example of posting an exception list item
❯ ./post_exception_list_item.sh
```ts
{
"_tags": [
"endpoint",
"process",
"malware",
"os:linux"
],
"_version": "Wzk4NywxXQ==",
"comments": [],
"created_at": "2020-07-17T18:59:30.286Z",
"created_by": "yo",
"description": "This is a sample endpoint type exception",
"entries": [
{
"field": "actingProcess.file.signer",
"operator": "excluded",
"type": "exists"
},
{
"field": "host.name",
"operator": "included",
"type": "match_any",
"value": [
"some host",
"another host"
]
}
],
"id": "a4f2b800-c85f-11ea-b1a6-c155df988a92",
"item_id": "simple_list_item",
"list_id": "simple_list",
"name": "Sample Endpoint Exception List",
"namespace_type": "single",
"tags": [
"user added string for a tag",
"malware"
],
"tie_breaker_id": "1dc456bc-7aa9-44b4-bca3-131689cf729f",
"type": "simple",
"updated_at": "2020-07-17T18:59:30.304Z",
"updated_by": "yo"
}
```
Output example of when you get an exception list:
❯ ./get_exception_list.sh simple_list
```ts
{
"_tags": [
"endpoint",
"process",
"malware",
"os:linux"
],
"_version": "WzEwNzcsMV0=",
"created_at": "2020-07-17T18:59:22.872Z",
"created_by": "yo",
"description": "Different description",
"id": "a08795b0-c85f-11ea-b1a6-c155df988a92",
"list_id": "simple_list",
"name": "Sample Endpoint Exception List",
"namespace_type": "single",
"tags": [
"user added string for a tag",
"malware"
],
"tie_breaker_id": "b789ec05-3e0f-4344-a156-0c0f5b6e2f9c",
"type": "endpoint",
"updated_at": "2020-07-17T20:01:24.958Z",
"updated_by": "yo"
}
```
Example of the error you get if you do an update of an exception list and someone else has changed it:
```ts
{
"message": "[exception-list:a08795b0-c85f-11ea-b1a6-c155df988a92]: version conflict, required seqNo [1074], primary term [1]. current document has seqNo [1077] and primary term [1]: [version_conflict_engine_exception] [exception-list:a08795b0-c85f-11ea-b1a6-c155df988a92]: version conflict, required seqNo [1074], primary term [1]. current document has seqNo [1077] and primary term [1], with { index_uuid=\"a2mgXBO6Tl2ULDq-MTs1Tw\" & shard=\"0\" & index=\".kibana-hassanabad_1\" }",
"status_code": 409
}
```
Lists are the same way and flavor, they encode the _version the same way that saved objects do. To see those work you run these scripts:
```ts
./post_list.sh
./post_list_item.sh
./find_list.sh
./find_list_item.sh
```
### Checklist
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
## Summary
* Adds the createEndpointListSchema to the create_endpoint_list_route for API boundary checks
* Adds unit tests to the requests
* Fixes a few bugs found in the find endpoints for types
* Fixes or deletes the skipped tests
* Updated TODO blocks and removed ones that are obsolete
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
* Adds specific endpoint_list REST API and API for abilities to autocreate the endpoint_list if it gets deleted
* Added the check against prepackaged list
* Updated to use LIST names
* Removed the namespace where it does not belong
* Updates per code review an extra space that was added
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
* Adds the ability for exception lists to be multi-list queried
* Fixes a bunch of script issues where I did not update everywhere I needed to use `ip_list` and deletes an old list that now lives within the new/lists folder
* Fixes a few io-ts issues with Encode Decode while I was in there.
* Adds two more types and their tests for supporting converting between comma separated strings and arrays for GET calls.
* Fixes one weird circular dep issue while adding more types.
You now send into the find an optional comma separated list of exception lists their namespace type and any filters like so:
```ts
GET /api/exception_lists/items/_find?list_id=simple_list,endpoint_list&namespace_type=single,agnostic&filtering=filter1,filter2"
```
And this will return the results of both together with each filter applied to each list. If you use a sort field and ordering it will order across the lists together as if they are one list. Filter is optional like before. If you provide less filters than there are lists, the lists will only apply the filters to each list until it runs out of filters and then not filter the other lists.
If at least one list is found this will _not_ return a 404 but it will _only_ query the list(s) it did find. If none of the lists are found, then this will return a 404 not found exception.
**Script testing**
See these files for more information:
* find_exception_list_items.sh
* find_exception_list_items_by_filter.sh
But basically you can create two lists and an item for each of the lists:
```ts
./post_exception_list.sh ./exception_lists/new/exception_list.json
./post_exception_list_item.sh ./exception_lists/new/exception_list_item.json
./post_exception_list.sh ./exception_lists/new/exception_list_agnostic.json
./post_exception_list_item.sh ./exception_lists/new/exception_list_item_agnostic.json
```
And then you can query these two lists together:
```ts
./find_exception_list_items.sh simple_list,endpoint_list single,agnostic
```
Or for filtering you can query both and add a filter for each one:
```ts
./find_exception_list_items_by_filter.sh simple_list,endpoint_list "exception-list.attributes.name:%20Sample%20Endpoint%20Exception%20List,exception-list-agnostic.attributes.name:%20Sample%20Endpoint%20Exception%20List" single,agnostic
```
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
## Summary
* Adds "wait_for" to all the create, update, patch, and delete endpoints
* Ran some quick tests against import and the performance still looks acceptable
* Updates the unit tests to reflect the addition
### Checklist
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
## Summary
* Changes the value list imports to use a streaming in model
* Adds a custom light hand spun multi-part parser for the incoming text
* Adds a buffer pause and resume which continues to buffer the incoming data if an async event such as creating a list from the attachment file needs to happen but does not emit the lines until the resume continues.
* Adds a data slicing if the buffer becomes larger than the maximum so that if we begin buffering too quickly within memory we don't blow up the limit of Elastic Search.
* Adds unit tests
### Checklist
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
## Summary
Adds these data types to the value based lists end points from [Elasticsearch field data types](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html):
Single value based list types:
* binary
* boolean
* byte
* date
* date_nanos
* date_range
* double
* float
* integer
* ip
* half_float
* keyword
* text
* long
* short
Range value based list types:
* double_range
* float_range
* integer_range
* ip_range
* long_range
Geo value based list types: (caveat is that you cannot query them using other geometry just yet ... you can only these and export them)
* geo_point
* geo_shape
* shape
For importing and exporting different values such as ranges, geo, or single values, this introduces a serialize and deserialize option for the endpoints.
For example if you want to serialize in an ip_range such as 192.168.0.1,192.168.0.3 which has a comma between the two would use the following:
```ts
POST /api/lists
{
"name": "List with an ip range",
"serializer": "(?<gte>.+),(?<lte>.+)",
"deserializer": "{{gte}},{{lte}}",
"description": "This list has ip ranges",
"type": "date_range"
}
```
If you want to serialize in keywords from a list that _only_ match a particular value you would use the following:
```ts
POST /api/lists
{
"id": "keyword_custom_format_list",
"name": "Simple list with a keyword using a custom format",
"description": "This parses the first found ipv4 only",
"serializer": "(?<value>((25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?).){3}(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))",
"deserializer": "{{value}}",
"type": "keyword"
}
```
The serializer is a [named capturing group](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/match) while the deserializer is using [MustacheJS](https://github.com/janl/mustache.js/). The range type, single value types, and geo types all have default captures for their serialize and default mustache templates if none are configured with an endpoint.
The default capture groups and mustache handles for each are:
* shape, geo_point, geo_shape: `(?<lat>.+),(?<lon>.+)`
* date_range: `(?<gte>.+),(?<lte>.+)|(?<value>.+)`
* other ranges are: `(?<gte>.+)-(?<lte>.+)|(?<value>.+)`
* all single data types: `(?<value>.+)`
For ranges you can use both `gte, lte`, and `value` together. If `gte` _and_ `lte` matches it will use that for the greater than, less than elastic range and ignore `value` even if `value` also matched. If _only_ `value` matches and `gte`, `lte` does not match then it will use `value` and put `value` as _both_ the `gte`, and `lte`.
For example, if you are serializing in a list of ip ranges as the list data type, `ip_range` and you have these 3 entries in the file:
```ts
127.0.0.1
127.0.0.2-5
```
The default `serializer` will use `(?<gte>.+)-(?<lte>.+)|(?<value>.+)` and you will get two elastic documents like so:
```ts
{
"_source" : {
"ip_range" : {
"gte" : "127.0.0.1",
"lte" : "127.0.0.1"
}
}
{
"_source" : {
"ip_range" : {
"gte" : "127.0.0.2",
"lte" : "127.0.0.5"
}
}
```
The default mustache handles for each are:
* shape, geo_point, geo_shape: `{{{lat}}},{{{lon}}}`
* date_range: `{{{gte}}},{{{lte}}}`
* other ranges are: `{{{gte}}}-{{{lte}}}`
* all values are: `{{{value}}}`
I use three instead of two handle bars (`{{{` vs.` {{`) so that HTML is not escaped for the lists. You can override and change it if you need or want the escaping.
If during the deserializer phase it detects that a `gte` and `lte` are exactly the same it will still output them as a two items and use the mustache deserialize value. Using the ip-range example above that will be outputted like so since it detects that the lte-gte are exactly the same value:
```ts
127.0.0.1-127.0.0.1
127.0.0.2-127.0.0.5
```
---
Interesting queries to run from the lists scripts folder for testing:
Load some small test files from `./lists/files` for example:
```ts
./import_list_items_by_filename.sh ip_range ./lists/files/ip_range_cidr.txt
./import_list_items_by_filename.sh ip_range ./lists/files/ip_range.txt
./import_list_items_by_filename.sh date ./lists/files/date.txt
./import_list_items_by_filename.sh ip_range ./lists/files/ip_range_mixed.txt
...
```
Export them
```ts
./export_list_items.sh ip_range_cidr.txt
./export_list_items.sh ip_range.txt
./export_list_items.sh date.txt
./export_list_items.sh ip_range_mixed.txt
...
```
Find on them
```ts
./find_list_items.sh ip_range_cidr.txt
./find_list_items.sh ip_range.txt
./find_list_items.sh date.txt
./find_list_items.sh ip_range_mixed.txt
...
```
Find specific values such as:
```ts
./get_list_item_by_value.sh ip_range_mixed.txt 192.168.0.1
./get_list_item_by_value.sh date.txt 2020-08-25T17:57:01.978Z
...
```
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
## Summary
* Removes the feature flag and turns on lists by default
* Applies to both exception lists and value lists
* Removes all scary messages about having it enabled
* Updates the unit tests to work with it on
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
## Summary
This PR tries to start to tie together the server and client changes for exceptions lists.
- Updates graphql types to allow UI access to a rule's `exceptions_list` property
- Updates the exception viewer component to now dynamically take the rule `exceptions_list`, up until now we just had an empty array in it's place
- Updates the viewer logic to check if a rule has an endpoint list associated with it. If it does, then it displays both detections and endpoint UIs (in the viewer), if it does not, then it only displays the detections UI
- Updates the viewer UI to better deal with spacing when an exception list item only has one or two entries (before the and badge with the antennas was stretching passed the exception items to fill the space)
- Updates the detections engine exceptions logic to fetch list items using an exception list's `id` as opposed to it's `list_id`, this now aligns with the UI using the same params on its end
- Adds exception list `type` to information kept by the rule for exception lists
- Updates the exception list type from `string` to `endpoint | detection`
- Updates the exception list _item_ type from `string` to `simple`
- Adds unit tests for the detection engine server side util that fetches the exception list items
* stub out task for the exceptions list packager
* Hits list code and pages
* refactor
* Begin adding saved object and type definitions
* Transforms to endpoint exceptions
* Get internal SO client
* update messaging
* cleanup
* Integrating with task manager
* Integrated with task manager properly
* Begin adding schemas
* Add multiple OS and schema version support
* filter by OS
* Fixing sort
* Move to security_solutions
* siem -> securitySolution
* Progress on downloads, cleanup
* Add config, update artifact creation, add TODOs
* Fixing buffer serialization problem
* Adding cleanup to task
* Handle HEAD req
* proper header
* More robust task management
* single -> agnostic
* Fix OS filtering
* Scaffolding digital signatures / tests
* Adds rotue for creating endpoint user
* Cleanup
* persisting user
* Adding route to fetch created user
* Addings tests for translating exceptions
* Adding test for download API
* Download tweaks + artifact generation fixes
* reorganize
* fix imports
* Fixing test
* Changes id of SO
* integration tests setup
* Add first integration tests
* Cache layer
* more schema validation
* Set up for manifest update
* minor change
* remove setup code
* add manifest schema
* refactoring
* manifest rewrite (partial)
* finish scaffolding new manifest logic
* syntax errors
* more refactoring
* Move to endpoint directory
* minor cleanup
* clean up old artifacts
* Use diff appropriately
* Fix download
* schedule task on interval
* Split up into client/manager
* more mocks
* config interval
* Fixing download tests and adding cache tests
* lint
* mo money, mo progress
* Converting to io-ts
* More tests and mocks
* even more tests and mocks
* Merging both refactors
* Adding more tests for the convertion layer
* fix conflicts
* Adding lzma types
* Bug fixes
* lint
* resolve some type errors
* Adding back in cache
* Fixing download test
* Changing cache to be sized
* Fix manifest manager initialization
* Hook up datasource service
* Fix download tests
* Incremental progress
* Adds integration with ingest manager for auth
* Update test fixture
* Add manifest dispatch
* Refactoring to use the same SO Client from ingest
* bug fixes
* build renovate config
* Fix endpoint_app_context_services tests
* Only index the fields that are necessary for searching
* Integ test progress
* mock and test city
* Add task tests
* Tests for artifact_client and manifest_client
* Add manifest_manager tests
* minor refactor
* Finish manifest_manager tests
* Type errors
* Update integ test
* Type errors, final cleanup
* Fix integration test and add test for invalid api key
* minor fixup
* Remove compression
* Update task interval
* Removing .text suffix from translated list
* Fixes hashes for unit tests
* clean up yarn.lock
* Remove lzma-native from package.json
* missed updating one of the tests
Co-authored-by: Alex Kahan <alexander.kahan@elastic.co>
* Add pure API functions and react hooks for value list APIs
This also adds a generic hook, useAsyncTask, that wraps an async
function to provide basic utilities:
* loading state
* error state
* abort/cancel function
* Fix type errors in hook tests
These were not caught locally as I was accidentally running typescript
without the full project.
* Document current limitations of useAsyncTask
* Defines a new validation function that returns an Either instead of a tuple
This allows callers to further leverage fp-ts functions as needed.
* Remove duplicated copyright comment
* WIP: Perform request/response validations in the FP style
* leverages new validateEither fn which returns an Either
* constructs a pipeline that:
* validates the payload
* performs the API call
* validates the response
and short-circuits if any of those produce a Left value.
It then converts the Either into a promise that either rejects with the
Left or resolves with the Right.
* Adds helper function to convert a TaskEither back to a Promise
This cleans up our validation pipeline considerably.
* Adds request/response validations to findLists
* refactors private API functions to accept the encoded request schema
(i.e. snake cased)
* refactors validateEither to use `schema.validate` instead of
`schema.decode` since we don't actually want the decoded value, we just
want to verify that it'll be able to be decoded on the backend.
* Refactor our API types
* Add request/response validation to import/export functions
* Fix type errors
* Continue to export decoded types without a qualifier
* pull types used by hooks from their new location
* Fix errors with usage of act()
* Attempting to reduce plugin bundle size
By pulling from the module directly instead of an index, we can
hopefully narrow down our dependencies until tree-shaking does this for
us.
* useAsyncFn's initiator does not return a promise
Rather than returning a promise and requiring the caller to handle a
rejection, we instead return nothing and require the user to watch the
hook's state.
* success can be handled with a useEffect on state.result
* errors can be handled with a useEffect on state.error
* Fix failing test
Assertion count wasn't updated following interface changes; we've now
got two inline expectations so this isn't needed.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* move last snapshot to inline
* move legacy files to legacy subfolder
* move request types out of legacy
* export Headers from http instead of elasticsearch
* renaming - first pass
* renaming - second pass
* fix core mocks
* adapt new calls
* update generated doc
* fix IT test mocks
* fix new usages
### Summary
This PR is a follow up to #68864 . That PR used a partial to differentiate between new and existing comments, this meant that comments could be updated when they shouldn't. It was decided in our discussion of exception list schemas that comments should be append only. This PR assures that's the case, but also leaves it open to editing comments (via API). It checks to make sure that users can only update their own comments.
* Updates list entry schema, exposes exception list client, updates tests
* create new de list schema and unit tests
* updated route unit tests and types to match new list schema
* updated existing DE exceptions code so it should now work as is with updated schema
* test and types cleanup
* cleanup
* update unit test
* updates per feedback
### Summary
This PR is meant to update the `ExceptionListItemSchema.entries` structure to align with the most recent conversations regarding the need for a more explicit depiction of `nested` fields. To summarize:
- Adds schema validation for requests and responses within `lists/public/exceptions/api.ts`. It was super helpful in catching existing bugs. Anyone that uses the api will run through this validation. If the client tries to send up a malformed request, the request will not be made and an error returned. If the request is successful, but somehow the response is malformed, an error is returned. There may be some UX things to figure out about how to best communicate these errors to the user, or if surfacing the raw error is fine.
- Updates `entries` structure in lists plugin api
- Updates hooks and tests within `lists/public` that make reference to new structure
- Updates and adds unit tests for updated schemas
- Removes unused temporary types in `security_solution/public/common/components/exceptions/` to now reference updated schema
- Updates UI tests
- Updates `lists/server/scripts`
### Summary
This is part of a series of upcoming changes to the exception list item structure. This PR focuses solely on updating exception_item.comment. The hope is to keep these PRs relatively small.
- Updates exception_item.comment structure which was previously a string to exception_item.comments which is an array of { comment: string; created_by: string; created_at: string; }
- Adds a few unit tests server side
- Fixes some minor misspellings
- Updates ExceptionViewer component in the UI to account for new structure
## Summary
* Smaller follow ups and bug fixes from: https://github.com/elastic/kibana/pull/68127
* Added unknown to `findDifferencesRecursive`
* Added linter rule to catch NodeJS code in the common folders for both `lists` and `security_solution`
* Removed the Hapi server type from the common folder of lists
### Checklist
* Added unknown to the correct locations
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
### Summary
This PR is a follow up to #68027 . It brings it all together to complete the exceptions viewer component. This component is meant to display all exception items and allow a user to create, edit, delete, and search these exception items.
- Moves ExceptionItem (from part 1) into its own folder
- Adds exceptions_viewer_header component that includes the search, list toggle, and add exception buttons
- Adds actual ExceptionViewer component
- Updates the useExceptionList hook refresh function logic. Noticed that the previous version was creating some issues
* rename siem to security_solution
* rename siem to security solution inside of code
* rename translation keys
* fix snapshot
* replace siem for security solution in tutorial
* missing translation to be renamed
* fix types for api test integration
* updates runner file to match the new path
* change category for kibana settings
* miss renaming in advance settings
* fixes cypress tests
* fix api integration test
* fix new translation
* fix unit test
* update translation i18n
* update translation i18n II
Co-authored-by: Gloria Hornero <snootchie.boochies@gmail.com>
