## Summary
Addresses feedback from https://github.com/elastic/kibana/pull/72748
- Updates `plugins/lists` tests text from `should not validate` to `should FAIL validation` after feedback that previous text is a bit confusing and can be interpreted to mean that validation is not conducted
- Remove unnecessary spreads from one of my late night PRs
- Removes `siem_common_deps` in favor of `shared_imports` in `plugins/lists`
- Updates `build_exceptions_query.test.ts` to use existing mocks
## Summary
This PR focuses on addressing issues around the pagination and functionality of rules with numerous (2+) exception lists.
- Updated the `use_exception_list.ts` hook to make use of the new multi list find API
- Updated the viewer to make use of the new multi list find API
- Previously was doing a lot of the filtering and paging manually (and badly) in the UI, now the _find takes care of all that
- Added logic for showing `No results` text if user filter/search returns no items
- Previously would show the `This rule has not exceptions` text
## Summary
This PR addresses the following:
- Adds `list_id` to `rule.exceptions_list` - this is needed in a number of features
- Updated `getExceptions` in `x-pack/plugins/security_solution/server/lib/detection_engine/signals/utils.ts` to use the latest exception item find endpoint that accepts an array of lists (previously was looping through lists and conducting a `find` for each)
- Updated prepackaged rule that makes reference to global endpoint list to include `list_id`
- Updates `formatAboutStepData` in `x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/create/helpers.ts` to include exception list `list_id`
* Validate exception list size when adding new items
* Update comment
* Extract list size validation and apply to endpoint route also
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
By default the upload time limit for payloads is 10 seconds. This is really too short and we were getting internal QA bug reports that uploads are timing out on large value list importing. This PR adds the plumbing and unit tests to make the timeout configurable for routes.
* Adds a single timeout option for routes and then normalizes that through Hapi for the socket, payload, and server timeouts.
* Adds unit tests which test the various options
* Adds integration tests which test the various options
* Adds some NOTES about where there are odd behaviors/bugs within Hapi around validations and the timeouts
* Adds a configurable 5 minute timeout to the large value lists route
**Manual testing of the feature**
You can manually test this by adding a configurable option to your chrome network throttle like so below where you throttle upload by some configurable amount. I chose to use 300 kbs/s upload
<img width="556" alt="Screen Shot 2020-07-23 at 11 26 01 AM" src="https://user-images.githubusercontent.com/1151048/88318015-5ab3f700-ccd7-11ea-9d9b-7e3649ec65de.png">
And then run an import of large value lists using a large enough file that it will exceed 5 minutes:
After 5 minutes you should see this message within your server side messages if you have configured your kibana.dev.yml to allow for these messages:
```ts
server respons [10:52:31.377] [access:lists-all] POST /api/lists/items/_import?type=keyword 408 318292ms - 9.0B
```
Note that it should show you that it is trying to return a `408` after `318292ms` the timeout period. Sometimes you will get the 408 in the browser and sometimes the browser actually will not respect the 408 and continue staying in a pending state forever. This seems to be browser side issue and not a client/user land issue. If you get the browser message it will be this error toaster
### Checklist
- [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)
- [x] [Documentation](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#writing-documentation) was added for features that require explanation or tutorials
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
### For maintainers
- [x] This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)
Add validation to reject when value list and other exception type are entries in the same exception item. Also adds tests for this situation on the schema validation
## Summary
This PR is somewhat of an intermediary step. Comments on exception list items are denormalized. We initially decided that we would not add `uuid` to comments, but found that it is in fact necessary. This is intermediary in the sense that what we ideally want to have is a dedicated `comments` CRUD route.
Also just note that I added a callout for when a version conflict occurs (ie: exception item was updated by someone else while a user is editing the same item).
With this PR users are able to:
- Create comments when creating exception list items
- Add new comments on exception item update
Users will currently be blocked from:
- Deleting comments
- Updating comments
- Updating exception item if version conflict is found
* adapt retryCallCluster for new ES client
* review comments
* retry on 408 ResponseError
* remove legacy retry functions
* use Migrator Es client in SO migration
* update migration tests
* improve ES typings and mocks
* migrate decorate ES errors
* add repository es client
* use new es client in so repository
* update repository tests
* fix migrator integration tests
* declare _seq_no & _primary_term on get response. _source expect to be a string
* make _sourceIncludes and refresh compatible with the client
* add test for repository_es_client
* move ApiResponse to es client mocks
* TEMP: handle wait_for as true for deleteByNamespace
* add tests for migration_es_client
* TEMP: skip test for deleteByNamespace refresh
* pass ignore as transport option in mget
* log both es client and response errors
* fix update method test failures
* update deleteByNamespace refresh settings
es doesn't support 'refresh: wait_for' for `updateByQuery` endpoint
* update repository tests. we do not allow customising wait_for
* do not delegate retry logic to es client
* fix type errors after master merged
* fix repository tests
* fix security solutions code
SO doesn't throw Error with status code anymore. Always use SO error helpers
* switch error conditions to use SO error helpers
* cleanup
* address comments about mocks
* use isResponseError helper
* address comments
* fix type errors
Co-authored-by: pgayvallet <pierre.gayvallet@elastic.co>
* Refresh rule details when exception list modal modifies the rule
This addresses a bug where, when opening the exceptions modal for the first
time and creating exceptions, the details page does not reflect these
created exceptions until a full refresh.
This is due to the hook performing the refresh being dependent on the
rule's exceptions_list attribute, which is not populated until after
opening the modal. Because the UI is not informed of the rule update, it
did not know to refresh the rule.
This adds the machinery necessary to make the above work. It:
* adds a new hook for fetching/refreshing a rule
* Adds an onRuleChange callback to both the ExceptionsViewer and the
mutating AddExceptionModal
* passes the refresh function in as the onRuleChange callback
There's currently a gross intermediate state here where the loading screen is
displayed while the rule refreshes in the background; I'll be fixing
that shortly.
* Do not show loading/blank state while refreshing rule
On Rule Details, when the Add Exceptions modal creates the rule's
exception list, we refresh quietly in the background by setting our rule
from null -> ruleA -> ruleB instead of null -> ruleA -> null -> ruleB.
This also simplifies the loading logic in a few places now that we're
using our new rule: we mainly care whether or not our rule is populated.
* Display toast error if rule fetch fails
This should now have feature parity with useRule, while additionally
providing a function to refresh the rule.
* Refactor tests to leverage existing helpers
* Add return type to our callback function
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Limits the lists to 9 megs upload size so we don't blow up smaller Kibana installs. Users can change/override this using the switch of `xpack.lists.maxImportPayloadBytes` like so:
```
xpack.lists.maxImportPayloadBytes: 40000000
```
That will increase the amount of bytes that can pushed through REST endpoints from 9 megs to something like 40 megs if the end users want to increase the size of their lists and have enough memory in Kibana.
Metrics and suggestions from testing looks like:
```ts
Kibana with 1 gig of memory can upload ~10 megs of a list before possible out of memory issue
Kibana with 2 gig of memory can upload ~20 megs of a list before possible out of memory issue
```
Things can vary depending on the speed of the uploads of the lists where faster connections to Kibana but slower connections from Kibana to Elastic Search can influence the numbers.
### Checklist
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
* wip - comment and sample json for exceptions
* promise.all for OR-ing exception items and quick-start script
* logging, added/updated json sample scripts, fixed missing await on filter with lists
* WIP
* bug fix where two lists when 'anded' together were not filtering down result set
* undo changes from testing
* fix changes to example json and fixes missed conflict with master
* update log message and fix type errors
* change log statement and add unit test for when exception items without a value list are passed in to the filter function
* fix failing test
* update expect on one test and adds a new test to ensure anding of value lists when appearing in different exception items
* update test after rebasing with master
* properly ands exception item entries together with proper test cases
* fix test (log statement tests - need to come up with a better way to cover these)
* cleans up json examples
* rename test and use 'every' in lieu of 'some' when determining if the filter logic should execute
## Summary
This PR updates the exception list entries schemas.
- **Prior:** `entries` could be `undefined` or empty array on `ExceptionListItemSchema`
- **Now:** `entries` is a required field that cannot be empty - there's really no use for an item without `entries`
- **Prior:** `field` and `value` could be empty string in `EntryMatch`
- **Now:** `field` and `value` can no longer be empty strings
- **Prior:** `field` could be empty string and `value` could be empty array in `EntryMatchAny`
- **Now:** `field` and `value` can no longer be empty string and array respectively
- **Prior:** `field` and `list.id` could be empty string in `EntryList`
- **Now:** `field` and `list.id` can no longer be empty strings
- **Prior:** `field` could be empty string in `EntryExists`
- **Now:** `field` can no longer be empty string
- **Prior:** `field` could be empty string in `EntryNested`
- **Now:** `field` can no longer be empty string
- **Prior:** `entries` could be empty array in `EntryNested`
- **Now:** `entries` can no longer be empty array
### Summary
The intent is to get the data structures in similar to rules so that we can have eventually immutable and versioned lists in later releases without too much hassle of upgrading the list and list item data structures.
* Adds version and immutability data structures to the exception lists and the value lists.
* Adds an optional version number to the update route of each so that you can modify the number either direction or you can omit it and it works like the detection rules where it will auto-increment the number.
* Does _not_ add a version and immutability to the exception list items and value list items.
* Does _not_ update the version number when you add a new exception list item or value list item.
**Examples:**
❯ ./post_list.sh
```json
{
"_version": "WzAsMV0=",
"id": "ip_list",
"created_at": "2020-07-21T20:31:11.679Z",
"created_by": "yo",
"description": "This list describes bad internet ip",
"immutable": false,
"name": "Simple list with an ip",
"tie_breaker_id": "d6bd7552-84d1-4f95-88c4-cc504517b4e5",
"type": "ip",
"updated_at": "2020-07-21T20:31:11.679Z",
"updated_by": "yo",
"version": 1
}
```
❯ ./post_exception_list.sh
```json
{
"_tags": [
"endpoint",
"process",
"malware",
"os:linux"
],
"_version": "WzMzOTgsMV0=",
"created_at": "2020-07-21T20:31:35.933Z",
"created_by": "yo",
"description": "This is a sample endpoint type exception",
"id": "2c24b100-cb91-11ea-a872-adfddf68361e",
"immutable": false,
"list_id": "simple_list",
"name": "Sample Endpoint Exception List",
"namespace_type": "single",
"tags": [
"user added string for a tag",
"malware"
],
"tie_breaker_id": "c11c4d53-d0be-4904-870e-d33ec7ca387f",
"type": "detection",
"updated_at": "2020-07-21T20:31:35.952Z",
"updated_by": "yo",
"version": 1
}
```
```json
❯ ./update_list.sh
{
"_version": "WzEsMV0=",
"created_at": "2020-07-21T20:31:11.679Z",
"created_by": "yo",
"description": "Some other description here for you",
"id": "ip_list",
"immutable": false,
"name": "Changed the name here to something else",
"tie_breaker_id": "d6bd7552-84d1-4f95-88c4-cc504517b4e5",
"type": "ip",
"updated_at": "2020-07-21T20:31:47.089Z",
"updated_by": "yo",
"version": 2
}
```
```json
❯ ./update_exception_list.sh
{
"_tags": [
"endpoint",
"process",
"malware",
"os:linux"
],
"_version": "WzMzOTksMV0=",
"created_at": "2020-07-21T20:31:35.933Z",
"created_by": "yo",
"description": "Different description",
"id": "2c24b100-cb91-11ea-a872-adfddf68361e",
"immutable": false,
"list_id": "simple_list",
"name": "Sample Endpoint Exception List",
"namespace_type": "single",
"tags": [
"user added string for a tag",
"malware"
],
"tie_breaker_id": "c11c4d53-d0be-4904-870e-d33ec7ca387f",
"type": "endpoint",
"updated_at": "2020-07-21T20:31:56.628Z",
"updated_by": "yo",
"version": 2
}
```
### Checklist
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
* Add loading spinners to Value Lists modal
While export or a delete is pending, we display a loading spinner
instead of the button that was clicked.
Since state is controlled in the parent, we must pass this additional
state in the same way; the table component simply reacts to this state.
* Fix bug with useAsync and multiple calls
Multiple calls to start() would not previously reset the hook's state,
where useEffect on the hook's state would fire improperly as subsequent
calls would not travel the same undefined -> result path.
* Fix style of loading spinner
This fits the size of the button it's replacing, so no shifting occurs
when replacing elements.
* Better styling of spinner
Keep it roughly the same size as the icons themselves, and fill the
space with margin.
* Fix circular dependency in value lists modal
Moves our shared types into a separate module to prevent a circular
dependency.
## Summary
* Adds list permissions as a feature control to SIEM.
* Separates the controls between two, one of which is `access:lists-all` and the other is `access:lists-read`
* Grants SIEM the ability to utilize both depending on which feature mode the space is in.
## Summary
* Adds conflict versioning by exposing the "_version" from the saved object system. It renames "version" to "_version" so that we can use regular "version" later for versioning things for pre-packaged lists abilities.
* Utilizes `t.OutputOf` in the requests and the data types to give us more correctly types
* Removes the `Identity` utility as that is adding confusion and can confuse vs code rather than improves things
* Removes extra types that were causing confusion which was an idiom from io-ts
* Changes the wording of `Partial` by removing that and instead focuses the request types on either client side or server side at this point.
NOTE: The UI can migrate to holding onto the `_version` and then push it back down when it wants to migrate to using the conflict resolution. If the UI does not push it down, then a value of undefined will be used which is indicating that no conflict errors are wanted.
Output example of posting an exception list:
❯ ./post_exception_list.sh
```ts
{
"_tags": [
"endpoint",
"process",
"malware",
"os:linux"
],
"_version": "Wzk4NiwxXQ==",
"created_at": "2020-07-17T18:59:22.872Z",
"created_by": "yo",
"description": "This is a sample endpoint type exception",
"id": "a08795b0-c85f-11ea-b1a6-c155df988a92",
"list_id": "simple_list",
"name": "Sample Endpoint Exception List",
"namespace_type": "single",
"tags": [
"user added string for a tag",
"malware"
],
"tie_breaker_id": "b789ec05-3e0f-4344-a156-0c0f5b6e2f9c",
"type": "detection",
"updated_at": "2020-07-17T18:59:22.891Z",
"updated_by": "yo"
}
```
Output example of posting an exception list item
❯ ./post_exception_list_item.sh
```ts
{
"_tags": [
"endpoint",
"process",
"malware",
"os:linux"
],
"_version": "Wzk4NywxXQ==",
"comments": [],
"created_at": "2020-07-17T18:59:30.286Z",
"created_by": "yo",
"description": "This is a sample endpoint type exception",
"entries": [
{
"field": "actingProcess.file.signer",
"operator": "excluded",
"type": "exists"
},
{
"field": "host.name",
"operator": "included",
"type": "match_any",
"value": [
"some host",
"another host"
]
}
],
"id": "a4f2b800-c85f-11ea-b1a6-c155df988a92",
"item_id": "simple_list_item",
"list_id": "simple_list",
"name": "Sample Endpoint Exception List",
"namespace_type": "single",
"tags": [
"user added string for a tag",
"malware"
],
"tie_breaker_id": "1dc456bc-7aa9-44b4-bca3-131689cf729f",
"type": "simple",
"updated_at": "2020-07-17T18:59:30.304Z",
"updated_by": "yo"
}
```
Output example of when you get an exception list:
❯ ./get_exception_list.sh simple_list
```ts
{
"_tags": [
"endpoint",
"process",
"malware",
"os:linux"
],
"_version": "WzEwNzcsMV0=",
"created_at": "2020-07-17T18:59:22.872Z",
"created_by": "yo",
"description": "Different description",
"id": "a08795b0-c85f-11ea-b1a6-c155df988a92",
"list_id": "simple_list",
"name": "Sample Endpoint Exception List",
"namespace_type": "single",
"tags": [
"user added string for a tag",
"malware"
],
"tie_breaker_id": "b789ec05-3e0f-4344-a156-0c0f5b6e2f9c",
"type": "endpoint",
"updated_at": "2020-07-17T20:01:24.958Z",
"updated_by": "yo"
}
```
Example of the error you get if you do an update of an exception list and someone else has changed it:
```ts
{
"message": "[exception-list:a08795b0-c85f-11ea-b1a6-c155df988a92]: version conflict, required seqNo [1074], primary term [1]. current document has seqNo [1077] and primary term [1]: [version_conflict_engine_exception] [exception-list:a08795b0-c85f-11ea-b1a6-c155df988a92]: version conflict, required seqNo [1074], primary term [1]. current document has seqNo [1077] and primary term [1], with { index_uuid=\"a2mgXBO6Tl2ULDq-MTs1Tw\" & shard=\"0\" & index=\".kibana-hassanabad_1\" }",
"status_code": 409
}
```
Lists are the same way and flavor, they encode the _version the same way that saved objects do. To see those work you run these scripts:
```ts
./post_list.sh
./post_list_item.sh
./find_list.sh
./find_list_item.sh
```
### Checklist
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
* Overview Alerts Histogram stacking defaults to signal.rule.name
Since this is now the default for all AlertsHistograms, I've moved this
default upstream into the histogram itself.
* Replace magic strings with our constant ENDPOINT_LIST_ID
Also replaced a few unintentional uses of this string with the
non-reserved 'endpoint_list_id'.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
* Adds the createEndpointListSchema to the create_endpoint_list_route for API boundary checks
* Adds unit tests to the requests
* Fixes a few bugs found in the find endpoints for types
* Fixes or deletes the skipped tests
* Updated TODO blocks and removed ones that are obsolete
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
* Adds specific endpoint_list REST API and API for abilities to autocreate the endpoint_list if it gets deleted
* Added the check against prepackaged list
* Updated to use LIST names
* Removed the namespace where it does not belong
* Updates per code review an extra space that was added
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Add Frontend components for Value Lists Management Modal
Imports and uses the hooks provided by the lists plugin. Tests coming
next.
* Update value list components to use newest Lists API
* uses useEffect on a task's state instead of promise chaining
* handles the fact that API calls can be rejected with strings
* uses exportList function instead of hook
* Close modal on outside click
* Add hook for using a cursor with paged API calls.
For e.g. findLists, we can send along a cursor to optimize our query. On
the backend, this cursor is used as part of a search_after query.
* Better implementation of useCursor
* Does not require args for setCursor as they're already passed to the
hook
* Finds nearest cursor for the same page size
Eventually this logic will also include sortField as part of the
hash/lookup, but we do not currently use that on the frontend.
* Fixes useCursor hook functionality
We were previously storing the cursor on the _current_ page, when it's
only truly valid for the _next_ page (and beyond).
This was causing a few issues, but now that it's fixed everything works
great.
* Add cursor to lists query
This allows us to search_after a previous page's search, if available.
* Do not validate response of export
This is just a blob, so we have nothing to validate.
* Fix double callback post-import
After uploading a list, the modal was being shown twice. Declaring the
constituent state dependencies separately fixed the issue.
* Update ValueListsForm to manually abort import request
These hooks no longer care about/expose an abort function. In this one
case where we need that functionality, we can do it ourselves relatively
simply.
* Default modal table to five rows
* Update translation keys following plugin rename
* Try to fit table contents on a single row
Dates were wrapping (and raw), and so were wrapped in a FormattedDate
component. However, since this component didn't wrap, we needed to
shrink/truncate the uploaded_by field as well as allow the fileName to
truncate.
* Add helper function to prevent tests from logging errors
https://github.com/enzymejs/enzyme/issues/2073 seems to be an ongoing
issue, and causes components with useEffect to update after the test is
completed.
waitForUpdates ensures that updates have completed within an act()
before continuing on.
* Add jest tests for our form, table, and modal components
* Fix translation conflict
* Add more waitForUpdates to new overview page tests
Each of these logs a console.error without them.
* Fix bad merge resolution
That resulted in duplicate exports.
* Make cursor an optional parameter to findLists
This param is an optimization and not required for basic functionality.
* Tweaking Table column sizes
Makes actions column smaller, leaving more room for everything else.
* Fix bug where onSuccess is called upon pagination change
Because fetchLists changes when pagination does, and handleUploadSuccess
changes with fetchLists, our useEffect in Form was being fired on every
pagination change due to its onSuccess changing.
The solution in this instance is to remove fetchLists from
handleUploadSuccess's dependencies, as we merely want to invoke
fetchLists from it, not change our reference.
* Fix failing test
It looks like this broke because EuiTable's pagination changed from a
button to an anchor tag.
* Hide page size options on ValueLists modal table
These have style issues, and anything above 5 rows causes the modal to
scroll, so we're going to disable it for now.
* Update error callbacks now that we have Errors
We don't display the nice errors in the case of an ApiError right now,
but this is better than it was.
* Synchronize delete with the subsequent fetch
Our start() no longer resolves in a meaningful way, so we instead need
to perform the refetch in an effect watching the result of our delete.
* Cast our unknown error to an Error
useAsync generally does not know how what its tasks are going to be
rejected with, hence the unknown.
For these API calls we know that it will be an Error, but I don't
currently have a way to type that generally. For now, we'll cast it
where we use it.
* Import lists code from our new, standardized modules
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
* Adds the ability for exception lists to be multi-list queried
* Fixes a bunch of script issues where I did not update everywhere I needed to use `ip_list` and deletes an old list that now lives within the new/lists folder
* Fixes a few io-ts issues with Encode Decode while I was in there.
* Adds two more types and their tests for supporting converting between comma separated strings and arrays for GET calls.
* Fixes one weird circular dep issue while adding more types.
You now send into the find an optional comma separated list of exception lists their namespace type and any filters like so:
```ts
GET /api/exception_lists/items/_find?list_id=simple_list,endpoint_list&namespace_type=single,agnostic&filtering=filter1,filter2"
```
And this will return the results of both together with each filter applied to each list. If you use a sort field and ordering it will order across the lists together as if they are one list. Filter is optional like before. If you provide less filters than there are lists, the lists will only apply the filters to each list until it runs out of filters and then not filter the other lists.
If at least one list is found this will _not_ return a 404 but it will _only_ query the list(s) it did find. If none of the lists are found, then this will return a 404 not found exception.
**Script testing**
See these files for more information:
* find_exception_list_items.sh
* find_exception_list_items_by_filter.sh
But basically you can create two lists and an item for each of the lists:
```ts
./post_exception_list.sh ./exception_lists/new/exception_list.json
./post_exception_list_item.sh ./exception_lists/new/exception_list_item.json
./post_exception_list.sh ./exception_lists/new/exception_list_agnostic.json
./post_exception_list_item.sh ./exception_lists/new/exception_list_item_agnostic.json
```
And then you can query these two lists together:
```ts
./find_exception_list_items.sh simple_list,endpoint_list single,agnostic
```
Or for filtering you can query both and add a filter for each one:
```ts
./find_exception_list_items_by_filter.sh simple_list,endpoint_list "exception-list.attributes.name:%20Sample%20Endpoint%20Exception%20List,exception-list-agnostic.attributes.name:%20Sample%20Endpoint%20Exception%20List" single,agnostic
```
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
* Add API functions and hooks for reading and creating the lists index
* Ensure KibanaApiError extends the Error interface
It has a name, so we should type it as such. This way, we can use it
anywhere that an Error is accepted.
* Return an Error from validationEither and thus from our useAsync hooks
Because an io-ts pipeline needs a consistent type across its left value,
and validateEither was returning a string, we were forcing all our
errors to strings. In the case of an API error, however, this meant a
loss of data, since the original error's extra fields were lost.
By returning an Error from validateEither, we can now pass through Api
errors from useAsync and thus use them directly in kibana utilities like
toasts.addError.
* WIP: implements checking for and consequent creation of lists index
This adds most of the machinery that I think we're going to need. Not
featured here:
* lists privileges (stubbed out currently)
* handling when lists is disabled
* tests
* Add frontend plugin for lists
We need this to deteremine in security_solution whether lists is enabled
or not. There's no other functionality here, just boilerplate.
* Fix cross-plugin imports/exports
Now that lists has a client plugin, the optimizer cares about code
coming into and out of it.
By default, you cannot import another plugin's common/ folder into your
own common/ nor public/ folders. This is fixed by adding 'common' to
extraPublicDirs, however: extraPublicDirs need to resolve to modules.
Rather than adding each folder from which we export modules to
extraPublicDirs, I've added common/index.ts and exporting everything
through there.
By convention, I'm adding shared_exports.ts as an index of these exported modules,
and shared_imports.ts is used to import on the other end.
For now, I've left the ad hoc _deps files so as to limit the changes
here, but we should come back through and remove them at some point. NB
that I did remove lists_common_deps as it was only used in one or two
spots.
* Fix test failing due to lack of context
This component now uses useKibana indirectly through useListsConfig.
* Lists and securitySolution require each other's bundles
Without lists being a requiredBundle of securitySolution, we cannot
import its code when the plugin is disabled. The opposite is also true,
but there's no lists "app" to break.
* Fix logic in useListsConfig
Lists needs configuration if the index explicitly does not exist. If it
is true (already exists) or null (lists is disabled or we could not read
the index), we're good.
* useList* behavior when lists plugin is disabled
When the lists plugin is disabled, our calls in useListsIndex become no-ops so that:
* useListsIndex state does not change
* useListsConfig.needsConfiguration remains false as indexExists is
never non-null
This also removes use of our `useIsMounted` hook. Since the effects
we're consuming come from useAsync hooks, state will (already) not be
updated if the component is unmounted.
* Fix warning due to dynamic creation of a styled component
* Revert "Fix warning due to dynamic creation of a styled component"
This reverts commit 7124a8fbd9.
(This was already fixed on master)
* Check user's lists index privileges when determining configuration status
If there is no lists index and the user cannot create it, we will
display a configuration message in lieu of Detections
* Adds a lists hook to read privileges (missing schemae)
* Adds security hook useListsPrivileges to perform and parse the
privileges request
* Updates useListsConfig to use useListsPrivileges hook
* Move lists hooks to their own subfolder
* Redirect to main detections page if lists needs configuration
If:
* lists are enabled, and
* lists indexes DNE, and
* user cannot manage the lists indexes
Then they will be redirected to the main detections page where they'll
be instructed to configure detections. If any of the above is false,
things work as normal.
* Lock out of detections when user cannot write to value lists
Rather than add conditional logic to all our UI components dealing with
lists, we're going the heavy-handed route for now.
* Mock lists config hook in relevant Detections page tests
* Disable Detections when Lists is enabled
This refactors useListsConfig.needsConfiguration to mean:
* lists plugin is disabled, OR
* lists indexes DNE and can't be created, OR,
* user can't write to the lists index
In any of these situations, we want to disable detections, and so we
export that as a single boolean, needsConfiguration.
* Remove unneeded complexity exception
We refactored this to work 👍
* Remove outdated TODO
We link to our documentation, which will describe the lists aspects of
configuration.
## Summary
* Adds "wait_for" to all the create, update, patch, and delete endpoints
* Ran some quick tests against import and the performance still looks acceptable
* Updates the unit tests to reflect the addition
### Checklist
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
## Summary
* Changes the value list imports to use a streaming in model
* Adds a custom light hand spun multi-part parser for the incoming text
* Adds a buffer pause and resume which continues to buffer the incoming data if an async event such as creating a list from the attachment file needs to happen but does not emit the lines until the resume continues.
* Adds a data slicing if the buffer becomes larger than the maximum so that if we begin buffering too quickly within memory we don't blow up the limit of Elastic Search.
* Adds unit tests
### Checklist
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
* adds 2 menu items to alert page, progress on exception modal
* adds enriching
* remove unused useExceptionList()
* implements some types
* move add exception modal files
* Exception builder changes to support latest schema
* Changes to lists plugin schemas and fix api bug
Needed to make the schemas more forgiving. Before this change they required name,
description, etc for creation and update.
The update item API was using the wrong url.
* Adding and editing exceptions working
- Modifies add_exception_modal component
- Creates edit_exception_modal component
- Creates shared comments component
- Creates use_add_exception api hook for adding or editing exceptions
- Updates viewer code to support adding and editing exceptions
- Updates alerts table code to use updated version of add_exception_modal
* fixes duplicate types
* updates os tag input
* fixes comment style
* removes checkbox programatically
* grahpql updates to expose exceptions_list
* Add fetch_or_create_exception_list hook
* fixes data population
* refactor use_add_exception hook, add tests
* fix rebase issues, pending updates to edit modal
* fix edit modal and default endpoint exceptions
* adds second checkbox
* adds signal index stuff
* switches boolean logic
* fix some type errors
* remove unnecesary code
* fixes checkbox logic in edit modal
* fixes recursive prop passing
* addresses comments/fixes types
* Revert schema type changes
* type fixes
* fixes regular exception modal
* fix more type errors, remove console log
* fix tests
* move add exception hook, lint
* close alert checkbox closes alert
* address PR comments
* add type to patch rule call, fix ts errors
* fix lint
* fix merge problems after conflict
* Address PR comments
* undo graphql type change
Co-authored-by: Davis Plumlee <davis.plumlee@elastic.co>
## Summary
Adds these data types to the value based lists end points from [Elasticsearch field data types](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html):
Single value based list types:
* binary
* boolean
* byte
* date
* date_nanos
* date_range
* double
* float
* integer
* ip
* half_float
* keyword
* text
* long
* short
Range value based list types:
* double_range
* float_range
* integer_range
* ip_range
* long_range
Geo value based list types: (caveat is that you cannot query them using other geometry just yet ... you can only these and export them)
* geo_point
* geo_shape
* shape
For importing and exporting different values such as ranges, geo, or single values, this introduces a serialize and deserialize option for the endpoints.
For example if you want to serialize in an ip_range such as 192.168.0.1,192.168.0.3 which has a comma between the two would use the following:
```ts
POST /api/lists
{
"name": "List with an ip range",
"serializer": "(?<gte>.+),(?<lte>.+)",
"deserializer": "{{gte}},{{lte}}",
"description": "This list has ip ranges",
"type": "date_range"
}
```
If you want to serialize in keywords from a list that _only_ match a particular value you would use the following:
```ts
POST /api/lists
{
"id": "keyword_custom_format_list",
"name": "Simple list with a keyword using a custom format",
"description": "This parses the first found ipv4 only",
"serializer": "(?<value>((25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?).){3}(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))",
"deserializer": "{{value}}",
"type": "keyword"
}
```
The serializer is a [named capturing group](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/match) while the deserializer is using [MustacheJS](https://github.com/janl/mustache.js/). The range type, single value types, and geo types all have default captures for their serialize and default mustache templates if none are configured with an endpoint.
The default capture groups and mustache handles for each are:
* shape, geo_point, geo_shape: `(?<lat>.+),(?<lon>.+)`
* date_range: `(?<gte>.+),(?<lte>.+)|(?<value>.+)`
* other ranges are: `(?<gte>.+)-(?<lte>.+)|(?<value>.+)`
* all single data types: `(?<value>.+)`
For ranges you can use both `gte, lte`, and `value` together. If `gte` _and_ `lte` matches it will use that for the greater than, less than elastic range and ignore `value` even if `value` also matched. If _only_ `value` matches and `gte`, `lte` does not match then it will use `value` and put `value` as _both_ the `gte`, and `lte`.
For example, if you are serializing in a list of ip ranges as the list data type, `ip_range` and you have these 3 entries in the file:
```ts
127.0.0.1
127.0.0.2-5
```
The default `serializer` will use `(?<gte>.+)-(?<lte>.+)|(?<value>.+)` and you will get two elastic documents like so:
```ts
{
"_source" : {
"ip_range" : {
"gte" : "127.0.0.1",
"lte" : "127.0.0.1"
}
}
{
"_source" : {
"ip_range" : {
"gte" : "127.0.0.2",
"lte" : "127.0.0.5"
}
}
```
The default mustache handles for each are:
* shape, geo_point, geo_shape: `{{{lat}}},{{{lon}}}`
* date_range: `{{{gte}}},{{{lte}}}`
* other ranges are: `{{{gte}}}-{{{lte}}}`
* all values are: `{{{value}}}`
I use three instead of two handle bars (`{{{` vs.` {{`) so that HTML is not escaped for the lists. You can override and change it if you need or want the escaping.
If during the deserializer phase it detects that a `gte` and `lte` are exactly the same it will still output them as a two items and use the mustache deserialize value. Using the ip-range example above that will be outputted like so since it detects that the lte-gte are exactly the same value:
```ts
127.0.0.1-127.0.0.1
127.0.0.2-127.0.0.5
```
---
Interesting queries to run from the lists scripts folder for testing:
Load some small test files from `./lists/files` for example:
```ts
./import_list_items_by_filename.sh ip_range ./lists/files/ip_range_cidr.txt
./import_list_items_by_filename.sh ip_range ./lists/files/ip_range.txt
./import_list_items_by_filename.sh date ./lists/files/date.txt
./import_list_items_by_filename.sh ip_range ./lists/files/ip_range_mixed.txt
...
```
Export them
```ts
./export_list_items.sh ip_range_cidr.txt
./export_list_items.sh ip_range.txt
./export_list_items.sh date.txt
./export_list_items.sh ip_range_mixed.txt
...
```
Find on them
```ts
./find_list_items.sh ip_range_cidr.txt
./find_list_items.sh ip_range.txt
./find_list_items.sh date.txt
./find_list_items.sh ip_range_mixed.txt
...
```
Find specific values such as:
```ts
./get_list_item_by_value.sh ip_range_mixed.txt 192.168.0.1
./get_list_item_by_value.sh date.txt 2020-08-25T17:57:01.978Z
...
```
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
## Summary
* Removes the feature flag and turns on lists by default
* Applies to both exception lists and value lists
* Removes all scary messages about having it enabled
* Updates the unit tests to work with it on
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
## Summary
This PR tries to start to tie together the server and client changes for exceptions lists.
- Updates graphql types to allow UI access to a rule's `exceptions_list` property
- Updates the exception viewer component to now dynamically take the rule `exceptions_list`, up until now we just had an empty array in it's place
- Updates the viewer logic to check if a rule has an endpoint list associated with it. If it does, then it displays both detections and endpoint UIs (in the viewer), if it does not, then it only displays the detections UI
- Updates the viewer UI to better deal with spacing when an exception list item only has one or two entries (before the and badge with the antennas was stretching passed the exception items to fill the space)
- Updates the detections engine exceptions logic to fetch list items using an exception list's `id` as opposed to it's `list_id`, this now aligns with the UI using the same params on its end
- Adds exception list `type` to information kept by the rule for exception lists
- Updates the exception list type from `string` to `endpoint | detection`
- Updates the exception list _item_ type from `string` to `simple`
- Adds unit tests for the detection engine server side util that fetches the exception list items
* stub out task for the exceptions list packager
* Hits list code and pages
* refactor
* Begin adding saved object and type definitions
* Transforms to endpoint exceptions
* Get internal SO client
* update messaging
* cleanup
* Integrating with task manager
* Integrated with task manager properly
* Begin adding schemas
* Add multiple OS and schema version support
* filter by OS
* Fixing sort
* Move to security_solutions
* siem -> securitySolution
* Progress on downloads, cleanup
* Add config, update artifact creation, add TODOs
* Fixing buffer serialization problem
* Adding cleanup to task
* Handle HEAD req
* proper header
* More robust task management
* single -> agnostic
* Fix OS filtering
* Scaffolding digital signatures / tests
* Adds rotue for creating endpoint user
* Cleanup
* persisting user
* Adding route to fetch created user
* Addings tests for translating exceptions
* Adding test for download API
* Download tweaks + artifact generation fixes
* reorganize
* fix imports
* Fixing test
* Changes id of SO
* integration tests setup
* Add first integration tests
* Cache layer
* more schema validation
* Set up for manifest update
* minor change
* remove setup code
* add manifest schema
* refactoring
* manifest rewrite (partial)
* finish scaffolding new manifest logic
* syntax errors
* more refactoring
* Move to endpoint directory
* minor cleanup
* clean up old artifacts
* Use diff appropriately
* Fix download
* schedule task on interval
* Split up into client/manager
* more mocks
* config interval
* Fixing download tests and adding cache tests
* lint
* mo money, mo progress
* Converting to io-ts
* More tests and mocks
* even more tests and mocks
* Merging both refactors
* Adding more tests for the convertion layer
* fix conflicts
* Adding lzma types
* Bug fixes
* lint
* resolve some type errors
* Adding back in cache
* Fixing download test
* Changing cache to be sized
* Fix manifest manager initialization
* Hook up datasource service
* Fix download tests
* Incremental progress
* Adds integration with ingest manager for auth
* Update test fixture
* Add manifest dispatch
* Refactoring to use the same SO Client from ingest
* bug fixes
* build renovate config
* Fix endpoint_app_context_services tests
* Only index the fields that are necessary for searching
* Integ test progress
* mock and test city
* Add task tests
* Tests for artifact_client and manifest_client
* Add manifest_manager tests
* minor refactor
* Finish manifest_manager tests
* Type errors
* Update integ test
* Type errors, final cleanup
* Fix integration test and add test for invalid api key
* minor fixup
* Remove compression
* Update task interval
* Removing .text suffix from translated list
* Fixes hashes for unit tests
* clean up yarn.lock
* Remove lzma-native from package.json
* missed updating one of the tests
Co-authored-by: Alex Kahan <alexander.kahan@elastic.co>
### Summary
This PR creates the bulk functionality of the exception builder. The exception builder is the component that will be used to create exception list items. It does not deal with the actual API creation/deletion/update of exceptions, it does contain an `onChange` handler that can be used to access the exceptions. The builder is able to:
- accept `ExceptionListItem` and render them correctly
- allow user to add exception list item and exception list item entries
- accept an `indexPattern` and use it to fetch relevant field and autocomplete field values
- disable `Or` button if user is only allowed to edit/add to exception list item (not add additional exception list items)
- displays `Add new exception` button if no exception items exist
- An exception item can be created without entries, the `add new exception` button will show in the case that an exception list contains exception list item(s) with an empty `entries` array (as long as there is one exception list item with an item in `entries`, button does not show)
- debounces field value autocomplete searches
- bubble up exceptions to parent component, stripping out any empty entries
* Add wrapper function to make an AbortSignal arg optional
Components commonly do not care about aborting a request, but are
required to pass `{ signal: new AbortController().signal }` anyway. This
addresses that use case.
* Adds hook for retrieving the component's mount status
This is useful for dealing with asynchronous tasks that may complete
after the invoking component has been unmounted. Using this hook,
callbacks can determine whether they're currently unmounted, i.e.
whether it's safe to set state or not.
* Add our own implemetation of useAsync
This does not suffer from the Typescript issues that the react-use
implementation had, and is generally a cleaner hook than useAsyncTask as
it makes no assumptions about the underlying function.
* Update exported Lists API hooks to use useAsync and withOptionalSignal
Removes the now-unused useAsyncTask as well.
* Add some JSDoc for our new functions
* Add pure API functions and react hooks for value list APIs
This also adds a generic hook, useAsyncTask, that wraps an async
function to provide basic utilities:
* loading state
* error state
* abort/cancel function
* Fix type errors in hook tests
These were not caught locally as I was accidentally running typescript
without the full project.
* Document current limitations of useAsyncTask
* Defines a new validation function that returns an Either instead of a tuple
This allows callers to further leverage fp-ts functions as needed.
* Remove duplicated copyright comment
* WIP: Perform request/response validations in the FP style
* leverages new validateEither fn which returns an Either
* constructs a pipeline that:
* validates the payload
* performs the API call
* validates the response
and short-circuits if any of those produce a Left value.
It then converts the Either into a promise that either rejects with the
Left or resolves with the Right.
* Adds helper function to convert a TaskEither back to a Promise
This cleans up our validation pipeline considerably.
* Adds request/response validations to findLists
* refactors private API functions to accept the encoded request schema
(i.e. snake cased)
* refactors validateEither to use `schema.validate` instead of
`schema.decode` since we don't actually want the decoded value, we just
want to verify that it'll be able to be decoded on the backend.
* Refactor our API types
* Add request/response validation to import/export functions
* Fix type errors
* Continue to export decoded types without a qualifier
* pull types used by hooks from their new location
* Fix errors with usage of act()
* Attempting to reduce plugin bundle size
By pulling from the module directly instead of an index, we can
hopefully narrow down our dependencies until tree-shaking does this for
us.
* useAsyncFn's initiator does not return a promise
Rather than returning a promise and requiring the caller to handle a
rejection, we instead return nothing and require the user to watch the
hook's state.
* success can be handled with a useEffect on state.result
* errors can be handled with a useEffect on state.error
* Fix failing test
Assertion count wasn't updated following interface changes; we've now
got two inline expectations so this isn't needed.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* move last snapshot to inline
* move legacy files to legacy subfolder
* move request types out of legacy
* export Headers from http instead of elasticsearch
* renaming - first pass
* renaming - second pass
* fix core mocks
* adapt new calls
* update generated doc
* fix IT test mocks
* fix new usages
### Summary
This PR is a follow up to #68864 . That PR used a partial to differentiate between new and existing comments, this meant that comments could be updated when they shouldn't. It was decided in our discussion of exception list schemas that comments should be append only. This PR assures that's the case, but also leaves it open to editing comments (via API). It checks to make sure that users can only update their own comments.
* Updates list entry schema, exposes exception list client, updates tests
* create new de list schema and unit tests
* updated route unit tests and types to match new list schema
* updated existing DE exceptions code so it should now work as is with updated schema
* test and types cleanup
* cleanup
* update unit test
* updates per feedback
### Summary
This PR is meant to update the `ExceptionListItemSchema.entries` structure to align with the most recent conversations regarding the need for a more explicit depiction of `nested` fields. To summarize:
- Adds schema validation for requests and responses within `lists/public/exceptions/api.ts`. It was super helpful in catching existing bugs. Anyone that uses the api will run through this validation. If the client tries to send up a malformed request, the request will not be made and an error returned. If the request is successful, but somehow the response is malformed, an error is returned. There may be some UX things to figure out about how to best communicate these errors to the user, or if surfacing the raw error is fine.
- Updates `entries` structure in lists plugin api
- Updates hooks and tests within `lists/public` that make reference to new structure
- Updates and adds unit tests for updated schemas
- Removes unused temporary types in `security_solution/public/common/components/exceptions/` to now reference updated schema
- Updates UI tests
- Updates `lists/server/scripts`
## Summary
Fixes io-ts formatter bugs for REST and validation by:
* First trying to get the correct key from the io-ts context. If no keys are found, then it will fall back on trying to get the first name from the context.
* If the key is a value and an object then this will do a `JSON.stringify()` on the value object
* This fixes a few places where `formatError` was not being used within the code base resulting in `[object Object]` within the validations to show up.
### Checklist
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
### Summary
This is part of a series of upcoming changes to the exception list item structure. This PR focuses solely on updating exception_item.comment. The hope is to keep these PRs relatively small.
- Updates exception_item.comment structure which was previously a string to exception_item.comments which is an array of { comment: string; created_by: string; created_at: string; }
- Adds a few unit tests server side
- Fixes some minor misspellings
- Updates ExceptionViewer component in the UI to account for new structure
* add babel support for export type
* bump ts version to 3.9.3
* rebuild kbn-pm
* bump typescript-eslint
* fix error in security plugin UI
* check export as works
* fix app migration type
* use correct test subj attribute
* fix errors from the old PR
* embeddable is already passed in props
* explicitly define type of fetch
* add some types for viz
* fix fetch type p.2
* add null to allow spreading without type errors due to override
* add type guard to fix type error
* cast to any, since cannot assign unknown
* add timestamp to known types
* fix type error in fetch
* fix type error. id is always defined in attibutes
* declare a type
* move ts-ignore to the lines with errors
* declare tuple type explicitly
* mute type error. cannot assign unknown
* fix errors. id is always defined
* fix error type
* fix override errors. id is always defined
* fix error. extends any doesn't work anymore
* fix type error. type is always defined
* env doesn't always contain values
* fix type error
* cast to string
* add: logs is already declared in getNodeLogsUrl
* state is already passed in props
* fix some errors in timelion
* number of fragments is always defined
* 'absolute' is not just string, but value
* TEMP: option is always defined
* always true if cast to promise manually
* both props are always defined
* explicitly define returned SO type
* workaround type
* bump tslib to be compatible with ts v3.9
* test private property
* rebuild kbn-pm
* Fix ts errors for beats management
* Fix type inference broken by the TS 3.9 upgrade
* Fix ingest manager saved object attributes typings
* Fix TS errors in cross_cluster_replication and index_management.
* Fix TS error in Watcher.
* roll back colorRange wrong type
* fix security plugin types
* TypeScript 3.9 fixes for APM
* Fix ColorRange types.
* fix actions & alerts errors. ByGidi
* fix lists error
* More APM fixes
* Remove paramaterization from `removeEmpty in agent config SettingsPage component (it's only used there and doesn't need to be parameterized.)
* Add option chain for case in registerTransactionDurationAlertType
* Cast `overallValue` in transform_metrics_chart
* Use more specific type for custom link filters
* Add more option chaining for local UI filters buckets response
* Remove unused parameters from routes
* Fix getProjection type parameter
* Use destructuring in serviceNodesLocalFiltersRoute to hide `never` error
* Revert `UnionToIntersection` change in `AggregationResponseMap`
Fixes#67804.
* fix platform type error
* Fix visualizations types.
* Fix data plugin types.
* bump TS version to 3.9.5
* Fix telemetry TS errors
* Fix dashboard code
* Adding Canvas Fixes for TS 3.9
* Fix case and security_solution types
* roll back to the old export syntax. new one might cause problems in api-extractor
* update docs
* Fix timelion code
* Fix meta
* Fix types
* fix type errors om ingest_manager
* bump babel deps
* enable private props & methods syntax
* update kbn-pm dist
* whitelist 0BSD license
* use @babel/plugin-proposal-private-methods in default set as well
* disable new babel plugins
* Revert "disable new babel plugins"
This reverts commit 04d959431d.
* cleanup security_solution types
* Fixes type error for newer TypeScript
* update docs
Co-authored-by: Nicolas Chaulet <nicolas.chaulet@elastic.co>
Co-authored-by: Felix Stürmer <stuermer@weltenwort.de>
Co-authored-by: CJ Cenizal <cj@cenizal.com>
Co-authored-by: Larry Gregory <larry.gregory@elastic.co>
Co-authored-by: Nathan L Smith <smith@nlsmith.com>
Co-authored-by: Walter Rafelsberger <walter@elastic.co>
Co-authored-by: Luke Elmers <luke.elmers@elastic.co>
Co-authored-by: Alejandro Fernández Haro <alejandro.haro@elastic.co>
Co-authored-by: Tim Roes <tim.roes@elastic.co>
Co-authored-by: Clint Andrew Hall <clint.hall@elastic.co>
Co-authored-by: Patryk Kopycinski <contact@patrykkopycinski.com>
Co-authored-by: FrankHassanabad <frank.hassanabad@elastic.co>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
### Summary
- Adds missing unit tests for relevant files missing them
- Changes filter search to fire request on 'Enter'
- Breaks out the main ExceptionViewer component into smaller components to make more readable and better tested
- Updates utility bar to have the specific list description text next to it as proposed by @spong in #68294 (comment)
- Adds loading state any time async request occurs
- Now fetches list on list type toggle (if user selects to view either only detections or endpoint items), before was simply filtering already fetched items
## Summary
* Smaller follow ups and bug fixes from: https://github.com/elastic/kibana/pull/68127
* Added unknown to `findDifferencesRecursive`
* Added linter rule to catch NodeJS code in the common folders for both `lists` and `security_solution`
* Removed the Hapi server type from the common folder of lists
### Checklist
* Added unknown to the correct locations
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
### Summary
This PR is a follow up to #68027 . It brings it all together to complete the exceptions viewer component. This component is meant to display all exception items and allow a user to create, edit, delete, and search these exception items.
- Moves ExceptionItem (from part 1) into its own folder
- Adds exceptions_viewer_header component that includes the search, list toggle, and add exception buttons
- Adds actual ExceptionViewer component
- Updates the useExceptionList hook refresh function logic. Noticed that the previous version was creating some issues
## Summary
* https://github.com/elastic/siem-team/issues/646
* Converts the detection rules and REST to use io-ts
* Removes their joi counterparts
* Updates all tests to use it
* Fixes a bug with the risk_score that was being sent in as a string from the UI instead of a number
* Fixes a bug within the exactCheck validating where it can now accept null value types for optional body messages.
* Fixes a bug in the FindRoute where it did not send down fields from REST
* Changes the lists plugin to utilize the io-ts types from siem rather than having them duplicated.
* Makes some stronger validations
* Adds a lot of codecs
**Things to look out for:**
* Generic testing to ensure I didn't break something that was not part of the tests.
* Fix for the risk_score from string to number is in:
```
x-pack/plugins/security_solution/public/alerts/components/rules/step_about_rule/index.test.tsx
```
* Fix for the exact check (unit tests are written and added)
```
x-pack/plugins/security_solution/public/alerts/components/rules/step_about_rule/index.test.tsx
```
* Within all the types I added are there any misspelled things or copy-pasta mistakes with strings:
x-pack/plugins/security_solution/common/detection_engine/schemas/types
* Fix for `find_rules_route.ts:58`
```
x-pack/plugins/security_solution/server/lib/detection_engine/routes/rules/find_rules_route.ts
```
**Follow on things that this PR doesn't do we need to:**
* Add linter rule to forbid NodeJS code within common section
* The `[object Object]` formatter issues seen in the code such as:
```
// TODO: Fix/Change the formatErrors to be better able to handle objects
'Invalid value "[object Object]" supplied to "note"',
```
* Formatter issues such as: `'Invalid value "" supplied to ""'`
* Remove the hapi server object from lists plugin
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
* rename siem to security_solution
* rename siem to security solution inside of code
* rename translation keys
* fix snapshot
* replace siem for security solution in tutorial
* missing translation to be renamed
* fix types for api test integration
* updates runner file to match the new path
* change category for kibana settings
* miss renaming in advance settings
* fixes cypress tests
* fix api integration test
* fix new translation
* fix unit test
* update translation i18n
* update translation i18n II
Co-authored-by: Gloria Hornero <snootchie.boochies@gmail.com>
### Summary
- Updates exception list hooks to include filtering options and updates corresponding unit tests.
- Adds refreshList callback to hook that fetches the list and its items
- Updates hooks tests to test onError callback
- Updates tests to use type checking more effectively per feedback from @FrankHassanabad (thanks!)
* introduce lists plugin for use by executor
* adds getListClient function on setup
* refactors searchAfterBulkCreate to integrate with the lists plugin so we only generate signals from events not in the list
* fixes type check issues
* fixes unit tests, adds field and other parameters for using lists in executor.
* cleaning up types and exports, updates to match new contracts with lists client from master
* prior to this commit the refactored while loop was doing more search after loops than it needed to and this fixes two bugs in the list filter function where we were returning the wrong count, and we were not accessing the right field on the event
* exception lists are optional
* use exceptions list format, this works with given sample query in scripts
* updates tests and fixes type issues
* updates README doc in detection engine with example for rule with list exception
* adds one test and removes commented out code
* fix sample rule json from 30s to 5m
* fix sample rule json from 30s to 5m
* remove unused import
* more cleanup
* e2e test for prepackaged rules was failing because lists was undefined in the siem plugin and was preventing the registration of the rule alert type. I removed this but once lists is ready for prime time we should consider adding the null check back
* can't reuse the same env var since the tests are setting the ELASTIC_XPACK_SIEM_LISTS_FEATURE env var to true without enabling the lists plugin
* fixes from pr review, still needs more TLC
* exports listspluginsetup type from top-level in lists plugin, fixes logic for empty exceptions list, updates types
* utilize type.is to remove as casting, also do null checks and throw an error when exceptionItem is malformed. This will change in the very near future once the new json format for exception lists is incorporated
* fix type issues after merging master into branch
* update mock
* remove bad null check for ml plugin before registering rule alert type in siem plugin
* prettier linting
* adds test for filter events with list
* pr comments
* adds logic for included vs excluded and updates tests
* update test cases for search after bulk create to default to included for exception lists
* filter out non-list exception items from the loop
## Summary
Adds the REST and API routes for find and filter for exception lists and value lists
* Fixes bugs with string parameters for the _find with exception lists
* Adds the _find for the value based lists
* More scripts for how to filter things for both list values and exception lists
* Misc type script fixes
* Adds a cursor to move from the previous page to the next page
* Adds name space 'agnostic' vs. 'single' feature for exception_lists
**REST API's:**
```ts
POST /api/lists/_find
POST /api/lists/items/_find
POST /api/exception_lists/_find
POST /api/exception_lists/items/_find
```
**Parameters you can send:**
* sort
* sort_order
* filter
* page
* per_page
* list_id (for list items only and required)
* cursor (for finding the next page or advancing to deep pages)
**See test scripts below:**
```sh
find_exception_list_items_by_filter.sh
find_exception_lists_by_filter.sh
find_list_items.sh
find_list_items_with_cursor.sh
find_list_items_with_sort.sh
find_list_items_with_sort_cursor.sh
find_lists.sh
find_lists_with_cursor.sh
find_lists_with_filter.sh
find_lists_with_sort.sh
find_lists_with_sort_cursor.sh
```
### Checklist
Note: Unit tests are left out as this is blocking people but I will be adding tests as this is being reviewed unless someone needs these features now. This is still all behind a feature flag and considered to be in the area of proof of concept and not production ready until more tests and end to tests are added.
- [ ] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
Added some basic functionality to help exception list UI work move forward. Wired up to exception list api and created hooks. This PR includes:
- UI api functions for basic exception list and exception list item CRUD
- useExceptionList hook to fetch the list and its items
- usePersistExceptionList hook to create or update an exception list
- usePersistExceptionListItem hook to create or update an exception item
- list_plugin_deps.tsx in the siem folder to import the lists plugin hooks
## Summary
See for more details:
https://github.com/elastic/kibana/issues/65938
Adds pieces of the `exception list` and `exception list item` and refactors/cleans the code up where I had parts incorrect with little things such as the javascript library io-ts. Some unit tests were added but I am holding off until more of the operations solidify before adding the unit tests. Everything is still behind a feature flag that must be enabled and not advised still at this point to use so I feel ok pushing these parts forward.
Adds to the API:
- Create exception list
- Read exception list
- Update exception list
- Delete exception list (and exception list items that are associated with it)
- Create exception list item
- Find exception list (/_find)
- Read exception list item
- Update exception list item
- Delete exception list items individually
- Find exception list item (/_find)
What is still missing from the REST and client API?
- Patch exception list
- Patch exception list item
- Bulk versions of everything
- Import/Export options for these exception lists and list items
### Manual testing and REST API endpoints
Go here:
```sh
/projects/kibana/x-pack/plugins/lists/server/scripts
```
See the files:
```sh
delete_all_exception_lists.sh
delete_exception_list.sh
delete_exception_list_by_id.sh
delete_exception_list_item.sh
delete_exception_list_item_by_id.sh
exception_lists
find_exception_list_items.sh
find_exception_lists.sh
get_exception_list.sh
get_exception_list_by_id.sh
get_exception_list_item.sh
get_exception_list_item_by_id.sh
post_exception_list.sh
post_exception_list_item.sh
update_exception_list.sh
update_exception_list_item.sh
```
Ensure you first run:
```sh
./hard_reset
```
and ensure you have setup your kibana.dev.yml to have:
```yml
# Enable lists feature
xpack.lists.enabled: true
xpack.lists.listIndex: '.lists-frank'
xpack.lists.listItemIndex: '.items-frank'
```
Then you can use the above scripts to create, read, update, and delete exception list and exception list items as well as perform find commands against them all.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
Note: Some but limited unit tests at this point.
* line restricted zones for export exressions
* more robust rule
* fix or mute eslint errors
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Fixes up the API contracts to work outside of a request and as a regular plugin.
* Removes space and request stuff that is not needed
* Adds in plugin ability with space id and user name being pushed down
* Adds unit tests to the schema for input/output validation
* Changes the mocks to use a `file_name.mock.ts` pattern
* Introduces io-ts partials _carefully_ where I get both the partials and the required undefined in the types
* Introduces an Identity type to remove weird intersection types and make plain types when using io-ts.
* I Introduces a RequiredKeepUndefined in order to work with partials and keep the undefined as required for when the type is used directly within the code. This makes it simpler to force new functions/methods to have to push down `undefined`
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
* Removes plugin dependencies for better integration outside of Requests such as alerting
* Adds more unit tests
* Fixes more TypeScript types to be more normalized
* Makes this work with the user 'elastic' if security is turned off
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
## Summary
* Adds large list support using REST endpoints.
Status:
---
* Currently ready to be merged behind the feature flag of it being disabled with ongoing work happening after it is merged.
* REST Endpoints shouldn't have large refactoring at this point
* Team meeting occurred where the pieces were discussed in person.
What is left?
---
- [ ] Add other data types. At the moment `ip` and `keyword` are the two types of lists. See: https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html
- [x] Unit tests
- [x] Lots of misc TODO's in the code base still
- [ ] Import loads everything into memory first when it should attempt streaming
- [ ] Add end to end backend tests
- [x] Add transform and io-ts validation for returns
Testing
---
Ensure you set this in your ENV before starting Kibana:
```ts
export ELASTIC_XPACK_SIEM_LISTS_FEATURE=true
```
Download or create a large list file such as this one filled with IP's:
https://cinsscore.com/list/ci-badguys.txt
Go to your REST endpoint folder of scripts:
```ts
cd kibana/x-pack/plugins/lists/server/scripts
```
Do a hard reset:
```ts
./hard_reset
```
Then import it as either a data type of `ip`:
```ts
./import_list_items_by_filename.sh ip ~/Downloads/ci-badguys-smaller.txt
```
Or as a `keyword`
```ts
./import_list_items_by_filename.sh keyword ~/Downloads/ci-badguys-smaller.txt
```
Then you can export it through:
```ts
./export_list_items.sh ci-badgusy-smaller.txt
```
For all the other endpoints and testing of the CRUD operations you have access to:
```ts
delete_all_lists.sh
delete_list.sh
delete_list_index.sh
delete_list_item.sh
delete_list_item_by_id.sh
delete_list_item_by_value.sh
export_list_items.sh
export_list_items_to_file.sh
get_list.sh
get_list_item_by_id.sh
get_list_item_by_value.sh
import_list_items.sh
import_list_items_by_filename.sh
lists_index_exists.sh
patch_list.sh
patch_list_item.sh
post_list.sh
post_list_index.sh
post_list_item.sh
```
### Checklist
Delete any items that are not applicable to this PR.
- [ ] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
### For maintainers
- [ ] This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)
