* Adds new SO type for persisting our signals migrations
* WIP: Migration status SO client
Trying to forge a patterrn using io-ts to validate at runtime. I think
I've got it working but I want to refactor the pipeline out into a
reusable function(s).
* Implements our SavedObjects service for signals migrations
* Defines a simple client that delegates to the base SO client with
our SO type
* Defines a service that consumes the simpler client, adding validations
and data transforms on top.
* Refactoring migration code to work with saved objects
As opposed to the previous ephemeral, encoded tokens, we now retrieve migration
info from saved objects.
At the API level, this means that both the create and finalize endpoints
receive a list of concrete indices. No more passing around tokens.
As both endpoints are idempotent, users can hammer them as much as they
want with the same lists of indices. Redundant creates and finalizes
will be met with inline 400 messages, and as one continues to poll the
finalize endpoint they should see more and more indices respond with
"completed: true"
* Fixing integration tests first, and anything upstream breaking them
* Clean up API integration tests
* standardize assignment of responses (with types)
* deletes migration SOs as test cleanup
* Split API tests into separate files
This was getting big and unwieldy; this splits these into one file per
endpoint.
* Refactor: split existing migration service functionality into atomic functions
This will allow us to repurpose the service to compose more
functionality and be more specifically useful, while keeping the
component logic separate.
* WIP: moving logic into migrationService.create
* Splitting get_migration_status into component functions
getMigrationStatus was really two separate aggregations, so I split them
out and we recompose them in the necessary routes.
* Move finalization logic into function
* migrationService exposes this as .finalize()
* adds an error field to our migration SO
* We currently only have one error that we persist there, but it would
be very time-consuming to track down that information were it not
there.
* Adds function for migration "deletion" logic
* migrationService leverages this function
* adds new boolean to our savedObject
* deletes unused function (deleteMigrationSavedObject)
* Adds route for soft-deletion of migrations
* Updating tests related to migration status
* Adding/updating mocks/unit tests necessary to satisfy the things I
need to test
* I mainly wanted to test that the the status endpoint filtered out the
deleted migrations; this was accomplished with a unit test after
fleshing out some mocks/sample data.
* Move old migration service tests to the relevant function tests
This logic was previously moved out into component functions; this moves
the tests accordingly.
* Add some unit tests around our reindex call
* Fix create migration route tests
Mocks out our migration functions, rather than stubbing ES calls
directly.
* Updates finalize route unit tests
Addresses functionality that hasn't been moved to finalizeMigration()
* Unit tests our finalization logic
Fixes a bug where we weren't accounting for soft-deleted migrations.
ALso updates our test migration SO to have a status of 'pending' as
that's a more useful default.
* Fixes finalization integration tests
These were failing due:
* a change in the migration status API response
* a bug I introduced in the finalize route
* Adds tests for our migration deletion endpoint
* unit tests
* API integration tests
* Caught/fixed bug with deleting a successful migration
* Fixes types
Removes unused code.
* Prevent race condition due to template rollover during migration
If a user has an out of date index (v1) relative to the template (v2), but the
template itself is out of date (newest is v3), then it's possible that
the template is rolled over to v3 after the v1-v2 migration has been
created but before the new index has been created.
In such a case, the new index would receive the v3 mappings but would
incorrectl be marked as v2. This shouldn't necessarily be an issue, but
it's an unnecessary state that can easily be prevented with the guard
introduced here.
* Add real usernames to migration savedObjects
In addition to the SOs themselves giving us observability into what
migration actions were performed, this gives us the additional info of
_who_ performed the action.
* Index minimal migration SO fields needed for current functionality
* Add additional migration info to status endpoint
This will allow users to finalize a migration if they've lost the
response to their POST call.
* Finalize endpoint receives an array of migration IDs, not indices
This disambiguates _which_ migrations we were finalizing if you passed
an index (which was previously: the most recent migration).
* Fix type errors in tests after we threaded through username
* Update responsibilities of migration finalize/delete endpoints
Discussions with @marshallmain lead to the following refactor:
* finalize does not delete tasks
* finalize only applies cleanup policy to a failed migration
* delete takes an array of migration ids (like finalize)
* delete hard-deletes the SavedObject of a completed (failed or
successful) migration
This gives a bit more flexibility with the endpoints, as well as
disambiguates the semantics: it just deletes migrations!
* Fix tests that were broken during refactoring
* Fix type errors
I removed some logic here but forgot the imports :(
* Move outdated integration test
In the case of a successful migration, application of the cleanup policy
is done by the deletion endpoint. In the interest of data preservation,
we do not delete a sourceIndex unless it is explicitly deleted.
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Fixes different bugs/issues when using exceptions with value based lists for both the UI, the backend, and the large value based lists. See https://github.com/elastic/kibana/issues/79516, but this also fixes several other bugs found mentioned below.
For the front end UI:
* Adds the ability to specify value based lists that are IP Ranges when the source event is an IP. Before you could only match IP to IP and the IP Ranges lists could not be used.
* Breaks down a few functions into smaller functions for unit test writing abilities.
You can now add ip ranges as list values for the UI when before it would not show up:
<img width="1035" alt="Screen Shot 2020-12-07 at 2 15 39 PM" src="https://user-images.githubusercontent.com/1151048/101406552-d6819b00-3896-11eb-9fb5-4c7c2ad93b2e.png">
For value based lists:
* Fixes text data type to use "and" between matching using `operator: 'and'` and changes it from a `terms query to a `match` query
* Adds new API for searching against types called `searchListItemByValues ` so that numeric, text, array based, and other non-stringable types can be sent and then the value based lists will push that to ElasticSearch. This shifts as many corner cases and string/numeric coercions to ElasticSearch rather than Kibana client side code.
* Adds ability to handle arrays within arrays through a `flatten` call.
* Utilizes the `named queries` from ElasticSearch for the new API so that clients can get which parts matched and then use that for their exception list logic rather than in-memory string to string checks. This fixes CIDR and ranges as well as works with arrays.
For the backend exception lists that used value based lists:
* Broke down the `filterEventsAgainstList` function into a folder called `filters` and the functions into other files for better unit based testing.
* Changed the calls from `getListItemByValues` to `searchListItemByValues` which can return exactly what it matched against and this will not break anyone using the existing REST API for `getListItemByValues` since that REST API and client side API stays the same.
* Cleaned up extra promises being used in a few spots that async/await automatically will create.
* Removed the stringabilities and stringify in favor of just a simpler exact check using `JSON.stringify()`
For the tests:
* Adds unit tests to broken down functions
* Adds ip_array, keyword_array, text_array, FTR tests for the backend.
* Adds more CIDR and range based FTR tests for the backend.
* Unskips and fixes all the numeric tests and range tests that could not operate previously from bugs.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
- [ ] Any UI touched in this PR is usable by keyboard only (learn more about [keyboard accessibility](https://webaim.org/techniques/keyboard/))
- [ ] Any UI touched in this PR does not create any new axe failures (run axe in browser: [FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/), [Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US))
- [ ] This renders correctly on smaller devices using a responsive layout. (You can test this [in your browser](https://www.browserstack.com/guide/responsive-testing-on-local-server))
- [ ] This was checked for [cross-browser compatibility](https://www.elastic.co/support/matrix#matrix_browsers)
* Separated out service layer for trusted apps.
* Improved the type structure a bit to avoid using explicit string literals and to add possibility to return OS specific parts of trusted app object in type safe manner.
* Added support for mapping of trusted app to exception item and back.
* Changed schema to support signer in the API.
* Renamed utils to mapping.
* Exported some types in lists plugin and used them in trusted apps.
* Added tests for mapping.
* Added tests for service.
* Switched deletion to use exceptions for not found case.
* Added resetting of the mocks in service layer tests.
* Added handlers tests.
* Refactored mapping tests to be more granular based on the case.
* Restored lowercasing of hash.
* Added schema tests for signer field.
* Removed the grouped tests (they were split into tests for separate concerns).
* Corrected the tests.
* Lowercased the hashes in the service test.
* Moved the lowercasing to the right location.
* Fixed the tests.
* Added test for lowercasing hash value.
* Introduced OperatingSystem enum instead of current types.
* Removed os list constant in favour of separate lists in places that use it (each place has own needs to the ordering).
* Fixed the missed OperatingSystem enum usage.
* bump version to 4.1.1-rc
* fix code to run kbn bootstrap
* fix errors
* DO NOT MERGE. mute errors and ping teams to fix them
* Address EuiSelectableProps configuration in discover sidebar
* use explicit type for EuiSelectable
* update to ts v4.1.2
* fix ts error in EuiSelectable
* update docs
* update prettier with ts version support
* Revert "update prettier with ts version support"
This reverts commit 3de48db3ec.
* address another new problem
Co-authored-by: Chandler Prall <chandler.prall@gmail.com>
## Summary
Adds support to the end to end (e2e) functional test runner (FTR) support for rule runtime tests as well as 213 tests for the exception lists which include value based lists. Previously we had limited runtime support, but as I scaled up runtime tests from 5 to 200+ I noticed in a lot of areas we had to use improved techniques for determinism.
The runtime support being added is our next step of tests. Up to now most of our e2e FTR tests have been structural testing of REST and API integration tests. Basically up to now 95% tests are API structural as:
* Call REST input related to a rule such as GET/PUT/POST/PATCH/DELETE.
* Check REST output of the rule, did it match expected output body and status code?
* In some rare cases we check if the the rule can be executed and we get a status of 'succeeded'
With only a small part of our tests ~5%, `generating_signals.ts` was checking the signals being produced. However, we cannot have confidence in runtime based tests until the structural tests have been built up and run through the weeks against PR's to ensure that those are stable and deterministic.
Now that we have confidence and 90%+ coverage of the structural REST based tests, we are building up newer sets of tests which allow us to do runtime based validation tests to increase confidence that:
* Detection engine produces signals as expected
* Structure of the signals are as expected, including signal on signals
* Exceptions to signals are working as expected
* Most runtime bugs can be TDD'ed with e2e FTR's and regressions
* Whack-a-mole will not happen
* Consistency and predictability of signals is validated
* Refactoring can occur with stronger confidence
* Runtime tests are reference points for answering questions about existing bugs or adding new ones to test if users are experiencing unexpected behaviors
* Scaling tests can happen without failures
* Velocity for creating tests increases as the utilities and examples increase
Lastly, this puts us within striking distance of creating FTR's for different common class of runtime situations such as:
* Creating tests that exercise each rule against a set of data criteria and get signal hits
* Creating tests that validate the rule overrides operate as expected against data sets
* Creating tests that validate malfunctions, corner cases, or misuse cases such as data sets that are _all_ arrays or data sets that put numbers as strings or throws in an expected `null` instead of a value.
These tests follow the pattern of:
* Add the smallest data set to a folder in data.json (not gzip format)
* Add the smallest mapping to that folder (mapping.json)
* Call REST input related to exception lists, value lists, adding prepackaged rules, etc...
* Call REST input related endpoint with utilities to create and activate the rule
* Wait for the rule to go into the `succeeded` phase
* Wait for the N exact signals specific to that rule to be available
* Check against the set of signals to ensure that the matches are exactly as expected
Example of one runtime test:
A keyword data set is added to a folder called "keyword" but you can add one anywhere you want under `es_archives`, I just grouped mine depending on the situation of the runtime. Small non-gzipped tests `data.json` and `mappings.json` are the best approach for small focused tests. For _larger_ tests and cases I would and sometimes do use things such as auditbeat but try to avoid using larger data sets in favor of smaller focused test cases to validate the runtime is operating as expected.
```ts
{
"type": "doc",
"value": {
"id": "1",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:00:53.000Z",
"long": 1
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "2",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:01:53.000Z",
"long": 2
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "3",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:02:53.000Z",
"long": 3
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "4",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:03:53.000Z",
"long": 4
},
"type": "_doc"
}
}
```
Mapping is added. Note that this is "ECS tolerant" but not necessarily all ECS meaning I can and will try to keep things simple where I can, but I have ensured that `"@timestamp"` is at least there.
```ts
{
"type": "index",
"value": {
"index": "long",
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"long": { "type": "long" }
}
},
"settings": {
"index": {
"number_of_replicas": "1",
"number_of_shards": "1"
}
}
}
}
```
Test is written with test utilities where the `beforeEach` and `afterEach` try and clean up the indexes and load/unload the archives to keep one test from effecting another. Note this is never going to be 100% possible so see below on how we add more determinism in case something escapes the sandbox.
```ts
beforeEach(async () => {
await createSignalsIndex(supertest);
await createListsIndex(supertest);
await esArchiver.load('rule_exceptions/keyword');
});
afterEach(async () => {
await deleteSignalsIndex(supertest);
await deleteAllAlerts(supertest);
await deleteAllExceptions(es);
await deleteListsIndex(supertest);
await esArchiver.unload('rule_exceptions/keyword');
});
describe('"is" operator', () => {
it('should filter 1 single keyword if it is set as an exception', async () => {
const rule = getRuleForSignalTesting(['keyword']);
const { id } = await createRuleWithExceptionEntries(supertest, rule, [
[
{
field: 'keyword',
operator: 'included',
type: 'match',
value: 'word one',
},
],
]);
await waitForRuleSuccess(supertest, id);
await waitForSignalsToBePresent(supertest, 3, [id]);
const signalsOpen = await getSignalsById(supertest, id);
const hits = signalsOpen.hits.hits.map((hit) => hit._source.keyword).sort();
expect(hits).to.eql(['word four', 'word three', 'word two']);
});
});
```
### Changes for better determinism
To support more determinism there are changes and utilities added which can be tuned during any sporadic failures we might encounter as well as better support unexpected changes to other Elastic Stack pieces such as alerting, task manager, etc...
Get simple rule and others are now defaulting to false, meaning that the structural tests will no longer activate a rule and run it on task manger. This should cut down on error outputs as well as reduce stress and potentials for left over rules interfering with the runtime rules.
```ts
export const getSimpleRule = (ruleId = 'rule-1', enabled = false): QueryCreateSchema => ({
```
Not mandatory to use, but for most tests that should be runtime based tests, I use this function below which will enable it by default and run it using settings such as `type: 'query'`, `query: '*:*',` `from: '1900-01-01T00:00:00.000Z'`, to cut down on boiler plate noise. However, people can use whatever they want out of the grab bag or if their test is more readable to hand craft a REST request to create signals, or if they just want to call this and override where they want to, then 👍 .
```ts
export const getRuleForSignalTesting = (index: string[], ruleId = 'rule-1', enabled = true)
```
This waits for a rule to succeed before continuing
```ts
await waitForRuleSuccess(supertest, id);
```
I added a required array of id that _waits_ only for that particular id here. This is useful in case another test did not cleanup and you are getting signals being produced or left behind but need to wait specifically for yours.
```ts
await waitForSignalsToBePresent(supertest, 4, [id]);
```
I only get the signals for a particular rule id using either the auto-generated id or the rule_id. It's safer to use the ones from the auto-generated id but either of these are fine if you're careful enough.
```ts
const signalsOpen = await getSignalsById(supertest, id);
const signalsOpen = await getSignalsByIds(supertest, [createdId]);
const signalsOpen = await getSignalsByRuleIds(supertest, ['signal-on-signal']);
```
I delete all alerts now through a series of steps where it properly removes all rules using the rules bulk_delete and does it in such a way that all the API keys and alerting will be the best it can destroyed as well as double check that the alerts are showing up as being cleaned up before continuing.
```ts
deleteAllAlerts()
```
When not explicitly testing something structural, prefer to use the utilities which can and will do retries in case there are over the wire failures or es failures. Examples are:
```ts
installPrePackagedRules()
waitForRuleSuccess()
importFile() // This does a _lot_ of checks to ensure that the file is fully imported before continuing
```
Some of these utilities might still do a `expect(200);` but as we are and should use regular structural tests to cover those problems, these will probably be more and more removed when/if we hit test failures in favor of doing retries, waitFor, and countDowns.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
* ExceptionIdentifiers -> ExceptionListIdentifiers
* Pass refreshRule through to alert context menu
* Fix type errors and rename refreshRule to onRuleChange for consistency
* Finish adding .lower to exceptionable fields
* Add back migrations
* .lower -> .caseless
* Add separate field for os type
* updates
* Type updates
* Switch over to osTypes
* get rid of _tags
* Add tests for schema validation
* Remove remaining references to _tags
* Another round of test fixes
* DefaultArray tests
* More test fixes
* Fix remaining test failures
* types / tests
* more test updates
* lowercase os values
* Address feedback + fix test failure
* tests
* Fix integration test
* process.executable.path -> process.executable.caseless
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
This is a bug that was introduced by moi in 76537. This previous PR added a sort_field and sort_order to the call for fetching exception lists' items so that the exception item order in the viewer wouldn't jump around any time there was an update. I noticed however that when a rule had both endpoint and detection lists associated with it, when trying to fetch items from both types of lists, the following error shows:
```
"Unable to sort multiple types by field created_at, not a root property"
```
## Summary
Mistmatch caught between the io-ts type and the corresponding typescript enum. Currently, io-ts does not have support for enums - as a workaround I had made a matching typescript enum type. Tests were added to try to ensure the two stayed in sync, but didn't do a straight up comparison of the two.
Updated the tests to now error if the keys do not match.
## Summary
**Components affected:** ExceptionsViewer
**Current behavior:**
- when a user edits an exception item, the order of the exception items in the viewer changes. This creates confusion and looks like updates weren't applied (even though they were, just item order changed)
**New behavior:**
- when a user edits an exception item, the order of the exception items in the viewer don't change. Sort order is now based on `created_at`
* bump ts to v4
* MOAR RAM
* fix type errors for OSS
* first pass on x-pack errors
* second pass on x-pack type errors
* 3rd pass on x-pack type-errors
* mute errors if complex cases
* don't delete if spread suffices
* mute other complex cases
* make User fields optional
* fix optional types
* fix tests
* fix typings for time_range
* fix type errors in x-pack/tests
* rebuild kbn-pm
* remove leftovers from master update
* fix alert tests
* [Telemetry Checker] TS4 Fixes
* bump to 4.0.1-rc
* fix new errors in master
* bump typescript-eslint to version supporting TS v4 syntax
* fix merge commit errors
* update to the stable TS version 4.0.2
* bump ts-eslint to version supporting ts v4
* fix typo
* fix type errors after merge
* update ts in another new package.json
* TEMP: remove me
* Revert "TEMP: remove me"
This reverts commit dc0fc3bae6.
* [Telemetry] Update snapshot for new TS4 SyntaxKind
* bump prettier to support TS v4 syntax
* fix prettier rules
* last style change
* fix new type errors
Co-authored-by: Alejandro Fernández Haro <alejandro.haro@elastic.co>
* Trusted Apps initial setup for route registration
* Added types for TrustedApp entries
* trusted apps list API returns results
* use methods and const from latest PR merge to lists
* a quick generator for trusted apps entries
* support cli options for trusted app data loader
* Add mocked `createTrustedAppsList()` method to `ExceptionListClientMock`
* tests fro trusted apps route handlers
* tests for trusted apps schema
* Correct name of mock method
* Fix service to ensure return value of `getExceptionList` service throws if service not available
* Fix types
* Refactor TrustedApp type + code review feedback
* Simplify our kibana mocks
* Simpler mock factory that returns an object instead of a thunk
* We can use mockReturnValue instead of mockImplementation to
accomplish the same
* Allows us to replace createStartServices mock
* Uses unknown instead of any for mocks
* Clean up our manual use of kibana mocks in tests
* Since our useKibana mock returns a consistent mock, we can modify its
return value instead of re-mocking the entire thing
* Removes unnecessary uses of clearing/resetting mocks
* If your mocks are configured at the beginning of each test this is
usually unnecessary.
* I left one case of clearAllMocks in all_cases/index.test since it
defined several mock functions that were persistent across tests, and
it was easier than moving their definitions to a beforeEach
* Removes some unnecessary overrides that seemed due to storage
previously not being mocked
* Rename some old occurrences of SIEM
* Cross-reference similar hooks via JSDoc
There's a good chance that the consumer might want the OTHER hook, so
let's make that discoverable.
* Adds jest tests for our useListsConfig hook
* adds mocks for the hooks upon which it depends
* Add a mock for our useListsConfig hook
Leverages this mock factory in our manual mock for this hook.
* Remove unneeded eslint exception
* Move kibana_react mocks into their own .mock file
We're trying to consolidate mocks to this pattern so that they're easier
to find and reuse.
* Remove intermediate mock factory
This was only being consumed by our general createStartServicesMock.
* Replace security_solution's alias for a core mock
This is just noise/tech debt, we should use the core mock directly when
we can.
* Remove unnecessary wrapper around core mocks
Instead let's just reference the core mocks themselves.
* Remove outdated references from upstream
* More accurate mock
Throw an error of the same type if an unexpected key is used.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Fixes a bug where the list privileges was returning the `.list` privileges twice instead of returning it once and returning the `.items` privileges second with the call. No UI has to change as the way it was written was dynamic to grab the first key found.
This also adds the functional tests to `x-pack/scripts/functional_tests.js` which was not there originally so the end to tend tests should actually run on the CI machine where it was not running on CI before.
Adds the functional tests to the code owners file as well.
Ensure that you go to the test results page from the Jenkins build:
<img width="901" alt="Screen Shot 2020-08-18 at 1 13 18 AM" src="https://user-images.githubusercontent.com/1151048/90482180-13f7c800-e0f0-11ea-92f2-b30a8fffe84e.png">
And ensure you see the tests under:
```
X-Pack Lists Integration Tests
```
Then click through it and ensure they are shown as running and passing
### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
* Route options timeout -> timeout.payload
* timeout.idleSocket can now be specified per route
* Removing nested ternary
* Fixing integration tests
* Trying to actually fix the integration tests. Existing tests are hitting
idle socket timeout, not the payload timeout
* Fixing payload post timeout integration test
* Fixing PUT and DELETE payload sending too long tests
* Fixing type-script errors
* GET routes can't specify the payload timeout, they can't accept payloads
* Removing some redundancy in the tests
* Adding 'Transfer-Encoding: chunked' to the POST test
* Fixing POST/GET/PUT quick tests
* Adding idleSocket timeout test
* Removing unnecessary `isSafeMethod` call
* Updating documentation
* Removing PUT/DELETE integration tests
* Working around the HapiJS bug
* Deleting unused type import
* The socket can be undefined...
This occurs when using @hapi/shot directly or indirectly via
Server.inject. In these scenarios, there isn't a socket. This can also
occur when a "fake request" is used by the hacky background jobs:
Reporting and Alerting...
* Update src/core/server/http/http_server.ts
Co-authored-by: Josh Dover <me@joshdover.com>
* Adding payload timeout functional tests
* Adding idle socket timeout functional tests
* Adding better comments, using ?? instead of ||
* Fixing the plugin fixture TS
* Fixing some typescript errors
* Fixing plugin fixture tsconfig.json
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Josh Dover <me@joshdover.com>
* Add separate io-ts types for endpoint exception entries
* Fix text typos
* Fix test
* address review comments
* Add extra entry validation when adding items to endpoint_list
* fix test
* fix tests again
* really actually fix the tests
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
This is the basics of end to end tests, so there could be a lot more, but this ties to cover the basics of the tests.
Test with:
```ts
node scripts/functional_tests --config x-pack/test/lists_api_integration/security_and_spaces/config.ts
```
Adds these tests for the route counter parts:
* create_exception_list_items.ts
* create_exception_lists.ts
* delete_exception_list_items.ts
* delete_exception_lists.ts
* find_exception_list_items.ts
* find_exception_lists.ts
* read_exception_list_items.ts
* read_exception_lists.ts
* update_exception_list_items.ts
* update_exception_lists.ts
Fixes a few minor strings, other tests, but no large bugs found with these tests
### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
## Summary
Adds initial set of end to end tests for lists
You can run all of these with the command from kibana root:
```ts
node scripts/functional_tests --config x-pack/test/lists_api_integration/security_and_spaces/config.ts
```
Fixes a few minor bugs found such as...
* Validation for importing lists was not checking if the indexes were created first
* Some wording for the error messages had duplicate words within them
### Checklist
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
## Summary
Using the `formatErrors` util would result in duplicate error messages sometimes. Was noticing this in particular when using union types, where the type validation would check every item in a union and report an error for each one. This resulted in large, repeating errors.
Used `uniq` to filter out duplicates. Updated unit tests.
## Summary
Addresses feedback from https://github.com/elastic/kibana/pull/72748
- Updates `plugins/lists` tests text from `should not validate` to `should FAIL validation` after feedback that previous text is a bit confusing and can be interpreted to mean that validation is not conducted
- Remove unnecessary spreads from one of my late night PRs
- Removes `siem_common_deps` in favor of `shared_imports` in `plugins/lists`
- Updates `build_exceptions_query.test.ts` to use existing mocks
## Summary
This PR focuses on addressing issues around the pagination and functionality of rules with numerous (2+) exception lists.
- Updated the `use_exception_list.ts` hook to make use of the new multi list find API
- Updated the viewer to make use of the new multi list find API
- Previously was doing a lot of the filtering and paging manually (and badly) in the UI, now the _find takes care of all that
- Added logic for showing `No results` text if user filter/search returns no items
- Previously would show the `This rule has not exceptions` text
## Summary
This PR addresses the following:
- Adds `list_id` to `rule.exceptions_list` - this is needed in a number of features
- Updated `getExceptions` in `x-pack/plugins/security_solution/server/lib/detection_engine/signals/utils.ts` to use the latest exception item find endpoint that accepts an array of lists (previously was looping through lists and conducting a `find` for each)
- Updated prepackaged rule that makes reference to global endpoint list to include `list_id`
- Updates `formatAboutStepData` in `x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/create/helpers.ts` to include exception list `list_id`
* Validate exception list size when adding new items
* Update comment
* Extract list size validation and apply to endpoint route also
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
By default the upload time limit for payloads is 10 seconds. This is really too short and we were getting internal QA bug reports that uploads are timing out on large value list importing. This PR adds the plumbing and unit tests to make the timeout configurable for routes.
* Adds a single timeout option for routes and then normalizes that through Hapi for the socket, payload, and server timeouts.
* Adds unit tests which test the various options
* Adds integration tests which test the various options
* Adds some NOTES about where there are odd behaviors/bugs within Hapi around validations and the timeouts
* Adds a configurable 5 minute timeout to the large value lists route
**Manual testing of the feature**
You can manually test this by adding a configurable option to your chrome network throttle like so below where you throttle upload by some configurable amount. I chose to use 300 kbs/s upload
<img width="556" alt="Screen Shot 2020-07-23 at 11 26 01 AM" src="https://user-images.githubusercontent.com/1151048/88318015-5ab3f700-ccd7-11ea-9d9b-7e3649ec65de.png">
And then run an import of large value lists using a large enough file that it will exceed 5 minutes:
After 5 minutes you should see this message within your server side messages if you have configured your kibana.dev.yml to allow for these messages:
```ts
server respons [10:52:31.377] [access:lists-all] POST /api/lists/items/_import?type=keyword 408 318292ms - 9.0B
```
Note that it should show you that it is trying to return a `408` after `318292ms` the timeout period. Sometimes you will get the 408 in the browser and sometimes the browser actually will not respect the 408 and continue staying in a pending state forever. This seems to be browser side issue and not a client/user land issue. If you get the browser message it will be this error toaster
### Checklist
- [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)
- [x] [Documentation](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#writing-documentation) was added for features that require explanation or tutorials
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
### For maintainers
- [x] This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)
Add validation to reject when value list and other exception type are entries in the same exception item. Also adds tests for this situation on the schema validation
## Summary
This PR is somewhat of an intermediary step. Comments on exception list items are denormalized. We initially decided that we would not add `uuid` to comments, but found that it is in fact necessary. This is intermediary in the sense that what we ideally want to have is a dedicated `comments` CRUD route.
Also just note that I added a callout for when a version conflict occurs (ie: exception item was updated by someone else while a user is editing the same item).
With this PR users are able to:
- Create comments when creating exception list items
- Add new comments on exception item update
Users will currently be blocked from:
- Deleting comments
- Updating comments
- Updating exception item if version conflict is found
* adapt retryCallCluster for new ES client
* review comments
* retry on 408 ResponseError
* remove legacy retry functions
* use Migrator Es client in SO migration
* update migration tests
* improve ES typings and mocks
* migrate decorate ES errors
* add repository es client
* use new es client in so repository
* update repository tests
* fix migrator integration tests
* declare _seq_no & _primary_term on get response. _source expect to be a string
* make _sourceIncludes and refresh compatible with the client
* add test for repository_es_client
* move ApiResponse to es client mocks
* TEMP: handle wait_for as true for deleteByNamespace
* add tests for migration_es_client
* TEMP: skip test for deleteByNamespace refresh
* pass ignore as transport option in mget
* log both es client and response errors
* fix update method test failures
* update deleteByNamespace refresh settings
es doesn't support 'refresh: wait_for' for `updateByQuery` endpoint
* update repository tests. we do not allow customising wait_for
* do not delegate retry logic to es client
* fix type errors after master merged
* fix repository tests
* fix security solutions code
SO doesn't throw Error with status code anymore. Always use SO error helpers
* switch error conditions to use SO error helpers
* cleanup
* address comments about mocks
* use isResponseError helper
* address comments
* fix type errors
Co-authored-by: pgayvallet <pierre.gayvallet@elastic.co>
* Refresh rule details when exception list modal modifies the rule
This addresses a bug where, when opening the exceptions modal for the first
time and creating exceptions, the details page does not reflect these
created exceptions until a full refresh.
This is due to the hook performing the refresh being dependent on the
rule's exceptions_list attribute, which is not populated until after
opening the modal. Because the UI is not informed of the rule update, it
did not know to refresh the rule.
This adds the machinery necessary to make the above work. It:
* adds a new hook for fetching/refreshing a rule
* Adds an onRuleChange callback to both the ExceptionsViewer and the
mutating AddExceptionModal
* passes the refresh function in as the onRuleChange callback
There's currently a gross intermediate state here where the loading screen is
displayed while the rule refreshes in the background; I'll be fixing
that shortly.
* Do not show loading/blank state while refreshing rule
On Rule Details, when the Add Exceptions modal creates the rule's
exception list, we refresh quietly in the background by setting our rule
from null -> ruleA -> ruleB instead of null -> ruleA -> null -> ruleB.
This also simplifies the loading logic in a few places now that we're
using our new rule: we mainly care whether or not our rule is populated.
* Display toast error if rule fetch fails
This should now have feature parity with useRule, while additionally
providing a function to refresh the rule.
* Refactor tests to leverage existing helpers
* Add return type to our callback function
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
Limits the lists to 9 megs upload size so we don't blow up smaller Kibana installs. Users can change/override this using the switch of `xpack.lists.maxImportPayloadBytes` like so:
```
xpack.lists.maxImportPayloadBytes: 40000000
```
That will increase the amount of bytes that can pushed through REST endpoints from 9 megs to something like 40 megs if the end users want to increase the size of their lists and have enough memory in Kibana.
Metrics and suggestions from testing looks like:
```ts
Kibana with 1 gig of memory can upload ~10 megs of a list before possible out of memory issue
Kibana with 2 gig of memory can upload ~20 megs of a list before possible out of memory issue
```
Things can vary depending on the speed of the uploads of the lists where faster connections to Kibana but slower connections from Kibana to Elastic Search can influence the numbers.
### Checklist
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
* wip - comment and sample json for exceptions
* promise.all for OR-ing exception items and quick-start script
* logging, added/updated json sample scripts, fixed missing await on filter with lists
* WIP
* bug fix where two lists when 'anded' together were not filtering down result set
* undo changes from testing
* fix changes to example json and fixes missed conflict with master
* update log message and fix type errors
* change log statement and add unit test for when exception items without a value list are passed in to the filter function
* fix failing test
* update expect on one test and adds a new test to ensure anding of value lists when appearing in different exception items
* update test after rebasing with master
* properly ands exception item entries together with proper test cases
* fix test (log statement tests - need to come up with a better way to cover these)
* cleans up json examples
* rename test and use 'every' in lieu of 'some' when determining if the filter logic should execute
## Summary
This PR updates the exception list entries schemas.
- **Prior:** `entries` could be `undefined` or empty array on `ExceptionListItemSchema`
- **Now:** `entries` is a required field that cannot be empty - there's really no use for an item without `entries`
- **Prior:** `field` and `value` could be empty string in `EntryMatch`
- **Now:** `field` and `value` can no longer be empty strings
- **Prior:** `field` could be empty string and `value` could be empty array in `EntryMatchAny`
- **Now:** `field` and `value` can no longer be empty string and array respectively
- **Prior:** `field` and `list.id` could be empty string in `EntryList`
- **Now:** `field` and `list.id` can no longer be empty strings
- **Prior:** `field` could be empty string in `EntryExists`
- **Now:** `field` can no longer be empty string
- **Prior:** `field` could be empty string in `EntryNested`
- **Now:** `field` can no longer be empty string
- **Prior:** `entries` could be empty array in `EntryNested`
- **Now:** `entries` can no longer be empty array
### Summary
The intent is to get the data structures in similar to rules so that we can have eventually immutable and versioned lists in later releases without too much hassle of upgrading the list and list item data structures.
* Adds version and immutability data structures to the exception lists and the value lists.
* Adds an optional version number to the update route of each so that you can modify the number either direction or you can omit it and it works like the detection rules where it will auto-increment the number.
* Does _not_ add a version and immutability to the exception list items and value list items.
* Does _not_ update the version number when you add a new exception list item or value list item.
**Examples:**
❯ ./post_list.sh
```json
{
"_version": "WzAsMV0=",
"id": "ip_list",
"created_at": "2020-07-21T20:31:11.679Z",
"created_by": "yo",
"description": "This list describes bad internet ip",
"immutable": false,
"name": "Simple list with an ip",
"tie_breaker_id": "d6bd7552-84d1-4f95-88c4-cc504517b4e5",
"type": "ip",
"updated_at": "2020-07-21T20:31:11.679Z",
"updated_by": "yo",
"version": 1
}
```
❯ ./post_exception_list.sh
```json
{
"_tags": [
"endpoint",
"process",
"malware",
"os:linux"
],
"_version": "WzMzOTgsMV0=",
"created_at": "2020-07-21T20:31:35.933Z",
"created_by": "yo",
"description": "This is a sample endpoint type exception",
"id": "2c24b100-cb91-11ea-a872-adfddf68361e",
"immutable": false,
"list_id": "simple_list",
"name": "Sample Endpoint Exception List",
"namespace_type": "single",
"tags": [
"user added string for a tag",
"malware"
],
"tie_breaker_id": "c11c4d53-d0be-4904-870e-d33ec7ca387f",
"type": "detection",
"updated_at": "2020-07-21T20:31:35.952Z",
"updated_by": "yo",
"version": 1
}
```
```json
❯ ./update_list.sh
{
"_version": "WzEsMV0=",
"created_at": "2020-07-21T20:31:11.679Z",
"created_by": "yo",
"description": "Some other description here for you",
"id": "ip_list",
"immutable": false,
"name": "Changed the name here to something else",
"tie_breaker_id": "d6bd7552-84d1-4f95-88c4-cc504517b4e5",
"type": "ip",
"updated_at": "2020-07-21T20:31:47.089Z",
"updated_by": "yo",
"version": 2
}
```
```json
❯ ./update_exception_list.sh
{
"_tags": [
"endpoint",
"process",
"malware",
"os:linux"
],
"_version": "WzMzOTksMV0=",
"created_at": "2020-07-21T20:31:35.933Z",
"created_by": "yo",
"description": "Different description",
"id": "2c24b100-cb91-11ea-a872-adfddf68361e",
"immutable": false,
"list_id": "simple_list",
"name": "Sample Endpoint Exception List",
"namespace_type": "single",
"tags": [
"user added string for a tag",
"malware"
],
"tie_breaker_id": "c11c4d53-d0be-4904-870e-d33ec7ca387f",
"type": "endpoint",
"updated_at": "2020-07-21T20:31:56.628Z",
"updated_by": "yo",
"version": 2
}
```
### Checklist
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
* Add loading spinners to Value Lists modal
While export or a delete is pending, we display a loading spinner
instead of the button that was clicked.
Since state is controlled in the parent, we must pass this additional
state in the same way; the table component simply reacts to this state.
* Fix bug with useAsync and multiple calls
Multiple calls to start() would not previously reset the hook's state,
where useEffect on the hook's state would fire improperly as subsequent
calls would not travel the same undefined -> result path.
* Fix style of loading spinner
This fits the size of the button it's replacing, so no shifting occurs
when replacing elements.
* Better styling of spinner
Keep it roughly the same size as the icons themselves, and fill the
space with margin.
* Fix circular dependency in value lists modal
Moves our shared types into a separate module to prevent a circular
dependency.
## Summary
* Adds list permissions as a feature control to SIEM.
* Separates the controls between two, one of which is `access:lists-all` and the other is `access:lists-read`
* Grants SIEM the ability to utilize both depending on which feature mode the space is in.
## Summary
* Adds conflict versioning by exposing the "_version" from the saved object system. It renames "version" to "_version" so that we can use regular "version" later for versioning things for pre-packaged lists abilities.
* Utilizes `t.OutputOf` in the requests and the data types to give us more correctly types
* Removes the `Identity` utility as that is adding confusion and can confuse vs code rather than improves things
* Removes extra types that were causing confusion which was an idiom from io-ts
* Changes the wording of `Partial` by removing that and instead focuses the request types on either client side or server side at this point.
NOTE: The UI can migrate to holding onto the `_version` and then push it back down when it wants to migrate to using the conflict resolution. If the UI does not push it down, then a value of undefined will be used which is indicating that no conflict errors are wanted.
Output example of posting an exception list:
❯ ./post_exception_list.sh
```ts
{
"_tags": [
"endpoint",
"process",
"malware",
"os:linux"
],
"_version": "Wzk4NiwxXQ==",
"created_at": "2020-07-17T18:59:22.872Z",
"created_by": "yo",
"description": "This is a sample endpoint type exception",
"id": "a08795b0-c85f-11ea-b1a6-c155df988a92",
"list_id": "simple_list",
"name": "Sample Endpoint Exception List",
"namespace_type": "single",
"tags": [
"user added string for a tag",
"malware"
],
"tie_breaker_id": "b789ec05-3e0f-4344-a156-0c0f5b6e2f9c",
"type": "detection",
"updated_at": "2020-07-17T18:59:22.891Z",
"updated_by": "yo"
}
```
Output example of posting an exception list item
❯ ./post_exception_list_item.sh
```ts
{
"_tags": [
"endpoint",
"process",
"malware",
"os:linux"
],
"_version": "Wzk4NywxXQ==",
"comments": [],
"created_at": "2020-07-17T18:59:30.286Z",
"created_by": "yo",
"description": "This is a sample endpoint type exception",
"entries": [
{
"field": "actingProcess.file.signer",
"operator": "excluded",
"type": "exists"
},
{
"field": "host.name",
"operator": "included",
"type": "match_any",
"value": [
"some host",
"another host"
]
}
],
"id": "a4f2b800-c85f-11ea-b1a6-c155df988a92",
"item_id": "simple_list_item",
"list_id": "simple_list",
"name": "Sample Endpoint Exception List",
"namespace_type": "single",
"tags": [
"user added string for a tag",
"malware"
],
"tie_breaker_id": "1dc456bc-7aa9-44b4-bca3-131689cf729f",
"type": "simple",
"updated_at": "2020-07-17T18:59:30.304Z",
"updated_by": "yo"
}
```
Output example of when you get an exception list:
❯ ./get_exception_list.sh simple_list
```ts
{
"_tags": [
"endpoint",
"process",
"malware",
"os:linux"
],
"_version": "WzEwNzcsMV0=",
"created_at": "2020-07-17T18:59:22.872Z",
"created_by": "yo",
"description": "Different description",
"id": "a08795b0-c85f-11ea-b1a6-c155df988a92",
"list_id": "simple_list",
"name": "Sample Endpoint Exception List",
"namespace_type": "single",
"tags": [
"user added string for a tag",
"malware"
],
"tie_breaker_id": "b789ec05-3e0f-4344-a156-0c0f5b6e2f9c",
"type": "endpoint",
"updated_at": "2020-07-17T20:01:24.958Z",
"updated_by": "yo"
}
```
Example of the error you get if you do an update of an exception list and someone else has changed it:
```ts
{
"message": "[exception-list:a08795b0-c85f-11ea-b1a6-c155df988a92]: version conflict, required seqNo [1074], primary term [1]. current document has seqNo [1077] and primary term [1]: [version_conflict_engine_exception] [exception-list:a08795b0-c85f-11ea-b1a6-c155df988a92]: version conflict, required seqNo [1074], primary term [1]. current document has seqNo [1077] and primary term [1], with { index_uuid=\"a2mgXBO6Tl2ULDq-MTs1Tw\" & shard=\"0\" & index=\".kibana-hassanabad_1\" }",
"status_code": 409
}
```
Lists are the same way and flavor, they encode the _version the same way that saved objects do. To see those work you run these scripts:
```ts
./post_list.sh
./post_list_item.sh
./find_list.sh
./find_list_item.sh
```
### Checklist
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
* Overview Alerts Histogram stacking defaults to signal.rule.name
Since this is now the default for all AlertsHistograms, I've moved this
default upstream into the histogram itself.
* Replace magic strings with our constant ENDPOINT_LIST_ID
Also replaced a few unintentional uses of this string with the
non-reserved 'endpoint_list_id'.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
* Adds the createEndpointListSchema to the create_endpoint_list_route for API boundary checks
* Adds unit tests to the requests
* Fixes a few bugs found in the find endpoints for types
* Fixes or deletes the skipped tests
* Updated TODO blocks and removed ones that are obsolete
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
