2020-02-11 23:24:30 +01:00
/ *
* Copyright Elasticsearch B . V . and / or licensed to Elasticsearch B . V . under one
2021-02-04 03:12:39 +01:00
* or more contributor license agreements . Licensed under the Elastic License
* 2.0 ; you may not use this file except in compliance with the Elastic License
* 2.0 .
2020-02-11 23:24:30 +01:00
* /
2021-02-26 00:13:27 +01:00
import { KbnClient } from '@kbn/test' ;
2021-03-25 09:47:16 +01:00
import type { ApiResponse } from '@elastic/elasticsearch' ;
import type { KibanaClient } from '@elastic/elasticsearch/api/kibana' ;
2020-05-07 19:18:27 +02:00
import { SuperTest } from 'supertest' ;
import supertestAsPromised from 'supertest-as-promised' ;
2020-10-15 21:21:03 +02:00
import { Context } from '@elastic/elasticsearch/lib/Transport' ;
2020-10-20 03:14:09 +02:00
import { SearchResponse } from 'elasticsearch' ;
2021-05-19 23:59:52 +02:00
import type { NonEmptyEntriesArray } from '@kbn/securitysolution-io-ts-list-types' ;
import type {
CreateExceptionListItemSchema ,
CreateExceptionListSchema ,
ExceptionListItemSchema ,
ExceptionListSchema ,
} from '@kbn/securitysolution-io-ts-list-types' ;
2021-05-25 02:38:14 +02:00
import { EXCEPTION_LIST_ITEM_URL , EXCEPTION_LIST_URL } from '@kbn/securitysolution-list-constants' ;
2021-02-05 03:07:14 +01:00
import { PrePackagedRulesAndTimelinesStatusSchema } from '../../plugins/security_solution/common/detection_engine/schemas/response' ;
[Security Solutions][Detection Engine] Adds e2e FTR runtime support and 213 tests for exception lists (#83764)
## Summary
Adds support to the end to end (e2e) functional test runner (FTR) support for rule runtime tests as well as 213 tests for the exception lists which include value based lists. Previously we had limited runtime support, but as I scaled up runtime tests from 5 to 200+ I noticed in a lot of areas we had to use improved techniques for determinism.
The runtime support being added is our next step of tests. Up to now most of our e2e FTR tests have been structural testing of REST and API integration tests. Basically up to now 95% tests are API structural as:
* Call REST input related to a rule such as GET/PUT/POST/PATCH/DELETE.
* Check REST output of the rule, did it match expected output body and status code?
* In some rare cases we check if the the rule can be executed and we get a status of 'succeeded'
With only a small part of our tests ~5%, `generating_signals.ts` was checking the signals being produced. However, we cannot have confidence in runtime based tests until the structural tests have been built up and run through the weeks against PR's to ensure that those are stable and deterministic.
Now that we have confidence and 90%+ coverage of the structural REST based tests, we are building up newer sets of tests which allow us to do runtime based validation tests to increase confidence that:
* Detection engine produces signals as expected
* Structure of the signals are as expected, including signal on signals
* Exceptions to signals are working as expected
* Most runtime bugs can be TDD'ed with e2e FTR's and regressions
* Whack-a-mole will not happen
* Consistency and predictability of signals is validated
* Refactoring can occur with stronger confidence
* Runtime tests are reference points for answering questions about existing bugs or adding new ones to test if users are experiencing unexpected behaviors
* Scaling tests can happen without failures
* Velocity for creating tests increases as the utilities and examples increase
Lastly, this puts us within striking distance of creating FTR's for different common class of runtime situations such as:
* Creating tests that exercise each rule against a set of data criteria and get signal hits
* Creating tests that validate the rule overrides operate as expected against data sets
* Creating tests that validate malfunctions, corner cases, or misuse cases such as data sets that are _all_ arrays or data sets that put numbers as strings or throws in an expected `null` instead of a value.
These tests follow the pattern of:
* Add the smallest data set to a folder in data.json (not gzip format)
* Add the smallest mapping to that folder (mapping.json)
* Call REST input related to exception lists, value lists, adding prepackaged rules, etc...
* Call REST input related endpoint with utilities to create and activate the rule
* Wait for the rule to go into the `succeeded` phase
* Wait for the N exact signals specific to that rule to be available
* Check against the set of signals to ensure that the matches are exactly as expected
Example of one runtime test:
A keyword data set is added to a folder called "keyword" but you can add one anywhere you want under `es_archives`, I just grouped mine depending on the situation of the runtime. Small non-gzipped tests `data.json` and `mappings.json` are the best approach for small focused tests. For _larger_ tests and cases I would and sometimes do use things such as auditbeat but try to avoid using larger data sets in favor of smaller focused test cases to validate the runtime is operating as expected.
```ts
{
"type": "doc",
"value": {
"id": "1",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:00:53.000Z",
"long": 1
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "2",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:01:53.000Z",
"long": 2
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "3",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:02:53.000Z",
"long": 3
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "4",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:03:53.000Z",
"long": 4
},
"type": "_doc"
}
}
```
Mapping is added. Note that this is "ECS tolerant" but not necessarily all ECS meaning I can and will try to keep things simple where I can, but I have ensured that `"@timestamp"` is at least there.
```ts
{
"type": "index",
"value": {
"index": "long",
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"long": { "type": "long" }
}
},
"settings": {
"index": {
"number_of_replicas": "1",
"number_of_shards": "1"
}
}
}
}
```
Test is written with test utilities where the `beforeEach` and `afterEach` try and clean up the indexes and load/unload the archives to keep one test from effecting another. Note this is never going to be 100% possible so see below on how we add more determinism in case something escapes the sandbox.
```ts
beforeEach(async () => {
await createSignalsIndex(supertest);
await createListsIndex(supertest);
await esArchiver.load('rule_exceptions/keyword');
});
afterEach(async () => {
await deleteSignalsIndex(supertest);
await deleteAllAlerts(supertest);
await deleteAllExceptions(es);
await deleteListsIndex(supertest);
await esArchiver.unload('rule_exceptions/keyword');
});
describe('"is" operator', () => {
it('should filter 1 single keyword if it is set as an exception', async () => {
const rule = getRuleForSignalTesting(['keyword']);
const { id } = await createRuleWithExceptionEntries(supertest, rule, [
[
{
field: 'keyword',
operator: 'included',
type: 'match',
value: 'word one',
},
],
]);
await waitForRuleSuccess(supertest, id);
await waitForSignalsToBePresent(supertest, 3, [id]);
const signalsOpen = await getSignalsById(supertest, id);
const hits = signalsOpen.hits.hits.map((hit) => hit._source.keyword).sort();
expect(hits).to.eql(['word four', 'word three', 'word two']);
});
});
```
### Changes for better determinism
To support more determinism there are changes and utilities added which can be tuned during any sporadic failures we might encounter as well as better support unexpected changes to other Elastic Stack pieces such as alerting, task manager, etc...
Get simple rule and others are now defaulting to false, meaning that the structural tests will no longer activate a rule and run it on task manger. This should cut down on error outputs as well as reduce stress and potentials for left over rules interfering with the runtime rules.
```ts
export const getSimpleRule = (ruleId = 'rule-1', enabled = false): QueryCreateSchema => ({
```
Not mandatory to use, but for most tests that should be runtime based tests, I use this function below which will enable it by default and run it using settings such as `type: 'query'`, `query: '*:*',` `from: '1900-01-01T00:00:00.000Z'`, to cut down on boiler plate noise. However, people can use whatever they want out of the grab bag or if their test is more readable to hand craft a REST request to create signals, or if they just want to call this and override where they want to, then 👍 .
```ts
export const getRuleForSignalTesting = (index: string[], ruleId = 'rule-1', enabled = true)
```
This waits for a rule to succeed before continuing
```ts
await waitForRuleSuccess(supertest, id);
```
I added a required array of id that _waits_ only for that particular id here. This is useful in case another test did not cleanup and you are getting signals being produced or left behind but need to wait specifically for yours.
```ts
await waitForSignalsToBePresent(supertest, 4, [id]);
```
I only get the signals for a particular rule id using either the auto-generated id or the rule_id. It's safer to use the ones from the auto-generated id but either of these are fine if you're careful enough.
```ts
const signalsOpen = await getSignalsById(supertest, id);
const signalsOpen = await getSignalsByIds(supertest, [createdId]);
const signalsOpen = await getSignalsByRuleIds(supertest, ['signal-on-signal']);
```
I delete all alerts now through a series of steps where it properly removes all rules using the rules bulk_delete and does it in such a way that all the API keys and alerting will be the best it can destroyed as well as double check that the alerts are showing up as being cleaned up before continuing.
```ts
deleteAllAlerts()
```
When not explicitly testing something structural, prefer to use the utilities which can and will do retries in case there are over the wire failures or es failures. Examples are:
```ts
installPrePackagedRules()
waitForRuleSuccess()
importFile() // This does a _lot_ of checks to ensure that the file is fully imported before continuing
```
Some of these utilities might still do a `expect(200);` but as we are and should use regular structural tests to cover those problems, these will probably be more and more removed when/if we hit test failures in favor of doing retries, waitFor, and countDowns.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2020-11-20 20:09:38 +01:00
import { getCreateExceptionListDetectionSchemaMock } from '../../plugins/lists/common/schemas/request/create_exception_list_schema.mock' ;
2020-11-13 03:22:13 +01:00
import {
CreateRulesSchema ,
UpdateRulesSchema ,
FullResponseSchema ,
QueryCreateSchema ,
} from '../../plugins/security_solution/common/detection_engine/schemas/request' ;
2020-10-20 03:14:09 +02:00
import { Signal } from '../../plugins/security_solution/server/lib/detection_engine/signals/types' ;
[SecuritySolution][Detections] Adds SavedObject persistence to Signals Migrations (#85690)
* Adds new SO type for persisting our signals migrations
* WIP: Migration status SO client
Trying to forge a patterrn using io-ts to validate at runtime. I think
I've got it working but I want to refactor the pipeline out into a
reusable function(s).
* Implements our SavedObjects service for signals migrations
* Defines a simple client that delegates to the base SO client with
our SO type
* Defines a service that consumes the simpler client, adding validations
and data transforms on top.
* Refactoring migration code to work with saved objects
As opposed to the previous ephemeral, encoded tokens, we now retrieve migration
info from saved objects.
At the API level, this means that both the create and finalize endpoints
receive a list of concrete indices. No more passing around tokens.
As both endpoints are idempotent, users can hammer them as much as they
want with the same lists of indices. Redundant creates and finalizes
will be met with inline 400 messages, and as one continues to poll the
finalize endpoint they should see more and more indices respond with
"completed: true"
* Fixing integration tests first, and anything upstream breaking them
* Clean up API integration tests
* standardize assignment of responses (with types)
* deletes migration SOs as test cleanup
* Split API tests into separate files
This was getting big and unwieldy; this splits these into one file per
endpoint.
* Refactor: split existing migration service functionality into atomic functions
This will allow us to repurpose the service to compose more
functionality and be more specifically useful, while keeping the
component logic separate.
* WIP: moving logic into migrationService.create
* Splitting get_migration_status into component functions
getMigrationStatus was really two separate aggregations, so I split them
out and we recompose them in the necessary routes.
* Move finalization logic into function
* migrationService exposes this as .finalize()
* adds an error field to our migration SO
* We currently only have one error that we persist there, but it would
be very time-consuming to track down that information were it not
there.
* Adds function for migration "deletion" logic
* migrationService leverages this function
* adds new boolean to our savedObject
* deletes unused function (deleteMigrationSavedObject)
* Adds route for soft-deletion of migrations
* Updating tests related to migration status
* Adding/updating mocks/unit tests necessary to satisfy the things I
need to test
* I mainly wanted to test that the the status endpoint filtered out the
deleted migrations; this was accomplished with a unit test after
fleshing out some mocks/sample data.
* Move old migration service tests to the relevant function tests
This logic was previously moved out into component functions; this moves
the tests accordingly.
* Add some unit tests around our reindex call
* Fix create migration route tests
Mocks out our migration functions, rather than stubbing ES calls
directly.
* Updates finalize route unit tests
Addresses functionality that hasn't been moved to finalizeMigration()
* Unit tests our finalization logic
Fixes a bug where we weren't accounting for soft-deleted migrations.
ALso updates our test migration SO to have a status of 'pending' as
that's a more useful default.
* Fixes finalization integration tests
These were failing due:
* a change in the migration status API response
* a bug I introduced in the finalize route
* Adds tests for our migration deletion endpoint
* unit tests
* API integration tests
* Caught/fixed bug with deleting a successful migration
* Fixes types
Removes unused code.
* Prevent race condition due to template rollover during migration
If a user has an out of date index (v1) relative to the template (v2), but the
template itself is out of date (newest is v3), then it's possible that
the template is rolled over to v3 after the v1-v2 migration has been
created but before the new index has been created.
In such a case, the new index would receive the v3 mappings but would
incorrectl be marked as v2. This shouldn't necessarily be an issue, but
it's an unnecessary state that can easily be prevented with the guard
introduced here.
* Add real usernames to migration savedObjects
In addition to the SOs themselves giving us observability into what
migration actions were performed, this gives us the additional info of
_who_ performed the action.
* Index minimal migration SO fields needed for current functionality
* Add additional migration info to status endpoint
This will allow users to finalize a migration if they've lost the
response to their POST call.
* Finalize endpoint receives an array of migration IDs, not indices
This disambiguates _which_ migrations we were finalizing if you passed
an index (which was previously: the most recent migration).
* Fix type errors in tests after we threaded through username
* Update responsibilities of migration finalize/delete endpoints
Discussions with @marshallmain lead to the following refactor:
* finalize does not delete tasks
* finalize only applies cleanup policy to a failed migration
* delete takes an array of migration ids (like finalize)
* delete hard-deletes the SavedObject of a completed (failed or
successful) migration
This gives a bit more flexibility with the endpoints, as well as
disambiguates the semantics: it just deletes migrations!
* Fix tests that were broken during refactoring
* Fix type errors
I removed some logic here but forgot the imports :(
* Move outdated integration test
In the case of a successful migration, application of the cleanup policy
is done by the deletion endpoint. In the interest of data preservation,
we do not delete a sourceIndex unless it is explicitly deleted.
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2020-12-15 10:25:39 +01:00
import { signalsMigrationType } from '../../plugins/security_solution/server/lib/detection_engine/migrations/saved_objects' ;
2020-06-16 16:41:59 +02:00
import {
Status ,
SignalIds ,
} from '../../plugins/security_solution/common/detection_engine/schemas/common/schemas' ;
2020-06-10 16:54:27 +02:00
import { RulesSchema } from '../../plugins/security_solution/common/detection_engine/schemas/response/rules_schema' ;
2020-10-15 21:21:03 +02:00
import {
DETECTION_ENGINE_INDEX_URL ,
2020-10-20 03:14:09 +02:00
DETECTION_ENGINE_PREPACKAGED_URL ,
DETECTION_ENGINE_QUERY_SIGNALS_URL ,
DETECTION_ENGINE_RULES_URL ,
2021-02-05 03:07:14 +01:00
INTERNAL_IMMUTABLE_KEY ,
2020-10-15 21:21:03 +02:00
INTERNAL_RULE_ID_KEY ,
} from '../../plugins/security_solution/common/constants' ;
[Security Solutions][Detection Engine] Adds e2e FTR runtime support and 213 tests for exception lists (#83764)
## Summary
Adds support to the end to end (e2e) functional test runner (FTR) support for rule runtime tests as well as 213 tests for the exception lists which include value based lists. Previously we had limited runtime support, but as I scaled up runtime tests from 5 to 200+ I noticed in a lot of areas we had to use improved techniques for determinism.
The runtime support being added is our next step of tests. Up to now most of our e2e FTR tests have been structural testing of REST and API integration tests. Basically up to now 95% tests are API structural as:
* Call REST input related to a rule such as GET/PUT/POST/PATCH/DELETE.
* Check REST output of the rule, did it match expected output body and status code?
* In some rare cases we check if the the rule can be executed and we get a status of 'succeeded'
With only a small part of our tests ~5%, `generating_signals.ts` was checking the signals being produced. However, we cannot have confidence in runtime based tests until the structural tests have been built up and run through the weeks against PR's to ensure that those are stable and deterministic.
Now that we have confidence and 90%+ coverage of the structural REST based tests, we are building up newer sets of tests which allow us to do runtime based validation tests to increase confidence that:
* Detection engine produces signals as expected
* Structure of the signals are as expected, including signal on signals
* Exceptions to signals are working as expected
* Most runtime bugs can be TDD'ed with e2e FTR's and regressions
* Whack-a-mole will not happen
* Consistency and predictability of signals is validated
* Refactoring can occur with stronger confidence
* Runtime tests are reference points for answering questions about existing bugs or adding new ones to test if users are experiencing unexpected behaviors
* Scaling tests can happen without failures
* Velocity for creating tests increases as the utilities and examples increase
Lastly, this puts us within striking distance of creating FTR's for different common class of runtime situations such as:
* Creating tests that exercise each rule against a set of data criteria and get signal hits
* Creating tests that validate the rule overrides operate as expected against data sets
* Creating tests that validate malfunctions, corner cases, or misuse cases such as data sets that are _all_ arrays or data sets that put numbers as strings or throws in an expected `null` instead of a value.
These tests follow the pattern of:
* Add the smallest data set to a folder in data.json (not gzip format)
* Add the smallest mapping to that folder (mapping.json)
* Call REST input related to exception lists, value lists, adding prepackaged rules, etc...
* Call REST input related endpoint with utilities to create and activate the rule
* Wait for the rule to go into the `succeeded` phase
* Wait for the N exact signals specific to that rule to be available
* Check against the set of signals to ensure that the matches are exactly as expected
Example of one runtime test:
A keyword data set is added to a folder called "keyword" but you can add one anywhere you want under `es_archives`, I just grouped mine depending on the situation of the runtime. Small non-gzipped tests `data.json` and `mappings.json` are the best approach for small focused tests. For _larger_ tests and cases I would and sometimes do use things such as auditbeat but try to avoid using larger data sets in favor of smaller focused test cases to validate the runtime is operating as expected.
```ts
{
"type": "doc",
"value": {
"id": "1",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:00:53.000Z",
"long": 1
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "2",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:01:53.000Z",
"long": 2
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "3",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:02:53.000Z",
"long": 3
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "4",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:03:53.000Z",
"long": 4
},
"type": "_doc"
}
}
```
Mapping is added. Note that this is "ECS tolerant" but not necessarily all ECS meaning I can and will try to keep things simple where I can, but I have ensured that `"@timestamp"` is at least there.
```ts
{
"type": "index",
"value": {
"index": "long",
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"long": { "type": "long" }
}
},
"settings": {
"index": {
"number_of_replicas": "1",
"number_of_shards": "1"
}
}
}
}
```
Test is written with test utilities where the `beforeEach` and `afterEach` try and clean up the indexes and load/unload the archives to keep one test from effecting another. Note this is never going to be 100% possible so see below on how we add more determinism in case something escapes the sandbox.
```ts
beforeEach(async () => {
await createSignalsIndex(supertest);
await createListsIndex(supertest);
await esArchiver.load('rule_exceptions/keyword');
});
afterEach(async () => {
await deleteSignalsIndex(supertest);
await deleteAllAlerts(supertest);
await deleteAllExceptions(es);
await deleteListsIndex(supertest);
await esArchiver.unload('rule_exceptions/keyword');
});
describe('"is" operator', () => {
it('should filter 1 single keyword if it is set as an exception', async () => {
const rule = getRuleForSignalTesting(['keyword']);
const { id } = await createRuleWithExceptionEntries(supertest, rule, [
[
{
field: 'keyword',
operator: 'included',
type: 'match',
value: 'word one',
},
],
]);
await waitForRuleSuccess(supertest, id);
await waitForSignalsToBePresent(supertest, 3, [id]);
const signalsOpen = await getSignalsById(supertest, id);
const hits = signalsOpen.hits.hits.map((hit) => hit._source.keyword).sort();
expect(hits).to.eql(['word four', 'word three', 'word two']);
});
});
```
### Changes for better determinism
To support more determinism there are changes and utilities added which can be tuned during any sporadic failures we might encounter as well as better support unexpected changes to other Elastic Stack pieces such as alerting, task manager, etc...
Get simple rule and others are now defaulting to false, meaning that the structural tests will no longer activate a rule and run it on task manger. This should cut down on error outputs as well as reduce stress and potentials for left over rules interfering with the runtime rules.
```ts
export const getSimpleRule = (ruleId = 'rule-1', enabled = false): QueryCreateSchema => ({
```
Not mandatory to use, but for most tests that should be runtime based tests, I use this function below which will enable it by default and run it using settings such as `type: 'query'`, `query: '*:*',` `from: '1900-01-01T00:00:00.000Z'`, to cut down on boiler plate noise. However, people can use whatever they want out of the grab bag or if their test is more readable to hand craft a REST request to create signals, or if they just want to call this and override where they want to, then 👍 .
```ts
export const getRuleForSignalTesting = (index: string[], ruleId = 'rule-1', enabled = true)
```
This waits for a rule to succeed before continuing
```ts
await waitForRuleSuccess(supertest, id);
```
I added a required array of id that _waits_ only for that particular id here. This is useful in case another test did not cleanup and you are getting signals being produced or left behind but need to wait specifically for yours.
```ts
await waitForSignalsToBePresent(supertest, 4, [id]);
```
I only get the signals for a particular rule id using either the auto-generated id or the rule_id. It's safer to use the ones from the auto-generated id but either of these are fine if you're careful enough.
```ts
const signalsOpen = await getSignalsById(supertest, id);
const signalsOpen = await getSignalsByIds(supertest, [createdId]);
const signalsOpen = await getSignalsByRuleIds(supertest, ['signal-on-signal']);
```
I delete all alerts now through a series of steps where it properly removes all rules using the rules bulk_delete and does it in such a way that all the API keys and alerting will be the best it can destroyed as well as double check that the alerts are showing up as being cleaned up before continuing.
```ts
deleteAllAlerts()
```
When not explicitly testing something structural, prefer to use the utilities which can and will do retries in case there are over the wire failures or es failures. Examples are:
```ts
installPrePackagedRules()
waitForRuleSuccess()
importFile() // This does a _lot_ of checks to ensure that the file is fully imported before continuing
```
Some of these utilities might still do a `expect(200);` but as we are and should use regular structural tests to cover those problems, these will probably be more and more removed when/if we hit test failures in favor of doing retries, waitFor, and countDowns.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2020-11-20 20:09:38 +01:00
import { getCreateExceptionListItemMinimalSchemaMockWithoutId } from '../../plugins/lists/common/schemas/request/create_exception_list_item_schema.mock' ;
2020-02-11 23:24:30 +01:00
/ * *
* This will remove server generated properties such as date times , etc . . .
* @param rule Rule to pass in to remove typical server generated properties
* /
export const removeServerGeneratedProperties = (
2020-11-13 03:22:13 +01:00
rule : FullResponseSchema
) : Partial < FullResponseSchema > = > {
2020-02-11 23:24:30 +01:00
const {
2020-08-05 17:32:19 +02:00
/* eslint-disable @typescript-eslint/naming-convention */
2020-02-11 23:24:30 +01:00
created_at ,
updated_at ,
id ,
2020-04-01 03:09:16 +02:00
last_failure_at ,
last_failure_message ,
2020-02-11 23:24:30 +01:00
last_success_at ,
last_success_message ,
status ,
status_date ,
2020-08-05 17:32:19 +02:00
/* eslint-enable @typescript-eslint/naming-convention */
2020-02-11 23:24:30 +01:00
. . . removedProperties
} = rule ;
return removedProperties ;
} ;
/ * *
* This will remove server generated properties such as date times , etc . . . including the rule_id
* @param rule Rule to pass in to remove typical server generated properties
* /
export const removeServerGeneratedPropertiesIncludingRuleId = (
2020-11-13 03:22:13 +01:00
rule : FullResponseSchema
) : Partial < FullResponseSchema > = > {
2020-02-11 23:24:30 +01:00
const ruleWithRemovedProperties = removeServerGeneratedProperties ( rule ) ;
2020-08-05 17:32:19 +02:00
// eslint-disable-next-line @typescript-eslint/naming-convention
2020-02-11 23:24:30 +01:00
const { rule_id , . . . additionalRuledIdRemoved } = ruleWithRemovedProperties ;
return additionalRuledIdRemoved ;
} ;
/ * *
* This is a typical simple rule for testing that is easy for most basic testing
* @param ruleId
[Security Solutions][Detection Engine] Adds e2e FTR runtime support and 213 tests for exception lists (#83764)
## Summary
Adds support to the end to end (e2e) functional test runner (FTR) support for rule runtime tests as well as 213 tests for the exception lists which include value based lists. Previously we had limited runtime support, but as I scaled up runtime tests from 5 to 200+ I noticed in a lot of areas we had to use improved techniques for determinism.
The runtime support being added is our next step of tests. Up to now most of our e2e FTR tests have been structural testing of REST and API integration tests. Basically up to now 95% tests are API structural as:
* Call REST input related to a rule such as GET/PUT/POST/PATCH/DELETE.
* Check REST output of the rule, did it match expected output body and status code?
* In some rare cases we check if the the rule can be executed and we get a status of 'succeeded'
With only a small part of our tests ~5%, `generating_signals.ts` was checking the signals being produced. However, we cannot have confidence in runtime based tests until the structural tests have been built up and run through the weeks against PR's to ensure that those are stable and deterministic.
Now that we have confidence and 90%+ coverage of the structural REST based tests, we are building up newer sets of tests which allow us to do runtime based validation tests to increase confidence that:
* Detection engine produces signals as expected
* Structure of the signals are as expected, including signal on signals
* Exceptions to signals are working as expected
* Most runtime bugs can be TDD'ed with e2e FTR's and regressions
* Whack-a-mole will not happen
* Consistency and predictability of signals is validated
* Refactoring can occur with stronger confidence
* Runtime tests are reference points for answering questions about existing bugs or adding new ones to test if users are experiencing unexpected behaviors
* Scaling tests can happen without failures
* Velocity for creating tests increases as the utilities and examples increase
Lastly, this puts us within striking distance of creating FTR's for different common class of runtime situations such as:
* Creating tests that exercise each rule against a set of data criteria and get signal hits
* Creating tests that validate the rule overrides operate as expected against data sets
* Creating tests that validate malfunctions, corner cases, or misuse cases such as data sets that are _all_ arrays or data sets that put numbers as strings or throws in an expected `null` instead of a value.
These tests follow the pattern of:
* Add the smallest data set to a folder in data.json (not gzip format)
* Add the smallest mapping to that folder (mapping.json)
* Call REST input related to exception lists, value lists, adding prepackaged rules, etc...
* Call REST input related endpoint with utilities to create and activate the rule
* Wait for the rule to go into the `succeeded` phase
* Wait for the N exact signals specific to that rule to be available
* Check against the set of signals to ensure that the matches are exactly as expected
Example of one runtime test:
A keyword data set is added to a folder called "keyword" but you can add one anywhere you want under `es_archives`, I just grouped mine depending on the situation of the runtime. Small non-gzipped tests `data.json` and `mappings.json` are the best approach for small focused tests. For _larger_ tests and cases I would and sometimes do use things such as auditbeat but try to avoid using larger data sets in favor of smaller focused test cases to validate the runtime is operating as expected.
```ts
{
"type": "doc",
"value": {
"id": "1",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:00:53.000Z",
"long": 1
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "2",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:01:53.000Z",
"long": 2
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "3",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:02:53.000Z",
"long": 3
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "4",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:03:53.000Z",
"long": 4
},
"type": "_doc"
}
}
```
Mapping is added. Note that this is "ECS tolerant" but not necessarily all ECS meaning I can and will try to keep things simple where I can, but I have ensured that `"@timestamp"` is at least there.
```ts
{
"type": "index",
"value": {
"index": "long",
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"long": { "type": "long" }
}
},
"settings": {
"index": {
"number_of_replicas": "1",
"number_of_shards": "1"
}
}
}
}
```
Test is written with test utilities where the `beforeEach` and `afterEach` try and clean up the indexes and load/unload the archives to keep one test from effecting another. Note this is never going to be 100% possible so see below on how we add more determinism in case something escapes the sandbox.
```ts
beforeEach(async () => {
await createSignalsIndex(supertest);
await createListsIndex(supertest);
await esArchiver.load('rule_exceptions/keyword');
});
afterEach(async () => {
await deleteSignalsIndex(supertest);
await deleteAllAlerts(supertest);
await deleteAllExceptions(es);
await deleteListsIndex(supertest);
await esArchiver.unload('rule_exceptions/keyword');
});
describe('"is" operator', () => {
it('should filter 1 single keyword if it is set as an exception', async () => {
const rule = getRuleForSignalTesting(['keyword']);
const { id } = await createRuleWithExceptionEntries(supertest, rule, [
[
{
field: 'keyword',
operator: 'included',
type: 'match',
value: 'word one',
},
],
]);
await waitForRuleSuccess(supertest, id);
await waitForSignalsToBePresent(supertest, 3, [id]);
const signalsOpen = await getSignalsById(supertest, id);
const hits = signalsOpen.hits.hits.map((hit) => hit._source.keyword).sort();
expect(hits).to.eql(['word four', 'word three', 'word two']);
});
});
```
### Changes for better determinism
To support more determinism there are changes and utilities added which can be tuned during any sporadic failures we might encounter as well as better support unexpected changes to other Elastic Stack pieces such as alerting, task manager, etc...
Get simple rule and others are now defaulting to false, meaning that the structural tests will no longer activate a rule and run it on task manger. This should cut down on error outputs as well as reduce stress and potentials for left over rules interfering with the runtime rules.
```ts
export const getSimpleRule = (ruleId = 'rule-1', enabled = false): QueryCreateSchema => ({
```
Not mandatory to use, but for most tests that should be runtime based tests, I use this function below which will enable it by default and run it using settings such as `type: 'query'`, `query: '*:*',` `from: '1900-01-01T00:00:00.000Z'`, to cut down on boiler plate noise. However, people can use whatever they want out of the grab bag or if their test is more readable to hand craft a REST request to create signals, or if they just want to call this and override where they want to, then 👍 .
```ts
export const getRuleForSignalTesting = (index: string[], ruleId = 'rule-1', enabled = true)
```
This waits for a rule to succeed before continuing
```ts
await waitForRuleSuccess(supertest, id);
```
I added a required array of id that _waits_ only for that particular id here. This is useful in case another test did not cleanup and you are getting signals being produced or left behind but need to wait specifically for yours.
```ts
await waitForSignalsToBePresent(supertest, 4, [id]);
```
I only get the signals for a particular rule id using either the auto-generated id or the rule_id. It's safer to use the ones from the auto-generated id but either of these are fine if you're careful enough.
```ts
const signalsOpen = await getSignalsById(supertest, id);
const signalsOpen = await getSignalsByIds(supertest, [createdId]);
const signalsOpen = await getSignalsByRuleIds(supertest, ['signal-on-signal']);
```
I delete all alerts now through a series of steps where it properly removes all rules using the rules bulk_delete and does it in such a way that all the API keys and alerting will be the best it can destroyed as well as double check that the alerts are showing up as being cleaned up before continuing.
```ts
deleteAllAlerts()
```
When not explicitly testing something structural, prefer to use the utilities which can and will do retries in case there are over the wire failures or es failures. Examples are:
```ts
installPrePackagedRules()
waitForRuleSuccess()
importFile() // This does a _lot_ of checks to ensure that the file is fully imported before continuing
```
Some of these utilities might still do a `expect(200);` but as we are and should use regular structural tests to cover those problems, these will probably be more and more removed when/if we hit test failures in favor of doing retries, waitFor, and countDowns.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2020-11-20 20:09:38 +01:00
* @param enabled Enables the rule on creation or not . Defaulted to true .
2020-02-11 23:24:30 +01:00
* /
[Security Solutions][Detection Engine] Adds e2e FTR runtime support and 213 tests for exception lists (#83764)
## Summary
Adds support to the end to end (e2e) functional test runner (FTR) support for rule runtime tests as well as 213 tests for the exception lists which include value based lists. Previously we had limited runtime support, but as I scaled up runtime tests from 5 to 200+ I noticed in a lot of areas we had to use improved techniques for determinism.
The runtime support being added is our next step of tests. Up to now most of our e2e FTR tests have been structural testing of REST and API integration tests. Basically up to now 95% tests are API structural as:
* Call REST input related to a rule such as GET/PUT/POST/PATCH/DELETE.
* Check REST output of the rule, did it match expected output body and status code?
* In some rare cases we check if the the rule can be executed and we get a status of 'succeeded'
With only a small part of our tests ~5%, `generating_signals.ts` was checking the signals being produced. However, we cannot have confidence in runtime based tests until the structural tests have been built up and run through the weeks against PR's to ensure that those are stable and deterministic.
Now that we have confidence and 90%+ coverage of the structural REST based tests, we are building up newer sets of tests which allow us to do runtime based validation tests to increase confidence that:
* Detection engine produces signals as expected
* Structure of the signals are as expected, including signal on signals
* Exceptions to signals are working as expected
* Most runtime bugs can be TDD'ed with e2e FTR's and regressions
* Whack-a-mole will not happen
* Consistency and predictability of signals is validated
* Refactoring can occur with stronger confidence
* Runtime tests are reference points for answering questions about existing bugs or adding new ones to test if users are experiencing unexpected behaviors
* Scaling tests can happen without failures
* Velocity for creating tests increases as the utilities and examples increase
Lastly, this puts us within striking distance of creating FTR's for different common class of runtime situations such as:
* Creating tests that exercise each rule against a set of data criteria and get signal hits
* Creating tests that validate the rule overrides operate as expected against data sets
* Creating tests that validate malfunctions, corner cases, or misuse cases such as data sets that are _all_ arrays or data sets that put numbers as strings or throws in an expected `null` instead of a value.
These tests follow the pattern of:
* Add the smallest data set to a folder in data.json (not gzip format)
* Add the smallest mapping to that folder (mapping.json)
* Call REST input related to exception lists, value lists, adding prepackaged rules, etc...
* Call REST input related endpoint with utilities to create and activate the rule
* Wait for the rule to go into the `succeeded` phase
* Wait for the N exact signals specific to that rule to be available
* Check against the set of signals to ensure that the matches are exactly as expected
Example of one runtime test:
A keyword data set is added to a folder called "keyword" but you can add one anywhere you want under `es_archives`, I just grouped mine depending on the situation of the runtime. Small non-gzipped tests `data.json` and `mappings.json` are the best approach for small focused tests. For _larger_ tests and cases I would and sometimes do use things such as auditbeat but try to avoid using larger data sets in favor of smaller focused test cases to validate the runtime is operating as expected.
```ts
{
"type": "doc",
"value": {
"id": "1",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:00:53.000Z",
"long": 1
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "2",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:01:53.000Z",
"long": 2
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "3",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:02:53.000Z",
"long": 3
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "4",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:03:53.000Z",
"long": 4
},
"type": "_doc"
}
}
```
Mapping is added. Note that this is "ECS tolerant" but not necessarily all ECS meaning I can and will try to keep things simple where I can, but I have ensured that `"@timestamp"` is at least there.
```ts
{
"type": "index",
"value": {
"index": "long",
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"long": { "type": "long" }
}
},
"settings": {
"index": {
"number_of_replicas": "1",
"number_of_shards": "1"
}
}
}
}
```
Test is written with test utilities where the `beforeEach` and `afterEach` try and clean up the indexes and load/unload the archives to keep one test from effecting another. Note this is never going to be 100% possible so see below on how we add more determinism in case something escapes the sandbox.
```ts
beforeEach(async () => {
await createSignalsIndex(supertest);
await createListsIndex(supertest);
await esArchiver.load('rule_exceptions/keyword');
});
afterEach(async () => {
await deleteSignalsIndex(supertest);
await deleteAllAlerts(supertest);
await deleteAllExceptions(es);
await deleteListsIndex(supertest);
await esArchiver.unload('rule_exceptions/keyword');
});
describe('"is" operator', () => {
it('should filter 1 single keyword if it is set as an exception', async () => {
const rule = getRuleForSignalTesting(['keyword']);
const { id } = await createRuleWithExceptionEntries(supertest, rule, [
[
{
field: 'keyword',
operator: 'included',
type: 'match',
value: 'word one',
},
],
]);
await waitForRuleSuccess(supertest, id);
await waitForSignalsToBePresent(supertest, 3, [id]);
const signalsOpen = await getSignalsById(supertest, id);
const hits = signalsOpen.hits.hits.map((hit) => hit._source.keyword).sort();
expect(hits).to.eql(['word four', 'word three', 'word two']);
});
});
```
### Changes for better determinism
To support more determinism there are changes and utilities added which can be tuned during any sporadic failures we might encounter as well as better support unexpected changes to other Elastic Stack pieces such as alerting, task manager, etc...
Get simple rule and others are now defaulting to false, meaning that the structural tests will no longer activate a rule and run it on task manger. This should cut down on error outputs as well as reduce stress and potentials for left over rules interfering with the runtime rules.
```ts
export const getSimpleRule = (ruleId = 'rule-1', enabled = false): QueryCreateSchema => ({
```
Not mandatory to use, but for most tests that should be runtime based tests, I use this function below which will enable it by default and run it using settings such as `type: 'query'`, `query: '*:*',` `from: '1900-01-01T00:00:00.000Z'`, to cut down on boiler plate noise. However, people can use whatever they want out of the grab bag or if their test is more readable to hand craft a REST request to create signals, or if they just want to call this and override where they want to, then 👍 .
```ts
export const getRuleForSignalTesting = (index: string[], ruleId = 'rule-1', enabled = true)
```
This waits for a rule to succeed before continuing
```ts
await waitForRuleSuccess(supertest, id);
```
I added a required array of id that _waits_ only for that particular id here. This is useful in case another test did not cleanup and you are getting signals being produced or left behind but need to wait specifically for yours.
```ts
await waitForSignalsToBePresent(supertest, 4, [id]);
```
I only get the signals for a particular rule id using either the auto-generated id or the rule_id. It's safer to use the ones from the auto-generated id but either of these are fine if you're careful enough.
```ts
const signalsOpen = await getSignalsById(supertest, id);
const signalsOpen = await getSignalsByIds(supertest, [createdId]);
const signalsOpen = await getSignalsByRuleIds(supertest, ['signal-on-signal']);
```
I delete all alerts now through a series of steps where it properly removes all rules using the rules bulk_delete and does it in such a way that all the API keys and alerting will be the best it can destroyed as well as double check that the alerts are showing up as being cleaned up before continuing.
```ts
deleteAllAlerts()
```
When not explicitly testing something structural, prefer to use the utilities which can and will do retries in case there are over the wire failures or es failures. Examples are:
```ts
installPrePackagedRules()
waitForRuleSuccess()
importFile() // This does a _lot_ of checks to ensure that the file is fully imported before continuing
```
Some of these utilities might still do a `expect(200);` but as we are and should use regular structural tests to cover those problems, these will probably be more and more removed when/if we hit test failures in favor of doing retries, waitFor, and countDowns.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2020-11-20 20:09:38 +01:00
export const getSimpleRule = ( ruleId = 'rule-1' , enabled = false ) : QueryCreateSchema = > ( {
2020-06-10 16:54:27 +02:00
name : 'Simple Rule Query' ,
description : 'Simple Rule Query' ,
2020-09-17 22:27:44 +02:00
enabled ,
2020-06-10 16:54:27 +02:00
risk_score : 1 ,
rule_id : ruleId ,
severity : 'high' ,
index : [ 'auditbeat-*' ] ,
type : 'query' ,
query : 'user.name: root or user.name: admin' ,
} ) ;
[Security Solutions][Detection Engine] Adds e2e FTR runtime support and 213 tests for exception lists (#83764)
## Summary
Adds support to the end to end (e2e) functional test runner (FTR) support for rule runtime tests as well as 213 tests for the exception lists which include value based lists. Previously we had limited runtime support, but as I scaled up runtime tests from 5 to 200+ I noticed in a lot of areas we had to use improved techniques for determinism.
The runtime support being added is our next step of tests. Up to now most of our e2e FTR tests have been structural testing of REST and API integration tests. Basically up to now 95% tests are API structural as:
* Call REST input related to a rule such as GET/PUT/POST/PATCH/DELETE.
* Check REST output of the rule, did it match expected output body and status code?
* In some rare cases we check if the the rule can be executed and we get a status of 'succeeded'
With only a small part of our tests ~5%, `generating_signals.ts` was checking the signals being produced. However, we cannot have confidence in runtime based tests until the structural tests have been built up and run through the weeks against PR's to ensure that those are stable and deterministic.
Now that we have confidence and 90%+ coverage of the structural REST based tests, we are building up newer sets of tests which allow us to do runtime based validation tests to increase confidence that:
* Detection engine produces signals as expected
* Structure of the signals are as expected, including signal on signals
* Exceptions to signals are working as expected
* Most runtime bugs can be TDD'ed with e2e FTR's and regressions
* Whack-a-mole will not happen
* Consistency and predictability of signals is validated
* Refactoring can occur with stronger confidence
* Runtime tests are reference points for answering questions about existing bugs or adding new ones to test if users are experiencing unexpected behaviors
* Scaling tests can happen without failures
* Velocity for creating tests increases as the utilities and examples increase
Lastly, this puts us within striking distance of creating FTR's for different common class of runtime situations such as:
* Creating tests that exercise each rule against a set of data criteria and get signal hits
* Creating tests that validate the rule overrides operate as expected against data sets
* Creating tests that validate malfunctions, corner cases, or misuse cases such as data sets that are _all_ arrays or data sets that put numbers as strings or throws in an expected `null` instead of a value.
These tests follow the pattern of:
* Add the smallest data set to a folder in data.json (not gzip format)
* Add the smallest mapping to that folder (mapping.json)
* Call REST input related to exception lists, value lists, adding prepackaged rules, etc...
* Call REST input related endpoint with utilities to create and activate the rule
* Wait for the rule to go into the `succeeded` phase
* Wait for the N exact signals specific to that rule to be available
* Check against the set of signals to ensure that the matches are exactly as expected
Example of one runtime test:
A keyword data set is added to a folder called "keyword" but you can add one anywhere you want under `es_archives`, I just grouped mine depending on the situation of the runtime. Small non-gzipped tests `data.json` and `mappings.json` are the best approach for small focused tests. For _larger_ tests and cases I would and sometimes do use things such as auditbeat but try to avoid using larger data sets in favor of smaller focused test cases to validate the runtime is operating as expected.
```ts
{
"type": "doc",
"value": {
"id": "1",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:00:53.000Z",
"long": 1
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "2",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:01:53.000Z",
"long": 2
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "3",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:02:53.000Z",
"long": 3
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "4",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:03:53.000Z",
"long": 4
},
"type": "_doc"
}
}
```
Mapping is added. Note that this is "ECS tolerant" but not necessarily all ECS meaning I can and will try to keep things simple where I can, but I have ensured that `"@timestamp"` is at least there.
```ts
{
"type": "index",
"value": {
"index": "long",
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"long": { "type": "long" }
}
},
"settings": {
"index": {
"number_of_replicas": "1",
"number_of_shards": "1"
}
}
}
}
```
Test is written with test utilities where the `beforeEach` and `afterEach` try and clean up the indexes and load/unload the archives to keep one test from effecting another. Note this is never going to be 100% possible so see below on how we add more determinism in case something escapes the sandbox.
```ts
beforeEach(async () => {
await createSignalsIndex(supertest);
await createListsIndex(supertest);
await esArchiver.load('rule_exceptions/keyword');
});
afterEach(async () => {
await deleteSignalsIndex(supertest);
await deleteAllAlerts(supertest);
await deleteAllExceptions(es);
await deleteListsIndex(supertest);
await esArchiver.unload('rule_exceptions/keyword');
});
describe('"is" operator', () => {
it('should filter 1 single keyword if it is set as an exception', async () => {
const rule = getRuleForSignalTesting(['keyword']);
const { id } = await createRuleWithExceptionEntries(supertest, rule, [
[
{
field: 'keyword',
operator: 'included',
type: 'match',
value: 'word one',
},
],
]);
await waitForRuleSuccess(supertest, id);
await waitForSignalsToBePresent(supertest, 3, [id]);
const signalsOpen = await getSignalsById(supertest, id);
const hits = signalsOpen.hits.hits.map((hit) => hit._source.keyword).sort();
expect(hits).to.eql(['word four', 'word three', 'word two']);
});
});
```
### Changes for better determinism
To support more determinism there are changes and utilities added which can be tuned during any sporadic failures we might encounter as well as better support unexpected changes to other Elastic Stack pieces such as alerting, task manager, etc...
Get simple rule and others are now defaulting to false, meaning that the structural tests will no longer activate a rule and run it on task manger. This should cut down on error outputs as well as reduce stress and potentials for left over rules interfering with the runtime rules.
```ts
export const getSimpleRule = (ruleId = 'rule-1', enabled = false): QueryCreateSchema => ({
```
Not mandatory to use, but for most tests that should be runtime based tests, I use this function below which will enable it by default and run it using settings such as `type: 'query'`, `query: '*:*',` `from: '1900-01-01T00:00:00.000Z'`, to cut down on boiler plate noise. However, people can use whatever they want out of the grab bag or if their test is more readable to hand craft a REST request to create signals, or if they just want to call this and override where they want to, then 👍 .
```ts
export const getRuleForSignalTesting = (index: string[], ruleId = 'rule-1', enabled = true)
```
This waits for a rule to succeed before continuing
```ts
await waitForRuleSuccess(supertest, id);
```
I added a required array of id that _waits_ only for that particular id here. This is useful in case another test did not cleanup and you are getting signals being produced or left behind but need to wait specifically for yours.
```ts
await waitForSignalsToBePresent(supertest, 4, [id]);
```
I only get the signals for a particular rule id using either the auto-generated id or the rule_id. It's safer to use the ones from the auto-generated id but either of these are fine if you're careful enough.
```ts
const signalsOpen = await getSignalsById(supertest, id);
const signalsOpen = await getSignalsByIds(supertest, [createdId]);
const signalsOpen = await getSignalsByRuleIds(supertest, ['signal-on-signal']);
```
I delete all alerts now through a series of steps where it properly removes all rules using the rules bulk_delete and does it in such a way that all the API keys and alerting will be the best it can destroyed as well as double check that the alerts are showing up as being cleaned up before continuing.
```ts
deleteAllAlerts()
```
When not explicitly testing something structural, prefer to use the utilities which can and will do retries in case there are over the wire failures or es failures. Examples are:
```ts
installPrePackagedRules()
waitForRuleSuccess()
importFile() // This does a _lot_ of checks to ensure that the file is fully imported before continuing
```
Some of these utilities might still do a `expect(200);` but as we are and should use regular structural tests to cover those problems, these will probably be more and more removed when/if we hit test failures in favor of doing retries, waitFor, and countDowns.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2020-11-20 20:09:38 +01:00
/ * *
* This is a typical signal testing rule that is easy for most basic testing of output of signals .
* It starts out in an enabled true state . The from is set very far back to test the basics of signal
* creation and testing by getting all the signals at once .
* @param ruleId The optional ruleId which is rule - 1 by default .
* @param enabled Enables the rule on creation or not . Defaulted to true .
* /
export const getRuleForSignalTesting = (
index : string [ ] ,
ruleId = 'rule-1' ,
enabled = true
) : QueryCreateSchema = > ( {
name : 'Signal Testing Query' ,
description : 'Tests a simple query' ,
enabled ,
risk_score : 1 ,
rule_id : ruleId ,
severity : 'high' ,
index ,
type : 'query' ,
query : '*:*' ,
from : '1900-01-01T00:00:00.000Z' ,
} ) ;
2020-12-24 03:44:57 +01:00
export const getRuleForSignalTestingWithTimestampOverride = (
index : string [ ] ,
ruleId = 'rule-1' ,
enabled = true ,
timestampOverride = 'event.ingested'
) : QueryCreateSchema = > ( {
name : 'Signal Testing Query' ,
description : 'Tests a simple query' ,
enabled ,
risk_score : 1 ,
rule_id : ruleId ,
severity : 'high' ,
index ,
type : 'query' ,
query : '*:*' ,
timestamp_override : timestampOverride ,
from : '1900-01-01T00:00:00.000Z' ,
} ) ;
2020-06-10 16:54:27 +02:00
/ * *
* This is a typical simple rule for testing that is easy for most basic testing
[Security Solutions][Detection Engine] Adds e2e FTR runtime support and 213 tests for exception lists (#83764)
## Summary
Adds support to the end to end (e2e) functional test runner (FTR) support for rule runtime tests as well as 213 tests for the exception lists which include value based lists. Previously we had limited runtime support, but as I scaled up runtime tests from 5 to 200+ I noticed in a lot of areas we had to use improved techniques for determinism.
The runtime support being added is our next step of tests. Up to now most of our e2e FTR tests have been structural testing of REST and API integration tests. Basically up to now 95% tests are API structural as:
* Call REST input related to a rule such as GET/PUT/POST/PATCH/DELETE.
* Check REST output of the rule, did it match expected output body and status code?
* In some rare cases we check if the the rule can be executed and we get a status of 'succeeded'
With only a small part of our tests ~5%, `generating_signals.ts` was checking the signals being produced. However, we cannot have confidence in runtime based tests until the structural tests have been built up and run through the weeks against PR's to ensure that those are stable and deterministic.
Now that we have confidence and 90%+ coverage of the structural REST based tests, we are building up newer sets of tests which allow us to do runtime based validation tests to increase confidence that:
* Detection engine produces signals as expected
* Structure of the signals are as expected, including signal on signals
* Exceptions to signals are working as expected
* Most runtime bugs can be TDD'ed with e2e FTR's and regressions
* Whack-a-mole will not happen
* Consistency and predictability of signals is validated
* Refactoring can occur with stronger confidence
* Runtime tests are reference points for answering questions about existing bugs or adding new ones to test if users are experiencing unexpected behaviors
* Scaling tests can happen without failures
* Velocity for creating tests increases as the utilities and examples increase
Lastly, this puts us within striking distance of creating FTR's for different common class of runtime situations such as:
* Creating tests that exercise each rule against a set of data criteria and get signal hits
* Creating tests that validate the rule overrides operate as expected against data sets
* Creating tests that validate malfunctions, corner cases, or misuse cases such as data sets that are _all_ arrays or data sets that put numbers as strings or throws in an expected `null` instead of a value.
These tests follow the pattern of:
* Add the smallest data set to a folder in data.json (not gzip format)
* Add the smallest mapping to that folder (mapping.json)
* Call REST input related to exception lists, value lists, adding prepackaged rules, etc...
* Call REST input related endpoint with utilities to create and activate the rule
* Wait for the rule to go into the `succeeded` phase
* Wait for the N exact signals specific to that rule to be available
* Check against the set of signals to ensure that the matches are exactly as expected
Example of one runtime test:
A keyword data set is added to a folder called "keyword" but you can add one anywhere you want under `es_archives`, I just grouped mine depending on the situation of the runtime. Small non-gzipped tests `data.json` and `mappings.json` are the best approach for small focused tests. For _larger_ tests and cases I would and sometimes do use things such as auditbeat but try to avoid using larger data sets in favor of smaller focused test cases to validate the runtime is operating as expected.
```ts
{
"type": "doc",
"value": {
"id": "1",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:00:53.000Z",
"long": 1
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "2",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:01:53.000Z",
"long": 2
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "3",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:02:53.000Z",
"long": 3
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "4",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:03:53.000Z",
"long": 4
},
"type": "_doc"
}
}
```
Mapping is added. Note that this is "ECS tolerant" but not necessarily all ECS meaning I can and will try to keep things simple where I can, but I have ensured that `"@timestamp"` is at least there.
```ts
{
"type": "index",
"value": {
"index": "long",
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"long": { "type": "long" }
}
},
"settings": {
"index": {
"number_of_replicas": "1",
"number_of_shards": "1"
}
}
}
}
```
Test is written with test utilities where the `beforeEach` and `afterEach` try and clean up the indexes and load/unload the archives to keep one test from effecting another. Note this is never going to be 100% possible so see below on how we add more determinism in case something escapes the sandbox.
```ts
beforeEach(async () => {
await createSignalsIndex(supertest);
await createListsIndex(supertest);
await esArchiver.load('rule_exceptions/keyword');
});
afterEach(async () => {
await deleteSignalsIndex(supertest);
await deleteAllAlerts(supertest);
await deleteAllExceptions(es);
await deleteListsIndex(supertest);
await esArchiver.unload('rule_exceptions/keyword');
});
describe('"is" operator', () => {
it('should filter 1 single keyword if it is set as an exception', async () => {
const rule = getRuleForSignalTesting(['keyword']);
const { id } = await createRuleWithExceptionEntries(supertest, rule, [
[
{
field: 'keyword',
operator: 'included',
type: 'match',
value: 'word one',
},
],
]);
await waitForRuleSuccess(supertest, id);
await waitForSignalsToBePresent(supertest, 3, [id]);
const signalsOpen = await getSignalsById(supertest, id);
const hits = signalsOpen.hits.hits.map((hit) => hit._source.keyword).sort();
expect(hits).to.eql(['word four', 'word three', 'word two']);
});
});
```
### Changes for better determinism
To support more determinism there are changes and utilities added which can be tuned during any sporadic failures we might encounter as well as better support unexpected changes to other Elastic Stack pieces such as alerting, task manager, etc...
Get simple rule and others are now defaulting to false, meaning that the structural tests will no longer activate a rule and run it on task manger. This should cut down on error outputs as well as reduce stress and potentials for left over rules interfering with the runtime rules.
```ts
export const getSimpleRule = (ruleId = 'rule-1', enabled = false): QueryCreateSchema => ({
```
Not mandatory to use, but for most tests that should be runtime based tests, I use this function below which will enable it by default and run it using settings such as `type: 'query'`, `query: '*:*',` `from: '1900-01-01T00:00:00.000Z'`, to cut down on boiler plate noise. However, people can use whatever they want out of the grab bag or if their test is more readable to hand craft a REST request to create signals, or if they just want to call this and override where they want to, then 👍 .
```ts
export const getRuleForSignalTesting = (index: string[], ruleId = 'rule-1', enabled = true)
```
This waits for a rule to succeed before continuing
```ts
await waitForRuleSuccess(supertest, id);
```
I added a required array of id that _waits_ only for that particular id here. This is useful in case another test did not cleanup and you are getting signals being produced or left behind but need to wait specifically for yours.
```ts
await waitForSignalsToBePresent(supertest, 4, [id]);
```
I only get the signals for a particular rule id using either the auto-generated id or the rule_id. It's safer to use the ones from the auto-generated id but either of these are fine if you're careful enough.
```ts
const signalsOpen = await getSignalsById(supertest, id);
const signalsOpen = await getSignalsByIds(supertest, [createdId]);
const signalsOpen = await getSignalsByRuleIds(supertest, ['signal-on-signal']);
```
I delete all alerts now through a series of steps where it properly removes all rules using the rules bulk_delete and does it in such a way that all the API keys and alerting will be the best it can destroyed as well as double check that the alerts are showing up as being cleaned up before continuing.
```ts
deleteAllAlerts()
```
When not explicitly testing something structural, prefer to use the utilities which can and will do retries in case there are over the wire failures or es failures. Examples are:
```ts
installPrePackagedRules()
waitForRuleSuccess()
importFile() // This does a _lot_ of checks to ensure that the file is fully imported before continuing
```
Some of these utilities might still do a `expect(200);` but as we are and should use regular structural tests to cover those problems, these will probably be more and more removed when/if we hit test failures in favor of doing retries, waitFor, and countDowns.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2020-11-20 20:09:38 +01:00
* @param ruleId The rule id
* @param enabled Set to tru to enable it , by default it is off
2020-06-10 16:54:27 +02:00
* /
[Security Solutions][Detection Engine] Adds e2e FTR runtime support and 213 tests for exception lists (#83764)
## Summary
Adds support to the end to end (e2e) functional test runner (FTR) support for rule runtime tests as well as 213 tests for the exception lists which include value based lists. Previously we had limited runtime support, but as I scaled up runtime tests from 5 to 200+ I noticed in a lot of areas we had to use improved techniques for determinism.
The runtime support being added is our next step of tests. Up to now most of our e2e FTR tests have been structural testing of REST and API integration tests. Basically up to now 95% tests are API structural as:
* Call REST input related to a rule such as GET/PUT/POST/PATCH/DELETE.
* Check REST output of the rule, did it match expected output body and status code?
* In some rare cases we check if the the rule can be executed and we get a status of 'succeeded'
With only a small part of our tests ~5%, `generating_signals.ts` was checking the signals being produced. However, we cannot have confidence in runtime based tests until the structural tests have been built up and run through the weeks against PR's to ensure that those are stable and deterministic.
Now that we have confidence and 90%+ coverage of the structural REST based tests, we are building up newer sets of tests which allow us to do runtime based validation tests to increase confidence that:
* Detection engine produces signals as expected
* Structure of the signals are as expected, including signal on signals
* Exceptions to signals are working as expected
* Most runtime bugs can be TDD'ed with e2e FTR's and regressions
* Whack-a-mole will not happen
* Consistency and predictability of signals is validated
* Refactoring can occur with stronger confidence
* Runtime tests are reference points for answering questions about existing bugs or adding new ones to test if users are experiencing unexpected behaviors
* Scaling tests can happen without failures
* Velocity for creating tests increases as the utilities and examples increase
Lastly, this puts us within striking distance of creating FTR's for different common class of runtime situations such as:
* Creating tests that exercise each rule against a set of data criteria and get signal hits
* Creating tests that validate the rule overrides operate as expected against data sets
* Creating tests that validate malfunctions, corner cases, or misuse cases such as data sets that are _all_ arrays or data sets that put numbers as strings or throws in an expected `null` instead of a value.
These tests follow the pattern of:
* Add the smallest data set to a folder in data.json (not gzip format)
* Add the smallest mapping to that folder (mapping.json)
* Call REST input related to exception lists, value lists, adding prepackaged rules, etc...
* Call REST input related endpoint with utilities to create and activate the rule
* Wait for the rule to go into the `succeeded` phase
* Wait for the N exact signals specific to that rule to be available
* Check against the set of signals to ensure that the matches are exactly as expected
Example of one runtime test:
A keyword data set is added to a folder called "keyword" but you can add one anywhere you want under `es_archives`, I just grouped mine depending on the situation of the runtime. Small non-gzipped tests `data.json` and `mappings.json` are the best approach for small focused tests. For _larger_ tests and cases I would and sometimes do use things such as auditbeat but try to avoid using larger data sets in favor of smaller focused test cases to validate the runtime is operating as expected.
```ts
{
"type": "doc",
"value": {
"id": "1",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:00:53.000Z",
"long": 1
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "2",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:01:53.000Z",
"long": 2
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "3",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:02:53.000Z",
"long": 3
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "4",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:03:53.000Z",
"long": 4
},
"type": "_doc"
}
}
```
Mapping is added. Note that this is "ECS tolerant" but not necessarily all ECS meaning I can and will try to keep things simple where I can, but I have ensured that `"@timestamp"` is at least there.
```ts
{
"type": "index",
"value": {
"index": "long",
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"long": { "type": "long" }
}
},
"settings": {
"index": {
"number_of_replicas": "1",
"number_of_shards": "1"
}
}
}
}
```
Test is written with test utilities where the `beforeEach` and `afterEach` try and clean up the indexes and load/unload the archives to keep one test from effecting another. Note this is never going to be 100% possible so see below on how we add more determinism in case something escapes the sandbox.
```ts
beforeEach(async () => {
await createSignalsIndex(supertest);
await createListsIndex(supertest);
await esArchiver.load('rule_exceptions/keyword');
});
afterEach(async () => {
await deleteSignalsIndex(supertest);
await deleteAllAlerts(supertest);
await deleteAllExceptions(es);
await deleteListsIndex(supertest);
await esArchiver.unload('rule_exceptions/keyword');
});
describe('"is" operator', () => {
it('should filter 1 single keyword if it is set as an exception', async () => {
const rule = getRuleForSignalTesting(['keyword']);
const { id } = await createRuleWithExceptionEntries(supertest, rule, [
[
{
field: 'keyword',
operator: 'included',
type: 'match',
value: 'word one',
},
],
]);
await waitForRuleSuccess(supertest, id);
await waitForSignalsToBePresent(supertest, 3, [id]);
const signalsOpen = await getSignalsById(supertest, id);
const hits = signalsOpen.hits.hits.map((hit) => hit._source.keyword).sort();
expect(hits).to.eql(['word four', 'word three', 'word two']);
});
});
```
### Changes for better determinism
To support more determinism there are changes and utilities added which can be tuned during any sporadic failures we might encounter as well as better support unexpected changes to other Elastic Stack pieces such as alerting, task manager, etc...
Get simple rule and others are now defaulting to false, meaning that the structural tests will no longer activate a rule and run it on task manger. This should cut down on error outputs as well as reduce stress and potentials for left over rules interfering with the runtime rules.
```ts
export const getSimpleRule = (ruleId = 'rule-1', enabled = false): QueryCreateSchema => ({
```
Not mandatory to use, but for most tests that should be runtime based tests, I use this function below which will enable it by default and run it using settings such as `type: 'query'`, `query: '*:*',` `from: '1900-01-01T00:00:00.000Z'`, to cut down on boiler plate noise. However, people can use whatever they want out of the grab bag or if their test is more readable to hand craft a REST request to create signals, or if they just want to call this and override where they want to, then 👍 .
```ts
export const getRuleForSignalTesting = (index: string[], ruleId = 'rule-1', enabled = true)
```
This waits for a rule to succeed before continuing
```ts
await waitForRuleSuccess(supertest, id);
```
I added a required array of id that _waits_ only for that particular id here. This is useful in case another test did not cleanup and you are getting signals being produced or left behind but need to wait specifically for yours.
```ts
await waitForSignalsToBePresent(supertest, 4, [id]);
```
I only get the signals for a particular rule id using either the auto-generated id or the rule_id. It's safer to use the ones from the auto-generated id but either of these are fine if you're careful enough.
```ts
const signalsOpen = await getSignalsById(supertest, id);
const signalsOpen = await getSignalsByIds(supertest, [createdId]);
const signalsOpen = await getSignalsByRuleIds(supertest, ['signal-on-signal']);
```
I delete all alerts now through a series of steps where it properly removes all rules using the rules bulk_delete and does it in such a way that all the API keys and alerting will be the best it can destroyed as well as double check that the alerts are showing up as being cleaned up before continuing.
```ts
deleteAllAlerts()
```
When not explicitly testing something structural, prefer to use the utilities which can and will do retries in case there are over the wire failures or es failures. Examples are:
```ts
installPrePackagedRules()
waitForRuleSuccess()
importFile() // This does a _lot_ of checks to ensure that the file is fully imported before continuing
```
Some of these utilities might still do a `expect(200);` but as we are and should use regular structural tests to cover those problems, these will probably be more and more removed when/if we hit test failures in favor of doing retries, waitFor, and countDowns.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2020-11-20 20:09:38 +01:00
export const getSimpleRuleUpdate = ( ruleId = 'rule-1' , enabled = false ) : UpdateRulesSchema = > ( {
2020-02-11 23:24:30 +01:00
name : 'Simple Rule Query' ,
description : 'Simple Rule Query' ,
[Security Solutions][Detection Engine] Adds e2e FTR runtime support and 213 tests for exception lists (#83764)
## Summary
Adds support to the end to end (e2e) functional test runner (FTR) support for rule runtime tests as well as 213 tests for the exception lists which include value based lists. Previously we had limited runtime support, but as I scaled up runtime tests from 5 to 200+ I noticed in a lot of areas we had to use improved techniques for determinism.
The runtime support being added is our next step of tests. Up to now most of our e2e FTR tests have been structural testing of REST and API integration tests. Basically up to now 95% tests are API structural as:
* Call REST input related to a rule such as GET/PUT/POST/PATCH/DELETE.
* Check REST output of the rule, did it match expected output body and status code?
* In some rare cases we check if the the rule can be executed and we get a status of 'succeeded'
With only a small part of our tests ~5%, `generating_signals.ts` was checking the signals being produced. However, we cannot have confidence in runtime based tests until the structural tests have been built up and run through the weeks against PR's to ensure that those are stable and deterministic.
Now that we have confidence and 90%+ coverage of the structural REST based tests, we are building up newer sets of tests which allow us to do runtime based validation tests to increase confidence that:
* Detection engine produces signals as expected
* Structure of the signals are as expected, including signal on signals
* Exceptions to signals are working as expected
* Most runtime bugs can be TDD'ed with e2e FTR's and regressions
* Whack-a-mole will not happen
* Consistency and predictability of signals is validated
* Refactoring can occur with stronger confidence
* Runtime tests are reference points for answering questions about existing bugs or adding new ones to test if users are experiencing unexpected behaviors
* Scaling tests can happen without failures
* Velocity for creating tests increases as the utilities and examples increase
Lastly, this puts us within striking distance of creating FTR's for different common class of runtime situations such as:
* Creating tests that exercise each rule against a set of data criteria and get signal hits
* Creating tests that validate the rule overrides operate as expected against data sets
* Creating tests that validate malfunctions, corner cases, or misuse cases such as data sets that are _all_ arrays or data sets that put numbers as strings or throws in an expected `null` instead of a value.
These tests follow the pattern of:
* Add the smallest data set to a folder in data.json (not gzip format)
* Add the smallest mapping to that folder (mapping.json)
* Call REST input related to exception lists, value lists, adding prepackaged rules, etc...
* Call REST input related endpoint with utilities to create and activate the rule
* Wait for the rule to go into the `succeeded` phase
* Wait for the N exact signals specific to that rule to be available
* Check against the set of signals to ensure that the matches are exactly as expected
Example of one runtime test:
A keyword data set is added to a folder called "keyword" but you can add one anywhere you want under `es_archives`, I just grouped mine depending on the situation of the runtime. Small non-gzipped tests `data.json` and `mappings.json` are the best approach for small focused tests. For _larger_ tests and cases I would and sometimes do use things such as auditbeat but try to avoid using larger data sets in favor of smaller focused test cases to validate the runtime is operating as expected.
```ts
{
"type": "doc",
"value": {
"id": "1",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:00:53.000Z",
"long": 1
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "2",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:01:53.000Z",
"long": 2
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "3",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:02:53.000Z",
"long": 3
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "4",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:03:53.000Z",
"long": 4
},
"type": "_doc"
}
}
```
Mapping is added. Note that this is "ECS tolerant" but not necessarily all ECS meaning I can and will try to keep things simple where I can, but I have ensured that `"@timestamp"` is at least there.
```ts
{
"type": "index",
"value": {
"index": "long",
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"long": { "type": "long" }
}
},
"settings": {
"index": {
"number_of_replicas": "1",
"number_of_shards": "1"
}
}
}
}
```
Test is written with test utilities where the `beforeEach` and `afterEach` try and clean up the indexes and load/unload the archives to keep one test from effecting another. Note this is never going to be 100% possible so see below on how we add more determinism in case something escapes the sandbox.
```ts
beforeEach(async () => {
await createSignalsIndex(supertest);
await createListsIndex(supertest);
await esArchiver.load('rule_exceptions/keyword');
});
afterEach(async () => {
await deleteSignalsIndex(supertest);
await deleteAllAlerts(supertest);
await deleteAllExceptions(es);
await deleteListsIndex(supertest);
await esArchiver.unload('rule_exceptions/keyword');
});
describe('"is" operator', () => {
it('should filter 1 single keyword if it is set as an exception', async () => {
const rule = getRuleForSignalTesting(['keyword']);
const { id } = await createRuleWithExceptionEntries(supertest, rule, [
[
{
field: 'keyword',
operator: 'included',
type: 'match',
value: 'word one',
},
],
]);
await waitForRuleSuccess(supertest, id);
await waitForSignalsToBePresent(supertest, 3, [id]);
const signalsOpen = await getSignalsById(supertest, id);
const hits = signalsOpen.hits.hits.map((hit) => hit._source.keyword).sort();
expect(hits).to.eql(['word four', 'word three', 'word two']);
});
});
```
### Changes for better determinism
To support more determinism there are changes and utilities added which can be tuned during any sporadic failures we might encounter as well as better support unexpected changes to other Elastic Stack pieces such as alerting, task manager, etc...
Get simple rule and others are now defaulting to false, meaning that the structural tests will no longer activate a rule and run it on task manger. This should cut down on error outputs as well as reduce stress and potentials for left over rules interfering with the runtime rules.
```ts
export const getSimpleRule = (ruleId = 'rule-1', enabled = false): QueryCreateSchema => ({
```
Not mandatory to use, but for most tests that should be runtime based tests, I use this function below which will enable it by default and run it using settings such as `type: 'query'`, `query: '*:*',` `from: '1900-01-01T00:00:00.000Z'`, to cut down on boiler plate noise. However, people can use whatever they want out of the grab bag or if their test is more readable to hand craft a REST request to create signals, or if they just want to call this and override where they want to, then 👍 .
```ts
export const getRuleForSignalTesting = (index: string[], ruleId = 'rule-1', enabled = true)
```
This waits for a rule to succeed before continuing
```ts
await waitForRuleSuccess(supertest, id);
```
I added a required array of id that _waits_ only for that particular id here. This is useful in case another test did not cleanup and you are getting signals being produced or left behind but need to wait specifically for yours.
```ts
await waitForSignalsToBePresent(supertest, 4, [id]);
```
I only get the signals for a particular rule id using either the auto-generated id or the rule_id. It's safer to use the ones from the auto-generated id but either of these are fine if you're careful enough.
```ts
const signalsOpen = await getSignalsById(supertest, id);
const signalsOpen = await getSignalsByIds(supertest, [createdId]);
const signalsOpen = await getSignalsByRuleIds(supertest, ['signal-on-signal']);
```
I delete all alerts now through a series of steps where it properly removes all rules using the rules bulk_delete and does it in such a way that all the API keys and alerting will be the best it can destroyed as well as double check that the alerts are showing up as being cleaned up before continuing.
```ts
deleteAllAlerts()
```
When not explicitly testing something structural, prefer to use the utilities which can and will do retries in case there are over the wire failures or es failures. Examples are:
```ts
installPrePackagedRules()
waitForRuleSuccess()
importFile() // This does a _lot_ of checks to ensure that the file is fully imported before continuing
```
Some of these utilities might still do a `expect(200);` but as we are and should use regular structural tests to cover those problems, these will probably be more and more removed when/if we hit test failures in favor of doing retries, waitFor, and countDowns.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2020-11-20 20:09:38 +01:00
enabled ,
2020-02-11 23:24:30 +01:00
risk_score : 1 ,
rule_id : ruleId ,
severity : 'high' ,
[SIEM] Create ML Rules (#58053)
* Remove unnecessary linter exceptions
Not sure what was causing issues here, but it's gone now.
* WIP: Simple form to test creation of ML rules
This will be integrated into the regular rule creation workflow, but for
now this simple form should allow us to exercise the full ML rule
workflow.
* WIP: Adds POST to backend, and type/payload changes necessary to make that work
* Simplify logic with Math.min
* WIP: Failed spike of making an http call
* WIP: Hacking together an ML client
The rest of this is going to be easier if I have actual data. For now
this is mostly copy/pasted and simplified ML code. I've hardcoded time
ranges to a period I know has data for a particular job.
* Threading through our new ML Rule params
It's a bummer that we normalize our rule alert params across all rule
types currently, but that's the deal.
* Retrieve our anomalies during rule execution
Next step: generate signals
* WIP: Generate ECS-compatible ML Signals
This uses as much of the existing signal-creation code as possible. I
skipped the search_after stuff for now because it would require us
recreating the anomalies query which we really shouldn't own. For now,
here's how it works:
* Adds a separate branch of the rule executor for machine_learning rules
* In that branch, we call our new bulkCreateMlSignal function
* This function first transforms the anomaly document into ECS fields
* We then pass the transformed documents to singleBulkCreate, which
does the rest
* After both branches, we update the rule's status appropriately.
We need to do some more work on the anomaly transformation, but this
works!
* Extract setting of rule failure to helper function
We were doing this identically in three places.
* Remove unused import
* Define a field for our Rule Type selection
This adds most of the markup and logic to allow an ML rule type to be
selected. We still need to add things like license-checking and
showing/hiding of fields based on type.
* Hide Query Fields when ML is selected
These are still getting set on the form. We'll need to filter these
fields before we send off the data, and not show them on the readonly
display either.
ALso, edit is majorly broken.
* Add input field for anomaly threshold
* Display numberic values in the readonly view of a step
TIL that isEmpty returns false for numbers and other non-iterable
values. I don't think it's exactly what we want here, but until I figure
out the intention this gets our anomalyThreshold showing up without a
separate logic branch here. Removes the unnecessary branch that was
redundant with the 'else' clause.
* Add field for selecting an ML job
This is not the same as the mockups and lacks some functionality, but
it'll allow us to select a job for now.
* Format our new ML Fields when sending them to the server
So that we don't get rejected due to snake case vs camelcase.
* Put back code that respects a rule's schedule
It was previously hardcoded to a time period I knew had anomalies.
* ML fields are optional in our creation step
In that we don't initialize them like we do the query (default) fields.
* Only send along type-specific Rule fields from form
This makes any query- or ML-specific fields optional on a Rule, and
performs some logic on the frontend to group and include these fieldsets
conditionally based on the user's selection. The one place we don't
handle this well is on the readonly view of a completed step in the
rules creation, but we'll address that.
* Rename anomalies query
It's no longer tabular data. If we need that, we can use the ML client.
* Remove spike page with simple form
* Remove unneeded ES option
This response isn't going to HTTP, which is where this option would
matter.
* Fix bulk create logic
I made a happy accident and flipped the logic here, which meant we
weren't capping the signals we created.
* Rename argument
Value is a little more ambiguous than data, here: this is our step data.
* Create Rule form stores all values, but filters by type for use
When sending off to the backend, or displaying on the readonly view, we
inspect which rule type we've currently selected, and filter our form
values appropriately.
* Fix editing of ML fields on Rule Create
We need to inherit the field value from our form on initial render, and
everything works as expected.
* Clear form errors when switching between rule types
Validation errors prevent us from moving to the next step, so it was
previously possible to get an error for Query fields, switch to an ML
rule, and be unable to continue because the form had Query errors.
This also adds a helper for checking whether a ruleType is ML, to
prevent having to change all these references if the type string
changes.
* Validate the selection of an ML Job
* Fix type errors on frontend
According to the types, this is essentially the opposite of formatRule,
so we need to reinflate all potential form values from the rule.
* Don't set defaults for query-specific rules
For ML rules these types should not be included.
* Return ML Fields in Rule responses
This adds these fields to our rule serialization, and then adds
conditional validation around those fields if the rule type is ML.
Conversely, we moved the 'language' and 'query' fields to be
conditionally validated if the rule is a query/saved_query rule.
* Fix editing of ML rules by changing who controls the field values
The source of truth for their state is the parent form object; these
inputs should not have local state.
* Fix type errors related to new ML fields
In adding the new ML fields, some other fields (e.g. `query` and
`index`) that were previously required but implicitly part of Query
Rules are now marked as optional.
Consequently, any downstream code that actually required these fields
started to complain. In general, the fix was to verify that those fields
exist, and throw an error otherwise as to appease the linter.
Runtime-wise, the new ML rules/signals follow a separate code path and
both branches should be unaffected by these changes; the issue is simply
that our conditional types don't work well with Typescript.
* Fix failing route tests
Error message changed.
* Fix integration tests
We were not sending required properties when creating a rule(index and
language).
* Fix non-ML Rule creation
I was accidentally dropping this parameter for our POST payload. Whoops.
* More informative logging during ML signal generation
The messaging diverged from the normal path here because we don't have
index patterns to display. However, we have the rest of the rule
context, and should report it appropriately.
* Prefer keyof for string union types
* Tidy up our new form components
* Type them as React.FCs
* Remove unnecessary use of styled-components
* Prefer destructuring to lodash's omit
* Fix mock params for helper functions
These were updated to take simpler parameters.
* Remove any type
This could have been a boolean all along, whoops
* Fix mock types
* Update outdated tests
These were added on master, but behavior has been changed on my branch.
* Add some tests around our helper function
I need to refactor it, so this is as good a time as any to pin down the
behavior.
* Remove uses of any in favor of actual types
Mainly leverages ML typings instead of our placeholder types. This
required handling a null case in our formatting of anomalies.
* Annotate our anomalies with @timestamp field
We were notably lacking this ECS field in our post-conversion anomalies,
and typescript was rightly complaining about it.
* ml_job_id -> machine_learning_job_id
* PR Feedback
* Stricter threshold type
* More robust date parsing
* More informative log/error messages
* Remove redundant runtime checks
* Cleaning up our new ML types
* Fix types on our Rest types
* Use less ambiguous machineLearningJobId over mlJobId
* Declare our ML params as required keys, and ensure we pass them around
everywhere we might need them (creating, importing, updating rules).
* Use implicit type to avoid the need for a ts-ignore
FormSchema has a very generic index signature such that our
filterRuleFieldsForType helper cannot infer that it has our necessary
rule fields (when in fact it does). By removing the FormSchema hint we
get the actual keys of our schema, and things work as expected.
All other uses of schema continue to work because they're expecting
FormSchema, which is effectively { [key: string]: any }.
* New ML params are not nullable
Rather than setting a null and then never using it, let's just make it
truly optional in terms of default values.
* Query and language are conditional based on rule type
For ML Rules, we don't use them.
* Remove defaulted parameter in API test
We don't need to specify this, and we should continue not to for
backwards compatibility.
* Use explicit types over implicit ones
The concern is that not typing our schemae as FormSchema could break our
form if there are upstream changes. For now, we simply use the
intersection of FormSchema and our generic parameter to satisfy our use
within the function.
* Add integration test for creation of ML Rule
* Add ML fields to route schemae
* threshold and job id are conditional on type
* makes query and language mutually exclusive with above
* Fix router test for creating an ML rule
We were sending invalid parameters.
* Remove null check against index for query rules
We support not having an index here, as getInputIndex will return the
current UI setting if none is specified.
* Add regression test for API compatibility
We were previously able to create a rule without an input index; we
should continue to support that, as verified by this test!
* Respect the index pattern determined at runtime when performing search_after
If a rule does not specify an input index pattern on creation, we use
the current UI default when the rule is evaluated. This ensures that any
subsequent searches use that same index.
We're not currently persisting that runtime index to the generated
signal, but we should.
* Fix type errors in our bulk create tests
We added a new argument, but didn't update the tests.
2020-03-19 01:26:42 +01:00
index : [ 'auditbeat-*' ] ,
2020-02-11 23:24:30 +01:00
type : 'query' ,
query : 'user.name: root or user.name: admin' ,
} ) ;
[SIEM] Create ML Rules (#58053)
* Remove unnecessary linter exceptions
Not sure what was causing issues here, but it's gone now.
* WIP: Simple form to test creation of ML rules
This will be integrated into the regular rule creation workflow, but for
now this simple form should allow us to exercise the full ML rule
workflow.
* WIP: Adds POST to backend, and type/payload changes necessary to make that work
* Simplify logic with Math.min
* WIP: Failed spike of making an http call
* WIP: Hacking together an ML client
The rest of this is going to be easier if I have actual data. For now
this is mostly copy/pasted and simplified ML code. I've hardcoded time
ranges to a period I know has data for a particular job.
* Threading through our new ML Rule params
It's a bummer that we normalize our rule alert params across all rule
types currently, but that's the deal.
* Retrieve our anomalies during rule execution
Next step: generate signals
* WIP: Generate ECS-compatible ML Signals
This uses as much of the existing signal-creation code as possible. I
skipped the search_after stuff for now because it would require us
recreating the anomalies query which we really shouldn't own. For now,
here's how it works:
* Adds a separate branch of the rule executor for machine_learning rules
* In that branch, we call our new bulkCreateMlSignal function
* This function first transforms the anomaly document into ECS fields
* We then pass the transformed documents to singleBulkCreate, which
does the rest
* After both branches, we update the rule's status appropriately.
We need to do some more work on the anomaly transformation, but this
works!
* Extract setting of rule failure to helper function
We were doing this identically in three places.
* Remove unused import
* Define a field for our Rule Type selection
This adds most of the markup and logic to allow an ML rule type to be
selected. We still need to add things like license-checking and
showing/hiding of fields based on type.
* Hide Query Fields when ML is selected
These are still getting set on the form. We'll need to filter these
fields before we send off the data, and not show them on the readonly
display either.
ALso, edit is majorly broken.
* Add input field for anomaly threshold
* Display numberic values in the readonly view of a step
TIL that isEmpty returns false for numbers and other non-iterable
values. I don't think it's exactly what we want here, but until I figure
out the intention this gets our anomalyThreshold showing up without a
separate logic branch here. Removes the unnecessary branch that was
redundant with the 'else' clause.
* Add field for selecting an ML job
This is not the same as the mockups and lacks some functionality, but
it'll allow us to select a job for now.
* Format our new ML Fields when sending them to the server
So that we don't get rejected due to snake case vs camelcase.
* Put back code that respects a rule's schedule
It was previously hardcoded to a time period I knew had anomalies.
* ML fields are optional in our creation step
In that we don't initialize them like we do the query (default) fields.
* Only send along type-specific Rule fields from form
This makes any query- or ML-specific fields optional on a Rule, and
performs some logic on the frontend to group and include these fieldsets
conditionally based on the user's selection. The one place we don't
handle this well is on the readonly view of a completed step in the
rules creation, but we'll address that.
* Rename anomalies query
It's no longer tabular data. If we need that, we can use the ML client.
* Remove spike page with simple form
* Remove unneeded ES option
This response isn't going to HTTP, which is where this option would
matter.
* Fix bulk create logic
I made a happy accident and flipped the logic here, which meant we
weren't capping the signals we created.
* Rename argument
Value is a little more ambiguous than data, here: this is our step data.
* Create Rule form stores all values, but filters by type for use
When sending off to the backend, or displaying on the readonly view, we
inspect which rule type we've currently selected, and filter our form
values appropriately.
* Fix editing of ML fields on Rule Create
We need to inherit the field value from our form on initial render, and
everything works as expected.
* Clear form errors when switching between rule types
Validation errors prevent us from moving to the next step, so it was
previously possible to get an error for Query fields, switch to an ML
rule, and be unable to continue because the form had Query errors.
This also adds a helper for checking whether a ruleType is ML, to
prevent having to change all these references if the type string
changes.
* Validate the selection of an ML Job
* Fix type errors on frontend
According to the types, this is essentially the opposite of formatRule,
so we need to reinflate all potential form values from the rule.
* Don't set defaults for query-specific rules
For ML rules these types should not be included.
* Return ML Fields in Rule responses
This adds these fields to our rule serialization, and then adds
conditional validation around those fields if the rule type is ML.
Conversely, we moved the 'language' and 'query' fields to be
conditionally validated if the rule is a query/saved_query rule.
* Fix editing of ML rules by changing who controls the field values
The source of truth for their state is the parent form object; these
inputs should not have local state.
* Fix type errors related to new ML fields
In adding the new ML fields, some other fields (e.g. `query` and
`index`) that were previously required but implicitly part of Query
Rules are now marked as optional.
Consequently, any downstream code that actually required these fields
started to complain. In general, the fix was to verify that those fields
exist, and throw an error otherwise as to appease the linter.
Runtime-wise, the new ML rules/signals follow a separate code path and
both branches should be unaffected by these changes; the issue is simply
that our conditional types don't work well with Typescript.
* Fix failing route tests
Error message changed.
* Fix integration tests
We were not sending required properties when creating a rule(index and
language).
* Fix non-ML Rule creation
I was accidentally dropping this parameter for our POST payload. Whoops.
* More informative logging during ML signal generation
The messaging diverged from the normal path here because we don't have
index patterns to display. However, we have the rest of the rule
context, and should report it appropriately.
* Prefer keyof for string union types
* Tidy up our new form components
* Type them as React.FCs
* Remove unnecessary use of styled-components
* Prefer destructuring to lodash's omit
* Fix mock params for helper functions
These were updated to take simpler parameters.
* Remove any type
This could have been a boolean all along, whoops
* Fix mock types
* Update outdated tests
These were added on master, but behavior has been changed on my branch.
* Add some tests around our helper function
I need to refactor it, so this is as good a time as any to pin down the
behavior.
* Remove uses of any in favor of actual types
Mainly leverages ML typings instead of our placeholder types. This
required handling a null case in our formatting of anomalies.
* Annotate our anomalies with @timestamp field
We were notably lacking this ECS field in our post-conversion anomalies,
and typescript was rightly complaining about it.
* ml_job_id -> machine_learning_job_id
* PR Feedback
* Stricter threshold type
* More robust date parsing
* More informative log/error messages
* Remove redundant runtime checks
* Cleaning up our new ML types
* Fix types on our Rest types
* Use less ambiguous machineLearningJobId over mlJobId
* Declare our ML params as required keys, and ensure we pass them around
everywhere we might need them (creating, importing, updating rules).
* Use implicit type to avoid the need for a ts-ignore
FormSchema has a very generic index signature such that our
filterRuleFieldsForType helper cannot infer that it has our necessary
rule fields (when in fact it does). By removing the FormSchema hint we
get the actual keys of our schema, and things work as expected.
All other uses of schema continue to work because they're expecting
FormSchema, which is effectively { [key: string]: any }.
* New ML params are not nullable
Rather than setting a null and then never using it, let's just make it
truly optional in terms of default values.
* Query and language are conditional based on rule type
For ML Rules, we don't use them.
* Remove defaulted parameter in API test
We don't need to specify this, and we should continue not to for
backwards compatibility.
* Use explicit types over implicit ones
The concern is that not typing our schemae as FormSchema could break our
form if there are upstream changes. For now, we simply use the
intersection of FormSchema and our generic parameter to satisfy our use
within the function.
* Add integration test for creation of ML Rule
* Add ML fields to route schemae
* threshold and job id are conditional on type
* makes query and language mutually exclusive with above
* Fix router test for creating an ML rule
We were sending invalid parameters.
* Remove null check against index for query rules
We support not having an index here, as getInputIndex will return the
current UI setting if none is specified.
* Add regression test for API compatibility
We were previously able to create a rule without an input index; we
should continue to support that, as verified by this test!
* Respect the index pattern determined at runtime when performing search_after
If a rule does not specify an input index pattern on creation, we use
the current UI default when the rule is evaluated. This ensures that any
subsequent searches use that same index.
We're not currently persisting that runtime index to the generated
signal, but we should.
* Fix type errors in our bulk create tests
We added a new argument, but didn't update the tests.
2020-03-19 01:26:42 +01:00
/ * *
* This is a representative ML rule payload as expected by the server
[Security Solutions][Detection Engine] Adds e2e FTR runtime support and 213 tests for exception lists (#83764)
## Summary
Adds support to the end to end (e2e) functional test runner (FTR) support for rule runtime tests as well as 213 tests for the exception lists which include value based lists. Previously we had limited runtime support, but as I scaled up runtime tests from 5 to 200+ I noticed in a lot of areas we had to use improved techniques for determinism.
The runtime support being added is our next step of tests. Up to now most of our e2e FTR tests have been structural testing of REST and API integration tests. Basically up to now 95% tests are API structural as:
* Call REST input related to a rule such as GET/PUT/POST/PATCH/DELETE.
* Check REST output of the rule, did it match expected output body and status code?
* In some rare cases we check if the the rule can be executed and we get a status of 'succeeded'
With only a small part of our tests ~5%, `generating_signals.ts` was checking the signals being produced. However, we cannot have confidence in runtime based tests until the structural tests have been built up and run through the weeks against PR's to ensure that those are stable and deterministic.
Now that we have confidence and 90%+ coverage of the structural REST based tests, we are building up newer sets of tests which allow us to do runtime based validation tests to increase confidence that:
* Detection engine produces signals as expected
* Structure of the signals are as expected, including signal on signals
* Exceptions to signals are working as expected
* Most runtime bugs can be TDD'ed with e2e FTR's and regressions
* Whack-a-mole will not happen
* Consistency and predictability of signals is validated
* Refactoring can occur with stronger confidence
* Runtime tests are reference points for answering questions about existing bugs or adding new ones to test if users are experiencing unexpected behaviors
* Scaling tests can happen without failures
* Velocity for creating tests increases as the utilities and examples increase
Lastly, this puts us within striking distance of creating FTR's for different common class of runtime situations such as:
* Creating tests that exercise each rule against a set of data criteria and get signal hits
* Creating tests that validate the rule overrides operate as expected against data sets
* Creating tests that validate malfunctions, corner cases, or misuse cases such as data sets that are _all_ arrays or data sets that put numbers as strings or throws in an expected `null` instead of a value.
These tests follow the pattern of:
* Add the smallest data set to a folder in data.json (not gzip format)
* Add the smallest mapping to that folder (mapping.json)
* Call REST input related to exception lists, value lists, adding prepackaged rules, etc...
* Call REST input related endpoint with utilities to create and activate the rule
* Wait for the rule to go into the `succeeded` phase
* Wait for the N exact signals specific to that rule to be available
* Check against the set of signals to ensure that the matches are exactly as expected
Example of one runtime test:
A keyword data set is added to a folder called "keyword" but you can add one anywhere you want under `es_archives`, I just grouped mine depending on the situation of the runtime. Small non-gzipped tests `data.json` and `mappings.json` are the best approach for small focused tests. For _larger_ tests and cases I would and sometimes do use things such as auditbeat but try to avoid using larger data sets in favor of smaller focused test cases to validate the runtime is operating as expected.
```ts
{
"type": "doc",
"value": {
"id": "1",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:00:53.000Z",
"long": 1
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "2",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:01:53.000Z",
"long": 2
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "3",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:02:53.000Z",
"long": 3
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "4",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:03:53.000Z",
"long": 4
},
"type": "_doc"
}
}
```
Mapping is added. Note that this is "ECS tolerant" but not necessarily all ECS meaning I can and will try to keep things simple where I can, but I have ensured that `"@timestamp"` is at least there.
```ts
{
"type": "index",
"value": {
"index": "long",
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"long": { "type": "long" }
}
},
"settings": {
"index": {
"number_of_replicas": "1",
"number_of_shards": "1"
}
}
}
}
```
Test is written with test utilities where the `beforeEach` and `afterEach` try and clean up the indexes and load/unload the archives to keep one test from effecting another. Note this is never going to be 100% possible so see below on how we add more determinism in case something escapes the sandbox.
```ts
beforeEach(async () => {
await createSignalsIndex(supertest);
await createListsIndex(supertest);
await esArchiver.load('rule_exceptions/keyword');
});
afterEach(async () => {
await deleteSignalsIndex(supertest);
await deleteAllAlerts(supertest);
await deleteAllExceptions(es);
await deleteListsIndex(supertest);
await esArchiver.unload('rule_exceptions/keyword');
});
describe('"is" operator', () => {
it('should filter 1 single keyword if it is set as an exception', async () => {
const rule = getRuleForSignalTesting(['keyword']);
const { id } = await createRuleWithExceptionEntries(supertest, rule, [
[
{
field: 'keyword',
operator: 'included',
type: 'match',
value: 'word one',
},
],
]);
await waitForRuleSuccess(supertest, id);
await waitForSignalsToBePresent(supertest, 3, [id]);
const signalsOpen = await getSignalsById(supertest, id);
const hits = signalsOpen.hits.hits.map((hit) => hit._source.keyword).sort();
expect(hits).to.eql(['word four', 'word three', 'word two']);
});
});
```
### Changes for better determinism
To support more determinism there are changes and utilities added which can be tuned during any sporadic failures we might encounter as well as better support unexpected changes to other Elastic Stack pieces such as alerting, task manager, etc...
Get simple rule and others are now defaulting to false, meaning that the structural tests will no longer activate a rule and run it on task manger. This should cut down on error outputs as well as reduce stress and potentials for left over rules interfering with the runtime rules.
```ts
export const getSimpleRule = (ruleId = 'rule-1', enabled = false): QueryCreateSchema => ({
```
Not mandatory to use, but for most tests that should be runtime based tests, I use this function below which will enable it by default and run it using settings such as `type: 'query'`, `query: '*:*',` `from: '1900-01-01T00:00:00.000Z'`, to cut down on boiler plate noise. However, people can use whatever they want out of the grab bag or if their test is more readable to hand craft a REST request to create signals, or if they just want to call this and override where they want to, then 👍 .
```ts
export const getRuleForSignalTesting = (index: string[], ruleId = 'rule-1', enabled = true)
```
This waits for a rule to succeed before continuing
```ts
await waitForRuleSuccess(supertest, id);
```
I added a required array of id that _waits_ only for that particular id here. This is useful in case another test did not cleanup and you are getting signals being produced or left behind but need to wait specifically for yours.
```ts
await waitForSignalsToBePresent(supertest, 4, [id]);
```
I only get the signals for a particular rule id using either the auto-generated id or the rule_id. It's safer to use the ones from the auto-generated id but either of these are fine if you're careful enough.
```ts
const signalsOpen = await getSignalsById(supertest, id);
const signalsOpen = await getSignalsByIds(supertest, [createdId]);
const signalsOpen = await getSignalsByRuleIds(supertest, ['signal-on-signal']);
```
I delete all alerts now through a series of steps where it properly removes all rules using the rules bulk_delete and does it in such a way that all the API keys and alerting will be the best it can destroyed as well as double check that the alerts are showing up as being cleaned up before continuing.
```ts
deleteAllAlerts()
```
When not explicitly testing something structural, prefer to use the utilities which can and will do retries in case there are over the wire failures or es failures. Examples are:
```ts
installPrePackagedRules()
waitForRuleSuccess()
importFile() // This does a _lot_ of checks to ensure that the file is fully imported before continuing
```
Some of these utilities might still do a `expect(200);` but as we are and should use regular structural tests to cover those problems, these will probably be more and more removed when/if we hit test failures in favor of doing retries, waitFor, and countDowns.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2020-11-20 20:09:38 +01:00
* @param ruleId The rule id
* @param enabled Set to tru to enable it , by default it is off
[SIEM] Create ML Rules (#58053)
* Remove unnecessary linter exceptions
Not sure what was causing issues here, but it's gone now.
* WIP: Simple form to test creation of ML rules
This will be integrated into the regular rule creation workflow, but for
now this simple form should allow us to exercise the full ML rule
workflow.
* WIP: Adds POST to backend, and type/payload changes necessary to make that work
* Simplify logic with Math.min
* WIP: Failed spike of making an http call
* WIP: Hacking together an ML client
The rest of this is going to be easier if I have actual data. For now
this is mostly copy/pasted and simplified ML code. I've hardcoded time
ranges to a period I know has data for a particular job.
* Threading through our new ML Rule params
It's a bummer that we normalize our rule alert params across all rule
types currently, but that's the deal.
* Retrieve our anomalies during rule execution
Next step: generate signals
* WIP: Generate ECS-compatible ML Signals
This uses as much of the existing signal-creation code as possible. I
skipped the search_after stuff for now because it would require us
recreating the anomalies query which we really shouldn't own. For now,
here's how it works:
* Adds a separate branch of the rule executor for machine_learning rules
* In that branch, we call our new bulkCreateMlSignal function
* This function first transforms the anomaly document into ECS fields
* We then pass the transformed documents to singleBulkCreate, which
does the rest
* After both branches, we update the rule's status appropriately.
We need to do some more work on the anomaly transformation, but this
works!
* Extract setting of rule failure to helper function
We were doing this identically in three places.
* Remove unused import
* Define a field for our Rule Type selection
This adds most of the markup and logic to allow an ML rule type to be
selected. We still need to add things like license-checking and
showing/hiding of fields based on type.
* Hide Query Fields when ML is selected
These are still getting set on the form. We'll need to filter these
fields before we send off the data, and not show them on the readonly
display either.
ALso, edit is majorly broken.
* Add input field for anomaly threshold
* Display numberic values in the readonly view of a step
TIL that isEmpty returns false for numbers and other non-iterable
values. I don't think it's exactly what we want here, but until I figure
out the intention this gets our anomalyThreshold showing up without a
separate logic branch here. Removes the unnecessary branch that was
redundant with the 'else' clause.
* Add field for selecting an ML job
This is not the same as the mockups and lacks some functionality, but
it'll allow us to select a job for now.
* Format our new ML Fields when sending them to the server
So that we don't get rejected due to snake case vs camelcase.
* Put back code that respects a rule's schedule
It was previously hardcoded to a time period I knew had anomalies.
* ML fields are optional in our creation step
In that we don't initialize them like we do the query (default) fields.
* Only send along type-specific Rule fields from form
This makes any query- or ML-specific fields optional on a Rule, and
performs some logic on the frontend to group and include these fieldsets
conditionally based on the user's selection. The one place we don't
handle this well is on the readonly view of a completed step in the
rules creation, but we'll address that.
* Rename anomalies query
It's no longer tabular data. If we need that, we can use the ML client.
* Remove spike page with simple form
* Remove unneeded ES option
This response isn't going to HTTP, which is where this option would
matter.
* Fix bulk create logic
I made a happy accident and flipped the logic here, which meant we
weren't capping the signals we created.
* Rename argument
Value is a little more ambiguous than data, here: this is our step data.
* Create Rule form stores all values, but filters by type for use
When sending off to the backend, or displaying on the readonly view, we
inspect which rule type we've currently selected, and filter our form
values appropriately.
* Fix editing of ML fields on Rule Create
We need to inherit the field value from our form on initial render, and
everything works as expected.
* Clear form errors when switching between rule types
Validation errors prevent us from moving to the next step, so it was
previously possible to get an error for Query fields, switch to an ML
rule, and be unable to continue because the form had Query errors.
This also adds a helper for checking whether a ruleType is ML, to
prevent having to change all these references if the type string
changes.
* Validate the selection of an ML Job
* Fix type errors on frontend
According to the types, this is essentially the opposite of formatRule,
so we need to reinflate all potential form values from the rule.
* Don't set defaults for query-specific rules
For ML rules these types should not be included.
* Return ML Fields in Rule responses
This adds these fields to our rule serialization, and then adds
conditional validation around those fields if the rule type is ML.
Conversely, we moved the 'language' and 'query' fields to be
conditionally validated if the rule is a query/saved_query rule.
* Fix editing of ML rules by changing who controls the field values
The source of truth for their state is the parent form object; these
inputs should not have local state.
* Fix type errors related to new ML fields
In adding the new ML fields, some other fields (e.g. `query` and
`index`) that were previously required but implicitly part of Query
Rules are now marked as optional.
Consequently, any downstream code that actually required these fields
started to complain. In general, the fix was to verify that those fields
exist, and throw an error otherwise as to appease the linter.
Runtime-wise, the new ML rules/signals follow a separate code path and
both branches should be unaffected by these changes; the issue is simply
that our conditional types don't work well with Typescript.
* Fix failing route tests
Error message changed.
* Fix integration tests
We were not sending required properties when creating a rule(index and
language).
* Fix non-ML Rule creation
I was accidentally dropping this parameter for our POST payload. Whoops.
* More informative logging during ML signal generation
The messaging diverged from the normal path here because we don't have
index patterns to display. However, we have the rest of the rule
context, and should report it appropriately.
* Prefer keyof for string union types
* Tidy up our new form components
* Type them as React.FCs
* Remove unnecessary use of styled-components
* Prefer destructuring to lodash's omit
* Fix mock params for helper functions
These were updated to take simpler parameters.
* Remove any type
This could have been a boolean all along, whoops
* Fix mock types
* Update outdated tests
These were added on master, but behavior has been changed on my branch.
* Add some tests around our helper function
I need to refactor it, so this is as good a time as any to pin down the
behavior.
* Remove uses of any in favor of actual types
Mainly leverages ML typings instead of our placeholder types. This
required handling a null case in our formatting of anomalies.
* Annotate our anomalies with @timestamp field
We were notably lacking this ECS field in our post-conversion anomalies,
and typescript was rightly complaining about it.
* ml_job_id -> machine_learning_job_id
* PR Feedback
* Stricter threshold type
* More robust date parsing
* More informative log/error messages
* Remove redundant runtime checks
* Cleaning up our new ML types
* Fix types on our Rest types
* Use less ambiguous machineLearningJobId over mlJobId
* Declare our ML params as required keys, and ensure we pass them around
everywhere we might need them (creating, importing, updating rules).
* Use implicit type to avoid the need for a ts-ignore
FormSchema has a very generic index signature such that our
filterRuleFieldsForType helper cannot infer that it has our necessary
rule fields (when in fact it does). By removing the FormSchema hint we
get the actual keys of our schema, and things work as expected.
All other uses of schema continue to work because they're expecting
FormSchema, which is effectively { [key: string]: any }.
* New ML params are not nullable
Rather than setting a null and then never using it, let's just make it
truly optional in terms of default values.
* Query and language are conditional based on rule type
For ML Rules, we don't use them.
* Remove defaulted parameter in API test
We don't need to specify this, and we should continue not to for
backwards compatibility.
* Use explicit types over implicit ones
The concern is that not typing our schemae as FormSchema could break our
form if there are upstream changes. For now, we simply use the
intersection of FormSchema and our generic parameter to satisfy our use
within the function.
* Add integration test for creation of ML Rule
* Add ML fields to route schemae
* threshold and job id are conditional on type
* makes query and language mutually exclusive with above
* Fix router test for creating an ML rule
We were sending invalid parameters.
* Remove null check against index for query rules
We support not having an index here, as getInputIndex will return the
current UI setting if none is specified.
* Add regression test for API compatibility
We were previously able to create a rule without an input index; we
should continue to support that, as verified by this test!
* Respect the index pattern determined at runtime when performing search_after
If a rule does not specify an input index pattern on creation, we use
the current UI default when the rule is evaluated. This ensures that any
subsequent searches use that same index.
We're not currently persisting that runtime index to the generated
signal, but we should.
* Fix type errors in our bulk create tests
We added a new argument, but didn't update the tests.
2020-03-19 01:26:42 +01:00
* /
[Security Solutions][Detection Engine] Adds e2e FTR runtime support and 213 tests for exception lists (#83764)
## Summary
Adds support to the end to end (e2e) functional test runner (FTR) support for rule runtime tests as well as 213 tests for the exception lists which include value based lists. Previously we had limited runtime support, but as I scaled up runtime tests from 5 to 200+ I noticed in a lot of areas we had to use improved techniques for determinism.
The runtime support being added is our next step of tests. Up to now most of our e2e FTR tests have been structural testing of REST and API integration tests. Basically up to now 95% tests are API structural as:
* Call REST input related to a rule such as GET/PUT/POST/PATCH/DELETE.
* Check REST output of the rule, did it match expected output body and status code?
* In some rare cases we check if the the rule can be executed and we get a status of 'succeeded'
With only a small part of our tests ~5%, `generating_signals.ts` was checking the signals being produced. However, we cannot have confidence in runtime based tests until the structural tests have been built up and run through the weeks against PR's to ensure that those are stable and deterministic.
Now that we have confidence and 90%+ coverage of the structural REST based tests, we are building up newer sets of tests which allow us to do runtime based validation tests to increase confidence that:
* Detection engine produces signals as expected
* Structure of the signals are as expected, including signal on signals
* Exceptions to signals are working as expected
* Most runtime bugs can be TDD'ed with e2e FTR's and regressions
* Whack-a-mole will not happen
* Consistency and predictability of signals is validated
* Refactoring can occur with stronger confidence
* Runtime tests are reference points for answering questions about existing bugs or adding new ones to test if users are experiencing unexpected behaviors
* Scaling tests can happen without failures
* Velocity for creating tests increases as the utilities and examples increase
Lastly, this puts us within striking distance of creating FTR's for different common class of runtime situations such as:
* Creating tests that exercise each rule against a set of data criteria and get signal hits
* Creating tests that validate the rule overrides operate as expected against data sets
* Creating tests that validate malfunctions, corner cases, or misuse cases such as data sets that are _all_ arrays or data sets that put numbers as strings or throws in an expected `null` instead of a value.
These tests follow the pattern of:
* Add the smallest data set to a folder in data.json (not gzip format)
* Add the smallest mapping to that folder (mapping.json)
* Call REST input related to exception lists, value lists, adding prepackaged rules, etc...
* Call REST input related endpoint with utilities to create and activate the rule
* Wait for the rule to go into the `succeeded` phase
* Wait for the N exact signals specific to that rule to be available
* Check against the set of signals to ensure that the matches are exactly as expected
Example of one runtime test:
A keyword data set is added to a folder called "keyword" but you can add one anywhere you want under `es_archives`, I just grouped mine depending on the situation of the runtime. Small non-gzipped tests `data.json` and `mappings.json` are the best approach for small focused tests. For _larger_ tests and cases I would and sometimes do use things such as auditbeat but try to avoid using larger data sets in favor of smaller focused test cases to validate the runtime is operating as expected.
```ts
{
"type": "doc",
"value": {
"id": "1",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:00:53.000Z",
"long": 1
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "2",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:01:53.000Z",
"long": 2
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "3",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:02:53.000Z",
"long": 3
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "4",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:03:53.000Z",
"long": 4
},
"type": "_doc"
}
}
```
Mapping is added. Note that this is "ECS tolerant" but not necessarily all ECS meaning I can and will try to keep things simple where I can, but I have ensured that `"@timestamp"` is at least there.
```ts
{
"type": "index",
"value": {
"index": "long",
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"long": { "type": "long" }
}
},
"settings": {
"index": {
"number_of_replicas": "1",
"number_of_shards": "1"
}
}
}
}
```
Test is written with test utilities where the `beforeEach` and `afterEach` try and clean up the indexes and load/unload the archives to keep one test from effecting another. Note this is never going to be 100% possible so see below on how we add more determinism in case something escapes the sandbox.
```ts
beforeEach(async () => {
await createSignalsIndex(supertest);
await createListsIndex(supertest);
await esArchiver.load('rule_exceptions/keyword');
});
afterEach(async () => {
await deleteSignalsIndex(supertest);
await deleteAllAlerts(supertest);
await deleteAllExceptions(es);
await deleteListsIndex(supertest);
await esArchiver.unload('rule_exceptions/keyword');
});
describe('"is" operator', () => {
it('should filter 1 single keyword if it is set as an exception', async () => {
const rule = getRuleForSignalTesting(['keyword']);
const { id } = await createRuleWithExceptionEntries(supertest, rule, [
[
{
field: 'keyword',
operator: 'included',
type: 'match',
value: 'word one',
},
],
]);
await waitForRuleSuccess(supertest, id);
await waitForSignalsToBePresent(supertest, 3, [id]);
const signalsOpen = await getSignalsById(supertest, id);
const hits = signalsOpen.hits.hits.map((hit) => hit._source.keyword).sort();
expect(hits).to.eql(['word four', 'word three', 'word two']);
});
});
```
### Changes for better determinism
To support more determinism there are changes and utilities added which can be tuned during any sporadic failures we might encounter as well as better support unexpected changes to other Elastic Stack pieces such as alerting, task manager, etc...
Get simple rule and others are now defaulting to false, meaning that the structural tests will no longer activate a rule and run it on task manger. This should cut down on error outputs as well as reduce stress and potentials for left over rules interfering with the runtime rules.
```ts
export const getSimpleRule = (ruleId = 'rule-1', enabled = false): QueryCreateSchema => ({
```
Not mandatory to use, but for most tests that should be runtime based tests, I use this function below which will enable it by default and run it using settings such as `type: 'query'`, `query: '*:*',` `from: '1900-01-01T00:00:00.000Z'`, to cut down on boiler plate noise. However, people can use whatever they want out of the grab bag or if their test is more readable to hand craft a REST request to create signals, or if they just want to call this and override where they want to, then 👍 .
```ts
export const getRuleForSignalTesting = (index: string[], ruleId = 'rule-1', enabled = true)
```
This waits for a rule to succeed before continuing
```ts
await waitForRuleSuccess(supertest, id);
```
I added a required array of id that _waits_ only for that particular id here. This is useful in case another test did not cleanup and you are getting signals being produced or left behind but need to wait specifically for yours.
```ts
await waitForSignalsToBePresent(supertest, 4, [id]);
```
I only get the signals for a particular rule id using either the auto-generated id or the rule_id. It's safer to use the ones from the auto-generated id but either of these are fine if you're careful enough.
```ts
const signalsOpen = await getSignalsById(supertest, id);
const signalsOpen = await getSignalsByIds(supertest, [createdId]);
const signalsOpen = await getSignalsByRuleIds(supertest, ['signal-on-signal']);
```
I delete all alerts now through a series of steps where it properly removes all rules using the rules bulk_delete and does it in such a way that all the API keys and alerting will be the best it can destroyed as well as double check that the alerts are showing up as being cleaned up before continuing.
```ts
deleteAllAlerts()
```
When not explicitly testing something structural, prefer to use the utilities which can and will do retries in case there are over the wire failures or es failures. Examples are:
```ts
installPrePackagedRules()
waitForRuleSuccess()
importFile() // This does a _lot_ of checks to ensure that the file is fully imported before continuing
```
Some of these utilities might still do a `expect(200);` but as we are and should use regular structural tests to cover those problems, these will probably be more and more removed when/if we hit test failures in favor of doing retries, waitFor, and countDowns.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2020-11-20 20:09:38 +01:00
export const getSimpleMlRule = ( ruleId = 'rule-1' , enabled = false ) : CreateRulesSchema = > ( {
2020-06-10 16:54:27 +02:00
name : 'Simple ML Rule' ,
description : 'Simple Machine Learning Rule' ,
[Security Solutions][Detection Engine] Adds e2e FTR runtime support and 213 tests for exception lists (#83764)
## Summary
Adds support to the end to end (e2e) functional test runner (FTR) support for rule runtime tests as well as 213 tests for the exception lists which include value based lists. Previously we had limited runtime support, but as I scaled up runtime tests from 5 to 200+ I noticed in a lot of areas we had to use improved techniques for determinism.
The runtime support being added is our next step of tests. Up to now most of our e2e FTR tests have been structural testing of REST and API integration tests. Basically up to now 95% tests are API structural as:
* Call REST input related to a rule such as GET/PUT/POST/PATCH/DELETE.
* Check REST output of the rule, did it match expected output body and status code?
* In some rare cases we check if the the rule can be executed and we get a status of 'succeeded'
With only a small part of our tests ~5%, `generating_signals.ts` was checking the signals being produced. However, we cannot have confidence in runtime based tests until the structural tests have been built up and run through the weeks against PR's to ensure that those are stable and deterministic.
Now that we have confidence and 90%+ coverage of the structural REST based tests, we are building up newer sets of tests which allow us to do runtime based validation tests to increase confidence that:
* Detection engine produces signals as expected
* Structure of the signals are as expected, including signal on signals
* Exceptions to signals are working as expected
* Most runtime bugs can be TDD'ed with e2e FTR's and regressions
* Whack-a-mole will not happen
* Consistency and predictability of signals is validated
* Refactoring can occur with stronger confidence
* Runtime tests are reference points for answering questions about existing bugs or adding new ones to test if users are experiencing unexpected behaviors
* Scaling tests can happen without failures
* Velocity for creating tests increases as the utilities and examples increase
Lastly, this puts us within striking distance of creating FTR's for different common class of runtime situations such as:
* Creating tests that exercise each rule against a set of data criteria and get signal hits
* Creating tests that validate the rule overrides operate as expected against data sets
* Creating tests that validate malfunctions, corner cases, or misuse cases such as data sets that are _all_ arrays or data sets that put numbers as strings or throws in an expected `null` instead of a value.
These tests follow the pattern of:
* Add the smallest data set to a folder in data.json (not gzip format)
* Add the smallest mapping to that folder (mapping.json)
* Call REST input related to exception lists, value lists, adding prepackaged rules, etc...
* Call REST input related endpoint with utilities to create and activate the rule
* Wait for the rule to go into the `succeeded` phase
* Wait for the N exact signals specific to that rule to be available
* Check against the set of signals to ensure that the matches are exactly as expected
Example of one runtime test:
A keyword data set is added to a folder called "keyword" but you can add one anywhere you want under `es_archives`, I just grouped mine depending on the situation of the runtime. Small non-gzipped tests `data.json` and `mappings.json` are the best approach for small focused tests. For _larger_ tests and cases I would and sometimes do use things such as auditbeat but try to avoid using larger data sets in favor of smaller focused test cases to validate the runtime is operating as expected.
```ts
{
"type": "doc",
"value": {
"id": "1",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:00:53.000Z",
"long": 1
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "2",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:01:53.000Z",
"long": 2
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "3",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:02:53.000Z",
"long": 3
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "4",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:03:53.000Z",
"long": 4
},
"type": "_doc"
}
}
```
Mapping is added. Note that this is "ECS tolerant" but not necessarily all ECS meaning I can and will try to keep things simple where I can, but I have ensured that `"@timestamp"` is at least there.
```ts
{
"type": "index",
"value": {
"index": "long",
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"long": { "type": "long" }
}
},
"settings": {
"index": {
"number_of_replicas": "1",
"number_of_shards": "1"
}
}
}
}
```
Test is written with test utilities where the `beforeEach` and `afterEach` try and clean up the indexes and load/unload the archives to keep one test from effecting another. Note this is never going to be 100% possible so see below on how we add more determinism in case something escapes the sandbox.
```ts
beforeEach(async () => {
await createSignalsIndex(supertest);
await createListsIndex(supertest);
await esArchiver.load('rule_exceptions/keyword');
});
afterEach(async () => {
await deleteSignalsIndex(supertest);
await deleteAllAlerts(supertest);
await deleteAllExceptions(es);
await deleteListsIndex(supertest);
await esArchiver.unload('rule_exceptions/keyword');
});
describe('"is" operator', () => {
it('should filter 1 single keyword if it is set as an exception', async () => {
const rule = getRuleForSignalTesting(['keyword']);
const { id } = await createRuleWithExceptionEntries(supertest, rule, [
[
{
field: 'keyword',
operator: 'included',
type: 'match',
value: 'word one',
},
],
]);
await waitForRuleSuccess(supertest, id);
await waitForSignalsToBePresent(supertest, 3, [id]);
const signalsOpen = await getSignalsById(supertest, id);
const hits = signalsOpen.hits.hits.map((hit) => hit._source.keyword).sort();
expect(hits).to.eql(['word four', 'word three', 'word two']);
});
});
```
### Changes for better determinism
To support more determinism there are changes and utilities added which can be tuned during any sporadic failures we might encounter as well as better support unexpected changes to other Elastic Stack pieces such as alerting, task manager, etc...
Get simple rule and others are now defaulting to false, meaning that the structural tests will no longer activate a rule and run it on task manger. This should cut down on error outputs as well as reduce stress and potentials for left over rules interfering with the runtime rules.
```ts
export const getSimpleRule = (ruleId = 'rule-1', enabled = false): QueryCreateSchema => ({
```
Not mandatory to use, but for most tests that should be runtime based tests, I use this function below which will enable it by default and run it using settings such as `type: 'query'`, `query: '*:*',` `from: '1900-01-01T00:00:00.000Z'`, to cut down on boiler plate noise. However, people can use whatever they want out of the grab bag or if their test is more readable to hand craft a REST request to create signals, or if they just want to call this and override where they want to, then 👍 .
```ts
export const getRuleForSignalTesting = (index: string[], ruleId = 'rule-1', enabled = true)
```
This waits for a rule to succeed before continuing
```ts
await waitForRuleSuccess(supertest, id);
```
I added a required array of id that _waits_ only for that particular id here. This is useful in case another test did not cleanup and you are getting signals being produced or left behind but need to wait specifically for yours.
```ts
await waitForSignalsToBePresent(supertest, 4, [id]);
```
I only get the signals for a particular rule id using either the auto-generated id or the rule_id. It's safer to use the ones from the auto-generated id but either of these are fine if you're careful enough.
```ts
const signalsOpen = await getSignalsById(supertest, id);
const signalsOpen = await getSignalsByIds(supertest, [createdId]);
const signalsOpen = await getSignalsByRuleIds(supertest, ['signal-on-signal']);
```
I delete all alerts now through a series of steps where it properly removes all rules using the rules bulk_delete and does it in such a way that all the API keys and alerting will be the best it can destroyed as well as double check that the alerts are showing up as being cleaned up before continuing.
```ts
deleteAllAlerts()
```
When not explicitly testing something structural, prefer to use the utilities which can and will do retries in case there are over the wire failures or es failures. Examples are:
```ts
installPrePackagedRules()
waitForRuleSuccess()
importFile() // This does a _lot_ of checks to ensure that the file is fully imported before continuing
```
Some of these utilities might still do a `expect(200);` but as we are and should use regular structural tests to cover those problems, these will probably be more and more removed when/if we hit test failures in favor of doing retries, waitFor, and countDowns.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2020-11-20 20:09:38 +01:00
enabled ,
2020-06-10 16:54:27 +02:00
anomaly_threshold : 44 ,
risk_score : 1 ,
rule_id : ruleId ,
severity : 'high' ,
[Security Solution][Detections] ML Rules accept multiple ML Job IDs (#97073)
* Adds helper to normalize legacy ML rule field to an array
This will be used on read of rules, to normalize legacy rules while
avoiding an explicit migration.
* Fix our detection-specific ML search function
Luckily this was just a translation layer to our anomaly call, and the
underlying functions already accepted an array of strings.
* WIP: Run rules against multiple ML Job IDs
We don't yet support creation of rules with multiple job ids, either on
the API or the UI, but when we do they will work.
Note: the logic was previously to generate an error if the underlying
job was not running, but to still query and generate alerts. Extending
that logic to multiple jobs: if any are not running, we generate an
error but continue querying and generating alerts.
* WIP: updating ml rule schemas to support multiple job IDs
* Simplify normalization method
We don't care about null or empty string values here; those were
holdovers from copying the logic of normalizeThreshold and don't apply
to this situation.
* Move normalized types to separate file to fix circular dependency
Our use of NonEmptyArray within common/schemas seemed to be causing the
above; this fixes it for now.
* Normalize ML job_ids param at the API layer
Previous changes to the base types already covered the majority of
routes; this updates the miscellaneous helpers that don't leverage those
shared utilities.
At the DB level, the forthcoming migration will ensure that we always
have "normalized" job IDs as an array.
* Count stopped ML Jobs as partial failure during ML Rule execution
Since we continue to query anomalies and potentially generate alerts, a
"failure" status is no longer the most accurate for this situation.
* Update 7.13 alerts migration to allow multi-job ML Rules
This ensures that we can assume string[] for this field during rule
execution.
* Display N job statuses on rule details
* WIP: converts MLJobSelect to a multiselect
Unfortunately, the SuperSelect does not allow multiselect so we need to
convert this to a combobox. Luckily we can reuse most of the code here
and remain relatively clean.
Since all combobox options must be the same (fixed) height, we're
somewhat more limited than before for displaying the rows. The
truncation appears fine, but I need to figure out a way to display the
full description as well.
* Update client-side logic to handle an array of ML job_ids
* Marginally more legible error message
* Conditionally call our normalize helper only if we have a value
This fixes a type error where TS could not infer that the return value
would not be undefined despite knowing that the argument was never
undefined. I tried some fancy conditional generic types, but that didn't
work.
This is more analogous to normalizeThresholdObject now, anyway.
* Fix remaining type error
* Clean up our ML executor tests with existing contract mocks
* Update ML Executor tests with new logic
We now record a partial failure instead of an error.
* Add and update tests for new ML normalization logic
* Add and update integration tests for ML Rules
Ensures that dealing with legacy job formats continues to work in the
API.
* Fix a type error
These params can no longer be strings.
* Update ML cypress test to create a rule with 2 ML jobs
If we can create a rule with 2 jobs, we should also be able to create a
rule with 1 job.
* Remove unused constant
* Persist a partial failure message written by a rule executor
We added the result.warning field as a way to indicate that a partial
failure was written to the rule, but neglected to account for that in the
main rule execution code, which caused a success status to immediately
overwrite the partial failure if the rule execution did not otherwise
fail/short-circuit.
2021-04-16 04:27:43 +02:00
machine_learning_job_id : [ 'some_job_id' ] ,
2020-06-10 16:54:27 +02:00
type : 'machine_learning' ,
} ) ;
[Security Solutions][Detection Engine] Adds e2e FTR runtime support and 213 tests for exception lists (#83764)
## Summary
Adds support to the end to end (e2e) functional test runner (FTR) support for rule runtime tests as well as 213 tests for the exception lists which include value based lists. Previously we had limited runtime support, but as I scaled up runtime tests from 5 to 200+ I noticed in a lot of areas we had to use improved techniques for determinism.
The runtime support being added is our next step of tests. Up to now most of our e2e FTR tests have been structural testing of REST and API integration tests. Basically up to now 95% tests are API structural as:
* Call REST input related to a rule such as GET/PUT/POST/PATCH/DELETE.
* Check REST output of the rule, did it match expected output body and status code?
* In some rare cases we check if the the rule can be executed and we get a status of 'succeeded'
With only a small part of our tests ~5%, `generating_signals.ts` was checking the signals being produced. However, we cannot have confidence in runtime based tests until the structural tests have been built up and run through the weeks against PR's to ensure that those are stable and deterministic.
Now that we have confidence and 90%+ coverage of the structural REST based tests, we are building up newer sets of tests which allow us to do runtime based validation tests to increase confidence that:
* Detection engine produces signals as expected
* Structure of the signals are as expected, including signal on signals
* Exceptions to signals are working as expected
* Most runtime bugs can be TDD'ed with e2e FTR's and regressions
* Whack-a-mole will not happen
* Consistency and predictability of signals is validated
* Refactoring can occur with stronger confidence
* Runtime tests are reference points for answering questions about existing bugs or adding new ones to test if users are experiencing unexpected behaviors
* Scaling tests can happen without failures
* Velocity for creating tests increases as the utilities and examples increase
Lastly, this puts us within striking distance of creating FTR's for different common class of runtime situations such as:
* Creating tests that exercise each rule against a set of data criteria and get signal hits
* Creating tests that validate the rule overrides operate as expected against data sets
* Creating tests that validate malfunctions, corner cases, or misuse cases such as data sets that are _all_ arrays or data sets that put numbers as strings or throws in an expected `null` instead of a value.
These tests follow the pattern of:
* Add the smallest data set to a folder in data.json (not gzip format)
* Add the smallest mapping to that folder (mapping.json)
* Call REST input related to exception lists, value lists, adding prepackaged rules, etc...
* Call REST input related endpoint with utilities to create and activate the rule
* Wait for the rule to go into the `succeeded` phase
* Wait for the N exact signals specific to that rule to be available
* Check against the set of signals to ensure that the matches are exactly as expected
Example of one runtime test:
A keyword data set is added to a folder called "keyword" but you can add one anywhere you want under `es_archives`, I just grouped mine depending on the situation of the runtime. Small non-gzipped tests `data.json` and `mappings.json` are the best approach for small focused tests. For _larger_ tests and cases I would and sometimes do use things such as auditbeat but try to avoid using larger data sets in favor of smaller focused test cases to validate the runtime is operating as expected.
```ts
{
"type": "doc",
"value": {
"id": "1",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:00:53.000Z",
"long": 1
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "2",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:01:53.000Z",
"long": 2
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "3",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:02:53.000Z",
"long": 3
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "4",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:03:53.000Z",
"long": 4
},
"type": "_doc"
}
}
```
Mapping is added. Note that this is "ECS tolerant" but not necessarily all ECS meaning I can and will try to keep things simple where I can, but I have ensured that `"@timestamp"` is at least there.
```ts
{
"type": "index",
"value": {
"index": "long",
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"long": { "type": "long" }
}
},
"settings": {
"index": {
"number_of_replicas": "1",
"number_of_shards": "1"
}
}
}
}
```
Test is written with test utilities where the `beforeEach` and `afterEach` try and clean up the indexes and load/unload the archives to keep one test from effecting another. Note this is never going to be 100% possible so see below on how we add more determinism in case something escapes the sandbox.
```ts
beforeEach(async () => {
await createSignalsIndex(supertest);
await createListsIndex(supertest);
await esArchiver.load('rule_exceptions/keyword');
});
afterEach(async () => {
await deleteSignalsIndex(supertest);
await deleteAllAlerts(supertest);
await deleteAllExceptions(es);
await deleteListsIndex(supertest);
await esArchiver.unload('rule_exceptions/keyword');
});
describe('"is" operator', () => {
it('should filter 1 single keyword if it is set as an exception', async () => {
const rule = getRuleForSignalTesting(['keyword']);
const { id } = await createRuleWithExceptionEntries(supertest, rule, [
[
{
field: 'keyword',
operator: 'included',
type: 'match',
value: 'word one',
},
],
]);
await waitForRuleSuccess(supertest, id);
await waitForSignalsToBePresent(supertest, 3, [id]);
const signalsOpen = await getSignalsById(supertest, id);
const hits = signalsOpen.hits.hits.map((hit) => hit._source.keyword).sort();
expect(hits).to.eql(['word four', 'word three', 'word two']);
});
});
```
### Changes for better determinism
To support more determinism there are changes and utilities added which can be tuned during any sporadic failures we might encounter as well as better support unexpected changes to other Elastic Stack pieces such as alerting, task manager, etc...
Get simple rule and others are now defaulting to false, meaning that the structural tests will no longer activate a rule and run it on task manger. This should cut down on error outputs as well as reduce stress and potentials for left over rules interfering with the runtime rules.
```ts
export const getSimpleRule = (ruleId = 'rule-1', enabled = false): QueryCreateSchema => ({
```
Not mandatory to use, but for most tests that should be runtime based tests, I use this function below which will enable it by default and run it using settings such as `type: 'query'`, `query: '*:*',` `from: '1900-01-01T00:00:00.000Z'`, to cut down on boiler plate noise. However, people can use whatever they want out of the grab bag or if their test is more readable to hand craft a REST request to create signals, or if they just want to call this and override where they want to, then 👍 .
```ts
export const getRuleForSignalTesting = (index: string[], ruleId = 'rule-1', enabled = true)
```
This waits for a rule to succeed before continuing
```ts
await waitForRuleSuccess(supertest, id);
```
I added a required array of id that _waits_ only for that particular id here. This is useful in case another test did not cleanup and you are getting signals being produced or left behind but need to wait specifically for yours.
```ts
await waitForSignalsToBePresent(supertest, 4, [id]);
```
I only get the signals for a particular rule id using either the auto-generated id or the rule_id. It's safer to use the ones from the auto-generated id but either of these are fine if you're careful enough.
```ts
const signalsOpen = await getSignalsById(supertest, id);
const signalsOpen = await getSignalsByIds(supertest, [createdId]);
const signalsOpen = await getSignalsByRuleIds(supertest, ['signal-on-signal']);
```
I delete all alerts now through a series of steps where it properly removes all rules using the rules bulk_delete and does it in such a way that all the API keys and alerting will be the best it can destroyed as well as double check that the alerts are showing up as being cleaned up before continuing.
```ts
deleteAllAlerts()
```
When not explicitly testing something structural, prefer to use the utilities which can and will do retries in case there are over the wire failures or es failures. Examples are:
```ts
installPrePackagedRules()
waitForRuleSuccess()
importFile() // This does a _lot_ of checks to ensure that the file is fully imported before continuing
```
Some of these utilities might still do a `expect(200);` but as we are and should use regular structural tests to cover those problems, these will probably be more and more removed when/if we hit test failures in favor of doing retries, waitFor, and countDowns.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2020-11-20 20:09:38 +01:00
/ * *
* This is a representative ML rule payload as expected by the server for an update
* @param ruleId The rule id
* @param enabled Set to tru to enable it , by default it is off
* /
export const getSimpleMlRuleUpdate = ( ruleId = 'rule-1' , enabled = false ) : UpdateRulesSchema = > ( {
[SIEM] Create ML Rules (#58053)
* Remove unnecessary linter exceptions
Not sure what was causing issues here, but it's gone now.
* WIP: Simple form to test creation of ML rules
This will be integrated into the regular rule creation workflow, but for
now this simple form should allow us to exercise the full ML rule
workflow.
* WIP: Adds POST to backend, and type/payload changes necessary to make that work
* Simplify logic with Math.min
* WIP: Failed spike of making an http call
* WIP: Hacking together an ML client
The rest of this is going to be easier if I have actual data. For now
this is mostly copy/pasted and simplified ML code. I've hardcoded time
ranges to a period I know has data for a particular job.
* Threading through our new ML Rule params
It's a bummer that we normalize our rule alert params across all rule
types currently, but that's the deal.
* Retrieve our anomalies during rule execution
Next step: generate signals
* WIP: Generate ECS-compatible ML Signals
This uses as much of the existing signal-creation code as possible. I
skipped the search_after stuff for now because it would require us
recreating the anomalies query which we really shouldn't own. For now,
here's how it works:
* Adds a separate branch of the rule executor for machine_learning rules
* In that branch, we call our new bulkCreateMlSignal function
* This function first transforms the anomaly document into ECS fields
* We then pass the transformed documents to singleBulkCreate, which
does the rest
* After both branches, we update the rule's status appropriately.
We need to do some more work on the anomaly transformation, but this
works!
* Extract setting of rule failure to helper function
We were doing this identically in three places.
* Remove unused import
* Define a field for our Rule Type selection
This adds most of the markup and logic to allow an ML rule type to be
selected. We still need to add things like license-checking and
showing/hiding of fields based on type.
* Hide Query Fields when ML is selected
These are still getting set on the form. We'll need to filter these
fields before we send off the data, and not show them on the readonly
display either.
ALso, edit is majorly broken.
* Add input field for anomaly threshold
* Display numberic values in the readonly view of a step
TIL that isEmpty returns false for numbers and other non-iterable
values. I don't think it's exactly what we want here, but until I figure
out the intention this gets our anomalyThreshold showing up without a
separate logic branch here. Removes the unnecessary branch that was
redundant with the 'else' clause.
* Add field for selecting an ML job
This is not the same as the mockups and lacks some functionality, but
it'll allow us to select a job for now.
* Format our new ML Fields when sending them to the server
So that we don't get rejected due to snake case vs camelcase.
* Put back code that respects a rule's schedule
It was previously hardcoded to a time period I knew had anomalies.
* ML fields are optional in our creation step
In that we don't initialize them like we do the query (default) fields.
* Only send along type-specific Rule fields from form
This makes any query- or ML-specific fields optional on a Rule, and
performs some logic on the frontend to group and include these fieldsets
conditionally based on the user's selection. The one place we don't
handle this well is on the readonly view of a completed step in the
rules creation, but we'll address that.
* Rename anomalies query
It's no longer tabular data. If we need that, we can use the ML client.
* Remove spike page with simple form
* Remove unneeded ES option
This response isn't going to HTTP, which is where this option would
matter.
* Fix bulk create logic
I made a happy accident and flipped the logic here, which meant we
weren't capping the signals we created.
* Rename argument
Value is a little more ambiguous than data, here: this is our step data.
* Create Rule form stores all values, but filters by type for use
When sending off to the backend, or displaying on the readonly view, we
inspect which rule type we've currently selected, and filter our form
values appropriately.
* Fix editing of ML fields on Rule Create
We need to inherit the field value from our form on initial render, and
everything works as expected.
* Clear form errors when switching between rule types
Validation errors prevent us from moving to the next step, so it was
previously possible to get an error for Query fields, switch to an ML
rule, and be unable to continue because the form had Query errors.
This also adds a helper for checking whether a ruleType is ML, to
prevent having to change all these references if the type string
changes.
* Validate the selection of an ML Job
* Fix type errors on frontend
According to the types, this is essentially the opposite of formatRule,
so we need to reinflate all potential form values from the rule.
* Don't set defaults for query-specific rules
For ML rules these types should not be included.
* Return ML Fields in Rule responses
This adds these fields to our rule serialization, and then adds
conditional validation around those fields if the rule type is ML.
Conversely, we moved the 'language' and 'query' fields to be
conditionally validated if the rule is a query/saved_query rule.
* Fix editing of ML rules by changing who controls the field values
The source of truth for their state is the parent form object; these
inputs should not have local state.
* Fix type errors related to new ML fields
In adding the new ML fields, some other fields (e.g. `query` and
`index`) that were previously required but implicitly part of Query
Rules are now marked as optional.
Consequently, any downstream code that actually required these fields
started to complain. In general, the fix was to verify that those fields
exist, and throw an error otherwise as to appease the linter.
Runtime-wise, the new ML rules/signals follow a separate code path and
both branches should be unaffected by these changes; the issue is simply
that our conditional types don't work well with Typescript.
* Fix failing route tests
Error message changed.
* Fix integration tests
We were not sending required properties when creating a rule(index and
language).
* Fix non-ML Rule creation
I was accidentally dropping this parameter for our POST payload. Whoops.
* More informative logging during ML signal generation
The messaging diverged from the normal path here because we don't have
index patterns to display. However, we have the rest of the rule
context, and should report it appropriately.
* Prefer keyof for string union types
* Tidy up our new form components
* Type them as React.FCs
* Remove unnecessary use of styled-components
* Prefer destructuring to lodash's omit
* Fix mock params for helper functions
These were updated to take simpler parameters.
* Remove any type
This could have been a boolean all along, whoops
* Fix mock types
* Update outdated tests
These were added on master, but behavior has been changed on my branch.
* Add some tests around our helper function
I need to refactor it, so this is as good a time as any to pin down the
behavior.
* Remove uses of any in favor of actual types
Mainly leverages ML typings instead of our placeholder types. This
required handling a null case in our formatting of anomalies.
* Annotate our anomalies with @timestamp field
We were notably lacking this ECS field in our post-conversion anomalies,
and typescript was rightly complaining about it.
* ml_job_id -> machine_learning_job_id
* PR Feedback
* Stricter threshold type
* More robust date parsing
* More informative log/error messages
* Remove redundant runtime checks
* Cleaning up our new ML types
* Fix types on our Rest types
* Use less ambiguous machineLearningJobId over mlJobId
* Declare our ML params as required keys, and ensure we pass them around
everywhere we might need them (creating, importing, updating rules).
* Use implicit type to avoid the need for a ts-ignore
FormSchema has a very generic index signature such that our
filterRuleFieldsForType helper cannot infer that it has our necessary
rule fields (when in fact it does). By removing the FormSchema hint we
get the actual keys of our schema, and things work as expected.
All other uses of schema continue to work because they're expecting
FormSchema, which is effectively { [key: string]: any }.
* New ML params are not nullable
Rather than setting a null and then never using it, let's just make it
truly optional in terms of default values.
* Query and language are conditional based on rule type
For ML Rules, we don't use them.
* Remove defaulted parameter in API test
We don't need to specify this, and we should continue not to for
backwards compatibility.
* Use explicit types over implicit ones
The concern is that not typing our schemae as FormSchema could break our
form if there are upstream changes. For now, we simply use the
intersection of FormSchema and our generic parameter to satisfy our use
within the function.
* Add integration test for creation of ML Rule
* Add ML fields to route schemae
* threshold and job id are conditional on type
* makes query and language mutually exclusive with above
* Fix router test for creating an ML rule
We were sending invalid parameters.
* Remove null check against index for query rules
We support not having an index here, as getInputIndex will return the
current UI setting if none is specified.
* Add regression test for API compatibility
We were previously able to create a rule without an input index; we
should continue to support that, as verified by this test!
* Respect the index pattern determined at runtime when performing search_after
If a rule does not specify an input index pattern on creation, we use
the current UI default when the rule is evaluated. This ensures that any
subsequent searches use that same index.
We're not currently persisting that runtime index to the generated
signal, but we should.
* Fix type errors in our bulk create tests
We added a new argument, but didn't update the tests.
2020-03-19 01:26:42 +01:00
name : 'Simple ML Rule' ,
description : 'Simple Machine Learning Rule' ,
[Security Solutions][Detection Engine] Adds e2e FTR runtime support and 213 tests for exception lists (#83764)
## Summary
Adds support to the end to end (e2e) functional test runner (FTR) support for rule runtime tests as well as 213 tests for the exception lists which include value based lists. Previously we had limited runtime support, but as I scaled up runtime tests from 5 to 200+ I noticed in a lot of areas we had to use improved techniques for determinism.
The runtime support being added is our next step of tests. Up to now most of our e2e FTR tests have been structural testing of REST and API integration tests. Basically up to now 95% tests are API structural as:
* Call REST input related to a rule such as GET/PUT/POST/PATCH/DELETE.
* Check REST output of the rule, did it match expected output body and status code?
* In some rare cases we check if the the rule can be executed and we get a status of 'succeeded'
With only a small part of our tests ~5%, `generating_signals.ts` was checking the signals being produced. However, we cannot have confidence in runtime based tests until the structural tests have been built up and run through the weeks against PR's to ensure that those are stable and deterministic.
Now that we have confidence and 90%+ coverage of the structural REST based tests, we are building up newer sets of tests which allow us to do runtime based validation tests to increase confidence that:
* Detection engine produces signals as expected
* Structure of the signals are as expected, including signal on signals
* Exceptions to signals are working as expected
* Most runtime bugs can be TDD'ed with e2e FTR's and regressions
* Whack-a-mole will not happen
* Consistency and predictability of signals is validated
* Refactoring can occur with stronger confidence
* Runtime tests are reference points for answering questions about existing bugs or adding new ones to test if users are experiencing unexpected behaviors
* Scaling tests can happen without failures
* Velocity for creating tests increases as the utilities and examples increase
Lastly, this puts us within striking distance of creating FTR's for different common class of runtime situations such as:
* Creating tests that exercise each rule against a set of data criteria and get signal hits
* Creating tests that validate the rule overrides operate as expected against data sets
* Creating tests that validate malfunctions, corner cases, or misuse cases such as data sets that are _all_ arrays or data sets that put numbers as strings or throws in an expected `null` instead of a value.
These tests follow the pattern of:
* Add the smallest data set to a folder in data.json (not gzip format)
* Add the smallest mapping to that folder (mapping.json)
* Call REST input related to exception lists, value lists, adding prepackaged rules, etc...
* Call REST input related endpoint with utilities to create and activate the rule
* Wait for the rule to go into the `succeeded` phase
* Wait for the N exact signals specific to that rule to be available
* Check against the set of signals to ensure that the matches are exactly as expected
Example of one runtime test:
A keyword data set is added to a folder called "keyword" but you can add one anywhere you want under `es_archives`, I just grouped mine depending on the situation of the runtime. Small non-gzipped tests `data.json` and `mappings.json` are the best approach for small focused tests. For _larger_ tests and cases I would and sometimes do use things such as auditbeat but try to avoid using larger data sets in favor of smaller focused test cases to validate the runtime is operating as expected.
```ts
{
"type": "doc",
"value": {
"id": "1",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:00:53.000Z",
"long": 1
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "2",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:01:53.000Z",
"long": 2
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "3",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:02:53.000Z",
"long": 3
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "4",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:03:53.000Z",
"long": 4
},
"type": "_doc"
}
}
```
Mapping is added. Note that this is "ECS tolerant" but not necessarily all ECS meaning I can and will try to keep things simple where I can, but I have ensured that `"@timestamp"` is at least there.
```ts
{
"type": "index",
"value": {
"index": "long",
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"long": { "type": "long" }
}
},
"settings": {
"index": {
"number_of_replicas": "1",
"number_of_shards": "1"
}
}
}
}
```
Test is written with test utilities where the `beforeEach` and `afterEach` try and clean up the indexes and load/unload the archives to keep one test from effecting another. Note this is never going to be 100% possible so see below on how we add more determinism in case something escapes the sandbox.
```ts
beforeEach(async () => {
await createSignalsIndex(supertest);
await createListsIndex(supertest);
await esArchiver.load('rule_exceptions/keyword');
});
afterEach(async () => {
await deleteSignalsIndex(supertest);
await deleteAllAlerts(supertest);
await deleteAllExceptions(es);
await deleteListsIndex(supertest);
await esArchiver.unload('rule_exceptions/keyword');
});
describe('"is" operator', () => {
it('should filter 1 single keyword if it is set as an exception', async () => {
const rule = getRuleForSignalTesting(['keyword']);
const { id } = await createRuleWithExceptionEntries(supertest, rule, [
[
{
field: 'keyword',
operator: 'included',
type: 'match',
value: 'word one',
},
],
]);
await waitForRuleSuccess(supertest, id);
await waitForSignalsToBePresent(supertest, 3, [id]);
const signalsOpen = await getSignalsById(supertest, id);
const hits = signalsOpen.hits.hits.map((hit) => hit._source.keyword).sort();
expect(hits).to.eql(['word four', 'word three', 'word two']);
});
});
```
### Changes for better determinism
To support more determinism there are changes and utilities added which can be tuned during any sporadic failures we might encounter as well as better support unexpected changes to other Elastic Stack pieces such as alerting, task manager, etc...
Get simple rule and others are now defaulting to false, meaning that the structural tests will no longer activate a rule and run it on task manger. This should cut down on error outputs as well as reduce stress and potentials for left over rules interfering with the runtime rules.
```ts
export const getSimpleRule = (ruleId = 'rule-1', enabled = false): QueryCreateSchema => ({
```
Not mandatory to use, but for most tests that should be runtime based tests, I use this function below which will enable it by default and run it using settings such as `type: 'query'`, `query: '*:*',` `from: '1900-01-01T00:00:00.000Z'`, to cut down on boiler plate noise. However, people can use whatever they want out of the grab bag or if their test is more readable to hand craft a REST request to create signals, or if they just want to call this and override where they want to, then 👍 .
```ts
export const getRuleForSignalTesting = (index: string[], ruleId = 'rule-1', enabled = true)
```
This waits for a rule to succeed before continuing
```ts
await waitForRuleSuccess(supertest, id);
```
I added a required array of id that _waits_ only for that particular id here. This is useful in case another test did not cleanup and you are getting signals being produced or left behind but need to wait specifically for yours.
```ts
await waitForSignalsToBePresent(supertest, 4, [id]);
```
I only get the signals for a particular rule id using either the auto-generated id or the rule_id. It's safer to use the ones from the auto-generated id but either of these are fine if you're careful enough.
```ts
const signalsOpen = await getSignalsById(supertest, id);
const signalsOpen = await getSignalsByIds(supertest, [createdId]);
const signalsOpen = await getSignalsByRuleIds(supertest, ['signal-on-signal']);
```
I delete all alerts now through a series of steps where it properly removes all rules using the rules bulk_delete and does it in such a way that all the API keys and alerting will be the best it can destroyed as well as double check that the alerts are showing up as being cleaned up before continuing.
```ts
deleteAllAlerts()
```
When not explicitly testing something structural, prefer to use the utilities which can and will do retries in case there are over the wire failures or es failures. Examples are:
```ts
installPrePackagedRules()
waitForRuleSuccess()
importFile() // This does a _lot_ of checks to ensure that the file is fully imported before continuing
```
Some of these utilities might still do a `expect(200);` but as we are and should use regular structural tests to cover those problems, these will probably be more and more removed when/if we hit test failures in favor of doing retries, waitFor, and countDowns.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2020-11-20 20:09:38 +01:00
enabled ,
[SIEM] Create ML Rules (#58053)
* Remove unnecessary linter exceptions
Not sure what was causing issues here, but it's gone now.
* WIP: Simple form to test creation of ML rules
This will be integrated into the regular rule creation workflow, but for
now this simple form should allow us to exercise the full ML rule
workflow.
* WIP: Adds POST to backend, and type/payload changes necessary to make that work
* Simplify logic with Math.min
* WIP: Failed spike of making an http call
* WIP: Hacking together an ML client
The rest of this is going to be easier if I have actual data. For now
this is mostly copy/pasted and simplified ML code. I've hardcoded time
ranges to a period I know has data for a particular job.
* Threading through our new ML Rule params
It's a bummer that we normalize our rule alert params across all rule
types currently, but that's the deal.
* Retrieve our anomalies during rule execution
Next step: generate signals
* WIP: Generate ECS-compatible ML Signals
This uses as much of the existing signal-creation code as possible. I
skipped the search_after stuff for now because it would require us
recreating the anomalies query which we really shouldn't own. For now,
here's how it works:
* Adds a separate branch of the rule executor for machine_learning rules
* In that branch, we call our new bulkCreateMlSignal function
* This function first transforms the anomaly document into ECS fields
* We then pass the transformed documents to singleBulkCreate, which
does the rest
* After both branches, we update the rule's status appropriately.
We need to do some more work on the anomaly transformation, but this
works!
* Extract setting of rule failure to helper function
We were doing this identically in three places.
* Remove unused import
* Define a field for our Rule Type selection
This adds most of the markup and logic to allow an ML rule type to be
selected. We still need to add things like license-checking and
showing/hiding of fields based on type.
* Hide Query Fields when ML is selected
These are still getting set on the form. We'll need to filter these
fields before we send off the data, and not show them on the readonly
display either.
ALso, edit is majorly broken.
* Add input field for anomaly threshold
* Display numberic values in the readonly view of a step
TIL that isEmpty returns false for numbers and other non-iterable
values. I don't think it's exactly what we want here, but until I figure
out the intention this gets our anomalyThreshold showing up without a
separate logic branch here. Removes the unnecessary branch that was
redundant with the 'else' clause.
* Add field for selecting an ML job
This is not the same as the mockups and lacks some functionality, but
it'll allow us to select a job for now.
* Format our new ML Fields when sending them to the server
So that we don't get rejected due to snake case vs camelcase.
* Put back code that respects a rule's schedule
It was previously hardcoded to a time period I knew had anomalies.
* ML fields are optional in our creation step
In that we don't initialize them like we do the query (default) fields.
* Only send along type-specific Rule fields from form
This makes any query- or ML-specific fields optional on a Rule, and
performs some logic on the frontend to group and include these fieldsets
conditionally based on the user's selection. The one place we don't
handle this well is on the readonly view of a completed step in the
rules creation, but we'll address that.
* Rename anomalies query
It's no longer tabular data. If we need that, we can use the ML client.
* Remove spike page with simple form
* Remove unneeded ES option
This response isn't going to HTTP, which is where this option would
matter.
* Fix bulk create logic
I made a happy accident and flipped the logic here, which meant we
weren't capping the signals we created.
* Rename argument
Value is a little more ambiguous than data, here: this is our step data.
* Create Rule form stores all values, but filters by type for use
When sending off to the backend, or displaying on the readonly view, we
inspect which rule type we've currently selected, and filter our form
values appropriately.
* Fix editing of ML fields on Rule Create
We need to inherit the field value from our form on initial render, and
everything works as expected.
* Clear form errors when switching between rule types
Validation errors prevent us from moving to the next step, so it was
previously possible to get an error for Query fields, switch to an ML
rule, and be unable to continue because the form had Query errors.
This also adds a helper for checking whether a ruleType is ML, to
prevent having to change all these references if the type string
changes.
* Validate the selection of an ML Job
* Fix type errors on frontend
According to the types, this is essentially the opposite of formatRule,
so we need to reinflate all potential form values from the rule.
* Don't set defaults for query-specific rules
For ML rules these types should not be included.
* Return ML Fields in Rule responses
This adds these fields to our rule serialization, and then adds
conditional validation around those fields if the rule type is ML.
Conversely, we moved the 'language' and 'query' fields to be
conditionally validated if the rule is a query/saved_query rule.
* Fix editing of ML rules by changing who controls the field values
The source of truth for their state is the parent form object; these
inputs should not have local state.
* Fix type errors related to new ML fields
In adding the new ML fields, some other fields (e.g. `query` and
`index`) that were previously required but implicitly part of Query
Rules are now marked as optional.
Consequently, any downstream code that actually required these fields
started to complain. In general, the fix was to verify that those fields
exist, and throw an error otherwise as to appease the linter.
Runtime-wise, the new ML rules/signals follow a separate code path and
both branches should be unaffected by these changes; the issue is simply
that our conditional types don't work well with Typescript.
* Fix failing route tests
Error message changed.
* Fix integration tests
We were not sending required properties when creating a rule(index and
language).
* Fix non-ML Rule creation
I was accidentally dropping this parameter for our POST payload. Whoops.
* More informative logging during ML signal generation
The messaging diverged from the normal path here because we don't have
index patterns to display. However, we have the rest of the rule
context, and should report it appropriately.
* Prefer keyof for string union types
* Tidy up our new form components
* Type them as React.FCs
* Remove unnecessary use of styled-components
* Prefer destructuring to lodash's omit
* Fix mock params for helper functions
These were updated to take simpler parameters.
* Remove any type
This could have been a boolean all along, whoops
* Fix mock types
* Update outdated tests
These were added on master, but behavior has been changed on my branch.
* Add some tests around our helper function
I need to refactor it, so this is as good a time as any to pin down the
behavior.
* Remove uses of any in favor of actual types
Mainly leverages ML typings instead of our placeholder types. This
required handling a null case in our formatting of anomalies.
* Annotate our anomalies with @timestamp field
We were notably lacking this ECS field in our post-conversion anomalies,
and typescript was rightly complaining about it.
* ml_job_id -> machine_learning_job_id
* PR Feedback
* Stricter threshold type
* More robust date parsing
* More informative log/error messages
* Remove redundant runtime checks
* Cleaning up our new ML types
* Fix types on our Rest types
* Use less ambiguous machineLearningJobId over mlJobId
* Declare our ML params as required keys, and ensure we pass them around
everywhere we might need them (creating, importing, updating rules).
* Use implicit type to avoid the need for a ts-ignore
FormSchema has a very generic index signature such that our
filterRuleFieldsForType helper cannot infer that it has our necessary
rule fields (when in fact it does). By removing the FormSchema hint we
get the actual keys of our schema, and things work as expected.
All other uses of schema continue to work because they're expecting
FormSchema, which is effectively { [key: string]: any }.
* New ML params are not nullable
Rather than setting a null and then never using it, let's just make it
truly optional in terms of default values.
* Query and language are conditional based on rule type
For ML Rules, we don't use them.
* Remove defaulted parameter in API test
We don't need to specify this, and we should continue not to for
backwards compatibility.
* Use explicit types over implicit ones
The concern is that not typing our schemae as FormSchema could break our
form if there are upstream changes. For now, we simply use the
intersection of FormSchema and our generic parameter to satisfy our use
within the function.
* Add integration test for creation of ML Rule
* Add ML fields to route schemae
* threshold and job id are conditional on type
* makes query and language mutually exclusive with above
* Fix router test for creating an ML rule
We were sending invalid parameters.
* Remove null check against index for query rules
We support not having an index here, as getInputIndex will return the
current UI setting if none is specified.
* Add regression test for API compatibility
We were previously able to create a rule without an input index; we
should continue to support that, as verified by this test!
* Respect the index pattern determined at runtime when performing search_after
If a rule does not specify an input index pattern on creation, we use
the current UI default when the rule is evaluated. This ensures that any
subsequent searches use that same index.
We're not currently persisting that runtime index to the generated
signal, but we should.
* Fix type errors in our bulk create tests
We added a new argument, but didn't update the tests.
2020-03-19 01:26:42 +01:00
anomaly_threshold : 44 ,
risk_score : 1 ,
rule_id : ruleId ,
severity : 'high' ,
[Security Solution][Detections] ML Rules accept multiple ML Job IDs (#97073)
* Adds helper to normalize legacy ML rule field to an array
This will be used on read of rules, to normalize legacy rules while
avoiding an explicit migration.
* Fix our detection-specific ML search function
Luckily this was just a translation layer to our anomaly call, and the
underlying functions already accepted an array of strings.
* WIP: Run rules against multiple ML Job IDs
We don't yet support creation of rules with multiple job ids, either on
the API or the UI, but when we do they will work.
Note: the logic was previously to generate an error if the underlying
job was not running, but to still query and generate alerts. Extending
that logic to multiple jobs: if any are not running, we generate an
error but continue querying and generating alerts.
* WIP: updating ml rule schemas to support multiple job IDs
* Simplify normalization method
We don't care about null or empty string values here; those were
holdovers from copying the logic of normalizeThreshold and don't apply
to this situation.
* Move normalized types to separate file to fix circular dependency
Our use of NonEmptyArray within common/schemas seemed to be causing the
above; this fixes it for now.
* Normalize ML job_ids param at the API layer
Previous changes to the base types already covered the majority of
routes; this updates the miscellaneous helpers that don't leverage those
shared utilities.
At the DB level, the forthcoming migration will ensure that we always
have "normalized" job IDs as an array.
* Count stopped ML Jobs as partial failure during ML Rule execution
Since we continue to query anomalies and potentially generate alerts, a
"failure" status is no longer the most accurate for this situation.
* Update 7.13 alerts migration to allow multi-job ML Rules
This ensures that we can assume string[] for this field during rule
execution.
* Display N job statuses on rule details
* WIP: converts MLJobSelect to a multiselect
Unfortunately, the SuperSelect does not allow multiselect so we need to
convert this to a combobox. Luckily we can reuse most of the code here
and remain relatively clean.
Since all combobox options must be the same (fixed) height, we're
somewhat more limited than before for displaying the rows. The
truncation appears fine, but I need to figure out a way to display the
full description as well.
* Update client-side logic to handle an array of ML job_ids
* Marginally more legible error message
* Conditionally call our normalize helper only if we have a value
This fixes a type error where TS could not infer that the return value
would not be undefined despite knowing that the argument was never
undefined. I tried some fancy conditional generic types, but that didn't
work.
This is more analogous to normalizeThresholdObject now, anyway.
* Fix remaining type error
* Clean up our ML executor tests with existing contract mocks
* Update ML Executor tests with new logic
We now record a partial failure instead of an error.
* Add and update tests for new ML normalization logic
* Add and update integration tests for ML Rules
Ensures that dealing with legacy job formats continues to work in the
API.
* Fix a type error
These params can no longer be strings.
* Update ML cypress test to create a rule with 2 ML jobs
If we can create a rule with 2 jobs, we should also be able to create a
rule with 1 job.
* Remove unused constant
* Persist a partial failure message written by a rule executor
We added the result.warning field as a way to indicate that a partial
failure was written to the rule, but neglected to account for that in the
main rule execution code, which caused a success status to immediately
overwrite the partial failure if the rule execution did not otherwise
fail/short-circuit.
2021-04-16 04:27:43 +02:00
machine_learning_job_id : [ 'some_job_id' ] ,
[SIEM] Create ML Rules (#58053)
* Remove unnecessary linter exceptions
Not sure what was causing issues here, but it's gone now.
* WIP: Simple form to test creation of ML rules
This will be integrated into the regular rule creation workflow, but for
now this simple form should allow us to exercise the full ML rule
workflow.
* WIP: Adds POST to backend, and type/payload changes necessary to make that work
* Simplify logic with Math.min
* WIP: Failed spike of making an http call
* WIP: Hacking together an ML client
The rest of this is going to be easier if I have actual data. For now
this is mostly copy/pasted and simplified ML code. I've hardcoded time
ranges to a period I know has data for a particular job.
* Threading through our new ML Rule params
It's a bummer that we normalize our rule alert params across all rule
types currently, but that's the deal.
* Retrieve our anomalies during rule execution
Next step: generate signals
* WIP: Generate ECS-compatible ML Signals
This uses as much of the existing signal-creation code as possible. I
skipped the search_after stuff for now because it would require us
recreating the anomalies query which we really shouldn't own. For now,
here's how it works:
* Adds a separate branch of the rule executor for machine_learning rules
* In that branch, we call our new bulkCreateMlSignal function
* This function first transforms the anomaly document into ECS fields
* We then pass the transformed documents to singleBulkCreate, which
does the rest
* After both branches, we update the rule's status appropriately.
We need to do some more work on the anomaly transformation, but this
works!
* Extract setting of rule failure to helper function
We were doing this identically in three places.
* Remove unused import
* Define a field for our Rule Type selection
This adds most of the markup and logic to allow an ML rule type to be
selected. We still need to add things like license-checking and
showing/hiding of fields based on type.
* Hide Query Fields when ML is selected
These are still getting set on the form. We'll need to filter these
fields before we send off the data, and not show them on the readonly
display either.
ALso, edit is majorly broken.
* Add input field for anomaly threshold
* Display numberic values in the readonly view of a step
TIL that isEmpty returns false for numbers and other non-iterable
values. I don't think it's exactly what we want here, but until I figure
out the intention this gets our anomalyThreshold showing up without a
separate logic branch here. Removes the unnecessary branch that was
redundant with the 'else' clause.
* Add field for selecting an ML job
This is not the same as the mockups and lacks some functionality, but
it'll allow us to select a job for now.
* Format our new ML Fields when sending them to the server
So that we don't get rejected due to snake case vs camelcase.
* Put back code that respects a rule's schedule
It was previously hardcoded to a time period I knew had anomalies.
* ML fields are optional in our creation step
In that we don't initialize them like we do the query (default) fields.
* Only send along type-specific Rule fields from form
This makes any query- or ML-specific fields optional on a Rule, and
performs some logic on the frontend to group and include these fieldsets
conditionally based on the user's selection. The one place we don't
handle this well is on the readonly view of a completed step in the
rules creation, but we'll address that.
* Rename anomalies query
It's no longer tabular data. If we need that, we can use the ML client.
* Remove spike page with simple form
* Remove unneeded ES option
This response isn't going to HTTP, which is where this option would
matter.
* Fix bulk create logic
I made a happy accident and flipped the logic here, which meant we
weren't capping the signals we created.
* Rename argument
Value is a little more ambiguous than data, here: this is our step data.
* Create Rule form stores all values, but filters by type for use
When sending off to the backend, or displaying on the readonly view, we
inspect which rule type we've currently selected, and filter our form
values appropriately.
* Fix editing of ML fields on Rule Create
We need to inherit the field value from our form on initial render, and
everything works as expected.
* Clear form errors when switching between rule types
Validation errors prevent us from moving to the next step, so it was
previously possible to get an error for Query fields, switch to an ML
rule, and be unable to continue because the form had Query errors.
This also adds a helper for checking whether a ruleType is ML, to
prevent having to change all these references if the type string
changes.
* Validate the selection of an ML Job
* Fix type errors on frontend
According to the types, this is essentially the opposite of formatRule,
so we need to reinflate all potential form values from the rule.
* Don't set defaults for query-specific rules
For ML rules these types should not be included.
* Return ML Fields in Rule responses
This adds these fields to our rule serialization, and then adds
conditional validation around those fields if the rule type is ML.
Conversely, we moved the 'language' and 'query' fields to be
conditionally validated if the rule is a query/saved_query rule.
* Fix editing of ML rules by changing who controls the field values
The source of truth for their state is the parent form object; these
inputs should not have local state.
* Fix type errors related to new ML fields
In adding the new ML fields, some other fields (e.g. `query` and
`index`) that were previously required but implicitly part of Query
Rules are now marked as optional.
Consequently, any downstream code that actually required these fields
started to complain. In general, the fix was to verify that those fields
exist, and throw an error otherwise as to appease the linter.
Runtime-wise, the new ML rules/signals follow a separate code path and
both branches should be unaffected by these changes; the issue is simply
that our conditional types don't work well with Typescript.
* Fix failing route tests
Error message changed.
* Fix integration tests
We were not sending required properties when creating a rule(index and
language).
* Fix non-ML Rule creation
I was accidentally dropping this parameter for our POST payload. Whoops.
* More informative logging during ML signal generation
The messaging diverged from the normal path here because we don't have
index patterns to display. However, we have the rest of the rule
context, and should report it appropriately.
* Prefer keyof for string union types
* Tidy up our new form components
* Type them as React.FCs
* Remove unnecessary use of styled-components
* Prefer destructuring to lodash's omit
* Fix mock params for helper functions
These were updated to take simpler parameters.
* Remove any type
This could have been a boolean all along, whoops
* Fix mock types
* Update outdated tests
These were added on master, but behavior has been changed on my branch.
* Add some tests around our helper function
I need to refactor it, so this is as good a time as any to pin down the
behavior.
* Remove uses of any in favor of actual types
Mainly leverages ML typings instead of our placeholder types. This
required handling a null case in our formatting of anomalies.
* Annotate our anomalies with @timestamp field
We were notably lacking this ECS field in our post-conversion anomalies,
and typescript was rightly complaining about it.
* ml_job_id -> machine_learning_job_id
* PR Feedback
* Stricter threshold type
* More robust date parsing
* More informative log/error messages
* Remove redundant runtime checks
* Cleaning up our new ML types
* Fix types on our Rest types
* Use less ambiguous machineLearningJobId over mlJobId
* Declare our ML params as required keys, and ensure we pass them around
everywhere we might need them (creating, importing, updating rules).
* Use implicit type to avoid the need for a ts-ignore
FormSchema has a very generic index signature such that our
filterRuleFieldsForType helper cannot infer that it has our necessary
rule fields (when in fact it does). By removing the FormSchema hint we
get the actual keys of our schema, and things work as expected.
All other uses of schema continue to work because they're expecting
FormSchema, which is effectively { [key: string]: any }.
* New ML params are not nullable
Rather than setting a null and then never using it, let's just make it
truly optional in terms of default values.
* Query and language are conditional based on rule type
For ML Rules, we don't use them.
* Remove defaulted parameter in API test
We don't need to specify this, and we should continue not to for
backwards compatibility.
* Use explicit types over implicit ones
The concern is that not typing our schemae as FormSchema could break our
form if there are upstream changes. For now, we simply use the
intersection of FormSchema and our generic parameter to satisfy our use
within the function.
* Add integration test for creation of ML Rule
* Add ML fields to route schemae
* threshold and job id are conditional on type
* makes query and language mutually exclusive with above
* Fix router test for creating an ML rule
We were sending invalid parameters.
* Remove null check against index for query rules
We support not having an index here, as getInputIndex will return the
current UI setting if none is specified.
* Add regression test for API compatibility
We were previously able to create a rule without an input index; we
should continue to support that, as verified by this test!
* Respect the index pattern determined at runtime when performing search_after
If a rule does not specify an input index pattern on creation, we use
the current UI default when the rule is evaluated. This ensures that any
subsequent searches use that same index.
We're not currently persisting that runtime index to the generated
signal, but we should.
* Fix type errors in our bulk create tests
We added a new argument, but didn't update the tests.
2020-03-19 01:26:42 +01:00
type : 'machine_learning' ,
} ) ;
2020-02-15 04:14:59 +01:00
export const getSignalStatus = ( ) = > ( {
aggs : { statuses : { terms : { field : 'signal.status' , size : 10 } } } ,
} ) ;
2020-06-16 16:41:59 +02:00
export const getQueryAllSignals = ( ) = > ( {
query : { match_all : { } } ,
} ) ;
export const getQuerySignalIds = ( signalIds : SignalIds ) = > ( {
query : {
terms : {
_id : signalIds ,
} ,
} ,
} ) ;
[Security Solutions][Detection Engine] Fixes critical clashing with source indexes that already contain a "signal" field (#82191)
## Summary
Fixes: https://github.com/elastic/kibana/issues/82148
We have errors and do not generate a signal when a source index already has utilized and reserved the "signal" field for their own data purposes. This fix is a bit tricky and has one medium sized risk which is we also support "signals generated on top of existing signals". Therefore we have to be careful and do a small runtime detection of the "data shape" of the signal's data type. If it looks like the user is using the "signal" field within their mapping instead of us, we move the customer's signal into "original_signal" inside our "signal" structure we create when we copy their data set when creating a signal.
To help mitigate the risks associated with this critical bug with regards to breaking signals on top of signals I have:
* This adds unit tests
* This adds end to end tests for testing generating signals including signals on signals to help mitigate risk
The key test for this shape in the PR are in the file:
```
detection_engine/signals/build_event_type_signal.ts
```
like so:
```ts
export const isEventTypeSignal = (doc: BaseSignalHit): boolean => {
return doc._source.signal?.rule?.id != null && typeof doc._source.signal?.rule?.id === 'string';
};
```
Example of what happens when it does a "move" of an existing numeric signal keyword type:
```ts
# This causes a clash with us using the name signal as a numeric.
PUT clashing-index/_doc/1
{
"@timestamp": "2020-10-28T05:08:53.000Z",
"signal": 1
}
```
Before, this was an error. With this PR it now will restructure this data like so when creating a signal along with additional signal ancestor information, meta data. I omitted some of the data from the output signal for this example.
```ts
{
... Other data copied ...
"signal":
{
"original_signal": 1 <--- We "move it" here now
"parents":
[
{
"id": "BhbXBmkBR346wHgn4PeZ",
"type": "event",
"index": "your-index-name",
"depth": 0
},
],
"ancestors":
[
{
"id": "BhbXBmkBR346wHgn4PeZ",
"type": "event",
"index": "your-index-name",
"depth": 0
},
],
"status": "open",
"depth": 1,
"parent":
{
"id": "BhbXBmkBR346wHgn4PeZ",
type: "event",
"index": "your-index-name",
"depth": 0
},
"original_time": "2019-02-19T17:40:03.790Z",
"original_event":
{
"action": "socket_closed",
"dataset": "socket",
"kind": "event",
"module": "system"
},
}
```
### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2020-11-06 15:47:57 +01:00
/ * *
* Given an array of ruleIds for a test this will get the signals
* created from that rule_id .
* @param ruleIds The rule_id to search for signals
* /
export const getQuerySignalsRuleId = ( ruleIds : string [ ] ) = > ( {
query : {
terms : {
'signal.rule.rule_id' : ruleIds ,
} ,
} ,
} ) ;
[Security Solutions][Detection Engine] Adds e2e FTR runtime support and 213 tests for exception lists (#83764)
## Summary
Adds support to the end to end (e2e) functional test runner (FTR) support for rule runtime tests as well as 213 tests for the exception lists which include value based lists. Previously we had limited runtime support, but as I scaled up runtime tests from 5 to 200+ I noticed in a lot of areas we had to use improved techniques for determinism.
The runtime support being added is our next step of tests. Up to now most of our e2e FTR tests have been structural testing of REST and API integration tests. Basically up to now 95% tests are API structural as:
* Call REST input related to a rule such as GET/PUT/POST/PATCH/DELETE.
* Check REST output of the rule, did it match expected output body and status code?
* In some rare cases we check if the the rule can be executed and we get a status of 'succeeded'
With only a small part of our tests ~5%, `generating_signals.ts` was checking the signals being produced. However, we cannot have confidence in runtime based tests until the structural tests have been built up and run through the weeks against PR's to ensure that those are stable and deterministic.
Now that we have confidence and 90%+ coverage of the structural REST based tests, we are building up newer sets of tests which allow us to do runtime based validation tests to increase confidence that:
* Detection engine produces signals as expected
* Structure of the signals are as expected, including signal on signals
* Exceptions to signals are working as expected
* Most runtime bugs can be TDD'ed with e2e FTR's and regressions
* Whack-a-mole will not happen
* Consistency and predictability of signals is validated
* Refactoring can occur with stronger confidence
* Runtime tests are reference points for answering questions about existing bugs or adding new ones to test if users are experiencing unexpected behaviors
* Scaling tests can happen without failures
* Velocity for creating tests increases as the utilities and examples increase
Lastly, this puts us within striking distance of creating FTR's for different common class of runtime situations such as:
* Creating tests that exercise each rule against a set of data criteria and get signal hits
* Creating tests that validate the rule overrides operate as expected against data sets
* Creating tests that validate malfunctions, corner cases, or misuse cases such as data sets that are _all_ arrays or data sets that put numbers as strings or throws in an expected `null` instead of a value.
These tests follow the pattern of:
* Add the smallest data set to a folder in data.json (not gzip format)
* Add the smallest mapping to that folder (mapping.json)
* Call REST input related to exception lists, value lists, adding prepackaged rules, etc...
* Call REST input related endpoint with utilities to create and activate the rule
* Wait for the rule to go into the `succeeded` phase
* Wait for the N exact signals specific to that rule to be available
* Check against the set of signals to ensure that the matches are exactly as expected
Example of one runtime test:
A keyword data set is added to a folder called "keyword" but you can add one anywhere you want under `es_archives`, I just grouped mine depending on the situation of the runtime. Small non-gzipped tests `data.json` and `mappings.json` are the best approach for small focused tests. For _larger_ tests and cases I would and sometimes do use things such as auditbeat but try to avoid using larger data sets in favor of smaller focused test cases to validate the runtime is operating as expected.
```ts
{
"type": "doc",
"value": {
"id": "1",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:00:53.000Z",
"long": 1
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "2",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:01:53.000Z",
"long": 2
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "3",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:02:53.000Z",
"long": 3
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "4",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:03:53.000Z",
"long": 4
},
"type": "_doc"
}
}
```
Mapping is added. Note that this is "ECS tolerant" but not necessarily all ECS meaning I can and will try to keep things simple where I can, but I have ensured that `"@timestamp"` is at least there.
```ts
{
"type": "index",
"value": {
"index": "long",
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"long": { "type": "long" }
}
},
"settings": {
"index": {
"number_of_replicas": "1",
"number_of_shards": "1"
}
}
}
}
```
Test is written with test utilities where the `beforeEach` and `afterEach` try and clean up the indexes and load/unload the archives to keep one test from effecting another. Note this is never going to be 100% possible so see below on how we add more determinism in case something escapes the sandbox.
```ts
beforeEach(async () => {
await createSignalsIndex(supertest);
await createListsIndex(supertest);
await esArchiver.load('rule_exceptions/keyword');
});
afterEach(async () => {
await deleteSignalsIndex(supertest);
await deleteAllAlerts(supertest);
await deleteAllExceptions(es);
await deleteListsIndex(supertest);
await esArchiver.unload('rule_exceptions/keyword');
});
describe('"is" operator', () => {
it('should filter 1 single keyword if it is set as an exception', async () => {
const rule = getRuleForSignalTesting(['keyword']);
const { id } = await createRuleWithExceptionEntries(supertest, rule, [
[
{
field: 'keyword',
operator: 'included',
type: 'match',
value: 'word one',
},
],
]);
await waitForRuleSuccess(supertest, id);
await waitForSignalsToBePresent(supertest, 3, [id]);
const signalsOpen = await getSignalsById(supertest, id);
const hits = signalsOpen.hits.hits.map((hit) => hit._source.keyword).sort();
expect(hits).to.eql(['word four', 'word three', 'word two']);
});
});
```
### Changes for better determinism
To support more determinism there are changes and utilities added which can be tuned during any sporadic failures we might encounter as well as better support unexpected changes to other Elastic Stack pieces such as alerting, task manager, etc...
Get simple rule and others are now defaulting to false, meaning that the structural tests will no longer activate a rule and run it on task manger. This should cut down on error outputs as well as reduce stress and potentials for left over rules interfering with the runtime rules.
```ts
export const getSimpleRule = (ruleId = 'rule-1', enabled = false): QueryCreateSchema => ({
```
Not mandatory to use, but for most tests that should be runtime based tests, I use this function below which will enable it by default and run it using settings such as `type: 'query'`, `query: '*:*',` `from: '1900-01-01T00:00:00.000Z'`, to cut down on boiler plate noise. However, people can use whatever they want out of the grab bag or if their test is more readable to hand craft a REST request to create signals, or if they just want to call this and override where they want to, then 👍 .
```ts
export const getRuleForSignalTesting = (index: string[], ruleId = 'rule-1', enabled = true)
```
This waits for a rule to succeed before continuing
```ts
await waitForRuleSuccess(supertest, id);
```
I added a required array of id that _waits_ only for that particular id here. This is useful in case another test did not cleanup and you are getting signals being produced or left behind but need to wait specifically for yours.
```ts
await waitForSignalsToBePresent(supertest, 4, [id]);
```
I only get the signals for a particular rule id using either the auto-generated id or the rule_id. It's safer to use the ones from the auto-generated id but either of these are fine if you're careful enough.
```ts
const signalsOpen = await getSignalsById(supertest, id);
const signalsOpen = await getSignalsByIds(supertest, [createdId]);
const signalsOpen = await getSignalsByRuleIds(supertest, ['signal-on-signal']);
```
I delete all alerts now through a series of steps where it properly removes all rules using the rules bulk_delete and does it in such a way that all the API keys and alerting will be the best it can destroyed as well as double check that the alerts are showing up as being cleaned up before continuing.
```ts
deleteAllAlerts()
```
When not explicitly testing something structural, prefer to use the utilities which can and will do retries in case there are over the wire failures or es failures. Examples are:
```ts
installPrePackagedRules()
waitForRuleSuccess()
importFile() // This does a _lot_ of checks to ensure that the file is fully imported before continuing
```
Some of these utilities might still do a `expect(200);` but as we are and should use regular structural tests to cover those problems, these will probably be more and more removed when/if we hit test failures in favor of doing retries, waitFor, and countDowns.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2020-11-20 20:09:38 +01:00
/ * *
* Given an array of ids for a test this will get the signals
* created from that rule ' s regular id .
* @param ruleIds The rule_id to search for signals
* /
2021-02-23 23:16:48 +01:00
export const getQuerySignalsId = ( ids : string [ ] , size = 10 ) = > ( {
size ,
[Security Solutions][Detection Engine] Adds e2e FTR runtime support and 213 tests for exception lists (#83764)
## Summary
Adds support to the end to end (e2e) functional test runner (FTR) support for rule runtime tests as well as 213 tests for the exception lists which include value based lists. Previously we had limited runtime support, but as I scaled up runtime tests from 5 to 200+ I noticed in a lot of areas we had to use improved techniques for determinism.
The runtime support being added is our next step of tests. Up to now most of our e2e FTR tests have been structural testing of REST and API integration tests. Basically up to now 95% tests are API structural as:
* Call REST input related to a rule such as GET/PUT/POST/PATCH/DELETE.
* Check REST output of the rule, did it match expected output body and status code?
* In some rare cases we check if the the rule can be executed and we get a status of 'succeeded'
With only a small part of our tests ~5%, `generating_signals.ts` was checking the signals being produced. However, we cannot have confidence in runtime based tests until the structural tests have been built up and run through the weeks against PR's to ensure that those are stable and deterministic.
Now that we have confidence and 90%+ coverage of the structural REST based tests, we are building up newer sets of tests which allow us to do runtime based validation tests to increase confidence that:
* Detection engine produces signals as expected
* Structure of the signals are as expected, including signal on signals
* Exceptions to signals are working as expected
* Most runtime bugs can be TDD'ed with e2e FTR's and regressions
* Whack-a-mole will not happen
* Consistency and predictability of signals is validated
* Refactoring can occur with stronger confidence
* Runtime tests are reference points for answering questions about existing bugs or adding new ones to test if users are experiencing unexpected behaviors
* Scaling tests can happen without failures
* Velocity for creating tests increases as the utilities and examples increase
Lastly, this puts us within striking distance of creating FTR's for different common class of runtime situations such as:
* Creating tests that exercise each rule against a set of data criteria and get signal hits
* Creating tests that validate the rule overrides operate as expected against data sets
* Creating tests that validate malfunctions, corner cases, or misuse cases such as data sets that are _all_ arrays or data sets that put numbers as strings or throws in an expected `null` instead of a value.
These tests follow the pattern of:
* Add the smallest data set to a folder in data.json (not gzip format)
* Add the smallest mapping to that folder (mapping.json)
* Call REST input related to exception lists, value lists, adding prepackaged rules, etc...
* Call REST input related endpoint with utilities to create and activate the rule
* Wait for the rule to go into the `succeeded` phase
* Wait for the N exact signals specific to that rule to be available
* Check against the set of signals to ensure that the matches are exactly as expected
Example of one runtime test:
A keyword data set is added to a folder called "keyword" but you can add one anywhere you want under `es_archives`, I just grouped mine depending on the situation of the runtime. Small non-gzipped tests `data.json` and `mappings.json` are the best approach for small focused tests. For _larger_ tests and cases I would and sometimes do use things such as auditbeat but try to avoid using larger data sets in favor of smaller focused test cases to validate the runtime is operating as expected.
```ts
{
"type": "doc",
"value": {
"id": "1",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:00:53.000Z",
"long": 1
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "2",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:01:53.000Z",
"long": 2
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "3",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:02:53.000Z",
"long": 3
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "4",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:03:53.000Z",
"long": 4
},
"type": "_doc"
}
}
```
Mapping is added. Note that this is "ECS tolerant" but not necessarily all ECS meaning I can and will try to keep things simple where I can, but I have ensured that `"@timestamp"` is at least there.
```ts
{
"type": "index",
"value": {
"index": "long",
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"long": { "type": "long" }
}
},
"settings": {
"index": {
"number_of_replicas": "1",
"number_of_shards": "1"
}
}
}
}
```
Test is written with test utilities where the `beforeEach` and `afterEach` try and clean up the indexes and load/unload the archives to keep one test from effecting another. Note this is never going to be 100% possible so see below on how we add more determinism in case something escapes the sandbox.
```ts
beforeEach(async () => {
await createSignalsIndex(supertest);
await createListsIndex(supertest);
await esArchiver.load('rule_exceptions/keyword');
});
afterEach(async () => {
await deleteSignalsIndex(supertest);
await deleteAllAlerts(supertest);
await deleteAllExceptions(es);
await deleteListsIndex(supertest);
await esArchiver.unload('rule_exceptions/keyword');
});
describe('"is" operator', () => {
it('should filter 1 single keyword if it is set as an exception', async () => {
const rule = getRuleForSignalTesting(['keyword']);
const { id } = await createRuleWithExceptionEntries(supertest, rule, [
[
{
field: 'keyword',
operator: 'included',
type: 'match',
value: 'word one',
},
],
]);
await waitForRuleSuccess(supertest, id);
await waitForSignalsToBePresent(supertest, 3, [id]);
const signalsOpen = await getSignalsById(supertest, id);
const hits = signalsOpen.hits.hits.map((hit) => hit._source.keyword).sort();
expect(hits).to.eql(['word four', 'word three', 'word two']);
});
});
```
### Changes for better determinism
To support more determinism there are changes and utilities added which can be tuned during any sporadic failures we might encounter as well as better support unexpected changes to other Elastic Stack pieces such as alerting, task manager, etc...
Get simple rule and others are now defaulting to false, meaning that the structural tests will no longer activate a rule and run it on task manger. This should cut down on error outputs as well as reduce stress and potentials for left over rules interfering with the runtime rules.
```ts
export const getSimpleRule = (ruleId = 'rule-1', enabled = false): QueryCreateSchema => ({
```
Not mandatory to use, but for most tests that should be runtime based tests, I use this function below which will enable it by default and run it using settings such as `type: 'query'`, `query: '*:*',` `from: '1900-01-01T00:00:00.000Z'`, to cut down on boiler plate noise. However, people can use whatever they want out of the grab bag or if their test is more readable to hand craft a REST request to create signals, or if they just want to call this and override where they want to, then 👍 .
```ts
export const getRuleForSignalTesting = (index: string[], ruleId = 'rule-1', enabled = true)
```
This waits for a rule to succeed before continuing
```ts
await waitForRuleSuccess(supertest, id);
```
I added a required array of id that _waits_ only for that particular id here. This is useful in case another test did not cleanup and you are getting signals being produced or left behind but need to wait specifically for yours.
```ts
await waitForSignalsToBePresent(supertest, 4, [id]);
```
I only get the signals for a particular rule id using either the auto-generated id or the rule_id. It's safer to use the ones from the auto-generated id but either of these are fine if you're careful enough.
```ts
const signalsOpen = await getSignalsById(supertest, id);
const signalsOpen = await getSignalsByIds(supertest, [createdId]);
const signalsOpen = await getSignalsByRuleIds(supertest, ['signal-on-signal']);
```
I delete all alerts now through a series of steps where it properly removes all rules using the rules bulk_delete and does it in such a way that all the API keys and alerting will be the best it can destroyed as well as double check that the alerts are showing up as being cleaned up before continuing.
```ts
deleteAllAlerts()
```
When not explicitly testing something structural, prefer to use the utilities which can and will do retries in case there are over the wire failures or es failures. Examples are:
```ts
installPrePackagedRules()
waitForRuleSuccess()
importFile() // This does a _lot_ of checks to ensure that the file is fully imported before continuing
```
Some of these utilities might still do a `expect(200);` but as we are and should use regular structural tests to cover those problems, these will probably be more and more removed when/if we hit test failures in favor of doing retries, waitFor, and countDowns.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2020-11-20 20:09:38 +01:00
query : {
terms : {
'signal.rule.id' : ids ,
} ,
} ,
} ) ;
2020-02-15 04:14:59 +01:00
export const setSignalStatus = ( {
signalIds ,
status ,
} : {
2020-06-16 16:41:59 +02:00
signalIds : SignalIds ;
status : Status ;
2020-02-15 04:14:59 +01:00
} ) = > ( {
signal_ids : signalIds ,
status ,
} ) ;
export const getSignalStatusEmptyResponse = ( ) = > ( {
timed_out : false ,
total : 0 ,
updated : 0 ,
deleted : 0 ,
batches : 0 ,
version_conflicts : 0 ,
noops : 0 ,
retries : { bulk : 0 , search : 0 } ,
throttled_millis : 0 ,
requests_per_second : - 1 ,
throttled_until_millis : 0 ,
failures : [ ] ,
} ) ;
2020-02-11 23:24:30 +01:00
/ * *
* This is a typical simple rule for testing that is easy for most basic testing
* /
2020-06-10 16:54:27 +02:00
export const getSimpleRuleWithoutRuleId = ( ) : CreateRulesSchema = > {
2020-02-11 23:24:30 +01:00
const simpleRule = getSimpleRule ( ) ;
2020-08-05 17:32:19 +02:00
// eslint-disable-next-line @typescript-eslint/naming-convention
2020-02-11 23:24:30 +01:00
const { rule_id , . . . ruleWithoutId } = simpleRule ;
return ruleWithoutId ;
} ;
/ * *
* Useful for export_api testing to convert from a multi - part binary back to a string
* @param res Response
* @param callback Callback
* /
export const binaryToString = ( res : any , callback : any ) : void = > {
res . setEncoding ( 'binary' ) ;
res . data = '' ;
res . on ( 'data' , ( chunk : any ) = > {
res . data += chunk ;
} ) ;
res . on ( 'end' , ( ) = > {
callback ( null , Buffer . from ( res . data ) ) ;
} ) ;
} ;
/ * *
2020-06-10 16:54:27 +02:00
* This is the typical output of a simple rule that Kibana will output with all the defaults
* except for the server generated properties . Useful for testing end to end tests .
2020-02-11 23:24:30 +01:00
* /
[Security Solutions][Detection Engine] Adds e2e FTR runtime support and 213 tests for exception lists (#83764)
## Summary
Adds support to the end to end (e2e) functional test runner (FTR) support for rule runtime tests as well as 213 tests for the exception lists which include value based lists. Previously we had limited runtime support, but as I scaled up runtime tests from 5 to 200+ I noticed in a lot of areas we had to use improved techniques for determinism.
The runtime support being added is our next step of tests. Up to now most of our e2e FTR tests have been structural testing of REST and API integration tests. Basically up to now 95% tests are API structural as:
* Call REST input related to a rule such as GET/PUT/POST/PATCH/DELETE.
* Check REST output of the rule, did it match expected output body and status code?
* In some rare cases we check if the the rule can be executed and we get a status of 'succeeded'
With only a small part of our tests ~5%, `generating_signals.ts` was checking the signals being produced. However, we cannot have confidence in runtime based tests until the structural tests have been built up and run through the weeks against PR's to ensure that those are stable and deterministic.
Now that we have confidence and 90%+ coverage of the structural REST based tests, we are building up newer sets of tests which allow us to do runtime based validation tests to increase confidence that:
* Detection engine produces signals as expected
* Structure of the signals are as expected, including signal on signals
* Exceptions to signals are working as expected
* Most runtime bugs can be TDD'ed with e2e FTR's and regressions
* Whack-a-mole will not happen
* Consistency and predictability of signals is validated
* Refactoring can occur with stronger confidence
* Runtime tests are reference points for answering questions about existing bugs or adding new ones to test if users are experiencing unexpected behaviors
* Scaling tests can happen without failures
* Velocity for creating tests increases as the utilities and examples increase
Lastly, this puts us within striking distance of creating FTR's for different common class of runtime situations such as:
* Creating tests that exercise each rule against a set of data criteria and get signal hits
* Creating tests that validate the rule overrides operate as expected against data sets
* Creating tests that validate malfunctions, corner cases, or misuse cases such as data sets that are _all_ arrays or data sets that put numbers as strings or throws in an expected `null` instead of a value.
These tests follow the pattern of:
* Add the smallest data set to a folder in data.json (not gzip format)
* Add the smallest mapping to that folder (mapping.json)
* Call REST input related to exception lists, value lists, adding prepackaged rules, etc...
* Call REST input related endpoint with utilities to create and activate the rule
* Wait for the rule to go into the `succeeded` phase
* Wait for the N exact signals specific to that rule to be available
* Check against the set of signals to ensure that the matches are exactly as expected
Example of one runtime test:
A keyword data set is added to a folder called "keyword" but you can add one anywhere you want under `es_archives`, I just grouped mine depending on the situation of the runtime. Small non-gzipped tests `data.json` and `mappings.json` are the best approach for small focused tests. For _larger_ tests and cases I would and sometimes do use things such as auditbeat but try to avoid using larger data sets in favor of smaller focused test cases to validate the runtime is operating as expected.
```ts
{
"type": "doc",
"value": {
"id": "1",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:00:53.000Z",
"long": 1
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "2",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:01:53.000Z",
"long": 2
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "3",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:02:53.000Z",
"long": 3
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "4",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:03:53.000Z",
"long": 4
},
"type": "_doc"
}
}
```
Mapping is added. Note that this is "ECS tolerant" but not necessarily all ECS meaning I can and will try to keep things simple where I can, but I have ensured that `"@timestamp"` is at least there.
```ts
{
"type": "index",
"value": {
"index": "long",
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"long": { "type": "long" }
}
},
"settings": {
"index": {
"number_of_replicas": "1",
"number_of_shards": "1"
}
}
}
}
```
Test is written with test utilities where the `beforeEach` and `afterEach` try and clean up the indexes and load/unload the archives to keep one test from effecting another. Note this is never going to be 100% possible so see below on how we add more determinism in case something escapes the sandbox.
```ts
beforeEach(async () => {
await createSignalsIndex(supertest);
await createListsIndex(supertest);
await esArchiver.load('rule_exceptions/keyword');
});
afterEach(async () => {
await deleteSignalsIndex(supertest);
await deleteAllAlerts(supertest);
await deleteAllExceptions(es);
await deleteListsIndex(supertest);
await esArchiver.unload('rule_exceptions/keyword');
});
describe('"is" operator', () => {
it('should filter 1 single keyword if it is set as an exception', async () => {
const rule = getRuleForSignalTesting(['keyword']);
const { id } = await createRuleWithExceptionEntries(supertest, rule, [
[
{
field: 'keyword',
operator: 'included',
type: 'match',
value: 'word one',
},
],
]);
await waitForRuleSuccess(supertest, id);
await waitForSignalsToBePresent(supertest, 3, [id]);
const signalsOpen = await getSignalsById(supertest, id);
const hits = signalsOpen.hits.hits.map((hit) => hit._source.keyword).sort();
expect(hits).to.eql(['word four', 'word three', 'word two']);
});
});
```
### Changes for better determinism
To support more determinism there are changes and utilities added which can be tuned during any sporadic failures we might encounter as well as better support unexpected changes to other Elastic Stack pieces such as alerting, task manager, etc...
Get simple rule and others are now defaulting to false, meaning that the structural tests will no longer activate a rule and run it on task manger. This should cut down on error outputs as well as reduce stress and potentials for left over rules interfering with the runtime rules.
```ts
export const getSimpleRule = (ruleId = 'rule-1', enabled = false): QueryCreateSchema => ({
```
Not mandatory to use, but for most tests that should be runtime based tests, I use this function below which will enable it by default and run it using settings such as `type: 'query'`, `query: '*:*',` `from: '1900-01-01T00:00:00.000Z'`, to cut down on boiler plate noise. However, people can use whatever they want out of the grab bag or if their test is more readable to hand craft a REST request to create signals, or if they just want to call this and override where they want to, then 👍 .
```ts
export const getRuleForSignalTesting = (index: string[], ruleId = 'rule-1', enabled = true)
```
This waits for a rule to succeed before continuing
```ts
await waitForRuleSuccess(supertest, id);
```
I added a required array of id that _waits_ only for that particular id here. This is useful in case another test did not cleanup and you are getting signals being produced or left behind but need to wait specifically for yours.
```ts
await waitForSignalsToBePresent(supertest, 4, [id]);
```
I only get the signals for a particular rule id using either the auto-generated id or the rule_id. It's safer to use the ones from the auto-generated id but either of these are fine if you're careful enough.
```ts
const signalsOpen = await getSignalsById(supertest, id);
const signalsOpen = await getSignalsByIds(supertest, [createdId]);
const signalsOpen = await getSignalsByRuleIds(supertest, ['signal-on-signal']);
```
I delete all alerts now through a series of steps where it properly removes all rules using the rules bulk_delete and does it in such a way that all the API keys and alerting will be the best it can destroyed as well as double check that the alerts are showing up as being cleaned up before continuing.
```ts
deleteAllAlerts()
```
When not explicitly testing something structural, prefer to use the utilities which can and will do retries in case there are over the wire failures or es failures. Examples are:
```ts
installPrePackagedRules()
waitForRuleSuccess()
importFile() // This does a _lot_ of checks to ensure that the file is fully imported before continuing
```
Some of these utilities might still do a `expect(200);` but as we are and should use regular structural tests to cover those problems, these will probably be more and more removed when/if we hit test failures in favor of doing retries, waitFor, and countDowns.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2020-11-20 20:09:38 +01:00
export const getSimpleRuleOutput = ( ruleId = 'rule-1' , enabled = false ) : Partial < RulesSchema > = > ( {
2020-03-20 10:54:51 +01:00
actions : [ ] ,
2020-07-02 06:49:30 +02:00
author : [ ] ,
2020-02-11 23:24:30 +01:00
created_by : 'elastic' ,
description : 'Simple Rule Query' ,
[Security Solutions][Detection Engine] Adds e2e FTR runtime support and 213 tests for exception lists (#83764)
## Summary
Adds support to the end to end (e2e) functional test runner (FTR) support for rule runtime tests as well as 213 tests for the exception lists which include value based lists. Previously we had limited runtime support, but as I scaled up runtime tests from 5 to 200+ I noticed in a lot of areas we had to use improved techniques for determinism.
The runtime support being added is our next step of tests. Up to now most of our e2e FTR tests have been structural testing of REST and API integration tests. Basically up to now 95% tests are API structural as:
* Call REST input related to a rule such as GET/PUT/POST/PATCH/DELETE.
* Check REST output of the rule, did it match expected output body and status code?
* In some rare cases we check if the the rule can be executed and we get a status of 'succeeded'
With only a small part of our tests ~5%, `generating_signals.ts` was checking the signals being produced. However, we cannot have confidence in runtime based tests until the structural tests have been built up and run through the weeks against PR's to ensure that those are stable and deterministic.
Now that we have confidence and 90%+ coverage of the structural REST based tests, we are building up newer sets of tests which allow us to do runtime based validation tests to increase confidence that:
* Detection engine produces signals as expected
* Structure of the signals are as expected, including signal on signals
* Exceptions to signals are working as expected
* Most runtime bugs can be TDD'ed with e2e FTR's and regressions
* Whack-a-mole will not happen
* Consistency and predictability of signals is validated
* Refactoring can occur with stronger confidence
* Runtime tests are reference points for answering questions about existing bugs or adding new ones to test if users are experiencing unexpected behaviors
* Scaling tests can happen without failures
* Velocity for creating tests increases as the utilities and examples increase
Lastly, this puts us within striking distance of creating FTR's for different common class of runtime situations such as:
* Creating tests that exercise each rule against a set of data criteria and get signal hits
* Creating tests that validate the rule overrides operate as expected against data sets
* Creating tests that validate malfunctions, corner cases, or misuse cases such as data sets that are _all_ arrays or data sets that put numbers as strings or throws in an expected `null` instead of a value.
These tests follow the pattern of:
* Add the smallest data set to a folder in data.json (not gzip format)
* Add the smallest mapping to that folder (mapping.json)
* Call REST input related to exception lists, value lists, adding prepackaged rules, etc...
* Call REST input related endpoint with utilities to create and activate the rule
* Wait for the rule to go into the `succeeded` phase
* Wait for the N exact signals specific to that rule to be available
* Check against the set of signals to ensure that the matches are exactly as expected
Example of one runtime test:
A keyword data set is added to a folder called "keyword" but you can add one anywhere you want under `es_archives`, I just grouped mine depending on the situation of the runtime. Small non-gzipped tests `data.json` and `mappings.json` are the best approach for small focused tests. For _larger_ tests and cases I would and sometimes do use things such as auditbeat but try to avoid using larger data sets in favor of smaller focused test cases to validate the runtime is operating as expected.
```ts
{
"type": "doc",
"value": {
"id": "1",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:00:53.000Z",
"long": 1
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "2",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:01:53.000Z",
"long": 2
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "3",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:02:53.000Z",
"long": 3
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "4",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:03:53.000Z",
"long": 4
},
"type": "_doc"
}
}
```
Mapping is added. Note that this is "ECS tolerant" but not necessarily all ECS meaning I can and will try to keep things simple where I can, but I have ensured that `"@timestamp"` is at least there.
```ts
{
"type": "index",
"value": {
"index": "long",
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"long": { "type": "long" }
}
},
"settings": {
"index": {
"number_of_replicas": "1",
"number_of_shards": "1"
}
}
}
}
```
Test is written with test utilities where the `beforeEach` and `afterEach` try and clean up the indexes and load/unload the archives to keep one test from effecting another. Note this is never going to be 100% possible so see below on how we add more determinism in case something escapes the sandbox.
```ts
beforeEach(async () => {
await createSignalsIndex(supertest);
await createListsIndex(supertest);
await esArchiver.load('rule_exceptions/keyword');
});
afterEach(async () => {
await deleteSignalsIndex(supertest);
await deleteAllAlerts(supertest);
await deleteAllExceptions(es);
await deleteListsIndex(supertest);
await esArchiver.unload('rule_exceptions/keyword');
});
describe('"is" operator', () => {
it('should filter 1 single keyword if it is set as an exception', async () => {
const rule = getRuleForSignalTesting(['keyword']);
const { id } = await createRuleWithExceptionEntries(supertest, rule, [
[
{
field: 'keyword',
operator: 'included',
type: 'match',
value: 'word one',
},
],
]);
await waitForRuleSuccess(supertest, id);
await waitForSignalsToBePresent(supertest, 3, [id]);
const signalsOpen = await getSignalsById(supertest, id);
const hits = signalsOpen.hits.hits.map((hit) => hit._source.keyword).sort();
expect(hits).to.eql(['word four', 'word three', 'word two']);
});
});
```
### Changes for better determinism
To support more determinism there are changes and utilities added which can be tuned during any sporadic failures we might encounter as well as better support unexpected changes to other Elastic Stack pieces such as alerting, task manager, etc...
Get simple rule and others are now defaulting to false, meaning that the structural tests will no longer activate a rule and run it on task manger. This should cut down on error outputs as well as reduce stress and potentials for left over rules interfering with the runtime rules.
```ts
export const getSimpleRule = (ruleId = 'rule-1', enabled = false): QueryCreateSchema => ({
```
Not mandatory to use, but for most tests that should be runtime based tests, I use this function below which will enable it by default and run it using settings such as `type: 'query'`, `query: '*:*',` `from: '1900-01-01T00:00:00.000Z'`, to cut down on boiler plate noise. However, people can use whatever they want out of the grab bag or if their test is more readable to hand craft a REST request to create signals, or if they just want to call this and override where they want to, then 👍 .
```ts
export const getRuleForSignalTesting = (index: string[], ruleId = 'rule-1', enabled = true)
```
This waits for a rule to succeed before continuing
```ts
await waitForRuleSuccess(supertest, id);
```
I added a required array of id that _waits_ only for that particular id here. This is useful in case another test did not cleanup and you are getting signals being produced or left behind but need to wait specifically for yours.
```ts
await waitForSignalsToBePresent(supertest, 4, [id]);
```
I only get the signals for a particular rule id using either the auto-generated id or the rule_id. It's safer to use the ones from the auto-generated id but either of these are fine if you're careful enough.
```ts
const signalsOpen = await getSignalsById(supertest, id);
const signalsOpen = await getSignalsByIds(supertest, [createdId]);
const signalsOpen = await getSignalsByRuleIds(supertest, ['signal-on-signal']);
```
I delete all alerts now through a series of steps where it properly removes all rules using the rules bulk_delete and does it in such a way that all the API keys and alerting will be the best it can destroyed as well as double check that the alerts are showing up as being cleaned up before continuing.
```ts
deleteAllAlerts()
```
When not explicitly testing something structural, prefer to use the utilities which can and will do retries in case there are over the wire failures or es failures. Examples are:
```ts
installPrePackagedRules()
waitForRuleSuccess()
importFile() // This does a _lot_ of checks to ensure that the file is fully imported before continuing
```
Some of these utilities might still do a `expect(200);` but as we are and should use regular structural tests to cover those problems, these will probably be more and more removed when/if we hit test failures in favor of doing retries, waitFor, and countDowns.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2020-11-20 20:09:38 +01:00
enabled ,
2020-02-11 23:24:30 +01:00
false_positives : [ ] ,
from : 'now-6m' ,
immutable : false ,
[SIEM] Create ML Rules (#58053)
* Remove unnecessary linter exceptions
Not sure what was causing issues here, but it's gone now.
* WIP: Simple form to test creation of ML rules
This will be integrated into the regular rule creation workflow, but for
now this simple form should allow us to exercise the full ML rule
workflow.
* WIP: Adds POST to backend, and type/payload changes necessary to make that work
* Simplify logic with Math.min
* WIP: Failed spike of making an http call
* WIP: Hacking together an ML client
The rest of this is going to be easier if I have actual data. For now
this is mostly copy/pasted and simplified ML code. I've hardcoded time
ranges to a period I know has data for a particular job.
* Threading through our new ML Rule params
It's a bummer that we normalize our rule alert params across all rule
types currently, but that's the deal.
* Retrieve our anomalies during rule execution
Next step: generate signals
* WIP: Generate ECS-compatible ML Signals
This uses as much of the existing signal-creation code as possible. I
skipped the search_after stuff for now because it would require us
recreating the anomalies query which we really shouldn't own. For now,
here's how it works:
* Adds a separate branch of the rule executor for machine_learning rules
* In that branch, we call our new bulkCreateMlSignal function
* This function first transforms the anomaly document into ECS fields
* We then pass the transformed documents to singleBulkCreate, which
does the rest
* After both branches, we update the rule's status appropriately.
We need to do some more work on the anomaly transformation, but this
works!
* Extract setting of rule failure to helper function
We were doing this identically in three places.
* Remove unused import
* Define a field for our Rule Type selection
This adds most of the markup and logic to allow an ML rule type to be
selected. We still need to add things like license-checking and
showing/hiding of fields based on type.
* Hide Query Fields when ML is selected
These are still getting set on the form. We'll need to filter these
fields before we send off the data, and not show them on the readonly
display either.
ALso, edit is majorly broken.
* Add input field for anomaly threshold
* Display numberic values in the readonly view of a step
TIL that isEmpty returns false for numbers and other non-iterable
values. I don't think it's exactly what we want here, but until I figure
out the intention this gets our anomalyThreshold showing up without a
separate logic branch here. Removes the unnecessary branch that was
redundant with the 'else' clause.
* Add field for selecting an ML job
This is not the same as the mockups and lacks some functionality, but
it'll allow us to select a job for now.
* Format our new ML Fields when sending them to the server
So that we don't get rejected due to snake case vs camelcase.
* Put back code that respects a rule's schedule
It was previously hardcoded to a time period I knew had anomalies.
* ML fields are optional in our creation step
In that we don't initialize them like we do the query (default) fields.
* Only send along type-specific Rule fields from form
This makes any query- or ML-specific fields optional on a Rule, and
performs some logic on the frontend to group and include these fieldsets
conditionally based on the user's selection. The one place we don't
handle this well is on the readonly view of a completed step in the
rules creation, but we'll address that.
* Rename anomalies query
It's no longer tabular data. If we need that, we can use the ML client.
* Remove spike page with simple form
* Remove unneeded ES option
This response isn't going to HTTP, which is where this option would
matter.
* Fix bulk create logic
I made a happy accident and flipped the logic here, which meant we
weren't capping the signals we created.
* Rename argument
Value is a little more ambiguous than data, here: this is our step data.
* Create Rule form stores all values, but filters by type for use
When sending off to the backend, or displaying on the readonly view, we
inspect which rule type we've currently selected, and filter our form
values appropriately.
* Fix editing of ML fields on Rule Create
We need to inherit the field value from our form on initial render, and
everything works as expected.
* Clear form errors when switching between rule types
Validation errors prevent us from moving to the next step, so it was
previously possible to get an error for Query fields, switch to an ML
rule, and be unable to continue because the form had Query errors.
This also adds a helper for checking whether a ruleType is ML, to
prevent having to change all these references if the type string
changes.
* Validate the selection of an ML Job
* Fix type errors on frontend
According to the types, this is essentially the opposite of formatRule,
so we need to reinflate all potential form values from the rule.
* Don't set defaults for query-specific rules
For ML rules these types should not be included.
* Return ML Fields in Rule responses
This adds these fields to our rule serialization, and then adds
conditional validation around those fields if the rule type is ML.
Conversely, we moved the 'language' and 'query' fields to be
conditionally validated if the rule is a query/saved_query rule.
* Fix editing of ML rules by changing who controls the field values
The source of truth for their state is the parent form object; these
inputs should not have local state.
* Fix type errors related to new ML fields
In adding the new ML fields, some other fields (e.g. `query` and
`index`) that were previously required but implicitly part of Query
Rules are now marked as optional.
Consequently, any downstream code that actually required these fields
started to complain. In general, the fix was to verify that those fields
exist, and throw an error otherwise as to appease the linter.
Runtime-wise, the new ML rules/signals follow a separate code path and
both branches should be unaffected by these changes; the issue is simply
that our conditional types don't work well with Typescript.
* Fix failing route tests
Error message changed.
* Fix integration tests
We were not sending required properties when creating a rule(index and
language).
* Fix non-ML Rule creation
I was accidentally dropping this parameter for our POST payload. Whoops.
* More informative logging during ML signal generation
The messaging diverged from the normal path here because we don't have
index patterns to display. However, we have the rest of the rule
context, and should report it appropriately.
* Prefer keyof for string union types
* Tidy up our new form components
* Type them as React.FCs
* Remove unnecessary use of styled-components
* Prefer destructuring to lodash's omit
* Fix mock params for helper functions
These were updated to take simpler parameters.
* Remove any type
This could have been a boolean all along, whoops
* Fix mock types
* Update outdated tests
These were added on master, but behavior has been changed on my branch.
* Add some tests around our helper function
I need to refactor it, so this is as good a time as any to pin down the
behavior.
* Remove uses of any in favor of actual types
Mainly leverages ML typings instead of our placeholder types. This
required handling a null case in our formatting of anomalies.
* Annotate our anomalies with @timestamp field
We were notably lacking this ECS field in our post-conversion anomalies,
and typescript was rightly complaining about it.
* ml_job_id -> machine_learning_job_id
* PR Feedback
* Stricter threshold type
* More robust date parsing
* More informative log/error messages
* Remove redundant runtime checks
* Cleaning up our new ML types
* Fix types on our Rest types
* Use less ambiguous machineLearningJobId over mlJobId
* Declare our ML params as required keys, and ensure we pass them around
everywhere we might need them (creating, importing, updating rules).
* Use implicit type to avoid the need for a ts-ignore
FormSchema has a very generic index signature such that our
filterRuleFieldsForType helper cannot infer that it has our necessary
rule fields (when in fact it does). By removing the FormSchema hint we
get the actual keys of our schema, and things work as expected.
All other uses of schema continue to work because they're expecting
FormSchema, which is effectively { [key: string]: any }.
* New ML params are not nullable
Rather than setting a null and then never using it, let's just make it
truly optional in terms of default values.
* Query and language are conditional based on rule type
For ML Rules, we don't use them.
* Remove defaulted parameter in API test
We don't need to specify this, and we should continue not to for
backwards compatibility.
* Use explicit types over implicit ones
The concern is that not typing our schemae as FormSchema could break our
form if there are upstream changes. For now, we simply use the
intersection of FormSchema and our generic parameter to satisfy our use
within the function.
* Add integration test for creation of ML Rule
* Add ML fields to route schemae
* threshold and job id are conditional on type
* makes query and language mutually exclusive with above
* Fix router test for creating an ML rule
We were sending invalid parameters.
* Remove null check against index for query rules
We support not having an index here, as getInputIndex will return the
current UI setting if none is specified.
* Add regression test for API compatibility
We were previously able to create a rule without an input index; we
should continue to support that, as verified by this test!
* Respect the index pattern determined at runtime when performing search_after
If a rule does not specify an input index pattern on creation, we use
the current UI default when the rule is evaluated. This ensures that any
subsequent searches use that same index.
We're not currently persisting that runtime index to the generated
signal, but we should.
* Fix type errors in our bulk create tests
We added a new argument, but didn't update the tests.
2020-03-19 01:26:42 +01:00
index : [ 'auditbeat-*' ] ,
2020-02-11 23:24:30 +01:00
interval : '5m' ,
rule_id : ruleId ,
language : 'kuery' ,
output_index : '.siem-signals-default' ,
max_signals : 100 ,
risk_score : 1 ,
2020-07-02 06:49:30 +02:00
risk_score_mapping : [ ] ,
2020-02-11 23:24:30 +01:00
name : 'Simple Rule Query' ,
query : 'user.name: root or user.name: admin' ,
references : [ ] ,
severity : 'high' ,
2020-07-02 06:49:30 +02:00
severity_mapping : [ ] ,
2020-02-11 23:24:30 +01:00
updated_by : 'elastic' ,
tags : [ ] ,
to : 'now' ,
type : 'query' ,
threat : [ ] ,
2020-03-31 12:54:58 +02:00
throttle : 'no_actions' ,
2020-04-18 00:58:35 +02:00
exceptions_list : [ ] ,
2020-02-11 23:24:30 +01:00
version : 1 ,
} ) ;
/ * *
2020-06-10 16:54:27 +02:00
* This is the typical output of a simple rule that Kibana will output with all the defaults except
* for all the server generated properties such as created_by . Useful for testing end to end tests .
2020-02-11 23:24:30 +01:00
* /
2020-06-10 16:54:27 +02:00
export const getSimpleRuleOutputWithoutRuleId = ( ruleId = 'rule-1' ) : Partial < RulesSchema > = > {
2020-02-11 23:24:30 +01:00
const rule = getSimpleRuleOutput ( ruleId ) ;
2020-08-05 17:32:19 +02:00
// eslint-disable-next-line @typescript-eslint/naming-convention
2020-02-11 23:24:30 +01:00
const { rule_id , . . . ruleWithoutRuleId } = rule ;
return ruleWithoutRuleId ;
} ;
2020-06-10 16:54:27 +02:00
export const getSimpleMlRuleOutput = ( ruleId = 'rule-1' ) : Partial < RulesSchema > = > {
[SIEM] Create ML Rules (#58053)
* Remove unnecessary linter exceptions
Not sure what was causing issues here, but it's gone now.
* WIP: Simple form to test creation of ML rules
This will be integrated into the regular rule creation workflow, but for
now this simple form should allow us to exercise the full ML rule
workflow.
* WIP: Adds POST to backend, and type/payload changes necessary to make that work
* Simplify logic with Math.min
* WIP: Failed spike of making an http call
* WIP: Hacking together an ML client
The rest of this is going to be easier if I have actual data. For now
this is mostly copy/pasted and simplified ML code. I've hardcoded time
ranges to a period I know has data for a particular job.
* Threading through our new ML Rule params
It's a bummer that we normalize our rule alert params across all rule
types currently, but that's the deal.
* Retrieve our anomalies during rule execution
Next step: generate signals
* WIP: Generate ECS-compatible ML Signals
This uses as much of the existing signal-creation code as possible. I
skipped the search_after stuff for now because it would require us
recreating the anomalies query which we really shouldn't own. For now,
here's how it works:
* Adds a separate branch of the rule executor for machine_learning rules
* In that branch, we call our new bulkCreateMlSignal function
* This function first transforms the anomaly document into ECS fields
* We then pass the transformed documents to singleBulkCreate, which
does the rest
* After both branches, we update the rule's status appropriately.
We need to do some more work on the anomaly transformation, but this
works!
* Extract setting of rule failure to helper function
We were doing this identically in three places.
* Remove unused import
* Define a field for our Rule Type selection
This adds most of the markup and logic to allow an ML rule type to be
selected. We still need to add things like license-checking and
showing/hiding of fields based on type.
* Hide Query Fields when ML is selected
These are still getting set on the form. We'll need to filter these
fields before we send off the data, and not show them on the readonly
display either.
ALso, edit is majorly broken.
* Add input field for anomaly threshold
* Display numberic values in the readonly view of a step
TIL that isEmpty returns false for numbers and other non-iterable
values. I don't think it's exactly what we want here, but until I figure
out the intention this gets our anomalyThreshold showing up without a
separate logic branch here. Removes the unnecessary branch that was
redundant with the 'else' clause.
* Add field for selecting an ML job
This is not the same as the mockups and lacks some functionality, but
it'll allow us to select a job for now.
* Format our new ML Fields when sending them to the server
So that we don't get rejected due to snake case vs camelcase.
* Put back code that respects a rule's schedule
It was previously hardcoded to a time period I knew had anomalies.
* ML fields are optional in our creation step
In that we don't initialize them like we do the query (default) fields.
* Only send along type-specific Rule fields from form
This makes any query- or ML-specific fields optional on a Rule, and
performs some logic on the frontend to group and include these fieldsets
conditionally based on the user's selection. The one place we don't
handle this well is on the readonly view of a completed step in the
rules creation, but we'll address that.
* Rename anomalies query
It's no longer tabular data. If we need that, we can use the ML client.
* Remove spike page with simple form
* Remove unneeded ES option
This response isn't going to HTTP, which is where this option would
matter.
* Fix bulk create logic
I made a happy accident and flipped the logic here, which meant we
weren't capping the signals we created.
* Rename argument
Value is a little more ambiguous than data, here: this is our step data.
* Create Rule form stores all values, but filters by type for use
When sending off to the backend, or displaying on the readonly view, we
inspect which rule type we've currently selected, and filter our form
values appropriately.
* Fix editing of ML fields on Rule Create
We need to inherit the field value from our form on initial render, and
everything works as expected.
* Clear form errors when switching between rule types
Validation errors prevent us from moving to the next step, so it was
previously possible to get an error for Query fields, switch to an ML
rule, and be unable to continue because the form had Query errors.
This also adds a helper for checking whether a ruleType is ML, to
prevent having to change all these references if the type string
changes.
* Validate the selection of an ML Job
* Fix type errors on frontend
According to the types, this is essentially the opposite of formatRule,
so we need to reinflate all potential form values from the rule.
* Don't set defaults for query-specific rules
For ML rules these types should not be included.
* Return ML Fields in Rule responses
This adds these fields to our rule serialization, and then adds
conditional validation around those fields if the rule type is ML.
Conversely, we moved the 'language' and 'query' fields to be
conditionally validated if the rule is a query/saved_query rule.
* Fix editing of ML rules by changing who controls the field values
The source of truth for their state is the parent form object; these
inputs should not have local state.
* Fix type errors related to new ML fields
In adding the new ML fields, some other fields (e.g. `query` and
`index`) that were previously required but implicitly part of Query
Rules are now marked as optional.
Consequently, any downstream code that actually required these fields
started to complain. In general, the fix was to verify that those fields
exist, and throw an error otherwise as to appease the linter.
Runtime-wise, the new ML rules/signals follow a separate code path and
both branches should be unaffected by these changes; the issue is simply
that our conditional types don't work well with Typescript.
* Fix failing route tests
Error message changed.
* Fix integration tests
We were not sending required properties when creating a rule(index and
language).
* Fix non-ML Rule creation
I was accidentally dropping this parameter for our POST payload. Whoops.
* More informative logging during ML signal generation
The messaging diverged from the normal path here because we don't have
index patterns to display. However, we have the rest of the rule
context, and should report it appropriately.
* Prefer keyof for string union types
* Tidy up our new form components
* Type them as React.FCs
* Remove unnecessary use of styled-components
* Prefer destructuring to lodash's omit
* Fix mock params for helper functions
These were updated to take simpler parameters.
* Remove any type
This could have been a boolean all along, whoops
* Fix mock types
* Update outdated tests
These were added on master, but behavior has been changed on my branch.
* Add some tests around our helper function
I need to refactor it, so this is as good a time as any to pin down the
behavior.
* Remove uses of any in favor of actual types
Mainly leverages ML typings instead of our placeholder types. This
required handling a null case in our formatting of anomalies.
* Annotate our anomalies with @timestamp field
We were notably lacking this ECS field in our post-conversion anomalies,
and typescript was rightly complaining about it.
* ml_job_id -> machine_learning_job_id
* PR Feedback
* Stricter threshold type
* More robust date parsing
* More informative log/error messages
* Remove redundant runtime checks
* Cleaning up our new ML types
* Fix types on our Rest types
* Use less ambiguous machineLearningJobId over mlJobId
* Declare our ML params as required keys, and ensure we pass them around
everywhere we might need them (creating, importing, updating rules).
* Use implicit type to avoid the need for a ts-ignore
FormSchema has a very generic index signature such that our
filterRuleFieldsForType helper cannot infer that it has our necessary
rule fields (when in fact it does). By removing the FormSchema hint we
get the actual keys of our schema, and things work as expected.
All other uses of schema continue to work because they're expecting
FormSchema, which is effectively { [key: string]: any }.
* New ML params are not nullable
Rather than setting a null and then never using it, let's just make it
truly optional in terms of default values.
* Query and language are conditional based on rule type
For ML Rules, we don't use them.
* Remove defaulted parameter in API test
We don't need to specify this, and we should continue not to for
backwards compatibility.
* Use explicit types over implicit ones
The concern is that not typing our schemae as FormSchema could break our
form if there are upstream changes. For now, we simply use the
intersection of FormSchema and our generic parameter to satisfy our use
within the function.
* Add integration test for creation of ML Rule
* Add ML fields to route schemae
* threshold and job id are conditional on type
* makes query and language mutually exclusive with above
* Fix router test for creating an ML rule
We were sending invalid parameters.
* Remove null check against index for query rules
We support not having an index here, as getInputIndex will return the
current UI setting if none is specified.
* Add regression test for API compatibility
We were previously able to create a rule without an input index; we
should continue to support that, as verified by this test!
* Respect the index pattern determined at runtime when performing search_after
If a rule does not specify an input index pattern on creation, we use
the current UI default when the rule is evaluated. This ensures that any
subsequent searches use that same index.
We're not currently persisting that runtime index to the generated
signal, but we should.
* Fix type errors in our bulk create tests
We added a new argument, but didn't update the tests.
2020-03-19 01:26:42 +01:00
const rule = getSimpleRuleOutput ( ruleId ) ;
const { query , language , index , . . . rest } = rule ;
return {
. . . rest ,
name : 'Simple ML Rule' ,
description : 'Simple Machine Learning Rule' ,
anomaly_threshold : 44 ,
[Security Solution][Detections] ML Rules accept multiple ML Job IDs (#97073)
* Adds helper to normalize legacy ML rule field to an array
This will be used on read of rules, to normalize legacy rules while
avoiding an explicit migration.
* Fix our detection-specific ML search function
Luckily this was just a translation layer to our anomaly call, and the
underlying functions already accepted an array of strings.
* WIP: Run rules against multiple ML Job IDs
We don't yet support creation of rules with multiple job ids, either on
the API or the UI, but when we do they will work.
Note: the logic was previously to generate an error if the underlying
job was not running, but to still query and generate alerts. Extending
that logic to multiple jobs: if any are not running, we generate an
error but continue querying and generating alerts.
* WIP: updating ml rule schemas to support multiple job IDs
* Simplify normalization method
We don't care about null or empty string values here; those were
holdovers from copying the logic of normalizeThreshold and don't apply
to this situation.
* Move normalized types to separate file to fix circular dependency
Our use of NonEmptyArray within common/schemas seemed to be causing the
above; this fixes it for now.
* Normalize ML job_ids param at the API layer
Previous changes to the base types already covered the majority of
routes; this updates the miscellaneous helpers that don't leverage those
shared utilities.
At the DB level, the forthcoming migration will ensure that we always
have "normalized" job IDs as an array.
* Count stopped ML Jobs as partial failure during ML Rule execution
Since we continue to query anomalies and potentially generate alerts, a
"failure" status is no longer the most accurate for this situation.
* Update 7.13 alerts migration to allow multi-job ML Rules
This ensures that we can assume string[] for this field during rule
execution.
* Display N job statuses on rule details
* WIP: converts MLJobSelect to a multiselect
Unfortunately, the SuperSelect does not allow multiselect so we need to
convert this to a combobox. Luckily we can reuse most of the code here
and remain relatively clean.
Since all combobox options must be the same (fixed) height, we're
somewhat more limited than before for displaying the rows. The
truncation appears fine, but I need to figure out a way to display the
full description as well.
* Update client-side logic to handle an array of ML job_ids
* Marginally more legible error message
* Conditionally call our normalize helper only if we have a value
This fixes a type error where TS could not infer that the return value
would not be undefined despite knowing that the argument was never
undefined. I tried some fancy conditional generic types, but that didn't
work.
This is more analogous to normalizeThresholdObject now, anyway.
* Fix remaining type error
* Clean up our ML executor tests with existing contract mocks
* Update ML Executor tests with new logic
We now record a partial failure instead of an error.
* Add and update tests for new ML normalization logic
* Add and update integration tests for ML Rules
Ensures that dealing with legacy job formats continues to work in the
API.
* Fix a type error
These params can no longer be strings.
* Update ML cypress test to create a rule with 2 ML jobs
If we can create a rule with 2 jobs, we should also be able to create a
rule with 1 job.
* Remove unused constant
* Persist a partial failure message written by a rule executor
We added the result.warning field as a way to indicate that a partial
failure was written to the rule, but neglected to account for that in the
main rule execution code, which caused a success status to immediately
overwrite the partial failure if the rule execution did not otherwise
fail/short-circuit.
2021-04-16 04:27:43 +02:00
machine_learning_job_id : [ 'some_job_id' ] ,
[SIEM] Create ML Rules (#58053)
* Remove unnecessary linter exceptions
Not sure what was causing issues here, but it's gone now.
* WIP: Simple form to test creation of ML rules
This will be integrated into the regular rule creation workflow, but for
now this simple form should allow us to exercise the full ML rule
workflow.
* WIP: Adds POST to backend, and type/payload changes necessary to make that work
* Simplify logic with Math.min
* WIP: Failed spike of making an http call
* WIP: Hacking together an ML client
The rest of this is going to be easier if I have actual data. For now
this is mostly copy/pasted and simplified ML code. I've hardcoded time
ranges to a period I know has data for a particular job.
* Threading through our new ML Rule params
It's a bummer that we normalize our rule alert params across all rule
types currently, but that's the deal.
* Retrieve our anomalies during rule execution
Next step: generate signals
* WIP: Generate ECS-compatible ML Signals
This uses as much of the existing signal-creation code as possible. I
skipped the search_after stuff for now because it would require us
recreating the anomalies query which we really shouldn't own. For now,
here's how it works:
* Adds a separate branch of the rule executor for machine_learning rules
* In that branch, we call our new bulkCreateMlSignal function
* This function first transforms the anomaly document into ECS fields
* We then pass the transformed documents to singleBulkCreate, which
does the rest
* After both branches, we update the rule's status appropriately.
We need to do some more work on the anomaly transformation, but this
works!
* Extract setting of rule failure to helper function
We were doing this identically in three places.
* Remove unused import
* Define a field for our Rule Type selection
This adds most of the markup and logic to allow an ML rule type to be
selected. We still need to add things like license-checking and
showing/hiding of fields based on type.
* Hide Query Fields when ML is selected
These are still getting set on the form. We'll need to filter these
fields before we send off the data, and not show them on the readonly
display either.
ALso, edit is majorly broken.
* Add input field for anomaly threshold
* Display numberic values in the readonly view of a step
TIL that isEmpty returns false for numbers and other non-iterable
values. I don't think it's exactly what we want here, but until I figure
out the intention this gets our anomalyThreshold showing up without a
separate logic branch here. Removes the unnecessary branch that was
redundant with the 'else' clause.
* Add field for selecting an ML job
This is not the same as the mockups and lacks some functionality, but
it'll allow us to select a job for now.
* Format our new ML Fields when sending them to the server
So that we don't get rejected due to snake case vs camelcase.
* Put back code that respects a rule's schedule
It was previously hardcoded to a time period I knew had anomalies.
* ML fields are optional in our creation step
In that we don't initialize them like we do the query (default) fields.
* Only send along type-specific Rule fields from form
This makes any query- or ML-specific fields optional on a Rule, and
performs some logic on the frontend to group and include these fieldsets
conditionally based on the user's selection. The one place we don't
handle this well is on the readonly view of a completed step in the
rules creation, but we'll address that.
* Rename anomalies query
It's no longer tabular data. If we need that, we can use the ML client.
* Remove spike page with simple form
* Remove unneeded ES option
This response isn't going to HTTP, which is where this option would
matter.
* Fix bulk create logic
I made a happy accident and flipped the logic here, which meant we
weren't capping the signals we created.
* Rename argument
Value is a little more ambiguous than data, here: this is our step data.
* Create Rule form stores all values, but filters by type for use
When sending off to the backend, or displaying on the readonly view, we
inspect which rule type we've currently selected, and filter our form
values appropriately.
* Fix editing of ML fields on Rule Create
We need to inherit the field value from our form on initial render, and
everything works as expected.
* Clear form errors when switching between rule types
Validation errors prevent us from moving to the next step, so it was
previously possible to get an error for Query fields, switch to an ML
rule, and be unable to continue because the form had Query errors.
This also adds a helper for checking whether a ruleType is ML, to
prevent having to change all these references if the type string
changes.
* Validate the selection of an ML Job
* Fix type errors on frontend
According to the types, this is essentially the opposite of formatRule,
so we need to reinflate all potential form values from the rule.
* Don't set defaults for query-specific rules
For ML rules these types should not be included.
* Return ML Fields in Rule responses
This adds these fields to our rule serialization, and then adds
conditional validation around those fields if the rule type is ML.
Conversely, we moved the 'language' and 'query' fields to be
conditionally validated if the rule is a query/saved_query rule.
* Fix editing of ML rules by changing who controls the field values
The source of truth for their state is the parent form object; these
inputs should not have local state.
* Fix type errors related to new ML fields
In adding the new ML fields, some other fields (e.g. `query` and
`index`) that were previously required but implicitly part of Query
Rules are now marked as optional.
Consequently, any downstream code that actually required these fields
started to complain. In general, the fix was to verify that those fields
exist, and throw an error otherwise as to appease the linter.
Runtime-wise, the new ML rules/signals follow a separate code path and
both branches should be unaffected by these changes; the issue is simply
that our conditional types don't work well with Typescript.
* Fix failing route tests
Error message changed.
* Fix integration tests
We were not sending required properties when creating a rule(index and
language).
* Fix non-ML Rule creation
I was accidentally dropping this parameter for our POST payload. Whoops.
* More informative logging during ML signal generation
The messaging diverged from the normal path here because we don't have
index patterns to display. However, we have the rest of the rule
context, and should report it appropriately.
* Prefer keyof for string union types
* Tidy up our new form components
* Type them as React.FCs
* Remove unnecessary use of styled-components
* Prefer destructuring to lodash's omit
* Fix mock params for helper functions
These were updated to take simpler parameters.
* Remove any type
This could have been a boolean all along, whoops
* Fix mock types
* Update outdated tests
These were added on master, but behavior has been changed on my branch.
* Add some tests around our helper function
I need to refactor it, so this is as good a time as any to pin down the
behavior.
* Remove uses of any in favor of actual types
Mainly leverages ML typings instead of our placeholder types. This
required handling a null case in our formatting of anomalies.
* Annotate our anomalies with @timestamp field
We were notably lacking this ECS field in our post-conversion anomalies,
and typescript was rightly complaining about it.
* ml_job_id -> machine_learning_job_id
* PR Feedback
* Stricter threshold type
* More robust date parsing
* More informative log/error messages
* Remove redundant runtime checks
* Cleaning up our new ML types
* Fix types on our Rest types
* Use less ambiguous machineLearningJobId over mlJobId
* Declare our ML params as required keys, and ensure we pass them around
everywhere we might need them (creating, importing, updating rules).
* Use implicit type to avoid the need for a ts-ignore
FormSchema has a very generic index signature such that our
filterRuleFieldsForType helper cannot infer that it has our necessary
rule fields (when in fact it does). By removing the FormSchema hint we
get the actual keys of our schema, and things work as expected.
All other uses of schema continue to work because they're expecting
FormSchema, which is effectively { [key: string]: any }.
* New ML params are not nullable
Rather than setting a null and then never using it, let's just make it
truly optional in terms of default values.
* Query and language are conditional based on rule type
For ML Rules, we don't use them.
* Remove defaulted parameter in API test
We don't need to specify this, and we should continue not to for
backwards compatibility.
* Use explicit types over implicit ones
The concern is that not typing our schemae as FormSchema could break our
form if there are upstream changes. For now, we simply use the
intersection of FormSchema and our generic parameter to satisfy our use
within the function.
* Add integration test for creation of ML Rule
* Add ML fields to route schemae
* threshold and job id are conditional on type
* makes query and language mutually exclusive with above
* Fix router test for creating an ML rule
We were sending invalid parameters.
* Remove null check against index for query rules
We support not having an index here, as getInputIndex will return the
current UI setting if none is specified.
* Add regression test for API compatibility
We were previously able to create a rule without an input index; we
should continue to support that, as verified by this test!
* Respect the index pattern determined at runtime when performing search_after
If a rule does not specify an input index pattern on creation, we use
the current UI default when the rule is evaluated. This ensures that any
subsequent searches use that same index.
We're not currently persisting that runtime index to the generated
signal, but we should.
* Fix type errors in our bulk create tests
We added a new argument, but didn't update the tests.
2020-03-19 01:26:42 +01:00
type : 'machine_learning' ,
} ;
} ;
2020-02-11 23:24:30 +01:00
/ * *
[Security Solutions][Detection Engine] Adds e2e FTR runtime support and 213 tests for exception lists (#83764)
## Summary
Adds support to the end to end (e2e) functional test runner (FTR) support for rule runtime tests as well as 213 tests for the exception lists which include value based lists. Previously we had limited runtime support, but as I scaled up runtime tests from 5 to 200+ I noticed in a lot of areas we had to use improved techniques for determinism.
The runtime support being added is our next step of tests. Up to now most of our e2e FTR tests have been structural testing of REST and API integration tests. Basically up to now 95% tests are API structural as:
* Call REST input related to a rule such as GET/PUT/POST/PATCH/DELETE.
* Check REST output of the rule, did it match expected output body and status code?
* In some rare cases we check if the the rule can be executed and we get a status of 'succeeded'
With only a small part of our tests ~5%, `generating_signals.ts` was checking the signals being produced. However, we cannot have confidence in runtime based tests until the structural tests have been built up and run through the weeks against PR's to ensure that those are stable and deterministic.
Now that we have confidence and 90%+ coverage of the structural REST based tests, we are building up newer sets of tests which allow us to do runtime based validation tests to increase confidence that:
* Detection engine produces signals as expected
* Structure of the signals are as expected, including signal on signals
* Exceptions to signals are working as expected
* Most runtime bugs can be TDD'ed with e2e FTR's and regressions
* Whack-a-mole will not happen
* Consistency and predictability of signals is validated
* Refactoring can occur with stronger confidence
* Runtime tests are reference points for answering questions about existing bugs or adding new ones to test if users are experiencing unexpected behaviors
* Scaling tests can happen without failures
* Velocity for creating tests increases as the utilities and examples increase
Lastly, this puts us within striking distance of creating FTR's for different common class of runtime situations such as:
* Creating tests that exercise each rule against a set of data criteria and get signal hits
* Creating tests that validate the rule overrides operate as expected against data sets
* Creating tests that validate malfunctions, corner cases, or misuse cases such as data sets that are _all_ arrays or data sets that put numbers as strings or throws in an expected `null` instead of a value.
These tests follow the pattern of:
* Add the smallest data set to a folder in data.json (not gzip format)
* Add the smallest mapping to that folder (mapping.json)
* Call REST input related to exception lists, value lists, adding prepackaged rules, etc...
* Call REST input related endpoint with utilities to create and activate the rule
* Wait for the rule to go into the `succeeded` phase
* Wait for the N exact signals specific to that rule to be available
* Check against the set of signals to ensure that the matches are exactly as expected
Example of one runtime test:
A keyword data set is added to a folder called "keyword" but you can add one anywhere you want under `es_archives`, I just grouped mine depending on the situation of the runtime. Small non-gzipped tests `data.json` and `mappings.json` are the best approach for small focused tests. For _larger_ tests and cases I would and sometimes do use things such as auditbeat but try to avoid using larger data sets in favor of smaller focused test cases to validate the runtime is operating as expected.
```ts
{
"type": "doc",
"value": {
"id": "1",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:00:53.000Z",
"long": 1
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "2",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:01:53.000Z",
"long": 2
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "3",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:02:53.000Z",
"long": 3
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "4",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:03:53.000Z",
"long": 4
},
"type": "_doc"
}
}
```
Mapping is added. Note that this is "ECS tolerant" but not necessarily all ECS meaning I can and will try to keep things simple where I can, but I have ensured that `"@timestamp"` is at least there.
```ts
{
"type": "index",
"value": {
"index": "long",
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"long": { "type": "long" }
}
},
"settings": {
"index": {
"number_of_replicas": "1",
"number_of_shards": "1"
}
}
}
}
```
Test is written with test utilities where the `beforeEach` and `afterEach` try and clean up the indexes and load/unload the archives to keep one test from effecting another. Note this is never going to be 100% possible so see below on how we add more determinism in case something escapes the sandbox.
```ts
beforeEach(async () => {
await createSignalsIndex(supertest);
await createListsIndex(supertest);
await esArchiver.load('rule_exceptions/keyword');
});
afterEach(async () => {
await deleteSignalsIndex(supertest);
await deleteAllAlerts(supertest);
await deleteAllExceptions(es);
await deleteListsIndex(supertest);
await esArchiver.unload('rule_exceptions/keyword');
});
describe('"is" operator', () => {
it('should filter 1 single keyword if it is set as an exception', async () => {
const rule = getRuleForSignalTesting(['keyword']);
const { id } = await createRuleWithExceptionEntries(supertest, rule, [
[
{
field: 'keyword',
operator: 'included',
type: 'match',
value: 'word one',
},
],
]);
await waitForRuleSuccess(supertest, id);
await waitForSignalsToBePresent(supertest, 3, [id]);
const signalsOpen = await getSignalsById(supertest, id);
const hits = signalsOpen.hits.hits.map((hit) => hit._source.keyword).sort();
expect(hits).to.eql(['word four', 'word three', 'word two']);
});
});
```
### Changes for better determinism
To support more determinism there are changes and utilities added which can be tuned during any sporadic failures we might encounter as well as better support unexpected changes to other Elastic Stack pieces such as alerting, task manager, etc...
Get simple rule and others are now defaulting to false, meaning that the structural tests will no longer activate a rule and run it on task manger. This should cut down on error outputs as well as reduce stress and potentials for left over rules interfering with the runtime rules.
```ts
export const getSimpleRule = (ruleId = 'rule-1', enabled = false): QueryCreateSchema => ({
```
Not mandatory to use, but for most tests that should be runtime based tests, I use this function below which will enable it by default and run it using settings such as `type: 'query'`, `query: '*:*',` `from: '1900-01-01T00:00:00.000Z'`, to cut down on boiler plate noise. However, people can use whatever they want out of the grab bag or if their test is more readable to hand craft a REST request to create signals, or if they just want to call this and override where they want to, then 👍 .
```ts
export const getRuleForSignalTesting = (index: string[], ruleId = 'rule-1', enabled = true)
```
This waits for a rule to succeed before continuing
```ts
await waitForRuleSuccess(supertest, id);
```
I added a required array of id that _waits_ only for that particular id here. This is useful in case another test did not cleanup and you are getting signals being produced or left behind but need to wait specifically for yours.
```ts
await waitForSignalsToBePresent(supertest, 4, [id]);
```
I only get the signals for a particular rule id using either the auto-generated id or the rule_id. It's safer to use the ones from the auto-generated id but either of these are fine if you're careful enough.
```ts
const signalsOpen = await getSignalsById(supertest, id);
const signalsOpen = await getSignalsByIds(supertest, [createdId]);
const signalsOpen = await getSignalsByRuleIds(supertest, ['signal-on-signal']);
```
I delete all alerts now through a series of steps where it properly removes all rules using the rules bulk_delete and does it in such a way that all the API keys and alerting will be the best it can destroyed as well as double check that the alerts are showing up as being cleaned up before continuing.
```ts
deleteAllAlerts()
```
When not explicitly testing something structural, prefer to use the utilities which can and will do retries in case there are over the wire failures or es failures. Examples are:
```ts
installPrePackagedRules()
waitForRuleSuccess()
importFile() // This does a _lot_ of checks to ensure that the file is fully imported before continuing
```
Some of these utilities might still do a `expect(200);` but as we are and should use regular structural tests to cover those problems, these will probably be more and more removed when/if we hit test failures in favor of doing retries, waitFor, and countDowns.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2020-11-20 20:09:38 +01:00
* Removes all rules by looping over any found and removing them from REST .
* @param supertest The supertest agent .
2020-02-11 23:24:30 +01:00
* /
[Security Solutions][Detection Engine] Adds e2e FTR runtime support and 213 tests for exception lists (#83764)
## Summary
Adds support to the end to end (e2e) functional test runner (FTR) support for rule runtime tests as well as 213 tests for the exception lists which include value based lists. Previously we had limited runtime support, but as I scaled up runtime tests from 5 to 200+ I noticed in a lot of areas we had to use improved techniques for determinism.
The runtime support being added is our next step of tests. Up to now most of our e2e FTR tests have been structural testing of REST and API integration tests. Basically up to now 95% tests are API structural as:
* Call REST input related to a rule such as GET/PUT/POST/PATCH/DELETE.
* Check REST output of the rule, did it match expected output body and status code?
* In some rare cases we check if the the rule can be executed and we get a status of 'succeeded'
With only a small part of our tests ~5%, `generating_signals.ts` was checking the signals being produced. However, we cannot have confidence in runtime based tests until the structural tests have been built up and run through the weeks against PR's to ensure that those are stable and deterministic.
Now that we have confidence and 90%+ coverage of the structural REST based tests, we are building up newer sets of tests which allow us to do runtime based validation tests to increase confidence that:
* Detection engine produces signals as expected
* Structure of the signals are as expected, including signal on signals
* Exceptions to signals are working as expected
* Most runtime bugs can be TDD'ed with e2e FTR's and regressions
* Whack-a-mole will not happen
* Consistency and predictability of signals is validated
* Refactoring can occur with stronger confidence
* Runtime tests are reference points for answering questions about existing bugs or adding new ones to test if users are experiencing unexpected behaviors
* Scaling tests can happen without failures
* Velocity for creating tests increases as the utilities and examples increase
Lastly, this puts us within striking distance of creating FTR's for different common class of runtime situations such as:
* Creating tests that exercise each rule against a set of data criteria and get signal hits
* Creating tests that validate the rule overrides operate as expected against data sets
* Creating tests that validate malfunctions, corner cases, or misuse cases such as data sets that are _all_ arrays or data sets that put numbers as strings or throws in an expected `null` instead of a value.
These tests follow the pattern of:
* Add the smallest data set to a folder in data.json (not gzip format)
* Add the smallest mapping to that folder (mapping.json)
* Call REST input related to exception lists, value lists, adding prepackaged rules, etc...
* Call REST input related endpoint with utilities to create and activate the rule
* Wait for the rule to go into the `succeeded` phase
* Wait for the N exact signals specific to that rule to be available
* Check against the set of signals to ensure that the matches are exactly as expected
Example of one runtime test:
A keyword data set is added to a folder called "keyword" but you can add one anywhere you want under `es_archives`, I just grouped mine depending on the situation of the runtime. Small non-gzipped tests `data.json` and `mappings.json` are the best approach for small focused tests. For _larger_ tests and cases I would and sometimes do use things such as auditbeat but try to avoid using larger data sets in favor of smaller focused test cases to validate the runtime is operating as expected.
```ts
{
"type": "doc",
"value": {
"id": "1",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:00:53.000Z",
"long": 1
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "2",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:01:53.000Z",
"long": 2
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "3",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:02:53.000Z",
"long": 3
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "4",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:03:53.000Z",
"long": 4
},
"type": "_doc"
}
}
```
Mapping is added. Note that this is "ECS tolerant" but not necessarily all ECS meaning I can and will try to keep things simple where I can, but I have ensured that `"@timestamp"` is at least there.
```ts
{
"type": "index",
"value": {
"index": "long",
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"long": { "type": "long" }
}
},
"settings": {
"index": {
"number_of_replicas": "1",
"number_of_shards": "1"
}
}
}
}
```
Test is written with test utilities where the `beforeEach` and `afterEach` try and clean up the indexes and load/unload the archives to keep one test from effecting another. Note this is never going to be 100% possible so see below on how we add more determinism in case something escapes the sandbox.
```ts
beforeEach(async () => {
await createSignalsIndex(supertest);
await createListsIndex(supertest);
await esArchiver.load('rule_exceptions/keyword');
});
afterEach(async () => {
await deleteSignalsIndex(supertest);
await deleteAllAlerts(supertest);
await deleteAllExceptions(es);
await deleteListsIndex(supertest);
await esArchiver.unload('rule_exceptions/keyword');
});
describe('"is" operator', () => {
it('should filter 1 single keyword if it is set as an exception', async () => {
const rule = getRuleForSignalTesting(['keyword']);
const { id } = await createRuleWithExceptionEntries(supertest, rule, [
[
{
field: 'keyword',
operator: 'included',
type: 'match',
value: 'word one',
},
],
]);
await waitForRuleSuccess(supertest, id);
await waitForSignalsToBePresent(supertest, 3, [id]);
const signalsOpen = await getSignalsById(supertest, id);
const hits = signalsOpen.hits.hits.map((hit) => hit._source.keyword).sort();
expect(hits).to.eql(['word four', 'word three', 'word two']);
});
});
```
### Changes for better determinism
To support more determinism there are changes and utilities added which can be tuned during any sporadic failures we might encounter as well as better support unexpected changes to other Elastic Stack pieces such as alerting, task manager, etc...
Get simple rule and others are now defaulting to false, meaning that the structural tests will no longer activate a rule and run it on task manger. This should cut down on error outputs as well as reduce stress and potentials for left over rules interfering with the runtime rules.
```ts
export const getSimpleRule = (ruleId = 'rule-1', enabled = false): QueryCreateSchema => ({
```
Not mandatory to use, but for most tests that should be runtime based tests, I use this function below which will enable it by default and run it using settings such as `type: 'query'`, `query: '*:*',` `from: '1900-01-01T00:00:00.000Z'`, to cut down on boiler plate noise. However, people can use whatever they want out of the grab bag or if their test is more readable to hand craft a REST request to create signals, or if they just want to call this and override where they want to, then 👍 .
```ts
export const getRuleForSignalTesting = (index: string[], ruleId = 'rule-1', enabled = true)
```
This waits for a rule to succeed before continuing
```ts
await waitForRuleSuccess(supertest, id);
```
I added a required array of id that _waits_ only for that particular id here. This is useful in case another test did not cleanup and you are getting signals being produced or left behind but need to wait specifically for yours.
```ts
await waitForSignalsToBePresent(supertest, 4, [id]);
```
I only get the signals for a particular rule id using either the auto-generated id or the rule_id. It's safer to use the ones from the auto-generated id but either of these are fine if you're careful enough.
```ts
const signalsOpen = await getSignalsById(supertest, id);
const signalsOpen = await getSignalsByIds(supertest, [createdId]);
const signalsOpen = await getSignalsByRuleIds(supertest, ['signal-on-signal']);
```
I delete all alerts now through a series of steps where it properly removes all rules using the rules bulk_delete and does it in such a way that all the API keys and alerting will be the best it can destroyed as well as double check that the alerts are showing up as being cleaned up before continuing.
```ts
deleteAllAlerts()
```
When not explicitly testing something structural, prefer to use the utilities which can and will do retries in case there are over the wire failures or es failures. Examples are:
```ts
installPrePackagedRules()
waitForRuleSuccess()
importFile() // This does a _lot_ of checks to ensure that the file is fully imported before continuing
```
Some of these utilities might still do a `expect(200);` but as we are and should use regular structural tests to cover those problems, these will probably be more and more removed when/if we hit test failures in favor of doing retries, waitFor, and countDowns.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2020-11-20 20:09:38 +01:00
export const deleteAllAlerts = async (
supertest : SuperTest < supertestAsPromised.Test >
) : Promise < void > = > {
await countDownTest (
async ( ) = > {
const { body } = await supertest
. get ( ` ${ DETECTION_ENGINE_RULES_URL } /_find?per_page=9999 ` )
. set ( 'kbn-xsrf' , 'true' )
. send ( ) ;
const ids = body . data . map ( ( rule : FullResponseSchema ) = > ( {
id : rule.id ,
} ) ) ;
await supertest
. post ( ` ${ DETECTION_ENGINE_RULES_URL } /_bulk_delete ` )
. send ( ids )
. set ( 'kbn-xsrf' , 'true' ) ;
const { body : finalCheck } = await supertest
. get ( ` ${ DETECTION_ENGINE_RULES_URL } /_find ` )
. set ( 'kbn-xsrf' , 'true' )
. send ( ) ;
return finalCheck . data . length === 0 ;
} ,
'deleteAllAlerts' ,
50 ,
1000
) ;
2020-10-15 21:21:03 +02:00
} ;
2020-10-02 00:06:26 +02:00
2021-03-25 09:47:16 +01:00
export const downgradeImmutableRule = async ( es : KibanaClient , ruleId : string ) : Promise < void > = > {
2020-10-15 21:21:03 +02:00
return countDownES ( async ( ) = > {
return es . updateByQuery ( {
index : '.kibana' ,
refresh : true ,
wait_for_completion : true ,
body : {
script : {
lang : 'painless' ,
source : 'ctx._source.alert.params.version--' ,
} ,
query : {
term : {
'alert.tags' : ` ${ INTERNAL_RULE_ID_KEY } : ${ ruleId } ` ,
} ,
} ,
} ,
} ) ;
} , 'downgradeImmutableRule' ) ;
2020-02-11 23:24:30 +01:00
} ;
2020-07-14 18:18:32 +02:00
/ * *
* Remove all timelines from the . kibana index
* @param es The ElasticSearch handle
* /
2021-03-25 09:47:16 +01:00
export const deleteAllTimelines = async ( es : KibanaClient ) : Promise < void > = > {
2020-07-14 18:18:32 +02:00
await es . deleteByQuery ( {
index : '.kibana' ,
q : 'type:siem-ui-timeline' ,
wait_for_completion : true ,
refresh : true ,
body : { } ,
} ) ;
} ;
2020-04-16 16:52:43 +02:00
/ * *
* Remove all rules statuses from the . kibana index
2020-07-10 04:36:51 +02:00
* This will retry 20 times before giving up and hopefully still not interfere with other tests
2020-04-16 16:52:43 +02:00
* @param es The ElasticSearch handle
* /
2021-03-25 09:47:16 +01:00
export const deleteAllRulesStatuses = async ( es : KibanaClient ) : Promise < void > = > {
2020-10-15 21:21:03 +02:00
return countDownES ( async ( ) = > {
return es . deleteByQuery ( {
index : '.kibana' ,
q : 'type:siem-detection-engine-rule-status' ,
wait_for_completion : true ,
refresh : true ,
body : { } ,
} ) ;
} , 'deleteAllRulesStatuses' ) ;
2020-04-16 16:52:43 +02:00
} ;
2020-02-11 23:24:30 +01:00
/ * *
* Creates the signals index for use inside of beforeEach blocks of tests
2020-07-10 04:36:51 +02:00
* This will retry 20 times before giving up and hopefully still not interfere with other tests
2020-02-11 23:24:30 +01:00
* @param supertest The supertest client library
* /
2020-05-07 19:18:27 +02:00
export const createSignalsIndex = async (
2020-10-15 21:21:03 +02:00
supertest : SuperTest < supertestAsPromised.Test >
2020-05-07 19:18:27 +02:00
) : Promise < void > = > {
2020-10-15 21:21:03 +02:00
await countDownTest ( async ( ) = > {
await supertest . post ( DETECTION_ENGINE_INDEX_URL ) . set ( 'kbn-xsrf' , 'true' ) . send ( ) ;
return true ;
} , 'createSignalsIndex' ) ;
2020-02-11 23:24:30 +01:00
} ;
/ * *
* Deletes the signals index for use inside of afterEach blocks of tests
* @param supertest The supertest client library
* /
2020-05-07 19:18:27 +02:00
export const deleteSignalsIndex = async (
2020-10-15 21:21:03 +02:00
supertest : SuperTest < supertestAsPromised.Test >
2020-05-07 19:18:27 +02:00
) : Promise < void > = > {
2020-10-15 21:21:03 +02:00
await countDownTest ( async ( ) = > {
await supertest . delete ( DETECTION_ENGINE_INDEX_URL ) . set ( 'kbn-xsrf' , 'true' ) . send ( ) ;
return true ;
} , 'deleteSignalsIndex' ) ;
2020-02-11 23:24:30 +01:00
} ;
/ * *
* Given an array of rule_id strings this will return a ndjson buffer which is useful
* for testing uploads .
* @param ruleIds Array of strings of rule_ids
* /
2020-09-17 22:27:44 +02:00
export const getSimpleRuleAsNdjson = ( ruleIds : string [ ] , enabled = false ) : Buffer = > {
2020-05-22 09:08:58 +02:00
const stringOfRules = ruleIds . map ( ( ruleId ) = > {
2020-09-17 22:27:44 +02:00
const simpleRule = getSimpleRule ( ruleId , enabled ) ;
2020-02-11 23:24:30 +01:00
return JSON . stringify ( simpleRule ) ;
} ) ;
return Buffer . from ( stringOfRules . join ( '\n' ) ) ;
} ;
/ * *
* Given a rule this will convert it to an ndjson buffer which is useful for
* testing upload features .
* @param rule The rule to convert to ndjson
* /
2020-11-13 03:22:13 +01:00
export const ruleToNdjson = ( rule : CreateRulesSchema ) : Buffer = > {
2020-02-11 23:24:30 +01:00
const stringified = JSON . stringify ( rule ) ;
return Buffer . from ( ` ${ stringified } \ n ` ) ;
} ;
/ * *
* This will return a complex rule with all the outputs possible
* @param ruleId The ruleId to set which is optional and defaults to rule - 1
* /
2020-06-10 16:54:27 +02:00
export const getComplexRule = ( ruleId = 'rule-1' ) : Partial < RulesSchema > = > ( {
2020-03-20 10:54:51 +01:00
actions : [ ] ,
2020-07-02 06:49:30 +02:00
author : [ ] ,
2020-02-11 23:24:30 +01:00
name : 'Complex Rule Query' ,
description : 'Complex Rule Query' ,
false_positives : [
'https://www.example.com/some-article-about-a-false-positive' ,
'some text string about why another condition could be a false positive' ,
] ,
risk_score : 1 ,
2020-07-02 06:49:30 +02:00
risk_score_mapping : [ ] ,
2020-02-11 23:24:30 +01:00
rule_id : ruleId ,
filters : [
{
query : {
match_phrase : {
'host.name' : 'siem-windows' ,
} ,
} ,
} ,
] ,
enabled : false ,
index : [ 'auditbeat-*' , 'filebeat-*' ] ,
interval : '5m' ,
output_index : '.siem-signals-default' ,
meta : {
anything_you_want_ui_related_or_otherwise : {
as_deep_structured_as_you_need : {
any_data_type : { } ,
} ,
} ,
} ,
max_signals : 10 ,
tags : [ 'tag 1' , 'tag 2' , 'any tag you want' ] ,
to : 'now' ,
from : 'now-6m' ,
severity : 'high' ,
2020-07-02 06:49:30 +02:00
severity_mapping : [ ] ,
2020-02-11 23:24:30 +01:00
language : 'kuery' ,
type : 'query' ,
threat : [
{
framework : 'MITRE ATT&CK' ,
tactic : {
id : 'TA0040' ,
name : 'impact' ,
reference : 'https://attack.mitre.org/tactics/TA0040/' ,
} ,
technique : [
{
id : 'T1499' ,
name : 'endpoint denial of service' ,
reference : 'https://attack.mitre.org/techniques/T1499/' ,
} ,
] ,
} ,
{
framework : 'Some other Framework you want' ,
tactic : {
id : 'some-other-id' ,
name : 'Some other name' ,
reference : 'https://example.com' ,
} ,
technique : [
{
id : 'some-other-id' ,
name : 'some other technique name' ,
reference : 'https://example.com' ,
} ,
] ,
} ,
] ,
references : [
'http://www.example.com/some-article-about-attack' ,
'Some plain text string here explaining why this is a valid thing to look out for' ,
] ,
timeline_id : 'timeline_id' ,
timeline_title : 'timeline_title' ,
2020-03-11 20:15:38 +01:00
note : '# some investigation documentation' ,
2020-02-11 23:24:30 +01:00
version : 1 ,
query : 'user.name: root or user.name: admin' ,
} ) ;
/ * *
* This will return a complex rule with all the outputs possible
* @param ruleId The ruleId to set which is optional and defaults to rule - 1
* /
2020-06-10 16:54:27 +02:00
export const getComplexRuleOutput = ( ruleId = 'rule-1' ) : Partial < RulesSchema > = > ( {
2020-03-20 10:54:51 +01:00
actions : [ ] ,
2020-07-02 06:49:30 +02:00
author : [ ] ,
2020-02-11 23:24:30 +01:00
created_by : 'elastic' ,
name : 'Complex Rule Query' ,
description : 'Complex Rule Query' ,
false_positives : [
'https://www.example.com/some-article-about-a-false-positive' ,
'some text string about why another condition could be a false positive' ,
] ,
risk_score : 1 ,
2020-07-02 06:49:30 +02:00
risk_score_mapping : [ ] ,
2020-02-11 23:24:30 +01:00
rule_id : ruleId ,
filters : [
{
query : {
match_phrase : {
'host.name' : 'siem-windows' ,
} ,
} ,
} ,
] ,
enabled : false ,
index : [ 'auditbeat-*' , 'filebeat-*' ] ,
immutable : false ,
interval : '5m' ,
output_index : '.siem-signals-default' ,
meta : {
anything_you_want_ui_related_or_otherwise : {
as_deep_structured_as_you_need : {
any_data_type : { } ,
} ,
} ,
} ,
max_signals : 10 ,
tags : [ 'tag 1' , 'tag 2' , 'any tag you want' ] ,
to : 'now' ,
from : 'now-6m' ,
severity : 'high' ,
2020-07-02 06:49:30 +02:00
severity_mapping : [ ] ,
2020-02-11 23:24:30 +01:00
language : 'kuery' ,
type : 'query' ,
threat : [
{
framework : 'MITRE ATT&CK' ,
tactic : {
id : 'TA0040' ,
name : 'impact' ,
reference : 'https://attack.mitre.org/tactics/TA0040/' ,
} ,
technique : [
{
id : 'T1499' ,
name : 'endpoint denial of service' ,
reference : 'https://attack.mitre.org/techniques/T1499/' ,
} ,
] ,
} ,
{
framework : 'Some other Framework you want' ,
tactic : {
id : 'some-other-id' ,
name : 'Some other name' ,
reference : 'https://example.com' ,
} ,
technique : [
{
id : 'some-other-id' ,
name : 'some other technique name' ,
reference : 'https://example.com' ,
} ,
] ,
} ,
] ,
references : [
'http://www.example.com/some-article-about-attack' ,
'Some plain text string here explaining why this is a valid thing to look out for' ,
] ,
2020-03-31 12:54:58 +02:00
throttle : 'no_actions' ,
2020-02-11 23:24:30 +01:00
timeline_id : 'timeline_id' ,
timeline_title : 'timeline_title' ,
updated_by : 'elastic' ,
2020-03-11 20:15:38 +01:00
note : '# some investigation documentation' ,
2020-02-11 23:24:30 +01:00
version : 1 ,
query : 'user.name: root or user.name: admin' ,
2020-04-18 00:58:35 +02:00
exceptions_list : [ ] ,
2020-02-11 23:24:30 +01:00
} ) ;
2020-06-16 16:41:59 +02:00
2020-08-26 17:01:32 +02:00
export const getWebHookAction = ( ) = > ( {
actionTypeId : '.webhook' ,
config : {
method : 'post' ,
url : 'http://localhost' ,
} ,
secrets : {
user : 'example' ,
password : 'example' ,
} ,
name : 'Some connector' ,
} ) ;
2021-02-05 03:07:14 +01:00
export const getRuleWithWebHookAction = (
id : string ,
enabled = false ,
2021-05-04 23:42:41 +02:00
rule? : CreateRulesSchema
2021-02-05 03:07:14 +01:00
) : CreateRulesSchema | UpdateRulesSchema = > {
const finalRule = rule != null ? { . . . rule , enabled } : getSimpleRule ( 'rule-1' , enabled ) ;
return {
. . . finalRule ,
throttle : 'rule' ,
actions : [
{
group : 'default' ,
id ,
params : {
body : '{}' ,
} ,
action_type_id : '.webhook' ,
2020-08-26 17:01:32 +02:00
} ,
2021-02-05 03:07:14 +01:00
] ,
} ;
} ;
2020-08-26 17:01:32 +02:00
export const getSimpleRuleOutputWithWebHookAction = ( actionId : string ) : Partial < RulesSchema > = > ( {
. . . getSimpleRuleOutput ( ) ,
throttle : 'rule' ,
actions : [
{
action_type_id : '.webhook' ,
group : 'default' ,
id : actionId ,
params : {
body : '{}' ,
} ,
} ,
] ,
} ) ;
2020-06-16 16:41:59 +02:00
// Similar to ReactJs's waitFor from here: https://testing-library.com/docs/dom-testing-library/api-async#waitfor
export const waitFor = async (
functionToTest : ( ) = > Promise < boolean > ,
[Security Solutions][Detection Engine] Adds e2e FTR runtime support and 213 tests for exception lists (#83764)
## Summary
Adds support to the end to end (e2e) functional test runner (FTR) support for rule runtime tests as well as 213 tests for the exception lists which include value based lists. Previously we had limited runtime support, but as I scaled up runtime tests from 5 to 200+ I noticed in a lot of areas we had to use improved techniques for determinism.
The runtime support being added is our next step of tests. Up to now most of our e2e FTR tests have been structural testing of REST and API integration tests. Basically up to now 95% tests are API structural as:
* Call REST input related to a rule such as GET/PUT/POST/PATCH/DELETE.
* Check REST output of the rule, did it match expected output body and status code?
* In some rare cases we check if the the rule can be executed and we get a status of 'succeeded'
With only a small part of our tests ~5%, `generating_signals.ts` was checking the signals being produced. However, we cannot have confidence in runtime based tests until the structural tests have been built up and run through the weeks against PR's to ensure that those are stable and deterministic.
Now that we have confidence and 90%+ coverage of the structural REST based tests, we are building up newer sets of tests which allow us to do runtime based validation tests to increase confidence that:
* Detection engine produces signals as expected
* Structure of the signals are as expected, including signal on signals
* Exceptions to signals are working as expected
* Most runtime bugs can be TDD'ed with e2e FTR's and regressions
* Whack-a-mole will not happen
* Consistency and predictability of signals is validated
* Refactoring can occur with stronger confidence
* Runtime tests are reference points for answering questions about existing bugs or adding new ones to test if users are experiencing unexpected behaviors
* Scaling tests can happen without failures
* Velocity for creating tests increases as the utilities and examples increase
Lastly, this puts us within striking distance of creating FTR's for different common class of runtime situations such as:
* Creating tests that exercise each rule against a set of data criteria and get signal hits
* Creating tests that validate the rule overrides operate as expected against data sets
* Creating tests that validate malfunctions, corner cases, or misuse cases such as data sets that are _all_ arrays or data sets that put numbers as strings or throws in an expected `null` instead of a value.
These tests follow the pattern of:
* Add the smallest data set to a folder in data.json (not gzip format)
* Add the smallest mapping to that folder (mapping.json)
* Call REST input related to exception lists, value lists, adding prepackaged rules, etc...
* Call REST input related endpoint with utilities to create and activate the rule
* Wait for the rule to go into the `succeeded` phase
* Wait for the N exact signals specific to that rule to be available
* Check against the set of signals to ensure that the matches are exactly as expected
Example of one runtime test:
A keyword data set is added to a folder called "keyword" but you can add one anywhere you want under `es_archives`, I just grouped mine depending on the situation of the runtime. Small non-gzipped tests `data.json` and `mappings.json` are the best approach for small focused tests. For _larger_ tests and cases I would and sometimes do use things such as auditbeat but try to avoid using larger data sets in favor of smaller focused test cases to validate the runtime is operating as expected.
```ts
{
"type": "doc",
"value": {
"id": "1",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:00:53.000Z",
"long": 1
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "2",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:01:53.000Z",
"long": 2
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "3",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:02:53.000Z",
"long": 3
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "4",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:03:53.000Z",
"long": 4
},
"type": "_doc"
}
}
```
Mapping is added. Note that this is "ECS tolerant" but not necessarily all ECS meaning I can and will try to keep things simple where I can, but I have ensured that `"@timestamp"` is at least there.
```ts
{
"type": "index",
"value": {
"index": "long",
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"long": { "type": "long" }
}
},
"settings": {
"index": {
"number_of_replicas": "1",
"number_of_shards": "1"
}
}
}
}
```
Test is written with test utilities where the `beforeEach` and `afterEach` try and clean up the indexes and load/unload the archives to keep one test from effecting another. Note this is never going to be 100% possible so see below on how we add more determinism in case something escapes the sandbox.
```ts
beforeEach(async () => {
await createSignalsIndex(supertest);
await createListsIndex(supertest);
await esArchiver.load('rule_exceptions/keyword');
});
afterEach(async () => {
await deleteSignalsIndex(supertest);
await deleteAllAlerts(supertest);
await deleteAllExceptions(es);
await deleteListsIndex(supertest);
await esArchiver.unload('rule_exceptions/keyword');
});
describe('"is" operator', () => {
it('should filter 1 single keyword if it is set as an exception', async () => {
const rule = getRuleForSignalTesting(['keyword']);
const { id } = await createRuleWithExceptionEntries(supertest, rule, [
[
{
field: 'keyword',
operator: 'included',
type: 'match',
value: 'word one',
},
],
]);
await waitForRuleSuccess(supertest, id);
await waitForSignalsToBePresent(supertest, 3, [id]);
const signalsOpen = await getSignalsById(supertest, id);
const hits = signalsOpen.hits.hits.map((hit) => hit._source.keyword).sort();
expect(hits).to.eql(['word four', 'word three', 'word two']);
});
});
```
### Changes for better determinism
To support more determinism there are changes and utilities added which can be tuned during any sporadic failures we might encounter as well as better support unexpected changes to other Elastic Stack pieces such as alerting, task manager, etc...
Get simple rule and others are now defaulting to false, meaning that the structural tests will no longer activate a rule and run it on task manger. This should cut down on error outputs as well as reduce stress and potentials for left over rules interfering with the runtime rules.
```ts
export const getSimpleRule = (ruleId = 'rule-1', enabled = false): QueryCreateSchema => ({
```
Not mandatory to use, but for most tests that should be runtime based tests, I use this function below which will enable it by default and run it using settings such as `type: 'query'`, `query: '*:*',` `from: '1900-01-01T00:00:00.000Z'`, to cut down on boiler plate noise. However, people can use whatever they want out of the grab bag or if their test is more readable to hand craft a REST request to create signals, or if they just want to call this and override where they want to, then 👍 .
```ts
export const getRuleForSignalTesting = (index: string[], ruleId = 'rule-1', enabled = true)
```
This waits for a rule to succeed before continuing
```ts
await waitForRuleSuccess(supertest, id);
```
I added a required array of id that _waits_ only for that particular id here. This is useful in case another test did not cleanup and you are getting signals being produced or left behind but need to wait specifically for yours.
```ts
await waitForSignalsToBePresent(supertest, 4, [id]);
```
I only get the signals for a particular rule id using either the auto-generated id or the rule_id. It's safer to use the ones from the auto-generated id but either of these are fine if you're careful enough.
```ts
const signalsOpen = await getSignalsById(supertest, id);
const signalsOpen = await getSignalsByIds(supertest, [createdId]);
const signalsOpen = await getSignalsByRuleIds(supertest, ['signal-on-signal']);
```
I delete all alerts now through a series of steps where it properly removes all rules using the rules bulk_delete and does it in such a way that all the API keys and alerting will be the best it can destroyed as well as double check that the alerts are showing up as being cleaned up before continuing.
```ts
deleteAllAlerts()
```
When not explicitly testing something structural, prefer to use the utilities which can and will do retries in case there are over the wire failures or es failures. Examples are:
```ts
installPrePackagedRules()
waitForRuleSuccess()
importFile() // This does a _lot_ of checks to ensure that the file is fully imported before continuing
```
Some of these utilities might still do a `expect(200);` but as we are and should use regular structural tests to cover those problems, these will probably be more and more removed when/if we hit test failures in favor of doing retries, waitFor, and countDowns.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2020-11-20 20:09:38 +01:00
functionName : string ,
2021-05-18 20:03:49 +02:00
maxTimeout : number = 20000 ,
2020-06-16 16:41:59 +02:00
timeoutWait : number = 10
2020-10-15 21:21:03 +02:00
) : Promise < void > = > {
2020-11-24 16:04:33 +01:00
await new Promise < void > ( async ( resolve , reject ) = > {
2020-06-16 16:41:59 +02:00
let found = false ;
let numberOfTries = 0 ;
while ( ! found && numberOfTries < Math . floor ( maxTimeout / timeoutWait ) ) {
const itPasses = await functionToTest ( ) ;
if ( itPasses ) {
found = true ;
} else {
numberOfTries ++ ;
}
await new Promise ( ( resolveTimeout ) = > setTimeout ( resolveTimeout , timeoutWait ) ) ;
}
if ( found ) {
resolve ( ) ;
} else {
[Security Solutions][Detection Engine] Adds e2e FTR runtime support and 213 tests for exception lists (#83764)
## Summary
Adds support to the end to end (e2e) functional test runner (FTR) support for rule runtime tests as well as 213 tests for the exception lists which include value based lists. Previously we had limited runtime support, but as I scaled up runtime tests from 5 to 200+ I noticed in a lot of areas we had to use improved techniques for determinism.
The runtime support being added is our next step of tests. Up to now most of our e2e FTR tests have been structural testing of REST and API integration tests. Basically up to now 95% tests are API structural as:
* Call REST input related to a rule such as GET/PUT/POST/PATCH/DELETE.
* Check REST output of the rule, did it match expected output body and status code?
* In some rare cases we check if the the rule can be executed and we get a status of 'succeeded'
With only a small part of our tests ~5%, `generating_signals.ts` was checking the signals being produced. However, we cannot have confidence in runtime based tests until the structural tests have been built up and run through the weeks against PR's to ensure that those are stable and deterministic.
Now that we have confidence and 90%+ coverage of the structural REST based tests, we are building up newer sets of tests which allow us to do runtime based validation tests to increase confidence that:
* Detection engine produces signals as expected
* Structure of the signals are as expected, including signal on signals
* Exceptions to signals are working as expected
* Most runtime bugs can be TDD'ed with e2e FTR's and regressions
* Whack-a-mole will not happen
* Consistency and predictability of signals is validated
* Refactoring can occur with stronger confidence
* Runtime tests are reference points for answering questions about existing bugs or adding new ones to test if users are experiencing unexpected behaviors
* Scaling tests can happen without failures
* Velocity for creating tests increases as the utilities and examples increase
Lastly, this puts us within striking distance of creating FTR's for different common class of runtime situations such as:
* Creating tests that exercise each rule against a set of data criteria and get signal hits
* Creating tests that validate the rule overrides operate as expected against data sets
* Creating tests that validate malfunctions, corner cases, or misuse cases such as data sets that are _all_ arrays or data sets that put numbers as strings or throws in an expected `null` instead of a value.
These tests follow the pattern of:
* Add the smallest data set to a folder in data.json (not gzip format)
* Add the smallest mapping to that folder (mapping.json)
* Call REST input related to exception lists, value lists, adding prepackaged rules, etc...
* Call REST input related endpoint with utilities to create and activate the rule
* Wait for the rule to go into the `succeeded` phase
* Wait for the N exact signals specific to that rule to be available
* Check against the set of signals to ensure that the matches are exactly as expected
Example of one runtime test:
A keyword data set is added to a folder called "keyword" but you can add one anywhere you want under `es_archives`, I just grouped mine depending on the situation of the runtime. Small non-gzipped tests `data.json` and `mappings.json` are the best approach for small focused tests. For _larger_ tests and cases I would and sometimes do use things such as auditbeat but try to avoid using larger data sets in favor of smaller focused test cases to validate the runtime is operating as expected.
```ts
{
"type": "doc",
"value": {
"id": "1",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:00:53.000Z",
"long": 1
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "2",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:01:53.000Z",
"long": 2
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "3",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:02:53.000Z",
"long": 3
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "4",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:03:53.000Z",
"long": 4
},
"type": "_doc"
}
}
```
Mapping is added. Note that this is "ECS tolerant" but not necessarily all ECS meaning I can and will try to keep things simple where I can, but I have ensured that `"@timestamp"` is at least there.
```ts
{
"type": "index",
"value": {
"index": "long",
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"long": { "type": "long" }
}
},
"settings": {
"index": {
"number_of_replicas": "1",
"number_of_shards": "1"
}
}
}
}
```
Test is written with test utilities where the `beforeEach` and `afterEach` try and clean up the indexes and load/unload the archives to keep one test from effecting another. Note this is never going to be 100% possible so see below on how we add more determinism in case something escapes the sandbox.
```ts
beforeEach(async () => {
await createSignalsIndex(supertest);
await createListsIndex(supertest);
await esArchiver.load('rule_exceptions/keyword');
});
afterEach(async () => {
await deleteSignalsIndex(supertest);
await deleteAllAlerts(supertest);
await deleteAllExceptions(es);
await deleteListsIndex(supertest);
await esArchiver.unload('rule_exceptions/keyword');
});
describe('"is" operator', () => {
it('should filter 1 single keyword if it is set as an exception', async () => {
const rule = getRuleForSignalTesting(['keyword']);
const { id } = await createRuleWithExceptionEntries(supertest, rule, [
[
{
field: 'keyword',
operator: 'included',
type: 'match',
value: 'word one',
},
],
]);
await waitForRuleSuccess(supertest, id);
await waitForSignalsToBePresent(supertest, 3, [id]);
const signalsOpen = await getSignalsById(supertest, id);
const hits = signalsOpen.hits.hits.map((hit) => hit._source.keyword).sort();
expect(hits).to.eql(['word four', 'word three', 'word two']);
});
});
```
### Changes for better determinism
To support more determinism there are changes and utilities added which can be tuned during any sporadic failures we might encounter as well as better support unexpected changes to other Elastic Stack pieces such as alerting, task manager, etc...
Get simple rule and others are now defaulting to false, meaning that the structural tests will no longer activate a rule and run it on task manger. This should cut down on error outputs as well as reduce stress and potentials for left over rules interfering with the runtime rules.
```ts
export const getSimpleRule = (ruleId = 'rule-1', enabled = false): QueryCreateSchema => ({
```
Not mandatory to use, but for most tests that should be runtime based tests, I use this function below which will enable it by default and run it using settings such as `type: 'query'`, `query: '*:*',` `from: '1900-01-01T00:00:00.000Z'`, to cut down on boiler plate noise. However, people can use whatever they want out of the grab bag or if their test is more readable to hand craft a REST request to create signals, or if they just want to call this and override where they want to, then 👍 .
```ts
export const getRuleForSignalTesting = (index: string[], ruleId = 'rule-1', enabled = true)
```
This waits for a rule to succeed before continuing
```ts
await waitForRuleSuccess(supertest, id);
```
I added a required array of id that _waits_ only for that particular id here. This is useful in case another test did not cleanup and you are getting signals being produced or left behind but need to wait specifically for yours.
```ts
await waitForSignalsToBePresent(supertest, 4, [id]);
```
I only get the signals for a particular rule id using either the auto-generated id or the rule_id. It's safer to use the ones from the auto-generated id but either of these are fine if you're careful enough.
```ts
const signalsOpen = await getSignalsById(supertest, id);
const signalsOpen = await getSignalsByIds(supertest, [createdId]);
const signalsOpen = await getSignalsByRuleIds(supertest, ['signal-on-signal']);
```
I delete all alerts now through a series of steps where it properly removes all rules using the rules bulk_delete and does it in such a way that all the API keys and alerting will be the best it can destroyed as well as double check that the alerts are showing up as being cleaned up before continuing.
```ts
deleteAllAlerts()
```
When not explicitly testing something structural, prefer to use the utilities which can and will do retries in case there are over the wire failures or es failures. Examples are:
```ts
installPrePackagedRules()
waitForRuleSuccess()
importFile() // This does a _lot_ of checks to ensure that the file is fully imported before continuing
```
Some of these utilities might still do a `expect(200);` but as we are and should use regular structural tests to cover those problems, these will probably be more and more removed when/if we hit test failures in favor of doing retries, waitFor, and countDowns.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2020-11-20 20:09:38 +01:00
reject (
new Error ( ` timed out waiting for function condition to be true within ${ functionName } ` )
) ;
2020-06-16 16:41:59 +02:00
}
} ) ;
} ;
2020-10-15 21:21:03 +02:00
/ * *
* Does a plain countdown and checks against es queries for either conflicts in the error
* or for any over the wire issues such as timeouts or temp 404 ' s to make the tests more
* reliant .
* @param esFunction The function to test against
* @param esFunctionName The name of the function to print if we encounter errors
* @param retryCount The number of times to retry before giving up ( has default )
* @param timeoutWait Time to wait before trying again ( has default )
* /
export const countDownES = async (
esFunction : ( ) = > Promise < ApiResponse < Record < string , any > , Context >> ,
esFunctionName : string ,
retryCount : number = 20 ,
timeoutWait = 250
) : Promise < void > = > {
await countDownTest (
async ( ) = > {
const result = await esFunction ( ) ;
if ( result . body . version_conflicts !== 0 ) {
// eslint-disable-next-line no-console
console . log ( ` Version conflicts for ${ result . body . version_conflicts } ` ) ;
return false ;
} else {
return true ;
}
} ,
esFunctionName ,
retryCount ,
timeoutWait
) ;
} ;
2021-04-19 22:10:59 +02:00
/ * *
* Refresh an index , making changes available to search .
* Useful for tests where we want to ensure that a rule does NOT create alerts , e . g . testing exceptions .
* @param es The ElasticSearch handle
* /
export const refreshIndex = async ( es : KibanaClient , index? : string ) = > {
await es . indices . refresh ( {
index ,
} ) ;
} ;
2020-10-15 21:21:03 +02:00
/ * *
* Does a plain countdown and checks against a boolean to determine if to wait and try again .
* This is useful for over the wire things that can cause issues such as conflict or timeouts
* for testing resiliency .
* @param functionToTest The function to test against
* @param name The name of the function to print if we encounter errors
* @param retryCount The number of times to retry before giving up ( has default )
* @param timeoutWait Time to wait before trying again ( has default )
* /
export const countDownTest = async (
functionToTest : ( ) = > Promise < boolean > ,
name : string ,
retryCount : number = 20 ,
timeoutWait = 250 ,
ignoreThrow : boolean = false
) = > {
if ( retryCount > 0 ) {
try {
const passed = await functionToTest ( ) ;
if ( ! passed ) {
// eslint-disable-next-line no-console
console . log ( ` Failure trying to ${ name } , retries left are: ${ retryCount - 1 } ` ) ;
// retry, counting down, and delay a bit before
await new Promise ( ( resolve ) = > setTimeout ( resolve , timeoutWait ) ) ;
await countDownTest ( functionToTest , name , retryCount - 1 , timeoutWait , ignoreThrow ) ;
}
} catch ( err ) {
if ( ignoreThrow ) {
throw err ;
} else {
// eslint-disable-next-line no-console
console . log (
` Failure trying to ${ name } , with exception message of: ` ,
err . message ,
` retries left are: ${ retryCount - 1 } `
) ;
// retry, counting down, and delay a bit before
await new Promise ( ( resolve ) = > setTimeout ( resolve , timeoutWait ) ) ;
await countDownTest ( functionToTest , name , retryCount - 1 , timeoutWait , ignoreThrow ) ;
}
}
} else {
// eslint-disable-next-line no-console
console . log ( ` Could not ${ name } , no retries are left ` ) ;
}
} ;
2020-10-20 03:14:09 +02:00
/ * *
* Helper to cut down on the noise in some of the tests . This checks for
* an expected 200 still and does not try to any retries .
* @param supertest The supertest deps
* @param rule The rule to create
* /
export const createRule = async (
supertest : SuperTest < supertestAsPromised.Test > ,
rule : CreateRulesSchema
2020-11-13 03:22:13 +01:00
) : Promise < FullResponseSchema > = > {
2020-10-20 03:14:09 +02:00
const { body } = await supertest
. post ( DETECTION_ENGINE_RULES_URL )
. set ( 'kbn-xsrf' , 'true' )
. send ( rule )
. expect ( 200 ) ;
return body ;
} ;
2021-02-05 03:07:14 +01:00
/ * *
* Helper to cut down on the noise in some of the tests . This checks for
* an expected 200 still and does not do any retries .
* @param supertest The supertest deps
* @param rule The rule to create
* /
export const updateRule = async (
supertest : SuperTest < supertestAsPromised.Test > ,
updatedRule : UpdateRulesSchema
) : Promise < FullResponseSchema > = > {
const { body } = await supertest
. put ( DETECTION_ENGINE_RULES_URL )
. set ( 'kbn-xsrf' , 'true' )
. send ( updatedRule )
. expect ( 200 ) ;
return body ;
} ;
/ * *
* Helper to cut down on the noise in some of the tests . This
* creates a new action and expects a 200 and does not do any retries .
* @param supertest The supertest deps
* /
export const createNewAction = async ( supertest : SuperTest < supertestAsPromised.Test > ) = > {
const { body } = await supertest
. post ( '/api/actions/action' )
. set ( 'kbn-xsrf' , 'true' )
. send ( getWebHookAction ( ) )
. expect ( 200 ) ;
return body ;
} ;
/ * *
* Helper to cut down on the noise in some of the tests . This
2021-05-04 23:42:41 +02:00
* uses the find API to get an immutable rule by id .
2021-02-05 03:07:14 +01:00
* @param supertest The supertest deps
* /
export const findImmutableRuleById = async (
supertest : SuperTest < supertestAsPromised.Test > ,
ruleId : string
) : Promise < {
page : number ;
perPage : number ;
total : number ;
data : FullResponseSchema [ ] ;
} > = > {
const { body } = await supertest
. get (
` ${ DETECTION_ENGINE_RULES_URL } /_find?filter=alert.attributes.tags: " ${ INTERNAL_IMMUTABLE_KEY } :true" AND alert.attributes.tags: " ${ INTERNAL_RULE_ID_KEY } : ${ ruleId } " `
)
. set ( 'kbn-xsrf' , 'true' )
. send ( )
. expect ( 200 ) ;
return body ;
} ;
/ * *
* Helper to cut down on the noise in some of the tests . This
* creates a new action and expects a 200 and does not do any retries .
* @param supertest The supertest deps
* /
export const getPrePackagedRulesStatus = async (
supertest : SuperTest < supertestAsPromised.Test >
) : Promise < PrePackagedRulesAndTimelinesStatusSchema > = > {
const { body } = await supertest
. get ( ` ${ DETECTION_ENGINE_PREPACKAGED_URL } /_status ` )
. set ( 'kbn-xsrf' , 'true' )
. send ( )
. expect ( 200 ) ;
return body ;
} ;
2020-10-20 03:14:09 +02:00
/ * *
* Helper to cut down on the noise in some of the tests . This checks for
* an expected 200 still and does not try to any retries . Creates exception lists
* @param supertest The supertest deps
* @param rule The rule to create
* /
export const createExceptionList = async (
supertest : SuperTest < supertestAsPromised.Test > ,
exceptionList : CreateExceptionListSchema
) : Promise < ExceptionListSchema > = > {
const { body } = await supertest
. post ( EXCEPTION_LIST_URL )
. set ( 'kbn-xsrf' , 'true' )
. send ( exceptionList )
. expect ( 200 ) ;
return body ;
} ;
/ * *
* Helper to cut down on the noise in some of the tests . This checks for
* an expected 200 still and does not try to any retries . Creates exception lists
* @param supertest The supertest deps
* @param rule The rule to create
* /
export const createExceptionListItem = async (
supertest : SuperTest < supertestAsPromised.Test > ,
exceptionListItem : CreateExceptionListItemSchema
) : Promise < ExceptionListItemSchema > = > {
const { body } = await supertest
. post ( EXCEPTION_LIST_ITEM_URL )
. set ( 'kbn-xsrf' , 'true' )
. send ( exceptionListItem )
. expect ( 200 ) ;
return body ;
} ;
/ * *
* Helper to cut down on the noise in some of the tests . This gets
* a particular rule .
* @param supertest The supertest deps
* @param rule The rule to create
* /
export const getRule = async (
supertest : SuperTest < supertestAsPromised.Test > ,
ruleId : string
) : Promise < RulesSchema > = > {
const { body } = await supertest
. get ( ` ${ DETECTION_ENGINE_RULES_URL } ?rule_id= ${ ruleId } ` )
. set ( 'kbn-xsrf' , 'true' )
. expect ( 200 ) ;
return body ;
} ;
2021-02-17 23:08:41 +01:00
export const waitForAlertToComplete = async (
supertest : SuperTest < supertestAsPromised.Test > ,
id : string
) : Promise < void > = > {
await waitFor ( async ( ) = > {
const { body : alertBody } = await supertest
. get ( ` /api/alerts/alert/ ${ id } /state ` )
. set ( 'kbn-xsrf' , 'true' )
. expect ( 200 ) ;
return alertBody . previousStartedAt != null ;
} , 'waitForAlertToComplete' ) ;
} ;
2020-10-20 03:14:09 +02:00
/ * *
2020-12-24 03:44:57 +01:00
* Waits for the rule in find status to be 'succeeded'
* or the provided status , before continuing
2020-10-20 03:14:09 +02:00
* @param supertest Deps
* /
2020-12-24 03:44:57 +01:00
export const waitForRuleSuccessOrStatus = async (
2020-10-20 03:14:09 +02:00
supertest : SuperTest < supertestAsPromised.Test > ,
2020-12-24 03:44:57 +01:00
id : string ,
2021-02-17 02:24:19 +01:00
status : 'succeeded' | 'failed' | 'partial failure' | 'warning' = 'succeeded'
2020-10-20 03:14:09 +02:00
) : Promise < void > = > {
await waitFor ( async ( ) = > {
const { body } = await supertest
. post ( ` ${ DETECTION_ENGINE_RULES_URL } /_find_statuses ` )
. set ( 'kbn-xsrf' , 'true' )
. send ( { ids : [ id ] } )
. expect ( 200 ) ;
2020-12-24 03:44:57 +01:00
return body [ id ] ? . current_status ? . status === status ;
2021-06-09 01:31:54 +02:00
} , 'waitForRuleSuccessOrStatus' ) ;
2020-10-20 03:14:09 +02:00
} ;
/ * *
2020-10-20 18:40:00 +02:00
* Waits for the signal hits to be greater than the supplied number
* before continuing with a default of at least one signal
2020-10-20 03:14:09 +02:00
* @param supertest Deps
2020-10-20 18:40:00 +02:00
* @param numberOfSignals The number of signals to wait for , default is 1
2020-10-20 03:14:09 +02:00
* /
export const waitForSignalsToBePresent = async (
2020-10-20 18:40:00 +02:00
supertest : SuperTest < supertestAsPromised.Test > ,
[Security Solutions][Detection Engine] Adds e2e FTR runtime support and 213 tests for exception lists (#83764)
## Summary
Adds support to the end to end (e2e) functional test runner (FTR) support for rule runtime tests as well as 213 tests for the exception lists which include value based lists. Previously we had limited runtime support, but as I scaled up runtime tests from 5 to 200+ I noticed in a lot of areas we had to use improved techniques for determinism.
The runtime support being added is our next step of tests. Up to now most of our e2e FTR tests have been structural testing of REST and API integration tests. Basically up to now 95% tests are API structural as:
* Call REST input related to a rule such as GET/PUT/POST/PATCH/DELETE.
* Check REST output of the rule, did it match expected output body and status code?
* In some rare cases we check if the the rule can be executed and we get a status of 'succeeded'
With only a small part of our tests ~5%, `generating_signals.ts` was checking the signals being produced. However, we cannot have confidence in runtime based tests until the structural tests have been built up and run through the weeks against PR's to ensure that those are stable and deterministic.
Now that we have confidence and 90%+ coverage of the structural REST based tests, we are building up newer sets of tests which allow us to do runtime based validation tests to increase confidence that:
* Detection engine produces signals as expected
* Structure of the signals are as expected, including signal on signals
* Exceptions to signals are working as expected
* Most runtime bugs can be TDD'ed with e2e FTR's and regressions
* Whack-a-mole will not happen
* Consistency and predictability of signals is validated
* Refactoring can occur with stronger confidence
* Runtime tests are reference points for answering questions about existing bugs or adding new ones to test if users are experiencing unexpected behaviors
* Scaling tests can happen without failures
* Velocity for creating tests increases as the utilities and examples increase
Lastly, this puts us within striking distance of creating FTR's for different common class of runtime situations such as:
* Creating tests that exercise each rule against a set of data criteria and get signal hits
* Creating tests that validate the rule overrides operate as expected against data sets
* Creating tests that validate malfunctions, corner cases, or misuse cases such as data sets that are _all_ arrays or data sets that put numbers as strings or throws in an expected `null` instead of a value.
These tests follow the pattern of:
* Add the smallest data set to a folder in data.json (not gzip format)
* Add the smallest mapping to that folder (mapping.json)
* Call REST input related to exception lists, value lists, adding prepackaged rules, etc...
* Call REST input related endpoint with utilities to create and activate the rule
* Wait for the rule to go into the `succeeded` phase
* Wait for the N exact signals specific to that rule to be available
* Check against the set of signals to ensure that the matches are exactly as expected
Example of one runtime test:
A keyword data set is added to a folder called "keyword" but you can add one anywhere you want under `es_archives`, I just grouped mine depending on the situation of the runtime. Small non-gzipped tests `data.json` and `mappings.json` are the best approach for small focused tests. For _larger_ tests and cases I would and sometimes do use things such as auditbeat but try to avoid using larger data sets in favor of smaller focused test cases to validate the runtime is operating as expected.
```ts
{
"type": "doc",
"value": {
"id": "1",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:00:53.000Z",
"long": 1
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "2",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:01:53.000Z",
"long": 2
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "3",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:02:53.000Z",
"long": 3
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "4",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:03:53.000Z",
"long": 4
},
"type": "_doc"
}
}
```
Mapping is added. Note that this is "ECS tolerant" but not necessarily all ECS meaning I can and will try to keep things simple where I can, but I have ensured that `"@timestamp"` is at least there.
```ts
{
"type": "index",
"value": {
"index": "long",
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"long": { "type": "long" }
}
},
"settings": {
"index": {
"number_of_replicas": "1",
"number_of_shards": "1"
}
}
}
}
```
Test is written with test utilities where the `beforeEach` and `afterEach` try and clean up the indexes and load/unload the archives to keep one test from effecting another. Note this is never going to be 100% possible so see below on how we add more determinism in case something escapes the sandbox.
```ts
beforeEach(async () => {
await createSignalsIndex(supertest);
await createListsIndex(supertest);
await esArchiver.load('rule_exceptions/keyword');
});
afterEach(async () => {
await deleteSignalsIndex(supertest);
await deleteAllAlerts(supertest);
await deleteAllExceptions(es);
await deleteListsIndex(supertest);
await esArchiver.unload('rule_exceptions/keyword');
});
describe('"is" operator', () => {
it('should filter 1 single keyword if it is set as an exception', async () => {
const rule = getRuleForSignalTesting(['keyword']);
const { id } = await createRuleWithExceptionEntries(supertest, rule, [
[
{
field: 'keyword',
operator: 'included',
type: 'match',
value: 'word one',
},
],
]);
await waitForRuleSuccess(supertest, id);
await waitForSignalsToBePresent(supertest, 3, [id]);
const signalsOpen = await getSignalsById(supertest, id);
const hits = signalsOpen.hits.hits.map((hit) => hit._source.keyword).sort();
expect(hits).to.eql(['word four', 'word three', 'word two']);
});
});
```
### Changes for better determinism
To support more determinism there are changes and utilities added which can be tuned during any sporadic failures we might encounter as well as better support unexpected changes to other Elastic Stack pieces such as alerting, task manager, etc...
Get simple rule and others are now defaulting to false, meaning that the structural tests will no longer activate a rule and run it on task manger. This should cut down on error outputs as well as reduce stress and potentials for left over rules interfering with the runtime rules.
```ts
export const getSimpleRule = (ruleId = 'rule-1', enabled = false): QueryCreateSchema => ({
```
Not mandatory to use, but for most tests that should be runtime based tests, I use this function below which will enable it by default and run it using settings such as `type: 'query'`, `query: '*:*',` `from: '1900-01-01T00:00:00.000Z'`, to cut down on boiler plate noise. However, people can use whatever they want out of the grab bag or if their test is more readable to hand craft a REST request to create signals, or if they just want to call this and override where they want to, then 👍 .
```ts
export const getRuleForSignalTesting = (index: string[], ruleId = 'rule-1', enabled = true)
```
This waits for a rule to succeed before continuing
```ts
await waitForRuleSuccess(supertest, id);
```
I added a required array of id that _waits_ only for that particular id here. This is useful in case another test did not cleanup and you are getting signals being produced or left behind but need to wait specifically for yours.
```ts
await waitForSignalsToBePresent(supertest, 4, [id]);
```
I only get the signals for a particular rule id using either the auto-generated id or the rule_id. It's safer to use the ones from the auto-generated id but either of these are fine if you're careful enough.
```ts
const signalsOpen = await getSignalsById(supertest, id);
const signalsOpen = await getSignalsByIds(supertest, [createdId]);
const signalsOpen = await getSignalsByRuleIds(supertest, ['signal-on-signal']);
```
I delete all alerts now through a series of steps where it properly removes all rules using the rules bulk_delete and does it in such a way that all the API keys and alerting will be the best it can destroyed as well as double check that the alerts are showing up as being cleaned up before continuing.
```ts
deleteAllAlerts()
```
When not explicitly testing something structural, prefer to use the utilities which can and will do retries in case there are over the wire failures or es failures. Examples are:
```ts
installPrePackagedRules()
waitForRuleSuccess()
importFile() // This does a _lot_ of checks to ensure that the file is fully imported before continuing
```
Some of these utilities might still do a `expect(200);` but as we are and should use regular structural tests to cover those problems, these will probably be more and more removed when/if we hit test failures in favor of doing retries, waitFor, and countDowns.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2020-11-20 20:09:38 +01:00
numberOfSignals = 1 ,
signalIds : string [ ]
2020-10-20 03:14:09 +02:00
) : Promise < void > = > {
await waitFor ( async ( ) = > {
2021-02-23 23:16:48 +01:00
const signalsOpen = await getSignalsByIds ( supertest , signalIds , numberOfSignals ) ;
2020-10-20 18:40:00 +02:00
return signalsOpen . hits . hits . length >= numberOfSignals ;
[Security Solutions][Detection Engine] Adds e2e FTR runtime support and 213 tests for exception lists (#83764)
## Summary
Adds support to the end to end (e2e) functional test runner (FTR) support for rule runtime tests as well as 213 tests for the exception lists which include value based lists. Previously we had limited runtime support, but as I scaled up runtime tests from 5 to 200+ I noticed in a lot of areas we had to use improved techniques for determinism.
The runtime support being added is our next step of tests. Up to now most of our e2e FTR tests have been structural testing of REST and API integration tests. Basically up to now 95% tests are API structural as:
* Call REST input related to a rule such as GET/PUT/POST/PATCH/DELETE.
* Check REST output of the rule, did it match expected output body and status code?
* In some rare cases we check if the the rule can be executed and we get a status of 'succeeded'
With only a small part of our tests ~5%, `generating_signals.ts` was checking the signals being produced. However, we cannot have confidence in runtime based tests until the structural tests have been built up and run through the weeks against PR's to ensure that those are stable and deterministic.
Now that we have confidence and 90%+ coverage of the structural REST based tests, we are building up newer sets of tests which allow us to do runtime based validation tests to increase confidence that:
* Detection engine produces signals as expected
* Structure of the signals are as expected, including signal on signals
* Exceptions to signals are working as expected
* Most runtime bugs can be TDD'ed with e2e FTR's and regressions
* Whack-a-mole will not happen
* Consistency and predictability of signals is validated
* Refactoring can occur with stronger confidence
* Runtime tests are reference points for answering questions about existing bugs or adding new ones to test if users are experiencing unexpected behaviors
* Scaling tests can happen without failures
* Velocity for creating tests increases as the utilities and examples increase
Lastly, this puts us within striking distance of creating FTR's for different common class of runtime situations such as:
* Creating tests that exercise each rule against a set of data criteria and get signal hits
* Creating tests that validate the rule overrides operate as expected against data sets
* Creating tests that validate malfunctions, corner cases, or misuse cases such as data sets that are _all_ arrays or data sets that put numbers as strings or throws in an expected `null` instead of a value.
These tests follow the pattern of:
* Add the smallest data set to a folder in data.json (not gzip format)
* Add the smallest mapping to that folder (mapping.json)
* Call REST input related to exception lists, value lists, adding prepackaged rules, etc...
* Call REST input related endpoint with utilities to create and activate the rule
* Wait for the rule to go into the `succeeded` phase
* Wait for the N exact signals specific to that rule to be available
* Check against the set of signals to ensure that the matches are exactly as expected
Example of one runtime test:
A keyword data set is added to a folder called "keyword" but you can add one anywhere you want under `es_archives`, I just grouped mine depending on the situation of the runtime. Small non-gzipped tests `data.json` and `mappings.json` are the best approach for small focused tests. For _larger_ tests and cases I would and sometimes do use things such as auditbeat but try to avoid using larger data sets in favor of smaller focused test cases to validate the runtime is operating as expected.
```ts
{
"type": "doc",
"value": {
"id": "1",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:00:53.000Z",
"long": 1
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "2",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:01:53.000Z",
"long": 2
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "3",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:02:53.000Z",
"long": 3
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "4",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:03:53.000Z",
"long": 4
},
"type": "_doc"
}
}
```
Mapping is added. Note that this is "ECS tolerant" but not necessarily all ECS meaning I can and will try to keep things simple where I can, but I have ensured that `"@timestamp"` is at least there.
```ts
{
"type": "index",
"value": {
"index": "long",
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"long": { "type": "long" }
}
},
"settings": {
"index": {
"number_of_replicas": "1",
"number_of_shards": "1"
}
}
}
}
```
Test is written with test utilities where the `beforeEach` and `afterEach` try and clean up the indexes and load/unload the archives to keep one test from effecting another. Note this is never going to be 100% possible so see below on how we add more determinism in case something escapes the sandbox.
```ts
beforeEach(async () => {
await createSignalsIndex(supertest);
await createListsIndex(supertest);
await esArchiver.load('rule_exceptions/keyword');
});
afterEach(async () => {
await deleteSignalsIndex(supertest);
await deleteAllAlerts(supertest);
await deleteAllExceptions(es);
await deleteListsIndex(supertest);
await esArchiver.unload('rule_exceptions/keyword');
});
describe('"is" operator', () => {
it('should filter 1 single keyword if it is set as an exception', async () => {
const rule = getRuleForSignalTesting(['keyword']);
const { id } = await createRuleWithExceptionEntries(supertest, rule, [
[
{
field: 'keyword',
operator: 'included',
type: 'match',
value: 'word one',
},
],
]);
await waitForRuleSuccess(supertest, id);
await waitForSignalsToBePresent(supertest, 3, [id]);
const signalsOpen = await getSignalsById(supertest, id);
const hits = signalsOpen.hits.hits.map((hit) => hit._source.keyword).sort();
expect(hits).to.eql(['word four', 'word three', 'word two']);
});
});
```
### Changes for better determinism
To support more determinism there are changes and utilities added which can be tuned during any sporadic failures we might encounter as well as better support unexpected changes to other Elastic Stack pieces such as alerting, task manager, etc...
Get simple rule and others are now defaulting to false, meaning that the structural tests will no longer activate a rule and run it on task manger. This should cut down on error outputs as well as reduce stress and potentials for left over rules interfering with the runtime rules.
```ts
export const getSimpleRule = (ruleId = 'rule-1', enabled = false): QueryCreateSchema => ({
```
Not mandatory to use, but for most tests that should be runtime based tests, I use this function below which will enable it by default and run it using settings such as `type: 'query'`, `query: '*:*',` `from: '1900-01-01T00:00:00.000Z'`, to cut down on boiler plate noise. However, people can use whatever they want out of the grab bag or if their test is more readable to hand craft a REST request to create signals, or if they just want to call this and override where they want to, then 👍 .
```ts
export const getRuleForSignalTesting = (index: string[], ruleId = 'rule-1', enabled = true)
```
This waits for a rule to succeed before continuing
```ts
await waitForRuleSuccess(supertest, id);
```
I added a required array of id that _waits_ only for that particular id here. This is useful in case another test did not cleanup and you are getting signals being produced or left behind but need to wait specifically for yours.
```ts
await waitForSignalsToBePresent(supertest, 4, [id]);
```
I only get the signals for a particular rule id using either the auto-generated id or the rule_id. It's safer to use the ones from the auto-generated id but either of these are fine if you're careful enough.
```ts
const signalsOpen = await getSignalsById(supertest, id);
const signalsOpen = await getSignalsByIds(supertest, [createdId]);
const signalsOpen = await getSignalsByRuleIds(supertest, ['signal-on-signal']);
```
I delete all alerts now through a series of steps where it properly removes all rules using the rules bulk_delete and does it in such a way that all the API keys and alerting will be the best it can destroyed as well as double check that the alerts are showing up as being cleaned up before continuing.
```ts
deleteAllAlerts()
```
When not explicitly testing something structural, prefer to use the utilities which can and will do retries in case there are over the wire failures or es failures. Examples are:
```ts
installPrePackagedRules()
waitForRuleSuccess()
importFile() // This does a _lot_ of checks to ensure that the file is fully imported before continuing
```
Some of these utilities might still do a `expect(200);` but as we are and should use regular structural tests to cover those problems, these will probably be more and more removed when/if we hit test failures in favor of doing retries, waitFor, and countDowns.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2020-11-20 20:09:38 +01:00
} , 'waitForSignalsToBePresent' ) ;
2020-10-20 03:14:09 +02:00
} ;
/ * *
[Security Solutions][Detection Engine] Adds e2e FTR runtime support and 213 tests for exception lists (#83764)
## Summary
Adds support to the end to end (e2e) functional test runner (FTR) support for rule runtime tests as well as 213 tests for the exception lists which include value based lists. Previously we had limited runtime support, but as I scaled up runtime tests from 5 to 200+ I noticed in a lot of areas we had to use improved techniques for determinism.
The runtime support being added is our next step of tests. Up to now most of our e2e FTR tests have been structural testing of REST and API integration tests. Basically up to now 95% tests are API structural as:
* Call REST input related to a rule such as GET/PUT/POST/PATCH/DELETE.
* Check REST output of the rule, did it match expected output body and status code?
* In some rare cases we check if the the rule can be executed and we get a status of 'succeeded'
With only a small part of our tests ~5%, `generating_signals.ts` was checking the signals being produced. However, we cannot have confidence in runtime based tests until the structural tests have been built up and run through the weeks against PR's to ensure that those are stable and deterministic.
Now that we have confidence and 90%+ coverage of the structural REST based tests, we are building up newer sets of tests which allow us to do runtime based validation tests to increase confidence that:
* Detection engine produces signals as expected
* Structure of the signals are as expected, including signal on signals
* Exceptions to signals are working as expected
* Most runtime bugs can be TDD'ed with e2e FTR's and regressions
* Whack-a-mole will not happen
* Consistency and predictability of signals is validated
* Refactoring can occur with stronger confidence
* Runtime tests are reference points for answering questions about existing bugs or adding new ones to test if users are experiencing unexpected behaviors
* Scaling tests can happen without failures
* Velocity for creating tests increases as the utilities and examples increase
Lastly, this puts us within striking distance of creating FTR's for different common class of runtime situations such as:
* Creating tests that exercise each rule against a set of data criteria and get signal hits
* Creating tests that validate the rule overrides operate as expected against data sets
* Creating tests that validate malfunctions, corner cases, or misuse cases such as data sets that are _all_ arrays or data sets that put numbers as strings or throws in an expected `null` instead of a value.
These tests follow the pattern of:
* Add the smallest data set to a folder in data.json (not gzip format)
* Add the smallest mapping to that folder (mapping.json)
* Call REST input related to exception lists, value lists, adding prepackaged rules, etc...
* Call REST input related endpoint with utilities to create and activate the rule
* Wait for the rule to go into the `succeeded` phase
* Wait for the N exact signals specific to that rule to be available
* Check against the set of signals to ensure that the matches are exactly as expected
Example of one runtime test:
A keyword data set is added to a folder called "keyword" but you can add one anywhere you want under `es_archives`, I just grouped mine depending on the situation of the runtime. Small non-gzipped tests `data.json` and `mappings.json` are the best approach for small focused tests. For _larger_ tests and cases I would and sometimes do use things such as auditbeat but try to avoid using larger data sets in favor of smaller focused test cases to validate the runtime is operating as expected.
```ts
{
"type": "doc",
"value": {
"id": "1",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:00:53.000Z",
"long": 1
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "2",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:01:53.000Z",
"long": 2
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "3",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:02:53.000Z",
"long": 3
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "4",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:03:53.000Z",
"long": 4
},
"type": "_doc"
}
}
```
Mapping is added. Note that this is "ECS tolerant" but not necessarily all ECS meaning I can and will try to keep things simple where I can, but I have ensured that `"@timestamp"` is at least there.
```ts
{
"type": "index",
"value": {
"index": "long",
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"long": { "type": "long" }
}
},
"settings": {
"index": {
"number_of_replicas": "1",
"number_of_shards": "1"
}
}
}
}
```
Test is written with test utilities where the `beforeEach` and `afterEach` try and clean up the indexes and load/unload the archives to keep one test from effecting another. Note this is never going to be 100% possible so see below on how we add more determinism in case something escapes the sandbox.
```ts
beforeEach(async () => {
await createSignalsIndex(supertest);
await createListsIndex(supertest);
await esArchiver.load('rule_exceptions/keyword');
});
afterEach(async () => {
await deleteSignalsIndex(supertest);
await deleteAllAlerts(supertest);
await deleteAllExceptions(es);
await deleteListsIndex(supertest);
await esArchiver.unload('rule_exceptions/keyword');
});
describe('"is" operator', () => {
it('should filter 1 single keyword if it is set as an exception', async () => {
const rule = getRuleForSignalTesting(['keyword']);
const { id } = await createRuleWithExceptionEntries(supertest, rule, [
[
{
field: 'keyword',
operator: 'included',
type: 'match',
value: 'word one',
},
],
]);
await waitForRuleSuccess(supertest, id);
await waitForSignalsToBePresent(supertest, 3, [id]);
const signalsOpen = await getSignalsById(supertest, id);
const hits = signalsOpen.hits.hits.map((hit) => hit._source.keyword).sort();
expect(hits).to.eql(['word four', 'word three', 'word two']);
});
});
```
### Changes for better determinism
To support more determinism there are changes and utilities added which can be tuned during any sporadic failures we might encounter as well as better support unexpected changes to other Elastic Stack pieces such as alerting, task manager, etc...
Get simple rule and others are now defaulting to false, meaning that the structural tests will no longer activate a rule and run it on task manger. This should cut down on error outputs as well as reduce stress and potentials for left over rules interfering with the runtime rules.
```ts
export const getSimpleRule = (ruleId = 'rule-1', enabled = false): QueryCreateSchema => ({
```
Not mandatory to use, but for most tests that should be runtime based tests, I use this function below which will enable it by default and run it using settings such as `type: 'query'`, `query: '*:*',` `from: '1900-01-01T00:00:00.000Z'`, to cut down on boiler plate noise. However, people can use whatever they want out of the grab bag or if their test is more readable to hand craft a REST request to create signals, or if they just want to call this and override where they want to, then 👍 .
```ts
export const getRuleForSignalTesting = (index: string[], ruleId = 'rule-1', enabled = true)
```
This waits for a rule to succeed before continuing
```ts
await waitForRuleSuccess(supertest, id);
```
I added a required array of id that _waits_ only for that particular id here. This is useful in case another test did not cleanup and you are getting signals being produced or left behind but need to wait specifically for yours.
```ts
await waitForSignalsToBePresent(supertest, 4, [id]);
```
I only get the signals for a particular rule id using either the auto-generated id or the rule_id. It's safer to use the ones from the auto-generated id but either of these are fine if you're careful enough.
```ts
const signalsOpen = await getSignalsById(supertest, id);
const signalsOpen = await getSignalsByIds(supertest, [createdId]);
const signalsOpen = await getSignalsByRuleIds(supertest, ['signal-on-signal']);
```
I delete all alerts now through a series of steps where it properly removes all rules using the rules bulk_delete and does it in such a way that all the API keys and alerting will be the best it can destroyed as well as double check that the alerts are showing up as being cleaned up before continuing.
```ts
deleteAllAlerts()
```
When not explicitly testing something structural, prefer to use the utilities which can and will do retries in case there are over the wire failures or es failures. Examples are:
```ts
installPrePackagedRules()
waitForRuleSuccess()
importFile() // This does a _lot_ of checks to ensure that the file is fully imported before continuing
```
Some of these utilities might still do a `expect(200);` but as we are and should use regular structural tests to cover those problems, these will probably be more and more removed when/if we hit test failures in favor of doing retries, waitFor, and countDowns.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2020-11-20 20:09:38 +01:00
* Returns all signals both closed and opened by ruleId
2020-10-20 03:14:09 +02:00
* @param supertest Deps
* /
[Security Solutions][Detection Engine] Adds e2e FTR runtime support and 213 tests for exception lists (#83764)
## Summary
Adds support to the end to end (e2e) functional test runner (FTR) support for rule runtime tests as well as 213 tests for the exception lists which include value based lists. Previously we had limited runtime support, but as I scaled up runtime tests from 5 to 200+ I noticed in a lot of areas we had to use improved techniques for determinism.
The runtime support being added is our next step of tests. Up to now most of our e2e FTR tests have been structural testing of REST and API integration tests. Basically up to now 95% tests are API structural as:
* Call REST input related to a rule such as GET/PUT/POST/PATCH/DELETE.
* Check REST output of the rule, did it match expected output body and status code?
* In some rare cases we check if the the rule can be executed and we get a status of 'succeeded'
With only a small part of our tests ~5%, `generating_signals.ts` was checking the signals being produced. However, we cannot have confidence in runtime based tests until the structural tests have been built up and run through the weeks against PR's to ensure that those are stable and deterministic.
Now that we have confidence and 90%+ coverage of the structural REST based tests, we are building up newer sets of tests which allow us to do runtime based validation tests to increase confidence that:
* Detection engine produces signals as expected
* Structure of the signals are as expected, including signal on signals
* Exceptions to signals are working as expected
* Most runtime bugs can be TDD'ed with e2e FTR's and regressions
* Whack-a-mole will not happen
* Consistency and predictability of signals is validated
* Refactoring can occur with stronger confidence
* Runtime tests are reference points for answering questions about existing bugs or adding new ones to test if users are experiencing unexpected behaviors
* Scaling tests can happen without failures
* Velocity for creating tests increases as the utilities and examples increase
Lastly, this puts us within striking distance of creating FTR's for different common class of runtime situations such as:
* Creating tests that exercise each rule against a set of data criteria and get signal hits
* Creating tests that validate the rule overrides operate as expected against data sets
* Creating tests that validate malfunctions, corner cases, or misuse cases such as data sets that are _all_ arrays or data sets that put numbers as strings or throws in an expected `null` instead of a value.
These tests follow the pattern of:
* Add the smallest data set to a folder in data.json (not gzip format)
* Add the smallest mapping to that folder (mapping.json)
* Call REST input related to exception lists, value lists, adding prepackaged rules, etc...
* Call REST input related endpoint with utilities to create and activate the rule
* Wait for the rule to go into the `succeeded` phase
* Wait for the N exact signals specific to that rule to be available
* Check against the set of signals to ensure that the matches are exactly as expected
Example of one runtime test:
A keyword data set is added to a folder called "keyword" but you can add one anywhere you want under `es_archives`, I just grouped mine depending on the situation of the runtime. Small non-gzipped tests `data.json` and `mappings.json` are the best approach for small focused tests. For _larger_ tests and cases I would and sometimes do use things such as auditbeat but try to avoid using larger data sets in favor of smaller focused test cases to validate the runtime is operating as expected.
```ts
{
"type": "doc",
"value": {
"id": "1",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:00:53.000Z",
"long": 1
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "2",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:01:53.000Z",
"long": 2
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "3",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:02:53.000Z",
"long": 3
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "4",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:03:53.000Z",
"long": 4
},
"type": "_doc"
}
}
```
Mapping is added. Note that this is "ECS tolerant" but not necessarily all ECS meaning I can and will try to keep things simple where I can, but I have ensured that `"@timestamp"` is at least there.
```ts
{
"type": "index",
"value": {
"index": "long",
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"long": { "type": "long" }
}
},
"settings": {
"index": {
"number_of_replicas": "1",
"number_of_shards": "1"
}
}
}
}
```
Test is written with test utilities where the `beforeEach` and `afterEach` try and clean up the indexes and load/unload the archives to keep one test from effecting another. Note this is never going to be 100% possible so see below on how we add more determinism in case something escapes the sandbox.
```ts
beforeEach(async () => {
await createSignalsIndex(supertest);
await createListsIndex(supertest);
await esArchiver.load('rule_exceptions/keyword');
});
afterEach(async () => {
await deleteSignalsIndex(supertest);
await deleteAllAlerts(supertest);
await deleteAllExceptions(es);
await deleteListsIndex(supertest);
await esArchiver.unload('rule_exceptions/keyword');
});
describe('"is" operator', () => {
it('should filter 1 single keyword if it is set as an exception', async () => {
const rule = getRuleForSignalTesting(['keyword']);
const { id } = await createRuleWithExceptionEntries(supertest, rule, [
[
{
field: 'keyword',
operator: 'included',
type: 'match',
value: 'word one',
},
],
]);
await waitForRuleSuccess(supertest, id);
await waitForSignalsToBePresent(supertest, 3, [id]);
const signalsOpen = await getSignalsById(supertest, id);
const hits = signalsOpen.hits.hits.map((hit) => hit._source.keyword).sort();
expect(hits).to.eql(['word four', 'word three', 'word two']);
});
});
```
### Changes for better determinism
To support more determinism there are changes and utilities added which can be tuned during any sporadic failures we might encounter as well as better support unexpected changes to other Elastic Stack pieces such as alerting, task manager, etc...
Get simple rule and others are now defaulting to false, meaning that the structural tests will no longer activate a rule and run it on task manger. This should cut down on error outputs as well as reduce stress and potentials for left over rules interfering with the runtime rules.
```ts
export const getSimpleRule = (ruleId = 'rule-1', enabled = false): QueryCreateSchema => ({
```
Not mandatory to use, but for most tests that should be runtime based tests, I use this function below which will enable it by default and run it using settings such as `type: 'query'`, `query: '*:*',` `from: '1900-01-01T00:00:00.000Z'`, to cut down on boiler plate noise. However, people can use whatever they want out of the grab bag or if their test is more readable to hand craft a REST request to create signals, or if they just want to call this and override where they want to, then 👍 .
```ts
export const getRuleForSignalTesting = (index: string[], ruleId = 'rule-1', enabled = true)
```
This waits for a rule to succeed before continuing
```ts
await waitForRuleSuccess(supertest, id);
```
I added a required array of id that _waits_ only for that particular id here. This is useful in case another test did not cleanup and you are getting signals being produced or left behind but need to wait specifically for yours.
```ts
await waitForSignalsToBePresent(supertest, 4, [id]);
```
I only get the signals for a particular rule id using either the auto-generated id or the rule_id. It's safer to use the ones from the auto-generated id but either of these are fine if you're careful enough.
```ts
const signalsOpen = await getSignalsById(supertest, id);
const signalsOpen = await getSignalsByIds(supertest, [createdId]);
const signalsOpen = await getSignalsByRuleIds(supertest, ['signal-on-signal']);
```
I delete all alerts now through a series of steps where it properly removes all rules using the rules bulk_delete and does it in such a way that all the API keys and alerting will be the best it can destroyed as well as double check that the alerts are showing up as being cleaned up before continuing.
```ts
deleteAllAlerts()
```
When not explicitly testing something structural, prefer to use the utilities which can and will do retries in case there are over the wire failures or es failures. Examples are:
```ts
installPrePackagedRules()
waitForRuleSuccess()
importFile() // This does a _lot_ of checks to ensure that the file is fully imported before continuing
```
Some of these utilities might still do a `expect(200);` but as we are and should use regular structural tests to cover those problems, these will probably be more and more removed when/if we hit test failures in favor of doing retries, waitFor, and countDowns.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2020-11-20 20:09:38 +01:00
export const getSignalsByRuleIds = async (
supertest : SuperTest < supertestAsPromised.Test > ,
ruleIds : string [ ]
2020-10-20 03:14:09 +02:00
) : Promise <
SearchResponse < {
signal : Signal ;
[Security Solutions][Detection Engine] Adds e2e FTR runtime support and 213 tests for exception lists (#83764)
## Summary
Adds support to the end to end (e2e) functional test runner (FTR) support for rule runtime tests as well as 213 tests for the exception lists which include value based lists. Previously we had limited runtime support, but as I scaled up runtime tests from 5 to 200+ I noticed in a lot of areas we had to use improved techniques for determinism.
The runtime support being added is our next step of tests. Up to now most of our e2e FTR tests have been structural testing of REST and API integration tests. Basically up to now 95% tests are API structural as:
* Call REST input related to a rule such as GET/PUT/POST/PATCH/DELETE.
* Check REST output of the rule, did it match expected output body and status code?
* In some rare cases we check if the the rule can be executed and we get a status of 'succeeded'
With only a small part of our tests ~5%, `generating_signals.ts` was checking the signals being produced. However, we cannot have confidence in runtime based tests until the structural tests have been built up and run through the weeks against PR's to ensure that those are stable and deterministic.
Now that we have confidence and 90%+ coverage of the structural REST based tests, we are building up newer sets of tests which allow us to do runtime based validation tests to increase confidence that:
* Detection engine produces signals as expected
* Structure of the signals are as expected, including signal on signals
* Exceptions to signals are working as expected
* Most runtime bugs can be TDD'ed with e2e FTR's and regressions
* Whack-a-mole will not happen
* Consistency and predictability of signals is validated
* Refactoring can occur with stronger confidence
* Runtime tests are reference points for answering questions about existing bugs or adding new ones to test if users are experiencing unexpected behaviors
* Scaling tests can happen without failures
* Velocity for creating tests increases as the utilities and examples increase
Lastly, this puts us within striking distance of creating FTR's for different common class of runtime situations such as:
* Creating tests that exercise each rule against a set of data criteria and get signal hits
* Creating tests that validate the rule overrides operate as expected against data sets
* Creating tests that validate malfunctions, corner cases, or misuse cases such as data sets that are _all_ arrays or data sets that put numbers as strings or throws in an expected `null` instead of a value.
These tests follow the pattern of:
* Add the smallest data set to a folder in data.json (not gzip format)
* Add the smallest mapping to that folder (mapping.json)
* Call REST input related to exception lists, value lists, adding prepackaged rules, etc...
* Call REST input related endpoint with utilities to create and activate the rule
* Wait for the rule to go into the `succeeded` phase
* Wait for the N exact signals specific to that rule to be available
* Check against the set of signals to ensure that the matches are exactly as expected
Example of one runtime test:
A keyword data set is added to a folder called "keyword" but you can add one anywhere you want under `es_archives`, I just grouped mine depending on the situation of the runtime. Small non-gzipped tests `data.json` and `mappings.json` are the best approach for small focused tests. For _larger_ tests and cases I would and sometimes do use things such as auditbeat but try to avoid using larger data sets in favor of smaller focused test cases to validate the runtime is operating as expected.
```ts
{
"type": "doc",
"value": {
"id": "1",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:00:53.000Z",
"long": 1
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "2",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:01:53.000Z",
"long": 2
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "3",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:02:53.000Z",
"long": 3
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "4",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:03:53.000Z",
"long": 4
},
"type": "_doc"
}
}
```
Mapping is added. Note that this is "ECS tolerant" but not necessarily all ECS meaning I can and will try to keep things simple where I can, but I have ensured that `"@timestamp"` is at least there.
```ts
{
"type": "index",
"value": {
"index": "long",
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"long": { "type": "long" }
}
},
"settings": {
"index": {
"number_of_replicas": "1",
"number_of_shards": "1"
}
}
}
}
```
Test is written with test utilities where the `beforeEach` and `afterEach` try and clean up the indexes and load/unload the archives to keep one test from effecting another. Note this is never going to be 100% possible so see below on how we add more determinism in case something escapes the sandbox.
```ts
beforeEach(async () => {
await createSignalsIndex(supertest);
await createListsIndex(supertest);
await esArchiver.load('rule_exceptions/keyword');
});
afterEach(async () => {
await deleteSignalsIndex(supertest);
await deleteAllAlerts(supertest);
await deleteAllExceptions(es);
await deleteListsIndex(supertest);
await esArchiver.unload('rule_exceptions/keyword');
});
describe('"is" operator', () => {
it('should filter 1 single keyword if it is set as an exception', async () => {
const rule = getRuleForSignalTesting(['keyword']);
const { id } = await createRuleWithExceptionEntries(supertest, rule, [
[
{
field: 'keyword',
operator: 'included',
type: 'match',
value: 'word one',
},
],
]);
await waitForRuleSuccess(supertest, id);
await waitForSignalsToBePresent(supertest, 3, [id]);
const signalsOpen = await getSignalsById(supertest, id);
const hits = signalsOpen.hits.hits.map((hit) => hit._source.keyword).sort();
expect(hits).to.eql(['word four', 'word three', 'word two']);
});
});
```
### Changes for better determinism
To support more determinism there are changes and utilities added which can be tuned during any sporadic failures we might encounter as well as better support unexpected changes to other Elastic Stack pieces such as alerting, task manager, etc...
Get simple rule and others are now defaulting to false, meaning that the structural tests will no longer activate a rule and run it on task manger. This should cut down on error outputs as well as reduce stress and potentials for left over rules interfering with the runtime rules.
```ts
export const getSimpleRule = (ruleId = 'rule-1', enabled = false): QueryCreateSchema => ({
```
Not mandatory to use, but for most tests that should be runtime based tests, I use this function below which will enable it by default and run it using settings such as `type: 'query'`, `query: '*:*',` `from: '1900-01-01T00:00:00.000Z'`, to cut down on boiler plate noise. However, people can use whatever they want out of the grab bag or if their test is more readable to hand craft a REST request to create signals, or if they just want to call this and override where they want to, then 👍 .
```ts
export const getRuleForSignalTesting = (index: string[], ruleId = 'rule-1', enabled = true)
```
This waits for a rule to succeed before continuing
```ts
await waitForRuleSuccess(supertest, id);
```
I added a required array of id that _waits_ only for that particular id here. This is useful in case another test did not cleanup and you are getting signals being produced or left behind but need to wait specifically for yours.
```ts
await waitForSignalsToBePresent(supertest, 4, [id]);
```
I only get the signals for a particular rule id using either the auto-generated id or the rule_id. It's safer to use the ones from the auto-generated id but either of these are fine if you're careful enough.
```ts
const signalsOpen = await getSignalsById(supertest, id);
const signalsOpen = await getSignalsByIds(supertest, [createdId]);
const signalsOpen = await getSignalsByRuleIds(supertest, ['signal-on-signal']);
```
I delete all alerts now through a series of steps where it properly removes all rules using the rules bulk_delete and does it in such a way that all the API keys and alerting will be the best it can destroyed as well as double check that the alerts are showing up as being cleaned up before continuing.
```ts
deleteAllAlerts()
```
When not explicitly testing something structural, prefer to use the utilities which can and will do retries in case there are over the wire failures or es failures. Examples are:
```ts
installPrePackagedRules()
waitForRuleSuccess()
importFile() // This does a _lot_ of checks to ensure that the file is fully imported before continuing
```
Some of these utilities might still do a `expect(200);` but as we are and should use regular structural tests to cover those problems, these will probably be more and more removed when/if we hit test failures in favor of doing retries, waitFor, and countDowns.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2020-11-20 20:09:38 +01:00
[ x : string ] : unknown ;
2020-10-20 03:14:09 +02:00
} >
> = > {
const { body : signalsOpen } : { body : SearchResponse < { signal : Signal } > } = await supertest
. post ( DETECTION_ENGINE_QUERY_SIGNALS_URL )
. set ( 'kbn-xsrf' , 'true' )
[Security Solutions][Detection Engine] Adds e2e FTR runtime support and 213 tests for exception lists (#83764)
## Summary
Adds support to the end to end (e2e) functional test runner (FTR) support for rule runtime tests as well as 213 tests for the exception lists which include value based lists. Previously we had limited runtime support, but as I scaled up runtime tests from 5 to 200+ I noticed in a lot of areas we had to use improved techniques for determinism.
The runtime support being added is our next step of tests. Up to now most of our e2e FTR tests have been structural testing of REST and API integration tests. Basically up to now 95% tests are API structural as:
* Call REST input related to a rule such as GET/PUT/POST/PATCH/DELETE.
* Check REST output of the rule, did it match expected output body and status code?
* In some rare cases we check if the the rule can be executed and we get a status of 'succeeded'
With only a small part of our tests ~5%, `generating_signals.ts` was checking the signals being produced. However, we cannot have confidence in runtime based tests until the structural tests have been built up and run through the weeks against PR's to ensure that those are stable and deterministic.
Now that we have confidence and 90%+ coverage of the structural REST based tests, we are building up newer sets of tests which allow us to do runtime based validation tests to increase confidence that:
* Detection engine produces signals as expected
* Structure of the signals are as expected, including signal on signals
* Exceptions to signals are working as expected
* Most runtime bugs can be TDD'ed with e2e FTR's and regressions
* Whack-a-mole will not happen
* Consistency and predictability of signals is validated
* Refactoring can occur with stronger confidence
* Runtime tests are reference points for answering questions about existing bugs or adding new ones to test if users are experiencing unexpected behaviors
* Scaling tests can happen without failures
* Velocity for creating tests increases as the utilities and examples increase
Lastly, this puts us within striking distance of creating FTR's for different common class of runtime situations such as:
* Creating tests that exercise each rule against a set of data criteria and get signal hits
* Creating tests that validate the rule overrides operate as expected against data sets
* Creating tests that validate malfunctions, corner cases, or misuse cases such as data sets that are _all_ arrays or data sets that put numbers as strings or throws in an expected `null` instead of a value.
These tests follow the pattern of:
* Add the smallest data set to a folder in data.json (not gzip format)
* Add the smallest mapping to that folder (mapping.json)
* Call REST input related to exception lists, value lists, adding prepackaged rules, etc...
* Call REST input related endpoint with utilities to create and activate the rule
* Wait for the rule to go into the `succeeded` phase
* Wait for the N exact signals specific to that rule to be available
* Check against the set of signals to ensure that the matches are exactly as expected
Example of one runtime test:
A keyword data set is added to a folder called "keyword" but you can add one anywhere you want under `es_archives`, I just grouped mine depending on the situation of the runtime. Small non-gzipped tests `data.json` and `mappings.json` are the best approach for small focused tests. For _larger_ tests and cases I would and sometimes do use things such as auditbeat but try to avoid using larger data sets in favor of smaller focused test cases to validate the runtime is operating as expected.
```ts
{
"type": "doc",
"value": {
"id": "1",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:00:53.000Z",
"long": 1
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "2",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:01:53.000Z",
"long": 2
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "3",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:02:53.000Z",
"long": 3
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "4",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:03:53.000Z",
"long": 4
},
"type": "_doc"
}
}
```
Mapping is added. Note that this is "ECS tolerant" but not necessarily all ECS meaning I can and will try to keep things simple where I can, but I have ensured that `"@timestamp"` is at least there.
```ts
{
"type": "index",
"value": {
"index": "long",
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"long": { "type": "long" }
}
},
"settings": {
"index": {
"number_of_replicas": "1",
"number_of_shards": "1"
}
}
}
}
```
Test is written with test utilities where the `beforeEach` and `afterEach` try and clean up the indexes and load/unload the archives to keep one test from effecting another. Note this is never going to be 100% possible so see below on how we add more determinism in case something escapes the sandbox.
```ts
beforeEach(async () => {
await createSignalsIndex(supertest);
await createListsIndex(supertest);
await esArchiver.load('rule_exceptions/keyword');
});
afterEach(async () => {
await deleteSignalsIndex(supertest);
await deleteAllAlerts(supertest);
await deleteAllExceptions(es);
await deleteListsIndex(supertest);
await esArchiver.unload('rule_exceptions/keyword');
});
describe('"is" operator', () => {
it('should filter 1 single keyword if it is set as an exception', async () => {
const rule = getRuleForSignalTesting(['keyword']);
const { id } = await createRuleWithExceptionEntries(supertest, rule, [
[
{
field: 'keyword',
operator: 'included',
type: 'match',
value: 'word one',
},
],
]);
await waitForRuleSuccess(supertest, id);
await waitForSignalsToBePresent(supertest, 3, [id]);
const signalsOpen = await getSignalsById(supertest, id);
const hits = signalsOpen.hits.hits.map((hit) => hit._source.keyword).sort();
expect(hits).to.eql(['word four', 'word three', 'word two']);
});
});
```
### Changes for better determinism
To support more determinism there are changes and utilities added which can be tuned during any sporadic failures we might encounter as well as better support unexpected changes to other Elastic Stack pieces such as alerting, task manager, etc...
Get simple rule and others are now defaulting to false, meaning that the structural tests will no longer activate a rule and run it on task manger. This should cut down on error outputs as well as reduce stress and potentials for left over rules interfering with the runtime rules.
```ts
export const getSimpleRule = (ruleId = 'rule-1', enabled = false): QueryCreateSchema => ({
```
Not mandatory to use, but for most tests that should be runtime based tests, I use this function below which will enable it by default and run it using settings such as `type: 'query'`, `query: '*:*',` `from: '1900-01-01T00:00:00.000Z'`, to cut down on boiler plate noise. However, people can use whatever they want out of the grab bag or if their test is more readable to hand craft a REST request to create signals, or if they just want to call this and override where they want to, then 👍 .
```ts
export const getRuleForSignalTesting = (index: string[], ruleId = 'rule-1', enabled = true)
```
This waits for a rule to succeed before continuing
```ts
await waitForRuleSuccess(supertest, id);
```
I added a required array of id that _waits_ only for that particular id here. This is useful in case another test did not cleanup and you are getting signals being produced or left behind but need to wait specifically for yours.
```ts
await waitForSignalsToBePresent(supertest, 4, [id]);
```
I only get the signals for a particular rule id using either the auto-generated id or the rule_id. It's safer to use the ones from the auto-generated id but either of these are fine if you're careful enough.
```ts
const signalsOpen = await getSignalsById(supertest, id);
const signalsOpen = await getSignalsByIds(supertest, [createdId]);
const signalsOpen = await getSignalsByRuleIds(supertest, ['signal-on-signal']);
```
I delete all alerts now through a series of steps where it properly removes all rules using the rules bulk_delete and does it in such a way that all the API keys and alerting will be the best it can destroyed as well as double check that the alerts are showing up as being cleaned up before continuing.
```ts
deleteAllAlerts()
```
When not explicitly testing something structural, prefer to use the utilities which can and will do retries in case there are over the wire failures or es failures. Examples are:
```ts
installPrePackagedRules()
waitForRuleSuccess()
importFile() // This does a _lot_ of checks to ensure that the file is fully imported before continuing
```
Some of these utilities might still do a `expect(200);` but as we are and should use regular structural tests to cover those problems, these will probably be more and more removed when/if we hit test failures in favor of doing retries, waitFor, and countDowns.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2020-11-20 20:09:38 +01:00
. send ( getQuerySignalsRuleId ( ruleIds ) )
2020-10-20 03:14:09 +02:00
. expect ( 200 ) ;
return signalsOpen ;
} ;
[Security Solutions][Detection Engine] Adds e2e FTR runtime support and 213 tests for exception lists (#83764)
## Summary
Adds support to the end to end (e2e) functional test runner (FTR) support for rule runtime tests as well as 213 tests for the exception lists which include value based lists. Previously we had limited runtime support, but as I scaled up runtime tests from 5 to 200+ I noticed in a lot of areas we had to use improved techniques for determinism.
The runtime support being added is our next step of tests. Up to now most of our e2e FTR tests have been structural testing of REST and API integration tests. Basically up to now 95% tests are API structural as:
* Call REST input related to a rule such as GET/PUT/POST/PATCH/DELETE.
* Check REST output of the rule, did it match expected output body and status code?
* In some rare cases we check if the the rule can be executed and we get a status of 'succeeded'
With only a small part of our tests ~5%, `generating_signals.ts` was checking the signals being produced. However, we cannot have confidence in runtime based tests until the structural tests have been built up and run through the weeks against PR's to ensure that those are stable and deterministic.
Now that we have confidence and 90%+ coverage of the structural REST based tests, we are building up newer sets of tests which allow us to do runtime based validation tests to increase confidence that:
* Detection engine produces signals as expected
* Structure of the signals are as expected, including signal on signals
* Exceptions to signals are working as expected
* Most runtime bugs can be TDD'ed with e2e FTR's and regressions
* Whack-a-mole will not happen
* Consistency and predictability of signals is validated
* Refactoring can occur with stronger confidence
* Runtime tests are reference points for answering questions about existing bugs or adding new ones to test if users are experiencing unexpected behaviors
* Scaling tests can happen without failures
* Velocity for creating tests increases as the utilities and examples increase
Lastly, this puts us within striking distance of creating FTR's for different common class of runtime situations such as:
* Creating tests that exercise each rule against a set of data criteria and get signal hits
* Creating tests that validate the rule overrides operate as expected against data sets
* Creating tests that validate malfunctions, corner cases, or misuse cases such as data sets that are _all_ arrays or data sets that put numbers as strings or throws in an expected `null` instead of a value.
These tests follow the pattern of:
* Add the smallest data set to a folder in data.json (not gzip format)
* Add the smallest mapping to that folder (mapping.json)
* Call REST input related to exception lists, value lists, adding prepackaged rules, etc...
* Call REST input related endpoint with utilities to create and activate the rule
* Wait for the rule to go into the `succeeded` phase
* Wait for the N exact signals specific to that rule to be available
* Check against the set of signals to ensure that the matches are exactly as expected
Example of one runtime test:
A keyword data set is added to a folder called "keyword" but you can add one anywhere you want under `es_archives`, I just grouped mine depending on the situation of the runtime. Small non-gzipped tests `data.json` and `mappings.json` are the best approach for small focused tests. For _larger_ tests and cases I would and sometimes do use things such as auditbeat but try to avoid using larger data sets in favor of smaller focused test cases to validate the runtime is operating as expected.
```ts
{
"type": "doc",
"value": {
"id": "1",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:00:53.000Z",
"long": 1
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "2",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:01:53.000Z",
"long": 2
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "3",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:02:53.000Z",
"long": 3
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "4",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:03:53.000Z",
"long": 4
},
"type": "_doc"
}
}
```
Mapping is added. Note that this is "ECS tolerant" but not necessarily all ECS meaning I can and will try to keep things simple where I can, but I have ensured that `"@timestamp"` is at least there.
```ts
{
"type": "index",
"value": {
"index": "long",
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"long": { "type": "long" }
}
},
"settings": {
"index": {
"number_of_replicas": "1",
"number_of_shards": "1"
}
}
}
}
```
Test is written with test utilities where the `beforeEach` and `afterEach` try and clean up the indexes and load/unload the archives to keep one test from effecting another. Note this is never going to be 100% possible so see below on how we add more determinism in case something escapes the sandbox.
```ts
beforeEach(async () => {
await createSignalsIndex(supertest);
await createListsIndex(supertest);
await esArchiver.load('rule_exceptions/keyword');
});
afterEach(async () => {
await deleteSignalsIndex(supertest);
await deleteAllAlerts(supertest);
await deleteAllExceptions(es);
await deleteListsIndex(supertest);
await esArchiver.unload('rule_exceptions/keyword');
});
describe('"is" operator', () => {
it('should filter 1 single keyword if it is set as an exception', async () => {
const rule = getRuleForSignalTesting(['keyword']);
const { id } = await createRuleWithExceptionEntries(supertest, rule, [
[
{
field: 'keyword',
operator: 'included',
type: 'match',
value: 'word one',
},
],
]);
await waitForRuleSuccess(supertest, id);
await waitForSignalsToBePresent(supertest, 3, [id]);
const signalsOpen = await getSignalsById(supertest, id);
const hits = signalsOpen.hits.hits.map((hit) => hit._source.keyword).sort();
expect(hits).to.eql(['word four', 'word three', 'word two']);
});
});
```
### Changes for better determinism
To support more determinism there are changes and utilities added which can be tuned during any sporadic failures we might encounter as well as better support unexpected changes to other Elastic Stack pieces such as alerting, task manager, etc...
Get simple rule and others are now defaulting to false, meaning that the structural tests will no longer activate a rule and run it on task manger. This should cut down on error outputs as well as reduce stress and potentials for left over rules interfering with the runtime rules.
```ts
export const getSimpleRule = (ruleId = 'rule-1', enabled = false): QueryCreateSchema => ({
```
Not mandatory to use, but for most tests that should be runtime based tests, I use this function below which will enable it by default and run it using settings such as `type: 'query'`, `query: '*:*',` `from: '1900-01-01T00:00:00.000Z'`, to cut down on boiler plate noise. However, people can use whatever they want out of the grab bag or if their test is more readable to hand craft a REST request to create signals, or if they just want to call this and override where they want to, then 👍 .
```ts
export const getRuleForSignalTesting = (index: string[], ruleId = 'rule-1', enabled = true)
```
This waits for a rule to succeed before continuing
```ts
await waitForRuleSuccess(supertest, id);
```
I added a required array of id that _waits_ only for that particular id here. This is useful in case another test did not cleanup and you are getting signals being produced or left behind but need to wait specifically for yours.
```ts
await waitForSignalsToBePresent(supertest, 4, [id]);
```
I only get the signals for a particular rule id using either the auto-generated id or the rule_id. It's safer to use the ones from the auto-generated id but either of these are fine if you're careful enough.
```ts
const signalsOpen = await getSignalsById(supertest, id);
const signalsOpen = await getSignalsByIds(supertest, [createdId]);
const signalsOpen = await getSignalsByRuleIds(supertest, ['signal-on-signal']);
```
I delete all alerts now through a series of steps where it properly removes all rules using the rules bulk_delete and does it in such a way that all the API keys and alerting will be the best it can destroyed as well as double check that the alerts are showing up as being cleaned up before continuing.
```ts
deleteAllAlerts()
```
When not explicitly testing something structural, prefer to use the utilities which can and will do retries in case there are over the wire failures or es failures. Examples are:
```ts
installPrePackagedRules()
waitForRuleSuccess()
importFile() // This does a _lot_ of checks to ensure that the file is fully imported before continuing
```
Some of these utilities might still do a `expect(200);` but as we are and should use regular structural tests to cover those problems, these will probably be more and more removed when/if we hit test failures in favor of doing retries, waitFor, and countDowns.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2020-11-20 20:09:38 +01:00
/ * *
* Given an array of rule ids this will return only signals based on that rule id both
* open and closed
* @param supertest agent
* @param ids Array of the rule ids
* /
export const getSignalsByIds = async (
[Security Solutions][Detection Engine] Fixes critical clashing with source indexes that already contain a "signal" field (#82191)
## Summary
Fixes: https://github.com/elastic/kibana/issues/82148
We have errors and do not generate a signal when a source index already has utilized and reserved the "signal" field for their own data purposes. This fix is a bit tricky and has one medium sized risk which is we also support "signals generated on top of existing signals". Therefore we have to be careful and do a small runtime detection of the "data shape" of the signal's data type. If it looks like the user is using the "signal" field within their mapping instead of us, we move the customer's signal into "original_signal" inside our "signal" structure we create when we copy their data set when creating a signal.
To help mitigate the risks associated with this critical bug with regards to breaking signals on top of signals I have:
* This adds unit tests
* This adds end to end tests for testing generating signals including signals on signals to help mitigate risk
The key test for this shape in the PR are in the file:
```
detection_engine/signals/build_event_type_signal.ts
```
like so:
```ts
export const isEventTypeSignal = (doc: BaseSignalHit): boolean => {
return doc._source.signal?.rule?.id != null && typeof doc._source.signal?.rule?.id === 'string';
};
```
Example of what happens when it does a "move" of an existing numeric signal keyword type:
```ts
# This causes a clash with us using the name signal as a numeric.
PUT clashing-index/_doc/1
{
"@timestamp": "2020-10-28T05:08:53.000Z",
"signal": 1
}
```
Before, this was an error. With this PR it now will restructure this data like so when creating a signal along with additional signal ancestor information, meta data. I omitted some of the data from the output signal for this example.
```ts
{
... Other data copied ...
"signal":
{
"original_signal": 1 <--- We "move it" here now
"parents":
[
{
"id": "BhbXBmkBR346wHgn4PeZ",
"type": "event",
"index": "your-index-name",
"depth": 0
},
],
"ancestors":
[
{
"id": "BhbXBmkBR346wHgn4PeZ",
"type": "event",
"index": "your-index-name",
"depth": 0
},
],
"status": "open",
"depth": 1,
"parent":
{
"id": "BhbXBmkBR346wHgn4PeZ",
type: "event",
"index": "your-index-name",
"depth": 0
},
"original_time": "2019-02-19T17:40:03.790Z",
"original_event":
{
"action": "socket_closed",
"dataset": "socket",
"kind": "event",
"module": "system"
},
}
```
### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2020-11-06 15:47:57 +01:00
supertest : SuperTest < supertestAsPromised.Test > ,
2021-02-23 23:16:48 +01:00
ids : string [ ] ,
size? : number
[Security Solutions][Detection Engine] Fixes critical clashing with source indexes that already contain a "signal" field (#82191)
## Summary
Fixes: https://github.com/elastic/kibana/issues/82148
We have errors and do not generate a signal when a source index already has utilized and reserved the "signal" field for their own data purposes. This fix is a bit tricky and has one medium sized risk which is we also support "signals generated on top of existing signals". Therefore we have to be careful and do a small runtime detection of the "data shape" of the signal's data type. If it looks like the user is using the "signal" field within their mapping instead of us, we move the customer's signal into "original_signal" inside our "signal" structure we create when we copy their data set when creating a signal.
To help mitigate the risks associated with this critical bug with regards to breaking signals on top of signals I have:
* This adds unit tests
* This adds end to end tests for testing generating signals including signals on signals to help mitigate risk
The key test for this shape in the PR are in the file:
```
detection_engine/signals/build_event_type_signal.ts
```
like so:
```ts
export const isEventTypeSignal = (doc: BaseSignalHit): boolean => {
return doc._source.signal?.rule?.id != null && typeof doc._source.signal?.rule?.id === 'string';
};
```
Example of what happens when it does a "move" of an existing numeric signal keyword type:
```ts
# This causes a clash with us using the name signal as a numeric.
PUT clashing-index/_doc/1
{
"@timestamp": "2020-10-28T05:08:53.000Z",
"signal": 1
}
```
Before, this was an error. With this PR it now will restructure this data like so when creating a signal along with additional signal ancestor information, meta data. I omitted some of the data from the output signal for this example.
```ts
{
... Other data copied ...
"signal":
{
"original_signal": 1 <--- We "move it" here now
"parents":
[
{
"id": "BhbXBmkBR346wHgn4PeZ",
"type": "event",
"index": "your-index-name",
"depth": 0
},
],
"ancestors":
[
{
"id": "BhbXBmkBR346wHgn4PeZ",
"type": "event",
"index": "your-index-name",
"depth": 0
},
],
"status": "open",
"depth": 1,
"parent":
{
"id": "BhbXBmkBR346wHgn4PeZ",
type: "event",
"index": "your-index-name",
"depth": 0
},
"original_time": "2019-02-19T17:40:03.790Z",
"original_event":
{
"action": "socket_closed",
"dataset": "socket",
"kind": "event",
"module": "system"
},
}
```
### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2020-11-06 15:47:57 +01:00
) : Promise <
SearchResponse < {
signal : Signal ;
[Security Solutions][Detection Engine] Adds e2e FTR runtime support and 213 tests for exception lists (#83764)
## Summary
Adds support to the end to end (e2e) functional test runner (FTR) support for rule runtime tests as well as 213 tests for the exception lists which include value based lists. Previously we had limited runtime support, but as I scaled up runtime tests from 5 to 200+ I noticed in a lot of areas we had to use improved techniques for determinism.
The runtime support being added is our next step of tests. Up to now most of our e2e FTR tests have been structural testing of REST and API integration tests. Basically up to now 95% tests are API structural as:
* Call REST input related to a rule such as GET/PUT/POST/PATCH/DELETE.
* Check REST output of the rule, did it match expected output body and status code?
* In some rare cases we check if the the rule can be executed and we get a status of 'succeeded'
With only a small part of our tests ~5%, `generating_signals.ts` was checking the signals being produced. However, we cannot have confidence in runtime based tests until the structural tests have been built up and run through the weeks against PR's to ensure that those are stable and deterministic.
Now that we have confidence and 90%+ coverage of the structural REST based tests, we are building up newer sets of tests which allow us to do runtime based validation tests to increase confidence that:
* Detection engine produces signals as expected
* Structure of the signals are as expected, including signal on signals
* Exceptions to signals are working as expected
* Most runtime bugs can be TDD'ed with e2e FTR's and regressions
* Whack-a-mole will not happen
* Consistency and predictability of signals is validated
* Refactoring can occur with stronger confidence
* Runtime tests are reference points for answering questions about existing bugs or adding new ones to test if users are experiencing unexpected behaviors
* Scaling tests can happen without failures
* Velocity for creating tests increases as the utilities and examples increase
Lastly, this puts us within striking distance of creating FTR's for different common class of runtime situations such as:
* Creating tests that exercise each rule against a set of data criteria and get signal hits
* Creating tests that validate the rule overrides operate as expected against data sets
* Creating tests that validate malfunctions, corner cases, or misuse cases such as data sets that are _all_ arrays or data sets that put numbers as strings or throws in an expected `null` instead of a value.
These tests follow the pattern of:
* Add the smallest data set to a folder in data.json (not gzip format)
* Add the smallest mapping to that folder (mapping.json)
* Call REST input related to exception lists, value lists, adding prepackaged rules, etc...
* Call REST input related endpoint with utilities to create and activate the rule
* Wait for the rule to go into the `succeeded` phase
* Wait for the N exact signals specific to that rule to be available
* Check against the set of signals to ensure that the matches are exactly as expected
Example of one runtime test:
A keyword data set is added to a folder called "keyword" but you can add one anywhere you want under `es_archives`, I just grouped mine depending on the situation of the runtime. Small non-gzipped tests `data.json` and `mappings.json` are the best approach for small focused tests. For _larger_ tests and cases I would and sometimes do use things such as auditbeat but try to avoid using larger data sets in favor of smaller focused test cases to validate the runtime is operating as expected.
```ts
{
"type": "doc",
"value": {
"id": "1",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:00:53.000Z",
"long": 1
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "2",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:01:53.000Z",
"long": 2
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "3",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:02:53.000Z",
"long": 3
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "4",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:03:53.000Z",
"long": 4
},
"type": "_doc"
}
}
```
Mapping is added. Note that this is "ECS tolerant" but not necessarily all ECS meaning I can and will try to keep things simple where I can, but I have ensured that `"@timestamp"` is at least there.
```ts
{
"type": "index",
"value": {
"index": "long",
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"long": { "type": "long" }
}
},
"settings": {
"index": {
"number_of_replicas": "1",
"number_of_shards": "1"
}
}
}
}
```
Test is written with test utilities where the `beforeEach` and `afterEach` try and clean up the indexes and load/unload the archives to keep one test from effecting another. Note this is never going to be 100% possible so see below on how we add more determinism in case something escapes the sandbox.
```ts
beforeEach(async () => {
await createSignalsIndex(supertest);
await createListsIndex(supertest);
await esArchiver.load('rule_exceptions/keyword');
});
afterEach(async () => {
await deleteSignalsIndex(supertest);
await deleteAllAlerts(supertest);
await deleteAllExceptions(es);
await deleteListsIndex(supertest);
await esArchiver.unload('rule_exceptions/keyword');
});
describe('"is" operator', () => {
it('should filter 1 single keyword if it is set as an exception', async () => {
const rule = getRuleForSignalTesting(['keyword']);
const { id } = await createRuleWithExceptionEntries(supertest, rule, [
[
{
field: 'keyword',
operator: 'included',
type: 'match',
value: 'word one',
},
],
]);
await waitForRuleSuccess(supertest, id);
await waitForSignalsToBePresent(supertest, 3, [id]);
const signalsOpen = await getSignalsById(supertest, id);
const hits = signalsOpen.hits.hits.map((hit) => hit._source.keyword).sort();
expect(hits).to.eql(['word four', 'word three', 'word two']);
});
});
```
### Changes for better determinism
To support more determinism there are changes and utilities added which can be tuned during any sporadic failures we might encounter as well as better support unexpected changes to other Elastic Stack pieces such as alerting, task manager, etc...
Get simple rule and others are now defaulting to false, meaning that the structural tests will no longer activate a rule and run it on task manger. This should cut down on error outputs as well as reduce stress and potentials for left over rules interfering with the runtime rules.
```ts
export const getSimpleRule = (ruleId = 'rule-1', enabled = false): QueryCreateSchema => ({
```
Not mandatory to use, but for most tests that should be runtime based tests, I use this function below which will enable it by default and run it using settings such as `type: 'query'`, `query: '*:*',` `from: '1900-01-01T00:00:00.000Z'`, to cut down on boiler plate noise. However, people can use whatever they want out of the grab bag or if their test is more readable to hand craft a REST request to create signals, or if they just want to call this and override where they want to, then 👍 .
```ts
export const getRuleForSignalTesting = (index: string[], ruleId = 'rule-1', enabled = true)
```
This waits for a rule to succeed before continuing
```ts
await waitForRuleSuccess(supertest, id);
```
I added a required array of id that _waits_ only for that particular id here. This is useful in case another test did not cleanup and you are getting signals being produced or left behind but need to wait specifically for yours.
```ts
await waitForSignalsToBePresent(supertest, 4, [id]);
```
I only get the signals for a particular rule id using either the auto-generated id or the rule_id. It's safer to use the ones from the auto-generated id but either of these are fine if you're careful enough.
```ts
const signalsOpen = await getSignalsById(supertest, id);
const signalsOpen = await getSignalsByIds(supertest, [createdId]);
const signalsOpen = await getSignalsByRuleIds(supertest, ['signal-on-signal']);
```
I delete all alerts now through a series of steps where it properly removes all rules using the rules bulk_delete and does it in such a way that all the API keys and alerting will be the best it can destroyed as well as double check that the alerts are showing up as being cleaned up before continuing.
```ts
deleteAllAlerts()
```
When not explicitly testing something structural, prefer to use the utilities which can and will do retries in case there are over the wire failures or es failures. Examples are:
```ts
installPrePackagedRules()
waitForRuleSuccess()
importFile() // This does a _lot_ of checks to ensure that the file is fully imported before continuing
```
Some of these utilities might still do a `expect(200);` but as we are and should use regular structural tests to cover those problems, these will probably be more and more removed when/if we hit test failures in favor of doing retries, waitFor, and countDowns.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2020-11-20 20:09:38 +01:00
[ x : string ] : unknown ;
[Security Solutions][Detection Engine] Fixes critical clashing with source indexes that already contain a "signal" field (#82191)
## Summary
Fixes: https://github.com/elastic/kibana/issues/82148
We have errors and do not generate a signal when a source index already has utilized and reserved the "signal" field for their own data purposes. This fix is a bit tricky and has one medium sized risk which is we also support "signals generated on top of existing signals". Therefore we have to be careful and do a small runtime detection of the "data shape" of the signal's data type. If it looks like the user is using the "signal" field within their mapping instead of us, we move the customer's signal into "original_signal" inside our "signal" structure we create when we copy their data set when creating a signal.
To help mitigate the risks associated with this critical bug with regards to breaking signals on top of signals I have:
* This adds unit tests
* This adds end to end tests for testing generating signals including signals on signals to help mitigate risk
The key test for this shape in the PR are in the file:
```
detection_engine/signals/build_event_type_signal.ts
```
like so:
```ts
export const isEventTypeSignal = (doc: BaseSignalHit): boolean => {
return doc._source.signal?.rule?.id != null && typeof doc._source.signal?.rule?.id === 'string';
};
```
Example of what happens when it does a "move" of an existing numeric signal keyword type:
```ts
# This causes a clash with us using the name signal as a numeric.
PUT clashing-index/_doc/1
{
"@timestamp": "2020-10-28T05:08:53.000Z",
"signal": 1
}
```
Before, this was an error. With this PR it now will restructure this data like so when creating a signal along with additional signal ancestor information, meta data. I omitted some of the data from the output signal for this example.
```ts
{
... Other data copied ...
"signal":
{
"original_signal": 1 <--- We "move it" here now
"parents":
[
{
"id": "BhbXBmkBR346wHgn4PeZ",
"type": "event",
"index": "your-index-name",
"depth": 0
},
],
"ancestors":
[
{
"id": "BhbXBmkBR346wHgn4PeZ",
"type": "event",
"index": "your-index-name",
"depth": 0
},
],
"status": "open",
"depth": 1,
"parent":
{
"id": "BhbXBmkBR346wHgn4PeZ",
type: "event",
"index": "your-index-name",
"depth": 0
},
"original_time": "2019-02-19T17:40:03.790Z",
"original_event":
{
"action": "socket_closed",
"dataset": "socket",
"kind": "event",
"module": "system"
},
}
```
### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2020-11-06 15:47:57 +01:00
} >
> = > {
const { body : signalsOpen } : { body : SearchResponse < { signal : Signal } > } = await supertest
. post ( DETECTION_ENGINE_QUERY_SIGNALS_URL )
. set ( 'kbn-xsrf' , 'true' )
2021-02-23 23:16:48 +01:00
. send ( getQuerySignalsId ( ids , size ) )
[Security Solutions][Detection Engine] Adds e2e FTR runtime support and 213 tests for exception lists (#83764)
## Summary
Adds support to the end to end (e2e) functional test runner (FTR) support for rule runtime tests as well as 213 tests for the exception lists which include value based lists. Previously we had limited runtime support, but as I scaled up runtime tests from 5 to 200+ I noticed in a lot of areas we had to use improved techniques for determinism.
The runtime support being added is our next step of tests. Up to now most of our e2e FTR tests have been structural testing of REST and API integration tests. Basically up to now 95% tests are API structural as:
* Call REST input related to a rule such as GET/PUT/POST/PATCH/DELETE.
* Check REST output of the rule, did it match expected output body and status code?
* In some rare cases we check if the the rule can be executed and we get a status of 'succeeded'
With only a small part of our tests ~5%, `generating_signals.ts` was checking the signals being produced. However, we cannot have confidence in runtime based tests until the structural tests have been built up and run through the weeks against PR's to ensure that those are stable and deterministic.
Now that we have confidence and 90%+ coverage of the structural REST based tests, we are building up newer sets of tests which allow us to do runtime based validation tests to increase confidence that:
* Detection engine produces signals as expected
* Structure of the signals are as expected, including signal on signals
* Exceptions to signals are working as expected
* Most runtime bugs can be TDD'ed with e2e FTR's and regressions
* Whack-a-mole will not happen
* Consistency and predictability of signals is validated
* Refactoring can occur with stronger confidence
* Runtime tests are reference points for answering questions about existing bugs or adding new ones to test if users are experiencing unexpected behaviors
* Scaling tests can happen without failures
* Velocity for creating tests increases as the utilities and examples increase
Lastly, this puts us within striking distance of creating FTR's for different common class of runtime situations such as:
* Creating tests that exercise each rule against a set of data criteria and get signal hits
* Creating tests that validate the rule overrides operate as expected against data sets
* Creating tests that validate malfunctions, corner cases, or misuse cases such as data sets that are _all_ arrays or data sets that put numbers as strings or throws in an expected `null` instead of a value.
These tests follow the pattern of:
* Add the smallest data set to a folder in data.json (not gzip format)
* Add the smallest mapping to that folder (mapping.json)
* Call REST input related to exception lists, value lists, adding prepackaged rules, etc...
* Call REST input related endpoint with utilities to create and activate the rule
* Wait for the rule to go into the `succeeded` phase
* Wait for the N exact signals specific to that rule to be available
* Check against the set of signals to ensure that the matches are exactly as expected
Example of one runtime test:
A keyword data set is added to a folder called "keyword" but you can add one anywhere you want under `es_archives`, I just grouped mine depending on the situation of the runtime. Small non-gzipped tests `data.json` and `mappings.json` are the best approach for small focused tests. For _larger_ tests and cases I would and sometimes do use things such as auditbeat but try to avoid using larger data sets in favor of smaller focused test cases to validate the runtime is operating as expected.
```ts
{
"type": "doc",
"value": {
"id": "1",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:00:53.000Z",
"long": 1
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "2",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:01:53.000Z",
"long": 2
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "3",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:02:53.000Z",
"long": 3
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "4",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:03:53.000Z",
"long": 4
},
"type": "_doc"
}
}
```
Mapping is added. Note that this is "ECS tolerant" but not necessarily all ECS meaning I can and will try to keep things simple where I can, but I have ensured that `"@timestamp"` is at least there.
```ts
{
"type": "index",
"value": {
"index": "long",
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"long": { "type": "long" }
}
},
"settings": {
"index": {
"number_of_replicas": "1",
"number_of_shards": "1"
}
}
}
}
```
Test is written with test utilities where the `beforeEach` and `afterEach` try and clean up the indexes and load/unload the archives to keep one test from effecting another. Note this is never going to be 100% possible so see below on how we add more determinism in case something escapes the sandbox.
```ts
beforeEach(async () => {
await createSignalsIndex(supertest);
await createListsIndex(supertest);
await esArchiver.load('rule_exceptions/keyword');
});
afterEach(async () => {
await deleteSignalsIndex(supertest);
await deleteAllAlerts(supertest);
await deleteAllExceptions(es);
await deleteListsIndex(supertest);
await esArchiver.unload('rule_exceptions/keyword');
});
describe('"is" operator', () => {
it('should filter 1 single keyword if it is set as an exception', async () => {
const rule = getRuleForSignalTesting(['keyword']);
const { id } = await createRuleWithExceptionEntries(supertest, rule, [
[
{
field: 'keyword',
operator: 'included',
type: 'match',
value: 'word one',
},
],
]);
await waitForRuleSuccess(supertest, id);
await waitForSignalsToBePresent(supertest, 3, [id]);
const signalsOpen = await getSignalsById(supertest, id);
const hits = signalsOpen.hits.hits.map((hit) => hit._source.keyword).sort();
expect(hits).to.eql(['word four', 'word three', 'word two']);
});
});
```
### Changes for better determinism
To support more determinism there are changes and utilities added which can be tuned during any sporadic failures we might encounter as well as better support unexpected changes to other Elastic Stack pieces such as alerting, task manager, etc...
Get simple rule and others are now defaulting to false, meaning that the structural tests will no longer activate a rule and run it on task manger. This should cut down on error outputs as well as reduce stress and potentials for left over rules interfering with the runtime rules.
```ts
export const getSimpleRule = (ruleId = 'rule-1', enabled = false): QueryCreateSchema => ({
```
Not mandatory to use, but for most tests that should be runtime based tests, I use this function below which will enable it by default and run it using settings such as `type: 'query'`, `query: '*:*',` `from: '1900-01-01T00:00:00.000Z'`, to cut down on boiler plate noise. However, people can use whatever they want out of the grab bag or if their test is more readable to hand craft a REST request to create signals, or if they just want to call this and override where they want to, then 👍 .
```ts
export const getRuleForSignalTesting = (index: string[], ruleId = 'rule-1', enabled = true)
```
This waits for a rule to succeed before continuing
```ts
await waitForRuleSuccess(supertest, id);
```
I added a required array of id that _waits_ only for that particular id here. This is useful in case another test did not cleanup and you are getting signals being produced or left behind but need to wait specifically for yours.
```ts
await waitForSignalsToBePresent(supertest, 4, [id]);
```
I only get the signals for a particular rule id using either the auto-generated id or the rule_id. It's safer to use the ones from the auto-generated id but either of these are fine if you're careful enough.
```ts
const signalsOpen = await getSignalsById(supertest, id);
const signalsOpen = await getSignalsByIds(supertest, [createdId]);
const signalsOpen = await getSignalsByRuleIds(supertest, ['signal-on-signal']);
```
I delete all alerts now through a series of steps where it properly removes all rules using the rules bulk_delete and does it in such a way that all the API keys and alerting will be the best it can destroyed as well as double check that the alerts are showing up as being cleaned up before continuing.
```ts
deleteAllAlerts()
```
When not explicitly testing something structural, prefer to use the utilities which can and will do retries in case there are over the wire failures or es failures. Examples are:
```ts
installPrePackagedRules()
waitForRuleSuccess()
importFile() // This does a _lot_ of checks to ensure that the file is fully imported before continuing
```
Some of these utilities might still do a `expect(200);` but as we are and should use regular structural tests to cover those problems, these will probably be more and more removed when/if we hit test failures in favor of doing retries, waitFor, and countDowns.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2020-11-20 20:09:38 +01:00
. expect ( 200 ) ;
return signalsOpen ;
} ;
/ * *
* Given a single rule id this will return only signals based on that rule id .
* @param supertest agent
* @param ids Rule id
* /
export const getSignalsById = async (
supertest : SuperTest < supertestAsPromised.Test > ,
id : string
) : Promise <
SearchResponse < {
signal : Signal ;
[ x : string ] : unknown ;
} >
> = > {
const { body : signalsOpen } : { body : SearchResponse < { signal : Signal } > } = await supertest
. post ( DETECTION_ENGINE_QUERY_SIGNALS_URL )
. set ( 'kbn-xsrf' , 'true' )
. send ( getQuerySignalsId ( [ id ] ) )
[Security Solutions][Detection Engine] Fixes critical clashing with source indexes that already contain a "signal" field (#82191)
## Summary
Fixes: https://github.com/elastic/kibana/issues/82148
We have errors and do not generate a signal when a source index already has utilized and reserved the "signal" field for their own data purposes. This fix is a bit tricky and has one medium sized risk which is we also support "signals generated on top of existing signals". Therefore we have to be careful and do a small runtime detection of the "data shape" of the signal's data type. If it looks like the user is using the "signal" field within their mapping instead of us, we move the customer's signal into "original_signal" inside our "signal" structure we create when we copy their data set when creating a signal.
To help mitigate the risks associated with this critical bug with regards to breaking signals on top of signals I have:
* This adds unit tests
* This adds end to end tests for testing generating signals including signals on signals to help mitigate risk
The key test for this shape in the PR are in the file:
```
detection_engine/signals/build_event_type_signal.ts
```
like so:
```ts
export const isEventTypeSignal = (doc: BaseSignalHit): boolean => {
return doc._source.signal?.rule?.id != null && typeof doc._source.signal?.rule?.id === 'string';
};
```
Example of what happens when it does a "move" of an existing numeric signal keyword type:
```ts
# This causes a clash with us using the name signal as a numeric.
PUT clashing-index/_doc/1
{
"@timestamp": "2020-10-28T05:08:53.000Z",
"signal": 1
}
```
Before, this was an error. With this PR it now will restructure this data like so when creating a signal along with additional signal ancestor information, meta data. I omitted some of the data from the output signal for this example.
```ts
{
... Other data copied ...
"signal":
{
"original_signal": 1 <--- We "move it" here now
"parents":
[
{
"id": "BhbXBmkBR346wHgn4PeZ",
"type": "event",
"index": "your-index-name",
"depth": 0
},
],
"ancestors":
[
{
"id": "BhbXBmkBR346wHgn4PeZ",
"type": "event",
"index": "your-index-name",
"depth": 0
},
],
"status": "open",
"depth": 1,
"parent":
{
"id": "BhbXBmkBR346wHgn4PeZ",
type: "event",
"index": "your-index-name",
"depth": 0
},
"original_time": "2019-02-19T17:40:03.790Z",
"original_event":
{
"action": "socket_closed",
"dataset": "socket",
"kind": "event",
"module": "system"
},
}
```
### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2020-11-06 15:47:57 +01:00
. expect ( 200 ) ;
return signalsOpen ;
} ;
2020-10-20 03:14:09 +02:00
export const installPrePackagedRules = async (
supertest : SuperTest < supertestAsPromised.Test >
) : Promise < void > = > {
[Security Solutions][Detection Engine] Adds e2e FTR runtime support and 213 tests for exception lists (#83764)
## Summary
Adds support to the end to end (e2e) functional test runner (FTR) support for rule runtime tests as well as 213 tests for the exception lists which include value based lists. Previously we had limited runtime support, but as I scaled up runtime tests from 5 to 200+ I noticed in a lot of areas we had to use improved techniques for determinism.
The runtime support being added is our next step of tests. Up to now most of our e2e FTR tests have been structural testing of REST and API integration tests. Basically up to now 95% tests are API structural as:
* Call REST input related to a rule such as GET/PUT/POST/PATCH/DELETE.
* Check REST output of the rule, did it match expected output body and status code?
* In some rare cases we check if the the rule can be executed and we get a status of 'succeeded'
With only a small part of our tests ~5%, `generating_signals.ts` was checking the signals being produced. However, we cannot have confidence in runtime based tests until the structural tests have been built up and run through the weeks against PR's to ensure that those are stable and deterministic.
Now that we have confidence and 90%+ coverage of the structural REST based tests, we are building up newer sets of tests which allow us to do runtime based validation tests to increase confidence that:
* Detection engine produces signals as expected
* Structure of the signals are as expected, including signal on signals
* Exceptions to signals are working as expected
* Most runtime bugs can be TDD'ed with e2e FTR's and regressions
* Whack-a-mole will not happen
* Consistency and predictability of signals is validated
* Refactoring can occur with stronger confidence
* Runtime tests are reference points for answering questions about existing bugs or adding new ones to test if users are experiencing unexpected behaviors
* Scaling tests can happen without failures
* Velocity for creating tests increases as the utilities and examples increase
Lastly, this puts us within striking distance of creating FTR's for different common class of runtime situations such as:
* Creating tests that exercise each rule against a set of data criteria and get signal hits
* Creating tests that validate the rule overrides operate as expected against data sets
* Creating tests that validate malfunctions, corner cases, or misuse cases such as data sets that are _all_ arrays or data sets that put numbers as strings or throws in an expected `null` instead of a value.
These tests follow the pattern of:
* Add the smallest data set to a folder in data.json (not gzip format)
* Add the smallest mapping to that folder (mapping.json)
* Call REST input related to exception lists, value lists, adding prepackaged rules, etc...
* Call REST input related endpoint with utilities to create and activate the rule
* Wait for the rule to go into the `succeeded` phase
* Wait for the N exact signals specific to that rule to be available
* Check against the set of signals to ensure that the matches are exactly as expected
Example of one runtime test:
A keyword data set is added to a folder called "keyword" but you can add one anywhere you want under `es_archives`, I just grouped mine depending on the situation of the runtime. Small non-gzipped tests `data.json` and `mappings.json` are the best approach for small focused tests. For _larger_ tests and cases I would and sometimes do use things such as auditbeat but try to avoid using larger data sets in favor of smaller focused test cases to validate the runtime is operating as expected.
```ts
{
"type": "doc",
"value": {
"id": "1",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:00:53.000Z",
"long": 1
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "2",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:01:53.000Z",
"long": 2
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "3",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:02:53.000Z",
"long": 3
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "4",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:03:53.000Z",
"long": 4
},
"type": "_doc"
}
}
```
Mapping is added. Note that this is "ECS tolerant" but not necessarily all ECS meaning I can and will try to keep things simple where I can, but I have ensured that `"@timestamp"` is at least there.
```ts
{
"type": "index",
"value": {
"index": "long",
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"long": { "type": "long" }
}
},
"settings": {
"index": {
"number_of_replicas": "1",
"number_of_shards": "1"
}
}
}
}
```
Test is written with test utilities where the `beforeEach` and `afterEach` try and clean up the indexes and load/unload the archives to keep one test from effecting another. Note this is never going to be 100% possible so see below on how we add more determinism in case something escapes the sandbox.
```ts
beforeEach(async () => {
await createSignalsIndex(supertest);
await createListsIndex(supertest);
await esArchiver.load('rule_exceptions/keyword');
});
afterEach(async () => {
await deleteSignalsIndex(supertest);
await deleteAllAlerts(supertest);
await deleteAllExceptions(es);
await deleteListsIndex(supertest);
await esArchiver.unload('rule_exceptions/keyword');
});
describe('"is" operator', () => {
it('should filter 1 single keyword if it is set as an exception', async () => {
const rule = getRuleForSignalTesting(['keyword']);
const { id } = await createRuleWithExceptionEntries(supertest, rule, [
[
{
field: 'keyword',
operator: 'included',
type: 'match',
value: 'word one',
},
],
]);
await waitForRuleSuccess(supertest, id);
await waitForSignalsToBePresent(supertest, 3, [id]);
const signalsOpen = await getSignalsById(supertest, id);
const hits = signalsOpen.hits.hits.map((hit) => hit._source.keyword).sort();
expect(hits).to.eql(['word four', 'word three', 'word two']);
});
});
```
### Changes for better determinism
To support more determinism there are changes and utilities added which can be tuned during any sporadic failures we might encounter as well as better support unexpected changes to other Elastic Stack pieces such as alerting, task manager, etc...
Get simple rule and others are now defaulting to false, meaning that the structural tests will no longer activate a rule and run it on task manger. This should cut down on error outputs as well as reduce stress and potentials for left over rules interfering with the runtime rules.
```ts
export const getSimpleRule = (ruleId = 'rule-1', enabled = false): QueryCreateSchema => ({
```
Not mandatory to use, but for most tests that should be runtime based tests, I use this function below which will enable it by default and run it using settings such as `type: 'query'`, `query: '*:*',` `from: '1900-01-01T00:00:00.000Z'`, to cut down on boiler plate noise. However, people can use whatever they want out of the grab bag or if their test is more readable to hand craft a REST request to create signals, or if they just want to call this and override where they want to, then 👍 .
```ts
export const getRuleForSignalTesting = (index: string[], ruleId = 'rule-1', enabled = true)
```
This waits for a rule to succeed before continuing
```ts
await waitForRuleSuccess(supertest, id);
```
I added a required array of id that _waits_ only for that particular id here. This is useful in case another test did not cleanup and you are getting signals being produced or left behind but need to wait specifically for yours.
```ts
await waitForSignalsToBePresent(supertest, 4, [id]);
```
I only get the signals for a particular rule id using either the auto-generated id or the rule_id. It's safer to use the ones from the auto-generated id but either of these are fine if you're careful enough.
```ts
const signalsOpen = await getSignalsById(supertest, id);
const signalsOpen = await getSignalsByIds(supertest, [createdId]);
const signalsOpen = await getSignalsByRuleIds(supertest, ['signal-on-signal']);
```
I delete all alerts now through a series of steps where it properly removes all rules using the rules bulk_delete and does it in such a way that all the API keys and alerting will be the best it can destroyed as well as double check that the alerts are showing up as being cleaned up before continuing.
```ts
deleteAllAlerts()
```
When not explicitly testing something structural, prefer to use the utilities which can and will do retries in case there are over the wire failures or es failures. Examples are:
```ts
installPrePackagedRules()
waitForRuleSuccess()
importFile() // This does a _lot_ of checks to ensure that the file is fully imported before continuing
```
Some of these utilities might still do a `expect(200);` but as we are and should use regular structural tests to cover those problems, these will probably be more and more removed when/if we hit test failures in favor of doing retries, waitFor, and countDowns.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2020-11-20 20:09:38 +01:00
await countDownTest ( async ( ) = > {
const { status } = await supertest
. put ( DETECTION_ENGINE_PREPACKAGED_URL )
. set ( 'kbn-xsrf' , 'true' )
. send ( ) ;
return status === 200 ;
} , 'installPrePackagedRules' ) ;
} ;
/ * *
* Convenience testing function where you can pass in just the entries and you will
* get a rule created with the entries added to an exception list and exception list item
* all auto - created at once .
* @param supertest super test agent
* @param rule The rule to create and attach an exception list to
* @param entries The entries to create the rule and exception list from
* /
export const createRuleWithExceptionEntries = async (
supertest : SuperTest < supertestAsPromised.Test > ,
2021-04-19 22:10:59 +02:00
rule : CreateRulesSchema ,
[Security Solutions][Detection Engine] Adds e2e FTR runtime support and 213 tests for exception lists (#83764)
## Summary
Adds support to the end to end (e2e) functional test runner (FTR) support for rule runtime tests as well as 213 tests for the exception lists which include value based lists. Previously we had limited runtime support, but as I scaled up runtime tests from 5 to 200+ I noticed in a lot of areas we had to use improved techniques for determinism.
The runtime support being added is our next step of tests. Up to now most of our e2e FTR tests have been structural testing of REST and API integration tests. Basically up to now 95% tests are API structural as:
* Call REST input related to a rule such as GET/PUT/POST/PATCH/DELETE.
* Check REST output of the rule, did it match expected output body and status code?
* In some rare cases we check if the the rule can be executed and we get a status of 'succeeded'
With only a small part of our tests ~5%, `generating_signals.ts` was checking the signals being produced. However, we cannot have confidence in runtime based tests until the structural tests have been built up and run through the weeks against PR's to ensure that those are stable and deterministic.
Now that we have confidence and 90%+ coverage of the structural REST based tests, we are building up newer sets of tests which allow us to do runtime based validation tests to increase confidence that:
* Detection engine produces signals as expected
* Structure of the signals are as expected, including signal on signals
* Exceptions to signals are working as expected
* Most runtime bugs can be TDD'ed with e2e FTR's and regressions
* Whack-a-mole will not happen
* Consistency and predictability of signals is validated
* Refactoring can occur with stronger confidence
* Runtime tests are reference points for answering questions about existing bugs or adding new ones to test if users are experiencing unexpected behaviors
* Scaling tests can happen without failures
* Velocity for creating tests increases as the utilities and examples increase
Lastly, this puts us within striking distance of creating FTR's for different common class of runtime situations such as:
* Creating tests that exercise each rule against a set of data criteria and get signal hits
* Creating tests that validate the rule overrides operate as expected against data sets
* Creating tests that validate malfunctions, corner cases, or misuse cases such as data sets that are _all_ arrays or data sets that put numbers as strings or throws in an expected `null` instead of a value.
These tests follow the pattern of:
* Add the smallest data set to a folder in data.json (not gzip format)
* Add the smallest mapping to that folder (mapping.json)
* Call REST input related to exception lists, value lists, adding prepackaged rules, etc...
* Call REST input related endpoint with utilities to create and activate the rule
* Wait for the rule to go into the `succeeded` phase
* Wait for the N exact signals specific to that rule to be available
* Check against the set of signals to ensure that the matches are exactly as expected
Example of one runtime test:
A keyword data set is added to a folder called "keyword" but you can add one anywhere you want under `es_archives`, I just grouped mine depending on the situation of the runtime. Small non-gzipped tests `data.json` and `mappings.json` are the best approach for small focused tests. For _larger_ tests and cases I would and sometimes do use things such as auditbeat but try to avoid using larger data sets in favor of smaller focused test cases to validate the runtime is operating as expected.
```ts
{
"type": "doc",
"value": {
"id": "1",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:00:53.000Z",
"long": 1
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "2",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:01:53.000Z",
"long": 2
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "3",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:02:53.000Z",
"long": 3
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "4",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:03:53.000Z",
"long": 4
},
"type": "_doc"
}
}
```
Mapping is added. Note that this is "ECS tolerant" but not necessarily all ECS meaning I can and will try to keep things simple where I can, but I have ensured that `"@timestamp"` is at least there.
```ts
{
"type": "index",
"value": {
"index": "long",
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"long": { "type": "long" }
}
},
"settings": {
"index": {
"number_of_replicas": "1",
"number_of_shards": "1"
}
}
}
}
```
Test is written with test utilities where the `beforeEach` and `afterEach` try and clean up the indexes and load/unload the archives to keep one test from effecting another. Note this is never going to be 100% possible so see below on how we add more determinism in case something escapes the sandbox.
```ts
beforeEach(async () => {
await createSignalsIndex(supertest);
await createListsIndex(supertest);
await esArchiver.load('rule_exceptions/keyword');
});
afterEach(async () => {
await deleteSignalsIndex(supertest);
await deleteAllAlerts(supertest);
await deleteAllExceptions(es);
await deleteListsIndex(supertest);
await esArchiver.unload('rule_exceptions/keyword');
});
describe('"is" operator', () => {
it('should filter 1 single keyword if it is set as an exception', async () => {
const rule = getRuleForSignalTesting(['keyword']);
const { id } = await createRuleWithExceptionEntries(supertest, rule, [
[
{
field: 'keyword',
operator: 'included',
type: 'match',
value: 'word one',
},
],
]);
await waitForRuleSuccess(supertest, id);
await waitForSignalsToBePresent(supertest, 3, [id]);
const signalsOpen = await getSignalsById(supertest, id);
const hits = signalsOpen.hits.hits.map((hit) => hit._source.keyword).sort();
expect(hits).to.eql(['word four', 'word three', 'word two']);
});
});
```
### Changes for better determinism
To support more determinism there are changes and utilities added which can be tuned during any sporadic failures we might encounter as well as better support unexpected changes to other Elastic Stack pieces such as alerting, task manager, etc...
Get simple rule and others are now defaulting to false, meaning that the structural tests will no longer activate a rule and run it on task manger. This should cut down on error outputs as well as reduce stress and potentials for left over rules interfering with the runtime rules.
```ts
export const getSimpleRule = (ruleId = 'rule-1', enabled = false): QueryCreateSchema => ({
```
Not mandatory to use, but for most tests that should be runtime based tests, I use this function below which will enable it by default and run it using settings such as `type: 'query'`, `query: '*:*',` `from: '1900-01-01T00:00:00.000Z'`, to cut down on boiler plate noise. However, people can use whatever they want out of the grab bag or if their test is more readable to hand craft a REST request to create signals, or if they just want to call this and override where they want to, then 👍 .
```ts
export const getRuleForSignalTesting = (index: string[], ruleId = 'rule-1', enabled = true)
```
This waits for a rule to succeed before continuing
```ts
await waitForRuleSuccess(supertest, id);
```
I added a required array of id that _waits_ only for that particular id here. This is useful in case another test did not cleanup and you are getting signals being produced or left behind but need to wait specifically for yours.
```ts
await waitForSignalsToBePresent(supertest, 4, [id]);
```
I only get the signals for a particular rule id using either the auto-generated id or the rule_id. It's safer to use the ones from the auto-generated id but either of these are fine if you're careful enough.
```ts
const signalsOpen = await getSignalsById(supertest, id);
const signalsOpen = await getSignalsByIds(supertest, [createdId]);
const signalsOpen = await getSignalsByRuleIds(supertest, ['signal-on-signal']);
```
I delete all alerts now through a series of steps where it properly removes all rules using the rules bulk_delete and does it in such a way that all the API keys and alerting will be the best it can destroyed as well as double check that the alerts are showing up as being cleaned up before continuing.
```ts
deleteAllAlerts()
```
When not explicitly testing something structural, prefer to use the utilities which can and will do retries in case there are over the wire failures or es failures. Examples are:
```ts
installPrePackagedRules()
waitForRuleSuccess()
importFile() // This does a _lot_ of checks to ensure that the file is fully imported before continuing
```
Some of these utilities might still do a `expect(200);` but as we are and should use regular structural tests to cover those problems, these will probably be more and more removed when/if we hit test failures in favor of doing retries, waitFor, and countDowns.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2020-11-20 20:09:38 +01:00
entries : NonEmptyEntriesArray [ ]
) : Promise < FullResponseSchema > = > {
// eslint-disable-next-line @typescript-eslint/naming-convention
const { id , list_id , namespace_type , type } = await createExceptionList (
supertest ,
getCreateExceptionListDetectionSchemaMock ( )
) ;
await Promise . all (
entries . map ( ( entry ) = > {
const exceptionListItem : CreateExceptionListItemSchema = {
. . . getCreateExceptionListItemMinimalSchemaMockWithoutId ( ) ,
entries : entry ,
} ;
return createExceptionListItem ( supertest , exceptionListItem ) ;
} )
) ;
// To reduce the odds of in-determinism and/or bugs we ensure we have
// the same length of entries before continuing.
await waitFor ( async ( ) = > {
const { body } = await supertest . get (
` ${ EXCEPTION_LIST_ITEM_URL } /_find?list_id= ${
getCreateExceptionListDetectionSchemaMock ( ) . list_id
} `
) ;
return body . data . length === entries . length ;
} , ` within createRuleWithExceptionEntries ${ EXCEPTION_LIST_ITEM_URL } /_find?list_id= ${ getCreateExceptionListDetectionSchemaMock ( ) . list_id } ` ) ;
// create the rule but don't run it immediately as running it immediately can cause
// the rule to sometimes not filter correctly the first time with an exception list
// or other timing issues. Then afterwards wait for the rule to have succeeded before
// returning.
2021-04-19 22:10:59 +02:00
const ruleWithException : CreateRulesSchema = {
[Security Solutions][Detection Engine] Adds e2e FTR runtime support and 213 tests for exception lists (#83764)
## Summary
Adds support to the end to end (e2e) functional test runner (FTR) support for rule runtime tests as well as 213 tests for the exception lists which include value based lists. Previously we had limited runtime support, but as I scaled up runtime tests from 5 to 200+ I noticed in a lot of areas we had to use improved techniques for determinism.
The runtime support being added is our next step of tests. Up to now most of our e2e FTR tests have been structural testing of REST and API integration tests. Basically up to now 95% tests are API structural as:
* Call REST input related to a rule such as GET/PUT/POST/PATCH/DELETE.
* Check REST output of the rule, did it match expected output body and status code?
* In some rare cases we check if the the rule can be executed and we get a status of 'succeeded'
With only a small part of our tests ~5%, `generating_signals.ts` was checking the signals being produced. However, we cannot have confidence in runtime based tests until the structural tests have been built up and run through the weeks against PR's to ensure that those are stable and deterministic.
Now that we have confidence and 90%+ coverage of the structural REST based tests, we are building up newer sets of tests which allow us to do runtime based validation tests to increase confidence that:
* Detection engine produces signals as expected
* Structure of the signals are as expected, including signal on signals
* Exceptions to signals are working as expected
* Most runtime bugs can be TDD'ed with e2e FTR's and regressions
* Whack-a-mole will not happen
* Consistency and predictability of signals is validated
* Refactoring can occur with stronger confidence
* Runtime tests are reference points for answering questions about existing bugs or adding new ones to test if users are experiencing unexpected behaviors
* Scaling tests can happen without failures
* Velocity for creating tests increases as the utilities and examples increase
Lastly, this puts us within striking distance of creating FTR's for different common class of runtime situations such as:
* Creating tests that exercise each rule against a set of data criteria and get signal hits
* Creating tests that validate the rule overrides operate as expected against data sets
* Creating tests that validate malfunctions, corner cases, or misuse cases such as data sets that are _all_ arrays or data sets that put numbers as strings or throws in an expected `null` instead of a value.
These tests follow the pattern of:
* Add the smallest data set to a folder in data.json (not gzip format)
* Add the smallest mapping to that folder (mapping.json)
* Call REST input related to exception lists, value lists, adding prepackaged rules, etc...
* Call REST input related endpoint with utilities to create and activate the rule
* Wait for the rule to go into the `succeeded` phase
* Wait for the N exact signals specific to that rule to be available
* Check against the set of signals to ensure that the matches are exactly as expected
Example of one runtime test:
A keyword data set is added to a folder called "keyword" but you can add one anywhere you want under `es_archives`, I just grouped mine depending on the situation of the runtime. Small non-gzipped tests `data.json` and `mappings.json` are the best approach for small focused tests. For _larger_ tests and cases I would and sometimes do use things such as auditbeat but try to avoid using larger data sets in favor of smaller focused test cases to validate the runtime is operating as expected.
```ts
{
"type": "doc",
"value": {
"id": "1",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:00:53.000Z",
"long": 1
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "2",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:01:53.000Z",
"long": 2
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "3",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:02:53.000Z",
"long": 3
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "4",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:03:53.000Z",
"long": 4
},
"type": "_doc"
}
}
```
Mapping is added. Note that this is "ECS tolerant" but not necessarily all ECS meaning I can and will try to keep things simple where I can, but I have ensured that `"@timestamp"` is at least there.
```ts
{
"type": "index",
"value": {
"index": "long",
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"long": { "type": "long" }
}
},
"settings": {
"index": {
"number_of_replicas": "1",
"number_of_shards": "1"
}
}
}
}
```
Test is written with test utilities where the `beforeEach` and `afterEach` try and clean up the indexes and load/unload the archives to keep one test from effecting another. Note this is never going to be 100% possible so see below on how we add more determinism in case something escapes the sandbox.
```ts
beforeEach(async () => {
await createSignalsIndex(supertest);
await createListsIndex(supertest);
await esArchiver.load('rule_exceptions/keyword');
});
afterEach(async () => {
await deleteSignalsIndex(supertest);
await deleteAllAlerts(supertest);
await deleteAllExceptions(es);
await deleteListsIndex(supertest);
await esArchiver.unload('rule_exceptions/keyword');
});
describe('"is" operator', () => {
it('should filter 1 single keyword if it is set as an exception', async () => {
const rule = getRuleForSignalTesting(['keyword']);
const { id } = await createRuleWithExceptionEntries(supertest, rule, [
[
{
field: 'keyword',
operator: 'included',
type: 'match',
value: 'word one',
},
],
]);
await waitForRuleSuccess(supertest, id);
await waitForSignalsToBePresent(supertest, 3, [id]);
const signalsOpen = await getSignalsById(supertest, id);
const hits = signalsOpen.hits.hits.map((hit) => hit._source.keyword).sort();
expect(hits).to.eql(['word four', 'word three', 'word two']);
});
});
```
### Changes for better determinism
To support more determinism there are changes and utilities added which can be tuned during any sporadic failures we might encounter as well as better support unexpected changes to other Elastic Stack pieces such as alerting, task manager, etc...
Get simple rule and others are now defaulting to false, meaning that the structural tests will no longer activate a rule and run it on task manger. This should cut down on error outputs as well as reduce stress and potentials for left over rules interfering with the runtime rules.
```ts
export const getSimpleRule = (ruleId = 'rule-1', enabled = false): QueryCreateSchema => ({
```
Not mandatory to use, but for most tests that should be runtime based tests, I use this function below which will enable it by default and run it using settings such as `type: 'query'`, `query: '*:*',` `from: '1900-01-01T00:00:00.000Z'`, to cut down on boiler plate noise. However, people can use whatever they want out of the grab bag or if their test is more readable to hand craft a REST request to create signals, or if they just want to call this and override where they want to, then 👍 .
```ts
export const getRuleForSignalTesting = (index: string[], ruleId = 'rule-1', enabled = true)
```
This waits for a rule to succeed before continuing
```ts
await waitForRuleSuccess(supertest, id);
```
I added a required array of id that _waits_ only for that particular id here. This is useful in case another test did not cleanup and you are getting signals being produced or left behind but need to wait specifically for yours.
```ts
await waitForSignalsToBePresent(supertest, 4, [id]);
```
I only get the signals for a particular rule id using either the auto-generated id or the rule_id. It's safer to use the ones from the auto-generated id but either of these are fine if you're careful enough.
```ts
const signalsOpen = await getSignalsById(supertest, id);
const signalsOpen = await getSignalsByIds(supertest, [createdId]);
const signalsOpen = await getSignalsByRuleIds(supertest, ['signal-on-signal']);
```
I delete all alerts now through a series of steps where it properly removes all rules using the rules bulk_delete and does it in such a way that all the API keys and alerting will be the best it can destroyed as well as double check that the alerts are showing up as being cleaned up before continuing.
```ts
deleteAllAlerts()
```
When not explicitly testing something structural, prefer to use the utilities which can and will do retries in case there are over the wire failures or es failures. Examples are:
```ts
installPrePackagedRules()
waitForRuleSuccess()
importFile() // This does a _lot_ of checks to ensure that the file is fully imported before continuing
```
Some of these utilities might still do a `expect(200);` but as we are and should use regular structural tests to cover those problems, these will probably be more and more removed when/if we hit test failures in favor of doing retries, waitFor, and countDowns.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2020-11-20 20:09:38 +01:00
. . . rule ,
enabled : false ,
exceptions_list : [
{
id ,
list_id ,
namespace_type ,
type ,
} ,
] ,
} ;
const ruleResponse = await createRule ( supertest , ruleWithException ) ;
await supertest
. patch ( DETECTION_ENGINE_RULES_URL )
. set ( 'kbn-xsrf' , 'true' )
. send ( { rule_id : ruleResponse.rule_id , enabled : true } )
. expect ( 200 ) ;
return ruleResponse ;
2020-10-20 03:14:09 +02:00
} ;
[Security Solution][Detections] Signals Migration API (#84721)
* WIP: basic reindexing works, lots of edge cases and TODOs to tackle
* Add note
* Add version metadata to signals documents
* WIP: Starting over from the ground up
* Removes obsolete endpoints/functions
* Adds endpoint for checking the migration status of signals indices
* Adds helper functions to represent the logical pieces of answering
that question
* Fleshing out upgrade of signals
* triggers reindex for each index
* starts implementing followup endpoint to "finalize" after reindexing
is finished
* Fleshing out more of the upgrade path
Still moving logic around a bunch.
* Pad the version number of our destination migration index
Instead of e.g. `.siem-signals-default-000001-r5`, this will generate
`.siem-signals-default-000001-r000005`.
This shouldn't matter much, but it may make it easier for users at a
glance to see the story of each index.
* Fleshing out more upgrade finalization
* Verifies that task matches the specified parameters
* Verifies that document counts are the same
* updates aliases
* finalization endpoint requires both source/dest indexes since we can't
determine that from the task itself.
* Ensure that new signals are generated with an appropriate schema_version
* Apply migration cleanup policy to obsolete signals indexes
After upgrading a particular signals index, we're left with both the old
and new copies of the index. While the former is unlinked, it's still
taking up disk space; this ensures that it will eventually be deleted,
but gives users enough time to recover data if necessary.
This also ensures that, as with the normal signals ILM policy, it is
present during our normal sanity checks.
* Move more logic into component functions
* Fix type errors
* Refactor to make things a little more organized
* Moves migration-related routes under signals/ to match their routing
* Generalizes migration-agnostic helpers, moves them to appropriate
folders (namely index/)
* Inlined getMigrationStatusInRange, a hyper-specific function with
limited utility elsewhere
* Add some JSDoc comments around our new functions
This is as much to get my thoughts in order as it is for posterity.
Next: tests!
* Adds integration tests around migration status route
* Adds io-ts schema for route params
* Adds es_archiver data to represent an outdated signals index
* Adds API integration tests for our signals upgrade endpoint
* Adds io-ts schema for route params
* Adds second signals index archive, updates docs
* Adds test helper to wait for a given index to have documents
* Adds test helper to retrieve the relevant index name from a call to
esArchive.load
* WIP: Fleshing out finalization tests
* Consolidate terminalogy around a migration
We're no longer making a distinction between an upgrade vs. an update
vs. a migration vs. a reindex: a migration is the concept that
encompasses this work. Both an index and individual documents can
require a migration, but both follow the same code path to migrate.
* Implement encoding of migration details
This will be a slightly better API: rather than having to pass all three
fields to finalize the migration, API users can instead send the token.
* Better transformation of errors thrown from the elasticsearch client
These often contain detailed information that we were previously
dropping. This will give better info on the migration finalization
endpoint, but should give more information across all detection_engine
endpoints in the case of an es client error.
* Finishing integration tests around finalization endpoint
This lead to a few changes in the responses from our different
endpoints; mainly, we pass both the migration token AND its constituent
parts to aid in debugging.
* Test an error case due to a reindexing failure
This would be really hard to reproduce with an integration test since
we'd need to generate a specific reindex failure. Much easier to stub
some ES calls to exercise that code in a unit test.
* Remove unnecessary version info from signals documents
We now record a single document-level version field. This represents the
version of the document's _source, which is generated by our rule
execution.
When either a mapping _or_ a transformation is added, this version will
be bumped such that new signals will contain the newest version, while
the index itself may still contain the old mappings.
The transformation pipeline will use the signal version to short-circuit
unnecessary transformations.
* Migrate an index relative to the ACTUAL template version
This handles the case where a user is attempting to migrate, but has not
yet rolled over to the newest template. Running rules may insert "new"
signals into an "old" index, but from the perspective of the app no
migration is necessary in that case.
If/when they roll over, the aforementioned index (and possibly older
ones) will be qualified as outdated, and can be migrated.
* Enrich our migration_status endpoint with an is_outdated qualification
This can be determined programatically, but for users manually
interpreting this response, the qualification will help.
* Update migration scripts
* More uniform version checking
* getIndexVersion always returns a number
* version comparisons use isOutdated
* Fix signal generation unit tests
We now generate a version field to indicate the version under which the
signal was created/migrated.
* Support reindex options to be sent to create_migration endpoint
Rather than having to perform a manual reindex, this should give API
users some control over the performance of their automated migration.
* Fix signal generation integration tests
These were failing on our new signal field.
* Add unit tests for getMigrationStatus
* Add a basic test for getSignalsIndicesInRange
Since this is ultimately just an aggregation query there's not much else
to test.
* Add unit test for the naming of our destination migration index
* Handle write indices in our migration logic
* Treat write indices as any other index in migration status endpoint
* Migration API rejects requests containing write indices
* Migration API rejects requests containing unknown/non-signals indices
* Add original hot phase to migration cleanup policy
Without this phase, ILM gets confused as it tries to move to the delete
phase and fails.
* Update old comment
The referenced field has changed.
* Delete task document as part of finalization
* Accurately report recoverable errors on create_signals_migration route
If we have a recoverable error: e.g. the destination index already
exists, or a specified index is a write index, we now report those
errors as part of the normal 200 response as these do not preclude other
specified indices from being migrated.
However, if non-signals indices are specified, we do continue to reject
the entire request, as that's indicative of misuse of the endpoint.
2020-12-10 20:12:39 +01:00
export const getIndexNameFromLoad = ( loadResponse : Record < string , unknown > ) : string = > {
const indexNames = Object . keys ( loadResponse ) ;
if ( indexNames . length > 1 ) {
throw new Error (
` expected load response to contain one index, but contained multiple: [ ${ indexNames } ] `
) ;
}
return indexNames [ 0 ] ;
} ;
/ * *
* Waits for the given index to contain documents
*
* @param esClient elasticsearch { @link Client }
* @param index name of the index to query
* /
2021-03-25 09:47:16 +01:00
export const waitForIndexToPopulate = async ( es : KibanaClient , index : string ) : Promise < void > = > {
[Security Solution][Detections] Signals Migration API (#84721)
* WIP: basic reindexing works, lots of edge cases and TODOs to tackle
* Add note
* Add version metadata to signals documents
* WIP: Starting over from the ground up
* Removes obsolete endpoints/functions
* Adds endpoint for checking the migration status of signals indices
* Adds helper functions to represent the logical pieces of answering
that question
* Fleshing out upgrade of signals
* triggers reindex for each index
* starts implementing followup endpoint to "finalize" after reindexing
is finished
* Fleshing out more of the upgrade path
Still moving logic around a bunch.
* Pad the version number of our destination migration index
Instead of e.g. `.siem-signals-default-000001-r5`, this will generate
`.siem-signals-default-000001-r000005`.
This shouldn't matter much, but it may make it easier for users at a
glance to see the story of each index.
* Fleshing out more upgrade finalization
* Verifies that task matches the specified parameters
* Verifies that document counts are the same
* updates aliases
* finalization endpoint requires both source/dest indexes since we can't
determine that from the task itself.
* Ensure that new signals are generated with an appropriate schema_version
* Apply migration cleanup policy to obsolete signals indexes
After upgrading a particular signals index, we're left with both the old
and new copies of the index. While the former is unlinked, it's still
taking up disk space; this ensures that it will eventually be deleted,
but gives users enough time to recover data if necessary.
This also ensures that, as with the normal signals ILM policy, it is
present during our normal sanity checks.
* Move more logic into component functions
* Fix type errors
* Refactor to make things a little more organized
* Moves migration-related routes under signals/ to match their routing
* Generalizes migration-agnostic helpers, moves them to appropriate
folders (namely index/)
* Inlined getMigrationStatusInRange, a hyper-specific function with
limited utility elsewhere
* Add some JSDoc comments around our new functions
This is as much to get my thoughts in order as it is for posterity.
Next: tests!
* Adds integration tests around migration status route
* Adds io-ts schema for route params
* Adds es_archiver data to represent an outdated signals index
* Adds API integration tests for our signals upgrade endpoint
* Adds io-ts schema for route params
* Adds second signals index archive, updates docs
* Adds test helper to wait for a given index to have documents
* Adds test helper to retrieve the relevant index name from a call to
esArchive.load
* WIP: Fleshing out finalization tests
* Consolidate terminalogy around a migration
We're no longer making a distinction between an upgrade vs. an update
vs. a migration vs. a reindex: a migration is the concept that
encompasses this work. Both an index and individual documents can
require a migration, but both follow the same code path to migrate.
* Implement encoding of migration details
This will be a slightly better API: rather than having to pass all three
fields to finalize the migration, API users can instead send the token.
* Better transformation of errors thrown from the elasticsearch client
These often contain detailed information that we were previously
dropping. This will give better info on the migration finalization
endpoint, but should give more information across all detection_engine
endpoints in the case of an es client error.
* Finishing integration tests around finalization endpoint
This lead to a few changes in the responses from our different
endpoints; mainly, we pass both the migration token AND its constituent
parts to aid in debugging.
* Test an error case due to a reindexing failure
This would be really hard to reproduce with an integration test since
we'd need to generate a specific reindex failure. Much easier to stub
some ES calls to exercise that code in a unit test.
* Remove unnecessary version info from signals documents
We now record a single document-level version field. This represents the
version of the document's _source, which is generated by our rule
execution.
When either a mapping _or_ a transformation is added, this version will
be bumped such that new signals will contain the newest version, while
the index itself may still contain the old mappings.
The transformation pipeline will use the signal version to short-circuit
unnecessary transformations.
* Migrate an index relative to the ACTUAL template version
This handles the case where a user is attempting to migrate, but has not
yet rolled over to the newest template. Running rules may insert "new"
signals into an "old" index, but from the perspective of the app no
migration is necessary in that case.
If/when they roll over, the aforementioned index (and possibly older
ones) will be qualified as outdated, and can be migrated.
* Enrich our migration_status endpoint with an is_outdated qualification
This can be determined programatically, but for users manually
interpreting this response, the qualification will help.
* Update migration scripts
* More uniform version checking
* getIndexVersion always returns a number
* version comparisons use isOutdated
* Fix signal generation unit tests
We now generate a version field to indicate the version under which the
signal was created/migrated.
* Support reindex options to be sent to create_migration endpoint
Rather than having to perform a manual reindex, this should give API
users some control over the performance of their automated migration.
* Fix signal generation integration tests
These were failing on our new signal field.
* Add unit tests for getMigrationStatus
* Add a basic test for getSignalsIndicesInRange
Since this is ultimately just an aggregation query there's not much else
to test.
* Add unit test for the naming of our destination migration index
* Handle write indices in our migration logic
* Treat write indices as any other index in migration status endpoint
* Migration API rejects requests containing write indices
* Migration API rejects requests containing unknown/non-signals indices
* Add original hot phase to migration cleanup policy
Without this phase, ILM gets confused as it tries to move to the delete
phase and fails.
* Update old comment
The referenced field has changed.
* Delete task document as part of finalization
* Accurately report recoverable errors on create_signals_migration route
If we have a recoverable error: e.g. the destination index already
exists, or a specified index is a write index, we now report those
errors as part of the normal 200 response as these do not preclude other
specified indices from being migrated.
However, if non-signals indices are specified, we do continue to reject
the entire request, as that's indicative of misuse of the endpoint.
2020-12-10 20:12:39 +01:00
await waitFor ( async ( ) = > {
const response = await es . count < { count : number } > ( { index } ) ;
return response . body . count > 0 ;
} , ` waitForIndexToPopulate: ${ index } ` ) ;
} ;
[SecuritySolution][Detections] Adds SavedObject persistence to Signals Migrations (#85690)
* Adds new SO type for persisting our signals migrations
* WIP: Migration status SO client
Trying to forge a patterrn using io-ts to validate at runtime. I think
I've got it working but I want to refactor the pipeline out into a
reusable function(s).
* Implements our SavedObjects service for signals migrations
* Defines a simple client that delegates to the base SO client with
our SO type
* Defines a service that consumes the simpler client, adding validations
and data transforms on top.
* Refactoring migration code to work with saved objects
As opposed to the previous ephemeral, encoded tokens, we now retrieve migration
info from saved objects.
At the API level, this means that both the create and finalize endpoints
receive a list of concrete indices. No more passing around tokens.
As both endpoints are idempotent, users can hammer them as much as they
want with the same lists of indices. Redundant creates and finalizes
will be met with inline 400 messages, and as one continues to poll the
finalize endpoint they should see more and more indices respond with
"completed: true"
* Fixing integration tests first, and anything upstream breaking them
* Clean up API integration tests
* standardize assignment of responses (with types)
* deletes migration SOs as test cleanup
* Split API tests into separate files
This was getting big and unwieldy; this splits these into one file per
endpoint.
* Refactor: split existing migration service functionality into atomic functions
This will allow us to repurpose the service to compose more
functionality and be more specifically useful, while keeping the
component logic separate.
* WIP: moving logic into migrationService.create
* Splitting get_migration_status into component functions
getMigrationStatus was really two separate aggregations, so I split them
out and we recompose them in the necessary routes.
* Move finalization logic into function
* migrationService exposes this as .finalize()
* adds an error field to our migration SO
* We currently only have one error that we persist there, but it would
be very time-consuming to track down that information were it not
there.
* Adds function for migration "deletion" logic
* migrationService leverages this function
* adds new boolean to our savedObject
* deletes unused function (deleteMigrationSavedObject)
* Adds route for soft-deletion of migrations
* Updating tests related to migration status
* Adding/updating mocks/unit tests necessary to satisfy the things I
need to test
* I mainly wanted to test that the the status endpoint filtered out the
deleted migrations; this was accomplished with a unit test after
fleshing out some mocks/sample data.
* Move old migration service tests to the relevant function tests
This logic was previously moved out into component functions; this moves
the tests accordingly.
* Add some unit tests around our reindex call
* Fix create migration route tests
Mocks out our migration functions, rather than stubbing ES calls
directly.
* Updates finalize route unit tests
Addresses functionality that hasn't been moved to finalizeMigration()
* Unit tests our finalization logic
Fixes a bug where we weren't accounting for soft-deleted migrations.
ALso updates our test migration SO to have a status of 'pending' as
that's a more useful default.
* Fixes finalization integration tests
These were failing due:
* a change in the migration status API response
* a bug I introduced in the finalize route
* Adds tests for our migration deletion endpoint
* unit tests
* API integration tests
* Caught/fixed bug with deleting a successful migration
* Fixes types
Removes unused code.
* Prevent race condition due to template rollover during migration
If a user has an out of date index (v1) relative to the template (v2), but the
template itself is out of date (newest is v3), then it's possible that
the template is rolled over to v3 after the v1-v2 migration has been
created but before the new index has been created.
In such a case, the new index would receive the v3 mappings but would
incorrectl be marked as v2. This shouldn't necessarily be an issue, but
it's an unnecessary state that can easily be prevented with the guard
introduced here.
* Add real usernames to migration savedObjects
In addition to the SOs themselves giving us observability into what
migration actions were performed, this gives us the additional info of
_who_ performed the action.
* Index minimal migration SO fields needed for current functionality
* Add additional migration info to status endpoint
This will allow users to finalize a migration if they've lost the
response to their POST call.
* Finalize endpoint receives an array of migration IDs, not indices
This disambiguates _which_ migrations we were finalizing if you passed
an index (which was previously: the most recent migration).
* Fix type errors in tests after we threaded through username
* Update responsibilities of migration finalize/delete endpoints
Discussions with @marshallmain lead to the following refactor:
* finalize does not delete tasks
* finalize only applies cleanup policy to a failed migration
* delete takes an array of migration ids (like finalize)
* delete hard-deletes the SavedObject of a completed (failed or
successful) migration
This gives a bit more flexibility with the endpoints, as well as
disambiguates the semantics: it just deletes migrations!
* Fix tests that were broken during refactoring
* Fix type errors
I removed some logic here but forgot the imports :(
* Move outdated integration test
In the case of a successful migration, application of the cleanup policy
is done by the deletion endpoint. In the interest of data preservation,
we do not delete a sourceIndex unless it is explicitly deleted.
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2020-12-15 10:25:39 +01:00
export const deleteMigrations = async ( {
ids ,
kbnClient ,
} : {
ids : string [ ] ;
kbnClient : KbnClient ;
} ) : Promise < void > = > {
await Promise . all (
ids . map ( ( id ) = >
kbnClient . savedObjects . delete ( {
id ,
type : signalsMigrationType ,
} )
)
) ;
} ;
2021-04-19 22:10:59 +02:00
export const getOpenSignals = async (
supertest : SuperTest < supertestAsPromised.Test > ,
es : KibanaClient ,
rule : FullResponseSchema
) = > {
await waitForRuleSuccessOrStatus ( supertest , rule . id ) ;
// Critically important that we wait for rule success AND refresh the write index in that order before we
// assert that no signals were created. Otherwise, signals could be written but not available to query yet
// when we search, causing tests that check that signals are NOT created to pass when they should fail.
await refreshIndex ( es , rule . output_index ) ;
return getSignalsByIds ( supertest , [ rule . id ] ) ;
} ;