## Summary
Adds support to the end to end (e2e) functional test runner (FTR) support for rule runtime tests as well as 213 tests for the exception lists which include value based lists. Previously we had limited runtime support, but as I scaled up runtime tests from 5 to 200+ I noticed in a lot of areas we had to use improved techniques for determinism.
The runtime support being added is our next step of tests. Up to now most of our e2e FTR tests have been structural testing of REST and API integration tests. Basically up to now 95% tests are API structural as:
* Call REST input related to a rule such as GET/PUT/POST/PATCH/DELETE.
* Check REST output of the rule, did it match expected output body and status code?
* In some rare cases we check if the the rule can be executed and we get a status of 'succeeded'
With only a small part of our tests ~5%, `generating_signals.ts` was checking the signals being produced. However, we cannot have confidence in runtime based tests until the structural tests have been built up and run through the weeks against PR's to ensure that those are stable and deterministic.
Now that we have confidence and 90%+ coverage of the structural REST based tests, we are building up newer sets of tests which allow us to do runtime based validation tests to increase confidence that:
* Detection engine produces signals as expected
* Structure of the signals are as expected, including signal on signals
* Exceptions to signals are working as expected
* Most runtime bugs can be TDD'ed with e2e FTR's and regressions
* Whack-a-mole will not happen
* Consistency and predictability of signals is validated
* Refactoring can occur with stronger confidence
* Runtime tests are reference points for answering questions about existing bugs or adding new ones to test if users are experiencing unexpected behaviors
* Scaling tests can happen without failures
* Velocity for creating tests increases as the utilities and examples increase
Lastly, this puts us within striking distance of creating FTR's for different common class of runtime situations such as:
* Creating tests that exercise each rule against a set of data criteria and get signal hits
* Creating tests that validate the rule overrides operate as expected against data sets
* Creating tests that validate malfunctions, corner cases, or misuse cases such as data sets that are _all_ arrays or data sets that put numbers as strings or throws in an expected `null` instead of a value.
These tests follow the pattern of:
* Add the smallest data set to a folder in data.json (not gzip format)
* Add the smallest mapping to that folder (mapping.json)
* Call REST input related to exception lists, value lists, adding prepackaged rules, etc...
* Call REST input related endpoint with utilities to create and activate the rule
* Wait for the rule to go into the `succeeded` phase
* Wait for the N exact signals specific to that rule to be available
* Check against the set of signals to ensure that the matches are exactly as expected
Example of one runtime test:
A keyword data set is added to a folder called "keyword" but you can add one anywhere you want under `es_archives`, I just grouped mine depending on the situation of the runtime. Small non-gzipped tests `data.json` and `mappings.json` are the best approach for small focused tests. For _larger_ tests and cases I would and sometimes do use things such as auditbeat but try to avoid using larger data sets in favor of smaller focused test cases to validate the runtime is operating as expected.
```ts
{
"type": "doc",
"value": {
"id": "1",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:00:53.000Z",
"long": 1
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "2",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:01:53.000Z",
"long": 2
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "3",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:02:53.000Z",
"long": 3
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "4",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:03:53.000Z",
"long": 4
},
"type": "_doc"
}
}
```
Mapping is added. Note that this is "ECS tolerant" but not necessarily all ECS meaning I can and will try to keep things simple where I can, but I have ensured that `"@timestamp"` is at least there.
```ts
{
"type": "index",
"value": {
"index": "long",
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"long": { "type": "long" }
}
},
"settings": {
"index": {
"number_of_replicas": "1",
"number_of_shards": "1"
}
}
}
}
```
Test is written with test utilities where the `beforeEach` and `afterEach` try and clean up the indexes and load/unload the archives to keep one test from effecting another. Note this is never going to be 100% possible so see below on how we add more determinism in case something escapes the sandbox.
```ts
beforeEach(async () => {
await createSignalsIndex(supertest);
await createListsIndex(supertest);
await esArchiver.load('rule_exceptions/keyword');
});
afterEach(async () => {
await deleteSignalsIndex(supertest);
await deleteAllAlerts(supertest);
await deleteAllExceptions(es);
await deleteListsIndex(supertest);
await esArchiver.unload('rule_exceptions/keyword');
});
describe('"is" operator', () => {
it('should filter 1 single keyword if it is set as an exception', async () => {
const rule = getRuleForSignalTesting(['keyword']);
const { id } = await createRuleWithExceptionEntries(supertest, rule, [
[
{
field: 'keyword',
operator: 'included',
type: 'match',
value: 'word one',
},
],
]);
await waitForRuleSuccess(supertest, id);
await waitForSignalsToBePresent(supertest, 3, [id]);
const signalsOpen = await getSignalsById(supertest, id);
const hits = signalsOpen.hits.hits.map((hit) => hit._source.keyword).sort();
expect(hits).to.eql(['word four', 'word three', 'word two']);
});
});
```
### Changes for better determinism
To support more determinism there are changes and utilities added which can be tuned during any sporadic failures we might encounter as well as better support unexpected changes to other Elastic Stack pieces such as alerting, task manager, etc...
Get simple rule and others are now defaulting to false, meaning that the structural tests will no longer activate a rule and run it on task manger. This should cut down on error outputs as well as reduce stress and potentials for left over rules interfering with the runtime rules.
```ts
export const getSimpleRule = (ruleId = 'rule-1', enabled = false): QueryCreateSchema => ({
```
Not mandatory to use, but for most tests that should be runtime based tests, I use this function below which will enable it by default and run it using settings such as `type: 'query'`, `query: '*:*',` `from: '1900-01-01T00:00:00.000Z'`, to cut down on boiler plate noise. However, people can use whatever they want out of the grab bag or if their test is more readable to hand craft a REST request to create signals, or if they just want to call this and override where they want to, then 👍 .
```ts
export const getRuleForSignalTesting = (index: string[], ruleId = 'rule-1', enabled = true)
```
This waits for a rule to succeed before continuing
```ts
await waitForRuleSuccess(supertest, id);
```
I added a required array of id that _waits_ only for that particular id here. This is useful in case another test did not cleanup and you are getting signals being produced or left behind but need to wait specifically for yours.
```ts
await waitForSignalsToBePresent(supertest, 4, [id]);
```
I only get the signals for a particular rule id using either the auto-generated id or the rule_id. It's safer to use the ones from the auto-generated id but either of these are fine if you're careful enough.
```ts
const signalsOpen = await getSignalsById(supertest, id);
const signalsOpen = await getSignalsByIds(supertest, [createdId]);
const signalsOpen = await getSignalsByRuleIds(supertest, ['signal-on-signal']);
```
I delete all alerts now through a series of steps where it properly removes all rules using the rules bulk_delete and does it in such a way that all the API keys and alerting will be the best it can destroyed as well as double check that the alerts are showing up as being cleaned up before continuing.
```ts
deleteAllAlerts()
```
When not explicitly testing something structural, prefer to use the utilities which can and will do retries in case there are over the wire failures or es failures. Examples are:
```ts
installPrePackagedRules()
waitForRuleSuccess()
importFile() // This does a _lot_ of checks to ensure that the file is fully imported before continuing
```
Some of these utilities might still do a `expect(200);` but as we are and should use regular structural tests to cover those problems, these will probably be more and more removed when/if we hit test failures in favor of doing retries, waitFor, and countDowns.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
5f4c211ea3