* Added deprecation for visualization:colorMapping
* Update text of deprecation
* Update message
* Delete from translations for re-translation
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Added restrictions and type to filtering filed for group by
* Use context instead of consumer
* Added bucket types
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* create kbn-legacy-logging package and start to move things
* fix rotator tests
* fix logging system test mocks
* move logging format to the package
* move logging setup to package
* adapt legacy logging server
* remove usage of legacy config in the legacy logging server
* move legacy logging server to package
* remove `??` syntax from package
* update generated doc
* fix a few things due to month old merge
* remove typings from project
* move reconfigureLogging to package
* add basic README file
* update generated doc
* remove old typings
* add typing for legacy logging events
* remove `??` from packages
* fix / improve event types usages
* remove suffix from tsconfig
* Added default dedupKey value as an {{alertInstanceId}} to provide grouping functionality for PagerDuty incidents. Set default savedObjectId as {{alertInstanceId}} for ServiceNow, Resilient and Jira
* fixed comment
* fixed due to comments
* fixed doc
* fixed due to comments
## Summary
Adds support to the end to end (e2e) functional test runner (FTR) support for rule runtime tests as well as 213 tests for the exception lists which include value based lists. Previously we had limited runtime support, but as I scaled up runtime tests from 5 to 200+ I noticed in a lot of areas we had to use improved techniques for determinism.
The runtime support being added is our next step of tests. Up to now most of our e2e FTR tests have been structural testing of REST and API integration tests. Basically up to now 95% tests are API structural as:
* Call REST input related to a rule such as GET/PUT/POST/PATCH/DELETE.
* Check REST output of the rule, did it match expected output body and status code?
* In some rare cases we check if the the rule can be executed and we get a status of 'succeeded'
With only a small part of our tests ~5%, `generating_signals.ts` was checking the signals being produced. However, we cannot have confidence in runtime based tests until the structural tests have been built up and run through the weeks against PR's to ensure that those are stable and deterministic.
Now that we have confidence and 90%+ coverage of the structural REST based tests, we are building up newer sets of tests which allow us to do runtime based validation tests to increase confidence that:
* Detection engine produces signals as expected
* Structure of the signals are as expected, including signal on signals
* Exceptions to signals are working as expected
* Most runtime bugs can be TDD'ed with e2e FTR's and regressions
* Whack-a-mole will not happen
* Consistency and predictability of signals is validated
* Refactoring can occur with stronger confidence
* Runtime tests are reference points for answering questions about existing bugs or adding new ones to test if users are experiencing unexpected behaviors
* Scaling tests can happen without failures
* Velocity for creating tests increases as the utilities and examples increase
Lastly, this puts us within striking distance of creating FTR's for different common class of runtime situations such as:
* Creating tests that exercise each rule against a set of data criteria and get signal hits
* Creating tests that validate the rule overrides operate as expected against data sets
* Creating tests that validate malfunctions, corner cases, or misuse cases such as data sets that are _all_ arrays or data sets that put numbers as strings or throws in an expected `null` instead of a value.
These tests follow the pattern of:
* Add the smallest data set to a folder in data.json (not gzip format)
* Add the smallest mapping to that folder (mapping.json)
* Call REST input related to exception lists, value lists, adding prepackaged rules, etc...
* Call REST input related endpoint with utilities to create and activate the rule
* Wait for the rule to go into the `succeeded` phase
* Wait for the N exact signals specific to that rule to be available
* Check against the set of signals to ensure that the matches are exactly as expected
Example of one runtime test:
A keyword data set is added to a folder called "keyword" but you can add one anywhere you want under `es_archives`, I just grouped mine depending on the situation of the runtime. Small non-gzipped tests `data.json` and `mappings.json` are the best approach for small focused tests. For _larger_ tests and cases I would and sometimes do use things such as auditbeat but try to avoid using larger data sets in favor of smaller focused test cases to validate the runtime is operating as expected.
```ts
{
"type": "doc",
"value": {
"id": "1",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:00:53.000Z",
"long": 1
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "2",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:01:53.000Z",
"long": 2
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "3",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:02:53.000Z",
"long": 3
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "4",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:03:53.000Z",
"long": 4
},
"type": "_doc"
}
}
```
Mapping is added. Note that this is "ECS tolerant" but not necessarily all ECS meaning I can and will try to keep things simple where I can, but I have ensured that `"@timestamp"` is at least there.
```ts
{
"type": "index",
"value": {
"index": "long",
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"long": { "type": "long" }
}
},
"settings": {
"index": {
"number_of_replicas": "1",
"number_of_shards": "1"
}
}
}
}
```
Test is written with test utilities where the `beforeEach` and `afterEach` try and clean up the indexes and load/unload the archives to keep one test from effecting another. Note this is never going to be 100% possible so see below on how we add more determinism in case something escapes the sandbox.
```ts
beforeEach(async () => {
await createSignalsIndex(supertest);
await createListsIndex(supertest);
await esArchiver.load('rule_exceptions/keyword');
});
afterEach(async () => {
await deleteSignalsIndex(supertest);
await deleteAllAlerts(supertest);
await deleteAllExceptions(es);
await deleteListsIndex(supertest);
await esArchiver.unload('rule_exceptions/keyword');
});
describe('"is" operator', () => {
it('should filter 1 single keyword if it is set as an exception', async () => {
const rule = getRuleForSignalTesting(['keyword']);
const { id } = await createRuleWithExceptionEntries(supertest, rule, [
[
{
field: 'keyword',
operator: 'included',
type: 'match',
value: 'word one',
},
],
]);
await waitForRuleSuccess(supertest, id);
await waitForSignalsToBePresent(supertest, 3, [id]);
const signalsOpen = await getSignalsById(supertest, id);
const hits = signalsOpen.hits.hits.map((hit) => hit._source.keyword).sort();
expect(hits).to.eql(['word four', 'word three', 'word two']);
});
});
```
### Changes for better determinism
To support more determinism there are changes and utilities added which can be tuned during any sporadic failures we might encounter as well as better support unexpected changes to other Elastic Stack pieces such as alerting, task manager, etc...
Get simple rule and others are now defaulting to false, meaning that the structural tests will no longer activate a rule and run it on task manger. This should cut down on error outputs as well as reduce stress and potentials for left over rules interfering with the runtime rules.
```ts
export const getSimpleRule = (ruleId = 'rule-1', enabled = false): QueryCreateSchema => ({
```
Not mandatory to use, but for most tests that should be runtime based tests, I use this function below which will enable it by default and run it using settings such as `type: 'query'`, `query: '*:*',` `from: '1900-01-01T00:00:00.000Z'`, to cut down on boiler plate noise. However, people can use whatever they want out of the grab bag or if their test is more readable to hand craft a REST request to create signals, or if they just want to call this and override where they want to, then 👍 .
```ts
export const getRuleForSignalTesting = (index: string[], ruleId = 'rule-1', enabled = true)
```
This waits for a rule to succeed before continuing
```ts
await waitForRuleSuccess(supertest, id);
```
I added a required array of id that _waits_ only for that particular id here. This is useful in case another test did not cleanup and you are getting signals being produced or left behind but need to wait specifically for yours.
```ts
await waitForSignalsToBePresent(supertest, 4, [id]);
```
I only get the signals for a particular rule id using either the auto-generated id or the rule_id. It's safer to use the ones from the auto-generated id but either of these are fine if you're careful enough.
```ts
const signalsOpen = await getSignalsById(supertest, id);
const signalsOpen = await getSignalsByIds(supertest, [createdId]);
const signalsOpen = await getSignalsByRuleIds(supertest, ['signal-on-signal']);
```
I delete all alerts now through a series of steps where it properly removes all rules using the rules bulk_delete and does it in such a way that all the API keys and alerting will be the best it can destroyed as well as double check that the alerts are showing up as being cleaned up before continuing.
```ts
deleteAllAlerts()
```
When not explicitly testing something structural, prefer to use the utilities which can and will do retries in case there are over the wire failures or es failures. Examples are:
```ts
installPrePackagedRules()
waitForRuleSuccess()
importFile() // This does a _lot_ of checks to ensure that the file is fully imported before continuing
```
Some of these utilities might still do a `expect(200);` but as we are and should use regular structural tests to cover those problems, these will probably be more and more removed when/if we hit test failures in favor of doing retries, waitFor, and countDowns.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
* [APM] Only fetch custom links when user has valid license
* Rename more files for consistency
* i18n fixes
* Fix tests
* Addess feedback and minor cleanup
* Fix test (and more cleanup)
* Make cachable
resolves#55634resolves#65746
Buffers event docs being written for a fixed interval / buffer size,
and indexes those docs via a bulk ES call.
Also now flushing those buffers at plugin stop() time, which
we couldn't do before with the single index calls, which were
run via `setImmediate()`.
This is a redo of PR https://github.com/elastic/kibana/pull/80941 which
had to be reverted.
* Initial agent log table
* Add data and storage services to Kibana context, rename useCore to useStartServices and replace usage of useStartDeps
* Initial attempt at adding query bar for log stream
* Adjust app layout to allow page content to be full height
* Dataset and log level filters, split into smaller files
* Use data plugin's `QueryStringInput` component for query bar
* Add open in Logs UI link
* Clean up unused storage dependency
* Remove agent activity log and related dead code
* Clean up i18n
* Clean up plugin deps, fix routing
* Add back storage dependency that data plugin components need
* Remove dependency on infra for logs UI link
* Change default date range to last one day
* Adjust panel padding
* Add comment
* Move module declarations to top-level x-pack/typings
* Fix missed renaming
* Remove unused path
* Move building of log stream query into separate service with unit tests
* Adjust test conditions
* update getPackgeInfo handler to fetch from install source
* add tests and modify fixtures to distinguish between registry and uploaded package
* improve error handling
* fix type
* fix test
* remove try/catch
* fix zip file test to have the right number of assets
* fix compressed files
* get all jobs from index node
* create map from modelId and enable url share
* highlight source node
* add map endpoint to api doc
* use variables in css.fix types.ensure map tab is shown
* fix translations
* Create Policies for each generated host
* Refactor Ingest setup to also setup Fleet
* Rename prop name
* Add generic response type to KbnClient.request + support for headers
* first attempt at adding fleet agent registration
* a little closer with fleet integration
* SUCCESS. Able to enroll agent and set it to online
* update names to be policy
* policy generator has advanced types in endpoint confit
* linting
* flesh out callback
* add submit button for verify_peer
* add verify hostname field
* 145 generalize cb
* 145 fix setAgain and getValue
* 145 merge conflict
* 145 add verify_hostname back, start loop for form
* 145 remove OS trick
* 145 make AdvancedPolicyForms its own component
* 145 grid partially working
* 145 back to basics
* 145 back to basics
* 145 rolled back grid
* 145 flex table working
* 145 undo accidental change
* 145 remove extra schema file
* 145 remove unused variable
* 145 kevin's PR feedback
* 145 fix type check and jest
* 145 EuiFlexGroups
* 145 use simple EuiFormRow and add show/hide buttons
* 145 move all advanced policy code to advanced file; remove unnec test code
* 145 fix IDs
* 145 take out unnecessary stuff
* 145 removed a couple more lines
* 145 add some fields back in
* 145 add spacer
* 145 start tests
* 145 add findAdvancedPolicyButton
* 145 test passing
* 145 remove comment
Co-authored-by: Paul Tavares <paul.tavares@elastic.co>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: kevinlog <kevin.logan@elastic.co>
Co-authored-by: Candace Park <candace.park@elastic.co>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Unskipping test
* Unskipping test
* Making action group name lookup dynamic
* Removing comment
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
