* Implement exceptions for ML rules
* Remove unused import
* Better implicit types
* Retrieve ML rule index pattern for exception field suggestions and autocomplete
* Add ML job logic to edit exception modal
* Remove unnecessary logic change
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Endpoint: add `withSecurityContext` HOC + refactor endpoint policy edit lazy component to use it
* Endpoint: refactor Policy Details to separate form from view
* Endpoint: Enable the Redux store for the Policy form when displayed via Fleet
* Fleet: Allow partial package policy updates to be sent via `onChange()`
* Changed the text for the description text in the antivirus registration form. Moved the form component to components folder and extracted translations into constants to make code more readable.
* Extracted EventsForm to reduce duplication among events forms.
This PR changes the behavior of severity and risk score overrides in two ways:
- adds support for arrays in the mapped event fields (so a rule can be triggered by an event where e.g. `event.custom_severity` has a value like `[45, 70, 90]`)
- makes the logic of overrides more flexible, resilient to the incoming values (filters out junk, extracts meaningful values, does its best to find a value that would fit the mapping)
* Do not render out-of-box integration policy configuration step if a custom UI extension is registered
* Remove endpoint specific logic from fleet and move it to UI extension
* Support for rendering a custom component in Integration Details
* Refactor Fleet app initialization contexts in order to support testing setup
* New test rendering helper tool
* refactor Endpoint to use mock builder from Fleet
* Separated out service layer for trusted apps.
* Improved the type structure a bit to avoid using explicit string literals and to add possibility to return OS specific parts of trusted app object in type safe manner.
* Added support for mapping of trusted app to exception item and back.
* Changed schema to support signer in the API.
* Renamed utils to mapping.
* Exported some types in lists plugin and used them in trusted apps.
* Added tests for mapping.
* Added tests for service.
* Switched deletion to use exceptions for not found case.
* Added resetting of the mocks in service layer tests.
* Added handlers tests.
* Refactored mapping tests to be more granular based on the case.
* Restored lowercasing of hash.
* Added schema tests for signer field.
* Removed the grouped tests (they were split into tests for separate concerns).
* Corrected the tests.
* Lowercased the hashes in the service test.
* Moved the lowercasing to the right location.
* Fixed the tests.
* Added test for lowercasing hash value.
* Introduced OperatingSystem enum instead of current types.
* Removed os list constant in favour of separate lists in places that use it (each place has own needs to the ordering).
* Fixed the missed OperatingSystem enum usage.
* improves 'Creates and activates a new custom rule' test
* fixes constant problem
* improves 'Creates and activates a new custom rule with override option' test
* improves 'Creates and activates a new threshold rule' test
* refactor
* fixes type check issue
* improves assertions
* removes unused code
* changes variables for constants
* improves 'waitForTheRuleToBeExecuted' test
* improves readability
* fixes jenkins error
* refactor
* blah
* more things
* finishes 'Creates an exception from rule details and deletes the excpetion' implementation
* implements 'Creates an exception from an alert and deletes the exception'
* updates VALUES_INPUT locator
* updates archiver
* refactor
* improves the code
* fixes CI error
* renames exceptions archive
* refactor
* fixes merge issue
* fixes CI issue
* debug
* refactor
* improves test data
* removes signals index after the execution
* removes unused line
* removes unused variable
* refactors 'numberOfauditbeatExceptionsAlerts' constant to camel case
* simplifies the archive
* waits for the rule to be executed after navigating to opened alerts tab
* cleaning data
* fixes tests flakiness
* cleans test data
* refactors code
* removes unsused archives
* cleans data
* simplifies data
* fixes CI issue
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Fix threshold rule synthetic signal generation
* Use top_hits aggregation
* Find signals and aggregate over search terms
* Exclude dupes
* Fixes to algorithm
* Sync timestamps with events/signals
* Add timestampOverride
* Revert changes in signal creation
* Simplify query, return 10k buckets
* Account for when threshold.field is not supplied
* Ensure we're getting the last event when threshold.field is not provided
* Add missing import
* Handle case where threshold field not supplied
* Fix type errors
* Handle non-ECS fields
* Regorganize
* Address comments
* Fix type error
* Add unit test for buildBulkBody on threshold results
* Add threshold_count back to mapping (and deprecate)
* Timestamp fixes
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Refactoring entity route to return schema
* Refactoring frontend middleware to pick off id field from entity route
* Refactoring schema and adding name and comments
* Adding name to schema mocks
* Fixing type issue
* Prevent error from being displayed when choosing action throttle
Addresses #83230.
This field was previously refactored to not require a callback prop;
simply updating the value via `field.setValue` is sufficient for our use
case.
This fix removes the errant code that assumed a callback prop, since
such a prop does not exist on the underlying component.
* Fix UI bug on ML Jobs popover
EUI links now add an SVG if they're an external link; our use of a div
was causing that to wrap. Since the div was only needed to change the
text size, a refactor makes this all work.
* Exercise editing of tags in E2E tests
These tests were recently skipped due to their improper teardown. Since
that's a broader issue across most of these tests, I'm reopening these
so we can get the coverage provided here for the time being.
* useFetchIndex defaults to isLoading: false
In the case where no index pattern is provided, the hook exits without
doing any work but does not update the loading state; this had the
downstream effect of disabling a form field that was waiting for this
hook to stop loading.
* Move situational action into ... the situation
We only need to clear existing tags in the case where we're editing the
rule (and it has tags); in all other cases, this method fails. This
fixes things by moving that conditional logic (clear the tags field)
into the test that needs it (editing custom rules).
* Trying to flesh out new tree route
* Working on the descendants query
* Almost working descendants
* Possible solution for aggs
* Working aggregations extraction
* Working on the ancestry array for descendants
* Making changes to the unique id for ancestr
* Implementing ancestry funcitonality
* Deleting the multiple edges
* Fleshing out the descendants loop for levels
* Writing tests for ancestors and descendants
* Fixing type errors and writing more tests
* Renaming validation variable and deprecating old tree routes
* Renaming tree integration test file
* Adding some integration tests
* Fixing ancestry to handle multiple nodes in the request and writing more tests
* Adding more tests
* Renaming new tree to handler file
* Renaming new tree directory
* Adding more unit tests
* Using doc value fields and working on types
* Adding comments and more tests
* Fixing timestamp test issue
* Adding more comments
* Fixing timestamp test issue take 2
* Adding id, parent, and name fields to the top level response
* Fixing generator start and end time generation
* Adding more comments
* Revert "Fixing generator start and end time generation"
This reverts commit 9e9abf68a6.
* Adding test for time
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* bump version to 4.1.1-rc
* fix code to run kbn bootstrap
* fix errors
* DO NOT MERGE. mute errors and ping teams to fix them
* Address EuiSelectableProps configuration in discover sidebar
* use explicit type for EuiSelectable
* update to ts v4.1.2
* fix ts error in EuiSelectable
* update docs
* update prettier with ts version support
* Revert "update prettier with ts version support"
This reverts commit 3de48db3ec.
* address another new problem
Co-authored-by: Chandler Prall <chandler.prall@gmail.com>
* move streams to kbn/std
* import streams from kbn/std
* fix styles
* remove unused shareWeakReplay
* move from kbn/std to kbn/utils
* import from subfolder since test mocks FS module and not compatible with kbn/utils
* remove new line at the end of json file
* Replaced AppContextProvider introduced by the plugin with KibanaContextProvider
* Removed unused files
* Fixed jest test
* Removed ActionsConnectorContext
* exposed addConnectorFlyout and editConnectorFlyouts as a plugin start result
* removed rest of unused connectors context
* fixed capabilities
* fixed jest tests
* fixed jest tests
* fixed jest tests
* fixed uptime
* fixed typecheck errors
* fixed typechecks
* fixed jest tests
* fixed type
* fixed uptime settings by pathing the correct plugin dependancy
* fixed security detection rules
* fixed due to commetns
* fixed jest tests
* fixed type check
* removed orig files
* fixed cases UI issues
* fixed due to comments
* fixed due to comments
* fixed kibana crash
* fixed es-lint
## Summary
This PR addresses a bug found where EQL preview was breaking. The preview was showing no hits, though the API response clearly had hits. Because EQL does not support aggregations, aggregations for preview were done manually, client side and missed this case where `@timestamp` is returned as unix epoch. We used `Date.parse` which expects a string, so when inserting a unix timestamp it returned `NaN`.
## Summary
Fixes flakey tests by adding explicit list value upload items through either the fixture that was uploaded or by a specific test value in case the uploaded list is a range value. Also filters out any empty values for more safeguards from prettier formatters that add them to fixture files.
https://github.com/elastic/kibana/issues/84014
### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
## Summary
Adds support to the end to end (e2e) functional test runner (FTR) support for rule runtime tests as well as 213 tests for the exception lists which include value based lists. Previously we had limited runtime support, but as I scaled up runtime tests from 5 to 200+ I noticed in a lot of areas we had to use improved techniques for determinism.
The runtime support being added is our next step of tests. Up to now most of our e2e FTR tests have been structural testing of REST and API integration tests. Basically up to now 95% tests are API structural as:
* Call REST input related to a rule such as GET/PUT/POST/PATCH/DELETE.
* Check REST output of the rule, did it match expected output body and status code?
* In some rare cases we check if the the rule can be executed and we get a status of 'succeeded'
With only a small part of our tests ~5%, `generating_signals.ts` was checking the signals being produced. However, we cannot have confidence in runtime based tests until the structural tests have been built up and run through the weeks against PR's to ensure that those are stable and deterministic.
Now that we have confidence and 90%+ coverage of the structural REST based tests, we are building up newer sets of tests which allow us to do runtime based validation tests to increase confidence that:
* Detection engine produces signals as expected
* Structure of the signals are as expected, including signal on signals
* Exceptions to signals are working as expected
* Most runtime bugs can be TDD'ed with e2e FTR's and regressions
* Whack-a-mole will not happen
* Consistency and predictability of signals is validated
* Refactoring can occur with stronger confidence
* Runtime tests are reference points for answering questions about existing bugs or adding new ones to test if users are experiencing unexpected behaviors
* Scaling tests can happen without failures
* Velocity for creating tests increases as the utilities and examples increase
Lastly, this puts us within striking distance of creating FTR's for different common class of runtime situations such as:
* Creating tests that exercise each rule against a set of data criteria and get signal hits
* Creating tests that validate the rule overrides operate as expected against data sets
* Creating tests that validate malfunctions, corner cases, or misuse cases such as data sets that are _all_ arrays or data sets that put numbers as strings or throws in an expected `null` instead of a value.
These tests follow the pattern of:
* Add the smallest data set to a folder in data.json (not gzip format)
* Add the smallest mapping to that folder (mapping.json)
* Call REST input related to exception lists, value lists, adding prepackaged rules, etc...
* Call REST input related endpoint with utilities to create and activate the rule
* Wait for the rule to go into the `succeeded` phase
* Wait for the N exact signals specific to that rule to be available
* Check against the set of signals to ensure that the matches are exactly as expected
Example of one runtime test:
A keyword data set is added to a folder called "keyword" but you can add one anywhere you want under `es_archives`, I just grouped mine depending on the situation of the runtime. Small non-gzipped tests `data.json` and `mappings.json` are the best approach for small focused tests. For _larger_ tests and cases I would and sometimes do use things such as auditbeat but try to avoid using larger data sets in favor of smaller focused test cases to validate the runtime is operating as expected.
```ts
{
"type": "doc",
"value": {
"id": "1",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:00:53.000Z",
"long": 1
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "2",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:01:53.000Z",
"long": 2
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "3",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:02:53.000Z",
"long": 3
},
"type": "_doc"
}
}
{
"type": "doc",
"value": {
"id": "4",
"index": "long",
"source": {
"@timestamp": "2020-10-28T05:03:53.000Z",
"long": 4
},
"type": "_doc"
}
}
```
Mapping is added. Note that this is "ECS tolerant" but not necessarily all ECS meaning I can and will try to keep things simple where I can, but I have ensured that `"@timestamp"` is at least there.
```ts
{
"type": "index",
"value": {
"index": "long",
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
},
"long": { "type": "long" }
}
},
"settings": {
"index": {
"number_of_replicas": "1",
"number_of_shards": "1"
}
}
}
}
```
Test is written with test utilities where the `beforeEach` and `afterEach` try and clean up the indexes and load/unload the archives to keep one test from effecting another. Note this is never going to be 100% possible so see below on how we add more determinism in case something escapes the sandbox.
```ts
beforeEach(async () => {
await createSignalsIndex(supertest);
await createListsIndex(supertest);
await esArchiver.load('rule_exceptions/keyword');
});
afterEach(async () => {
await deleteSignalsIndex(supertest);
await deleteAllAlerts(supertest);
await deleteAllExceptions(es);
await deleteListsIndex(supertest);
await esArchiver.unload('rule_exceptions/keyword');
});
describe('"is" operator', () => {
it('should filter 1 single keyword if it is set as an exception', async () => {
const rule = getRuleForSignalTesting(['keyword']);
const { id } = await createRuleWithExceptionEntries(supertest, rule, [
[
{
field: 'keyword',
operator: 'included',
type: 'match',
value: 'word one',
},
],
]);
await waitForRuleSuccess(supertest, id);
await waitForSignalsToBePresent(supertest, 3, [id]);
const signalsOpen = await getSignalsById(supertest, id);
const hits = signalsOpen.hits.hits.map((hit) => hit._source.keyword).sort();
expect(hits).to.eql(['word four', 'word three', 'word two']);
});
});
```
### Changes for better determinism
To support more determinism there are changes and utilities added which can be tuned during any sporadic failures we might encounter as well as better support unexpected changes to other Elastic Stack pieces such as alerting, task manager, etc...
Get simple rule and others are now defaulting to false, meaning that the structural tests will no longer activate a rule and run it on task manger. This should cut down on error outputs as well as reduce stress and potentials for left over rules interfering with the runtime rules.
```ts
export const getSimpleRule = (ruleId = 'rule-1', enabled = false): QueryCreateSchema => ({
```
Not mandatory to use, but for most tests that should be runtime based tests, I use this function below which will enable it by default and run it using settings such as `type: 'query'`, `query: '*:*',` `from: '1900-01-01T00:00:00.000Z'`, to cut down on boiler plate noise. However, people can use whatever they want out of the grab bag or if their test is more readable to hand craft a REST request to create signals, or if they just want to call this and override where they want to, then 👍 .
```ts
export const getRuleForSignalTesting = (index: string[], ruleId = 'rule-1', enabled = true)
```
This waits for a rule to succeed before continuing
```ts
await waitForRuleSuccess(supertest, id);
```
I added a required array of id that _waits_ only for that particular id here. This is useful in case another test did not cleanup and you are getting signals being produced or left behind but need to wait specifically for yours.
```ts
await waitForSignalsToBePresent(supertest, 4, [id]);
```
I only get the signals for a particular rule id using either the auto-generated id or the rule_id. It's safer to use the ones from the auto-generated id but either of these are fine if you're careful enough.
```ts
const signalsOpen = await getSignalsById(supertest, id);
const signalsOpen = await getSignalsByIds(supertest, [createdId]);
const signalsOpen = await getSignalsByRuleIds(supertest, ['signal-on-signal']);
```
I delete all alerts now through a series of steps where it properly removes all rules using the rules bulk_delete and does it in such a way that all the API keys and alerting will be the best it can destroyed as well as double check that the alerts are showing up as being cleaned up before continuing.
```ts
deleteAllAlerts()
```
When not explicitly testing something structural, prefer to use the utilities which can and will do retries in case there are over the wire failures or es failures. Examples are:
```ts
installPrePackagedRules()
waitForRuleSuccess()
importFile() // This does a _lot_ of checks to ensure that the file is fully imported before continuing
```
Some of these utilities might still do a `expect(200);` but as we are and should use regular structural tests to cover those problems, these will probably be more and more removed when/if we hit test failures in favor of doing retries, waitFor, and countDowns.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
* Create Policies for each generated host
* Refactor Ingest setup to also setup Fleet
* Rename prop name
* Add generic response type to KbnClient.request + support for headers
* first attempt at adding fleet agent registration
* a little closer with fleet integration
* SUCCESS. Able to enroll agent and set it to online
* update names to be policy
* policy generator has advanced types in endpoint confit
* linting
* flesh out callback
* add submit button for verify_peer
* add verify hostname field
* 145 generalize cb
* 145 fix setAgain and getValue
* 145 merge conflict
* 145 add verify_hostname back, start loop for form
* 145 remove OS trick
* 145 make AdvancedPolicyForms its own component
* 145 grid partially working
* 145 back to basics
* 145 back to basics
* 145 rolled back grid
* 145 flex table working
* 145 undo accidental change
* 145 remove extra schema file
* 145 remove unused variable
* 145 kevin's PR feedback
* 145 fix type check and jest
* 145 EuiFlexGroups
* 145 use simple EuiFormRow and add show/hide buttons
* 145 move all advanced policy code to advanced file; remove unnec test code
* 145 fix IDs
* 145 take out unnecessary stuff
* 145 removed a couple more lines
* 145 add some fields back in
* 145 add spacer
* 145 start tests
* 145 add findAdvancedPolicyButton
* 145 test passing
* 145 remove comment
Co-authored-by: Paul Tavares <paul.tavares@elastic.co>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: kevinlog <kevin.logan@elastic.co>
Co-authored-by: Candace Park <candace.park@elastic.co>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* shell scripts for creating roles + users for testing
* update readme's and updated privilege requirements based on testing with the users and inferring what the roles are supposed to do
* update role privileges based on feedback meeting yesterday
* updated scripts to accept filepath to role / user, added a test to ensure upload value list button is disabled
* updated role scripts to be parameterized
* adds login with role function and adds a sample test with a role to test that a t1 analyst user cannot upload a value list
* add object with corresponding roles
* fix spacing
* parameterize urls for basic auth with roles + users
* forgot to change the cy.visit string
* add KIBANA_URL env var for cli runner
* add env vars for curl script execution
* second script
* update readme's for each role and remove create_index from lists privilege for the soc manager role
* remove 'manage' cluster privilege for rule author
* remove 'create_index' privilege from soc_manager role since that is not parity with the security workflows spreadsheet
* update the login function logic with glo's feedback
* replace SIEM with Security Solution in markdown files
* make role param optional not just undefined
* remove unused file
* add copyright to scripts files
* update top-level README for roles scripts
* remove reference to internal spreadsheet and reference readme for this pr
* remove unnecessary -XPOST and remove verbose mode from post_detections_user script
* adds utils for running integration tests with other users and adds two sample tests showing example usage
* minor type updates and small refactor
* fix x-pack/test types
* use enum types instead of custom type
* fix path to json
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Xavier Mouligneau <189600+XavierM@users.noreply.github.com>
