2431a08d2b
**Ticket:** https://github.com/elastic/kibana/issues/106469, https://github.com/elastic/kibana/issues/101013 ## Summary TL;DR: New internal endpoint for reading data from Event Log (raw version), legacy status SO under the hood. With this PR we now read the Failure History (last 5 failures) on the Rule Details page from Event Log. We continue getting the Current Status from the legacy `siem-detection-engine-rule-status` saved objects. Rule Management page also gets data from the legacy saved objects. - [x] Deprecate existing methods for reading data in `IRuleExecutionLogClient`: `.find()` and `.findBulk()` - [x] Introduce new methods for reading data in IRuleExecutionLogClient: - for reading last N execution events for 1 rule from event log - for reading current status and metrics for 1 rule from legacy status SOs - for reading current statuses and metrics for N rules from legacy status SOs - [x] New methods should return data in the legacy status SO format. - [x] Update all the existing endpoints that depend on `IRuleExecutionLogClient` to use the new methods. - [x] Implement a new internal endpoint for fetching current status of the rule execution and execution events from Event Log for a given rule. - [x] The API of the new endpoint should be the same as `rules/_find_statuses` to minimise changes in the app. - [x] Use the new endpoint on the Rule Details page. ## Near-term plan for technical implementation of the Rule Execution Log (https://github.com/elastic/kibana/issues/101013) **Stage 1. Reading last 5 failures from Event Log v1 - raw implementation** - ✔️ done in this PR TL;DR: New internal endpoint for reading data from Event Log (raw version), legacy status SO under the hood. - Deprecate existing methods for reading data in `IRuleExecutionLogClient`: `.find()` and `.findBulk()` - Introduce new methods for reading data in IRuleExecutionLogClient: - for reading last N execution events for 1 rule from event log - for reading current status and metrics for 1 rule from legacy status SOs - for reading current statuses and metrics for N rules from legacy status SOs - New methods should return data in the legacy status SO format. - Update all the existing endpoints that depend on `IRuleExecutionLogClient` to use the new methods. - Implement a new internal endpoint for fetching current status of the rule execution and execution events from Event Log for a given rule. - The API of the new endpoint should be the same as `rules/_find_statuses` to minimise changes in the app. - Use the new endpoint on the Rule Details page. **Stage 2: Reading last 5 failures from Event Log v2 - clean implementation** TL;DR: Clean HTTP API, legacy Rule Status SO under the hood. 🚨🚨🚨 Possible breaking changes in Detections API 🚨🚨🚨 - Design a new data model for the Current Rule Execution Info (the TO-BE new SO type and later the TO-BE data in the rule object itself). - Design a new data model for the Rule Execution Event (read model to be used on the Rule Details page) - Think over changes in `IRuleExecutionLogClient` to support the new data model. - Think over changes in all the endpoints that return any data related to rule monitoring (statuses, metrics, etc). Make sure to check our docs to identify what's documented there regarding rule monitoring. - Update `IRuleExecutionLogClient` to return data in the new format. - Update all the endpoints (including the raw new one) to return data in the new format. - Update Rule Details page to consume data in the new format. - Update Rule Management page to consume data in the new format. **Stage 3: Reading last 5 failures from Event Log v3 - new SO** TL;DR: Clean HTTP API, new Rule Execution Info SO under the hood. - Implement a new SO type for storing the current rule execution info. Relation type: 1 rule - 1 current execution info. - Swap the legacy SO with the new SO in the implementation of `IRuleExecutionLogClient`. **Stage 4: Cleanup and misc** - Revisit the problem of deterministic ordering ([comment](https://github.com/elastic/kibana/pull/115574#discussion_r735803087)) - Remove rule execution log's glue code: adapters, feature switch. - Remove the legacy rule status SO. - Mark the legacy rule status SO as deleted in Kibana Core. - Encapsulate the current space id in the instance of IRuleExecutionLogClient. Remove it from parameters of its methods. - Introduce a Rule Execution Logger scoped to a rule instance. For use in rule executors. - Add test coverage. ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios |
||
|---|---|---|
| .. | ||
| build_chromium | ||
| dev-tools | ||
| examples | ||
| plugins | ||
| scripts | ||
| tasks | ||
| test | ||
| .gitignore | ||
| .i18nrc.json | ||
| .telemetryrc.json | ||
| gulpfile.js | ||
| package.json | ||
| README.md | ||
Elastic License Functionality
This directory tree contains files subject to the Elastic License 2.0. The files subject to the Elastic License 2.0 are grouped in this directory to clearly separate them from files dual-licensed under the Server Side Public License and the Elastic License 2.0.
Development
By default, Kibana will run with X-Pack installed as mentioned in the contributing guide.
Elasticsearch will run with a basic license. To run with a trial license, including security, you can specifying that with the yarn es command.
Example: yarn es snapshot --license trial --password changeme
By default, this will also set the password for native realm accounts to the password provided (changeme by default). This includes that of the kibana_system user which elasticsearch.username defaults to in development. If you wish to specify a password for a given native realm account, you can do that like so: --password.kibana_system=notsecure
Testing
For information on testing, see the Elastic functional test development guide.
Running functional tests
The functional UI tests, the API integration tests, and the SAML API integration tests are all run against a live browser, Kibana, and Elasticsearch install. Each set of tests is specified with a unique config that describes how to start the Elasticsearch server, the Kibana server, and what tests to run against them. The sets of tests that exist today are functional UI tests (specified by this config), API integration tests (specified by this config), and SAML API integration tests (specified by this config).
The script runs all sets of tests sequentially like so:
- builds Elasticsearch and X-Pack
- runs Elasticsearch with X-Pack
- starts up the Kibana server with X-Pack
- runs the functional UI tests against those servers
- tears down the servers
- repeats the same process for the API and SAML API integration test configs.
To do all of this in a single command run:
node scripts/functional_tests
Developing functional UI tests
If you are developing functional tests then you probably don't want to rebuild Elasticsearch and wait for all that setup on every test run, so instead use this command to build and start just the Elasticsearch and Kibana servers:
node scripts/functional_tests_server
After the servers are started, open a new terminal and run this command to run just the tests (without tearing down Elasticsearch or Kibana):
node scripts/functional_test_runner
For both of the above commands, it's crucial that you pass in --config to specify the same config file to both commands. This makes sure that the right tests will run against the right servers. Typically a set of tests and server configuration go together.
Read more about how the scripts work here.
For a deeper dive, read more about the way functional tests and servers work here.
Running API integration tests
API integration tests are run with a unique setup usually without UI assets built for the Kibana server.
API integration tests are intended to test only programmatic API exposed by Kibana. There is no need to run browser and simulate user actions, which significantly reduces execution time. In addition, the configuration for API integration tests typically sets optimize.enabled=false for Kibana because UI assets are usually not needed for these tests.
To run only the API integration tests:
node scripts/functional_tests --config test/api_integration/config
Running SAML API integration tests
We also have SAML API integration tests which set up Elasticsearch and Kibana with SAML support. Run only API integration tests with SAML enabled like so:
node scripts/functional_tests --config test/security_api_integration/saml.config
Running Jest integration tests
Jest integration tests can be used to test behavior with Elasticsearch and the Kibana server.
yarn test:jest_integration
Running Reporting functional tests
See here for more information on running reporting tests.
Running Security Solution Cypress E2E/integration tests
See here for information on running this test suite.