No description
9a459806ad
## Summary Adds a workaround for EQL bug: https://github.com/elastic/elasticsearch/issues/77152 Adds the safety feature mentioned here: https://github.com/elastic/kibana/issues/110802 Adds the ability to ignore particular [fields](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-fields.html#search-fields-param) when the field is merged with [_source](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-fields.html#source-filtering). Also fixes an EQL bug where EQL is introducing the meta field of `_ignored` within the fields and causing documents to not be indexable when we merge with the fields from EQL. Alerting document creation uses the fields API to get [runtime field](https://www.elastic.co/guide/en/elasticsearch/reference/current/runtime.html), [constant keyword](https://www.elastic.co/guide/en/elasticsearch/reference/master/keyword.html#constant-keyword-field-type), etc... that are only available within the [fields API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-fields.html#search-fields-param) and then merges the field values not found within the `_source` document with the `_source` document and then finally indexes this merged document as an alert document. This fix/ability is a "safety feature" in that if a problematic [runtime field](https://www.elastic.co/guide/en/elasticsearch/reference/current/runtime.html), [constant keyword](https://www.elastic.co/guide/en/elasticsearch/reference/master/keyword.html#constant-keyword-field-type) is discovered or another bug along the stack we can set a `kibana.yml` key/value pair to ignore the problematic field. This _WILL NOT_ remove problematic fields from the `_source` document. This will only ignore problematic constant keyword, runtime fields, aliases, or anything else found in the fields API that is causing merge issues. This PR: * Adds a `alertIgnoreFields` `kibana.yml` array key with a default of an empty array if not specified. * Plumbs the `alertIgnoreFields` through the stack and into the fields/_source merge strategies of `missingFields` and `allFields` * Adds a temporary `isEqlBug77152` where it hard codes an ignore of `_ignored` until the EQL problem is fixed and then we will remove the workaround * Adds unit tests * Adds e2e tests which covers the described use cases above. The `alertIgnoreFields` key/value within `kibana.yml` if set should be an array of strings of each field you want to ignore. This can also contain regular expressions as long as they are of the form, `"/regex/"` in the array. Example if you want to ignore fields that are problematic called "host.name" and then one in which you want to ignore all fields that start with "user." using a regular expression: ```yml xpack.securitySolution.alertIgnoreFields: ['host.name', '/user\..*/'] ``` Although there are e2e tests which exercise the use cases... If you want to manual test the EQL bug fix you would add these documents in dev tools: ```json # Delete and add a mapping with a small ignore_above. DELETE eql-issue-ignore-fields-delme PUT eql-issue-ignore-fields-delme { "mappings" : { "dynamic": "strict", "properties" : { "@timestamp": { "type": "date" }, "some_keyword" : { "ignore_above": 5, "type" : "keyword" }, "other_keyword" : { "ignore_above": 10, "type" : "keyword" } } } } # Add a single document with one field that will be truncated and a second that will not. PUT eql-issue-ignore-fields-delme/_doc/1 { "@timestamp": "2021-09-02T04:13:05.626Z", "some_keyword": "longer than normal", "other_keyword": "normal" } ``` Then create an alert which queries everything from it: <img width="1155" alt="Screen Shot 2021-09-01 at 10 15 06 PM" src="https://user-images.githubusercontent.com/1151048/131781042-faa424cf-65a5-4ebb-b801-3f188940c81d.png"> and ensure signals are created: <img width="2214" alt="Screen Shot 2021-09-01 at 10 30 18 PM" src="https://user-images.githubusercontent.com/1151048/131782069-b9ab959c-f22d-44d5-baf0-561fe349c037.png"> To test the manual exclusions of any other problematic fields, create any index which has runtime fields or `constant keywords` but does not have anything within the `_source` document using dev tools. For example you can use `constant keyword` like so ```json PUT constant-keywords-deleme { "mappings": { "dynamic": "strict", "properties": { "@timestamp": { "type": "date" }, "testing_ignored": { "properties": { "constant": { "type": "constant_keyword", "value": "constant_value" } } }, "testing_regex": { "type": "constant_keyword", "value": "constant_value" }, "normal_constant": { "type": "constant_keyword", "value": "constant_value" }, "small_field": { "type": "keyword", "ignore_above": 10 } } } } PUT constant-keywords-deleme/_doc/1 { "@timestamp": "2021-09-02T04:20:01.760Z" } ``` Set in your `kibana.yml` the key/value of: ```yml xpack.securitySolution.alertIgnoreFields: ['testing_ignored.constant', '/.*_regex/'] ``` Setup a rule to run: <img width="1083" alt="Screen Shot 2021-09-01 at 10 23 23 PM" src="https://user-images.githubusercontent.com/1151048/131781696-fea0d421-836f-465c-9be6-5289fbb622a4.png"> Once it runs you should notice that the constant values for testing are not on the signals table since it only typically exists in the fields API: <img width="1166" alt="Screen Shot 2021-09-01 at 10 26 16 PM" src="https://user-images.githubusercontent.com/1151048/131781782-1684fb1d-bed9-4cf0-be9a-0abe1f0f34d1.png"> But the normal one still exists: <img width="1136" alt="Screen Shot 2021-09-01 at 10 26 31 PM" src="https://user-images.githubusercontent.com/1151048/131781827-5450c693-de9e-4285-b082-9f7a2cbd5d07.png"> If you change the `xpack.securitySolution.alertIgnoreFields` by removing it and re-generate the signals you will see these values added back. ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/master/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) |
||
|---|---|---|
| .buildkite | ||
| .ci | ||
| .github | ||
| api_docs | ||
| config | ||
| dev_docs | ||
| docs | ||
| examples | ||
| licenses | ||
| packages | ||
| plugins | ||
| rfcs | ||
| scripts | ||
| src | ||
| tasks/config | ||
| test | ||
| typings | ||
| utilities | ||
| vars | ||
| x-pack | ||
| .backportrc.json | ||
| .bazelignore | ||
| .bazeliskversion | ||
| .bazelrc | ||
| .bazelrc.common | ||
| .bazelversion | ||
| .browserslistrc | ||
| .editorconfig | ||
| .eslintignore | ||
| .eslintrc.js | ||
| .fossa.yml | ||
| .gitattributes | ||
| .gitignore | ||
| .i18nrc.json | ||
| .node-version | ||
| .npmrc | ||
| .nvmrc | ||
| .prettierignore | ||
| .prettierrc | ||
| .stylelintignore | ||
| .stylelintrc | ||
| .telemetryrc.json | ||
| .yarnrc | ||
| api-documenter.json | ||
| BUILD.bazel | ||
| CODE_OF_CONDUCT.md | ||
| CONTRIBUTING.md | ||
| FAQ.md | ||
| github_checks_reporter.json | ||
| Gruntfile.js | ||
| Jenkinsfile | ||
| jest.config.integration.js | ||
| jest.config.js | ||
| kibana.d.ts | ||
| LICENSE.txt | ||
| NOTICE.txt | ||
| package.json | ||
| preinstall_check.js | ||
| README.md | ||
| renovate.json5 | ||
| RISK_MATRIX.mdx | ||
| SECURITY.md | ||
| STYLEGUIDE.mdx | ||
| tsconfig.base.json | ||
| tsconfig.bazel.json | ||
| tsconfig.browser.json | ||
| tsconfig.browser_bazel.json | ||
| tsconfig.json | ||
| tsconfig.types.json | ||
| TYPESCRIPT.md | ||
| WORKSPACE.bazel | ||
| yarn.lock | ||
Kibana
Kibana is your window into the Elastic Stack. Specifically, it's a browser-based analytics and search dashboard for Elasticsearch.
- Getting Started
- Documentation
- Version Compatibility with Elasticsearch
- Questions? Problems? Suggestions?
Getting Started
If you just want to try Kibana out, check out the Elastic Stack Getting Started Page to give it a whirl.
If you're interested in diving a bit deeper and getting a taste of Kibana's capabilities, head over to the Kibana Getting Started Page.
Using a Kibana Release
If you want to use a Kibana release in production, give it a test run, or just play around:
- Download the latest version on the Kibana Download Page.
- Learn more about Kibana's features and capabilities on the Kibana Product Page.
- We also offer a hosted version of Kibana on our Cloud Service.
Building and Running Kibana, and/or Contributing Code
You might want to build Kibana locally to contribute some code, test out the latest features, or try out an open PR:
- CONTRIBUTING.md will help you get Kibana up and running.
- If you would like to contribute code, please follow our STYLEGUIDE.mdx.
- For all other questions, check out the FAQ.md and wiki.
Documentation
Visit Elastic.co for the full Kibana documentation.
For information about building the documentation, see the README in elastic/docs.
Version Compatibility with Elasticsearch
Ideally, you should be running Elasticsearch and Kibana with matching version numbers. If your Elasticsearch has an older version number or a newer major number than Kibana, then Kibana will fail to run. If Elasticsearch has a newer minor or patch number than Kibana, then the Kibana Server will log a warning.
Note: The version numbers below are only examples, meant to illustrate the relationships between different types of version numbers.
| Situation | Example Kibana version | Example ES version | Outcome |
|---|---|---|---|
| Versions are the same. | 5.1.2 | 5.1.2 | 💚 OK |
| ES patch number is newer. | 5.1.2 | 5.1.5 | ⚠️ Logged warning |
| ES minor number is newer. | 5.1.2 | 5.5.0 | ⚠️ Logged warning |
| ES major number is newer. | 5.1.2 | 6.0.0 | 🚫 Fatal error |
| ES patch number is older. | 5.1.2 | 5.1.0 | ⚠️ Logged warning |
| ES minor number is older. | 5.1.2 | 5.0.0 | 🚫 Fatal error |
| ES major number is older. | 5.1.2 | 4.0.0 | 🚫 Fatal error |
Questions? Problems? Suggestions?
- If you've found a bug or want to request a feature, please create a GitHub Issue. Please check to make sure someone else hasn't already created an issue for the same topic.
- Need help using Kibana? Ask away on our Kibana Discuss Forum and a fellow community member or Elastic engineer will be glad to help you out.