* Move intialization to use effect
* Fixing fields can't get test working
* Fix tests
Co-authored-by: Christos Nasikas <christos.nasikas@elastic.co>
Co-authored-by: Jonathan Buttner <56361221+jonathan-buttner@users.noreply.github.com>
Co-authored-by: Christos Nasikas <christos.nasikas@elastic.co>
* [ML] Add API integration tests for start and stop datafeeds
* [ML] Edits to setup and clean-up steps following review
Co-authored-by: Pete Harverson <peteharverson@users.noreply.github.com>
* added initial version of locator
* removed unused params and added jest test
* updated functional test to expect PDF reports to be available when vis is new
* fix TS: remove unkown field
* added some docs and removed unused code
* AggsConfigOption -> AggsConfigSerialized
* moved locator to common
* fixed building of "create" path and updated test snapshots
* updated import
* update encoding behaviour
* added time range from timefilter to locator params request
* add index pattern and search id to URL params
* reading index pattern from search source if it is there for the locator
* remove "type" from locator params, update comments and test
* removed duplicate identifier
* remove unused type
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Adds a workaround for EQL bug: https://github.com/elastic/elasticsearch/issues/77152
Adds the safety feature mentioned here: https://github.com/elastic/kibana/issues/110802
Adds the ability to ignore particular [fields](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-fields.html#search-fields-param) when the field is merged with [_source](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-fields.html#source-filtering). Also fixes an EQL bug where EQL is introducing the meta field of `_ignored` within the fields and causing documents to not be indexable when we merge with the fields from EQL.
Alerting document creation uses the fields API to get [runtime field](https://www.elastic.co/guide/en/elasticsearch/reference/current/runtime.html), [constant keyword](https://www.elastic.co/guide/en/elasticsearch/reference/master/keyword.html#constant-keyword-field-type), etc... that are only available within the [fields API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-fields.html#search-fields-param) and then merges the field values not found within the `_source` document with the `_source` document and then finally indexes this merged document as an alert document.
This fix/ability is a "safety feature" in that if a problematic [runtime field](https://www.elastic.co/guide/en/elasticsearch/reference/current/runtime.html), [constant keyword](https://www.elastic.co/guide/en/elasticsearch/reference/master/keyword.html#constant-keyword-field-type) is discovered or another bug along the stack we can set a `kibana.yml` key/value pair to ignore the problematic field.
This _WILL NOT_ remove problematic fields from the `_source` document. This will only ignore problematic constant keyword, runtime fields, aliases, or anything else found in the fields API that is causing merge issues.
This PR:
* Adds a `alertIgnoreFields` `kibana.yml` array key with a default of an empty array if not specified.
* Plumbs the `alertIgnoreFields` through the stack and into the fields/_source merge strategies of `missingFields` and `allFields`
* Adds a temporary `isEqlBug77152` where it hard codes an ignore of `_ignored` until the EQL problem is fixed and then we will remove the workaround
* Adds unit tests
* Adds e2e tests which covers the described use cases above.
The `alertIgnoreFields` key/value within `kibana.yml` if set should be an array of strings of each field you want to ignore. This can also contain regular expressions as long as they are of the form, `"/regex/"` in the array.
Example if you want to ignore fields that are problematic called "host.name" and then one in which you want to ignore all fields that start with "user." using a regular expression:
```yml
xpack.securitySolution.alertIgnoreFields: ['host.name', '/user\..*/']
```
Although there are e2e tests which exercise the use cases...
If you want to manual test the EQL bug fix you would add these documents in dev tools:
```json
# Delete and add a mapping with a small ignore_above.
DELETE eql-issue-ignore-fields-delme
PUT eql-issue-ignore-fields-delme
{
"mappings" : {
"dynamic": "strict",
"properties" : {
"@timestamp": {
"type": "date"
},
"some_keyword" : {
"ignore_above": 5,
"type" : "keyword"
},
"other_keyword" : {
"ignore_above": 10,
"type" : "keyword"
}
}
}
}
# Add a single document with one field that will be truncated and a second that will not.
PUT eql-issue-ignore-fields-delme/_doc/1
{
"@timestamp": "2021-09-02T04:13:05.626Z",
"some_keyword": "longer than normal",
"other_keyword": "normal"
}
```
Then create an alert which queries everything from it:
<img width="1155" alt="Screen Shot 2021-09-01 at 10 15 06 PM" src="https://user-images.githubusercontent.com/1151048/131781042-faa424cf-65a5-4ebb-b801-3f188940c81d.png">
and ensure signals are created:
<img width="2214" alt="Screen Shot 2021-09-01 at 10 30 18 PM" src="https://user-images.githubusercontent.com/1151048/131782069-b9ab959c-f22d-44d5-baf0-561fe349c037.png">
To test the manual exclusions of any other problematic fields, create any index which has runtime fields or `constant keywords` but does not have anything within the `_source` document using dev tools. For example you can use `constant keyword` like so
```json
PUT constant-keywords-deleme
{
"mappings": {
"dynamic": "strict",
"properties": {
"@timestamp": {
"type": "date"
},
"testing_ignored": {
"properties": {
"constant": {
"type": "constant_keyword",
"value": "constant_value"
}
}
},
"testing_regex": {
"type": "constant_keyword",
"value": "constant_value"
},
"normal_constant": {
"type": "constant_keyword",
"value": "constant_value"
},
"small_field": {
"type": "keyword",
"ignore_above": 10
}
}
}
}
PUT constant-keywords-deleme/_doc/1
{
"@timestamp": "2021-09-02T04:20:01.760Z"
}
```
Set in your `kibana.yml` the key/value of:
```yml
xpack.securitySolution.alertIgnoreFields: ['testing_ignored.constant', '/.*_regex/']
```
Setup a rule to run:
<img width="1083" alt="Screen Shot 2021-09-01 at 10 23 23 PM" src="https://user-images.githubusercontent.com/1151048/131781696-fea0d421-836f-465c-9be6-5289fbb622a4.png">
Once it runs you should notice that the constant values for testing are not on the signals table since it only typically exists in the fields API:
<img width="1166" alt="Screen Shot 2021-09-01 at 10 26 16 PM" src="https://user-images.githubusercontent.com/1151048/131781782-1684fb1d-bed9-4cf0-be9a-0abe1f0f34d1.png">
But the normal one still exists:
<img width="1136" alt="Screen Shot 2021-09-01 at 10 26 31 PM" src="https://user-images.githubusercontent.com/1151048/131781827-5450c693-de9e-4285-b082-9f7a2cbd5d07.png">
If you change the `xpack.securitySolution.alertIgnoreFields` by removing it and re-generate the signals you will see these values added back.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
- [x] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/master/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
Co-authored-by: Frank Hassanabad <frank.hassanabad@elastic.co>
* Step 2: Update client code to use resolve() method instead of get()
Following sharing Saved Objects developer guide: Step 2
This step demonstrates the changes to update client code to use the new
SavedObjectsClient `resolve()` method instead of `get()`.
* Step 3 Lens
Co-authored-by: Marta Bondyra <marta.bondyra@gmail.com>
* Initial commit of serverType in email connector config
* Fleshing in route to get well known email service configs from nodemailer
* Adding elastic cloud to well known server type
* Cleaning up email constants and allowing for empty selection
* Showing error if user doesn't select server type
* Adding hook for setting email config based on server type
* Adding tests and making sure settings are not overwritten on edit
* Fixing functional test
* Adding migration
* Adding functional test for migration
* Repurposing service instead of adding serverType
* Cleanup
* Disabling host/port/secure form fields when settings retrieved from API
* Updating docs for service
* Filtering options based on whether cloud is enabled
* Initialize as disabled
* Fixing types
* Update docs/management/connectors/action-types/email.asciidoc
Co-authored-by: David Kilfoyle <41695641+kilfoyle@users.noreply.github.com>
Co-authored-by: David Kilfoyle <41695641+kilfoyle@users.noreply.github.com>
# Conflicts:
# x-pack/plugins/actions/server/saved_objects/actions_migrations.test.ts
# x-pack/plugins/actions/server/saved_objects/actions_migrations.ts
* first draft(work in progress)
* add back missing await
* disable require_alias flag only when we update
* cleanup
Co-authored-by: mgiota <giota85@gmail.com>
* Use super date picker instead of date range picker
fixes elastic/security-team/issues/1571
* fix test target
Super date picker's `data-test-subj` prop gets garbled and doesn't show up in rendered DOM. In other words, the component is entirely void of a data-test-subj attribute.
* make auto refresh work!!
fixes https://github.com/elastic/security-team/issues/1571
* set max width as per mock
fixes elastic/security-team/issues/1571
* show a callout to inform users to select different date ranges
fixes elastic/security-team/issues/1571
* persist recently used date ranges on the component only
fixes elastic/security-team/issues/1571
* use commonly used ranges from default common security solution ranges
fixes elastic/security-team/issues/1571
* Better align date picker
* full width panel for date picker so content flows below it
review comments
* mock time picker settings for tests
* use eui token for bg color
review comment
* persist recently used dates
fixes elastic/security-team/issues/1571
* persist date range selection over new endpoint selection
review comments
* remove obsolete local state since update button is not visible.
review comments
* fix bg color for dark mode and relative path
* update relative path
review comments
* cleanup - the action doesn't allow for undefined start and end dates anyway
refs 28a859ab3a
* fix types after sync
* update test title
* add a test for callout when empty data
* fix lint
* show update button when dates are changed
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Ashokaditya <am.struktr@gmail.com>
* remove unnecessary ts-ignore
* add context propagation to x-opaque-id header tests
* run tests on CI
* simplify logging. the action purpose follows from the context name
* extend tests with the assertion against execution_context from the Kibana logs
* split JSON log records only
* apply suggestions proposed by Spencer
# Conflicts:
# scripts/functional_tests.js
* Migrationsv2: limit batch sizes to migrations.batchSizeBytes (= 100mb by default) (#109540)
* Fix logging for existing integration test
* First stab at limiting batches to batchSizeBytes
* Fix tests
* Fix batch size calculation, NDJSON needs to be terminated by an empty line
* Integration tests
* Fix type failures
* rename migration integration tests and log files to be consistent & more descriptive
* Review feedback
* Remove duplication of fatal error reasons
* migrations.maxBatchSizeBytes to docker environment vars
* docs for migrations.maxBatchSizeBytes
# Conflicts:
# src/core/server/saved_objects/migrationsv2/integration_tests/7_13_0_unknown_types.test.ts
# src/core/server/saved_objects/migrationsv2/integration_tests/migration_from_v1.test.ts
* Fix tests on 7.x being off by one byte
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Move to vis_types folder part 2
* fix jest tests
* do some tests
* revert
* Test Tiago's fix
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
# Conflicts:
# .github/CODEOWNERS
* Make analyzer work with EuiDataGrid full screen
* Don't ever restrict the width, remove console.log
* Remove isEventViewer prop no longer used
* Make global filters appear below data grid
Co-authored-by: Kevin Qualters <56408403+kqualters-elastic@users.noreply.github.com>
* [Metrics UI] Filter out APM nodes from the inventory view
* Update jest snapshots
* Add tests for fs for filtering out APM nodes
Co-authored-by: Zacqary Adam Xeper <Zacqary@users.noreply.github.com>
With the merge of #98213, pid and log file settings were moved from
command line options to kibana.yml. Package tests use a non-default
kibana.yml and these settings were not applied. This updates our
configuration to include these settings.
Co-authored-by: Jonathan Budzenski <jon@budzenski.me>
* Add animals role to test user for functional tests in dashboard
* Fix two more dashboard tests by applying proper roles
* Restore test user defaults
* Fix one last dashboard test
* forgot security service
* Cleanup
Co-authored-by: Poff Poffenberger <poffdeluxe@gmail.com>
* refactor/reflatten server routes
* fix import
* fix any usage in server/lib
* clean up unused parameter
* remove any in server/browsers
* refactor handle request function into a class
* more cleanup
* [Metrics UI] Add integration tests for Metric Threshold and refactor to fire correctly
* Removing unused variables
* Fixing tests for metric_threshold_executor
* Fixing test for metric_query
* fixing test
* Changing type guard
Co-authored-by: Chris Cowan <chris@chriscowan.us>
