Code deduplication:
- combine 3 existing client side hooks into useSearchStrategy
- combine server side multiple search strategy files into shared search_strategy_provider.ts
- Clarified naming (client/server params, strategy naming etc.), improved types.
- None of the actual deeper internal logic changed, larger chunks of code that show up as new lines is mostly just moved code + additional types (e.g. function overloads) to support the different search strategies with the same server side code and client side hook.
* React version of angular license view
* Fix time handling and linking to license upload
* Use getPageData pattern instead of useClusters
* Add note about locked time picker
* Add disable support to monitoring toolbar
* Disable toolbar on license page only
* Remove old todo
* Clean up render setup method ordering
* Fix CI checks
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* [sample data] update web log geo.src field to match country code of geo.coordinates
* fix functional tests
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Working test that validate migrated artifact has same properties as SO artifact
* Checks if artifact is compressed and uncompress it if necessary before creating the new one from fleet
Co-authored-by: Paul Tavares <paul.tavares@elastic.co>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* [Maps] fix term join not updating when editing right field
* tslint
* clean up
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* removed anys and ran TS organize imports
* updated jest snapshots
* fix import paths for non-type imports
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
We can't display `response.totalCount` because it is the total number of
events the query returns. It doesn't take into account the aggregation.
It does include events with missing `stackedByField` and events that are
not included in the 10 top.
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* redirect to page adding transaction type
* skipping transaction type
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* trim comments so empty comments do not show up
fixes elastic/kibana/issues/111106
* not exclusive test
* update test to be more specific
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* first iteration of canvas reporting using v2 PDF generator
* updated jest test
* made v2 report URLs compatible with spaces and simplified some code
* remove non-existent import
* updated import of lib
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Added `tagCloud` to canvas.
* Added `icon` to the `tagCloud` element.
* Added column name support at `tag_cloud`.
* Added condition to `vis_dimension` not to pass invalid index.
Added check of accessor index, if such column exists at vis_dimension.
Removed checks of column existance from TagCloudChart.
Added test for accessing data by column name in addition to a column number.
Updated tag_cloud element in Canvas.
Fixed types. Removed almost all `any` and `as` types.
* Added test suites for `vis_dimension` function.
* Added tests for DatatableColumn accessors at tag_cloud_fn and to_ast.
* Refactored metrics, tagcloud and tests.
Added valid functional tests to metrics and tag_cloud.
Fixed types of metrics_vis.
Added handling of empty data at tag_cloud renderer.
* Added storybook ( still doesn't work ).
* Fixed some mistakes.
* Added working storybook with mocks.
* Added clear storybook for tag_cloud_vis_renderer.
* Updated the location of vis_dimension test after movement of the function.
* Fixed unused type.
* Fixed tests and added handling of the column name at `visualizations/**/*/prepare_log_table.ts`
* Reduced the complexity of checking the accessor at `tag_cloud_chart.tsx`
* Added comments at unclear places of code.
* Added the logic for disabling elements for renderers from disabled plugins.
* removed garbage from `kibana.yml`.
* Fixed element_strings.test error.
* Made changes, based on nits.
* Fixed mistake.
* Removed `disabled` flag for `expression_*` plugins.
* recovered lost comments at the unclear places.
* removed dead code.
* fixed test errors.
* Fixed test error, I hope.
* fixed more tests.
* fixed code, based on nits.
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* added initial version of locator
* removed unused params and added jest test
* updated functional test to expect PDF reports to be available when vis is new
* fix TS: remove unkown field
* added some docs and removed unused code
* AggsConfigOption -> AggsConfigSerialized
* moved locator to common
* fixed building of "create" path and updated test snapshots
* updated import
* update encoding behaviour
* added time range from timefilter to locator params request
* add index pattern and search id to URL params
* reading index pattern from search source if it is there for the locator
* remove "type" from locator params, update comments and test
* removed duplicate identifier
* remove unused type
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Step 2: Update client code to use resolve() method instead of get()
Following sharing Saved Objects developer guide: Step 2
This step demonstrates the changes to update client code to use the new
SavedObjectsClient `resolve()` method instead of `get()`.
* Step 3 Lens
## Summary
Adds a workaround for EQL bug: https://github.com/elastic/elasticsearch/issues/77152
Adds the safety feature mentioned here: https://github.com/elastic/kibana/issues/110802
Adds the ability to ignore particular [fields](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-fields.html#search-fields-param) when the field is merged with [_source](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-fields.html#source-filtering). Also fixes an EQL bug where EQL is introducing the meta field of `_ignored` within the fields and causing documents to not be indexable when we merge with the fields from EQL.
Alerting document creation uses the fields API to get [runtime field](https://www.elastic.co/guide/en/elasticsearch/reference/current/runtime.html), [constant keyword](https://www.elastic.co/guide/en/elasticsearch/reference/master/keyword.html#constant-keyword-field-type), etc... that are only available within the [fields API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-fields.html#search-fields-param) and then merges the field values not found within the `_source` document with the `_source` document and then finally indexes this merged document as an alert document.
This fix/ability is a "safety feature" in that if a problematic [runtime field](https://www.elastic.co/guide/en/elasticsearch/reference/current/runtime.html), [constant keyword](https://www.elastic.co/guide/en/elasticsearch/reference/master/keyword.html#constant-keyword-field-type) is discovered or another bug along the stack we can set a `kibana.yml` key/value pair to ignore the problematic field.
This _WILL NOT_ remove problematic fields from the `_source` document. This will only ignore problematic constant keyword, runtime fields, aliases, or anything else found in the fields API that is causing merge issues.
This PR:
* Adds a `alertIgnoreFields` `kibana.yml` array key with a default of an empty array if not specified.
* Plumbs the `alertIgnoreFields` through the stack and into the fields/_source merge strategies of `missingFields` and `allFields`
* Adds a temporary `isEqlBug77152` where it hard codes an ignore of `_ignored` until the EQL problem is fixed and then we will remove the workaround
* Adds unit tests
* Adds e2e tests which covers the described use cases above.
The `alertIgnoreFields` key/value within `kibana.yml` if set should be an array of strings of each field you want to ignore. This can also contain regular expressions as long as they are of the form, `"/regex/"` in the array.
Example if you want to ignore fields that are problematic called "host.name" and then one in which you want to ignore all fields that start with "user." using a regular expression:
```yml
xpack.securitySolution.alertIgnoreFields: ['host.name', '/user\..*/']
```
Although there are e2e tests which exercise the use cases...
If you want to manual test the EQL bug fix you would add these documents in dev tools:
```json
# Delete and add a mapping with a small ignore_above.
DELETE eql-issue-ignore-fields-delme
PUT eql-issue-ignore-fields-delme
{
"mappings" : {
"dynamic": "strict",
"properties" : {
"@timestamp": {
"type": "date"
},
"some_keyword" : {
"ignore_above": 5,
"type" : "keyword"
},
"other_keyword" : {
"ignore_above": 10,
"type" : "keyword"
}
}
}
}
# Add a single document with one field that will be truncated and a second that will not.
PUT eql-issue-ignore-fields-delme/_doc/1
{
"@timestamp": "2021-09-02T04:13:05.626Z",
"some_keyword": "longer than normal",
"other_keyword": "normal"
}
```
Then create an alert which queries everything from it:
<img width="1155" alt="Screen Shot 2021-09-01 at 10 15 06 PM" src="https://user-images.githubusercontent.com/1151048/131781042-faa424cf-65a5-4ebb-b801-3f188940c81d.png">
and ensure signals are created:
<img width="2214" alt="Screen Shot 2021-09-01 at 10 30 18 PM" src="https://user-images.githubusercontent.com/1151048/131782069-b9ab959c-f22d-44d5-baf0-561fe349c037.png">
To test the manual exclusions of any other problematic fields, create any index which has runtime fields or `constant keywords` but does not have anything within the `_source` document using dev tools. For example you can use `constant keyword` like so
```json
PUT constant-keywords-deleme
{
"mappings": {
"dynamic": "strict",
"properties": {
"@timestamp": {
"type": "date"
},
"testing_ignored": {
"properties": {
"constant": {
"type": "constant_keyword",
"value": "constant_value"
}
}
},
"testing_regex": {
"type": "constant_keyword",
"value": "constant_value"
},
"normal_constant": {
"type": "constant_keyword",
"value": "constant_value"
},
"small_field": {
"type": "keyword",
"ignore_above": 10
}
}
}
}
PUT constant-keywords-deleme/_doc/1
{
"@timestamp": "2021-09-02T04:20:01.760Z"
}
```
Set in your `kibana.yml` the key/value of:
```yml
xpack.securitySolution.alertIgnoreFields: ['testing_ignored.constant', '/.*_regex/']
```
Setup a rule to run:
<img width="1083" alt="Screen Shot 2021-09-01 at 10 23 23 PM" src="https://user-images.githubusercontent.com/1151048/131781696-fea0d421-836f-465c-9be6-5289fbb622a4.png">
Once it runs you should notice that the constant values for testing are not on the signals table since it only typically exists in the fields API:
<img width="1166" alt="Screen Shot 2021-09-01 at 10 26 16 PM" src="https://user-images.githubusercontent.com/1151048/131781782-1684fb1d-bed9-4cf0-be9a-0abe1f0f34d1.png">
But the normal one still exists:
<img width="1136" alt="Screen Shot 2021-09-01 at 10 26 31 PM" src="https://user-images.githubusercontent.com/1151048/131781827-5450c693-de9e-4285-b082-9f7a2cbd5d07.png">
If you change the `xpack.securitySolution.alertIgnoreFields` by removing it and re-generate the signals you will see these values added back.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
- [x] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/master/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
* Initial commit of serverType in email connector config
* Fleshing in route to get well known email service configs from nodemailer
* Adding elastic cloud to well known server type
* Cleaning up email constants and allowing for empty selection
* Showing error if user doesn't select server type
* Adding hook for setting email config based on server type
* Adding tests and making sure settings are not overwritten on edit
* Fixing functional test
* Adding migration
* Adding functional test for migration
* Repurposing service instead of adding serverType
* Cleanup
* Disabling host/port/secure form fields when settings retrieved from API
* Updating docs for service
* Filtering options based on whether cloud is enabled
* Initialize as disabled
* Fixing types
* Update docs/management/connectors/action-types/email.asciidoc
Co-authored-by: David Kilfoyle <41695641+kilfoyle@users.noreply.github.com>
Co-authored-by: David Kilfoyle <41695641+kilfoyle@users.noreply.github.com>
* Show setup mode button and setup bottom bar
* Adapt setup mode in react components to work without angular
* Add setup mode data update to react app
* Add missing functions from setup mode
* Revert setup mode changes from react components
* remove some empty lines
* Add setup button to monitoring toolbar
* Fix types
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Use super date picker instead of date range picker
fixes elastic/security-team/issues/1571
* fix test target
Super date picker's `data-test-subj` prop gets garbled and doesn't show up in rendered DOM. In other words, the component is entirely void of a data-test-subj attribute.
* make auto refresh work!!
fixes https://github.com/elastic/security-team/issues/1571
* set max width as per mock
fixes elastic/security-team/issues/1571
* show a callout to inform users to select different date ranges
fixes elastic/security-team/issues/1571
* persist recently used date ranges on the component only
fixes elastic/security-team/issues/1571
* use commonly used ranges from default common security solution ranges
fixes elastic/security-team/issues/1571
* Better align date picker
* full width panel for date picker so content flows below it
review comments
* mock time picker settings for tests
* use eui token for bg color
review comment
* persist recently used dates
fixes elastic/security-team/issues/1571
* persist date range selection over new endpoint selection
review comments
* remove obsolete local state since update button is not visible.
review comments
* fix bg color for dark mode and relative path
* update relative path
review comments
* cleanup - the action doesn't allow for undefined start and end dates anyway
refs 28a859ab3a
* fix types after sync
* update test title
* add a test for callout when empty data
* fix lint
* show update button when dates are changed
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* [Event log][7.x] Updated event log client to search across legacy IDs
* fixed tests
* extended kibana null version check
* added logic to alerting plugin
* fixed typechecks
* fixed typechecks
* Revert "fixed typechecks"
This reverts commit 6f6770fa4b.
* removed legacyId for routes
* fixed typechecks
* fixed position
* fixed query
* fixed query
* fixed tests
* fixed types place
* fixed due to comments
* fixed due to comments
* fixed eslint
* fixed due to comments
* splitted test data
* fixed test data
* increased the delay time to await the search
* removed version for 7.9 docs
* Update x-pack/plugins/event_log/server/es/cluster_client_adapter.ts
Co-authored-by: Mike Côté <mikecote@users.noreply.github.com>
* fixed unit test
* fixed test data
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Mike Côté <mikecote@users.noreply.github.com>
* refactor/reflatten server routes
* fix import
* fix any usage in server/lib
* clean up unused parameter
* remove any in server/browsers
* refactor handle request function into a class
* more cleanup
* Make analyzer work with EuiDataGrid full screen
* Don't ever restrict the width, remove console.log
* Remove isEventViewer prop no longer used
* Make global filters appear below data grid
* [Metrics UI] Add integration tests for Metric Threshold and refactor to fire correctly
* Removing unused variables
* Fixing tests for metric_threshold_executor
* Fixing test for metric_query
* fixing test
* Changing type guard
* [eslint] add rule to prevent export* in plugin index files
* deduplicate export names for types/instances with the same name
* attempt to auto-fix duplicate exports too
* capture exported enums too
* enforce no_export_all for core too
* disable rule by default, allow opting-in for help fixing
* update tests
* reduce yarn.lock duplication
* add rule but no fixes
* disable all existing violations
* update api docs with new line numbers
* revert unnecessary changes to yarn.lock which only had drawbacks
* remove unnecessary eslint-disable
* rework codegen to split type exports and use babel to generate valid code
* check for "export types" deeply
* improve test by using fixtures
* add comments to some helper functions
* disable fix for namespace exports including types
* label all eslint-disable comments with related team-specific issue
* ensure that child exports of `export type` are always tracked as types
Co-authored-by: spalger <spalger@users.noreply.github.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* [@pjhampton/@donaherc] Move sec telem tasks into own package.
* Split filter out into its own module, started abstracting ES interaction into a queries module
* Implemented querier and fixed some types
* Updated tests, moved receiver to plugin from sender to decouple them.
* fixed integration in detection engine, misc fixes
* [@pjhampton] Fix type ref problems. Update test defs.
* Make url transformer a member func of the sender class.
* [@pjhampton] clean up receiver commentary.
* [@pjhampton] add null check consistency.
* Fix bad formatting.
Co-authored-by: cdonaher <cdonaher@endgame.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
resolves#109095resolves#106854
Changes the way task manager and alerting perform their health / status
checks:
- no longer sets an `unavailable` status; now uses `degraded` instead
- change task manager "hot stats freshness" calculation to allow for
staler data before signalling a problem
- Changed the "Detected potential performance issue" message to sound
less scary, include a doc link to task manager health monitoring, and
log a debug instead of warning level
- add additional debug logging when task manager sets a status that's
not `available`, indicating why it's setting that status (in the code,
it's when task manager uses HealthStatus.Warning or Error)
* Upgrade EUI to v37.3.1
* Update i18n token mappings
* Skip i18n_eui_mapping defString checks for functions
* Update snapshots
* Update failing Security tests with extra nodes
* Remove hook cleanup now that elastic/eui#5068 is merged
* [i18n PR feedback] Prefer specific token skipping over all functions skipping
* Revert "Remove hook cleanup now that elastic/eui#5068 is merged"
This reverts commit e40ebfa929.
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* wip to remove rbac
* Revert "[Cases] Include rule registry client for updating alert statuses (#108588)"
This reverts commit 1fd7038b34.
This leaves the rule registry mock changes
* remove rbac on Trend/Count alert
* update detection api for status
* remove @kbn-alerts packages
* fix leftover
* Switching cases to leverage update by query for alert status
* Adding missed files
* fix bad logic
* updating tests for use_alerts_privileges
* remove index alias/fields
* fix types
* fix plugin to get the right index names
* left over of alis on template
* forget to use current user for create/read route index
* updated alerts page to not show table when no privileges and updates to tests
* fix bug when switching between o11y and security solution
* updates tests and move to use privileges page when user tries to access alerts without proper access
* updating jest tests
* pairing with yara
* bring back kbn-alerts after discussion with the team
* fix types
* fix index field for o11y
* fix bug with updating index priv state
* fix i18n issue and update api docs
* fix refresh on alerts
* fix render view on alerts
* updating tests and checking for null in alerts page to not show no privileges page before load
* fix details rules
Co-authored-by: Jonathan Buttner <jonathan.buttner@elastic.co>
Co-authored-by: Yara Tercero <yara.tercero@elastic.co>
* Fix bug when upgrading Windows package policies
Ensure package policy merge logics accounts for cases in which an
input/stream which previously had no variables declared but has
variables in a later package version.
Fixes#110202
* Refactor original var set into deepMergeVars
* [ML] Add index pattern info & select control for date time
* [ML] Update translations
* [ML] Gracefully handle when index pattern is not available
* [ML] Fix import
* [ML] Handle when unmounted
* [ML] Remove load index patterns because we don't really need it
* [ML] Add error obj to error toasts
* [ML] Update tests
* [ML] Update hook
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
When the observability:enableInspectEsQueries advanced setting is enabled, show an inspector that includes all queries through useFetcher.
Remove the callout.
## Summary
Ports over the existing Security Solution ML Rule to the RuleRegistry.
How to test this implementation
1. Enable the following in your `kibana.dev.yml`
```
xpack.ruleRegistry.enabled: true
xpack.ruleRegistry.write.enabled: true
xpack.securitySolution.enableExperimental: ['ruleRegistryEnabled']
```
2. Create a rule by running:
```
./x-pack/plugins/security_solution/server/lib/detection_engine/rule_types/scripts/create_rule_ml.sh
```
3. Push document to anomalies index (or trigger anomaly for job id from `create_rule_ml.sh` script)
### Checklist
Delete any items that are not applicable to this PR.
- [X] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
* Add header to page template
* add external config provider and overview content
* REmove unnecessary todos
* Remove non working section from header
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
**Ticket:** https://github.com/elastic/kibana/issues/110594
## Summary
This PR adds a feature flag around the logic that finds existing Alerts as Data indices and upgrades the mappings or rolls the index if the mappings can't be upgraded in place.
**IMPORTANT:**
- **The feature flag is switched off by default**. This is intentional, because we need to **disable the upgrade logic in 7.15.0**.
- **This is a temporary measure**. We're going to work on fixing the index upgrade logic asap and ship it before the next release that makes any mapping changes, possibly as soon as 7.15.1.
- Developers will need to enable it in their local kibana configs this way:
```yaml
xpack.ruleRegistry.unsafe.indexUpgrade.enabled: true
```
Please check the ticket for the background of this fix.
### Checklist
Delete any items that are not applicable to this PR.
- [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials
- [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
- [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/master/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
* [APM] Fix for no-data state for fallback from aggregated transactions (#109609)
* PR feedback and unit tests
* fixes lint error
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Fix policy upgrade from APM 0.3.0 to 0.4.0
Add debug log + logic to skip over any package variables that have been
removed from the base policy object. Issue was initially surfaced
testing upgrade from APM integration v0.3.0 to v0.4.0.
Ref #109907
* Fix type error in test
* Remove translation for validation debug log
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Adds distinct breadcrumbs for both "upgrade package policy" paths:
1. From the fleet agent policies list page
2. From the integrations package policie slist page
Closes#110434
* fix o11y privileges when rule created in stack
* fix merge
* fix dsl alerts
* fix privileges on o11y
* after discussion with o11y, we agree to simplify logic of the count of alerts
* remove unused variable
* fix one more type error
Co-authored-by: mgiota <panagiota.mitsopoulou@elastic.co>
* [Observability] Update AlertsSearchBar placeholder (#108179)
* [Observability] Remove default search query from Alerts page (#110242)
This is done to align with other placeholder texts found within the
Observability solution.
* first pass at renaming exports
* type fixes
* fix jest test
* look for correct error type
* remove transitional error
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* account for API errors and uninitialized state before fetching data
fixes elastic/kibana/issues/107129
* better name
refs elastic/kibana/pull/102261
* don't show date picker when loading data initially
fixes elastic/kibana/issues/107129
* use a readable selector instead
review changes
* remove redundant data fetch using paging action on tab switch.
refs elastic/kibana/pull/102261
* remove redundant validation
review comments
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* [Maps] replace IFieldType with IndexPatternField
* clean up imports
* import from public
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
- Correctly renders the empty chart state when no data is available.
- Hides the "Click drag to select" and trace samples message when the chart shows an empty state to avoid redundant info.
- Adds jest unit tests that would fail with the previously visible loading indicators.
- Fix a bug with cancelling search strategies.
