* Add learn more links to data streams, indices, and index templates tabs.
* Add tooltips to detail panel.
* Unify data streams description text.
* Fix bug in which index tab showed an empty list, by clearing the filter state on unmount.
* Add indices count to data stream detail panel.
* Add usage collector for telemetry.
* Make minimal usage collector work.
* Add all fields to Usage and schema
* Type packages as array.
* Temporarily remove schema.
* Temporarily exclude our collector from schema checks.
* Add fleet telemetry.
* Remove events from agent stats.
* Add package telemetry.
* Use correct import.
* Add telemetry about enabled packages.
* Clean up comments.
* Update x-pack/plugins/ingest_manager/server/collectors/package_collectors.ts
Co-authored-by: Alejandro Fernández Haro <afharo@gmail.com>
* Update x-pack/plugins/ingest_manager/server/collectors/package_collectors.ts
Co-authored-by: Nicolas Chaulet <n.chaulet@gmail.com>
* Correctly check for element in array.
* Use a real SavedObjectsClient.
* Remove useless use of undefined.
* Use less deep path to import SavedObjectsClient.
Co-authored-by: Alejandro Fernández Haro <afharo@gmail.com>
Co-authored-by: Nicolas Chaulet <n.chaulet@gmail.com>
Related to #70383 and #63455.
Refactors the action buttons of the transform and data frame analytics jobs list:
Previously custom actions included state and JSX for e.g. confirmation modals. Problem with that: If the actions list popover hides, the modal would unmount too. Since EUI's behaviour will change with the release/merge of #70383, we needed a refactor that solves that issue right now.
With this PR, state management for UI behaviour that follows after a button click like the confirmation modals was moved to a custom hook which is part of the outer level of the buttons itself. The modal now also gets mounted on the outer level. This way we won't loose the modals state and DOM rendering when the action button hides.
Note that this PR doesn't fix the nested buttons issue (#63455) yet. For that we need EUI issue #70383 to be in Kibana which will arrive with EUI v26.3.0 via #70243. So there will be one follow up to that which will focus on getting rid of the nested button structure.
* Fix base64 download bug
* Add test for artifact download
* Add more tests to ensure cached versions of artifacts are correct
* Convert to new format
* missed some refs
* partial fix to wrapper format
* update fixtures and integration test
* Fixing unit tests
Co-authored-by: Alex Kahan <alexander.kahan@elastic.co>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* adds 2 menu items to alert page, progress on exception modal
* adds enriching
* remove unused useExceptionList()
* implements some types
* move add exception modal files
* Exception builder changes to support latest schema
* Changes to lists plugin schemas and fix api bug
Needed to make the schemas more forgiving. Before this change they required name,
description, etc for creation and update.
The update item API was using the wrong url.
* Adding and editing exceptions working
- Modifies add_exception_modal component
- Creates edit_exception_modal component
- Creates shared comments component
- Creates use_add_exception api hook for adding or editing exceptions
- Updates viewer code to support adding and editing exceptions
- Updates alerts table code to use updated version of add_exception_modal
* fixes duplicate types
* updates os tag input
* fixes comment style
* removes checkbox programatically
* grahpql updates to expose exceptions_list
* Add fetch_or_create_exception_list hook
* fixes data population
* refactor use_add_exception hook, add tests
* fix rebase issues, pending updates to edit modal
* fix edit modal and default endpoint exceptions
* adds second checkbox
* adds signal index stuff
* switches boolean logic
* fix some type errors
* remove unnecesary code
* fixes checkbox logic in edit modal
* fixes recursive prop passing
* addresses comments/fixes types
* Revert schema type changes
* type fixes
* fixes regular exception modal
* fix more type errors, remove console log
* fix tests
* move add exception hook, lint
* close alert checkbox closes alert
* address PR comments
* add type to patch rule call, fix ts errors
* fix lint
* fix merge problems after conflict
* Address PR comments
* undo graphql type change
Co-authored-by: Davis Plumlee <davis.plumlee@elastic.co>
## Summary
Adds these data types to the value based lists end points from [Elasticsearch field data types](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html):
Single value based list types:
* binary
* boolean
* byte
* date
* date_nanos
* date_range
* double
* float
* integer
* ip
* half_float
* keyword
* text
* long
* short
Range value based list types:
* double_range
* float_range
* integer_range
* ip_range
* long_range
Geo value based list types: (caveat is that you cannot query them using other geometry just yet ... you can only these and export them)
* geo_point
* geo_shape
* shape
For importing and exporting different values such as ranges, geo, or single values, this introduces a serialize and deserialize option for the endpoints.
For example if you want to serialize in an ip_range such as 192.168.0.1,192.168.0.3 which has a comma between the two would use the following:
```ts
POST /api/lists
{
"name": "List with an ip range",
"serializer": "(?<gte>.+),(?<lte>.+)",
"deserializer": "{{gte}},{{lte}}",
"description": "This list has ip ranges",
"type": "date_range"
}
```
If you want to serialize in keywords from a list that _only_ match a particular value you would use the following:
```ts
POST /api/lists
{
"id": "keyword_custom_format_list",
"name": "Simple list with a keyword using a custom format",
"description": "This parses the first found ipv4 only",
"serializer": "(?<value>((25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?).){3}(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))",
"deserializer": "{{value}}",
"type": "keyword"
}
```
The serializer is a [named capturing group](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/match) while the deserializer is using [MustacheJS](https://github.com/janl/mustache.js/). The range type, single value types, and geo types all have default captures for their serialize and default mustache templates if none are configured with an endpoint.
The default capture groups and mustache handles for each are:
* shape, geo_point, geo_shape: `(?<lat>.+),(?<lon>.+)`
* date_range: `(?<gte>.+),(?<lte>.+)|(?<value>.+)`
* other ranges are: `(?<gte>.+)-(?<lte>.+)|(?<value>.+)`
* all single data types: `(?<value>.+)`
For ranges you can use both `gte, lte`, and `value` together. If `gte` _and_ `lte` matches it will use that for the greater than, less than elastic range and ignore `value` even if `value` also matched. If _only_ `value` matches and `gte`, `lte` does not match then it will use `value` and put `value` as _both_ the `gte`, and `lte`.
For example, if you are serializing in a list of ip ranges as the list data type, `ip_range` and you have these 3 entries in the file:
```ts
127.0.0.1
127.0.0.2-5
```
The default `serializer` will use `(?<gte>.+)-(?<lte>.+)|(?<value>.+)` and you will get two elastic documents like so:
```ts
{
"_source" : {
"ip_range" : {
"gte" : "127.0.0.1",
"lte" : "127.0.0.1"
}
}
{
"_source" : {
"ip_range" : {
"gte" : "127.0.0.2",
"lte" : "127.0.0.5"
}
}
```
The default mustache handles for each are:
* shape, geo_point, geo_shape: `{{{lat}}},{{{lon}}}`
* date_range: `{{{gte}}},{{{lte}}}`
* other ranges are: `{{{gte}}}-{{{lte}}}`
* all values are: `{{{value}}}`
I use three instead of two handle bars (`{{{` vs.` {{`) so that HTML is not escaped for the lists. You can override and change it if you need or want the escaping.
If during the deserializer phase it detects that a `gte` and `lte` are exactly the same it will still output them as a two items and use the mustache deserialize value. Using the ip-range example above that will be outputted like so since it detects that the lte-gte are exactly the same value:
```ts
127.0.0.1-127.0.0.1
127.0.0.2-127.0.0.5
```
---
Interesting queries to run from the lists scripts folder for testing:
Load some small test files from `./lists/files` for example:
```ts
./import_list_items_by_filename.sh ip_range ./lists/files/ip_range_cidr.txt
./import_list_items_by_filename.sh ip_range ./lists/files/ip_range.txt
./import_list_items_by_filename.sh date ./lists/files/date.txt
./import_list_items_by_filename.sh ip_range ./lists/files/ip_range_mixed.txt
...
```
Export them
```ts
./export_list_items.sh ip_range_cidr.txt
./export_list_items.sh ip_range.txt
./export_list_items.sh date.txt
./export_list_items.sh ip_range_mixed.txt
...
```
Find on them
```ts
./find_list_items.sh ip_range_cidr.txt
./find_list_items.sh ip_range.txt
./find_list_items.sh date.txt
./find_list_items.sh ip_range_mixed.txt
...
```
Find specific values such as:
```ts
./get_list_item_by_value.sh ip_range_mixed.txt 192.168.0.1
./get_list_item_by_value.sh date.txt 2020-08-25T17:57:01.978Z
...
```
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
## Summary
* Removes the feature flag and turns on lists by default
* Applies to both exception lists and value lists
* Removes all scary messages about having it enabled
* Updates the unit tests to work with it on
- [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
* [search] Refactor the way search strategies are registered/retrieved on the server
* Fix types and tests and update docs
* Fix failing test
* Fix build of example plugin
* Fix functional test
* Make server strategies sync
* Move strategy name into options
* docs
* Remove FE strategies
* TypeScript of hell
delete search explorer
* Fix search interceptor OSS tests
* typos
* test cleanup
* Delete search example
fix interceptor async tests to use fake timers
* docs
* fix
* return search wrapper
* Update search interceptor tests and abort utils
* ts
* jest test fix
* code review
* change how logs consume search API
Co-authored-by: Lukas Olson <olson.lukas@gmail.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
## Summary
This PR tries to start to tie together the server and client changes for exceptions lists.
- Updates graphql types to allow UI access to a rule's `exceptions_list` property
- Updates the exception viewer component to now dynamically take the rule `exceptions_list`, up until now we just had an empty array in it's place
- Updates the viewer logic to check if a rule has an endpoint list associated with it. If it does, then it displays both detections and endpoint UIs (in the viewer), if it does not, then it only displays the detections UI
- Updates the viewer UI to better deal with spacing when an exception list item only has one or two entries (before the and badge with the antennas was stretching passed the exception items to fill the space)
- Updates the detections engine exceptions logic to fetch list items using an exception list's `id` as opposed to it's `list_id`, this now aligns with the UI using the same params on its end
- Adds exception list `type` to information kept by the rule for exception lists
- Updates the exception list type from `string` to `endpoint | detection`
- Updates the exception list _item_ type from `string` to `simple`
- Adds unit tests for the detection engine server side util that fetches the exception list items
* add generic audit_trail service in core
* expose auditTraik service to plugins
* add auditTrail x-pack plugin
* fix type errors
* update mocks
* expose asScoped interface via start. auditor via request context
* use type from audit trail service
* wrap getActiveSpace in safeCall only. it throws exception for non-authz
* pass message to log explicitly
* update docs
* create one auditor per request
* wire es client up to auditor
* update docs
* withScope accepts only one scope
* use scoped client in context for callAsInternalUser
* use auditor in scoped cluster client
* adopt auditTrail plugin to new interface. configure log from config
* do not log audit events in console by default
* add audit trail functional tests
* cleanup
* add example
* add mocks for spaces plugin
* add unit tests
* update docs
* test description
* Apply suggestions from code review
apply @jportner suggestions
Co-authored-by: Joe Portner <5295965+jportner@users.noreply.github.com>
* add unit tests
* more robust tests
* make spaces optional
* address comments
* update docs
* fix WebStorm refactoring
Co-authored-by: Joe Portner <5295965+jportner@users.noreply.github.com>
* Small styling tweaks to processor items
- Moved the move button to the before the processor name
- Cancel button is still after description if there is one
- Made inline text description a bit taller and changed border
style
* Commit code that moves the cancel move button 🤦🏼♂️
* Do not completely hide the move button, prevent ui from jumping
* Update styling and UX of move button; EuiToggleButton
- Bring the styling of the button more in line with this comment
https://github.com/elastic/kibana/pull/70786#issuecomment-654222298
* use cross icon for cancelling move
* replace hard values with EUI values in SCSS
* Address rerendering triggered by context
- also prevent re-renders basded on contstructing objects on
each render
* Similarly move use of context to settings form container
We are only interested in the es docs path string in the settings
form component so no need to render for other updates.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Make some changes to how we deal with data telemetry in APM and reduce the number of fields we're storing in Saved Objects in the .kibana index.
Add a telemetry doc in dev_docs explaining how telemetry is collected and how to make updates. (In this PR the docs only cover data telemetry, but there's a space for the behavioral telemetry docs.)
Stop storing the mapping for the data telemetry in the Saved Object but instead use `{ dynamic: false }`.
This reduces the number of fields used by APM in the .kibana index (as requested in #43673.)
Before:
```bash
> curl -s -X GET "admin:changeme@localhost:9200/.kibana/_field_caps?fields=*&pretty=true" | jq '.fields|length'
653
```
After:
```bash
> curl -s -X GET "admin:changeme@localhost:9200/.kibana/_field_caps?fields=*&pretty=true" | jq '.fields|length'
415
```
We don't need the mapping anymore for storing the saved object, but we still do need to update the telemetry repository when the mapping changes, and the `upload-telemetry-data` script uses that mapping when generating data.
For these purposes the mapping in now defined in TypeScript in a function in common/apm_telemetry.ts.
It's broken down into some variables that and put together as the same mapping object that was there before, but having it in this form should make it easier to update.
A new script, `merge-telemetry-mapping`, takes the telemetry repository's xpack-phone-home.json mapping, merges in the result of our mapping and replaces the file. The result can be committed to the telemetry repo, making it easier to make changes to the mapping.
References #61583Fixes#67032
* [APM] Quote trace id to ensure a word is searched (#69500)
Signed-off-by: Mathis Raguin <mathis@cri.epita.fr>
* [APM] Fix TransactionActionMenu tests (one test was not updated)
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Add extractQueryParams to es_ui_shared/public/url. Update CCR, Remote Clusters, and Rollup to consume this service via shared_imports.
* Fix Data Streams bug in which clicking a data stream would apply a deep-link filter to the table.
* Fix Rollup Job deep-link bug.
* Adds 'Anomaly detection' settings page along with require API endpoints
to list and create the apm anomaly detection jobs per environment.
Some test data is hardcoded while the the required changes in the ML
plugin are in flight.
* Converts the environment name to a compatible ML id string and persist
in groups array. Also adds random token to the job ID to prevent
collisions for job ids where diffferent environment names convert to the
same string
* - Improve job creation with latest updates for the `apm_transaction` ML module
- Implements job list in settings by reading from `custom_settings.job_tags['service.environment']`
- Add ML module method `createModuleItem` for job configuration
- Don't allow user to type in duplicate environments
* Update x-pack/plugins/apm/public/components/app/Settings/anomaly_detection/add_environments.tsx
Co-authored-by: Casper Hübertz <casper@formgeist.com>
* Update x-pack/plugins/apm/public/components/app/Settings/anomaly_detection/index.tsx
Co-authored-by: Casper Hübertz <casper@formgeist.com>
* UX feedback, adds i18n, and handles failed state for ML jobs fetch.
* - Moves get_all_environments from agent_configuration dir to common dir
- makes the 'all' environment name ALL_OPTION_VALUE agent configuration-specific
- replace field literals with constants
* PR feedback
* Adds support to create jobs for environment which are not defined.
* Fixes description copy, rearranges settings links, and makes sure the 'Not defined' option is disabled if it already exists.
* Only show "Not defined" in environment selector if there are actually
documents without service.environment set
* get the indexPatternName for the ML job from the set of user-definned indices
* updated job_tags type definition
Co-authored-by: Casper Hübertz <casper@formgeist.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Added UI validation when creating a Webhook connector with invalid URL
* fixed tests
* Fixed due to comments
* fixed type check and extended error message for invalid URL
* Fixed whitelisting of URL
* fixed failing tests
* fixed str
* [Security Solution] Change default index pattern
Add `logs-*` to the Security Solution default index pattern. This should
allow the app to recognize events from the Elastic Endpoint.
* Draft ServiceNow generic implementation
* simple working servicenow incident per alert
* fixed running times
* rely on externalId for update incident on the next execution
* Added consumer to the action type to be able to split ServiceNow for Cases and Alerts
* Added subActions support for ServiceNow action form
* Basic version for Alerts part for ServiceNow
* Keep Case ServiceNow functionality working
* Revert changes on app_router
* Fixed type checks
* Fixed language check issues
* Fixed actions unit tests
* Fixed functional tests
* Fixed jest tests
* fixed tests
* Copied case mappings to alerting plugin
* made consumer optional
* Cleanup tests
* more cleanup
* Fixed jest tests and type checks
* fixed tests
* fixed servicenow validation tests
* Added ServiceNow unit tests
* Removed consumer for actions
* fixed client side isCaseOwned support
* fixed failing tests
* fixed jest tests
* Fixed URL validation
* fixed due to comments
* fixed tests
* fixed jest tests
* Fixed due to comments. Moved ServiceNow filtering in case plugin to server side
* fixed mock for ServiceNow
* fixed consumer config
* fixed test
* fixed type check
* Fixed jest test
* fixed type check
* Add API endpoint and hook for retrieving restricted packages
* Filter out restricted packages already in use from list of integrations available for an agent config
* Allow list agent configs to optionally return expanded package configs, re
* Filter out agent configs which already use the restricted package already from list of agent configs available for an integration
* Allow more than 20 agent configs to be shown
* Rename restricted to limited; add some common methods to DRY
* Add limited package check on server side
* Adjust copy wording
* Fix typings
* Add some package config api integration tests, update es archive mappings
* Move test to dockerized integation tests directory; move existing epm tests to their own directory
* Remove extra assignPackageConfigs() - already handled in packageConfigService.create()
* Review fixes
* Fix type, reenabled skipped test
* Move new EPM integration test file
The last thing we were using from it was configuring a static assets directory (which is only use for the EPM Integrations header graphic).
This is now provided by platform and is not configurable da28df5b15/src/core/MIGRATION.md (L1344)
Moved the header assets to the new directory & updated the `toAssets` helper