Make some changes to how we deal with data telemetry in APM and reduce the number of fields we're storing in Saved Objects in the .kibana index.
Add a telemetry doc in dev_docs explaining how telemetry is collected and how to make updates. (In this PR the docs only cover data telemetry, but there's a space for the behavioral telemetry docs.)
Stop storing the mapping for the data telemetry in the Saved Object but instead use `{ dynamic: false }`.
This reduces the number of fields used by APM in the .kibana index (as requested in #43673.)
Before:
```bash
> curl -s -X GET "admin:changeme@localhost:9200/.kibana/_field_caps?fields=*&pretty=true" | jq '.fields|length'
653
```
After:
```bash
> curl -s -X GET "admin:changeme@localhost:9200/.kibana/_field_caps?fields=*&pretty=true" | jq '.fields|length'
415
```
We don't need the mapping anymore for storing the saved object, but we still do need to update the telemetry repository when the mapping changes, and the `upload-telemetry-data` script uses that mapping when generating data.
For these purposes the mapping in now defined in TypeScript in a function in common/apm_telemetry.ts.
It's broken down into some variables that and put together as the same mapping object that was there before, but having it in this form should make it easier to update.
A new script, `merge-telemetry-mapping`, takes the telemetry repository's xpack-phone-home.json mapping, merges in the result of our mapping and replaces the file. The result can be committed to the telemetry repo, making it easier to make changes to the mapping.
References #61583Fixes#67032
* [APM] Quote trace id to ensure a word is searched (#69500)
Signed-off-by: Mathis Raguin <mathis@cri.epita.fr>
* [APM] Fix TransactionActionMenu tests (one test was not updated)
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* CI Reporter for saved objects field count
* Metrics needs to be an array
* Fix type failures
* Link to field count issue
* Revert "Link to field count issue"
This reverts commit 8c0126b838.
* Break down field count per type
* Don't log total metric as metrics report already calculates this
* Add saved objects field count ci metrics test to codeowners
* Address review comments
* Add field count CI metrics for disabled plugins
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Add extractQueryParams to es_ui_shared/public/url. Update CCR, Remote Clusters, and Rollup to consume this service via shared_imports.
* Fix Data Streams bug in which clicking a data stream would apply a deep-link filter to the table.
* Fix Rollup Job deep-link bug.
* Adds 'Anomaly detection' settings page along with require API endpoints
to list and create the apm anomaly detection jobs per environment.
Some test data is hardcoded while the the required changes in the ML
plugin are in flight.
* Converts the environment name to a compatible ML id string and persist
in groups array. Also adds random token to the job ID to prevent
collisions for job ids where diffferent environment names convert to the
same string
* - Improve job creation with latest updates for the `apm_transaction` ML module
- Implements job list in settings by reading from `custom_settings.job_tags['service.environment']`
- Add ML module method `createModuleItem` for job configuration
- Don't allow user to type in duplicate environments
* Update x-pack/plugins/apm/public/components/app/Settings/anomaly_detection/add_environments.tsx
Co-authored-by: Casper Hübertz <casper@formgeist.com>
* Update x-pack/plugins/apm/public/components/app/Settings/anomaly_detection/index.tsx
Co-authored-by: Casper Hübertz <casper@formgeist.com>
* UX feedback, adds i18n, and handles failed state for ML jobs fetch.
* - Moves get_all_environments from agent_configuration dir to common dir
- makes the 'all' environment name ALL_OPTION_VALUE agent configuration-specific
- replace field literals with constants
* PR feedback
* Adds support to create jobs for environment which are not defined.
* Fixes description copy, rearranges settings links, and makes sure the 'Not defined' option is disabled if it already exists.
* Only show "Not defined" in environment selector if there are actually
documents without service.environment set
* get the indexPatternName for the ML job from the set of user-definned indices
* updated job_tags type definition
Co-authored-by: Casper Hübertz <casper@formgeist.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Permit timezones not working in Elasticsearch
* Fix functional tests
* Use timezone without summer time for test
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Added UI validation when creating a Webhook connector with invalid URL
* fixed tests
* Fixed due to comments
* fixed type check and extended error message for invalid URL
* Fixed whitelisting of URL
* fixed failing tests
* fixed str
* [Security Solution] Change default index pattern
Add `logs-*` to the Security Solution default index pattern. This should
allow the app to recognize events from the Elastic Endpoint.
* Draft ServiceNow generic implementation
* simple working servicenow incident per alert
* fixed running times
* rely on externalId for update incident on the next execution
* Added consumer to the action type to be able to split ServiceNow for Cases and Alerts
* Added subActions support for ServiceNow action form
* Basic version for Alerts part for ServiceNow
* Keep Case ServiceNow functionality working
* Revert changes on app_router
* Fixed type checks
* Fixed language check issues
* Fixed actions unit tests
* Fixed functional tests
* Fixed jest tests
* fixed tests
* Copied case mappings to alerting plugin
* made consumer optional
* Cleanup tests
* more cleanup
* Fixed jest tests and type checks
* fixed tests
* fixed servicenow validation tests
* Added ServiceNow unit tests
* Removed consumer for actions
* fixed client side isCaseOwned support
* fixed failing tests
* fixed jest tests
* Fixed URL validation
* fixed due to comments
* fixed tests
* fixed jest tests
* Fixed due to comments. Moved ServiceNow filtering in case plugin to server side
* fixed mock for ServiceNow
* fixed consumer config
* fixed test
* fixed type check
* Fixed jest test
* fixed type check
* Add API endpoint and hook for retrieving restricted packages
* Filter out restricted packages already in use from list of integrations available for an agent config
* Allow list agent configs to optionally return expanded package configs, re
* Filter out agent configs which already use the restricted package already from list of agent configs available for an integration
* Allow more than 20 agent configs to be shown
* Rename restricted to limited; add some common methods to DRY
* Add limited package check on server side
* Adjust copy wording
* Fix typings
* Add some package config api integration tests, update es archive mappings
* Move test to dockerized integation tests directory; move existing epm tests to their own directory
* Remove extra assignPackageConfigs() - already handled in packageConfigService.create()
* Review fixes
* Fix type, reenabled skipped test
* Move new EPM integration test file
The last thing we were using from it was configuring a static assets directory (which is only use for the EPM Integrations header graphic).
This is now provided by platform and is not configurable da28df5b15/src/core/MIGRATION.md (L1344)
Moved the header assets to the new directory & updated the `toAssets` helper
* wip: add edit action to dfanalytics table
* add update endpoint and edit flyout
* show success and error toasts. close flyout and refresh on success
* show permission message in edit action
* update types
* disable update button if mml not valid
* show error in toast, init values are config values
* fix undefined check for allow lazy start
* prevent update if mml is empty
* Implementing dataset component templates
* Fixing test
* Temporary fix to include timestamp with any component template created
* Update package registry docker image for CI.
* Adapt to new registry filesystem layout.
* Adjust tests to changed registry behavior.
* Adding a test for mappings and settings overrides
* Wrap all the tests in the docker check
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Sonja Krause-Harder <sonja.krause-harder@elastic.co>
