maxmustermann/kibana
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
kibana/x-pack/plugins/apm/dev_docs/telemetry.md

73 lines
3.9 KiB
Markdown
Raw Normal View History

[APM] Improvements to data telemetry (#70524) Make some changes to how we deal with data telemetry in APM and reduce the number of fields we're storing in Saved Objects in the .kibana index. Add a telemetry doc in dev_docs explaining how telemetry is collected and how to make updates. (In this PR the docs only cover data telemetry, but there's a space for the behavioral telemetry docs.) Stop storing the mapping for the data telemetry in the Saved Object but instead use `{ dynamic: false }`. This reduces the number of fields used by APM in the .kibana index (as requested in #43673.) Before: ```bash > curl -s -X GET "admin:changeme@localhost:9200/.kibana/_field_caps?fields=*&pretty=true" | jq '.fields|length' 653 ``` After: ```bash > curl -s -X GET "admin:changeme@localhost:9200/.kibana/_field_caps?fields=*&pretty=true" | jq '.fields|length' 415 ``` We don't need the mapping anymore for storing the saved object, but we still do need to update the telemetry repository when the mapping changes, and the `upload-telemetry-data` script uses that mapping when generating data. For these purposes the mapping in now defined in TypeScript in a function in common/apm_telemetry.ts. It's broken down into some variables that and put together as the same mapping object that was there before, but having it in this form should make it easier to update. A new script, `merge-telemetry-mapping`, takes the telemetry repository's xpack-phone-home.json mapping, merges in the result of our mapping and replaces the file. The result can be committed to the telemetry repo, making it easier to make changes to the mapping. References #61583 Fixes #67032
2020-07-07 18:19:59 +02:00
# APM Telemetry
In order to learn about our customers' usage and experience of APM, we collect
two types of telemetry, which we'll refer to here as "Data Telemetry" and
"Behavioral Telemetry."
This document will explain how they are collected and how to make changes to
them.
[The telemetry repository has information about accessing the clusters](https://github.com/elastic/telemetry#kibana-access).
Telemetry data is uploaded to the "xpack-phone-home" indices.
## Data Telemetry
Information that can be derived from a cluster's APM indices is queried and sent
to the telemetry cluster using the
Converts usage collection README to .mdx (#92982) Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-03-02 10:40:43 +01:00
[Usage Collection plugin](../../../../src/plugins/usage_collection/README.mdx).
[APM] Improvements to data telemetry (#70524) Make some changes to how we deal with data telemetry in APM and reduce the number of fields we're storing in Saved Objects in the .kibana index. Add a telemetry doc in dev_docs explaining how telemetry is collected and how to make updates. (In this PR the docs only cover data telemetry, but there's a space for the behavioral telemetry docs.) Stop storing the mapping for the data telemetry in the Saved Object but instead use `{ dynamic: false }`. This reduces the number of fields used by APM in the .kibana index (as requested in #43673.) Before: ```bash > curl -s -X GET "admin:changeme@localhost:9200/.kibana/_field_caps?fields=*&pretty=true" | jq '.fields|length' 653 ``` After: ```bash > curl -s -X GET "admin:changeme@localhost:9200/.kibana/_field_caps?fields=*&pretty=true" | jq '.fields|length' 415 ``` We don't need the mapping anymore for storing the saved object, but we still do need to update the telemetry repository when the mapping changes, and the `upload-telemetry-data` script uses that mapping when generating data. For these purposes the mapping in now defined in TypeScript in a function in common/apm_telemetry.ts. It's broken down into some variables that and put together as the same mapping object that was there before, but having it in this form should make it easier to update. A new script, `merge-telemetry-mapping`, takes the telemetry repository's xpack-phone-home.json mapping, merges in the result of our mapping and replaces the file. The result can be committed to the telemetry repo, making it easier to make changes to the mapping. References #61583 Fixes #67032
2020-07-07 18:19:59 +02:00
During the APM server-side plugin's setup phase a
[Saved Object](https://www.elastic.co/guide/en/kibana/master/managing-saved-objects.html)
for APM telemetry is registered and a
[task manager](../../task_manager/server/README.md) task is registered and started.
The task periodically queries the APM indices and saves the results in the Saved
Object, and the usage collector periodically gets the data from the saved object
and uploads it to the telemetry cluster.
Once uploaded to the telemetry cluster, the data telemetry is stored in
`stack_stats.kibana.plugins.apm` in the xpack-phone-home index.
### Generating sample data
The script in `scripts/upload-telemetry-data` can generate sample telemetry data and upload it to a cluster of your choosing.
You'll need to set the `GITHUB_TOKEN` environment variable to a token that has `repo` scope so it can read from the
[elastic/telemetry](https://github.com/elastic/telemetry) repository. (You probably have a token that works for this in
~/.backport/config.json.)
The script will run as the `elastic` user using the elasticsearch hosts and password settings from the config/kibana.yml
and/or config/kibana.dev.yml files.
Running the script with `--clear` will delete the index first.
[APM] Add cloud attributes to data telemetry (#71008) Add cloud provider, region, and availability zone to the data telemetry mappings and queries. Fixes #70465
2020-07-08 01:48:51 +02:00
If you're using an Elasticsearch instance without TLS verification (if you have `elasticsearch.ssl.verificationMode: none` set in your kibana.yml)
you can run the script with `env NODE_TLS_REJECT_UNAUTHORIZED=0` to avoid TLS connection errors.
[APM] Improvements to data telemetry (#70524) Make some changes to how we deal with data telemetry in APM and reduce the number of fields we're storing in Saved Objects in the .kibana index. Add a telemetry doc in dev_docs explaining how telemetry is collected and how to make updates. (In this PR the docs only cover data telemetry, but there's a space for the behavioral telemetry docs.) Stop storing the mapping for the data telemetry in the Saved Object but instead use `{ dynamic: false }`. This reduces the number of fields used by APM in the .kibana index (as requested in #43673.) Before: ```bash > curl -s -X GET "admin:changeme@localhost:9200/.kibana/_field_caps?fields=*&pretty=true" | jq '.fields|length' 653 ``` After: ```bash > curl -s -X GET "admin:changeme@localhost:9200/.kibana/_field_caps?fields=*&pretty=true" | jq '.fields|length' 415 ``` We don't need the mapping anymore for storing the saved object, but we still do need to update the telemetry repository when the mapping changes, and the `upload-telemetry-data` script uses that mapping when generating data. For these purposes the mapping in now defined in TypeScript in a function in common/apm_telemetry.ts. It's broken down into some variables that and put together as the same mapping object that was there before, but having it in this form should make it easier to update. A new script, `merge-telemetry-mapping`, takes the telemetry repository's xpack-phone-home.json mapping, merges in the result of our mapping and replaces the file. The result can be committed to the telemetry repo, making it easier to make changes to the mapping. References #61583 Fixes #67032
2020-07-07 18:19:59 +02:00
After running the script you should see sample telemetry data in the "xpack-phone-home" index.
### Updating Data Telemetry Mappings
In order for fields to be searchable on the telemetry cluster, they need to be
added to the cluster's mapping. The mapping is defined in
[the telemetry repository's xpack-phone-home template](https://github.com/elastic/telemetry/blob/master/config/templates/xpack-phone-home.json).
The mapping for the telemetry data is here under `stack_stats.kibana.plugins.apm`.
Update APM telemetry docs (#79583) In #79000 the way we create the telemetry schema was updated and APM is no longer excluded from the telemetry scripts. Update the docs to be more correct about this.
2020-10-06 15:49:35 +02:00
The mapping used there corresponds with the the [`apmSchema`](../server/lib/apm_telemetry/schema.ts) object. The telemetry tooling parses this file to generate its schemas, so some operations in this file (like doing a `reduce` or `map` over an array of properties) will not work.
[APM] Improvements to data telemetry (#70524) Make some changes to how we deal with data telemetry in APM and reduce the number of fields we're storing in Saved Objects in the .kibana index. Add a telemetry doc in dev_docs explaining how telemetry is collected and how to make updates. (In this PR the docs only cover data telemetry, but there's a space for the behavioral telemetry docs.) Stop storing the mapping for the data telemetry in the Saved Object but instead use `{ dynamic: false }`. This reduces the number of fields used by APM in the .kibana index (as requested in #43673.) Before: ```bash > curl -s -X GET "admin:changeme@localhost:9200/.kibana/_field_caps?fields=*&pretty=true" | jq '.fields|length' 653 ``` After: ```bash > curl -s -X GET "admin:changeme@localhost:9200/.kibana/_field_caps?fields=*&pretty=true" | jq '.fields|length' 415 ``` We don't need the mapping anymore for storing the saved object, but we still do need to update the telemetry repository when the mapping changes, and the `upload-telemetry-data` script uses that mapping when generating data. For these purposes the mapping in now defined in TypeScript in a function in common/apm_telemetry.ts. It's broken down into some variables that and put together as the same mapping object that was there before, but having it in this form should make it easier to update. A new script, `merge-telemetry-mapping`, takes the telemetry repository's xpack-phone-home.json mapping, merges in the result of our mapping and replaces the file. The result can be committed to the telemetry repo, making it easier to make changes to the mapping. References #61583 Fixes #67032
2020-07-07 18:19:59 +02:00
Update APM telemetry docs (#79583) In #79000 the way we create the telemetry schema was updated and APM is no longer excluded from the telemetry scripts. Update the docs to be more correct about this.
2020-10-06 15:49:35 +02:00
The `schema` property of the `makeUsageCollector` call in the [`createApmTelemetry` function](../server/lib/apm_telemetry/index.ts) contains the `apmSchema`.
[APM] Improvements to data telemetry (#70524) Make some changes to how we deal with data telemetry in APM and reduce the number of fields we're storing in Saved Objects in the .kibana index. Add a telemetry doc in dev_docs explaining how telemetry is collected and how to make updates. (In this PR the docs only cover data telemetry, but there's a space for the behavioral telemetry docs.) Stop storing the mapping for the data telemetry in the Saved Object but instead use `{ dynamic: false }`. This reduces the number of fields used by APM in the .kibana index (as requested in #43673.) Before: ```bash > curl -s -X GET "admin:changeme@localhost:9200/.kibana/_field_caps?fields=*&pretty=true" | jq '.fields|length' 653 ``` After: ```bash > curl -s -X GET "admin:changeme@localhost:9200/.kibana/_field_caps?fields=*&pretty=true" | jq '.fields|length' 415 ``` We don't need the mapping anymore for storing the saved object, but we still do need to update the telemetry repository when the mapping changes, and the `upload-telemetry-data` script uses that mapping when generating data. For these purposes the mapping in now defined in TypeScript in a function in common/apm_telemetry.ts. It's broken down into some variables that and put together as the same mapping object that was there before, but having it in this form should make it easier to update. A new script, `merge-telemetry-mapping`, takes the telemetry repository's xpack-phone-home.json mapping, merges in the result of our mapping and replaces the file. The result can be committed to the telemetry repo, making it easier to make changes to the mapping. References #61583 Fixes #67032
2020-07-07 18:19:59 +02:00
[APM] Additional data telemetry changes (#71112) * Add a date range of `now-1d` to the cloud query * Add a timeout of 5m to all queries (we'll investigate using async queries to improve this in the future.) * Factor out the date range filter into a variable * Fix a bug with the `indices_stats` tasks when it doesn't return data * Update the merge mapping script to create a migration file
2020-07-14 15:16:20 +02:00
When adding a task, the key of the task and the `took` properties need to be added under the `tasks` properties in the mapping, as when tasks run they report the time they took.
[APM] Improvements to data telemetry (#70524) Make some changes to how we deal with data telemetry in APM and reduce the number of fields we're storing in Saved Objects in the .kibana index. Add a telemetry doc in dev_docs explaining how telemetry is collected and how to make updates. (In this PR the docs only cover data telemetry, but there's a space for the behavioral telemetry docs.) Stop storing the mapping for the data telemetry in the Saved Object but instead use `{ dynamic: false }`. This reduces the number of fields used by APM in the .kibana index (as requested in #43673.) Before: ```bash > curl -s -X GET "admin:changeme@localhost:9200/.kibana/_field_caps?fields=*&pretty=true" | jq '.fields|length' 653 ``` After: ```bash > curl -s -X GET "admin:changeme@localhost:9200/.kibana/_field_caps?fields=*&pretty=true" | jq '.fields|length' 415 ``` We don't need the mapping anymore for storing the saved object, but we still do need to update the telemetry repository when the mapping changes, and the `upload-telemetry-data` script uses that mapping when generating data. For these purposes the mapping in now defined in TypeScript in a function in common/apm_telemetry.ts. It's broken down into some variables that and put together as the same mapping object that was there before, but having it in this form should make it easier to update. A new script, `merge-telemetry-mapping`, takes the telemetry repository's xpack-phone-home.json mapping, merges in the result of our mapping and replaces the file. The result can be committed to the telemetry repo, making it easier to make changes to the mapping. References #61583 Fixes #67032
2020-07-07 18:19:59 +02:00
[APM] Add cloud attributes to data telemetry (#71008) Add cloud provider, region, and availability zone to the data telemetry mappings and queries. Fixes #70465
2020-07-08 01:48:51 +02:00
The queries for the stats are in the [collect data telemetry tasks](../server/lib/apm_telemetry/collect_data_telemetry/tasks.ts).
The collection tasks also use the [`APMDataTelemetry` type](../server/lib/apm_telemetry/types.ts) which also needs to be updated with any changes to the fields.
Update APM telemetry docs (#79583) In #79000 the way we create the telemetry schema was updated and APM is no longer excluded from the telemetry scripts. Update the docs to be more correct about this.
2020-10-06 15:49:35 +02:00
Running `node scripts/telemetry_check --fix` from the root Kibana directory will update the schemas which should automatically notify the Infra team when a pull request is opened so they can update the mapping in the telemetry clusters.
[APM] Additional data telemetry changes (#71112) * Add a date range of `now-1d` to the cloud query * Add a timeout of 5m to all queries (we'll investigate using async queries to improve this in the future.) * Factor out the date range filter into a variable * Fix a bug with the `indices_stats` tasks when it doesn't return data * Update the merge mapping script to create a migration file
2020-07-14 15:16:20 +02:00
[APM] Improvements to data telemetry (#70524) Make some changes to how we deal with data telemetry in APM and reduce the number of fields we're storing in Saved Objects in the .kibana index. Add a telemetry doc in dev_docs explaining how telemetry is collected and how to make updates. (In this PR the docs only cover data telemetry, but there's a space for the behavioral telemetry docs.) Stop storing the mapping for the data telemetry in the Saved Object but instead use `{ dynamic: false }`. This reduces the number of fields used by APM in the .kibana index (as requested in #43673.) Before: ```bash > curl -s -X GET "admin:changeme@localhost:9200/.kibana/_field_caps?fields=*&pretty=true" | jq '.fields|length' 653 ``` After: ```bash > curl -s -X GET "admin:changeme@localhost:9200/.kibana/_field_caps?fields=*&pretty=true" | jq '.fields|length' 415 ``` We don't need the mapping anymore for storing the saved object, but we still do need to update the telemetry repository when the mapping changes, and the `upload-telemetry-data` script uses that mapping when generating data. For these purposes the mapping in now defined in TypeScript in a function in common/apm_telemetry.ts. It's broken down into some variables that and put together as the same mapping object that was there before, but having it in this form should make it easier to update. A new script, `merge-telemetry-mapping`, takes the telemetry repository's xpack-phone-home.json mapping, merges in the result of our mapping and replaces the file. The result can be committed to the telemetry repo, making it easier to make changes to the mapping. References #61583 Fixes #67032
2020-07-07 18:19:59 +02:00
## Behavioral Telemetry
Behavioral telemetry is recorded with the ui_metrics and application_usage methods from the Usage Collection plugin.
Please fill this in with more details.
Reference in a new issue Copy permalink