[DOCS] Updates images and adds example to rollup jobs doc (#41839)
* [DOCS] Updates images and adds example to rollup jobs doc * [DOCS] Edits to rollup text
BIN
docs/images/management_create_rollup_job.png
Normal file → Executable file
Before Width: | Height: | Size: 147 KiB After Width: | Height: | Size: 182 KiB |
BIN
docs/images/management_create_rollup_menu.png
Normal file → Executable file
Before Width: | Height: | Size: 177 KiB After Width: | Height: | Size: 143 KiB |
BIN
docs/images/management_rolled_dashboard.png
Normal file → Executable file
Before Width: | Height: | Size: 251 KiB After Width: | Height: | Size: 86 KiB |
BIN
docs/images/management_rollup_job_dashboard.png
Executable file
After Width: | Height: | Size: 213 KiB |
BIN
docs/images/management_rollup_job_details.png
Normal file → Executable file
Before Width: | Height: | Size: 146 KiB After Width: | Height: | Size: 189 KiB |
BIN
docs/images/management_rollup_job_vis.png
Executable file
After Width: | Height: | Size: 217 KiB |
BIN
docs/images/management_rollup_list.png
Normal file → Executable file
Before Width: | Height: | Size: 96 KiB After Width: | Height: | Size: 129 KiB |
BIN
docs/images/management_rollups_visualization.png
Normal file → Executable file
Before Width: | Height: | Size: 301 KiB After Width: | Height: | Size: 137 KiB |
|
@ -17,8 +17,6 @@ include::management/index-patterns.asciidoc[]
|
||||||
|
|
||||||
include::management/rollups/create_and_manage_rollups.asciidoc[]
|
include::management/rollups/create_and_manage_rollups.asciidoc[]
|
||||||
|
|
||||||
include::management/rollups/visualize_rollup_data.asciidoc[]
|
|
||||||
|
|
||||||
include::management/index-lifecycle-policies/intro-to-lifecycle-policies.asciidoc[]
|
include::management/index-lifecycle-policies/intro-to-lifecycle-policies.asciidoc[]
|
||||||
|
|
||||||
include::management/index-lifecycle-policies/create-policy.asciidoc[]
|
include::management/index-lifecycle-policies/create-policy.asciidoc[]
|
||||||
|
|
|
@ -1,90 +1,148 @@
|
||||||
[role="xpack"]
|
[role="xpack"]
|
||||||
[[data-rollups]]
|
[[data-rollups]]
|
||||||
== Working with rollup indices
|
== Rollup jobs
|
||||||
|
|
||||||
The {ref}/xpack-rollup.html[rollup feature in {es}]
|
|
||||||
enables you to summarize historical data and store it compactly for future analysis,
|
|
||||||
so you can query, aggregate, and visualize the data using a fraction of the storage.
|
|
||||||
This is a good way to keep costs down when you need to store months or years of
|
|
||||||
historical data for use in visualizations and reports.
|
|
||||||
|
|
||||||
{kib} supports rolled up data in two ways:
|
|
||||||
|
|
||||||
* You can create and manage a rollup job in Management
|
|
||||||
* You can create a visualization using rolled up data in
|
|
||||||
Visualize and view it in a dashboard
|
|
||||||
|
|
||||||
|
|
||||||
[role="xpack"]
|
A rollup job is a periodic task that aggregates data from indices specified
|
||||||
[[create-and-manage-rollup-job]]
|
by an index pattern and rolls it into a new index. Rollup indices are a good way to
|
||||||
=== Create and manage rollup jobs
|
compactly store months or years of historical
|
||||||
|
data for use in visualizations and reports.
|
||||||
|
|
||||||
In Management, you'll find a UI for viewing, creating, starting, stopping, and
|
You’ll find *Rollup Jobs* under *Management > Elasticsearch*. With this UI,
|
||||||
deleting rollup jobs. A rollup job is a periodic task that summarizes data from
|
you can:
|
||||||
indices specified by an index pattern and rolls it into a new index. To navigate
|
|
||||||
to the UI, go to *Management*, and under *Elasticsearch*, click *Rollup Jobs*.
|
* <<create-and-manage-rollup-job, Create a rollup job>>
|
||||||
|
* <<manage-rollup-job, Start, stop, and delete rollup jobs>>
|
||||||
|
|
||||||
[role="screenshot"]
|
[role="screenshot"]
|
||||||
image::images/management_rollup_list.png[][List of currently active rollup jobs]
|
image::images/management_rollup_list.png[][List of currently active rollup jobs]
|
||||||
|
|
||||||
|
Before using this feature, you should be familiar with how rollups work.
|
||||||
|
{ref}/xpack-rollup.html[Rolling up historical data] is a good source for more detailed information.
|
||||||
|
|
||||||
[float]
|
[float]
|
||||||
[[create-rollup-job]]
|
[[create-and-manage-rollup-job]]
|
||||||
==== Creating a rollup job
|
=== Create a rollup job
|
||||||
|
|
||||||
{kib} makes it easy for you to create a rollup job by walking you through the
|
{kib} makes it easy for you to create a rollup job by walking you through
|
||||||
process step by step. The first step is to define the job logistics. These include
|
the process. You fill in the name, data flow, and how often you want to roll
|
||||||
the name of the rollup job, the index or indices to summarize, and the output rollup index.
|
up the data. Then you define a date histogram aggregation for the rollup job
|
||||||
|
and optionally terms, histogram, and metrics aggregations.
|
||||||
|
|
||||||
The index pattern cannot match the name of the output rollup index. For example,
|
When defining the index pattern, you must enter a name that is different than
|
||||||
if your index pattern is `metricbeat-*`, you cannot name your rollup index
|
the output rollup index. Otherwise, the job
|
||||||
`metricbeat-rollup`. Otherwise, the job will attempt to capture the data in the
|
will attempt to capture the data in the rollup index. For example, if your index pattern is `metricbeat-*`,
|
||||||
rollup index.
|
you can name your rollup index `rollup-metricbeat`, but not `metricbeat-rollup`.
|
||||||
|
|
||||||
[role="screenshot"]
|
[role="screenshot"]
|
||||||
image::images/management_create_rollup_job.png[][Wizard that walks you through creation of a rollup job]
|
image::images/management_create_rollup_job.png[][Wizard that walks you through creation of a rollup job]
|
||||||
|
|
||||||
You must set a schedule for the rollup job: how often to collect the data,
|
|
||||||
the number of documents to roll up at a time, and the duration of its latency.
|
|
||||||
The latency buffer field is provided to protect against the late arrival of data
|
|
||||||
from Beats or other sources. By delaying the rollup for the specified amount of
|
|
||||||
time from when the job starts, you allow for the inclusion of late-arriving data
|
|
||||||
in the rollup.
|
|
||||||
|
|
||||||
In the subsequent phases, you define the Date Histogram aggregation for the job
|
|
||||||
and optionally the Terms and Histogram aggregations.
|
|
||||||
|
|
||||||
* The Date Histogram aggregation defines the time intervals for summarizing the data.
|
|
||||||
This value is important because you cannot search the data with a smaller value
|
|
||||||
than this interval. However, you can aggregate buckets in a larger time interval.
|
|
||||||
|
|
||||||
* The Terms histogram enables you to split the time buckets into sub buckets for
|
|
||||||
term field values.
|
|
||||||
|
|
||||||
* The Histogram aggregation enables you to split the time buckets into sub buckets
|
|
||||||
for numeric field values.
|
|
||||||
|
|
||||||
The final step is to specify the fields for calculating metrics. For each selected
|
|
||||||
field, you can collect any or all of the following: value count, average, sum, min, and max.
|
|
||||||
|
|
||||||
Before you save the rollup job, {kib} displays a summary of the rollup job for
|
|
||||||
validation.
|
|
||||||
|
|
||||||
[float]
|
[float]
|
||||||
[[manage-rollup-job]]
|
[[manage-rollup-job]]
|
||||||
==== Managing rollup jobs
|
=== Start, stop, and delete rollup jobs
|
||||||
|
|
||||||
Selecting a job on the *Rollup jobs* page shows its details. The Manage menu in
|
Once you’ve saved a rollup job, you’ll see it the *Rollup Jobs* overview page,
|
||||||
|
where you can drill down for further investigation. The *Manage* menu in
|
||||||
the lower right enables you to start, stop, and delete the rollup job.
|
the lower right enables you to start, stop, and delete the rollup job.
|
||||||
You must first stop a rollup job before deleting it.
|
You must first stop a rollup job before deleting it.
|
||||||
|
|
||||||
[role="screenshot"]
|
[role="screenshot"]
|
||||||
image::images/management_rollup_job_details.png[][Rollup job details]
|
image::images/management_rollup_job_details.png[][Rollup job details]
|
||||||
|
|
||||||
You can start, stop, and delete an existing rollup job, but edits are not supported.
|
You can’t change a rollup job after you’ve created it. To select additional fields
|
||||||
If you want to make any changes, delete the existing job and create a new one with
|
or redefine terms, you must delete the existing job, and then create a new one
|
||||||
the updated specifications. Be sure to use a different name for the new rollup job;
|
with the updated specifications. Be sure to use a different name for the new rollup
|
||||||
reusing the same name could lead to problems with mismatched job configurations.
|
job—reusing the same name can lead to problems with mismatched job configurations.
|
||||||
More about logistical details for the {ref}/rollup-job-config.html[rollup job configuration]
|
You can read more at {ref}/rollup-job-config.html[rollup job configuration].
|
||||||
can be found in the {es} documentation.
|
|
||||||
|
[float]
|
||||||
|
=== Try it: Create and visualize rolled up data
|
||||||
|
|
||||||
|
This example creates a rollup job to capture log data from sample web logs.
|
||||||
|
To follow along, add the <<add-sample-data, sample web logs data set>>.
|
||||||
|
|
||||||
|
In this example, you want data that is older than 7 days in the target index pattern `kibana_sample_data_logs`
|
||||||
|
to roll up once a day into the index `rollup_logstash`. You’ll bucket the
|
||||||
|
rolled up data on an hourly basis, using 60m for the time bucket configuration.
|
||||||
|
This allows for more granular queries, such as 2h and 12h.
|
||||||
|
|
||||||
|
[float]
|
||||||
|
==== Create the rollup job
|
||||||
|
|
||||||
|
As you walk through the *Create rollup job* UI, enter the data shown in
|
||||||
|
the table below. The terms, histogram, and metrics fields reflect
|
||||||
|
the key information to retain in the rolled up data: where visitors are from (geo.src),
|
||||||
|
what operating system they are using (machine.os.keyword),
|
||||||
|
and how much data is being sent (bytes).
|
||||||
|
|
||||||
|
|===
|
||||||
|
|*Field* |*Value*
|
||||||
|
|
||||||
|
|Name
|
||||||
|
|logs_job
|
||||||
|
|
||||||
|
|Index pattern
|
||||||
|
|`kibana_sample_data_logs`
|
||||||
|
|
||||||
|
|Rollup index name
|
||||||
|
|`rollup_logstash`
|
||||||
|
|
||||||
|
|Frequency
|
||||||
|
|Every day at midnight
|
||||||
|
|
||||||
|
|Page size
|
||||||
|
|1000
|
||||||
|
|
||||||
|
|Delay (latency buffer)|7d
|
||||||
|
|
||||||
|
|Date field
|
||||||
|
|@timestamp
|
||||||
|
|
||||||
|
|Time bucket size
|
||||||
|
|60m
|
||||||
|
|
||||||
|
|Time zone
|
||||||
|
|UTC
|
||||||
|
|
||||||
|
|Terms
|
||||||
|
|geo.src, machine.os.keyword
|
||||||
|
|
||||||
|
|Histogram
|
||||||
|
|bytes, memory
|
||||||
|
|
||||||
|
|Histogram interval
|
||||||
|
|1000
|
||||||
|
|
||||||
|
|Metrics
|
||||||
|
|bytes (average)
|
||||||
|
|===
|
||||||
|
|
||||||
|
|
||||||
|
You can now use the rolled up data for analysis at a fraction of the storage cost
|
||||||
|
of the original index. The original data can live side by side with the new
|
||||||
|
rollup index, or you can remove or archive it using <<creating-index-lifecycle-policies,Index Lifecycle Management>>.
|
||||||
|
|
||||||
|
[float]
|
||||||
|
==== Visualize the rolled up data
|
||||||
|
|
||||||
|
Your next step is to visualize your rolled up data in a vertical bar chart.
|
||||||
|
Most visualizations support rolled up data, with the exception of Timelion, TSVB, and Vega visualizations.
|
||||||
|
|
||||||
|
Using the information from the example rollup configuration described above,
|
||||||
|
you can use `rollup_logstash` to match the rolled up index pattern,
|
||||||
|
and `kibana_sample_data_logs` to match the index pattern for raw data.
|
||||||
|
The notation for a combination index pattern with both raw and rolled up data
|
||||||
|
is `rollup_logstash,kibana_sample_data_logs`.
|
||||||
|
|
||||||
|
[role="screenshot"]
|
||||||
|
image::images/management_rollup_job_vis.png[][Visualization of rolled up data]
|
||||||
|
|
||||||
|
You can then create a dashboard that contains visualizations of the rolled up
|
||||||
|
data, raw data, or both. See <<visualize-rollup-data, Using rolled up data in a visualization>>
|
||||||
|
for more information.
|
||||||
|
|
||||||
|
[role="screenshot"]
|
||||||
|
image::images/management_rollup_job_dashboard.png[][Dashboard with rolled up data]
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
|
@ -140,6 +140,8 @@ Aggregation Execution Order, and You].
|
||||||
|
|
||||||
include::visualize/saving.asciidoc[]
|
include::visualize/saving.asciidoc[]
|
||||||
|
|
||||||
|
include::visualize/visualize_rollup_data.asciidoc[]
|
||||||
|
|
||||||
include::visualize/xychart.asciidoc[]
|
include::visualize/xychart.asciidoc[]
|
||||||
|
|
||||||
include::visualize/controls.asciidoc[]
|
include::visualize/controls.asciidoc[]
|
||||||
|
|
|
@ -1,6 +1,6 @@
|
||||||
[role="xpack"]
|
[role="xpack"]
|
||||||
[[visualize-rollup-data]]
|
[[visualize-rollup-data]]
|
||||||
=== Create a visualization using rolled up data
|
== Using rolled up data in a visualization
|
||||||
|
|
||||||
beta[]
|
beta[]
|
||||||
|
|
||||||
|
@ -8,9 +8,9 @@ You can visualize your rolled up data in a variety of charts, tables, maps, and
|
||||||
more. Most visualizations support rolled up data, with the exception of
|
more. Most visualizations support rolled up data, with the exception of
|
||||||
Timelion, TSVB, and Vega visualizations.
|
Timelion, TSVB, and Vega visualizations.
|
||||||
|
|
||||||
You create an index pattern for rolled up data the same way you do for any data,
|
To get started, go to *Management > Kibana > Index patterns.*
|
||||||
in *Management > Kibana > Index patterns*. Clicking *Create index pattern* includes
|
If a rollup index is detected in the cluster, *Create index pattern*
|
||||||
an item for creating a rollup index pattern, if a rollup index is detected in the cluster.
|
includes an item for creating a rollup index pattern.
|
||||||
|
|
||||||
[role="screenshot"]
|
[role="screenshot"]
|
||||||
image::images/management_create_rollup_menu.png[Create index pattern menu]
|
image::images/management_create_rollup_menu.png[Create index pattern menu]
|
||||||
|
@ -18,17 +18,8 @@ image::images/management_create_rollup_menu.png[Create index pattern menu]
|
||||||
You can match an index pattern to only rolled up data, or mix both rolled up
|
You can match an index pattern to only rolled up data, or mix both rolled up
|
||||||
and raw data to visualize all data together. An index pattern can match only one
|
and raw data to visualize all data together. An index pattern can match only one
|
||||||
rolled up index, not multiple. There is no restriction on the number of standard
|
rolled up index, not multiple. There is no restriction on the number of standard
|
||||||
indices that an index pattern can match.
|
indices that an index pattern can match. When matching multiple indices,
|
||||||
|
use a comma to separate the names, with no space after the comma.
|
||||||
Combination index patterns use the same
|
|
||||||
notation as other multiple indices in {es}. To match multiple indices to create a
|
|
||||||
combination index pattern, use a comma to separate the names, with no space after the comma.
|
|
||||||
The notation for wildcards (`*`) and the ability to "exclude" (`-`) also apply
|
|
||||||
(for example, `test*,-test3`).
|
|
||||||
|
|
||||||
When creating an index pattern, you’re asked to set a time field for filtering.
|
|
||||||
With a rollup index, the time filter field is the same field used for
|
|
||||||
the rolled up date histogram aggregation.
|
|
||||||
|
|
||||||
Keep the following in mind when creating a visualization from rolled up data:
|
Keep the following in mind when creating a visualization from rolled up data:
|
||||||
|
|
||||||
|
@ -39,15 +30,14 @@ numeric field values or terms. You can ask for a time aggregation that takes
|
||||||
several time buckets and combines them to lower granularity. For example,
|
several time buckets and combines them to lower granularity. For example,
|
||||||
if the rollup job was aggregated by hours, you can ask for buckets of days.
|
if the rollup job was aggregated by hours, you can ask for buckets of days.
|
||||||
|
|
||||||
The data represented in this visualization comes from a rollup index and
|
The following visualization of rolled up data shows the date histogram
|
||||||
standard indices.
|
interval multiple and the limited metrics aggregations.
|
||||||
|
|
||||||
[role="screenshot"]
|
[role="screenshot"]
|
||||||
image::images/management_rollups_visualization.png[][Rollups in visualizations]
|
image::images/management_rollups_visualization.png[][Rollups in visualizations]
|
||||||
|
|
||||||
You can mix rollup visualizations and regular visualizations in a dashboard.
|
Dashboards can have a mixture of rollup visualizations and regular visualizations,
|
||||||
The following dashboard shows this mix, along with a field filter. Note
|
as shown in the following figure. Note that not all queries and filters support rollups.
|
||||||
that not all queries and filters are supported by rollups.
|
|
||||||
|
|
||||||
[role="screenshot"]
|
[role="screenshot"]
|
||||||
image::images/management_rolled_dashboard.png[][Rollups in dashboards]
|
image::images/management_rolled_dashboard.png[][Rollups in dashboards]
|