2019-04-01 17:58:59 +02:00
|
|
|
|
[role="xpack"]
|
2018-11-09 21:57:59 +01:00
|
|
|
|
[[data-rollups]]
|
2020-02-06 18:16:32 +01:00
|
|
|
|
== Rollup Jobs
|
2018-11-09 21:57:59 +01:00
|
|
|
|
|
|
|
|
|
|
2020-02-06 18:16:32 +01:00
|
|
|
|
A rollup job is a periodic task that aggregates data from indices specified
|
|
|
|
|
by an index pattern, and then rolls it into a new index. Rollup indices are a good way to
|
|
|
|
|
compactly store months or years of historical
|
2019-07-24 17:16:03 +02:00
|
|
|
|
data for use in visualizations and reports.
|
2018-11-09 21:57:59 +01:00
|
|
|
|
|
2020-02-06 18:16:32 +01:00
|
|
|
|
You’ll find *Rollup Jobs* under *Management > Elasticsearch*. With this UI,
|
2019-07-24 17:16:03 +02:00
|
|
|
|
you can:
|
2018-11-09 21:57:59 +01:00
|
|
|
|
|
2019-07-24 17:16:03 +02:00
|
|
|
|
* <<create-and-manage-rollup-job, Create a rollup job>>
|
|
|
|
|
* <<manage-rollup-job, Start, stop, and delete rollup jobs>>
|
2018-11-09 21:57:59 +01:00
|
|
|
|
|
|
|
|
|
[role="screenshot"]
|
|
|
|
|
image::images/management_rollup_list.png[][List of currently active rollup jobs]
|
|
|
|
|
|
2020-02-06 18:16:32 +01:00
|
|
|
|
Before using this feature, you should be familiar with how rollups work.
|
|
|
|
|
{ref}/xpack-rollup.html[Rolling up historical data] is a good source for more detailed information.
|
2019-07-24 17:16:03 +02:00
|
|
|
|
|
2018-11-09 21:57:59 +01:00
|
|
|
|
[float]
|
2019-07-24 17:16:03 +02:00
|
|
|
|
[[create-and-manage-rollup-job]]
|
|
|
|
|
=== Create a rollup job
|
2018-11-09 21:57:59 +01:00
|
|
|
|
|
2020-02-06 18:16:32 +01:00
|
|
|
|
{kib} makes it easy for you to create a rollup job by walking you through
|
|
|
|
|
the process. You fill in the name, data flow, and how often you want to roll
|
|
|
|
|
up the data. Then you define a date histogram aggregation for the rollup job
|
|
|
|
|
and optionally define terms, histogram, and metrics aggregations.
|
2018-11-09 21:57:59 +01:00
|
|
|
|
|
2020-02-06 18:16:32 +01:00
|
|
|
|
When defining the index pattern, you must enter a name that is different than
|
|
|
|
|
the output rollup index. Otherwise, the job
|
|
|
|
|
will attempt to capture the data in the rollup index. For example, if your index pattern is `metricbeat-*`,
|
|
|
|
|
you can name your rollup index `rollup-metricbeat`, but not `metricbeat-rollup`.
|
2018-11-09 21:57:59 +01:00
|
|
|
|
|
|
|
|
|
[role="screenshot"]
|
|
|
|
|
image::images/management_create_rollup_job.png[][Wizard that walks you through creation of a rollup job]
|
|
|
|
|
|
2019-07-24 17:16:03 +02:00
|
|
|
|
[float]
|
|
|
|
|
[[manage-rollup-job]]
|
|
|
|
|
=== Start, stop, and delete rollup jobs
|
2018-11-09 21:57:59 +01:00
|
|
|
|
|
2020-02-06 18:16:32 +01:00
|
|
|
|
Once you’ve saved a rollup job, you’ll see it the *Rollup Jobs* overview page,
|
2020-04-09 19:42:30 +02:00
|
|
|
|
where you can drill down for further investigation. The *Manage* menu enables
|
|
|
|
|
you to start, stop, and delete the rollup job.
|
2019-07-24 17:16:03 +02:00
|
|
|
|
You must first stop a rollup job before deleting it.
|
2018-11-09 21:57:59 +01:00
|
|
|
|
|
2019-07-24 17:16:03 +02:00
|
|
|
|
[role="screenshot"]
|
|
|
|
|
image::images/management_rollup_job_details.png[][Rollup job details]
|
2018-11-09 21:57:59 +01:00
|
|
|
|
|
2020-02-06 18:16:32 +01:00
|
|
|
|
You can’t change a rollup job after you’ve created it. To select additional fields
|
|
|
|
|
or redefine terms, you must delete the existing job, and then create a new one
|
|
|
|
|
with the updated specifications. Be sure to use a different name for the new rollup
|
|
|
|
|
job—reusing the same name can lead to problems with mismatched job configurations.
|
|
|
|
|
You can read more at {ref}/rollup-job-config.html[rollup job configuration].
|
2018-11-09 21:57:59 +01:00
|
|
|
|
|
2019-07-24 17:16:03 +02:00
|
|
|
|
[float]
|
2020-03-10 22:37:17 +01:00
|
|
|
|
[[rollup-data-tutorial]]
|
2019-07-24 17:16:03 +02:00
|
|
|
|
=== Try it: Create and visualize rolled up data
|
2018-11-09 21:57:59 +01:00
|
|
|
|
|
2020-02-06 18:16:32 +01:00
|
|
|
|
This example creates a rollup job to capture log data from sample web logs.
|
2020-05-27 22:01:29 +02:00
|
|
|
|
To follow along, add the <<get-data-in, sample web logs data set>>.
|
2018-11-09 21:57:59 +01:00
|
|
|
|
|
2019-07-24 17:16:03 +02:00
|
|
|
|
In this example, you want data that is older than 7 days in the target index pattern `kibana_sample_data_logs`
|
2020-02-06 18:16:32 +01:00
|
|
|
|
to roll up once a day into the index `rollup_logstash`. You’ll bucket the
|
|
|
|
|
rolled up data on an hourly basis, using 60m for the time bucket configuration.
|
2019-07-24 17:16:03 +02:00
|
|
|
|
This allows for more granular queries, such as 2h and 12h.
|
2018-11-09 21:57:59 +01:00
|
|
|
|
|
|
|
|
|
[float]
|
2019-07-24 17:16:03 +02:00
|
|
|
|
==== Create the rollup job
|
2018-11-09 21:57:59 +01:00
|
|
|
|
|
2020-03-16 15:58:16 +01:00
|
|
|
|
As you walk through the *Create rollup job* UI, enter the data:
|
2019-07-24 17:16:03 +02:00
|
|
|
|
|
|
|
|
|
|===
|
|
|
|
|
|*Field* |*Value*
|
|
|
|
|
|
|
|
|
|
|Name
|
|
|
|
|
|logs_job
|
|
|
|
|
|
|
|
|
|
|Index pattern
|
|
|
|
|
|`kibana_sample_data_logs`
|
|
|
|
|
|
|
|
|
|
|Rollup index name
|
|
|
|
|
|`rollup_logstash`
|
|
|
|
|
|
|
|
|
|
|Frequency
|
|
|
|
|
|Every day at midnight
|
|
|
|
|
|
|
|
|
|
|Page size
|
|
|
|
|
|1000
|
|
|
|
|
|
|
|
|
|
|Delay (latency buffer)|7d
|
|
|
|
|
|
|
|
|
|
|Date field
|
|
|
|
|
|@timestamp
|
|
|
|
|
|
|
|
|
|
|Time bucket size
|
|
|
|
|
|60m
|
|
|
|
|
|
|
|
|
|
|Time zone
|
|
|
|
|
|UTC
|
|
|
|
|
|
|
|
|
|
|Terms
|
|
|
|
|
|geo.src, machine.os.keyword
|
|
|
|
|
|
|
|
|
|
|Histogram
|
|
|
|
|
|bytes, memory
|
|
|
|
|
|
|
|
|
|
|Histogram interval
|
|
|
|
|
|1000
|
|
|
|
|
|
|
|
|
|
|Metrics
|
|
|
|
|
|bytes (average)
|
|
|
|
|
|===
|
|
|
|
|
|
2020-03-16 15:58:16 +01:00
|
|
|
|
The terms, histogram, and metrics fields reflect
|
|
|
|
|
the key information to retain in the rolled up data: where visitors are from (geo.src),
|
|
|
|
|
what operating system they are using (machine.os.keyword),
|
|
|
|
|
and how much data is being sent (bytes).
|
2019-07-24 17:16:03 +02:00
|
|
|
|
|
2020-02-06 18:16:32 +01:00
|
|
|
|
You can now use the rolled up data for analysis at a fraction of the storage cost
|
|
|
|
|
of the original index. The original data can live side by side with the new
|
2019-07-24 17:16:03 +02:00
|
|
|
|
rollup index, or you can remove or archive it using <<creating-index-lifecycle-policies,Index Lifecycle Management>>.
|
|
|
|
|
|
|
|
|
|
[float]
|
|
|
|
|
==== Visualize the rolled up data
|
|
|
|
|
|
2020-02-06 18:16:32 +01:00
|
|
|
|
Your next step is to visualize your rolled up data in a vertical bar chart.
|
2020-05-27 22:01:29 +02:00
|
|
|
|
Most visualizations support rolled up data, with the exception of Timelion and Vega visualizations.
|
2019-07-24 17:16:03 +02:00
|
|
|
|
|
2020-05-27 22:01:29 +02:00
|
|
|
|
|
|
|
|
|
. Create the rollup index pattern in *Management > Index Patterns* so you can
|
|
|
|
|
select your rolled up data for visualizations. Click *Create index pattern*, and select *Rollup index pattern* from the dropdown.
|
|
|
|
|
+
|
|
|
|
|
[role="screenshot"]
|
|
|
|
|
image::images/management-rollup-index-pattern.png[][Create rollup index pattern]
|
|
|
|
|
|
|
|
|
|
. Enter *rollup_logstash,kibana_sample_logs* as your *Index Pattern* and `@timestamp`
|
|
|
|
|
as the *Time Filter field name*.
|
|
|
|
|
+
|
2020-02-06 18:16:32 +01:00
|
|
|
|
The notation for a combination index pattern with both raw and rolled up data
|
2020-05-27 22:01:29 +02:00
|
|
|
|
is `rollup_logstash,kibana_sample_data_logs`. In this index pattern, `rollup_logstash`
|
|
|
|
|
matches the rolled up index pattern and `kibana_sample_data_logs` matches the index
|
|
|
|
|
pattern for raw data.
|
2018-11-09 21:57:59 +01:00
|
|
|
|
|
2020-05-27 22:01:29 +02:00
|
|
|
|
. Go to *Visualize* and create a vertical bar chart. Choose `rollup_logstash,kibana_sample_data_logs`
|
|
|
|
|
as your source to see both the raw and rolled up data.
|
|
|
|
|
+
|
2018-11-09 21:57:59 +01:00
|
|
|
|
[role="screenshot"]
|
2020-05-27 22:01:29 +02:00
|
|
|
|
image::images/management-create-rollup-bar-chart.png[][Create visualization of rolled up data]
|
2019-07-24 17:16:03 +02:00
|
|
|
|
|
2020-05-27 22:01:29 +02:00
|
|
|
|
. Look at the data in your visualization.
|
|
|
|
|
+
|
|
|
|
|
[role="screenshot"]
|
|
|
|
|
image::images/management_rollup_job_vis.png[][Visualization of rolled up data]
|
2019-07-24 17:16:03 +02:00
|
|
|
|
|
2020-05-27 22:01:29 +02:00
|
|
|
|
. Optionally, create a dashboard that contains visualizations of the rolled up
|
|
|
|
|
data, raw data, or both.
|
|
|
|
|
+
|
2019-07-24 17:16:03 +02:00
|
|
|
|
[role="screenshot"]
|
|
|
|
|
image::images/management_rollup_job_dashboard.png[][Dashboard with rolled up data]
|