[DOCS] Updates images and adds example to rollup jobs doc (#41839)

* [DOCS] Updates images and adds example to rollup jobs doc

* [DOCS] Edits to rollup text
This commit is contained in:
gchaps 2019-07-24 08:16:03 -07:00 committed by GitHub
parent 3dc4595d6b
commit f1ca9b50e6
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
12 changed files with 133 additions and 85 deletions

BIN
docs/images/management_create_rollup_job.png Normal file → Executable file

Binary file not shown.

Before

Width:  |  Height:  |  Size: 147 KiB

After

Width:  |  Height:  |  Size: 182 KiB

BIN
docs/images/management_create_rollup_menu.png Normal file → Executable file

Binary file not shown.

Before

Width:  |  Height:  |  Size: 177 KiB

After

Width:  |  Height:  |  Size: 143 KiB

BIN
docs/images/management_rolled_dashboard.png Normal file → Executable file

Binary file not shown.

Before

Width:  |  Height:  |  Size: 251 KiB

After

Width:  |  Height:  |  Size: 86 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 213 KiB

BIN
docs/images/management_rollup_job_details.png Normal file → Executable file

Binary file not shown.

Before

Width:  |  Height:  |  Size: 146 KiB

After

Width:  |  Height:  |  Size: 189 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 217 KiB

BIN
docs/images/management_rollup_list.png Normal file → Executable file

Binary file not shown.

Before

Width:  |  Height:  |  Size: 96 KiB

After

Width:  |  Height:  |  Size: 129 KiB

BIN
docs/images/management_rollups_visualization.png Normal file → Executable file

Binary file not shown.

Before

Width:  |  Height:  |  Size: 301 KiB

After

Width:  |  Height:  |  Size: 137 KiB

View file

@ -17,8 +17,6 @@ include::management/index-patterns.asciidoc[]
include::management/rollups/create_and_manage_rollups.asciidoc[]
include::management/rollups/visualize_rollup_data.asciidoc[]
include::management/index-lifecycle-policies/intro-to-lifecycle-policies.asciidoc[]
include::management/index-lifecycle-policies/create-policy.asciidoc[]

View file

@ -1,90 +1,148 @@
[role="xpack"]
[[data-rollups]]
== Working with rollup indices
The {ref}/xpack-rollup.html[rollup feature in {es}]
enables you to summarize historical data and store it compactly for future analysis,
so you can query, aggregate, and visualize the data using a fraction of the storage.
This is a good way to keep costs down when you need to store months or years of
historical data for use in visualizations and reports.
{kib} supports rolled up data in two ways:
* You can create and manage a rollup job in Management
* You can create a visualization using rolled up data in
Visualize and view it in a dashboard
== Rollup jobs
[role="xpack"]
[[create-and-manage-rollup-job]]
=== Create and manage rollup jobs
A rollup job is a periodic task that aggregates data from indices specified
by an index pattern and rolls it into a new index. Rollup indices are a good way to
compactly store months or years of historical
data for use in visualizations and reports.
In Management, you'll find a UI for viewing, creating, starting, stopping, and
deleting rollup jobs. A rollup job is a periodic task that summarizes data from
indices specified by an index pattern and rolls it into a new index. To navigate
to the UI, go to *Management*, and under *Elasticsearch*, click *Rollup Jobs*.
Youll find *Rollup Jobs* under *Management > Elasticsearch*. With this UI,
you can:
* <<create-and-manage-rollup-job, Create a rollup job>>
* <<manage-rollup-job, Start&comma; stop&comma; and delete rollup jobs>>
[role="screenshot"]
image::images/management_rollup_list.png[][List of currently active rollup jobs]
Before using this feature, you should be familiar with how rollups work.
{ref}/xpack-rollup.html[Rolling up historical data] is a good source for more detailed information.
[float]
[[create-rollup-job]]
==== Creating a rollup job
[[create-and-manage-rollup-job]]
=== Create a rollup job
{kib} makes it easy for you to create a rollup job by walking you through the
process step by step. The first step is to define the job logistics. These include
the name of the rollup job, the index or indices to summarize, and the output rollup index.
{kib} makes it easy for you to create a rollup job by walking you through
the process. You fill in the name, data flow, and how often you want to roll
up the data. Then you define a date histogram aggregation for the rollup job
and optionally terms, histogram, and metrics aggregations.
The index pattern cannot match the name of the output rollup index. For example,
if your index pattern is `metricbeat-*`, you cannot name your rollup index
`metricbeat-rollup`. Otherwise, the job will attempt to capture the data in the
rollup index.
When defining the index pattern, you must enter a name that is different than
the output rollup index. Otherwise, the job
will attempt to capture the data in the rollup index. For example, if your index pattern is `metricbeat-*`,
you can name your rollup index `rollup-metricbeat`, but not `metricbeat-rollup`.
[role="screenshot"]
image::images/management_create_rollup_job.png[][Wizard that walks you through creation of a rollup job]
You must set a schedule for the rollup job: how often to collect the data,
the number of documents to roll up at a time, and the duration of its latency.
The latency buffer field is provided to protect against the late arrival of data
from Beats or other sources. By delaying the rollup for the specified amount of
time from when the job starts, you allow for the inclusion of late-arriving data
in the rollup.
In the subsequent phases, you define the Date Histogram aggregation for the job
and optionally the Terms and Histogram aggregations.
* The Date Histogram aggregation defines the time intervals for summarizing the data.
This value is important because you cannot search the data with a smaller value
than this interval. However, you can aggregate buckets in a larger time interval.
* The Terms histogram enables you to split the time buckets into sub buckets for
term field values.
* The Histogram aggregation enables you to split the time buckets into sub buckets
for numeric field values.
The final step is to specify the fields for calculating metrics. For each selected
field, you can collect any or all of the following: value count, average, sum, min, and max.
Before you save the rollup job, {kib} displays a summary of the rollup job for
validation.
[float]
[[manage-rollup-job]]
==== Managing rollup jobs
=== Start, stop, and delete rollup jobs
Selecting a job on the *Rollup jobs* page shows its details. The Manage menu in
Once youve saved a rollup job, youll see it the *Rollup Jobs* overview page,
where you can drill down for further investigation. The *Manage* menu in
the lower right enables you to start, stop, and delete the rollup job.
You must first stop a rollup job before deleting it.
[role="screenshot"]
image::images/management_rollup_job_details.png[][Rollup job details]
You can start, stop, and delete an existing rollup job, but edits are not supported.
If you want to make any changes, delete the existing job and create a new one with
the updated specifications. Be sure to use a different name for the new rollup job;
reusing the same name could lead to problems with mismatched job configurations.
More about logistical details for the {ref}/rollup-job-config.html[rollup job configuration]
can be found in the {es} documentation.
You cant change a rollup job after youve created it. To select additional fields
or redefine terms, you must delete the existing job, and then create a new one
with the updated specifications. Be sure to use a different name for the new rollup
job&mdash;reusing the same name can lead to problems with mismatched job configurations.
You can read more at {ref}/rollup-job-config.html[rollup job configuration].
[float]
=== Try it: Create and visualize rolled up data
This example creates a rollup job to capture log data from sample web logs.
To follow along, add the <<add-sample-data, sample web logs data set>>.
In this example, you want data that is older than 7 days in the target index pattern `kibana_sample_data_logs`
to roll up once a day into the index `rollup_logstash`. Youll bucket the
rolled up data on an hourly basis, using 60m for the time bucket configuration.
This allows for more granular queries, such as 2h and 12h.
[float]
==== Create the rollup job
As you walk through the *Create rollup job* UI, enter the data shown in
the table below. The terms, histogram, and metrics fields reflect
the key information to retain in the rolled up data: where visitors are from (geo.src),
what operating system they are using (machine.os.keyword),
and how much data is being sent (bytes).
|===
|*Field* |*Value*
|Name
|logs_job
|Index pattern
|`kibana_sample_data_logs`
|Rollup index name
|`rollup_logstash`
|Frequency
|Every day at midnight
|Page size
|1000
|Delay (latency buffer)|7d
|Date field
|@timestamp
|Time bucket size
|60m
|Time zone
|UTC
|Terms
|geo.src, machine.os.keyword
|Histogram
|bytes, memory
|Histogram interval
|1000
|Metrics
|bytes (average)
|===
You can now use the rolled up data for analysis at a fraction of the storage cost
of the original index. The original data can live side by side with the new
rollup index, or you can remove or archive it using <<creating-index-lifecycle-policies,Index Lifecycle Management>>.
[float]
==== Visualize the rolled up data
Your next step is to visualize your rolled up data in a vertical bar chart.
Most visualizations support rolled up data, with the exception of Timelion, TSVB, and Vega visualizations.
Using the information from the example rollup configuration described above,
you can use `rollup_logstash` to match the rolled up index pattern,
and `kibana_sample_data_logs` to match the index pattern for raw data.
The notation for a combination index pattern with both raw and rolled up data
is `rollup_logstash,kibana_sample_data_logs`.
[role="screenshot"]
image::images/management_rollup_job_vis.png[][Visualization of rolled up data]
You can then create a dashboard that contains visualizations of the rolled up
data, raw data, or both. See <<visualize-rollup-data, Using rolled up data in a visualization>>
for more information.
[role="screenshot"]
image::images/management_rollup_job_dashboard.png[][Dashboard with rolled up data]

View file

@ -140,6 +140,8 @@ Aggregation Execution Order, and You].
include::visualize/saving.asciidoc[]
include::visualize/visualize_rollup_data.asciidoc[]
include::visualize/xychart.asciidoc[]
include::visualize/controls.asciidoc[]

View file

@ -1,6 +1,6 @@
[role="xpack"]
[[visualize-rollup-data]]
=== Create a visualization using rolled up data
== Using rolled up data in a visualization
beta[]
@ -8,9 +8,9 @@ You can visualize your rolled up data in a variety of charts, tables, maps, and
more. Most visualizations support rolled up data, with the exception of
Timelion, TSVB, and Vega visualizations.
You create an index pattern for rolled up data the same way you do for any data,
in *Management > Kibana > Index patterns*. Clicking *Create index pattern* includes
an item for creating a rollup index pattern, if a rollup index is detected in the cluster.
To get started, go to *Management > Kibana > Index patterns.*
If a rollup index is detected in the cluster, *Create index pattern*
includes an item for creating a rollup index pattern.
[role="screenshot"]
image::images/management_create_rollup_menu.png[Create index pattern menu]
@ -18,17 +18,8 @@ image::images/management_create_rollup_menu.png[Create index pattern menu]
You can match an index pattern to only rolled up data, or mix both rolled up
and raw data to visualize all data together. An index pattern can match only one
rolled up index, not multiple. There is no restriction on the number of standard
indices that an index pattern can match.
Combination index patterns use the same
notation as other multiple indices in {es}. To match multiple indices to create a
combination index pattern, use a comma to separate the names, with no space after the comma.
The notation for wildcards (`*`) and the ability to "exclude" (`-`) also apply
(for example, `test*,-test3`).
When creating an index pattern, youre asked to set a time field for filtering.
With a rollup index, the time filter field is the same field used for
the rolled up date histogram aggregation.
indices that an index pattern can match. When matching multiple indices,
use a comma to separate the names, with no space after the comma.
Keep the following in mind when creating a visualization from rolled up data:
@ -39,15 +30,14 @@ numeric field values or terms. You can ask for a time aggregation that takes
several time buckets and combines them to lower granularity. For example,
if the rollup job was aggregated by hours, you can ask for buckets of days.
The data represented in this visualization comes from a rollup index and
standard indices.
The following visualization of rolled up data shows the date histogram
interval multiple and the limited metrics aggregations.
[role="screenshot"]
image::images/management_rollups_visualization.png[][Rollups in visualizations]
You can mix rollup visualizations and regular visualizations in a dashboard.
The following dashboard shows this mix, along with a field filter. Note
that not all queries and filters are supported by rollups.
Dashboards can have a mixture of rollup visualizations and regular visualizations,
as shown in the following figure. Note that not all queries and filters support rollups.
[role="screenshot"]
image::images/management_rolled_dashboard.png[][Rollups in dashboards]