[DOCS] Adds documentation for index lifecycle policies (#28705) (#28802)

* [DOCS] Adds documentation for index lifecycle policies

* [DOCS] Updated image for policy options to show all menu items

* Update create-policy.asciidoc

* [DOCS] Incorporated review comments on hot and warm phase

* [DOCS] Additional changes to warm phase

* [DOCS] Removed the word open in the warm phase
This commit is contained in:
gchaps 2019-01-16 10:47:05 -08:00 committed by GitHub
parent 5fd3742f5f
commit 5a3fcbd5be
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
9 changed files with 162 additions and 0 deletions

Binary file not shown.

After

Width:  |  Height:  |  Size: 145 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 90 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 189 KiB

View file

@ -19,6 +19,16 @@ include::management/rollups/create_and_manage_rollups.asciidoc[]
include::management/rollups/visualize_rollup_data.asciidoc[]
include::management/index-lifecycle-policies/intro-to-lifecycle-policies.asciidoc[]
include::management/index-lifecycle-policies/create-policy.asciidoc[]
include::management/index-lifecycle-policies/manage-policy.asciidoc[]
include::management/index-lifecycle-policies/add-policy-to-index.asciidoc[]
include::management/index-lifecycle-policies/example-index-lifecycle-policy.asciidoc[]
include::management/managing-fields.asciidoc[]
include::management/managing-indices.asciidoc[]

View file

@ -0,0 +1,17 @@
[[adding-policy-to-index]]
=== Adding a policy to an index
You can add a lifecycle policy to an index and view the status for indices
managed by a policy in *Management > {es} > Index Management*. This page lists your
{es} indices, which you can filter by lifecycle status and lifecycle phase.
To add a policy, select the index name and then select *Manage > Add lifecycle policy*.
Youll see the policy name, the phase the index is in, the current
action, and if any errors occurred performing that action.
To remove a policy from an index, select *Manage > Remove lifecycle policy*.
[role="screenshot"]
image::images/index_management_add_policy.png[][UI for adding a policy to an index]

View file

@ -0,0 +1,57 @@
[[creating-index-lifecycle-policies]]
=== Creating an index lifecycle policy
An index lifecycle policy enables you to define rules over when to perform
certain actions, such as a rollover or force merge, on an index. Index lifecycle
management automates execution of those actions at the right time.
When you create an index lifecycle policy, consider the tradeoffs between
performance and availability. As you move your index through the lifecycle,
youre likely moving your data to less performant hardware and reducing the
number of shards and replicas. Its important to ensure that the index
continues to have enough replicas to prevent data loss in the event of failures.
*Index Lifecycle Policies* is automatically enabled in {kib}. Go to
*Management > {es} > Index Lifecycle Policies*.
NOTE: If you dont want to use this feature, you can disable it by setting
`xpack.ilm.enabled` to false in your `kibana.yml` configuration file. If you
disable *Index Management*, then *Index Lifecycle Policies* is also disabled.
[role="screenshot"]
image::images/index-lifecycle-policies-create.png[][UI for creating an index lifecycle policy]
You can define up to four phases in the index lifecycle. For each phase, you
can enable actions to optimize performance for that phase. Transitioning
between phases is based on the age of the index.
The four phases of an index lifecycle policy are:
* *Hot.* The index is actively being queried and written to. You can optionally
roll over to a new index when the
original index reaches a specified size or age. When a rollover occurs, a new
index is created, added to the index alias, and designated as the new “hot”
index. You can still query the previous indices, but you only ever write to
the “hot” index. See {ref}/indices-rollover-index.html[Rollover index] for more information.
* *Warm.* The index is typically searched at a lower rate than when the data is
hot. The index is not used for storing for new data, but might occasionally add
late-arriving data, for example, from a Beat that had a network problem that's now fixed.
You can optionally shrink the number replicas and move the shards to a
different set of nodes with smaller or less performant hardware. You can also
reduce the number of primary shards and force merge the index into
smaller {ref}/indices-segments.html[segments].
* *Cold.* The index is no longer being updated and is seldom queried, but is
still searchable. If you have a big deployment, you can move it to even
less performant hardware. You might also reduce the number of replicas because
you expect the data to be queried less frequently.
* *Delete.* The index is no longer relevant. You can define when it is safe to
delete it.
The index lifecycle always includes an active hot phase. The warm, cold, and
delete phases are optional. For example, you might define all four phases for
one policy and only a hot and delete phase for another. See {ref}/_actions.html[Actions]
for more information on the actions available in each phase.

View file

@ -0,0 +1,22 @@
[[example-using-index-lifecycle-policy]]
=== Example of using an index lifecycle policy
A common use case for managing index lifecycle policies is when youre using
{beats-ref}/beats-reference.html[Beats] to continually send time-series data,
such as metrics and log data, to {es}. When you create the Beats packages, an
index template is installed. The template includes a default policy to apply
when new indices are created.
You can edit the policy in {kib}'s *Index Lifecycle Policies*. For example, you might:
* Rollover the index when it reaches 50 GB in size or is 30 days old. These
settings are the default for the Beats lifecycle policy. This avoids
having 1000s of tiny indices. When a rollover occurs, a new “hot” index is
created and added to the index alias.
* Move the index into the warm phase, shrink the index down to a single shard,
and force merge to a single segment.
* After 60 days, move the index into the cold phase and onto less expensive hardware.
* Delete the index after 90 days.

View file

@ -0,0 +1,29 @@
[[index-lifecycle-policies]]
== Index lifecycle policies
If you're working with time series data, you don't want to continually dump
everything into a single index. Instead, you might periodically roll over the
data to a new index to keep it from growing so big it's slow and expensive.
As the index ages and you query it less frequently, youll likely move it to
less expensive hardware and reduce the number of shards and replicas.
To automatically move an index through its lifecycle, you can create a policy
to define actions to perform on the index as it ages. Index lifecycle policies
are especially useful when working with {beats-ref}/beats-reference.html[Beats]
data shippers, which continually
send operational data, such as metrics and logs, to Elasticsearch. You can
automate a rollover to a new index when the existing index reaches a specified
size or age. This ensures that all indices have a similar size instead of having
daily indices where size can vary based on the number of Beats and the number
of events sent.
{kib}s *Index Lifecycle Policies* walks you through the process for creating
and configuring a policy. Before using this feature, you should be familiar
with index lifecycle management:
* For an introduction, see
{ref}/getting-started-index-lifecycle-management.html[Getting started with index
lifecycle management].
* To dig into the concepts and technical details, see
{ref}/index-lifecycle-management.html[Managing the index lifecycle].
* To check out the APIs, see {ref}/index-lifecycle-management-api.html[Index lifecycle management API].

View file

@ -0,0 +1,27 @@
[[managing-index-lifecycle-policies]]
=== Managing index lifecycle policies
Your configured policies appear on the *Index lifecycle policies* page.
You can update an existing index lifecycle policy to fix errors or change
strategies for newly created indices. To edit a policy, select its name.
[role="screenshot"]
image::images/index_lifecycle_policies_options.png[][UI for viewing and editing an index lifecycle policy]
In addition, you can:
* *View indices linked to the policy.* This is important when editing a policy.
Any changes you make affect all indices attached to the policy. The settings
for the current phase are cached, so the update doesnt affect that phase. This
prevents conflicts when youre modifying a phase that is currently executing on
an index. The changes takes effect when the next phase in the index lifecycle begins.
* *Add the policy to an index template.* When an index is automatically
created using the index template, the policy is applied. If the index is rolled
over, the policies for any matching index templates are applied to the newly
created index. For more information, see {ref}/indices-templates.html[Index templates].
* *Delete a policy.* You cant delete a policy that is currently in use or
recover a deleted index.