* Add tutorial for ILM with filebeat * Change screenshots and add additional steps * Update screenshots, add numbered steps, and other minor edits * Incorporate feedback: update links, formatting, and minor edits * Move tip inline with list * Apply suggestions from code review Co-Authored-By: James Rodewig <james.rodewig@elastic.co> * Move TIP inline . . . again * Put TIP inline Co-authored-by: James Rodewig <james.rodewig@elastic.co> Co-authored-by: James Rodewig <james.rodewig@elastic.co>
This commit is contained in:
parent
942eb4e420
commit
33eeb47ee5
BIN
docs/images/tutorial-ilm-custom-policy.png
Normal file
BIN
docs/images/tutorial-ilm-custom-policy.png
Normal file
Binary file not shown.
After Width: | Height: | Size: 471 KiB |
BIN
docs/images/tutorial-ilm-delete-phase-creation.png
Normal file
BIN
docs/images/tutorial-ilm-delete-phase-creation.png
Normal file
Binary file not shown.
After Width: | Height: | Size: 69 KiB |
BIN
docs/images/tutorial-ilm-delete-rollover.png
Normal file
BIN
docs/images/tutorial-ilm-delete-rollover.png
Normal file
Binary file not shown.
After Width: | Height: | Size: 121 KiB |
BIN
docs/images/tutorial-ilm-hotphaserollover-default.png
Normal file
BIN
docs/images/tutorial-ilm-hotphaserollover-default.png
Normal file
Binary file not shown.
After Width: | Height: | Size: 247 KiB |
BIN
docs/images/tutorial-ilm-modify-default-warm-phase-rollover.png
Normal file
BIN
docs/images/tutorial-ilm-modify-default-warm-phase-rollover.png
Normal file
Binary file not shown.
After Width: | Height: | Size: 444 KiB |
|
@ -1,23 +1,179 @@
|
|||
[role="xpack"]
|
||||
|
||||
[[example-using-index-lifecycle-policy]]
|
||||
=== Example of using an index lifecycle policy
|
||||
=== Tutorial: Use {ilm-init} to manage {filebeat} time-based indices
|
||||
|
||||
A common use case for managing index lifecycle policies is when you’re using
|
||||
{beats-ref}/beats-reference.html[Beats] to continually send time-series data,
|
||||
such as metrics and log data, to {es}. When you create the Beats packages, an
|
||||
index template is installed. The template includes a default policy to apply
|
||||
when new indices are created.
|
||||
With {ilm} ({ilm-init}), you can create policies that perform actions automatically
|
||||
on indices as they age and grow. {ilm-init} policies help you to manage
|
||||
performance, resilience, and retention of your data during its lifecycle. This tutorial shows
|
||||
you how to use {kib}’s *Index Lifecycle Policies* to modify and create {ilm-init}
|
||||
policies. You can learn more about all of the actions, benefits, and lifecycle
|
||||
phases in the {ref}/overview-index-lifecycle-management.html[{ilm-init} overview].
|
||||
|
||||
You can edit the policy in {kib}'s *Index Lifecycle Policies*. For example, you might:
|
||||
|
||||
* Rollover the index when it reaches 50 GB in size or is 30 days old. These
|
||||
settings are the default for the Beats lifecycle policy. This avoids
|
||||
having 1000s of tiny indices. When a rollover occurs, a new “hot” index is
|
||||
created and added to the index alias.
|
||||
[discrete]
|
||||
[[example-using-index-lifecycle-policy-scenario]]
|
||||
==== Scenario
|
||||
|
||||
* Move the index into the warm phase, shrink the index down to a single shard,
|
||||
and force merge to a single segment.
|
||||
You’re tasked with sending syslog files to an {es} cluster. This
|
||||
log data has the following data retention guidelines:
|
||||
|
||||
* After 60 days, move the index into the cold phase and onto less expensive hardware.
|
||||
* Keep logs on hot data nodes for 30 days
|
||||
* Roll over to a new index if the size reaches 50GB
|
||||
* After 30 days:
|
||||
** Move the logs to warm data nodes
|
||||
** Set {ref}/glossary.html#glossary-replica-shard[replica shards] to 1
|
||||
** {ref}/indices-forcemerge.html[Force merge] multiple index segments to free up the space used by deleted documents
|
||||
* Delete logs after 90 days
|
||||
|
||||
* Delete the index after 90 days.
|
||||
|
||||
[discrete]
|
||||
[[example-using-index-lifecycle-policy-prerequisites]]
|
||||
==== Prerequisites
|
||||
|
||||
To complete this tutorial, you'll need:
|
||||
|
||||
* An {es} cluster with hot and warm nodes configured for shard allocation
|
||||
awareness. If you’re using {cloud}/ec-getting-started-templates-hot-warm.html[{ess}],
|
||||
choose the hot-warm architecture deployment template.
|
||||
|
||||
+
|
||||
For a self-managed cluster, add node attributes as described for {ref}/shard-allocation-filtering.html[shard allocation filtering]
|
||||
to label data nodes as hot or warm. This step is required to migrate shards between
|
||||
nodes configured with specific hardware for the hot or warm phases.
|
||||
+
|
||||
For example, you can set this in your `elasticsearch.yml` for each data node:
|
||||
+
|
||||
[source,yaml]
|
||||
--------------------------------------------------------------------------------
|
||||
node.attr.data: "warm"
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
* A server with {filebeat} installed and configured to send logs to the `elasticsearch`
|
||||
output as described in {filebeat-ref}/filebeat-getting-started.html[Getting Started with {filebeat}].
|
||||
|
||||
[discrete]
|
||||
[[example-using-index-lifecycle-policy-view-fb-ilm-policy]]
|
||||
==== View the {filebeat} {ilm-init} policy
|
||||
|
||||
{filebeat} includes a default {ilm-init} policy that enables rollover. {ilm-init}
|
||||
is enabled automatically if you’re using the default `filebeat.yml` and index template.
|
||||
|
||||
To view the default policy in {kib}, go to *Management > Index Lifecycle Policies*,
|
||||
search for _filebeat_, and choose the _filebeat-version_ policy.
|
||||
|
||||
This policy initiates the rollover action when the index size reaches 50GB or
|
||||
becomes 30 days old.
|
||||
|
||||
[role="screenshot"]
|
||||
image::images/tutorial-ilm-hotphaserollover-default.png["Default policy"]
|
||||
|
||||
|
||||
[float]
|
||||
==== Modify the policy
|
||||
|
||||
The default policy is enough to prevent the creation of many tiny daily indices.
|
||||
You can modify the policy to meet more complex requirements.
|
||||
|
||||
. Activate the warm phase.
|
||||
|
||||
+
|
||||
. Set either of the following options to control when the index moves to the warm phase:
|
||||
|
||||
** Provide a value for *Timing for warm phase*. Setting this to *15* keeps the
|
||||
indices on hot nodes for a range of 15-45 days, depending on when the initial
|
||||
rollover occurred.
|
||||
|
||||
** Enable *Move to warm phase on rollover*. The index might move to the warm phase
|
||||
more quickly than intended if it reaches the *Maximum index size* before the
|
||||
the *Maximum age*.
|
||||
|
||||
. In the *Select a node attribute to control shard allocation* dropdown, select
|
||||
*data:warm(2)* to migrate shards to warm data nodes.
|
||||
|
||||
. Change *Number of replicas* to *1*.
|
||||
|
||||
. Enable *Force merge data* and set *Number of segments* to *1*.
|
||||
+
|
||||
NOTE: When rollover is enabled in the hot phase, action timing in the other phases
|
||||
is based on the rollover date.
|
||||
|
||||
+
|
||||
[role="screenshot"]
|
||||
image::images/tutorial-ilm-modify-default-warm-phase-rollover.png["Modify to add warm phase"]
|
||||
|
||||
. Activate the delete phase and set *Timing for delete phase* to *90* days.
|
||||
+
|
||||
[role="screenshot"]
|
||||
image::images/tutorial-ilm-delete-rollover.png["Add a delete phase"]
|
||||
|
||||
[float]
|
||||
==== Create a custom policy
|
||||
|
||||
If meeting a specific retention time period is most important, you can create a
|
||||
custom policy. For this option, you will use {filebeat} daily indices without
|
||||
rollover.
|
||||
|
||||
. Create a custom policy in {kib}, go to *Management > Index Lifecycle Policies >
|
||||
Create Policy*.
|
||||
|
||||
. Activate the warm phase and configure it as follows:
|
||||
+
|
||||
|===
|
||||
|*Setting* |*Value*
|
||||
|
||||
|Timing for warm phase
|
||||
|30 days from index creation
|
||||
|
||||
|Node attribute
|
||||
|`data:warm`
|
||||
|
||||
|Number of replicas
|
||||
|1
|
||||
|
||||
|Force merge data
|
||||
|enable
|
||||
|
||||
|Number of segments
|
||||
|1
|
||||
|===
|
||||
|
||||
+
|
||||
[role="screenshot"]
|
||||
image::images/tutorial-ilm-custom-policy.png["Modify the custom policy to add a warm phase"]
|
||||
|
||||
|
||||
+
|
||||
. Activate the delete phase and set the timing.
|
||||
+
|
||||
|===
|
||||
|*Setting* |*Value*
|
||||
|Timing for delete phase
|
||||
|90
|
||||
|===
|
||||
|
||||
+
|
||||
[role="screenshot"]
|
||||
image::images/tutorial-ilm-delete-phase-creation.png["Delete phase"]
|
||||
|
||||
. Configure the index to use the new policy in *{kib} > Management > Index Lifecycle
|
||||
Policies*
|
||||
|
||||
.. Find your {ilm-init} policy.
|
||||
.. Click the *Actions* link next to your policy name.
|
||||
.. Choose *Add policy to index template*.
|
||||
.. Select your {filebeat} index template name from the *Index template* list. For example, `filebeat-7.5.x`.
|
||||
.. Click *Add Policy* to save the changes.
|
||||
|
||||
+
|
||||
NOTE: If you initially used the default {filebeat} {ilm-init} policy, you will
|
||||
see a notice that the template already has a policy associated with it. Confirm
|
||||
that you want to overwrite that configuration.
|
||||
|
||||
+
|
||||
+
|
||||
TIP: When you change the policy associated with the index template, the active
|
||||
index will continue to use the policy it was associated with at index creation
|
||||
unless you manually update it. The next new index will use the updated policy.
|
||||
For more reasons that your {ilm-init} policy changes might be delayed, see
|
||||
{ref}/update-lifecycle-policy.html#update-lifecycle-policy[Update Lifecycle Policy].
|
||||
|
|
Loading…
Reference in a new issue