eed9383969
* Migrations v2 docs * Not all kibana distributions automatically restarted a killed process * Mention that we add a write block to the outdated index * Formating: collapse three notes into a single note with three bullet points * Update docs/setup/upgrade/upgrade-standard.asciidoc Co-authored-by: Josh Dover <1813008+joshdover@users.noreply.github.com> * Add table of outdated / upgraded indices per version of Kibana * Review feedback: separate section for multi-instance upgrade migrations * Review feedback: link to saved objects management * Review feedback: stronger wording for not deleting any indices Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Josh Dover <1813008+joshdover@users.noreply.github.com>
124 lines
8.9 KiB
Plaintext
124 lines
8.9 KiB
Plaintext
[[upgrade-migrations]]
|
|
=== Upgrade migrations
|
|
|
|
Every time {kib} is upgraded it will perform an upgrade migration to ensure that all <<managing-saved-objects,saved objects>> are compatible with the new version.
|
|
|
|
NOTE: 6.7 includes an https://www.elastic.co/guide/en/kibana/6.7/upgrade-assistant.html[Upgrade Assistant]
|
|
to help you prepare for your upgrade to 7.0. To access the assistant, go to *Management > 7.0 Upgrade Assistant*.
|
|
|
|
WARNING: {kib} 7.12.0 and later uses a new migration process and index naming scheme. Be sure to read the documentation for your version of {kib} before proceeding.
|
|
|
|
WARNING: The following instructions assumes {kib} is using the default index names. If the `kibana.index` or `xpack.tasks.index` configuration settings were changed these instructions will have to be adapted accordingly.
|
|
|
|
[float]
|
|
[[upgrade-migrations-process]]
|
|
==== Background
|
|
|
|
Saved objects are stored in two indices:
|
|
|
|
* `.kibana_{kibana_version}_001`, or if the `kibana.index` configuration setting is set `.{kibana.index}_{kibana_version}_001`. E.g. for Kibana v7.12.0 `.kibana_7.12.0_001`.
|
|
* `.kibana_task_manager_{kibana_version}_001`, or if the `xpack.tasks.index` configuration setting is set `.{xpack.tasks.index}_{kibana_version}_001` E.g. for Kibana v7.12.0 `.kibana_task_manager_7.12.0_001`.
|
|
|
|
The index aliases `.kibana` and `.kibana_task_manager` will always point to the most up-to-date version indices.
|
|
|
|
The first time a newer {kib} starts, it will first perform an upgrade migration before starting plugins or serving HTTP traffic. To prevent losing acknowledged writes old nodes should be shutdown before starting the upgrade. To reduce the likelihood of old nodes losing acknowledged writes, {kib} 7.12.0 and later will add a write block to the outdated index. Table 1 lists the saved objects indices used by previous versions of {kib}.
|
|
|
|
.Saved object indices and aliases per {kib} version
|
|
[options="header"]
|
|
[cols="a,a,a"]
|
|
|=======================
|
|
|Upgrading from version | Outdated index (alias) | Upgraded index (alias)
|
|
| 6.0.0 through 6.4.x | `.kibana` 1.3+^.^| `.kibana_7.12.0_001`
|
|
(`.kibana` alias)
|
|
|
|
`.kibana_task_manager_7.12.0_001` (`.kibana_task_manager` alias)
|
|
| 6.5.0 through 7.3.x | `.kibana_N` (`.kibana` alias)
|
|
| 7.4.0 through 7.11.x
|
|
| `.kibana_N` (`.kibana` alias)
|
|
|
|
`.kibana_task_manager_N` (`.kibana_task_manager` alias)
|
|
|=======================
|
|
|
|
==== Upgrading multiple {kib} instances
|
|
When upgrading several {kib} instances connected to the same {es} cluster, ensure that all outdated instances are shutdown before starting the upgrade.
|
|
|
|
Kibana does not support rolling upgrades. However, once outdated instances are shutdown, all upgraded instances can be started in parallel in which case all instances will participate in the upgrade migration in parallel.
|
|
|
|
For large deployments with more than 10 {kib} instances and more than 10 000 saved objects, the upgrade downtime can be reduced by bringing up a single {kib} instance and waiting for it to complete the upgrade migration before bringing up the remaining instances.
|
|
|
|
[float]
|
|
[[preventing-migration-failures]]
|
|
==== Preventing migration failures
|
|
This section highlights common causes of {kib} upgrade failures and how to prevent them.
|
|
|
|
[float]
|
|
===== Corrupt saved objects
|
|
We highly recommend testing your {kib} upgrade in a development cluster to discover and remedy problems caused by corrupt documents, especially when there are custom integrations creating saved objects in your environment. Saved objects that were corrupted through manual editing or integrations will cause migration failures with a log message like `Failed to transform document. Transform: index-pattern:7.0.0\n Doc: {...}` or `Unable to migrate the corrupt Saved Object document ...`. Corrupt documents will have to be fixed or deleted before an upgrade migration can succeed.
|
|
|
|
[float]
|
|
===== User defined index templates that causes new `.kibana*` indices to have incompatible settings or mappings
|
|
Matching index templates which specify `settings.refresh_interval` or `mappings` are known to interfere with {kib} upgrades.
|
|
|
|
Prevention: narrow down the index patterns of any user-defined index templates to ensure that these won't apply to new `.kibana*` indices.
|
|
|
|
Note: {kib} < 6.5 creates it's own index template called `kibana_index_template:.kibana` and index pattern `.kibana`. This index template will not interfere and does not need to be changed or removed.
|
|
|
|
[float]
|
|
===== An unhealthy {es} cluster
|
|
Problems with your {es} cluster can prevent {kib} upgrades from succeeding. Ensure that your cluster has:
|
|
|
|
* enough free disk space, at least twice the amount of storage taken up by the `.kibana` and `.kibana_task_manager` indices
|
|
* sufficient heap size
|
|
* a "green" cluster status
|
|
|
|
[float]
|
|
===== Different versions of {kib} connected to the same {es} index
|
|
When different versions of {kib} are attempting an upgrade migration in parallel this can lead to migration failures. Ensure that all {kib} instances are running the same version, configuration and plugins.
|
|
|
|
[float]
|
|
===== Incompatible `xpack.tasks.index` configuration setting
|
|
For {kib} versions prior to 7.5.1, if the task manager index is set to `.tasks` with the configuration setting `xpack.tasks.index: ".tasks"`, upgrade migrations will fail. {kib} 7.5.1 and later prevents this by refusing to start with an incompatible configuration setting.
|
|
|
|
[float]
|
|
[[resolve-migrations-failures]]
|
|
==== Resolving migration failures
|
|
|
|
If {kib} terminates unexpectedly while migrating a saved object index it will automatically attempt to perform the migration again once the process has restarted. Do not delete any saved objects indices to attempt to fix a failed migration. Unlike previous versions, {kib} version 7.12.0 and later does not require deleting any indices to release a failed migration lock.
|
|
|
|
If upgrade migrations fail repeatedly, follow the advice in (preventing migration failures)[preventing-migration-failures]. Once the root cause for the migration failure has been addressed, {kib} will automatically retry the migration without any further intervention. If you're unable to resolve a failed migration following these steps, please contact support.
|
|
|
|
[float]
|
|
[[upgrade-migrations-rolling-back]]
|
|
==== Rolling back to a previous version of {kib}
|
|
|
|
If you've followed the advice in (preventing migration failures)[preventing-migration-failures] and (resolving migration failures)[resolve-migrations-failures] and {kib} is still not able to upgrade successfully, you might choose to rollback {kib} until you're able to identify and fix the root cause.
|
|
|
|
WARNING: Before rolling back {kib}, ensure that the version you wish to rollback to is compatible with your {es} cluster. If the version you're rolling back to is not compatible, you will have to also rollback {es}. +
|
|
Any changes made after an upgrade will be lost when rolling back to a previous version.
|
|
|
|
In order to rollback after a failed upgrade migration, the saved object indices have to be rolled back to be compatible with the previous {kibana} version.
|
|
|
|
[float]
|
|
===== Rollback by restoring a backup snapshot:
|
|
|
|
1. Before proceeding ensure that you have a recent and successful backup snapshot of all `.kibana*` indices.
|
|
2. Shutdown all {kib} instances to be 100% sure that there are no instances currently performing a migration.
|
|
3. Delete all saved object indices with `DELETE /.kibana*`
|
|
4. Restore the `.kibana* indices and their aliases from the backup snapshot. See {es} {ref}/modules-snapshots.html[snapshots]
|
|
5. Start up all {kib} instances on the older version you wish to rollback to.
|
|
|
|
[float]
|
|
===== (Not recommended) Rollback without a backup snapshot:
|
|
|
|
1. Shutdown all {kib} instances to be 100% sure that there are no {kib} instances currently performing a migration.
|
|
2. Create a backup snapshot of the `.kibana*` indices.
|
|
3. Delete the version specific indices created by the failed upgrade migration. E.g. if you wish to rollback from a failed upgrade to v7.12.0 `DELETE /.kibana_7.12.0_*,.kibana_task_manager_7.12.0_*`
|
|
4. Inspect the output of `GET /_cat/aliases`. If either the `.kibana` and/or `.kibana_task_manager` alias is missing, these will have to be created manually. Find the latest index from the output of `GET /_cat/indices` and create the missing alias to point to the latest index. E.g. if the `.kibana` alias was missing and the latest index is `.kibana_3` create a new alias with `POST /.kibana_3/_aliases/.kibana`.
|
|
5. Remove the write block from the rollback indices. `PUT /.kibana,.kibana_task_manager/_settings {"index.blocks.write": false}`
|
|
6. Start up {kib} on the older version you wish to rollback to.
|
|
|
|
[float]
|
|
[[upgrade-migrations-old-indices]]
|
|
==== Handling old `.kibana_N` indices
|
|
|
|
After migrations have completed, there will be multiple {kib} indices in {es}: (`.kibana_1`, `.kibana_2`, `.kibana_7.12.0` etc). {kib} only uses the index that the `.kibana` and `.kibana_task_manager` alias points to. The other {kib} indices can be safely deleted, but are left around as a matter of historical record, and to facilitate rolling {kib} back to a previous version. |