kibana/docs/developer/advanced/development-es-snapshots.asciidoc

[[development-es-snapshots]]
== Daily {es} Snapshots

For local development and CI, {kib}, by default, uses {es} snapshots that are built daily when running tasks that require {es} (e.g. functional tests).

A snapshot is just a group of tarballs, one for each supported distribution/architecture/os of {es}, and a JSON-based manifest file containing metadata about the distributions.

https://ci.kibana.dev/es-snapshots[A dashboard] is available that shows the current status and compatibility of the latest {es} snapshots.

=== Process Overview

1. {es} snapshots are built for each current tracked branch of {kib}.
2. Each snapshot is uploaded to a public Google Cloud Storage bucket, `kibana-ci-es-snapshots-daily`.
** At this point, the snapshot is not automatically used in CI or local development. It needs to be tested/verified first.
3. Each snapshot is tested with the latest commit of the corresponding {kib} branch, using the full CI suite.
4. After CI
** If the snapshot passes, it is promoted and automatically used in CI and local development.
** If the snapshot fails, the issue must be investigated and resolved. A new incompatibility may exist between {es} and {kib}.

=== Using the latest snapshot

When developing locally, you may wish to use the most recent {es} snapshot, even if it's failing CI. To do so, prefix your commands with the follow environment variable:

["source","bash"]
-----------
KBN_ES_SNAPSHOT_USE_UNVERIFIED=true
-----------

You can use this flag with any command that downloads and runs {es} snapshots, such as `scripts/es` or the FTR.

For example, to run functional tests with the latest snapshot:

["source","bash"]
-----------
KBN_ES_SNAPSHOT_USE_UNVERIFIED=true node scripts/functional_tests_server
-----------

==== For Pull Requests

Currently, there is not a way to run your pull request with the latest unverified snapshot without a code change. You can, however, do it with a small code change.

1. Edit `Jenkinsfile` in the root of the {kib} repo
2. Add `env.KBN_ES_SNAPSHOT_USE_UNVERIFIED = 'true'` at the top of the file.
3. Commit the change

Your pull request should then use the latest snapshot the next time that it runs. Just don't merge the change to `Jenkinsfile`!

=== Google Cloud Storage buckets

==== kibana-ci-es-snapshots-daily

This bucket stores snapshots that are created on a daily basis, and is the primary location used by `kbn-es` to download snapshots.

Snapshots are automatically deleted after 10 days.

The file structure for this bucket looks like this:

* `<version>/manifest-latest.json`
* `<version>/manifest-latest-verified.json`
* `<version>/archives/<unique id>/*.tar.gz`
* `<version>/archives/<unique id>/*.tar.gz.sha512`
* `<version>/archives/<unique id>/manifest.json`

==== kibana-ci-es-snapshots-permanent

This bucket stores only the most recently promoted snapshot for each version. Old snapshots are only deleted when new ones are uploaded.

This bucket serves as permanent snapshot storage for old branches/versions that are no longer being built. `kbn-es` checks the daily bucket first, followed by this one if no snapshots were found.

The file structure for this bucket looks like this:

* `<version>/*.tar.gz`
* `<version>/*.tar.gz.sha512`
* `<version>/manifest.json`

=== How snapshots are built, tested, and promoted

Each day, a https://kibana-ci.elastic.co/job/elasticsearch+snapshots+trigger/[Jenkins job] runs that triggers {es} builds for each currently tracked branch/version. This job is automatically updated with the correct branches whenever we release new versions of {kib}.

==== Build

https://kibana-ci.elastic.co/job/elasticsearch+snapshots+build/[This Jenkins job] builds the {es} snapshots and uploads them to GCS.

The Jenkins job pipeline definition is https://github.com/elastic/kibana/blob/main/.ci/es-snapshots/Jenkinsfile_build_es[in the {kib} repo].

1. Checkout {es} repo for the given branch/version.
2. Run `./gradlew -p distribution/archives assemble --parallel` to create all of the {es} distributions.
3. Create a tarball for each distribution.
4. Create a manifest JSON file containing info about the distribution, as well as its download URL.
5. Upload the tarballs and manifest to a unique location in the GCS bucket `kibana-ci-es-snapshots-daily`.
** e.g. `<version>/archives/<unique id>`
6. Replace `<version>/manifest-latest.json` in GCS with this newest manifest.
** This allows the `KBN_ES_SNAPSHOT_USE_UNVERIFIED` flag to work.
7. Trigger the verification job, to run the full {kib} CI test suite with this snapshot.

==== Verification and Promotion

https://kibana-ci.elastic.co/job/elasticsearch+snapshots+verify/[This Jenkins job] tests the latest {es} snapshot with the full {kib} CI pipeline, and promotes if it there are no test failures.

The Jenkins job pipeline definition is https://github.com/elastic/kibana/blob/main/.ci/es-snapshots/Jenkinsfile_verify_es[in the {kib} repo].

1. Checkout {kib} and set up CI environment as normal.
2. Set the `ES_SNAPSHOT_MANIFEST` env var to point to the latest snapshot manifest.
3. Run CI (functional tests, integration tests, etc).
4. After CI
** If there was a test failure or other build error, send out an e-mail notification and stop.
** If there were no errors, promote the snapshot.

Promotion is done as part of the same pipeline:

1. Replace the manifest at `kibana-ci-es-snapshots-daily/<version>/manifest-latest-verified.json` with the manifest from the tested snapshot.
** At this point, the snapshot has been promoted and will automatically be used in CI and in local development.
2. Replace the snapshot at `kibana-ci-es-snapshots-permanent/<version>/` with the tested snapshot by copying all of the tarballs and the manifest file.