Go to file
John Schulz 81a340e681
[Fleet][EPM] Save installed package assets in ES (#83391)
## Summary
Store package assets (from Registry or local upload) in Elasticsearch. Related to proposal [issue](https://github.com/elastic/kibana/issues/83426) & [document](https://docs.google.com/document/d/18XoS6CSl9UxxPPBt9LXuJngf1Jv-4tl3jY6l19U1yH8)

 * New `epm-packages-assets` saved objects are stored on `.kibana` index, like our existing saved object `epm-packages`
 * Asset id is uuid v5 based on the package name, package version & file path. See 1974324
 * Add a list of IDs of all the installed assets, to `epm-packages` saved object. Like the existing `installed_` properties.  [Example](https://github.com/elastic/kibana/pull/83391/files#diff-fa07cac51b6a49bf1e4824bc2250c9a77dac6c7d6b0a56020f559ef1ff9be25fR491-R512) from a test

<details><summary>Mapping for new Saved Object</summary>

37f7b6ded7/x-pack%2Fplugins%2Ffleet%2Fserver%2Fsaved_objects%2Findex.ts (L329-L339)
</details>

<details><summary>Additional property on existing <code>epm-packages</code> Saved Object</summary>

c4f27ab257/x-pack/plugins/fleet/server/saved_objects/index.ts (L306-L312)

 I don't think the saved object changes are strictly required. It can be removed without changing much about how things work

- Pros: 
      - Preserves accurate record of the assets added at installation time. Separates what assets are currently available for package-version from what was installed. They _should_ be the same, but things happen.
      - Avoids a query to get the installed assets before operating on them
- Cons:
      - size/noise? Could be tens or hundreds of ids
      - migration?
</details>

### More details

**When are saved objects added?**
During installation, after all other actions have succeeded, just before marking the save object as installed, we commit all the files from the package to ES

37f7b6ded7/x-pack%2Fplugins%2Ffleet%2Fserver%2Fservices%2Fepm%2Fpackages%2F_install_package.ts (L193-L198)

**When are documents removed from the index?**

In the `removeInstallation` function which is called in response to a `DELETE /api/fleet/epm/packages/pkgkey`

37f7b6ded7/x-pack%2Fplugins%2Ffleet%2Fserver%2Fservices%2Fepm%2Fpackages%2Fremove.ts (L72)

or a failed package (re-)installation

bf068739ac/x-pack%2Fplugins%2Ffleet%2Fserver%2Fservices%2Fepm%2Fpackages%2Finstall.ts (L145)




**How are we using these assets?**
We're not, currently. Here's an example showing how we could update [`getFileHandler`](514b50e4c2/x-pack%2Fplugins%2Ffleet%2Fserver%2Froutes%2Fepm%2Fhandlers.ts (L101)) to check for local assets before reaching out to the Registry if we wished. It's not DRY, but it does work

```typescript
const esDocRoot = `http://elastic:changeme@localhost:9200/${PACKAGE_ASSETS_INDEX_NAME}/_doc`;
const escapedDocId = encodeURIComponent(`${pkgName}-${pkgVersion}/${filePath}`);
const esRes = await fetch(`${esDocRoot}/${escapedDocId}`);
const esJson = await esRes.json();
if (esJson.found) {
  const asset: PackageAsset = esJson._source;
  const body = asset.data_utf8 || Buffer.from(asset.data_base64, 'base64');
  return response.ok({
    body,
    headers: {
      'content-type': asset.media_type,
      // should add our own `cache-control` header here
      // kibana default is prevents caching: `private, no-cache, no-store, must-revalidate`
      // https://github.com/elastic/kibana/issues/83631
    },
  });
}
```

### Checklist
_updated tests to include new saved object output, no tests added yet_
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2020-12-07 15:11:09 -05:00
.ci chore(NA): remove scripts on plugins to find circular deps (#84852) 2020-12-07 19:41:38 +00:00
.github Attempt to more granularly separate App Search vs Workplace Search vs shared GitHub notifications (#84713) 2020-12-01 14:59:27 -08:00
.teamcity chore(NA): remove scripts on plugins to find circular deps (#84852) 2020-12-07 19:41:38 +00:00
common/graphql
config [deb/rpm] Remove /var prefix from tmpfiles.d (#82196) 2020-11-09 10:05:26 -06:00
data
docs [Docs] Adds docs on how to run Kibana with the APM agent locally (#84700) 2020-12-07 12:39:48 -07:00
examples [data.search.searchSource] Update SearchSource to use Fields API. (#82383) 2020-12-03 08:09:23 -07:00
licenses
packages Integrate painless autocomplete in runtime fields editor (#84943) 2020-12-07 12:55:53 -05:00
plugins [dev/cli] ensure plugins/ and all watch source dirs exist (#78973) 2020-09-30 10:20:44 -07:00
rfcs Use .kibana instead of .kibana_current to mark migration completion (#83373) 2020-11-25 15:20:56 +01:00
scripts Jest multi-project configuration (#77894) 2020-12-02 11:42:23 -08:00
src [cli/dev] make optimizer delays more obvious and hide proxy target url (#84835) 2020-12-07 11:34:59 -07:00
tasks oss tests: replace grunt task with node script (#82371) 2020-11-04 17:57:32 +01:00
test chore(NA): remove scripts on plugins to find circular deps (#84852) 2020-12-07 19:41:38 +00:00
typings Get rid of global types (#81739) 2020-10-28 11:03:04 +01:00
utilities
vars chore(NA): remove scripts on plugins to find circular deps (#84852) 2020-12-07 19:41:38 +00:00
x-pack [Fleet][EPM] Save installed package assets in ES (#83391) 2020-12-07 15:11:09 -05:00
.backportrc.json chore(NA): add missing branches into backportrc configuration file (#79848) 2020-10-07 19:51:44 +01:00
.browserslistrc [browserlist] Excludes browsers not supporting es6-class (#81431) 2020-10-28 16:38:59 -07:00
.editorconfig
.eslintignore chore(NA): move into single pkg json (#80015) 2020-11-02 21:18:52 +00:00
.eslintrc.js Update typescript eslint to v4.8 (#83520) 2020-11-18 18:23:08 +01:00
.fossa.yml
.gitattributes
.gitignore chore(NA): enable yarn prefer offline and local mirror for development (#84124) 2020-11-25 00:18:18 +00:00
.i18nrc.json Support for painless language autocomplete within monaco (#80577) 2020-11-30 10:14:33 -05:00
.node-version Upgrade Node.js to version 14 (#83425) 2020-12-02 23:40:06 +01:00
.nvmrc Upgrade Node.js to version 14 (#83425) 2020-12-02 23:40:06 +01:00
.prettierrc
.sass-lint.yml [Visualize] New visualization wizard (#79627) 2020-11-06 18:03:44 +02:00
.telemetryrc.json [Usage collection] Make schema mandatory (#79999) 2020-10-26 12:57:15 +02:00
.yarnrc chore(NA): enable yarn prefer offline and local mirror for development (#84124) 2020-11-25 00:18:18 +00:00
api-documenter.json
CONTRIBUTING.md
FAQ.md
github_checks_reporter.json
Gruntfile.js
Jenkinsfile chore(NA): remove usage of unverified es snapshots (#83589) 2020-11-18 00:18:31 +00:00
jest.config.integration.js Jest multi-project configuration (#77894) 2020-12-02 11:42:23 -08:00
jest.config.js Jest multi-project configuration (#77894) 2020-12-02 11:42:23 -08:00
jest.config.oss.js Jest multi-project configuration (#77894) 2020-12-02 11:42:23 -08:00
kibana.d.ts
LICENSE.txt
NOTICE.txt chore(NA): move into single pkg json (#80015) 2020-11-02 21:18:52 +00:00
package.json chore(NA): remove scripts on plugins to find circular deps (#84852) 2020-12-07 19:41:38 +00:00
preinstall_check.js
README.md Fix "Getting started" link in README (#84153) 2020-11-23 15:33:02 -05:00
renovate.json5 [renovate] update label config 2020-12-04 12:23:47 -07:00
STYLEGUIDE.md chore(NA): tool to find plugins circular dependencies between plugins (#82867) 2020-11-30 22:19:32 +00:00
tsconfig.base.json Cleanup tsconfig files (#84396) 2020-11-30 19:12:00 +01:00
tsconfig.browser.json
tsconfig.json Cleanup tsconfig files (#84396) 2020-11-30 19:12:00 +01:00
tsconfig.refs.json Cleanup tsconfig files (#84396) 2020-11-30 19:12:00 +01:00
tsconfig.types.json ui_actions service initial docs (#78902) 2020-09-30 16:44:29 +02:00
TYPESCRIPT.md
yarn.lock chore(NA): remove scripts on plugins to find circular deps (#84852) 2020-12-07 19:41:38 +00:00

Kibana

Kibana is your window into the Elastic Stack. Specifically, it's a browser-based analytics and search dashboard for Elasticsearch.

Getting Started

If you just want to try Kibana out, check out the Elastic Stack Getting Started Page to give it a whirl.

If you're interested in diving a bit deeper and getting a taste of Kibana's capabilities, head over to the Kibana Getting Started Page.

Using a Kibana Release

If you want to use a Kibana release in production, give it a test run, or just play around:

Building and Running Kibana, and/or Contributing Code

You might want to build Kibana locally to contribute some code, test out the latest features, or try out an open PR:

Documentation

Visit Elastic.co for the full Kibana documentation.

For information about building the documentation, see the README in elastic/docs.

Version Compatibility with Elasticsearch

Ideally, you should be running Elasticsearch and Kibana with matching version numbers. If your Elasticsearch has an older version number or a newer major number than Kibana, then Kibana will fail to run. If Elasticsearch has a newer minor or patch number than Kibana, then the Kibana Server will log a warning.

Note: The version numbers below are only examples, meant to illustrate the relationships between different types of version numbers.

Situation Example Kibana version Example ES version Outcome
Versions are the same. 5.1.2 5.1.2 💚 OK
ES patch number is newer. 5.1.2 5.1.5 ⚠️ Logged warning
ES minor number is newer. 5.1.2 5.5.0 ⚠️ Logged warning
ES major number is newer. 5.1.2 6.0.0 🚫 Fatal error
ES patch number is older. 5.1.2 5.1.0 ⚠️ Logged warning
ES minor number is older. 5.1.2 5.0.0 🚫 Fatal error
ES major number is older. 5.1.2 4.0.0 🚫 Fatal error

Questions? Problems? Suggestions?

  • If you've found a bug or want to request a feature, please create a GitHub Issue. Please check to make sure someone else hasn't already created an issue for the same topic.
  • Need help using Kibana? Ask away on our Kibana Discuss Forum and a fellow community member or Elastic engineer will be glad to help you out.