maxmustermann/kibana
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity
kibana/x-pack/plugins/fleet
History
John Schulz 81a340e681
[Fleet][EPM] Save installed package assets in ES (#83391) 
## Summary
Store package assets (from Registry or local upload) in Elasticsearch. Related to proposal [issue](https://github.com/elastic/kibana/issues/83426) & [document](https://docs.google.com/document/d/18XoS6CSl9UxxPPBt9LXuJngf1Jv-4tl3jY6l19U1yH8)

 * New `epm-packages-assets` saved objects are stored on `.kibana` index, like our existing saved object `epm-packages`
 * Asset id is uuid v5 based on the package name, package version & file path. See 1974324
 * Add a list of IDs of all the installed assets, to `epm-packages` saved object. Like the existing `installed_` properties.  [Example](https://github.com/elastic/kibana/pull/83391/files#diff-fa07cac51b6a49bf1e4824bc2250c9a77dac6c7d6b0a56020f559ef1ff9be25fR491-R512) from a test

<details><summary>Mapping for new Saved Object</summary>

37f7b6ded7/x-pack%2Fplugins%2Ffleet%2Fserver%2Fsaved_objects%2Findex.ts (L329-L339)
</details>

<details><summary>Additional property on existing <code>epm-packages</code> Saved Object</summary>

c4f27ab257/x-pack/plugins/fleet/server/saved_objects/index.ts (L306-L312)

 I don't think the saved object changes are strictly required. It can be removed without changing much about how things work

- Pros: 
      - Preserves accurate record of the assets added at installation time. Separates what assets are currently available for package-version from what was installed. They _should_ be the same, but things happen.
      - Avoids a query to get the installed assets before operating on them
- Cons:
      - size/noise? Could be tens or hundreds of ids
      - migration?
</details>

### More details

**When are saved objects added?**
During installation, after all other actions have succeeded, just before marking the save object as installed, we commit all the files from the package to ES

37f7b6ded7/x-pack%2Fplugins%2Ffleet%2Fserver%2Fservices%2Fepm%2Fpackages%2F_install_package.ts (L193-L198)

**When are documents removed from the index?**

In the `removeInstallation` function which is called in response to a `DELETE /api/fleet/epm/packages/pkgkey`

37f7b6ded7/x-pack%2Fplugins%2Ffleet%2Fserver%2Fservices%2Fepm%2Fpackages%2Fremove.ts (L72)

or a failed package (re-)installation

bf068739ac/x-pack%2Fplugins%2Ffleet%2Fserver%2Fservices%2Fepm%2Fpackages%2Finstall.ts (L145)




**How are we using these assets?**
We're not, currently. Here's an example showing how we could update [`getFileHandler`](514b50e4c2/x-pack%2Fplugins%2Ffleet%2Fserver%2Froutes%2Fepm%2Fhandlers.ts (L101)) to check for local assets before reaching out to the Registry if we wished. It's not DRY, but it does work

```typescript
const esDocRoot = `http://elastic:changeme@localhost:9200/${PACKAGE_ASSETS_INDEX_NAME}/_doc`;
const escapedDocId = encodeURIComponent(`${pkgName}-${pkgVersion}/${filePath}`);
const esRes = await fetch(`${esDocRoot}/${escapedDocId}`);
const esJson = await esRes.json();
if (esJson.found) {
  const asset: PackageAsset = esJson._source;
  const body = asset.data_utf8 || Buffer.from(asset.data_base64, 'base64');
  return response.ok({
    body,
    headers: {
      'content-type': asset.media_type,
      // should add our own `cache-control` header here
      // kibana default is prevents caching: `private, no-cache, no-store, must-revalidate`
      // https://github.com/elastic/kibana/issues/83631
    },
  });
}
```

### Checklist
_updated tests to include new saved object output, no tests added yet_
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2020-12-07 15:11:09 -05:00
..
common [Fleet][EPM] Save installed package assets in ES (#83391) 2020-12-07 15:11:09 -05:00
dev_docs
public [Fleet][EPM] Add TS type for package-spec. Clarify EPR relationship with spec (#84946) 2020-12-07 09:30:27 -05:00
scripts [DX] Bump TS version to v4.1 (#83397) 2020-11-24 16:04:33 +01:00
server [Fleet][EPM] Save installed package assets in ES (#83391) 2020-12-07 15:11:09 -05:00
CHANGELOG.md
jest.config.js Jest multi-project configuration (#77894) 2020-12-02 11:42:23 -08:00
kibana.json [Fleet] Agent logs UI (#83356) 2020-11-20 10:49:02 -08:00
package.json [Fleet] Rename ingestManager plugin ID fleet (#83200) 2020-11-19 08:43:14 -05:00
README.md [Fleet] Rename ingestManager plugin ID fleet (#83200) 2020-11-19 08:43:14 -05:00

README.md

Fleet

Plugin

  • The plugin is enabled by default. See the TypeScript type for the the available plugin configuration options
  • Adding xpack.fleet.enabled=false will disable the plugin including the EPM and Fleet features. It will also remove the PACKAGE_POLICY_API_ROUTES and AGENT_POLICY_API_ROUTES values in common/constants/routes.ts
  • Adding --xpack.fleet.agents.enabled=false will disable the Fleet API & UI
  • Both EPM and Fleet require ingestManager be enabled. They are not standalone features.
  • For Enterprise license, a custom package registry URL can be used by setting xpack.fleet.registryUrl=http://localhost:8080
    • This property is currently only for internal Elastic development and is unsupported

Fleet Requirements

Fleet needs to have Elasticsearch API keys enabled, and also to have TLS enabled on kibana, (if you want to run Kibana without TLS you can provide the following config flag --xpack.fleet.agents.tlsCheckDisabled=false)

Also you need to configure the hosts your agent is going to use to comunication with Elasticsearch and Kibana (Not needed if you use Elastic cloud). You can use the following flags:

--xpack.fleet.agents.elasticsearch.host=http://localhost:9200
--xpack.fleet.agents.kibana.host=http://localhost:5601

Development

Getting started

See the Kibana docs for how to set up your dev environment, run Elasticsearch, and start Kibana

One common development workflow is:

  • Bootstrap Kibana 
    yarn kbn bootstrap
  • Start Elasticsearch in one shell 
    yarn es snapshot -E xpack.security.authc.api_key.enabled=true
  • Start Kibana in another shell 
    yarn start --xpack.fleet.enabled=true --no-base-path

This plugin follows the common, server, public structure from the Architecture Style Guide . We also follow the pattern of developing feature branches under your personal fork of Kibana.

Note: The plugin was previously named Ingest Manager it's possible that some variables are still named with that old plugin name.

Tests

API integration tests

You need to have docker to run ingest manager api integration tests

  1. In one terminal, run the tests from the Kibana root directory with

    FLEET_PACKAGE_REGISTRY_PORT=12345 yarn test:ftr:server --config x-pack/test/fleet_api_integration/config.ts

  2. in a second terminal, run the tests from the Kibana root directory with

    FLEET_PACKAGE_REGISTRY_PORT=12345 yarn test:ftr:runner --config x-pack/test/fleet_api_integration/config.ts

    Optionally you can filter which tests you want to run using --grep

    FLEET_PACKAGE_REGISTRY_PORT=12345 yarn test:ftr:runner --config x-pack/test/fleet_api_integration/config.ts --grep='fleet'