Commit graph

68 commits

Author SHA1 Message Date
Spencer e3856a9e45
[security/apiKeys] migrate from id to ids in ES request (#87053)
Co-authored-by: spalger <spalger@users.noreply.github.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-01-04 15:46:37 -07:00
Kevin Logan 5e5a410204
[Fleet] add leading dot to hidden indices (#86277)
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: John Schulz <github.com@jfsiii.org>
2021-01-04 10:50:54 -05:00
Tiago Costa 4c3c2ac156
skip failing es promotion suite (#87166) 2021-01-04 15:23:18 +00:00
Matthew Kime c129f93083
[index patterns] Fleep app - Keep saved object field list until field caps provides fields (#85370) 2020-12-19 21:56:06 -06:00
Sandra Gonzales 7ae4f2d572
remove test that checks for data stream backing index names (#86052) 2020-12-15 20:34:55 -05:00
John Schulz ec4fbf2901
[Fleet][EPM] Revert es-storage-related commits (#85942)
* Revert "[Fleet][EPM] Save installed package assets in ES (#83391)"

This reverts commit 81a340e681.

* Revert 00c2e96

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2020-12-15 15:11:46 -05:00
Silvia Mitter 98308f8680
[Fleet] Change permissions for Fleet enroll role (#85802)
* Add APM traces index names to Fleet enroll role

* Removes fleet permissions for `events-*` as they became obsolete

fixes #85761
2020-12-15 09:31:39 +01:00
nnamdifrankie 23c5daa622
[Fleet] add ilm policy per data stream (#85492)
Co-authored-by: kevinlog <kevin.logan@elastic.co>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2020-12-14 16:15:55 -05:00
Nicolas Chaulet 943bce1512
[Fleet] Enforce superuser role for all fleet APIs (#85136) 2020-12-08 17:21:45 -05:00
John Schulz 81a340e681
[Fleet][EPM] Save installed package assets in ES (#83391)
## Summary
Store package assets (from Registry or local upload) in Elasticsearch. Related to proposal [issue](https://github.com/elastic/kibana/issues/83426) & [document](https://docs.google.com/document/d/18XoS6CSl9UxxPPBt9LXuJngf1Jv-4tl3jY6l19U1yH8)

 * New `epm-packages-assets` saved objects are stored on `.kibana` index, like our existing saved object `epm-packages`
 * Asset id is uuid v5 based on the package name, package version & file path. See 1974324
 * Add a list of IDs of all the installed assets, to `epm-packages` saved object. Like the existing `installed_` properties.  [Example](https://github.com/elastic/kibana/pull/83391/files#diff-fa07cac51b6a49bf1e4824bc2250c9a77dac6c7d6b0a56020f559ef1ff9be25fR491-R512) from a test

<details><summary>Mapping for new Saved Object</summary>

37f7b6ded7/x-pack%2Fplugins%2Ffleet%2Fserver%2Fsaved_objects%2Findex.ts (L329-L339)
</details>

<details><summary>Additional property on existing <code>epm-packages</code> Saved Object</summary>

c4f27ab257/x-pack/plugins/fleet/server/saved_objects/index.ts (L306-L312)

 I don't think the saved object changes are strictly required. It can be removed without changing much about how things work

- Pros: 
      - Preserves accurate record of the assets added at installation time. Separates what assets are currently available for package-version from what was installed. They _should_ be the same, but things happen.
      - Avoids a query to get the installed assets before operating on them
- Cons:
      - size/noise? Could be tens or hundreds of ids
      - migration?
</details>

### More details

**When are saved objects added?**
During installation, after all other actions have succeeded, just before marking the save object as installed, we commit all the files from the package to ES

37f7b6ded7/x-pack%2Fplugins%2Ffleet%2Fserver%2Fservices%2Fepm%2Fpackages%2F_install_package.ts (L193-L198)

**When are documents removed from the index?**

In the `removeInstallation` function which is called in response to a `DELETE /api/fleet/epm/packages/pkgkey`

37f7b6ded7/x-pack%2Fplugins%2Ffleet%2Fserver%2Fservices%2Fepm%2Fpackages%2Fremove.ts (L72)

or a failed package (re-)installation

bf068739ac/x-pack%2Fplugins%2Ffleet%2Fserver%2Fservices%2Fepm%2Fpackages%2Finstall.ts (L145)




**How are we using these assets?**
We're not, currently. Here's an example showing how we could update [`getFileHandler`](514b50e4c2/x-pack%2Fplugins%2Ffleet%2Fserver%2Froutes%2Fepm%2Fhandlers.ts (L101)) to check for local assets before reaching out to the Registry if we wished. It's not DRY, but it does work

```typescript
const esDocRoot = `http://elastic:changeme@localhost:9200/${PACKAGE_ASSETS_INDEX_NAME}/_doc`;
const escapedDocId = encodeURIComponent(`${pkgName}-${pkgVersion}/${filePath}`);
const esRes = await fetch(`${esDocRoot}/${escapedDocId}`);
const esJson = await esRes.json();
if (esJson.found) {
  const asset: PackageAsset = esJson._source;
  const body = asset.data_utf8 || Buffer.from(asset.data_base64, 'base64');
  return response.ok({
    body,
    headers: {
      'content-type': asset.media_type,
      // should add our own `cache-control` header here
      // kibana default is prevents caching: `private, no-cache, no-store, must-revalidate`
      // https://github.com/elastic/kibana/issues/83631
    },
  });
}
```

### Checklist
_updated tests to include new saved object output, no tests added yet_
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2020-12-07 15:11:09 -05:00
John Schulz ef7367ddb3
[Fleet][EPM] Add TS type for package-spec. Clarify EPR relationship with spec (#84946)
## Summary

 * Create a [TS type for `package-spec`](09cce235de/x-pack/plugins/fleet/common/types/models/package_spec.ts) to differentiate Registry shape from spec shape
 * Clarify EPR types relationship with package spec. i.e. what required properties it makes optional, what properties it adds, which property types it changes, etc.
5f2c4a5547/x-pack/plugins/fleet/common/types/models/epm.ts (L71-L80)
  * Updated `manifest.yml` in `apache_invalid_manifest_missing_field_0.1.4.zip`. Previously it was missing the `type` property, but that's optional [according to package-spec](3b4d820755/versions/1/manifest.spec.yml (L241-L248)). Removed the required `format_version` and re-zipped.

<details><summary>screenshots show errors if catches in a mock PackageInfo object</summary>
using `requirement`; not `conditions`
<img width="1084" alt="Screen Shot 2020-12-02 at 12 07 15 PM" src="https://user-images.githubusercontent.com/57655/101083986-4b915f80-357b-11eb-84c6-14605e1ea3d8.png">
using `versions`; not `version`
<img width="1395" alt="Screen Shot 2020-12-02 at 12 08 13 PM" src="https://user-images.githubusercontent.com/57655/101083988-4c29f600-357b-11eb-9ff2-2b73e313c130.png">
including `elasticsearch` when spec says it' not valid yet
<img width="1589" alt="Screen Shot 2020-12-02 at 12 08 36 PM" src="https://user-images.githubusercontent.com/57655/101083990-4c29f600-357b-11eb-8250-8926f7189af8.png">
</details>

<details><summary>screenshots showing editor autocomplete for registry response values like <code>categories</code></summary>

<img width="422" alt="Screen Shot 2020-12-02 at 12 25 11 PM" src="https://user-images.githubusercontent.com/57655/101083994-4cc28c80-357b-11eb-9013-ae208f7c311a.png">

<img width="345" alt="Screen Shot 2020-12-02 at 12 25 01 PM" src="https://user-images.githubusercontent.com/57655/101083991-4c29f600-357b-11eb-9468-bbb9d27513b1.png">
</details>

<details><summary>screenshot showing type check preventing adding properties which aren't explicitly listed as "additions"</summary>

<img width="1295" alt="Screen Shot 2020-12-03 at 11 38 43 AM" src="https://user-images.githubusercontent.com/57655/101083997-4cc28c80-357b-11eb-83b9-206b85521f11.png">
</details>


### Checklist


- [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2020-12-07 09:30:27 -05:00
Sandra Gonzales a5dd5b6998
[Fleet] EPM support to handle uploaded file paths (#84708)
* modify file route to handle uploaded packge file paths

* update messaging

* improve tests

* fix bug and add test to check the version of the uploaded package before failing

* fix similar bug for getting package info from registry when a different version is uploaded
2020-12-02 18:33:27 -05:00
Nicolas Chaulet 9b74fe6d39
[Fleet] Handler api key creation errors when Fleet Admin is invalid (#84576) 2020-12-01 12:55:15 -05:00
Sandra Gonzales 4e9afeebd5
[Fleet] index patterns to handle uploaded packages (#83994)
* fix index patterns, remove install_source from cache key, add missing test file

* remove unused function

* fix types

* fix comment

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2020-11-23 08:29:13 -05:00
Sandra Gonzales 39291e16b5
[Fleet] update getPackageInfo to handle uploaded packages (#83854)
* update getPackgeInfo handler to fetch from install source

* add tests and modify fixtures  to distinguish between registry and uploaded package

* improve error handling

* fix type

* fix test

* remove try/catch

* fix zip file test to have the right number of assets

* fix compressed files
2020-11-20 11:40:45 -05:00
Nicolas Chaulet c1a263cff4
[Fleet] Allow to send SETTINGS action (#83707) 2020-11-20 09:21:24 -05:00
Nicolas Chaulet 1b6cfe819d
[Fleet] Rename ingestManager plugin ID fleet (#83200) 2020-11-19 08:43:14 -05:00
Nicolas Chaulet fed9a4fddc
[Fleet] Rename ingest_manager_api_integration tests fleet_api_integration (#83011) 2020-11-12 13:50:59 -05:00