John Schulz
ec4fbf2901
[Fleet][EPM] Revert es-storage-related commits ( #85942 )
...
* Revert "[Fleet][EPM] Save installed package assets in ES (#83391 )"
This reverts commit 81a340e68100c2e96
2020-12-15 15:11:46 -05:00
Silvia Mitter
98308f8680
[Fleet] Change permissions for Fleet enroll role ( #85802 )
...
* Add APM traces index names to Fleet enroll role
* Removes fleet permissions for `events-*` as they became obsolete
fixes #85761
2020-12-15 09:31:39 +01:00
nnamdifrankie
23c5daa622
[Fleet] add ilm policy per data stream ( #85492 )
...
Co-authored-by: kevinlog <kevin.logan@elastic.co>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2020-12-14 16:15:55 -05:00
Kevin Logan
06993c469b
[Fleet] Installation of hidden field ( #85703 )
...
Co-authored-by: nnamdifrankie <franklin.ejoh@elastic.co>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2020-12-14 13:45:56 -05:00
Sonja Krause-Harder
1b3a1bb385
Don't rollback on saved objects conflict errors. ( #85131 )
2020-12-14 17:36:25 +01:00
Larry Gregory
96bb72f68d
Fix fleet route protections ( #85626 )
...
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2020-12-13 12:40:50 -05:00
Candace Park
4e4e55029c
[Security Solution][Endpoint][Admin] Api guard for paid policy features ( #84667 )
2020-12-10 18:53:55 -05:00
John Schulz
1c36ab9236
[Fleet][EPM] Move SO work from getFileHandler to service method ( #85594 )
...
## Summary
- Move registry URL details from handler to Registry service
- rename `archive/save_to_es` to `archive/storage` to better reflect its role
- Move SO handling from http into new `getAsset` method in `archive/storage`
- Rearrange code in HTTP handler to be more easily moved to one or two service method(s) which deal with the logic for getting the asset.
### Checklist
- [ ] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility ) were updated or added to match the most common scenarios
2020-12-10 13:28:41 -05:00
John Schulz
00c2e960b6
[Fleet][EPM] - Do cache || saved_object || network in http handler ( #85331 )
...
## Summary
1. cf0d567e33d7d4™️ quality version of "use cached or saved object or registry" file in HTTP handler. It's doing too much and should eventually move the logic to service method(s).
2020-12-10 11:23:03 -05:00
Nicolas Chaulet
c9b5ec7303
[Fleet] Update agent listing for better status reporting ( #84798 )
2020-12-09 21:07:31 -05:00
Aleh Zasypkin
88e61a6651
Migrate API keys functionality to a new Elasticsearch client. ( #85029 )
2020-12-09 20:43:24 +01:00
Nicolas Chaulet
943bce1512
[Fleet] Enforce superuser role for all fleet APIs ( #85136 )
2020-12-08 17:21:45 -05:00
John Schulz
81a340e681
[Fleet][EPM] Save installed package assets in ES ( #83391 )
...
## Summary
Store package assets (from Registry or local upload) in Elasticsearch. Related to proposal [issue](https://github.com/elastic/kibana/issues/83426 ) & [document](https://docs.google.com/document/d/18XoS6CSl9UxxPPBt9LXuJngf1Jv-4tl3jY6l19U1yH8 )
* New `epm-packages-assets` saved objects are stored on `.kibana` index, like our existing saved object `epm-packages`
* Asset id is uuid v5 based on the package name, package version & file path. See 1974324https://github.com/elastic/kibana/pull/83391/files#diff-fa07cac51b6a49bf1e4824bc2250c9a77dac6c7d6b0a56020f559ef1ff9be25fR491-R512 ) from a test
<details><summary>Mapping for new Saved Object</summary>
37f7b6ded7/x-pack%2Fplugins%2Ffleet%2Fserver%2Fsaved_objects%2Findex.ts (L329-L339)c4f27ab257/x-pack/plugins/fleet/server/saved_objects/index.ts (L306-L312)37f7b6ded7/x-pack%2Fplugins%2Ffleet%2Fserver%2Fservices%2Fepm%2Fpackages%2F_install_package.ts (L193-L198)37f7b6ded7/x-pack%2Fplugins%2Ffleet%2Fserver%2Fservices%2Fepm%2Fpackages%2Fremove.ts (L72)bf068739ac/x-pack%2Fplugins%2Ffleet%2Fserver%2Fservices%2Fepm%2Fpackages%2Finstall.ts (L145)514b50e4c2/x-pack%2Fplugins%2Ffleet%2Fserver%2Froutes%2Fepm%2Fhandlers.ts (L101)http://elastic:changeme@localhost:9200/${PACKAGE_ASSETS_INDEX_NAME}/_doc `;
const escapedDocId = encodeURIComponent(`${pkgName}-${pkgVersion}/${filePath}`);
const esRes = await fetch(`${esDocRoot}/${escapedDocId}`);
const esJson = await esRes.json();
if (esJson.found) {
const asset: PackageAsset = esJson._source;
const body = asset.data_utf8 || Buffer.from(asset.data_base64, 'base64');
return response.ok({
body,
headers: {
'content-type': asset.media_type,
// should add our own `cache-control` header here
// kibana default is prevents caching: `private, no-cache, no-store, must-revalidate`
// https://github.com/elastic/kibana/issues/83631
},
});
}
```
### Checklist
_updated tests to include new saved object output, no tests added yet_
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html ) were updated or added to match the most common scenarios
2020-12-07 15:11:09 -05:00
John Schulz
ef7367ddb3
[Fleet][EPM] Add TS type for package-spec. Clarify EPR relationship with spec ( #84946 )
...
## Summary
* Create a [TS type for `package-spec`](09cce235de/x-pack/plugins/fleet/common/types/models/package_spec.ts5f2c4a5547/x-pack/plugins/fleet/common/types/models/epm.ts (L71-L80)3b4d820755/versions/1/manifest.spec.yml (L241-L248)https://user-images.githubusercontent.com/57655/101083986-4b915f80-357b-11eb-84c6-14605e1ea3d8.png ">
using `versions`; not `version`
<img width="1395" alt="Screen Shot 2020-12-02 at 12 08 13 PM" src="https://user-images.githubusercontent.com/57655/101083988-4c29f600-357b-11eb-9ff2-2b73e313c130.png ">
including `elasticsearch` when spec says it' not valid yet
<img width="1589" alt="Screen Shot 2020-12-02 at 12 08 36 PM" src="https://user-images.githubusercontent.com/57655/101083990-4c29f600-357b-11eb-8250-8926f7189af8.png ">
</details>
<details><summary>screenshots showing editor autocomplete for registry response values like <code>categories</code></summary>
<img width="422" alt="Screen Shot 2020-12-02 at 12 25 11 PM" src="https://user-images.githubusercontent.com/57655/101083994-4cc28c80-357b-11eb-9013-ae208f7c311a.png ">
<img width="345" alt="Screen Shot 2020-12-02 at 12 25 01 PM" src="https://user-images.githubusercontent.com/57655/101083991-4c29f600-357b-11eb-9468-bbb9d27513b1.png ">
</details>
<details><summary>screenshot showing type check preventing adding properties which aren't explicitly listed as "additions"</summary>
<img width="1295" alt="Screen Shot 2020-12-03 at 11 38 43 AM" src="https://user-images.githubusercontent.com/57655/101083997-4cc28c80-357b-11eb-83b9-206b85521f11.png ">
</details>
### Checklist
- [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html ) were updated or added to match the most common scenarios
2020-12-07 09:30:27 -05:00
Sandra Gonzales
db70286bca
[Fleet] add readme to uploaded package info and fix images ( #84944 )
...
* use the relative src instead of path for package images
* add readme to package info for uploaded packages
* let toPackageImage take the image object and check path if no src
* use variable
2020-12-04 10:58:04 -05:00
Sandra Gonzales
a5dd5b6998
[Fleet] EPM support to handle uploaded file paths ( #84708 )
...
* modify file route to handle uploaded packge file paths
* update messaging
* improve tests
* fix bug and add test to check the version of the uploaded package before failing
* fix similar bug for getting package info from registry when a different version is uploaded
2020-12-02 18:33:27 -05:00
John Schulz
90a18cc15d
[Fleet][EPM] Pass through valid manifest values from upload ( #84703 )
...
* Add missing properties & improve type safety
* Break up types for better readability
2020-12-02 09:49:21 -05:00
Nicolas Chaulet
9b74fe6d39
[Fleet] Handler api key creation errors when Fleet Admin is invalid ( #84576 )
2020-12-01 12:55:15 -05:00
Sonja Krause-Harder
c9fc876da1
Return early when parallel install process detected ( #84190 )
2020-12-01 15:00:15 +01:00
Jen Huang
284b1046df
[Fleet] Support input-level vars & templates ( #83878 )
...
* Fix bug creating new policy on the fly
* Adjust UI for input with vars but no streams
* Revert "Fix bug creating new policy on the fly"
This reverts commit 34f7014d69
2020-11-25 10:13:36 -08:00
Mikhail Shustov
5ec6fe315f
[DX] Bump TS version to v4.1 ( #83397 )
...
* bump version to 4.1.1-rc
* fix code to run kbn bootstrap
* fix errors
* DO NOT MERGE. mute errors and ping teams to fix them
* Address EuiSelectableProps configuration in discover sidebar
* use explicit type for EuiSelectable
* update to ts v4.1.2
* fix ts error in EuiSelectable
* update docs
* update prettier with ts version support
* Revert "update prettier with ts version support"
This reverts commit 3de48db3ec
2020-11-24 16:04:33 +01:00
Sonja Krause-Harder
aa07f5c1d0
[Fleet] Rename usage collection object to 'fleet'. ( #83407 )
...
* Rename usage collection object to 'fleet'.
* Update telemetry mapping.
* Adjust naming.
* Rename ingestManager -> fleet in telemetry collector
2020-11-24 10:11:58 +01:00
Sandra Gonzales
4e9afeebd5
[Fleet] index patterns to handle uploaded packages ( #83994 )
...
* fix index patterns, remove install_source from cache key, add missing test file
* remove unused function
* fix types
* fix comment
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2020-11-23 08:29:13 -05:00
Mikhail Shustov
95861a0fb0
[DX] Prettier v2.2 ( #83899 )
...
* update prettier with ts version support
* mute type-error
* run prettier on codebase
* fix examples
* fix errors after master merged
2020-11-23 13:17:05 +01:00
Dan Panzarella
2cd2528ac8
[Security Solution] Give notice when endpoint policy is out of date ( #83469 )
2020-11-20 15:21:23 -05:00
Sandra Gonzales
39291e16b5
[Fleet] update getPackageInfo to handle uploaded packages ( #83854 )
...
* update getPackgeInfo handler to fetch from install source
* add tests and modify fixtures to distinguish between registry and uploaded package
* improve error handling
* fix type
* fix test
* remove try/catch
* fix zip file test to have the right number of assets
* fix compressed files
2020-11-20 11:40:45 -05:00
Nicolas Chaulet
c1a263cff4
[Fleet] Allow to send SETTINGS action ( #83707 )
2020-11-20 09:21:24 -05:00
Jonathan Budzenski
6c23302b36
[cli] Add bin/kibana-encryption-keys ( #82838 )
...
Co-authored-by: Aleh Zasypkin <aleh.zasypkin@gmail.com>
Co-authored-by: Tyler Smalley <tylersmalley@me.com>
2020-11-19 12:41:48 -06:00
Nicolas Chaulet
1b6cfe819d
[Fleet] Rename ingestManager plugin ID fleet ( #83200 )
2020-11-19 08:43:14 -05:00
John Schulz
514b50e4c2
Forward any registry cache-control header for files ( #83680 )
...
closes #83631
### Problem
Assets are served with a `cache-control` header that prevents any caching
<img src="https://user-images.githubusercontent.com/640/99534379-517d2300-2975-11eb-8c05-4fb3f127c52b.png "/>
### Root cause
Likely from this code 2a365ff632/src/core/server/http/http_tools.ts (L40-L43)b3eefb97da/src/core/server/http/integration_tests/router.test.ts (L510-L520)b3eefb97da/src/core/server/http/integration_tests/router.test.ts (L522-L536)https://user-images.githubusercontent.com/57655/99568352-4fc75580-299d-11eb-962f-6ff28fa9510d.png ">
which matches https://epr-snapshot.elastic.co/package/apache/0.2.6/img/logo_apache.svg
or using `epr.elastic.co`,
<img width="781" alt="Screen Shot 2020-11-18 at 12 31 56 PM" src="https://user-images.githubusercontent.com/57655/99568350-4fc75580-299d-11eb-966e-f3489c13edb5.png ">
which matches https://epr.elastic.co/package/apache/0.2.6/img/logo_apache.svg
2020-11-19 07:19:41 -05:00
John Schulz
cf7aacc1c0
[Fleet][EPM] Unified install and archive ( #83384 )
...
## Summary
* Further reduce differences between installing uploaded vs registry package
* Improve cache/store names, TS types, etc. Including key by name + version + source
* Add a cache/store for PackageInfo (e.g. results metadata from registry's /package/version/ response)
* Remove ensureCachedArchiveInfo
2020-11-17 10:42:00 -05:00
John Schulz
380fa5b3f6
[bundle optimization] Update to semver 7.x to get tree-shaking ( #83020 )
...
## What's changed in this PR
### Update to latest available `semver`: `7.3.2`
* `semver` 5.x pulls in the entire library in one large file (~38k uncompressed / ~9k gz), when we might only use 1-2K.
* `semver` 7.0+ supports tree-shaking: https://github.com/npm/node-semver/blob/master/CHANGELOG.md#700
### Update paths to only import individual function(s) used instead of the entire library
* Getting the smaller bundle requires a different import style [as shown in the docs](https://github.com/npm/node-semver#usage )
* Only changed code in `public` & `common` folders; not `server`. We could also update `server` as well for consistency, but I skipped because the new import style is more verbose and the filesize didn't seem as important on the server
### Results
The build stats show a 10K+ improvement for initial page bundles https://github.com/elastic/kibana/pull/83020#issuecomment-724724432
| id | [before](c6afc47f32213bb52a8chttps://github.com/elastic/kibana/pull/83020#issuecomment-724859130
### Possible issues
Moving 2 major versions. We're currently on 5.7 and the latest available is 7.3.
* changelog says 5.x (our current) to 6.0 should be safe: https://github.com/npm/node-semver/blob/master/CHANGELOG.md#60
* There 6.x & 7.x changes all appear to be new features or bugfixes around the `includePrerelease` flag added in 5.6, but I'm not sure if those "fixes" will break existing code
* https://github.com/npm/node-semver/blob/master/CHANGELOG.md#613
* https://github.com/npm/node-semver/blob/master/CHANGELOG.md#722
### Stats / screenshots
generated with `node scripts/build_kibana_platform_plugins.js --profile --focus=ingestManager`
<details><summary><b>Ingest Manager in `master`</b>: imports entire `semver` lib, totals 40k+, only 1 large file (orange arc below)</summary>
<img width="972" alt="Screen Shot 2020-11-09 at 6 50 23 PM" src="https://user-images.githubusercontent.com/57655/98666188-a50ac380-231a-11eb-9b8a-6ca784752714.png ">
</details>
<details><summary><b>Ingest Manager in PR after upgrade to 7</b>: still imports entire lib. file size *increased* to ~60k, but now individual files are imported (orange arcs below)</summary>
<img width="825" alt="Screen Shot 2020-11-09 at 5 46 30 PM" src="https://user-images.githubusercontent.com/57655/98666355-e602d800-231a-11eb-803f-bc04beb4eaf1.png ">
<img width="963" alt="Screen Shot 2020-11-09 at 5 47 06 PM" src="https://user-images.githubusercontent.com/57655/98666357-e69b6e80-231a-11eb-92d3-c66904f92c30.png ">
</details>
<details><summary><b>Ingest Manager in PR after changing `import`s:</b> total imported size down to ~20k. Can see individual imported files</summary>
<img width="926" alt="Screen Shot 2020-11-10 at 6 10 23 AM" src="https://user-images.githubusercontent.com/57655/98667058-e64fa300-231b-11eb-9690-5e36ed6475e0.png ">
<img width="895" alt="Screen Shot 2020-11-10 at 6 10 53 AM" src="https://user-images.githubusercontent.com/57655/98667059-e780d000-231b-11eb-8abf-98d8bdbcf061.png ">
</details>
### Checklist
- [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html ) were updated or added to match the most common scenarios
2020-11-14 07:32:02 -05:00
Thomas Watson
4c49d5d1be
Upgrade Node.js to version 12 ( #61587 )
2020-11-12 22:00:57 +01:00
John Schulz
208e86e66a
[Ingest Manager] Lift up registry/{stream,extract} functions ( #83239 )
...
## Summary
* Move stream utility functions from `server/services/epm/registry/streams.ts` to `server/services/epm/streams.ts`
* They're only used in registry at the moment but aren't specific to registry
* Move archive extraction functions from `server/services/epm/registry/extract.ts` to `server/services/epm/archive.ts`
* The Registry isn't the only service/code which needs to extract packages. Continue consolidating archive-related code under archive vs registry
2020-11-12 11:05:17 -05:00
John Schulz
4721b3211a
[Ingest Manger] Move asset getters out of registry ( #83214 )
...
## Summary
Packages/Archives aren't limited to the registry any longer. Continue moving file- & cache-related functions from services/registry to services/archive.
Move `getAsset` and `pathParts` to archive/index. The behavior is the same for now, but it's more accurate to separate these from registry namespace.
Registry has `fetch*` and other functions for dealing with the online service.
2020-11-12 05:47:43 -05:00
John Schulz
2d3261fc44
Don't make the caller do work the function can do ( #83180 )
2020-11-11 15:20:04 -05:00
John Schulz
d66ca491ed
[Fleet] Split unpacking an archive and caching its files into separate functions ( #83085 )
...
## Summary
* Separate unpacking an archive from caching it or other side-effects
* Parse and validate an archive before caching it
* Validation has no coupling with caching side-effects or any other code outside the `validation.ts` file
```diff
- const paths = await unpackArchiveToCache(archiveBuffer, contentType);
- const archivePackageInfo = parseAndVerifyArchive(paths);
+ const entries = await unpackArchiveEntries(archiveBuffer, contentType);
+ const { archivePackageInfo } = await parseAndVerifyArchiveEntries(entries);
+ const paths = addEntriesToMemoryStore(entries);
```
### Checklist
- [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html ) were updated or added to match the most common scenarios
2020-11-10 18:27:34 -05:00
Nicolas Chaulet
441a0d4ec9
[Fleet] Move ingestManager plugin to fleet ( #82886 )
2020-11-09 11:07:04 -05:00
