## Summary
* Add `installPackage` with `installSource` param, to provide a single interface the `install*` functions.
```diff
- const res = await installPackageFromRegistry({
+ const res = await installPackage({
+ installSource: 'registry',
```
and
```diff
- const res = await installPackageByUpload({
+ const res = await installPackage({
+ installSource: 'upload',
```
* Push some repeated work (`install`, `removable`) from `install*` into `_installPackage`. Which also simplifies its interface.
### installPackage
For now `installPackage` checks the `installSource` and calls the same `install*` functions to prevent any change in behavior but there's still a lot of overlap between `installPackageFromRegistry` & `installPackageByUpload`. I think we can bring them together into `installPackage` using the same branching on `installSource`.
### local checks with curl
<details><summary>curl request/responses for happy path:</summary>
```
## zip:
curl -X POST -u elastic:changeme http://localhost:5601/api/fleet/epm/packages --data-binary @$KIBANA_HOME/x-pack/test/ingest_manager_api_integration/apis/fixtures/direct_upload_packages/apache_0.1.4.zip -H 'kbn-xsrf: xyz' -H 'Content-Type: application/zip'
{"response":[{"id":"apache-Logs-Apache-Dashboard-ecs","type":"dashboard"},{"id":"apache-Metrics-Apache-HTTPD-server-status-ecs","type":"dashboard"},{"id":"Apache-HTTPD-CPU-ecs","type":"visualization"},{"id":"Apache-HTTPD-Hostname-list-ecs","type":"visualization"},{"id":"Apache-HTTPD-Load1-slash-5-slash-15-ecs","type":"visualization"},{"id":"Apache-HTTPD-Scoreboard-ecs","type":"visualization"},{"id":"Apache-HTTPD-Total-accesses-and-kbytes-ecs","type":"visualization"},{"id":"Apache-HTTPD-Uptime-ecs","type":"visualization"},{"id":"Apache-HTTPD-Workers-ecs","type":"visualization"},{"id":"Apache-access-unique-IPs-map-ecs","type":"visualization"},{"id":"Apache-browsers-ecs","type":"visualization"},{"id":"Apache-error-logs-over-time-ecs","type":"visualization"},{"id":"Apache-operating-systems-ecs","type":"visualization"},{"id":"Apache-response-codes-of-top-URLs-ecs","type":"visualization"},{"id":"Apache-response-codes-over-time-ecs","type":"visualization"},{"id":"Apache-HTTPD-ecs","type":"search"},{"id":"Apache-access-logs-ecs","type":"search"},{"id":"Apache-errors-log-ecs","type":"search"}]}
## Uploaded packages can be deleted as expected:
curl -X DELETE -u elastic:changeme http://localhost:5601/api/fleet/epm/packages/apache-0.1.4 -H 'kbn-xsrf: xxx'
{"response":[{"id":"apache-Logs-Apache-Dashboard-ecs","type":"dashboard"},{"id":"apache-Metrics-Apache-HTTPD-server-status-ecs","type":"dashboard"},{"id":"Apache-HTTPD-CPU-ecs","type":"visualization"},{"id":"Apache-HTTPD-Hostname-list-ecs","type":"visualization"},{"id":"Apache-HTTPD-Load1-slash-5-slash-15-ecs","type":"visualization"},{"id":"Apache-HTTPD-Scoreboard-ecs","type":"visualization"},{"id":"Apache-HTTPD-Total-accesses-and-kbytes-ecs","type":"visualization"},{"id":"Apache-HTTPD-Uptime-ecs","type":"visualization"},{"id":"Apache-HTTPD-Workers-ecs","type":"visualization"},{"id":"Apache-access-unique-IPs-map-ecs","type":"visualization"},{"id":"Apache-browsers-ecs","type":"visualization"},{"id":"Apache-error-logs-over-time-ecs","type":"visualization"},{"id":"Apache-operating-systems-ecs","type":"visualization"},{"id":"Apache-response-codes-of-top-URLs-ecs","type":"visualization"},{"id":"Apache-response-codes-over-time-ecs","type":"visualization"},{"id":"Apache-HTTPD-ecs","type":"search"},{"id":"Apache-access-logs-ecs","type":"search"},{"id":"Apache-errors-log-ecs","type":"search"}]}
## Now upload
curl -X POST -u elastic:changeme http://localhost:5601/api/fleet/epm/packages --data-binary @$KIBANA_HOME/x-pack/test/ingest_manager_api_integration/apis/fixtures/direct_upload_packages/apache_0.1.4.tar.gz -H 'kbn-xsrf: xyz' -H 'Content-Type: application/gzip'
{"response":[{"id":"apache-Metrics-Apache-HTTPD-server-status-ecs","type":"dashboard"},{"id":"apache-Logs-Apache-Dashboard-ecs","type":"dashboard"},{"id":"Apache-access-unique-IPs-map-ecs","type":"visualization"},{"id":"Apache-HTTPD-CPU-ecs","type":"visualization"},{"id":"Apache-HTTPD-Load1-slash-5-slash-15-ecs","type":"visualization"},{"id":"Apache-response-codes-over-time-ecs","type":"visualization"},{"id":"Apache-HTTPD-Workers-ecs","type":"visualization"},{"id":"Apache-HTTPD-Hostname-list-ecs","type":"visualization"},{"id":"Apache-error-logs-over-time-ecs","type":"visualization"},{"id":"Apache-HTTPD-Scoreboard-ecs","type":"visualization"},{"id":"Apache-HTTPD-Uptime-ecs","type":"visualization"},{"id":"Apache-operating-systems-ecs","type":"visualization"},{"id":"Apache-HTTPD-Total-accesses-and-kbytes-ecs","type":"visualization"},{"id":"Apache-browsers-ecs","type":"visualization"},{"id":"Apache-response-codes-of-top-URLs-ecs","type":"visualization"},{"id":"Apache-access-logs-ecs","type":"search"},{"id":"Apache-errors-log-ecs","type":"search"},{"id":"Apache-HTTPD-ecs","type":"search"},{"id":"logs-apache.error-0.1.4","type":"ingest_pipeline"},{"id":"logs-apache.access-0.1.4","type":"ingest_pipeline"},{"id":"logs-apache.error","type":"index_template"},{"id":"metrics-apache.status","type":"index_template"},{"id":"logs-apache.access","type":"index_template"}]}
```
</details>
<details><summary>curl request/responses for archive errors:</summary>
```
## Wrong content type:
### tar.gz with application/zip:
curl -X POST -u elastic:changeme http://localhost:5601/api/fleet/epm/packages --data-binary @$KIBANA_HOME/x-pack/test/ingest_manager_api_integration/apis/fixtures/direct_upload_packages/apache_0.1.4.tar.gz -H 'kbn-xsrf: xyz' -H 'Content-Type: application/zip'
{"statusCode":400,"error":"Bad Request","message":"Error during extraction of package: Error: end of central directory record signature not found. Assumed content type was application/zip, check if this matches the archive type."}
### zip with application/gzip:
curl -X POST -u elastic:changeme http://localhost:5601/api/fleet/epm/packages --data-binary @$KIBANA_HOME/x-pack/test/ingest_manager_api_integration/apis/fixtures/direct_upload_packages/apache_0.1.4.zip -H 'kbn-xsrf: xyz' -H 'Content-Type: application/gzip'
{"statusCode":400,"error":"Bad Request","message":"Archive seems empty. Assumed content type was application/gzip, check if this matches the archive type."}
## Invalid packages
### Two top-level directories:
curl -X POST -u elastic:changeme http://localhost:5601/api/fleet/epm/packages --data-binary @$KIBANA_HOME/x-pack/test/ingest_manager_api_integration/apis/fixtures/direct_upload_packages/apache_invalid_two_toplevels_0.1.4.zip -H 'kbn-xsrf: xyz' -H 'Content-Type: application/zip'
{"statusCode":400,"error":"Bad Request","message":"Package contains more than one top-level directory."}
### No manifest:
curl -X POST -u elastic:changeme http://localhost:5601/api/fleet/epm/packages --data-binary @$KIBANA_HOME/x-pack/test/ingest_manager_api_integration/apis/fixtures/direct_upload_packages/apache_invalid_no_manifest_0.1.4.zip -H 'kbn-xsrf: xyz' -H 'Content-Type: application/zip'
{"statusCode":400,"error":"Bad Request","message":"Package must contain a top-level manifest.yml file."}
### Invalid YAML in manifest:
curl -X POST -u elastic:changeme http://localhost:5601/api/fleet/epm/packages --data-binary @$KIBANA_HOME/x-pack/test/ingest_manager_api_integration/apis/fixtures/direct_upload_packages/apache_invalid_manifest_invalid_yaml_0.1.4.zip -H 'kbn-xsrf: xyz' -H 'Content-Type: application/zip'
{"statusCode":400,"error":"Bad Request","message":"Could not parse top-level package manifest: YAMLException: bad indentation of a mapping entry at line 2, column 7:\n name: apache\n ^."}
### Mandatory field missing in manifest:
curl -X POST -u elastic:changeme http://localhost:5601/api/fleet/epm/packages --data-binary @$KIBANA_HOME/x-pack/test/ingest_manager_api_integration/apis/fixtures/direct_upload_packages/apache_invalid_manifest_missing_field_0.1.4.zip -H 'kbn-xsrf: xyz' -H 'Content-Type: application/zip'
{"statusCode":400,"error":"Bad Request","message":"Invalid top-level package manifest: one or more fields missing of name, version, description, type, categories, format_version"}
### Top-level directory doesn't match name and version from manifest:
curl -X POST -u elastic:changeme http://localhost:5601/api/fleet/epm/packages --data-binary @$KIBANA_HOME/x-pack/test/ingest_manager_api_integration/apis/fixtures/direct_upload_packages/apache_invalid_toplevel_mismatch_0.1.4.zip -H 'kbn-xsrf: xyz' -H 'Content-Type: application/zip'
{"statusCode":400,"error":"Bad Request","message":"Name thisIsATypo and version 0.1.4 do not match top-level directory apache-0.1.4"}
```
</details>
#### TS type check examples on `installPackage`
<details><summary>screenshots</summary>
<img width="379" alt="Screen Shot 2020-11-08 at 4 00 14 PM" src="https://user-images.githubusercontent.com/57655/98484251-1d1e9f80-21dc-11eb-93f8-601036b45355.png">
<img width="890" alt="Screen Shot 2020-11-08 at 4 00 21 PM" src="https://user-images.githubusercontent.com/57655/98484252-1db73600-21dc-11eb-88d1-5faa498f94fc.png">
<img width="396" alt="Screen Shot 2020-11-08 at 4 01 06 PM" src="https://user-images.githubusercontent.com/57655/98484253-1db73600-21dc-11eb-8e2a-10a5762f4a95.png">
<img width="441" alt="Screen Shot 2020-11-08 at 4 01 25 PM" src="https://user-images.githubusercontent.com/57655/98484254-1db73600-21dc-11eb-9d9a-c1620dcad11e.png">
<img width="879" alt="Screen Shot 2020-11-08 at 4 02 54 PM" src="https://user-images.githubusercontent.com/57655/98484255-1db73600-21dc-11eb-8f36-7da3e9256feb.png">
</details>
### Checklist
- [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
## Summary
* Function rename to follow existing get/fetch convention
```diff
- export async function loadRegistryPackage(
+ export async function getRegistryPackage(
```
* Moved `unpackRegistryPackageToCache` into `getRegistryPackage`
* In my opinion, those three/four statements are more clear than the previous name
* It isn't used anywhere else
## Summary
Moving the memory store functions to `archive/cache.ts` to better express their role.
They are archive-related functions. The registry is just one possible source of an archive/assets. Also considered moving to `assets`, but can always come back to that.
`deletePackageCache` undoes side-effects from `unpackArchiveToCache` so put them in the same file
## Summary
Basic cut-and-paste of `parseAndVerify*` functions from `archive/index.ts` to `archive/validation.ts`.
Should be easier to mock now, replace later, etc.
## Summary
Update `unpackRegistryPackageToCache` to call `unpackArchiveToCache` instead of duplicating much of it.
Now an archive is iterated & put in cache via the same function regardless of its initial source.
## Summary
Move logic from `getBufferExtractorForContentType` into `getBufferExtractor` & change the interface so one function can be used.
### Diff showing old vs new call
```diff
- getBufferExtractorForContentType(contentType);
+ getBufferExtractor({ contentType });
```
```diff
- getBufferExtractor(archivePath);
+ getBufferExtractor({ archivePath });
```
### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
* Properly handle kibana assets with underscores in their path
* Recomment test
* Fix type check
* Don't install index patterns that are reserved
* Introduce SavedObjectType to use on AssetReference
* Fix Test
* Update install.ts
Use new `dataTypes` const which replaced `DataType` enum
* Update install.ts
Remove unused `indexPatternTypes` from outer scope
* Update install.ts
fix (?) bad updates from before where new/correct value was used but result wasn't exported
* Update install.ts
* Update install.ts
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: John Schulz <john.schulz@elastic.co>
## Summary
Follow pattern from #82188. Same behavior as now. Stronger type safety.
* `defaultPackages` is runtime value (JS object)
* `DefaultPackage` is TS type (`typeof defaultPackages`)
* `ValueOf<DefaultPackage>` is TS union type `'system' | 'endpoint'`
* Can access values directly as `defaultPackages.System` (JS) or `DefaultPackage['System']` (TS)
## Summary
Follow pattern from #82188. Same behavior as now. Stronger type safety.
* `installationSources` is runtime value (JS object)
* `InstallationSource` is TS type (`typeof installationSources`)
* `ValueOf<InstallationSource>` is TS union type `'installed' | 'not_installed'`
* Can access values directly as `installationSources.Installed` (JS) or `InstallationSource['Installed']` (TS)
* chore(NA): update gitignore to include first changes from moving into a single package.json
* chore(NA): update gitignore
* chore(NA): move all the dependencies into the single package.json and apply changes to bootstrap
* chore(NA): fix types problems after the single package json
* chore(NA): include code to find the dependencies used across the code
* chore(NA): introduce pure lockfile for install dependencies on build
* chore(NA): update clean task to not delete anything from xpack node_modules
* chore(NA): update gitignore to remove development temporary rules
* chore(NA): update notice file
* chore(NA): update jest snapshots
* chore(NA): fix whitelisted licenses to include a new specify form of an already included one
* chore(NA): remove check lockfile symlinks from child projects
* chore(NA): fix eslint and add missing declared deps on single pkg json
* chore(NA): correctly update notice
* chore(NA): fix failing jest test for storyshots.test.tsx
* chore(NA): fix cypress multi reporter path
* chore(NA): fix Project tests check
* chore(NA): fix problem with logic to detect used dependes on oss build
* chore(NA): include correct x-pack plugins dep discovery
* chore(NA): discover entries under dynamic requires on vis_type_timelion
* chore(NA): remove canvas
* test(NA): fix jest unit tests
* chore(NA): remove double react declaration from storyshot test file
* chore(NA): try removing isOSS check
* chore(NA): support for plugin development
* chore(NA): update logic to fix unit tests and typechecking
* chore(NA): support to run npm scripts in child kbn projects across all envs
* chore(NA): support github checks reporter on x-pack and remove cpy types as the package correctly provides them
* chore(NA): update cpy version
* chore(NA): include last kbn pm changes
* chore(NA): update style on build_production_projects.ts
* chore(NA): remove any cast fom telemetry opt in stats
* chore(NA): remove del and re-use rm -rf again
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* remove -SNAPSHOT from kibana version
* add integration tests with -SNAPSHOT version of kibana
* update isAgentUpgradeable to compare version numbers only
* continue to send the kibana version with snapshot suffix to agent
* cleanup code into one function
* fix test to check for snapshot before adding suffix
