Commit graph

79 commits

Author SHA1 Message Date
Spencer
21c54d38c3
[ftr] auto assign ciGroupDocker to suites with dockerServers (#99393)
Co-authored-by: spalger <spalger@users.noreply.github.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-05-06 12:42:29 -07:00
Zacqary Adam Xeper
a72b0393f3
[Fleet] Fix frozen key definition (#99232)
* [Fleet] Fix frozen key definition

* Add integration test for frozen vars
2021-05-04 17:24:11 -05:00
Nicolas Chaulet
ed8dc62f77
[Fleet] Add default http|https port to ES hosts (#99240) 2021-05-04 18:13:06 -04:00
Nicolas Chaulet
b2a20fc9af
[Fleet] Fix fleet server host port during settings creation (#99084) 2021-05-03 17:42:07 -04:00
Brian Seeders
f404305eaa
[CI] Create a ciGroup just for tests that use Docker (#98535) 2021-05-03 13:53:22 -04:00
Ross Wolf
0ddea04544
Make security rules optional (revert #97191) (#98854) 2021-04-29 19:59:52 -06:00
Zacqary Adam Xeper
57f84f8593
[Fleet] Refactor setup to load default packages/policies with preconfiguration (#97328)
Co-authored-by: Nicolas Chaulet <nicolas.chaulet@elastic.co>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-04-22 11:25:10 -05:00
Nicolas Chaulet
ee7ba71b43
[Fleet] Always create agent policy as active (#97874) 2021-04-21 17:13:48 -04:00
John Schulz
45255425f4
Add the more explicit & actionable text listed in ticket (#97857)
## Summary
closes #92591

Create a new `HostedAgentPolicyRestrictionRelatedError` which ensures the additional text from #92591 is included in all the appropriate places, but only specified once.

#### Some current examples
 * `Cannot update integrations of hosted agent policy ${id}`
 * `Cannot remove integrations of hosted agent policy ${id}`
 * `Cannot reassign an agent to hosted agent policy ${newAgentPolicy.id}`


#### In this PR are now
 * `Cannot update integrations of hosted agent policy ${id}  in Fleet because the agent policy is managed by an external orchestration solution, such as Elastic Cloud, Kubernetes, etc. Please make changes using your orchestration solution.`
 * `Cannot remove integrations of hosted agent policy ${id} in Fleet because the agent policy is managed by an external orchestration solution, such as Elastic Cloud, Kubernetes, etc. Please make changes using your orchestration solution.`
 * `Cannot reassign an agent to hosted agent policy ${newAgentPolicy.id} in Fleet because the agent policy is managed by an external orchestration solution, such as Elastic Cloud, Kubernetes, etc. Please make changes using your orchestration solution.`


### Checklist

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2021-04-21 15:52:02 -04:00
Jen Huang
10e52bb582
[Fleet] Add instructions and generation of a service token for Fleet Server onboarding (#97585)
* Don't block standalone agent instructions when not using Fleet server yet

* Add service token instructions - UI only

* Add route for regenerating fleet server service token

* generate tokens instead of regenerate and add error catching and tests

* fix i18n typo

* i18n fix, add sudo, copy edits

* Fix commands

* Add missing test file

Co-authored-by: Nicolas Chaulet <nicolas.chaulet@elastic.co>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-04-20 13:53:18 -04:00
Ross Wolf
1925cea9a2
[Security][Fleet] Install the security_detection_engine package automatically (#97191)
* Automatically install the security_detection_engine package via fleet
* Update dockerImage to include the security_detection_engine package
* Update api/fleet/setup install test
* Update test data for Endpoint package
* Fix JSON token
* Update firis json entry in destination_index
* Update destination_index structure
* Update destination_index structure
* Change KQL query to unblock testing
* Restore KQL and fix JSON instead
* update timestamps to pass tests
2021-04-20 13:07:11 -04:00
Sonja Krause-Harder
05bd1c0cdb
[Fleet] Finer-grained error information from install/upgrade API (#95649)
* Intercept installation errors and add meta info.

* Adjust mock.

* Catch errors in all steps of install/upgrade.

* Adjust handler for direct package upload.

* Don't throw not-found errors on assets during rollback.

* Correctly catch errors from _installPackage()

* Propagate error from installResult in bulk install case.

* Add tests for rollback.

* Remove unused code.

* Skipping test that doesn't test what it says.

* Fix and reenable test.
2021-04-18 16:47:24 +02:00
John Schulz
6e5c9278ba
[Fleet] Use 'hosted agent policy' naming in UI & API messages (#97287)
## Summary

### What this PR _does_ change
 - [x] Replace all UI & API instances of "managed policy" with "hosted agent policy" #93504
 - [x] Replace all UI & API instances of "unmanaged policy" with "regular agent policy" #93504
 - [x] Update most variable names containing `managed` to `hosted`

<details><summary>screenshots</summary>

<img width="489" alt="Screen Shot 2021-04-15 at 10 54 01 AM" src="https://user-images.githubusercontent.com/57655/114898051-cac35980-9ddf-11eb-9c55-3fb5c0211c63.png">
<img width="339" alt="Screen Shot 2021-04-15 at 11 40 47 AM" src="https://user-images.githubusercontent.com/57655/114898053-cac35980-9ddf-11eb-8c60-3459aed3db6c.png">
<img width="714" alt="Screen Shot 2021-04-15 at 11 40 59 AM" src="https://user-images.githubusercontent.com/57655/114898054-cac35980-9ddf-11eb-912a-623eba9c3860.png">
</details>

### What this PR _does not_ change
  - [ ] The `is_managed` property of the agent policy saved object*
  - [ ] The updated error messages in https://github.com/elastic/kibana/issues/92591

<details><summary>*why not change the <code>is_managed</code> property?</summary>

<p>Changing the property will require a migration from 7.12. That's not a problem, but we haven't decided on <a href="https://github.com/elastic/kibana/issues/93504#issuecomment-790060997">a new name/approach</a> yet. Rather than update to <code>is_hosted: boolean</code> now and potentially something like <code>managed_by: enum</code> later, we'll update it when we have a decision.

<p><strong>TL;DR:</strong> It's a trivial update to make later and a minuscule amount of debt to pay until then. Less than cost of doing it twice.
 
</details>

### Checklist

- [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2021-04-15 21:01:28 -04:00
Nicolas Chaulet
78937e3bb4
[Fleet] Remove fleet agent routes and related services (#97206) 2021-04-15 17:15:43 -04:00
Nicolas Chaulet
b96172e27c
[Fleet] Fleet server onboarding UI (#96867) 2021-04-15 01:22:32 -04:00
John Schulz
de4bcdb9d9
[Fleet] Rename force to revoke agent unenroll APIs (#97041)
## Summary

 - fcbc9d9 Rename `force` param to `revoke` for `/agents/{agent_id}/unenroll` & `/agents/bulk_unenroll`
 - 03b9b90 Add new `force` param

See https://github.com/elastic/kibana/issues/96873 for background

<table>
  <thead>
    <tr>
      <td rowspan="2"></td><td colspan="2">Unenroll Agent</td><td rowspan="2">Revoke API Keys</td>
    </tr>
    <tr>
      <td>Regular</td><td>Hosted</td></td>
    </tr>
  </thead>
  <tr><td colspan="4"><strong>Rename <code>force</code> to <code>revoke</code></strong></td></tr>
  <tr><td>Current <code>force=false|undefined</code></td><td></td><td></td><td></td></tr>
  <tr><td>Proposed <code>revoke=false|undefined</code></td><td></td><td></td><td></td></tr>
  <tr><td>Current <code>force=true</code></td><td></td><td></td><td></td></tr>
  <tr><td>Proposed <code>revoke=true</code></td><td></td><td></td><td></td></tr>
  <tr><td colspan="4"><strong>Change <code>force</code> param </strong></td></tr>
  <tr><td>Proposed <code>force=false|undefined</code></td><td></td><td></td><td></td></tr>
  <tr><td>Proposed <code>force=true</code></td><td></td><td></td><td></td></tr>
  <tr><td>Proposed <code>force=true</code> & <code>revoke=true</code></td><td></td><td></td><td></td></tr>
</table>

### Checklist

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

### Changes required for consumers
Any call to `/agents/{agent_id}/unenroll` & `/agents/bulk_unenroll` which passes the `force` param should change to `revoke` to maintain the current behavior.
2021-04-14 19:27:06 -04:00
John Schulz
58b1d10f0b
Copy esArchiver commands from ./reassign.ts to fix tests (#97012)
## Summary

Seeing failures like this locally for `x-pack/test/fleet_api_integration/apis/agents/unenroll.ts` tests
<details><summary>screenshot of error</summary>
<img width="1121" alt="Screen Shot 2021-04-13 at 10 06 51 AM" src="https://user-images.githubusercontent.com/57655/114592495-fec04280-9c58-11eb-939c-b84aa7469c2a.png">
</details>
Copied the `esArchiver` patterns from `x-pack/test/fleet_api_integration/apis/agents/reassign.ts` in https://github.com/elastic/kibana/pull/96837 and the error is gone

### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2021-04-13 15:42:36 -04:00
Nicolas Chaulet
0e7612dd1a
[Fleet] Fix Fleet API integration tests (#96837) 2021-04-13 11:16:32 -04:00
John Schulz
cb3c4e3a21
[Fleet] support force flag to add/remove package_policies (#96713)
## Summary

Can now pass a `force=true` parameter to add & remove integrations on hosted policies as originally intended [1] & [2]

 * Add `force` param for `POST` `/api/fleet/package_policies` & `/api/fleet/package_policies/delete` to a policy. Update tests to confirm
 * Not strictly required, but "while I was in there"
   * Updated a few places to throw `IngestManagerError` vs `Error` for `400` response vs `500`. Updated tests.
   * removed a few unnecessary `await`s of sync function

[1] https://github.com/elastic/kibana/issues/92426#issuecomment-785092670
[2] https://github.com/elastic/kibana/issues/90445

### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-04-12 14:05:39 -04:00
Rudolf Meijering
f544d8d458
Migrations v2 ignore fleet agent events (#96690)
* migrationsv2: ignore fleet agent events and tsvb telemetry

* migrationsv1: ignore tsvb-validation-telemetry

* Skip fleet test that depends on fleet-agent-events

* Fix typescript errors

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-04-12 18:19:42 +02:00
John Schulz
d338f1c3de
Allow integrations of hosted policies to be updated (#96705)
## Summary

Remove the restriction against updating integrations on hosted policies.

I described the current behavior and asked if it should change in [1]. Based on the responses in [2] & [3] and looking back at prior discussion around hosted policies, I don't think updates should be restricted.

Adding or removing integrations is still blocked for hosted policies. Updated API tests to confirm behavior. 


[1] https://github.com/elastic/kibana/issues/76843#issuecomment-816096760
[2] https://github.com/elastic/kibana/issues/76843#issuecomment-816153871
[3] https://github.com/elastic/kibana/issues/76843#issuecomment-816538672

## Screenshots
<details><summary>Current behavior</summary>

<h3>Error about updating integrations of a managed policy</h3>

<img width="1208" alt="Screen Shot 2021-04-08 at 3 23 37 PM" src="https://user-images.githubusercontent.com/57655/114084750-87686880-987e-11eb-91a9-081c45dbe871.png">

<details><summary>via flow A</summary>
<img width="1223" alt="Screen Shot 2021-04-08 at 3 01 32 PM" src="https://user-images.githubusercontent.com/57655/114082826-3a839280-987c-11eb-94d0-151ae93ab523.png">

<img width="1205" alt="Screen Shot 2021-04-08 at 3 13 24 PM" src="https://user-images.githubusercontent.com/57655/114083728-5c314980-987d-11eb-92be-195d7d44c037.png">
</details>

<details><summary>via flow B</summary>
<img width="1221" alt="Screen Shot 2021-04-08 at 3 19 52 PM" src="https://user-images.githubusercontent.com/57655/114084502-3fe1dc80-987e-11eb-8879-57718586ac95.png">
<img width="1198" alt="Screen Shot 2021-04-08 at 3 20 06 PM" src="https://user-images.githubusercontent.com/57655/114084503-3fe1dc80-987e-11eb-9fa9-512210b938cd.png">
</details>

</details>

<details><summary>This PR</summary>
<h3>Successful updates using either form</h3>
<img width="1301" alt="Screen Shot 2021-04-09 at 1 21 02 PM" src="https://user-images.githubusercontent.com/57655/114219370-8f84de80-9938-11eb-9b94-dfbeb18535b2.png">
<img width="1320" alt="Screen Shot 2021-04-09 at 1 05 10 PM" src="https://user-images.githubusercontent.com/57655/114219408-9f9cbe00-9938-11eb-96d2-2918332d1539.png">

</details>


### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-04-12 11:01:38 -04:00
Nicolas Chaulet
1de77ccb4e
[Fleet] Create enrollment API keys as current user (#96464) 2021-04-12 08:27:54 -04:00
spalger
afc1fd022e skip entire fleet_api_integration suite to unblock es promotion (#96515) 2021-04-08 08:14:01 -07:00
spalger
f9317281d1 skip suite blocking es promotion (#96515) 2021-04-08 02:09:29 -07:00
Ross Wolf
93965343e5
[Fleet] Install security_rule assets as saved objects (#95885)
* [Fleet] Install security_rule assets as saved objects

* Add security-rule to update_assets.ts

* Update UUIDs for security_rule asset

* Change .type to match the saved object type not the asset type

* Add saved object mapping for security-rule

* Make SO non-hidden

* Fix SO mapping for security-rule

* Make security-rule a non-hidden asset
2021-04-08 00:52:10 -04:00
spalger
0cf31ae22f skip suite block es promotion (#96515)
(cherry picked from commit f06be93a40)
2021-04-07 18:59:01 -07:00
spalger
fc9f97e03b skip suites failing es promotion (#96515)
(cherry picked from commit 7fdf7e1d79)
2021-04-07 16:10:19 -07:00
Zacqary Adam Xeper
f7caf44876
[Fleet] Match telemetry key names to UI agent states (#95567) 2021-03-31 15:10:46 -04:00
Zacqary Adam Xeper
28410539b5
[Fleet] Configure Fleet packages and integrations through endpoint (#94509) 2021-03-29 21:41:27 -04:00
James Gowdy
478ad3bad5
[Fleet] Adding ML module asset type (#94950)
* [Fleet] Adding ML module asset type

* adding test

* guessing asset ids

* better guess at IDs

* renaming asset ids

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-03-29 16:21:49 +01:00
John Schulz
7f15931725
[Fleet] /bulk_unenroll response matches other bulk action APIs (#95571)
## Summary
`/agents/bulk_unenroll` should return a response with a result for each agent given; including invalid or missing ids. It currently returns an empty object. https://github.com/elastic/kibana/issues/90437


[TS type diff for response](dd34e4c5ef/x-pack/plugins/fleet/common/types/rest_spec/agent.ts (L124-L130))

```diff
- // eslint-disable-next-line @typescript-eslint/no-empty-interface
- export interface PostBulkAgentUnenrollResponse {}
+ export type PostBulkAgentUnenrollResponse = Record<
+   Agent['id'],
+   {
+     success: boolean;
+     error?: string;
+   }
+ >;
```

### Checklist

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2021-03-26 17:36:39 -04:00
John Schulz
ab33df898a
[Fleet] Bulk upgrade api response change (#95236)
## Summary
`/agents/bulk_upgrade` should return a response with a result for each agent given; including invalid or missing ids. It currently returns an empty object.

This PR includes commits from open PR https://github.com/elastic/kibana/pull/95024. The additions from this PR are https://github.com/jfsiii/kibana/compare/bulk-reassign-response-should-include-all-given-agents..871ebcb

[TS type diff for response](https://github.com/jfsiii/kibana/compare/bulk-reassign-response-should-include-all-given-agents..871ebcb#diff-7006a6c170a608c8c7211fc218c0a6f4bc8ff642c170ea264db4b1b5545fb728)

```diff
- // eslint-disable-next-line @typescript-eslint/no-empty-interface
- export interface PostBulkAgentUpgradeResponse {}

+ export type PostBulkAgentUpgradeResponse = Record<
+   Agent['id'],
+   {
+     success: boolean;
+     error?: string;
+   }
+ >;
```

### Checklist

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-03-26 11:40:38 -05:00
Tomas Della Vedova
238791b942
ES client : use the new type definitions (#83808)
* Use client from branch

* Get type checking working in core

* Fix types in other plugins

* Update client types + remove type errors from core

* migrate Task Manager Elasticsearch typing from legacy library to client library

* use SortOrder instead o string in alerts

* Update client types + fix core type issues

* fix maps ts errors

* Update Lens types

* Convert Search Profiler body from a string to an object to conform to SearchRequest type.

* Fix SOT types

* Fix/mute Security/Spaces plugins type errors.

* Fix bootstrap types

* Fix painless_lab

* corrected es typing in Event Log

* Use new types from client for inferred search responses

* Latest type defs

* Integrate latest type defs for APM/UX

* fix core errors

* fix telemetry errors

* fix canvas errors

* fix data_enhanced errors

* fix event_log errors

* mute lens errors

* fix or mute maps errors

* fix reporting errors

* fix security errors

* mute errors in task_manager

* fix errors in telemetry_collection_xpack

* fix errors in data plugins

* fix errors in alerts

* mute errors in index_management

* fix task_manager errors

* mute or fix lens errors

* fix upgrade_assistant errors

* fix or mute errors in index_lifecycle_management

* fix discover errors

* fix core tests

* ML changes

* fix core type errors

* mute error in kbn-es-archiver

* fix error in data plugin

* fix error in telemetry plugin

* fix error in discover

* fix discover errors

* fix errors in task_manager

* fix security errors

* fix wrong conflict resolution

* address errors with upstream code

* update deps to the last commit

* remove outdated comments

* fix core errors

* fix errors after update

* adding more expect errors to ML

* pull the lastest changes

* fix core errors

* fix errors in infra plugin

* fix errors in uptime plugin

* fix errors in ml

* fix errors in xpack telemetry

* fix or mute errors in transform

* fix errors in upgrade assistant

* fix or mute fleet errors

* start fixing apm errors

* fix errors in osquery

* fix telemetry tests

* core cleanup

* fix asMutableArray imports

* cleanup

* data_enhanced cleanup

* cleanup events_log

* cleaup

* fix error in kbn-es-archiver

* fix errors in kbn-es-archiver

* fix errors in kbn-es-archiver

* fix ES typings for Hit

* fix SO

* fix actions plugin

* fix fleet

* fix maps

* fix stack_alerts

* fix eslint problems

* fix event_log unit tests

* fix failures in data_enhanced tests

* fix test failure in kbn-es-archiver

* fix test failures in index_pattern_management

* fixing ML test

* remove outdated comment in kbn-es-archiver

* fix error type in ml

* fix eslint errors in osquery plugin

* fix runtime error in infra plugin

* revert changes to event_log cluser exist check

* fix eslint error in osquery

* fixing ML endpoint argument types

* fx types

* Update api-extractor docs

* attempt fix for ese test

* Fix lint error

* Fix types for ts refs

* Fix data_enhanced unit test

* fix lens types

* generate docs

* Fix a number of type issues in monitoring and ml

* fix triggers_actions_ui

* Fix ILM functional test

* Put search.d.ts typings back

* fix data plugin

* Update typings in typings/elasticsearch

* Update snapshots

* mute errors in task_manager

* mute fleet errors

* lens. remove unnecessary ts-expect-errors

* fix errors in stack_alerts

* mute errors in osquery

* fix errors in security_solution

* fix errors in lists

* fix errors in cases

* mute errors in search_examples

* use KibanaClient to enforce promise-based API

* fix errors in test/ folder

* update comment

* fix errors in x-pack/test folder

* fix errors in ml plugin

* fix optional fields in ml api_integartoon tests

* fix another casting problem in ml tests

* fix another ml test failure

* fix fleet problem after conflict resolution

* rollback changes in security_solution. trying to fix test

* Update type for discover rows

* uncomment runtime_mappings as its outdated

* address comments from Wylie

* remove eslint error due to any

* mute error due to incompatibility

* Apply suggestions from code review

Co-authored-by: John Schulz <github.com@jfsiii.org>

* fix type error in lens tests

* Update x-pack/plugins/upgrade_assistant/server/lib/reindexing/reindex_service.ts

Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com>

* Update x-pack/plugins/upgrade_assistant/server/lib/reindexing/reindex_service.test.ts

Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com>

* update deps

* fix errors in core types

* fix errors for the new elastic/elasticsearch version

* remove unused type

* remove unnecessary manual type cast and put optional chaining back

* ML: mute Datafeed is missing indices_options

* Apply suggestions from code review

Co-authored-by: Josh Dover <1813008+joshdover@users.noreply.github.com>

* use canary pacakge instead of git commit

Co-authored-by: Josh Dover <me@joshdover.com>
Co-authored-by: Josh Dover <1813008+joshdover@users.noreply.github.com>
Co-authored-by: Gidi Meir Morris <github@gidi.io>
Co-authored-by: Nathan Reese <reese.nathan@gmail.com>
Co-authored-by: Wylie Conlon <wylieconlon@gmail.com>
Co-authored-by: CJ Cenizal <cj@cenizal.com>
Co-authored-by: Aleh Zasypkin <aleh.zasypkin@gmail.com>
Co-authored-by: Dario Gieselaar <dario.gieselaar@elastic.co>
Co-authored-by: restrry <restrry@gmail.com>
Co-authored-by: James Gowdy <jgowdy@elastic.co>
Co-authored-by: John Schulz <github.com@jfsiii.org>
Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com>
2021-03-25 04:47:16 -04:00
Sonja Krause-Harder
2e5b5debb5
[Fleet] Add force option to DELETE package endpoint. (#95051)
* Add force option to DELETE package endpoint.

* Add integration test.

* Adjust openapi spec.

* Run EPM tests before fleet setup tests.

* Run package delete tests first in EPM tests
2021-03-24 17:04:49 +01:00
John Schulz
3639aa4422
[Fleet] Bulk reassign response should include all given ids (#95024)
## Summary
`/agents/bulk_reassign` should return a response with a result for each agent given; including invalid or missing ids. It currently filters out missing or invalid before updating. This PR leaves them in and includes their error results in the response. 

[Added/updated tests](https://github.com/elastic/kibana/pull/95024/files#diff-7ec94bee3e2bae79e5d98b8c17c17b26fad14736143ffa144f3e035773d4cad1R113-R128) to confirm

### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-03-24 11:44:22 -04:00
Jen Huang
d886979e3b
[Fleet] Remove upgradePackage and consolidate it with installPackage, optimize calls to create index patterns (#94490)
* Add data plugin to server app context

* First attempt at switching to indexPatternService for EPM index pattern creation & deletion, instead of interacting directly with index pattern SOs

* Simplify bulk install package, remove upgradePackage() method in favor of consolidating with installPackage(), use installPackage() for bulk install instead

* Update tests

* Change cache logging of objects to trace level

* Install index patterns as a post-package installation operation, for bulk package installation, install index pattern only if one or more packages are actually new installs, add debug logging

* Allow getAsSavedObjectBody to return non-scripted fields when allowNoIndex is true

* Allow `getFieldsForWildcard` to return fields saved on index pattern when allowNoIndices is true

* Bring back passing custom ID for index pattern SO

* Fix tests

* Revert "Merge branch 'index-pattern/allow-no-index' into epm/missing-index-patterns"

This reverts commit 8e712e9c00, reversing
changes made to af0fb0eaa8.

* Allow getAsSavedObjectBody to return non-scripted fields when allowNoIndex is true

(cherry picked from commit 69b93da180)

* Update API docs

* Run post-install ops for install by upload too

* Remove allowedInstallTypes param

* Adjust force install conditions

* Revert "Update API docs"

This reverts commit b9770fdc56.

* Revert "Allow getAsSavedObjectBody to return non-scripted fields when allowNoIndex is true"

This reverts commit afc91ce32f.

* Go back to using SO client for index patterns :(

* Stringify attributes again for SO client saving

* Fix condition for reinstall same pkg version

* Remove unused type

* Adjust comment

* Update snapshot

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-03-23 11:34:20 -07:00
Nicolas Chaulet
76b55207f7
[Fleet] Fix agent status count (#95099) 2021-03-23 11:24:25 -04:00
John Schulz
c0c5fba4b0
[Fleet] Support force flag in bulk upgrade agents API (#94952)
## Summary

0cbbb41da2 is just a rearrangement of the tests.  5cad301e87 has the real changes: 
* Bug fix: `force: true` should bypass any restrictions re: managed policies
* Refactoring towards new response shape coming as part of https://github.com/elastic/kibana/issues/90437
* Added test to confirm new behavior


### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2021-03-23 11:22:03 -04:00
John Schulz
044a94ac46
[Fleet] Add test/fix for invalid/missing ids in bulk agent reassign (#94632)
## Problem
While working on changes for bulk reassign https://github.com/elastic/kibana/issues/90437, I found that the server has a runtime error and returns a 500 if given an invalid or missing id.

<details><summary>server error stack trace</summary>

```
   │ proc [kibana] server    log   [12:21:48.953] [error][fleet][plugins] TypeError: Cannot read property 'policy_revision_idx' of undefined
   │ proc [kibana]     at map (/Users/jfsiii/work/kibana/x-pack/plugins/fleet/server/services/agents/helpers.ts:15:34)
   │ proc [kibana]     at Array.map (<anonymous>)
   │ proc [kibana]     at getAgents (/Users/jfsiii/work/kibana/x-pack/plugins/fleet/server/services/agents/crud.ts:191:32)
   │ proc [kibana]     at runMicrotasks (<anonymous>)
   │ proc [kibana]     at processTicksAndRejections (internal/process/task_queues.js:93:5)
   │ proc [kibana]     at Object.reassignAgents (/Users/jfsiii/work/kibana/x-pack/plugins/fleet/server/services/agents/reassign.ts:91:9)
   │ proc [kibana]     at postBulkAgentsReassignHandler (/Users/jfsiii/work/kibana/x-pack/plugins/fleet/server/routes/agent/handlers.ts:314:21)
   │ proc [kibana]     at Router.handle (/Users/jfsiii/work/kibana/src/core/server/http/router/router.ts:272:30)
   │ proc [kibana]     at handler (/Users/jfsiii/work/kibana/src/core/server/http/router/router.ts:227:11)
   │ proc [kibana]     at exports.Manager.execute (/Users/jfsiii/work/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
   │ proc [kibana]     at Object.internals.handler (/Users/jfsiii/work/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)
   │ proc [kibana]     at exports.execute (/Users/jfsiii/work/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)
   │ proc [kibana]     at Request._lifecycle (/Users/jfsiii/work/kibana/node_modules/@hapi/hapi/lib/request.js:370:32)
   │ proc [kibana]     at Request._execute (/Users/jfsiii/work/kibana/node_modules/@hapi/hapi/lib/request.js:279:9)
```
</details>

<details><summary>see test added in this PR fail on master</summary>

```
1)    Fleet Endpoints
       reassign agent(s)
         bulk reassign agents
           should allow to reassign multiple agents by id -- some invalid:

      Error: expected 200 "OK", got 500 "Internal Server Error"
```
</details>

## Root cause
Debugging runtime error in `searchHitToAgent` found some TS type mismatches for the ES values being returned. Perhaps from one or more of the recent changes to ES client & Fleet Server. Based on `test:jest` and `test:ftr`, it appears the possible types are `GetResponse` or `SearchResponse`, instead of only an `ESSearchHit`.

https://github.com/elastic/kibana/pull/94632/files#diff-254d0f427979efc3b442f78762302eb28fb9c8857df68ea04f8d411e052f939cL11

While a `.search` result will include return matched values, a `.get` or `.mget` will return a row for each input and a `found: boolean`. e.g. `{ _id: "does-not-exist", found: false }`. The error occurs when [`searchHitToAgent`](1702cf98f0/x-pack/plugins/fleet/server/services/agents/helpers.ts (L11)) is run on a get miss instead of a search hit.

## PR Changes
* Added a test to ensure it doesn't fail if invalid or missing IDs are given
* Moved the `bulk_reassign` tests to their own test section
* Filter out any missing results before calling `searchHitToAgent`, to match current behavior
* Consolidate repeated arguments into and code for getting agents into single [function](https://github.com/elastic/kibana/pull/94632/files#diff-f7377ed9ad56eaa8ea188b64e957e771ccc7a7652fd1eaf44251c25b930f8448R70-R87):  and [TS type](https://github.com/elastic/kibana/pull/94632/files#diff-f7377ed9ad56eaa8ea188b64e957e771ccc7a7652fd1eaf44251c25b930f8448R61-R68)
* Rename some agent service functions to be more explicit (IMO) but behavior maintained. Same API names exported.

This moves toward the "one result (success or error) per given id" approach for https://github.com/elastic/kibana/issues/90437

### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2021-03-16 20:51:57 -04:00
Jen Huang
67e968304e
[Fleet] Remove coreMigrationVersion from snapshot (#94235)
* Remove coreMigrationVersion from snapshot

(cherry picked from commit a8b9b41a68)

* Remove `latestVersion` too

(cherry picked from commit 6f3949c8d8)
2021-03-09 18:59:36 -08:00
Zacqary Adam Xeper
5d119cfcbb
[Fleet] Return empty agents list when submitting a kuery with no keys (#93844) 2021-03-09 16:10:56 -06:00
Jen Huang
3946c4ad91
[Fleet] Fix package policies created with null var values (empty defaults) (#94040)
* Make package validation be based on types

* Add package info to cache after it is generated from ES storage

* Add logging around package info retrieval and when cache is set

* Add snapshot api integration test for uploaded package info

* Use the apache package for snapshot test instead

* Remove date field from snapshot

* Update docs

* Fix streams getting overridden

* Add back package field to data streams

* PR fixes
2021-03-09 14:53:22 -05:00
Nicolas Chaulet
f4caf8727d
[Fleet] Remove fleetServerEnabled feature flag and use fleet system indices (#92422) 2021-03-08 07:40:09 -05:00
Nicolas Chaulet
02be8c16c9
[Fleet] Improve error status code for package (#93696) 2021-03-05 13:50:14 -05:00
Jen Huang
0d4ca46cf0
[Fleet] Set all keyword and text fields for index.query.default_field index template setting (#91791)
* Set all keyword and text fields for `index.query.default_field` setting

* Update tests and snapshots

* Fix test

* Add default field limit safeguard

* Add logging when beyond limit

* Update tests to mock app context (because I added logger usage)

* Update api integration test

* Rename consts
2021-03-04 16:34:47 -05:00
Nicolas Chaulet
fe35e0de3b
[Fleet] Install Elastic Agent integration by default during setup (#91676) 2021-02-18 18:45:15 -05:00
John Schulz
0a5e054fdc
[Fleet] Don't upgrade agent(s) in a managed policy (#91303)
## Summary

 - Make sure any agents requesting to be upgraded, are not enrolled in a managed policy.
 - `force: true` will only bypass agent / kibana version checks. It will not bypass managed policy check. To workaround, the enrolled policy should be changed to unmanaged (`is_managed: false`) as we do with enroll, reassign, etc.
 - Took more efficient approach to bulk actions. One `bulkGet` for N agents/policies vs N `get`s  approach used for bulk reassignment of agents. See discussion in https://github.com/elastic/kibana/pull/88688/files#r568941761
 - [x] API
 - [ ] UI
 - [x] tests

### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios


### Manual tests
#### upgrade one
```
curl --location --request POST 'http://localhost:5601/api/fleet/agents/8d9748e0-6d52-11eb-8cbd-47e38cd1c8de/upgrade' --header 'kbn-xsrf: <string>' --header 'Content-Type: application/json' --header 'Authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==' --data-raw '{
    "version": "8.0.0"
}'
{"statusCode":400,"error":"Bad Request","message":"Cannot upgrade agent 8d9748e0-6d52-11eb-8cbd-47e38cd1c8de in managed policy bf319100-6d50-11eb-8859-15a87f509a99"}
```

```
curl --location --request POST 'http://localhost:5601/api/fleet/agents/8d9748e0-6d52-11eb-8cbd-47e38cd1c8de/upgrade' --header 'kbn-xsrf: <string>' --header 'Content-Type: application/json' --header 'Authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==' --data-raw '{
    "version": "8.0.0", "force": true
}'
{"statusCode":400,"error":"Bad Request","message":"Cannot upgrade agent 8d9748e0-6d52-11eb-8cbd-47e38cd1c8de in managed policy bf319100-6d50-11eb-8859-15a87f509a99"}
```

#### bulk upgrade
```
curl --location --request POST 'http://localhost:5601/api/fleet/agents/bulk_upgrade' --header 'kbn-xsrf: <string>' --header 'Content-Type: application/json' --header 'Authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==' --data-raw '{
    "version": "8.0.0",
    "agents": [
        "8d9748e0-6d52-11eb-8cbd-47e38cd1c8de"
    ]
}'
{}
```

```
curl --location --request POST 'http://localhost:5601/api/fleet/agents/bulk_upgrade' --header 'kbn-xsrf: <string>' --header 'Content-Type: application/json' --header 'Authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==' --data-raw '{
    "version": "8.0.0",
    "agents": [
        "8d9748e0-6d52-11eb-8cbd-47e38cd1c8de"
    ], "force": true
}'
{"statusCode":400,"error":"Bad Request","message":"Cannot update agent in managed policy bf319100-6d50-11eb-8859-15a87f509a99"}```
```
2021-02-15 13:36:39 -05:00
Sonja Krause-Harder
9870ade971
[Fleet] Reduce permissions. (#90302)
* Reduce permissions.

* Change permissions back.

* Reducing permissions on fleet_enroll role

- 'write', 'create_index' -> 'auto_configure', 'create_doc'

* Remove indices:admin/auto_create from privileges.
2021-02-11 09:33:51 +01:00
John Schulz
c92af5a4d5
[Fleet] Restrict integration changes for managed policies (#90675)
## Summary

- [x] Integrations cannot be added ~~, unless with a force flag~~
  - [x] API
  - [x] UI
  - [x] tests
- [x] Integrations cannot be removed ~~, unless with a force flag~~
  - [x] API
  - [x] UI
  - [x] tests

closes https://github.com/elastic/kibana/issues/90445
refs https://github.com/elastic/kibana/issues/89617

### Cannot add integrations to managed policy

<img height="400" alt="Screen Shot 2021-02-08 at 1 56 32 PM" src="https://user-images.githubusercontent.com/57655/107277261-25c48300-6a22-11eb-936a-0a7361667093.png">

### Cannot delete integrations from managed policy

<img  alt="Screen Shot 2021-02-08 at 3 05 16 PM" src="https://user-images.githubusercontent.com/57655/107277318-337a0880-6a22-11eb-836f-fc66b510d257.png">

### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2021-02-10 17:04:01 -05:00
John Schulz
810e4ab8e8
[Fleet] Prevent agents from enrolling in a managed policy (#90458)
## Summary
Add guard to `/agents/enroll` API preventing agents from enrolling in managed policies

closes #90435
- [x] No Agents can be enrolled into this policy by the user.
- [x] The install & enroll commands should print an error to the console if the enroll command fails (due to being a managed policy or any other reason)



#### So how do you associate an agent with a managed policy?
Enroll in an unmanaged policy then set that policy to managed. 

We don't restrict the agent policy, only what other things (agents, integrations) can do if they're associated with a managed policy.

A _force flag_ has been mentioned for some other actions. It might work here as well, but I'd like to handle discussion & implementation of those later.

### Manual testing
<details><summary>Prevent enroll for managed policies</summary>

1. Created a managed agent policy
    ```
    curl --user elastic:changeme -X POST localhost:5601/api/fleet/agent_policies -H 'Content-Type: application/json' -d'{ "name": "User created MANAGED", "namespace": "default", "is_managed": true}' -H 'kbn-xsrf: true'
    {"item":{"id":"3bd07db0-67d0-11eb-b656-21ad68ebfa8a","name":"User created MANAGED","namespace":"default","is_managed":true,"revision":1,"updated_at":"2021-02-05T16:36:01.931Z","updated_by":"elastic"}}
    ```
2. Try `install` command show in the UI
    ```
    sudo ./elastic-agent install -f --kibana-url=http://localhost:5601 --enrollment-token=WmcwTWMzY0IzWlBUUWJJUjZqRDA6UGRZelVlaS1STml1cVdjSUVwSkJRQQ== --insecure
    Password:
    The Elastic Agent is currently in BETA and should not be used in production

    Error: fail to enroll: fail to execute request to Kibana: Status code: 400, Kibana returned an error: Bad Request, message: Cannot enroll in managed policy 3bd07db0-67d0-11eb-b656-21ad68ebfa8a
    Error: enroll command failed with exit code: 1
    ```
3. Observe `Cannot enroll in managed policy 3bd07db0-67d0-11eb-b656-21ad68ebfa8a` error
4. Try `enroll` instead:
    ```
    sudo ./elastic-agent enroll http://localhost:5601 WmcwTWMzY0IzWlBUUWJJUjZqRDA6UGRZelVlaS1STml1cVdjSUVwSkJRQQ== --insecure
    The Elastic Agent is currently in BETA and should not be used in production

    This will replace your current settings. Do you want to continue? [Y/n]:
    Error: fail to enroll: fail to execute request to Kibana: Status code: 400, Kibana returned an error: Bad Request, message: Cannot enroll in managed policy 3bd07db0-67d0-11eb-b656-21ad68ebfa8a
    ```
5. Observe same `Cannot enroll in managed policy 3bd07db0-67d0-11eb-b656-21ad68ebfa8a` error

</details>

<details><summary>Enroll in unmanaged policy, then update it to managed</summary>

Agent policies are `is_managed: false` by default, or we can update the policy to `is_managed: false`, like:

```
curl --user elastic:changeme -X PUT localhost:5601/api/fleet/agent_policies/3bd07db0-67d0-11eb-b656-21ad68ebfa8a -H 'Content-Type: application/json' -d'{ "is_managed": false, "name": "xyz", "namespace": "default" }' -H 'kbn-xsrf: true'
{"item":{"id":"3bd07db0-67d0-11eb-b656-21ad68ebfa8a","name":"xyz","namespace":"default","is_managed":false,"revision":4,"updated_at":"2021-02-05T17:42:05.610Z","updated_by":"elastic","package_policies":[]}}
```

then enroll

```
 sudo ./elastic-agent install -f --kibana-url=http://localhost:5601 --enrollment-token=WmcwTWMzY0IzWlBUUWJJUjZqRDA6UGRZelVlaS1STml1cVdjSUVwSkJRQQ== --insecure
The Elastic Agent is currently in BETA and should not be used in production

Successfully enrolled the Elastic Agent.
Installation was successful and Elastic Agent is running.
```

and set the policy back to managed

```
curl --user elastic:changeme -X PUT localhost:5601/api/fleet/agent_policies/3bd07db0-67d0-11eb-b656-21ad68ebfa8a -H 'Content-Type: application/json' -d'{ "is_managed": true, "name": "xyz", "namespace": "default" }' -H 'kbn-xsrf: true'
{"item":{"id":"3bd07db0-67d0-11eb-b656-21ad68ebfa8a","name":"xyz","namespace":"default","is_managed":true,"revision":5,"updated_at":"2021-02-05T17:44:18.757Z","updated_by":"elastic","package_policies":[]}}
```

with all the restrictions that entails (cannot unenroll, reassign, etc)

```
curl --user elastic:changeme -X PUT 'http://localhost:5601/api/fleet/agents/8169f0a0-67d9-11eb-80f2-73dd45e7318e/reassign'   -X 'PUT'  -H 'kbn-xsrf: abc'  -H 'Content-Type: application/json'   --data-raw '{"policy_id":"729f8440-67cf-11eb-b656-21ad68ebfa8a"}'   
{
  "statusCode": 400,
  "error": "Bad Request",
  "message": "Cannot reassign an agent from managed agent policy 3bd07db0-67d0-11eb-b656-21ad68ebfa8a"
}
```

</details>

### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2021-02-09 08:54:51 -05:00