* Intercept installation errors and add meta info.
* Adjust mock.
* Catch errors in all steps of install/upgrade.
* Adjust handler for direct package upload.
* Don't throw not-found errors on assets during rollback.
* Correctly catch errors from _installPackage()
* Propagate error from installResult in bulk install case.
* Add tests for rollback.
* Remove unused code.
* Skipping test that doesn't test what it says.
* Fix and reenable test.
## Summary
- fcbc9d9 Rename `force` param to `revoke` for `/agents/{agent_id}/unenroll` & `/agents/bulk_unenroll`
- 03b9b90 Add new `force` param
See https://github.com/elastic/kibana/issues/96873 for background
<table>
<thead>
<tr>
<td rowspan="2"></td><td colspan="2">Unenroll Agent</td><td rowspan="2">Revoke API Keys</td>
</tr>
<tr>
<td>Regular</td><td>Hosted</td></td>
</tr>
</thead>
<tr><td colspan="4"><strong>Rename <code>force</code> to <code>revoke</code></strong></td></tr>
<tr><td>Current <code>force=false|undefined</code></td><td>✅</td><td>❌</td><td>❌</td></tr>
<tr><td>Proposed <code>revoke=false|undefined</code></td><td>✅</td><td>❌</td><td>❌</td></tr>
<tr><td>Current <code>force=true</code></td><td>✅</td><td>❌</td><td>✅</td></tr>
<tr><td>Proposed <code>revoke=true</code></td><td>✅</td><td>❌</td><td>✅</td></tr>
<tr><td colspan="4"><strong>Change <code>force</code> param </strong></td></tr>
<tr><td>Proposed <code>force=false|undefined</code></td><td>✅</td><td>❌</td><td>❌</td></tr>
<tr><td>Proposed <code>force=true</code></td><td>✅</td><td>✅</td><td>❌</td></tr>
<tr><td>Proposed <code>force=true</code> & <code>revoke=true</code></td><td>✅</td><td>✅</td><td>✅</td></tr>
</table>
### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
### Changes required for consumers
Any call to `/agents/{agent_id}/unenroll` & `/agents/bulk_unenroll` which passes the `force` param should change to `revoke` to maintain the current behavior.
## Summary
Can now pass a `force=true` parameter to add & remove integrations on hosted policies as originally intended [1] & [2]
* Add `force` param for `POST` `/api/fleet/package_policies` & `/api/fleet/package_policies/delete` to a policy. Update tests to confirm
* Not strictly required, but "while I was in there"
* Updated a few places to throw `IngestManagerError` vs `Error` for `400` response vs `500`. Updated tests.
* removed a few unnecessary `await`s of sync function
[1] https://github.com/elastic/kibana/issues/92426#issuecomment-785092670
[2] https://github.com/elastic/kibana/issues/90445
### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* [Fleet] Install security_rule assets as saved objects
* Add security-rule to update_assets.ts
* Update UUIDs for security_rule asset
* Change .type to match the saved object type not the asset type
* Add saved object mapping for security-rule
* Make SO non-hidden
* Fix SO mapping for security-rule
* Make security-rule a non-hidden asset
* Use client from branch
* Get type checking working in core
* Fix types in other plugins
* Update client types + remove type errors from core
* migrate Task Manager Elasticsearch typing from legacy library to client library
* use SortOrder instead o string in alerts
* Update client types + fix core type issues
* fix maps ts errors
* Update Lens types
* Convert Search Profiler body from a string to an object to conform to SearchRequest type.
* Fix SOT types
* Fix/mute Security/Spaces plugins type errors.
* Fix bootstrap types
* Fix painless_lab
* corrected es typing in Event Log
* Use new types from client for inferred search responses
* Latest type defs
* Integrate latest type defs for APM/UX
* fix core errors
* fix telemetry errors
* fix canvas errors
* fix data_enhanced errors
* fix event_log errors
* mute lens errors
* fix or mute maps errors
* fix reporting errors
* fix security errors
* mute errors in task_manager
* fix errors in telemetry_collection_xpack
* fix errors in data plugins
* fix errors in alerts
* mute errors in index_management
* fix task_manager errors
* mute or fix lens errors
* fix upgrade_assistant errors
* fix or mute errors in index_lifecycle_management
* fix discover errors
* fix core tests
* ML changes
* fix core type errors
* mute error in kbn-es-archiver
* fix error in data plugin
* fix error in telemetry plugin
* fix error in discover
* fix discover errors
* fix errors in task_manager
* fix security errors
* fix wrong conflict resolution
* address errors with upstream code
* update deps to the last commit
* remove outdated comments
* fix core errors
* fix errors after update
* adding more expect errors to ML
* pull the lastest changes
* fix core errors
* fix errors in infra plugin
* fix errors in uptime plugin
* fix errors in ml
* fix errors in xpack telemetry
* fix or mute errors in transform
* fix errors in upgrade assistant
* fix or mute fleet errors
* start fixing apm errors
* fix errors in osquery
* fix telemetry tests
* core cleanup
* fix asMutableArray imports
* cleanup
* data_enhanced cleanup
* cleanup events_log
* cleaup
* fix error in kbn-es-archiver
* fix errors in kbn-es-archiver
* fix errors in kbn-es-archiver
* fix ES typings for Hit
* fix SO
* fix actions plugin
* fix fleet
* fix maps
* fix stack_alerts
* fix eslint problems
* fix event_log unit tests
* fix failures in data_enhanced tests
* fix test failure in kbn-es-archiver
* fix test failures in index_pattern_management
* fixing ML test
* remove outdated comment in kbn-es-archiver
* fix error type in ml
* fix eslint errors in osquery plugin
* fix runtime error in infra plugin
* revert changes to event_log cluser exist check
* fix eslint error in osquery
* fixing ML endpoint argument types
* fx types
* Update api-extractor docs
* attempt fix for ese test
* Fix lint error
* Fix types for ts refs
* Fix data_enhanced unit test
* fix lens types
* generate docs
* Fix a number of type issues in monitoring and ml
* fix triggers_actions_ui
* Fix ILM functional test
* Put search.d.ts typings back
* fix data plugin
* Update typings in typings/elasticsearch
* Update snapshots
* mute errors in task_manager
* mute fleet errors
* lens. remove unnecessary ts-expect-errors
* fix errors in stack_alerts
* mute errors in osquery
* fix errors in security_solution
* fix errors in lists
* fix errors in cases
* mute errors in search_examples
* use KibanaClient to enforce promise-based API
* fix errors in test/ folder
* update comment
* fix errors in x-pack/test folder
* fix errors in ml plugin
* fix optional fields in ml api_integartoon tests
* fix another casting problem in ml tests
* fix another ml test failure
* fix fleet problem after conflict resolution
* rollback changes in security_solution. trying to fix test
* Update type for discover rows
* uncomment runtime_mappings as its outdated
* address comments from Wylie
* remove eslint error due to any
* mute error due to incompatibility
* Apply suggestions from code review
Co-authored-by: John Schulz <github.com@jfsiii.org>
* fix type error in lens tests
* Update x-pack/plugins/upgrade_assistant/server/lib/reindexing/reindex_service.ts
Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com>
* Update x-pack/plugins/upgrade_assistant/server/lib/reindexing/reindex_service.test.ts
Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com>
* update deps
* fix errors in core types
* fix errors for the new elastic/elasticsearch version
* remove unused type
* remove unnecessary manual type cast and put optional chaining back
* ML: mute Datafeed is missing indices_options
* Apply suggestions from code review
Co-authored-by: Josh Dover <1813008+joshdover@users.noreply.github.com>
* use canary pacakge instead of git commit
Co-authored-by: Josh Dover <me@joshdover.com>
Co-authored-by: Josh Dover <1813008+joshdover@users.noreply.github.com>
Co-authored-by: Gidi Meir Morris <github@gidi.io>
Co-authored-by: Nathan Reese <reese.nathan@gmail.com>
Co-authored-by: Wylie Conlon <wylieconlon@gmail.com>
Co-authored-by: CJ Cenizal <cj@cenizal.com>
Co-authored-by: Aleh Zasypkin <aleh.zasypkin@gmail.com>
Co-authored-by: Dario Gieselaar <dario.gieselaar@elastic.co>
Co-authored-by: restrry <restrry@gmail.com>
Co-authored-by: James Gowdy <jgowdy@elastic.co>
Co-authored-by: John Schulz <github.com@jfsiii.org>
Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com>
* Add force option to DELETE package endpoint.
* Add integration test.
* Adjust openapi spec.
* Run EPM tests before fleet setup tests.
* Run package delete tests first in EPM tests
* Add data plugin to server app context
* First attempt at switching to indexPatternService for EPM index pattern creation & deletion, instead of interacting directly with index pattern SOs
* Simplify bulk install package, remove upgradePackage() method in favor of consolidating with installPackage(), use installPackage() for bulk install instead
* Update tests
* Change cache logging of objects to trace level
* Install index patterns as a post-package installation operation, for bulk package installation, install index pattern only if one or more packages are actually new installs, add debug logging
* Allow getAsSavedObjectBody to return non-scripted fields when allowNoIndex is true
* Allow `getFieldsForWildcard` to return fields saved on index pattern when allowNoIndices is true
* Bring back passing custom ID for index pattern SO
* Fix tests
* Revert "Merge branch 'index-pattern/allow-no-index' into epm/missing-index-patterns"
This reverts commit 8e712e9c00, reversing
changes made to af0fb0eaa8.
* Allow getAsSavedObjectBody to return non-scripted fields when allowNoIndex is true
(cherry picked from commit 69b93da180)
* Update API docs
* Run post-install ops for install by upload too
* Remove allowedInstallTypes param
* Adjust force install conditions
* Revert "Update API docs"
This reverts commit b9770fdc56.
* Revert "Allow getAsSavedObjectBody to return non-scripted fields when allowNoIndex is true"
This reverts commit afc91ce32f.
* Go back to using SO client for index patterns :(
* Stringify attributes again for SO client saving
* Fix condition for reinstall same pkg version
* Remove unused type
* Adjust comment
* Update snapshot
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Problem
While working on changes for bulk reassign https://github.com/elastic/kibana/issues/90437, I found that the server has a runtime error and returns a 500 if given an invalid or missing id.
<details><summary>server error stack trace</summary>
```
│ proc [kibana] server log [12:21:48.953] [error][fleet][plugins] TypeError: Cannot read property 'policy_revision_idx' of undefined
│ proc [kibana] at map (/Users/jfsiii/work/kibana/x-pack/plugins/fleet/server/services/agents/helpers.ts:15:34)
│ proc [kibana] at Array.map (<anonymous>)
│ proc [kibana] at getAgents (/Users/jfsiii/work/kibana/x-pack/plugins/fleet/server/services/agents/crud.ts:191:32)
│ proc [kibana] at runMicrotasks (<anonymous>)
│ proc [kibana] at processTicksAndRejections (internal/process/task_queues.js:93:5)
│ proc [kibana] at Object.reassignAgents (/Users/jfsiii/work/kibana/x-pack/plugins/fleet/server/services/agents/reassign.ts:91:9)
│ proc [kibana] at postBulkAgentsReassignHandler (/Users/jfsiii/work/kibana/x-pack/plugins/fleet/server/routes/agent/handlers.ts:314:21)
│ proc [kibana] at Router.handle (/Users/jfsiii/work/kibana/src/core/server/http/router/router.ts:272:30)
│ proc [kibana] at handler (/Users/jfsiii/work/kibana/src/core/server/http/router/router.ts:227:11)
│ proc [kibana] at exports.Manager.execute (/Users/jfsiii/work/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
│ proc [kibana] at Object.internals.handler (/Users/jfsiii/work/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)
│ proc [kibana] at exports.execute (/Users/jfsiii/work/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)
│ proc [kibana] at Request._lifecycle (/Users/jfsiii/work/kibana/node_modules/@hapi/hapi/lib/request.js:370:32)
│ proc [kibana] at Request._execute (/Users/jfsiii/work/kibana/node_modules/@hapi/hapi/lib/request.js:279:9)
```
</details>
<details><summary>see test added in this PR fail on master</summary>
```
1) Fleet Endpoints
reassign agent(s)
bulk reassign agents
should allow to reassign multiple agents by id -- some invalid:
Error: expected 200 "OK", got 500 "Internal Server Error"
```
</details>
## Root cause
Debugging runtime error in `searchHitToAgent` found some TS type mismatches for the ES values being returned. Perhaps from one or more of the recent changes to ES client & Fleet Server. Based on `test:jest` and `test:ftr`, it appears the possible types are `GetResponse` or `SearchResponse`, instead of only an `ESSearchHit`.
https://github.com/elastic/kibana/pull/94632/files#diff-254d0f427979efc3b442f78762302eb28fb9c8857df68ea04f8d411e052f939cL11
While a `.search` result will include return matched values, a `.get` or `.mget` will return a row for each input and a `found: boolean`. e.g. `{ _id: "does-not-exist", found: false }`. The error occurs when [`searchHitToAgent`](1702cf98f0/x-pack/plugins/fleet/server/services/agents/helpers.ts (L11)) is run on a get miss instead of a search hit.
## PR Changes
* Added a test to ensure it doesn't fail if invalid or missing IDs are given
* Moved the `bulk_reassign` tests to their own test section
* Filter out any missing results before calling `searchHitToAgent`, to match current behavior
* Consolidate repeated arguments into and code for getting agents into single [function](https://github.com/elastic/kibana/pull/94632/files#diff-f7377ed9ad56eaa8ea188b64e957e771ccc7a7652fd1eaf44251c25b930f8448R70-R87): and [TS type](https://github.com/elastic/kibana/pull/94632/files#diff-f7377ed9ad56eaa8ea188b64e957e771ccc7a7652fd1eaf44251c25b930f8448R61-R68)
* Rename some agent service functions to be more explicit (IMO) but behavior maintained. Same API names exported.
This moves toward the "one result (success or error) per given id" approach for https://github.com/elastic/kibana/issues/90437
### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
* Make package validation be based on types
* Add package info to cache after it is generated from ES storage
* Add logging around package info retrieval and when cache is set
* Add snapshot api integration test for uploaded package info
* Use the apache package for snapshot test instead
* Remove date field from snapshot
* Update docs
* Fix streams getting overridden
* Add back package field to data streams
* PR fixes
* Set all keyword and text fields for `index.query.default_field` setting
* Update tests and snapshots
* Fix test
* Add default field limit safeguard
* Add logging when beyond limit
* Update tests to mock app context (because I added logger usage)
* Update api integration test
* Rename consts
## Summary
- Make sure any agents requesting to be upgraded, are not enrolled in a managed policy.
- `force: true` will only bypass agent / kibana version checks. It will not bypass managed policy check. To workaround, the enrolled policy should be changed to unmanaged (`is_managed: false`) as we do with enroll, reassign, etc.
- Took more efficient approach to bulk actions. One `bulkGet` for N agents/policies vs N `get`s approach used for bulk reassignment of agents. See discussion in https://github.com/elastic/kibana/pull/88688/files#r568941761
- [x] API
- [ ] UI
- [x] tests
### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
### Manual tests
#### upgrade one
```
curl --location --request POST 'http://localhost:5601/api/fleet/agents/8d9748e0-6d52-11eb-8cbd-47e38cd1c8de/upgrade' --header 'kbn-xsrf: <string>' --header 'Content-Type: application/json' --header 'Authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==' --data-raw '{
"version": "8.0.0"
}'
{"statusCode":400,"error":"Bad Request","message":"Cannot upgrade agent 8d9748e0-6d52-11eb-8cbd-47e38cd1c8de in managed policy bf319100-6d50-11eb-8859-15a87f509a99"}
```
```
curl --location --request POST 'http://localhost:5601/api/fleet/agents/8d9748e0-6d52-11eb-8cbd-47e38cd1c8de/upgrade' --header 'kbn-xsrf: <string>' --header 'Content-Type: application/json' --header 'Authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==' --data-raw '{
"version": "8.0.0", "force": true
}'
{"statusCode":400,"error":"Bad Request","message":"Cannot upgrade agent 8d9748e0-6d52-11eb-8cbd-47e38cd1c8de in managed policy bf319100-6d50-11eb-8859-15a87f509a99"}
```
#### bulk upgrade
```
curl --location --request POST 'http://localhost:5601/api/fleet/agents/bulk_upgrade' --header 'kbn-xsrf: <string>' --header 'Content-Type: application/json' --header 'Authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==' --data-raw '{
"version": "8.0.0",
"agents": [
"8d9748e0-6d52-11eb-8cbd-47e38cd1c8de"
]
}'
{}
```
```
curl --location --request POST 'http://localhost:5601/api/fleet/agents/bulk_upgrade' --header 'kbn-xsrf: <string>' --header 'Content-Type: application/json' --header 'Authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==' --data-raw '{
"version": "8.0.0",
"agents": [
"8d9748e0-6d52-11eb-8cbd-47e38cd1c8de"
], "force": true
}'
{"statusCode":400,"error":"Bad Request","message":"Cannot update agent in managed policy bf319100-6d50-11eb-8859-15a87f509a99"}```
```
## Summary
Add guard to `/agents/enroll` API preventing agents from enrolling in managed policies
closes#90435
- [x] No Agents can be enrolled into this policy by the user.
- [x] The install & enroll commands should print an error to the console if the enroll command fails (due to being a managed policy or any other reason)
#### So how do you associate an agent with a managed policy?
Enroll in an unmanaged policy then set that policy to managed.
We don't restrict the agent policy, only what other things (agents, integrations) can do if they're associated with a managed policy.
A _force flag_ has been mentioned for some other actions. It might work here as well, but I'd like to handle discussion & implementation of those later.
### Manual testing
<details><summary>Prevent enroll for managed policies</summary>
1. Created a managed agent policy
```
curl --user elastic:changeme -X POST localhost:5601/api/fleet/agent_policies -H 'Content-Type: application/json' -d'{ "name": "User created MANAGED", "namespace": "default", "is_managed": true}' -H 'kbn-xsrf: true'
{"item":{"id":"3bd07db0-67d0-11eb-b656-21ad68ebfa8a","name":"User created MANAGED","namespace":"default","is_managed":true,"revision":1,"updated_at":"2021-02-05T16:36:01.931Z","updated_by":"elastic"}}
```
2. Try `install` command show in the UI
```
sudo ./elastic-agent install -f --kibana-url=http://localhost:5601 --enrollment-token=WmcwTWMzY0IzWlBUUWJJUjZqRDA6UGRZelVlaS1STml1cVdjSUVwSkJRQQ== --insecure
Password:
The Elastic Agent is currently in BETA and should not be used in production
Error: fail to enroll: fail to execute request to Kibana: Status code: 400, Kibana returned an error: Bad Request, message: Cannot enroll in managed policy 3bd07db0-67d0-11eb-b656-21ad68ebfa8a
Error: enroll command failed with exit code: 1
```
3. Observe `Cannot enroll in managed policy 3bd07db0-67d0-11eb-b656-21ad68ebfa8a` error
4. Try `enroll` instead:
```
sudo ./elastic-agent enroll http://localhost:5601 WmcwTWMzY0IzWlBUUWJJUjZqRDA6UGRZelVlaS1STml1cVdjSUVwSkJRQQ== --insecure
The Elastic Agent is currently in BETA and should not be used in production
This will replace your current settings. Do you want to continue? [Y/n]:
Error: fail to enroll: fail to execute request to Kibana: Status code: 400, Kibana returned an error: Bad Request, message: Cannot enroll in managed policy 3bd07db0-67d0-11eb-b656-21ad68ebfa8a
```
5. Observe same `Cannot enroll in managed policy 3bd07db0-67d0-11eb-b656-21ad68ebfa8a` error
</details>
<details><summary>Enroll in unmanaged policy, then update it to managed</summary>
Agent policies are `is_managed: false` by default, or we can update the policy to `is_managed: false`, like:
```
curl --user elastic:changeme -X PUT localhost:5601/api/fleet/agent_policies/3bd07db0-67d0-11eb-b656-21ad68ebfa8a -H 'Content-Type: application/json' -d'{ "is_managed": false, "name": "xyz", "namespace": "default" }' -H 'kbn-xsrf: true'
{"item":{"id":"3bd07db0-67d0-11eb-b656-21ad68ebfa8a","name":"xyz","namespace":"default","is_managed":false,"revision":4,"updated_at":"2021-02-05T17:42:05.610Z","updated_by":"elastic","package_policies":[]}}
```
then enroll
```
sudo ./elastic-agent install -f --kibana-url=http://localhost:5601 --enrollment-token=WmcwTWMzY0IzWlBUUWJJUjZqRDA6UGRZelVlaS1STml1cVdjSUVwSkJRQQ== --insecure
The Elastic Agent is currently in BETA and should not be used in production
Successfully enrolled the Elastic Agent.
Installation was successful and Elastic Agent is running.
```
and set the policy back to managed
```
curl --user elastic:changeme -X PUT localhost:5601/api/fleet/agent_policies/3bd07db0-67d0-11eb-b656-21ad68ebfa8a -H 'Content-Type: application/json' -d'{ "is_managed": true, "name": "xyz", "namespace": "default" }' -H 'kbn-xsrf: true'
{"item":{"id":"3bd07db0-67d0-11eb-b656-21ad68ebfa8a","name":"xyz","namespace":"default","is_managed":true,"revision":5,"updated_at":"2021-02-05T17:44:18.757Z","updated_by":"elastic","package_policies":[]}}
```
with all the restrictions that entails (cannot unenroll, reassign, etc)
```
curl --user elastic:changeme -X PUT 'http://localhost:5601/api/fleet/agents/8169f0a0-67d9-11eb-80f2-73dd45e7318e/reassign' -X 'PUT' -H 'kbn-xsrf: abc' -H 'Content-Type: application/json' --data-raw '{"policy_id":"729f8440-67cf-11eb-b656-21ad68ebfa8a"}'
{
"statusCode": 400,
"error": "Bad Request",
"message": "Cannot reassign an agent from managed agent policy 3bd07db0-67d0-11eb-b656-21ad68ebfa8a"
}
```
</details>
### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
* Explicitly generate ES index pattern name.
* Adjust tests.
* Adjust and reenable tests.
* Set template priority based on dataset_is_prefix
* Refactor indexPatternName -> templateIndexPattern
* Add unit tests.
* Use more realistic index pattern in test.
* Fix unit test.
* Add unit test for installTemplate().
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Introduces the concept of a managed agent policy. Resolves most of the acceptance criteria from #76843. Remaining to be done in follow up PRs
- [x] Define hosted Agent Policy concept in Fleet.
- [x] Flag in policy? **_yes, added `is_managed: boolean`_ in agent policy SO**
- [x] Should not built only for cloud, an admin should be able to set theses restrictions.
- [x] We should have an API to configure it _**Can `POST` and `PUT` to `/api/fleet/agent_policies/{policy_id}`**_
- [x] Integration should be editable, we expect integration author to do the right thing and limit what can be edited.
- [x] Research if we can ensure the right behavior of Hosted Agent policy and restrict the super user.
- [ ] Capabilities restrictions
- [ ] An Agent enrolled in an Hosted Agent policy should not be able to be upgraded.
- [x] An Agent enrolled in an Hosted Agent policy should not be able to be unenrolled.
- [ ] No Agents cannot be enrolled into this policy by the user.
- Hide the enrollment key?
- Need to figure out the workflow.
- [x] An Agent enrolled in an Hosted Agent policy should not be able to be reassigned to a different configuration.
- [x] As a user I should be prevented to do theses action. _**No user-level checks. Only Agent Policy. No UI changes, but API errors are shown for failed actions like reassigning**_
- [x] As an API user I should receive error messages.
- [x] If making a single "flag" is easier/faster let's do it. _**Currently single `is_managed` property on agent policy SO.**_
Checks are implemented in service layer (is agent enrolled in a managed policy?)
No UI-specific changes added but UI is affected because HTTP requests (like `api/fleet/agents/{agentId}/reassign`) can fail. See screenshots below.
Tests at service (`yarn test:jest`) and http (`yarn test ftr`) layers for each of create policy, update policy, unenroll agent, and reassign agent
Bulk actions currently filter out restricted items. A follow-up PR will change them to throw an error and cause the request to fail.
## Managed Policy
Can create (`POST`) and update (`PUT`) an agent policy with an `is_managed` property. Each new saved object will have an `is_managed` property (default `false`)
<details><summary>HTTP commands</summary>
#### Create (`is_managed: false` by default)
```
curl --user elastic:changeme -X POST localhost:5601/api/fleet/agent_policies -H 'Content-Type: application/json' -d'{ "name": "User created policy", "namespace": "default"}' -H 'kbn-xsrf: true'
{"item":{"id":"edc236a0-5cbb-11eb-ab2c-0134aecb4ce8","name":"User created policy","namespace":"default","is_managed":false,"revision":1,"updated_at":"2021-01-22T14:12:58.250Z","updated_by":"elastic"}}
```
#### Create with `is_managed: true`
```
curl --user elastic:changeme -X POST localhost:5601/api/fleet/agent_policies -H 'Content-Type: application/json' -d'{ "name": "User created policy", "namespace": "default"}' -H 'kbn-xsrf: true'
{"item":{"id":"67c785b0-662e-11eb-bf6b-4790dc0178c0","name":"User created policy","namespace":"default","is_managed":false,"revision":1,"updated_at":"2021-02-03T14:45:06.059Z","updated_by":"elastic"}}
```
#### Update with `is_managed: true`
```
curl --user elastic:changeme -X PUT -H 'Content-Type: application/json' -H 'kbn-xsrf: 1234' localhost:5601/api/fleet/agent_policies/67c785b0-662e-11eb-bf6b-4790dc0178c0 -d '{ "name":"User created policy","namespace":"default","is_managed":true }'
{"item":{"id":"67c785b0-662e-11eb-bf6b-4790dc0178c0","name":"User created policy","namespace":"default","is_managed":true,"revision":2,"updated_at":"2021-02-03T14:47:28.471Z","updated_by":"elastic","package_policies":[]}}
```
</details>
## Enroll behavior
is not changed/addressed in this PR. Agents can still be enrolled in managed policies
## Unenroll Agent from managed policy behavior
#### Enrolled in managed agent policy, cannot be unenrolled
```
curl --user elastic:changeme -X POST http://localhost:5601/api/fleet/agents/441d4a40-6710-11eb-8f57-db14e8e41cff/unenroll -H 'kbn-xsrf: 1234' | jq
{
"statusCode": 400,
"error": "Bad Request",
"message": "Cannot unenroll 441d4a40-6710-11eb-8f57-db14e8e41cff from a managed agent policy af9b4970-6701-11eb-b55a-899b78cb64da"
}
```
<details><summary>Screenshots for managed & unmanaged policies</summary>
#### Enrolled in managed agent policy, cannot be unenrolled
<img width="1931" alt="Screen Shot 2021-01-19 at 1 22 53 PM" src="https://user-images.githubusercontent.com/57655/105081614-67d05980-5a60-11eb-8faa-07e4e722a5b5.png">
<img width="1199" alt="Screen Shot 2021-01-19 at 1 30 26 PM" src="https://user-images.githubusercontent.com/57655/105081617-67d05980-5a60-11eb-9099-832dc6e04eca.png">
<img width="1971" alt="Screen Shot 2021-01-19 at 1 30 42 PM" src="https://user-images.githubusercontent.com/57655/105081618-67d05980-5a60-11eb-9a84-b80b6295ba19.png">
#### Enrolled agent policy is not managed, agent can be unenrolled<img width="1917" alt="Screen Shot 2021-01-19 at 1 44 12 PM" src="https://user-images.githubusercontent.com/57655/105081951-e3caa180-5a60-11eb-9308-7741b8986e8e.png">
<img width="2183" alt="Screen Shot 2021-01-19 at 1 44 19 PM" src="https://user-images.githubusercontent.com/57655/105081952-e3caa180-5a60-11eb-9833-1c721be0a107.png">
</details>
## Reassign agent
#### No agent can be reassigned to a managed policy
```
curl --user elastic:changeme -X 'PUT' 'http://localhost:5601/api/fleet/agents/482760d0-6710-11eb-8f57-db14e8e41cff/reassign' -H 'kbn-xsrf: xxx' -H 'Content-Type: application/json' -d '{"policy_id":"af9b4970-6701-11eb-b55a-899b78cb64da"}'
{
"statusCode": 400,
"error": "Bad Request",
"message": "Cannot reassign an agent to managed agent policy 94129590-6707-11eb-b55a-899b78cb64da"
}
```
<details><summary>Screenshots</summary>
<img width="1350" alt="Screen Shot 2021-02-04 at 2 14 51 PM" src="https://user-images.githubusercontent.com/57655/106943490-8044a300-66f3-11eb-9d2c-4b1ceef2e783.png">
</details>
#### Enrolled in managed agent policy, cannot be reassigned
```
curl --user elastic:changeme -X 'PUT' 'http://localhost:5601/api/fleet/agents/482760d0-6710-11eb-8f57-db14e8e41cff/reassign' -H 'kbn-xsrf: xxx' -H 'Content-Type: application/json' -d '{"policy_id":"af9b4970-6701-11eb-b55a-899b78cb64da"}'
{
"statusCode": 400,
"error": "Bad Request",
"message": "Cannot reassign an agent from managed agent policy 94129590-6707-11eb-b55a-899b78cb64da"
}
```
<details><summary>Screenshots</summary>
<img width="1364" alt="Screen Shot 2021-01-19 at 2 58 38 PM" src="https://user-images.githubusercontent.com/57655/105086737-72dab800-5a67-11eb-8f5e-93cd7768b914.png">
<img width="1367" alt="Screen Shot 2021-01-19 at 2 58 44 PM" src="https://user-images.githubusercontent.com/57655/105086740-73734e80-5a67-11eb-8ef9-9c7005a0a4ea.png">
<img width="623" alt="Screen Shot 2021-01-19 at 2 59 27 PM" src="https://user-images.githubusercontent.com/57655/105086741-740be500-5a67-11eb-8fc2-721f8b5d178a.png">
</details>
#### Enrolled agent policy is unmanaged, agent can be reassigned to another unmanaged policy
<details><summary>Screenshots</summary>
<img width="1368" alt="Screen Shot 2021-01-19 at 3 00 01 PM" src="https://user-images.githubusercontent.com/57655/105086754-78d09900-5a67-11eb-86a5-9e3ac02d6e1f.png">
<img width="1363" alt="Screen Shot 2021-01-19 at 3 00 08 PM" src="https://user-images.githubusercontent.com/57655/105086761-7a01c600-5a67-11eb-991d-acf994e2a393.png">
<img width="625" alt="Screen Shot 2021-01-19 at 3 00 46 PM" src="https://user-images.githubusercontent.com/57655/105086764-7a9a5c80-5a67-11eb-8290-e79648d01579.png">
</details>
### Checklist
Delete any items that are not applicable to this PR.
- [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)
- [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
* Add API integration tests for data streams list, including one that is expected to fail due to reliance on number of backing indices
* Use ES data streams API as source of truth for list of data streams, and only query against backing indices afterwards
* Get package name from data stream meta info
* Increate retry timeout
* Move initial info requests inside Promise.all
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Update data streams mappings directly instead of querying for backing indices, update integration tests to test with multiple namespaces
* Add flag to only update mappings of the current write index
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
