* Use js-yaml.safeDump() to escape string values.
* Add unit test.
* Explicitly check for YAML special characters.
* Remove unnecessary imports.
* Use RegExp.prototype.test() for speed.
## Summary
- Make sure any agents requesting to be upgraded, are not enrolled in a managed policy.
- `force: true` will only bypass agent / kibana version checks. It will not bypass managed policy check. To workaround, the enrolled policy should be changed to unmanaged (`is_managed: false`) as we do with enroll, reassign, etc.
- Took more efficient approach to bulk actions. One `bulkGet` for N agents/policies vs N `get`s approach used for bulk reassignment of agents. See discussion in https://github.com/elastic/kibana/pull/88688/files#r568941761
- [x] API
- [ ] UI
- [x] tests
### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
### Manual tests
#### upgrade one
```
curl --location --request POST 'http://localhost:5601/api/fleet/agents/8d9748e0-6d52-11eb-8cbd-47e38cd1c8de/upgrade' --header 'kbn-xsrf: <string>' --header 'Content-Type: application/json' --header 'Authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==' --data-raw '{
"version": "8.0.0"
}'
{"statusCode":400,"error":"Bad Request","message":"Cannot upgrade agent 8d9748e0-6d52-11eb-8cbd-47e38cd1c8de in managed policy bf319100-6d50-11eb-8859-15a87f509a99"}
```
```
curl --location --request POST 'http://localhost:5601/api/fleet/agents/8d9748e0-6d52-11eb-8cbd-47e38cd1c8de/upgrade' --header 'kbn-xsrf: <string>' --header 'Content-Type: application/json' --header 'Authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==' --data-raw '{
"version": "8.0.0", "force": true
}'
{"statusCode":400,"error":"Bad Request","message":"Cannot upgrade agent 8d9748e0-6d52-11eb-8cbd-47e38cd1c8de in managed policy bf319100-6d50-11eb-8859-15a87f509a99"}
```
#### bulk upgrade
```
curl --location --request POST 'http://localhost:5601/api/fleet/agents/bulk_upgrade' --header 'kbn-xsrf: <string>' --header 'Content-Type: application/json' --header 'Authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==' --data-raw '{
"version": "8.0.0",
"agents": [
"8d9748e0-6d52-11eb-8cbd-47e38cd1c8de"
]
}'
{}
```
```
curl --location --request POST 'http://localhost:5601/api/fleet/agents/bulk_upgrade' --header 'kbn-xsrf: <string>' --header 'Content-Type: application/json' --header 'Authorization: Basic ZWxhc3RpYzpjaGFuZ2VtZQ==' --data-raw '{
"version": "8.0.0",
"agents": [
"8d9748e0-6d52-11eb-8cbd-47e38cd1c8de"
], "force": true
}'
{"statusCode":400,"error":"Bad Request","message":"Cannot update agent in managed policy bf319100-6d50-11eb-8859-15a87f509a99"}```
```
* exclude all the plugins from src/plugins
* move all the used fixtures to discover
* remove src/fixtures alias
* remove unused fixtures
* cleanup x-pack/tsconfig.json
* dont compile apm/scripts
* fix tests
* dont include infra in xpack/tsconfig.json
* update list of includes
The staging registry is used in Kibana builds which are not built of the master branch or release version. This means, any build ending with `-SNAPSHOT` not the master branch will use the staging registry.
Closes https://github.com/elastic/kibana/issues/90131
Co-authored-by: Jen Huang <its.jenetic@gmail.com>
## Summary
Add guard to `/agents/enroll` API preventing agents from enrolling in managed policies
closes#90435
- [x] No Agents can be enrolled into this policy by the user.
- [x] The install & enroll commands should print an error to the console if the enroll command fails (due to being a managed policy or any other reason)
#### So how do you associate an agent with a managed policy?
Enroll in an unmanaged policy then set that policy to managed.
We don't restrict the agent policy, only what other things (agents, integrations) can do if they're associated with a managed policy.
A _force flag_ has been mentioned for some other actions. It might work here as well, but I'd like to handle discussion & implementation of those later.
### Manual testing
<details><summary>Prevent enroll for managed policies</summary>
1. Created a managed agent policy
```
curl --user elastic:changeme -X POST localhost:5601/api/fleet/agent_policies -H 'Content-Type: application/json' -d'{ "name": "User created MANAGED", "namespace": "default", "is_managed": true}' -H 'kbn-xsrf: true'
{"item":{"id":"3bd07db0-67d0-11eb-b656-21ad68ebfa8a","name":"User created MANAGED","namespace":"default","is_managed":true,"revision":1,"updated_at":"2021-02-05T16:36:01.931Z","updated_by":"elastic"}}
```
2. Try `install` command show in the UI
```
sudo ./elastic-agent install -f --kibana-url=http://localhost:5601 --enrollment-token=WmcwTWMzY0IzWlBUUWJJUjZqRDA6UGRZelVlaS1STml1cVdjSUVwSkJRQQ== --insecure
Password:
The Elastic Agent is currently in BETA and should not be used in production
Error: fail to enroll: fail to execute request to Kibana: Status code: 400, Kibana returned an error: Bad Request, message: Cannot enroll in managed policy 3bd07db0-67d0-11eb-b656-21ad68ebfa8a
Error: enroll command failed with exit code: 1
```
3. Observe `Cannot enroll in managed policy 3bd07db0-67d0-11eb-b656-21ad68ebfa8a` error
4. Try `enroll` instead:
```
sudo ./elastic-agent enroll http://localhost:5601 WmcwTWMzY0IzWlBUUWJJUjZqRDA6UGRZelVlaS1STml1cVdjSUVwSkJRQQ== --insecure
The Elastic Agent is currently in BETA and should not be used in production
This will replace your current settings. Do you want to continue? [Y/n]:
Error: fail to enroll: fail to execute request to Kibana: Status code: 400, Kibana returned an error: Bad Request, message: Cannot enroll in managed policy 3bd07db0-67d0-11eb-b656-21ad68ebfa8a
```
5. Observe same `Cannot enroll in managed policy 3bd07db0-67d0-11eb-b656-21ad68ebfa8a` error
</details>
<details><summary>Enroll in unmanaged policy, then update it to managed</summary>
Agent policies are `is_managed: false` by default, or we can update the policy to `is_managed: false`, like:
```
curl --user elastic:changeme -X PUT localhost:5601/api/fleet/agent_policies/3bd07db0-67d0-11eb-b656-21ad68ebfa8a -H 'Content-Type: application/json' -d'{ "is_managed": false, "name": "xyz", "namespace": "default" }' -H 'kbn-xsrf: true'
{"item":{"id":"3bd07db0-67d0-11eb-b656-21ad68ebfa8a","name":"xyz","namespace":"default","is_managed":false,"revision":4,"updated_at":"2021-02-05T17:42:05.610Z","updated_by":"elastic","package_policies":[]}}
```
then enroll
```
sudo ./elastic-agent install -f --kibana-url=http://localhost:5601 --enrollment-token=WmcwTWMzY0IzWlBUUWJJUjZqRDA6UGRZelVlaS1STml1cVdjSUVwSkJRQQ== --insecure
The Elastic Agent is currently in BETA and should not be used in production
Successfully enrolled the Elastic Agent.
Installation was successful and Elastic Agent is running.
```
and set the policy back to managed
```
curl --user elastic:changeme -X PUT localhost:5601/api/fleet/agent_policies/3bd07db0-67d0-11eb-b656-21ad68ebfa8a -H 'Content-Type: application/json' -d'{ "is_managed": true, "name": "xyz", "namespace": "default" }' -H 'kbn-xsrf: true'
{"item":{"id":"3bd07db0-67d0-11eb-b656-21ad68ebfa8a","name":"xyz","namespace":"default","is_managed":true,"revision":5,"updated_at":"2021-02-05T17:44:18.757Z","updated_by":"elastic","package_policies":[]}}
```
with all the restrictions that entails (cannot unenroll, reassign, etc)
```
curl --user elastic:changeme -X PUT 'http://localhost:5601/api/fleet/agents/8169f0a0-67d9-11eb-80f2-73dd45e7318e/reassign' -X 'PUT' -H 'kbn-xsrf: abc' -H 'Content-Type: application/json' --data-raw '{"policy_id":"729f8440-67cf-11eb-b656-21ad68ebfa8a"}'
{
"statusCode": 400,
"error": "Bad Request",
"message": "Cannot reassign an agent from managed agent policy 3bd07db0-67d0-11eb-b656-21ad68ebfa8a"
}
```
</details>
### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
* Explicitly generate ES index pattern name.
* Adjust tests.
* Adjust and reenable tests.
* Set template priority based on dataset_is_prefix
* Refactor indexPatternName -> templateIndexPattern
* Add unit tests.
* Use more realistic index pattern in test.
* Fix unit test.
* Add unit test for installTemplate().
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* first pass
* migrate more plugins
* migrate yet more plugins
* more oss plugins
* fix test file
* change Plugin signature on the client-side too
* fix test types
* migrate OSS client-side plugins
* migrate OSS client-side test plugins
* migrate xpack client-side plugins
* revert fix attempt on fleet plugin
* fix presentation start signature
* fix yet another signature
* add warnings for server-side async plugins in dev mode
* remove unused import
* fix isPromise
* Add client-side deprecations
* update migration examples
* update generated doc
* fix xpack unit tests
* nit
* (will be reverted) explicitly await for license to be ready in the auth hook
* Revert "(will be reverted) explicitly await for license to be ready in the auth hook"
This reverts commit fdf73feb
* restore await on on promise contracts
* Revert "(will be reverted) explicitly await for license to be ready in the auth hook"
This reverts commit fdf73feb
* Revert "restore await on on promise contracts"
This reverts commit c5f2fe51
* add delay before starting tests in FTR
* update deprecation ts doc
* add explicit contract for monitoring setup
* migrate monitoring plugin to sync
* change plugin timeout to 10sec
* use delay instead of silence
* Convert user values back to string after yaml template compilation if they were strings originally
* Add better test cases and adjust patch
* Fix when field is undefined
* Handle array of strings too
## Summary
Introduces the concept of a managed agent policy. Resolves most of the acceptance criteria from #76843. Remaining to be done in follow up PRs
- [x] Define hosted Agent Policy concept in Fleet.
- [x] Flag in policy? **_yes, added `is_managed: boolean`_ in agent policy SO**
- [x] Should not built only for cloud, an admin should be able to set theses restrictions.
- [x] We should have an API to configure it _**Can `POST` and `PUT` to `/api/fleet/agent_policies/{policy_id}`**_
- [x] Integration should be editable, we expect integration author to do the right thing and limit what can be edited.
- [x] Research if we can ensure the right behavior of Hosted Agent policy and restrict the super user.
- [ ] Capabilities restrictions
- [ ] An Agent enrolled in an Hosted Agent policy should not be able to be upgraded.
- [x] An Agent enrolled in an Hosted Agent policy should not be able to be unenrolled.
- [ ] No Agents cannot be enrolled into this policy by the user.
- Hide the enrollment key?
- Need to figure out the workflow.
- [x] An Agent enrolled in an Hosted Agent policy should not be able to be reassigned to a different configuration.
- [x] As a user I should be prevented to do theses action. _**No user-level checks. Only Agent Policy. No UI changes, but API errors are shown for failed actions like reassigning**_
- [x] As an API user I should receive error messages.
- [x] If making a single "flag" is easier/faster let's do it. _**Currently single `is_managed` property on agent policy SO.**_
Checks are implemented in service layer (is agent enrolled in a managed policy?)
No UI-specific changes added but UI is affected because HTTP requests (like `api/fleet/agents/{agentId}/reassign`) can fail. See screenshots below.
Tests at service (`yarn test:jest`) and http (`yarn test ftr`) layers for each of create policy, update policy, unenroll agent, and reassign agent
Bulk actions currently filter out restricted items. A follow-up PR will change them to throw an error and cause the request to fail.
## Managed Policy
Can create (`POST`) and update (`PUT`) an agent policy with an `is_managed` property. Each new saved object will have an `is_managed` property (default `false`)
<details><summary>HTTP commands</summary>
#### Create (`is_managed: false` by default)
```
curl --user elastic:changeme -X POST localhost:5601/api/fleet/agent_policies -H 'Content-Type: application/json' -d'{ "name": "User created policy", "namespace": "default"}' -H 'kbn-xsrf: true'
{"item":{"id":"edc236a0-5cbb-11eb-ab2c-0134aecb4ce8","name":"User created policy","namespace":"default","is_managed":false,"revision":1,"updated_at":"2021-01-22T14:12:58.250Z","updated_by":"elastic"}}
```
#### Create with `is_managed: true`
```
curl --user elastic:changeme -X POST localhost:5601/api/fleet/agent_policies -H 'Content-Type: application/json' -d'{ "name": "User created policy", "namespace": "default"}' -H 'kbn-xsrf: true'
{"item":{"id":"67c785b0-662e-11eb-bf6b-4790dc0178c0","name":"User created policy","namespace":"default","is_managed":false,"revision":1,"updated_at":"2021-02-03T14:45:06.059Z","updated_by":"elastic"}}
```
#### Update with `is_managed: true`
```
curl --user elastic:changeme -X PUT -H 'Content-Type: application/json' -H 'kbn-xsrf: 1234' localhost:5601/api/fleet/agent_policies/67c785b0-662e-11eb-bf6b-4790dc0178c0 -d '{ "name":"User created policy","namespace":"default","is_managed":true }'
{"item":{"id":"67c785b0-662e-11eb-bf6b-4790dc0178c0","name":"User created policy","namespace":"default","is_managed":true,"revision":2,"updated_at":"2021-02-03T14:47:28.471Z","updated_by":"elastic","package_policies":[]}}
```
</details>
## Enroll behavior
is not changed/addressed in this PR. Agents can still be enrolled in managed policies
## Unenroll Agent from managed policy behavior
#### Enrolled in managed agent policy, cannot be unenrolled
```
curl --user elastic:changeme -X POST http://localhost:5601/api/fleet/agents/441d4a40-6710-11eb-8f57-db14e8e41cff/unenroll -H 'kbn-xsrf: 1234' | jq
{
"statusCode": 400,
"error": "Bad Request",
"message": "Cannot unenroll 441d4a40-6710-11eb-8f57-db14e8e41cff from a managed agent policy af9b4970-6701-11eb-b55a-899b78cb64da"
}
```
<details><summary>Screenshots for managed & unmanaged policies</summary>
#### Enrolled in managed agent policy, cannot be unenrolled
<img width="1931" alt="Screen Shot 2021-01-19 at 1 22 53 PM" src="https://user-images.githubusercontent.com/57655/105081614-67d05980-5a60-11eb-8faa-07e4e722a5b5.png">
<img width="1199" alt="Screen Shot 2021-01-19 at 1 30 26 PM" src="https://user-images.githubusercontent.com/57655/105081617-67d05980-5a60-11eb-9099-832dc6e04eca.png">
<img width="1971" alt="Screen Shot 2021-01-19 at 1 30 42 PM" src="https://user-images.githubusercontent.com/57655/105081618-67d05980-5a60-11eb-9a84-b80b6295ba19.png">
#### Enrolled agent policy is not managed, agent can be unenrolled<img width="1917" alt="Screen Shot 2021-01-19 at 1 44 12 PM" src="https://user-images.githubusercontent.com/57655/105081951-e3caa180-5a60-11eb-9308-7741b8986e8e.png">
<img width="2183" alt="Screen Shot 2021-01-19 at 1 44 19 PM" src="https://user-images.githubusercontent.com/57655/105081952-e3caa180-5a60-11eb-9833-1c721be0a107.png">
</details>
## Reassign agent
#### No agent can be reassigned to a managed policy
```
curl --user elastic:changeme -X 'PUT' 'http://localhost:5601/api/fleet/agents/482760d0-6710-11eb-8f57-db14e8e41cff/reassign' -H 'kbn-xsrf: xxx' -H 'Content-Type: application/json' -d '{"policy_id":"af9b4970-6701-11eb-b55a-899b78cb64da"}'
{
"statusCode": 400,
"error": "Bad Request",
"message": "Cannot reassign an agent to managed agent policy 94129590-6707-11eb-b55a-899b78cb64da"
}
```
<details><summary>Screenshots</summary>
<img width="1350" alt="Screen Shot 2021-02-04 at 2 14 51 PM" src="https://user-images.githubusercontent.com/57655/106943490-8044a300-66f3-11eb-9d2c-4b1ceef2e783.png">
</details>
#### Enrolled in managed agent policy, cannot be reassigned
```
curl --user elastic:changeme -X 'PUT' 'http://localhost:5601/api/fleet/agents/482760d0-6710-11eb-8f57-db14e8e41cff/reassign' -H 'kbn-xsrf: xxx' -H 'Content-Type: application/json' -d '{"policy_id":"af9b4970-6701-11eb-b55a-899b78cb64da"}'
{
"statusCode": 400,
"error": "Bad Request",
"message": "Cannot reassign an agent from managed agent policy 94129590-6707-11eb-b55a-899b78cb64da"
}
```
<details><summary>Screenshots</summary>
<img width="1364" alt="Screen Shot 2021-01-19 at 2 58 38 PM" src="https://user-images.githubusercontent.com/57655/105086737-72dab800-5a67-11eb-8f5e-93cd7768b914.png">
<img width="1367" alt="Screen Shot 2021-01-19 at 2 58 44 PM" src="https://user-images.githubusercontent.com/57655/105086740-73734e80-5a67-11eb-8ef9-9c7005a0a4ea.png">
<img width="623" alt="Screen Shot 2021-01-19 at 2 59 27 PM" src="https://user-images.githubusercontent.com/57655/105086741-740be500-5a67-11eb-8fc2-721f8b5d178a.png">
</details>
#### Enrolled agent policy is unmanaged, agent can be reassigned to another unmanaged policy
<details><summary>Screenshots</summary>
<img width="1368" alt="Screen Shot 2021-01-19 at 3 00 01 PM" src="https://user-images.githubusercontent.com/57655/105086754-78d09900-5a67-11eb-86a5-9e3ac02d6e1f.png">
<img width="1363" alt="Screen Shot 2021-01-19 at 3 00 08 PM" src="https://user-images.githubusercontent.com/57655/105086761-7a01c600-5a67-11eb-991d-acf994e2a393.png">
<img width="625" alt="Screen Shot 2021-01-19 at 3 00 46 PM" src="https://user-images.githubusercontent.com/57655/105086764-7a9a5c80-5a67-11eb-8290-e79648d01579.png">
</details>
### Checklist
Delete any items that are not applicable to this PR.
- [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)
- [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
* Add API integration tests for data streams list, including one that is expected to fail due to reliance on number of backing indices
* Use ES data streams API as source of truth for list of data streams, and only query against backing indices afterwards
* Get package name from data stream meta info
* Increate retry timeout
* Move initial info requests inside Promise.all
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Update data streams mappings directly instead of querying for backing indices, update integration tests to test with multiple namespaces
* Add flag to only update mappings of the current write index
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
The experimental registry was used for the 7.8 release but since then was not touched anymore. Because of this it should not show up in the code anymore even if it is commented out.
## Summary
A user received a `407` response when using a `registryProxyUrl` format like `http://user:pass@host:port`
I believe the issue is we're not including the [`auth` property as described in this issue](https://github.com/TooTallNate/node-https-proxy-agent/issues/12#issuecomment-216098644).
Add tests to ensure it only adds `auth` if username & password are given.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
```
getProxyAgentOptions
auth property
present & correct if given username and password
✓ proxy url is http (1 ms)
✓ proxy url is https
missing if not given username and password
✓ proxy url is http
✓ proxy url is https
```
