Commit graph

90 commits

Author SHA1 Message Date
John Schulz db34f0c6af
[Fleet] Update docker image of registry used in integration tests (#101911)
## Summary

 * Use a more recent Docker image for package registry used in integration/API tests. 
    * [New image](https://container-library.elastic.co/r/package-registry/distribution:fc104ac437370d80518e24da6d0b84370edf0c0c) is from Jun-14 [old image](https://container-library.elastic.co/r/package-registry/distribution:b6a53ac9300333a4a45f3f7d350c9aed72061a66) is from Apr-14
 * Add support for using a custom Docker registry using `FLEET_PACKAGE_REGISTRY_DOCKER_IMAGE` env var
    * e.g. `production` tag
    ```
    FLEET_PACKAGE_REGISTRY_DOCKER_IMAGE='docker.elastic.co/package-registry/distribution:production' FLEET_PACKAGE_REGISTRY_PORT=12345 yarn test:ftr:runner
    ```
    or a personal one
    ```
    FLEET_PACKAGE_REGISTRY_DOCKER_IMAGE='docker.elastic.co/employees/jfsiii/package-registry-distribution:kb90454' FLEET_PACKAGE_REGISTRY_PORT=12345 yarn test:ftr:runner

    ```

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2021-06-17 14:55:17 -04:00
Tyler Smalley c225aaaaa5 [FTR] Updates esArchive paths
PR's merged after #101345

Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
2021-06-08 21:10:36 -07:00
Nicolas Chaulet 8c2c83cb7d
[Fleet] Add fleet server telemetry (#101400) 2021-06-08 18:55:48 -04:00
Spencer f466ebf1a3
[esArchiver] drop support for --dir, use repo-relative paths instead (#101345)
Co-authored-by: spalger <spalger@users.noreply.github.com>
2021-06-08 17:37:42 -04:00
Mikhail Shustov d920682e4e
Update @elastic/elasticsearch to 8.0.0-canary13 (#98266)
* bump @elastic/elasticsearch to canary.7

* address errors in core

* address errors in data plugin

* address errors in Alerting team plugins

* remove outdated messages in Lens

* remove unnecessary comments in ML

* address errors in Observability plugin

* address errors in reporting plugin

* address errors in Rule registry plugin

* fix errors in Security plugins

* fix errors in ES-UI plugin

* remove unnecessary union.

* update core tests

* fix kbn-es-archiver

* update to canary 8

* bump to v9

* use new typings

* fix new errors in core

* fix errors in core typeings

* fix type errors in data plugin

* fix type errors in telemetray plugin

* fix data plugin tests

* fix search examples type error

* fix errors in discover plugin

* fix errors in index_pattern_management

* fix type errors in vis_type_*

* fix errors in typings/elasticsearch

* fix type errors in actions plugin

* fix type errors in alerting and apm plugins

* fix type errors in canvas and cases

* fix errors in event_log

* fix type errors in ILM and ingest_pipelines

* fix errors in lens plugin

* fix errors in lists plugin

* fix errors in logstash

* fix errors in metrics_entities

* fix errors in o11y

* fix errors in watcher

* fix errors in uptime

* fix errors in upgrade_assistant

* fix errors in task_manager

* fix errors in stack_alerts

* fix errors in security_solution

* fix errors in rule_registry

* fix errors in snapshot_restore

* fix remaining errors

* fix search intergration tests

* adjust assetion

* bump version to canary.10

* adapt code to new naming schema

* use mapping types provided by the client library

* Revert "adjust assetion"

This reverts commit 19b8fe0464.

* fix so intergration tests

* fix http integration tests

* bump version to canary 11

* fix login test

* fix http integration test

* fix apm test

* update docs

* fixing some ml types

* fix new errors in data plugin

* fix new errors in alerting plugin

* fix new errors in lists plugin

* fix new errors in reporting

* fix or mute errors in rule_registry plugin

* more ML type fixes

* bump to canary 12

* fix errors after merge conflict

* additional ML fixes

* bump to canary 13

* fix errors in apm plugin

* fix errors in fleet plugin

* fix errors in infra plugin

* fix errors in monitoring plugin

* fix errors in osquery plugin

* fix errors in security solution plugins

* fix errors in transform plugin

* Update type imports for ES

* fix errors in x-pack plugins

* fix errors in tests

* update docs

* fix errors in x-pack/test

* update error description

* fix errors after master merge

* update comment in infra plugin

* fix new errors on xpack tests/

Co-authored-by: James Gowdy <jgowdy@elastic.co>
Co-authored-by: Dario Gieselaar <dario.gieselaar@elastic.co>
2021-06-08 15:06:06 +02:00
Spencer be9fcad655
[fix] import from the root of @kbn/expect (#101321)
Co-authored-by: spalger <spalger@users.noreply.github.com>
2021-06-03 18:13:11 -04:00
Tiago Costa f69d63e8be
fix(NA): windows ts_project outside sandbox compilation (#100947)
* fix(NA): windows ts_project outside sandbox compilation adding tsconfig paths for packages

* chore(NA): missing @kbn paths for node_modules so types can work

* chore(NA): missing @kbn paths for node_modules so types can work

* chore(NA): organizing deps on non ts_project packages

* chore(NA): change order to find @kbn packages on node_modules first

* chore(NA): add @kbn/expect typings setting on package.json

* chore(NA): fix typechecking

* chore(NA): add missing change on tsconfig file

* chore(NA): unblock windows build by not depending on the pkg_npm rule symlink in the package.json

* chore(NA): add missing depedencies on BUILD.bazel file for io-ts-list-types

* chore(NA): remove rootDirs configs

* chore(NA): change kbn/monaco targets order

* chore(NA): update kbn-monaco build

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-06-03 17:53:39 +01:00
Nicolas Chaulet 3b1e8b03f1
[Fleet] Install final pipeline (#100973) 2021-06-03 12:27:29 -04:00
Spencer 808b44f2c1
[kbn/test] move types/ftr into src (#99555)
* [kbn/test] move types/ftr into src

* Apply eslint updates

* fix import of Lifecycle type

Co-authored-by: spalger <spalger@users.noreply.github.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-05-17 09:51:53 -07:00
John Schulz 0364e8f5aa
[Fleet] Fix error when searching for keys whose names have spaces (#100056)
## Summary
fixes #99895

Can reproduce #99895 with something like
```shell
curl 'http://localhost:5601/api/fleet/enrollment-api-keys' \
  -H 'content-type: application/json' \
  -H 'kbn-version: 8.0.0' \
  -u elastic:changeme \
  --data-raw '{"name":"with spaces","policy_id":"d6a93200-b1bd-11eb-90ac-052b474d74cd"}'
```

Kibana logs this stack trace

```
server    log   [10:57:07.863] [error][fleet][plugins] KQLSyntaxError: Expected AND, OR, end of input but "\" found.
policy_id:"d6a93200-b1bd-11eb-90ac-052b474d74cd" AND name:with\ spaces*
--------------------------------------------------------------^
    at Object.fromKueryExpression (/Users/jfsiii/work/kibana/src/plugins/data/common/es_query/kuery/ast/ast.ts:52:13)
    at listEnrollmentApiKeys (/Users/jfsiii/work/kibana/x-pack/plugins/fleet/server/services/api_keys/enrollment_api_key.ts:37:69)
    at Object.generateEnrollmentAPIKey (/Users/jfsiii/work/kibana/x-pack/plugins/fleet/server/services/api_keys/enrollment_api_key.ts:160:31)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at postEnrollmentApiKeyHandler (/Users/jfsiii/work/kibana/x-pack/plugins/fleet/server/routes/enrollment_api_key/handler.ts:53:20)
    at Router.handle (/Users/jfsiii/work/kibana/src/core/server/http/router/router.ts:273:30)
    at handler (/Users/jfsiii/work/kibana/src/core/server/http/router/router.ts:228:11)
    at exports.Manager.execute (/Users/jfsiii/work/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
    at Object.internals.handler (/Users/jfsiii/work/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)
    at exports.execute (/Users/jfsiii/work/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)
    at Request._lifecycle (/Users/jfsiii/work/kibana/node_modules/@hapi/hapi/lib/request.js:370:32)
    at Request._execute (/Users/jfsiii/work/kibana/node_modules/@hapi/hapi/lib/request.js:279:9) {
  shortMessage: 'Expected AND, OR, end of input but "\\" found.'
```

the `kuery` value which causes the `KQLSyntaxError` is
```
policy_id:\"d6a93200-b1bd-11eb-90ac-052b474d74cd\" AND name:with\\ spaces*
``` 

a value without spaces, e.g. `no_spaces` 

```
policy_id:\"d6a93200-b1bd-11eb-90ac-052b474d74cd\" AND name:no_spaces*
```

is converted to this query object

```
{
  "bool": {
    "filter": [
      {
        "bool": {
          "should": [
            {
              "match_phrase": {
                "policy_id": "d6a93200-b1bd-11eb-90ac-052b474d74cd"
              }
            }
          ],
          "minimum_should_match": 1
        }
      },
      {
        "bool": {
          "should": [
            {
              "query_string": {
                "fields": [
                  "name"
                ],
                "query": "no_spaces*"
              }
            }
          ],
          "minimum_should_match": 1
        }
      }
    ]
  }
```

I tried some other approaches for handling the spaces based on what I saw in the docs like `name:"\"with spaces\"` and `name:(with spaces)*`but they all failed as well, like

```
KQLSyntaxError: Expected AND, OR, end of input but "*" found.
policy_id:"d6a93200-b1bd-11eb-90ac-052b474d74cd" AND name:(with spaces)*
-----------------------------------------------------------------------^
    at Object.fromKueryExpression (/Users/jfsiii/work/kibana/src/plugins/data/common/es_query/kuery/ast/ast.ts:52:13)
    at listEnrollmentApiKeys (/Users/jfsiii/work/kibana/x-pack/plugins/fleet/server/services/api_keys/enrollment_api_key.ts:37:69)
    at Object.generateEnrollmentAPIKey (/Users/jfsiii/work/kibana/x-pack/plugins/fleet/server/services/api_keys/enrollment_api_key.ts:166:31)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at postEnrollmentApiKeyHandler (/Users/jfsiii/work/kibana/x-pack/plugins/fleet/server/routes/enrollment_api_key/handler.ts:53:20)
    at Router.handle (/Users/jfsiii/work/kibana/src/core/server/http/router/router.ts:273:30)
    at handler (/Users/jfsiii/work/kibana/src/core/server/http/router/router.ts:228:11)
    at exports.Manager.execute (/Users/jfsiii/work/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
    at Object.internals.handler (/Users/jfsiii/work/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)
    at exports.execute (/Users/jfsiii/work/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)
    at Request._lifecycle (/Users/jfsiii/work/kibana/node_modules/@hapi/hapi/lib/request.js:370:32)
    at Request._execute (/Users/jfsiii/work/kibana/node_modules/@hapi/hapi/lib/request.js:279:9) {
  shortMessage: 'Expected AND, OR, end of input but "*" found.'
```

So I logged out the query object for a successful request, and put that in a function

```
{
  "query": {
    "bool": {
      "filter": [
        {
          "bool": {
            "should": [
              {
                "match_phrase": {
                  "policy_id": "d6a93200-b1bd-11eb-90ac-052b474d74cd"
                }
              }
            ],
            "minimum_should_match": 1
          }
        },
        {
          "bool": {
            "should": [
              {
                "query_string": {
                  "fields": [
                    "name"
                  ],
                  "query": "(with spaces) *"
                }
              }
            ],
            "minimum_should_match": 1
          }
        }
      ]
    }
  }
}
```


### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2021-05-13 17:32:14 -04:00
Kyle Pollich baf11e8749
[Fleet] Remove timestamp field from component template (#99619)
* Remove timestamp field from component template

Elasticsearch bug fixed upstream allows us to remove this field from the
template: elastic/elasticsearch/#58956

Closes #71095

* Remove timestamp field from functional test
2021-05-10 14:11:23 -04:00
Spencer 21c54d38c3
[ftr] auto assign ciGroupDocker to suites with dockerServers (#99393)
Co-authored-by: spalger <spalger@users.noreply.github.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-05-06 12:42:29 -07:00
Zacqary Adam Xeper a72b0393f3
[Fleet] Fix frozen key definition (#99232)
* [Fleet] Fix frozen key definition

* Add integration test for frozen vars
2021-05-04 17:24:11 -05:00
Nicolas Chaulet ed8dc62f77
[Fleet] Add default http|https port to ES hosts (#99240) 2021-05-04 18:13:06 -04:00
Nicolas Chaulet b2a20fc9af
[Fleet] Fix fleet server host port during settings creation (#99084) 2021-05-03 17:42:07 -04:00
Brian Seeders f404305eaa
[CI] Create a ciGroup just for tests that use Docker (#98535) 2021-05-03 13:53:22 -04:00
Ross Wolf 0ddea04544
Make security rules optional (revert #97191) (#98854) 2021-04-29 19:59:52 -06:00
Zacqary Adam Xeper 57f84f8593
[Fleet] Refactor setup to load default packages/policies with preconfiguration (#97328)
Co-authored-by: Nicolas Chaulet <nicolas.chaulet@elastic.co>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-04-22 11:25:10 -05:00
Nicolas Chaulet ee7ba71b43
[Fleet] Always create agent policy as active (#97874) 2021-04-21 17:13:48 -04:00
John Schulz 45255425f4
Add the more explicit & actionable text listed in ticket (#97857)
## Summary
closes #92591

Create a new `HostedAgentPolicyRestrictionRelatedError` which ensures the additional text from #92591 is included in all the appropriate places, but only specified once.

#### Some current examples
 * `Cannot update integrations of hosted agent policy ${id}`
 * `Cannot remove integrations of hosted agent policy ${id}`
 * `Cannot reassign an agent to hosted agent policy ${newAgentPolicy.id}`


#### In this PR are now
 * `Cannot update integrations of hosted agent policy ${id}  in Fleet because the agent policy is managed by an external orchestration solution, such as Elastic Cloud, Kubernetes, etc. Please make changes using your orchestration solution.`
 * `Cannot remove integrations of hosted agent policy ${id} in Fleet because the agent policy is managed by an external orchestration solution, such as Elastic Cloud, Kubernetes, etc. Please make changes using your orchestration solution.`
 * `Cannot reassign an agent to hosted agent policy ${newAgentPolicy.id} in Fleet because the agent policy is managed by an external orchestration solution, such as Elastic Cloud, Kubernetes, etc. Please make changes using your orchestration solution.`


### Checklist

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2021-04-21 15:52:02 -04:00
Jen Huang 10e52bb582
[Fleet] Add instructions and generation of a service token for Fleet Server onboarding (#97585)
* Don't block standalone agent instructions when not using Fleet server yet

* Add service token instructions - UI only

* Add route for regenerating fleet server service token

* generate tokens instead of regenerate and add error catching and tests

* fix i18n typo

* i18n fix, add sudo, copy edits

* Fix commands

* Add missing test file

Co-authored-by: Nicolas Chaulet <nicolas.chaulet@elastic.co>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-04-20 13:53:18 -04:00
Ross Wolf 1925cea9a2
[Security][Fleet] Install the security_detection_engine package automatically (#97191)
* Automatically install the security_detection_engine package via fleet
* Update dockerImage to include the security_detection_engine package
* Update api/fleet/setup install test
* Update test data for Endpoint package
* Fix JSON token
* Update firis json entry in destination_index
* Update destination_index structure
* Update destination_index structure
* Change KQL query to unblock testing
* Restore KQL and fix JSON instead
* update timestamps to pass tests
2021-04-20 13:07:11 -04:00
Sonja Krause-Harder 05bd1c0cdb
[Fleet] Finer-grained error information from install/upgrade API (#95649)
* Intercept installation errors and add meta info.

* Adjust mock.

* Catch errors in all steps of install/upgrade.

* Adjust handler for direct package upload.

* Don't throw not-found errors on assets during rollback.

* Correctly catch errors from _installPackage()

* Propagate error from installResult in bulk install case.

* Add tests for rollback.

* Remove unused code.

* Skipping test that doesn't test what it says.

* Fix and reenable test.
2021-04-18 16:47:24 +02:00
John Schulz 6e5c9278ba
[Fleet] Use 'hosted agent policy' naming in UI & API messages (#97287)
## Summary

### What this PR _does_ change
 - [x] Replace all UI & API instances of "managed policy" with "hosted agent policy" #93504
 - [x] Replace all UI & API instances of "unmanaged policy" with "regular agent policy" #93504
 - [x] Update most variable names containing `managed` to `hosted`

<details><summary>screenshots</summary>

<img width="489" alt="Screen Shot 2021-04-15 at 10 54 01 AM" src="https://user-images.githubusercontent.com/57655/114898051-cac35980-9ddf-11eb-9c55-3fb5c0211c63.png">
<img width="339" alt="Screen Shot 2021-04-15 at 11 40 47 AM" src="https://user-images.githubusercontent.com/57655/114898053-cac35980-9ddf-11eb-8c60-3459aed3db6c.png">
<img width="714" alt="Screen Shot 2021-04-15 at 11 40 59 AM" src="https://user-images.githubusercontent.com/57655/114898054-cac35980-9ddf-11eb-912a-623eba9c3860.png">
</details>

### What this PR _does not_ change
  - [ ] The `is_managed` property of the agent policy saved object*
  - [ ] The updated error messages in https://github.com/elastic/kibana/issues/92591

<details><summary>*why not change the <code>is_managed</code> property?</summary>

<p>Changing the property will require a migration from 7.12. That's not a problem, but we haven't decided on <a href="https://github.com/elastic/kibana/issues/93504#issuecomment-790060997">a new name/approach</a> yet. Rather than update to <code>is_hosted: boolean</code> now and potentially something like <code>managed_by: enum</code> later, we'll update it when we have a decision.

<p><strong>TL;DR:</strong> It's a trivial update to make later and a minuscule amount of debt to pay until then. Less than cost of doing it twice.
 
</details>

### Checklist

- [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2021-04-15 21:01:28 -04:00
Nicolas Chaulet 78937e3bb4
[Fleet] Remove fleet agent routes and related services (#97206) 2021-04-15 17:15:43 -04:00
Nicolas Chaulet b96172e27c
[Fleet] Fleet server onboarding UI (#96867) 2021-04-15 01:22:32 -04:00
John Schulz de4bcdb9d9
[Fleet] Rename force to revoke agent unenroll APIs (#97041)
## Summary

 - fcbc9d9 Rename `force` param to `revoke` for `/agents/{agent_id}/unenroll` & `/agents/bulk_unenroll`
 - 03b9b90 Add new `force` param

See https://github.com/elastic/kibana/issues/96873 for background

<table>
  <thead>
    <tr>
      <td rowspan="2"></td><td colspan="2">Unenroll Agent</td><td rowspan="2">Revoke API Keys</td>
    </tr>
    <tr>
      <td>Regular</td><td>Hosted</td></td>
    </tr>
  </thead>
  <tr><td colspan="4"><strong>Rename <code>force</code> to <code>revoke</code></strong></td></tr>
  <tr><td>Current <code>force=false|undefined</code></td><td></td><td></td><td></td></tr>
  <tr><td>Proposed <code>revoke=false|undefined</code></td><td></td><td></td><td></td></tr>
  <tr><td>Current <code>force=true</code></td><td></td><td></td><td></td></tr>
  <tr><td>Proposed <code>revoke=true</code></td><td></td><td></td><td></td></tr>
  <tr><td colspan="4"><strong>Change <code>force</code> param </strong></td></tr>
  <tr><td>Proposed <code>force=false|undefined</code></td><td></td><td></td><td></td></tr>
  <tr><td>Proposed <code>force=true</code></td><td></td><td></td><td></td></tr>
  <tr><td>Proposed <code>force=true</code> & <code>revoke=true</code></td><td></td><td></td><td></td></tr>
</table>

### Checklist

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

### Changes required for consumers
Any call to `/agents/{agent_id}/unenroll` & `/agents/bulk_unenroll` which passes the `force` param should change to `revoke` to maintain the current behavior.
2021-04-14 19:27:06 -04:00
John Schulz 58b1d10f0b
Copy esArchiver commands from ./reassign.ts to fix tests (#97012)
## Summary

Seeing failures like this locally for `x-pack/test/fleet_api_integration/apis/agents/unenroll.ts` tests
<details><summary>screenshot of error</summary>
<img width="1121" alt="Screen Shot 2021-04-13 at 10 06 51 AM" src="https://user-images.githubusercontent.com/57655/114592495-fec04280-9c58-11eb-939c-b84aa7469c2a.png">
</details>
Copied the `esArchiver` patterns from `x-pack/test/fleet_api_integration/apis/agents/reassign.ts` in https://github.com/elastic/kibana/pull/96837 and the error is gone

### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2021-04-13 15:42:36 -04:00
Nicolas Chaulet 0e7612dd1a
[Fleet] Fix Fleet API integration tests (#96837) 2021-04-13 11:16:32 -04:00
John Schulz cb3c4e3a21
[Fleet] support force flag to add/remove package_policies (#96713)
## Summary

Can now pass a `force=true` parameter to add & remove integrations on hosted policies as originally intended [1] & [2]

 * Add `force` param for `POST` `/api/fleet/package_policies` & `/api/fleet/package_policies/delete` to a policy. Update tests to confirm
 * Not strictly required, but "while I was in there"
   * Updated a few places to throw `IngestManagerError` vs `Error` for `400` response vs `500`. Updated tests.
   * removed a few unnecessary `await`s of sync function

[1] https://github.com/elastic/kibana/issues/92426#issuecomment-785092670
[2] https://github.com/elastic/kibana/issues/90445

### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-04-12 14:05:39 -04:00
Rudolf Meijering f544d8d458
Migrations v2 ignore fleet agent events (#96690)
* migrationsv2: ignore fleet agent events and tsvb telemetry

* migrationsv1: ignore tsvb-validation-telemetry

* Skip fleet test that depends on fleet-agent-events

* Fix typescript errors

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-04-12 18:19:42 +02:00
John Schulz d338f1c3de
Allow integrations of hosted policies to be updated (#96705)
## Summary

Remove the restriction against updating integrations on hosted policies.

I described the current behavior and asked if it should change in [1]. Based on the responses in [2] & [3] and looking back at prior discussion around hosted policies, I don't think updates should be restricted.

Adding or removing integrations is still blocked for hosted policies. Updated API tests to confirm behavior. 


[1] https://github.com/elastic/kibana/issues/76843#issuecomment-816096760
[2] https://github.com/elastic/kibana/issues/76843#issuecomment-816153871
[3] https://github.com/elastic/kibana/issues/76843#issuecomment-816538672

## Screenshots
<details><summary>Current behavior</summary>

<h3>Error about updating integrations of a managed policy</h3>

<img width="1208" alt="Screen Shot 2021-04-08 at 3 23 37 PM" src="https://user-images.githubusercontent.com/57655/114084750-87686880-987e-11eb-91a9-081c45dbe871.png">

<details><summary>via flow A</summary>
<img width="1223" alt="Screen Shot 2021-04-08 at 3 01 32 PM" src="https://user-images.githubusercontent.com/57655/114082826-3a839280-987c-11eb-94d0-151ae93ab523.png">

<img width="1205" alt="Screen Shot 2021-04-08 at 3 13 24 PM" src="https://user-images.githubusercontent.com/57655/114083728-5c314980-987d-11eb-92be-195d7d44c037.png">
</details>

<details><summary>via flow B</summary>
<img width="1221" alt="Screen Shot 2021-04-08 at 3 19 52 PM" src="https://user-images.githubusercontent.com/57655/114084502-3fe1dc80-987e-11eb-8879-57718586ac95.png">
<img width="1198" alt="Screen Shot 2021-04-08 at 3 20 06 PM" src="https://user-images.githubusercontent.com/57655/114084503-3fe1dc80-987e-11eb-9fa9-512210b938cd.png">
</details>

</details>

<details><summary>This PR</summary>
<h3>Successful updates using either form</h3>
<img width="1301" alt="Screen Shot 2021-04-09 at 1 21 02 PM" src="https://user-images.githubusercontent.com/57655/114219370-8f84de80-9938-11eb-9b94-dfbeb18535b2.png">
<img width="1320" alt="Screen Shot 2021-04-09 at 1 05 10 PM" src="https://user-images.githubusercontent.com/57655/114219408-9f9cbe00-9938-11eb-96d2-2918332d1539.png">

</details>


### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-04-12 11:01:38 -04:00
Nicolas Chaulet 1de77ccb4e
[Fleet] Create enrollment API keys as current user (#96464) 2021-04-12 08:27:54 -04:00
spalger afc1fd022e skip entire fleet_api_integration suite to unblock es promotion (#96515) 2021-04-08 08:14:01 -07:00
spalger f9317281d1 skip suite blocking es promotion (#96515) 2021-04-08 02:09:29 -07:00
Ross Wolf 93965343e5
[Fleet] Install security_rule assets as saved objects (#95885)
* [Fleet] Install security_rule assets as saved objects

* Add security-rule to update_assets.ts

* Update UUIDs for security_rule asset

* Change .type to match the saved object type not the asset type

* Add saved object mapping for security-rule

* Make SO non-hidden

* Fix SO mapping for security-rule

* Make security-rule a non-hidden asset
2021-04-08 00:52:10 -04:00
spalger 0cf31ae22f skip suite block es promotion (#96515)
(cherry picked from commit f06be93a40)
2021-04-07 18:59:01 -07:00
spalger fc9f97e03b skip suites failing es promotion (#96515)
(cherry picked from commit 7fdf7e1d79)
2021-04-07 16:10:19 -07:00
Zacqary Adam Xeper f7caf44876
[Fleet] Match telemetry key names to UI agent states (#95567) 2021-03-31 15:10:46 -04:00
Zacqary Adam Xeper 28410539b5
[Fleet] Configure Fleet packages and integrations through endpoint (#94509) 2021-03-29 21:41:27 -04:00
James Gowdy 478ad3bad5
[Fleet] Adding ML module asset type (#94950)
* [Fleet] Adding ML module asset type

* adding test

* guessing asset ids

* better guess at IDs

* renaming asset ids

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-03-29 16:21:49 +01:00
John Schulz 7f15931725
[Fleet] /bulk_unenroll response matches other bulk action APIs (#95571)
## Summary
`/agents/bulk_unenroll` should return a response with a result for each agent given; including invalid or missing ids. It currently returns an empty object. https://github.com/elastic/kibana/issues/90437


[TS type diff for response](dd34e4c5ef/x-pack/plugins/fleet/common/types/rest_spec/agent.ts (L124-L130))

```diff
- // eslint-disable-next-line @typescript-eslint/no-empty-interface
- export interface PostBulkAgentUnenrollResponse {}
+ export type PostBulkAgentUnenrollResponse = Record<
+   Agent['id'],
+   {
+     success: boolean;
+     error?: string;
+   }
+ >;
```

### Checklist

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2021-03-26 17:36:39 -04:00
John Schulz ab33df898a
[Fleet] Bulk upgrade api response change (#95236)
## Summary
`/agents/bulk_upgrade` should return a response with a result for each agent given; including invalid or missing ids. It currently returns an empty object.

This PR includes commits from open PR https://github.com/elastic/kibana/pull/95024. The additions from this PR are https://github.com/jfsiii/kibana/compare/bulk-reassign-response-should-include-all-given-agents..871ebcb

[TS type diff for response](https://github.com/jfsiii/kibana/compare/bulk-reassign-response-should-include-all-given-agents..871ebcb#diff-7006a6c170a608c8c7211fc218c0a6f4bc8ff642c170ea264db4b1b5545fb728)

```diff
- // eslint-disable-next-line @typescript-eslint/no-empty-interface
- export interface PostBulkAgentUpgradeResponse {}

+ export type PostBulkAgentUpgradeResponse = Record<
+   Agent['id'],
+   {
+     success: boolean;
+     error?: string;
+   }
+ >;
```

### Checklist

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-03-26 11:40:38 -05:00
Tomas Della Vedova 238791b942
ES client : use the new type definitions (#83808)
* Use client from branch

* Get type checking working in core

* Fix types in other plugins

* Update client types + remove type errors from core

* migrate Task Manager Elasticsearch typing from legacy library to client library

* use SortOrder instead o string in alerts

* Update client types + fix core type issues

* fix maps ts errors

* Update Lens types

* Convert Search Profiler body from a string to an object to conform to SearchRequest type.

* Fix SOT types

* Fix/mute Security/Spaces plugins type errors.

* Fix bootstrap types

* Fix painless_lab

* corrected es typing in Event Log

* Use new types from client for inferred search responses

* Latest type defs

* Integrate latest type defs for APM/UX

* fix core errors

* fix telemetry errors

* fix canvas errors

* fix data_enhanced errors

* fix event_log errors

* mute lens errors

* fix or mute maps errors

* fix reporting errors

* fix security errors

* mute errors in task_manager

* fix errors in telemetry_collection_xpack

* fix errors in data plugins

* fix errors in alerts

* mute errors in index_management

* fix task_manager errors

* mute or fix lens errors

* fix upgrade_assistant errors

* fix or mute errors in index_lifecycle_management

* fix discover errors

* fix core tests

* ML changes

* fix core type errors

* mute error in kbn-es-archiver

* fix error in data plugin

* fix error in telemetry plugin

* fix error in discover

* fix discover errors

* fix errors in task_manager

* fix security errors

* fix wrong conflict resolution

* address errors with upstream code

* update deps to the last commit

* remove outdated comments

* fix core errors

* fix errors after update

* adding more expect errors to ML

* pull the lastest changes

* fix core errors

* fix errors in infra plugin

* fix errors in uptime plugin

* fix errors in ml

* fix errors in xpack telemetry

* fix or mute errors in transform

* fix errors in upgrade assistant

* fix or mute fleet errors

* start fixing apm errors

* fix errors in osquery

* fix telemetry tests

* core cleanup

* fix asMutableArray imports

* cleanup

* data_enhanced cleanup

* cleanup events_log

* cleaup

* fix error in kbn-es-archiver

* fix errors in kbn-es-archiver

* fix errors in kbn-es-archiver

* fix ES typings for Hit

* fix SO

* fix actions plugin

* fix fleet

* fix maps

* fix stack_alerts

* fix eslint problems

* fix event_log unit tests

* fix failures in data_enhanced tests

* fix test failure in kbn-es-archiver

* fix test failures in index_pattern_management

* fixing ML test

* remove outdated comment in kbn-es-archiver

* fix error type in ml

* fix eslint errors in osquery plugin

* fix runtime error in infra plugin

* revert changes to event_log cluser exist check

* fix eslint error in osquery

* fixing ML endpoint argument types

* fx types

* Update api-extractor docs

* attempt fix for ese test

* Fix lint error

* Fix types for ts refs

* Fix data_enhanced unit test

* fix lens types

* generate docs

* Fix a number of type issues in monitoring and ml

* fix triggers_actions_ui

* Fix ILM functional test

* Put search.d.ts typings back

* fix data plugin

* Update typings in typings/elasticsearch

* Update snapshots

* mute errors in task_manager

* mute fleet errors

* lens. remove unnecessary ts-expect-errors

* fix errors in stack_alerts

* mute errors in osquery

* fix errors in security_solution

* fix errors in lists

* fix errors in cases

* mute errors in search_examples

* use KibanaClient to enforce promise-based API

* fix errors in test/ folder

* update comment

* fix errors in x-pack/test folder

* fix errors in ml plugin

* fix optional fields in ml api_integartoon tests

* fix another casting problem in ml tests

* fix another ml test failure

* fix fleet problem after conflict resolution

* rollback changes in security_solution. trying to fix test

* Update type for discover rows

* uncomment runtime_mappings as its outdated

* address comments from Wylie

* remove eslint error due to any

* mute error due to incompatibility

* Apply suggestions from code review

Co-authored-by: John Schulz <github.com@jfsiii.org>

* fix type error in lens tests

* Update x-pack/plugins/upgrade_assistant/server/lib/reindexing/reindex_service.ts

Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com>

* Update x-pack/plugins/upgrade_assistant/server/lib/reindexing/reindex_service.test.ts

Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com>

* update deps

* fix errors in core types

* fix errors for the new elastic/elasticsearch version

* remove unused type

* remove unnecessary manual type cast and put optional chaining back

* ML: mute Datafeed is missing indices_options

* Apply suggestions from code review

Co-authored-by: Josh Dover <1813008+joshdover@users.noreply.github.com>

* use canary pacakge instead of git commit

Co-authored-by: Josh Dover <me@joshdover.com>
Co-authored-by: Josh Dover <1813008+joshdover@users.noreply.github.com>
Co-authored-by: Gidi Meir Morris <github@gidi.io>
Co-authored-by: Nathan Reese <reese.nathan@gmail.com>
Co-authored-by: Wylie Conlon <wylieconlon@gmail.com>
Co-authored-by: CJ Cenizal <cj@cenizal.com>
Co-authored-by: Aleh Zasypkin <aleh.zasypkin@gmail.com>
Co-authored-by: Dario Gieselaar <dario.gieselaar@elastic.co>
Co-authored-by: restrry <restrry@gmail.com>
Co-authored-by: James Gowdy <jgowdy@elastic.co>
Co-authored-by: John Schulz <github.com@jfsiii.org>
Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com>
2021-03-25 04:47:16 -04:00
Sonja Krause-Harder 2e5b5debb5
[Fleet] Add force option to DELETE package endpoint. (#95051)
* Add force option to DELETE package endpoint.

* Add integration test.

* Adjust openapi spec.

* Run EPM tests before fleet setup tests.

* Run package delete tests first in EPM tests
2021-03-24 17:04:49 +01:00
John Schulz 3639aa4422
[Fleet] Bulk reassign response should include all given ids (#95024)
## Summary
`/agents/bulk_reassign` should return a response with a result for each agent given; including invalid or missing ids. It currently filters out missing or invalid before updating. This PR leaves them in and includes their error results in the response. 

[Added/updated tests](https://github.com/elastic/kibana/pull/95024/files#diff-7ec94bee3e2bae79e5d98b8c17c17b26fad14736143ffa144f3e035773d4cad1R113-R128) to confirm

### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-03-24 11:44:22 -04:00
Jen Huang d886979e3b
[Fleet] Remove upgradePackage and consolidate it with installPackage, optimize calls to create index patterns (#94490)
* Add data plugin to server app context

* First attempt at switching to indexPatternService for EPM index pattern creation & deletion, instead of interacting directly with index pattern SOs

* Simplify bulk install package, remove upgradePackage() method in favor of consolidating with installPackage(), use installPackage() for bulk install instead

* Update tests

* Change cache logging of objects to trace level

* Install index patterns as a post-package installation operation, for bulk package installation, install index pattern only if one or more packages are actually new installs, add debug logging

* Allow getAsSavedObjectBody to return non-scripted fields when allowNoIndex is true

* Allow `getFieldsForWildcard` to return fields saved on index pattern when allowNoIndices is true

* Bring back passing custom ID for index pattern SO

* Fix tests

* Revert "Merge branch 'index-pattern/allow-no-index' into epm/missing-index-patterns"

This reverts commit 8e712e9c00, reversing
changes made to af0fb0eaa8.

* Allow getAsSavedObjectBody to return non-scripted fields when allowNoIndex is true

(cherry picked from commit 69b93da180)

* Update API docs

* Run post-install ops for install by upload too

* Remove allowedInstallTypes param

* Adjust force install conditions

* Revert "Update API docs"

This reverts commit b9770fdc56.

* Revert "Allow getAsSavedObjectBody to return non-scripted fields when allowNoIndex is true"

This reverts commit afc91ce32f.

* Go back to using SO client for index patterns :(

* Stringify attributes again for SO client saving

* Fix condition for reinstall same pkg version

* Remove unused type

* Adjust comment

* Update snapshot

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-03-23 11:34:20 -07:00
Nicolas Chaulet 76b55207f7
[Fleet] Fix agent status count (#95099) 2021-03-23 11:24:25 -04:00
John Schulz c0c5fba4b0
[Fleet] Support force flag in bulk upgrade agents API (#94952)
## Summary

0cbbb41da2 is just a rearrangement of the tests.  5cad301e87 has the real changes: 
* Bug fix: `force: true` should bypass any restrictions re: managed policies
* Refactoring towards new response shape coming as part of https://github.com/elastic/kibana/issues/90437
* Added test to confirm new behavior


### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2021-03-23 11:22:03 -04:00
John Schulz 044a94ac46
[Fleet] Add test/fix for invalid/missing ids in bulk agent reassign (#94632)
## Problem
While working on changes for bulk reassign https://github.com/elastic/kibana/issues/90437, I found that the server has a runtime error and returns a 500 if given an invalid or missing id.

<details><summary>server error stack trace</summary>

```
   │ proc [kibana] server    log   [12:21:48.953] [error][fleet][plugins] TypeError: Cannot read property 'policy_revision_idx' of undefined
   │ proc [kibana]     at map (/Users/jfsiii/work/kibana/x-pack/plugins/fleet/server/services/agents/helpers.ts:15:34)
   │ proc [kibana]     at Array.map (<anonymous>)
   │ proc [kibana]     at getAgents (/Users/jfsiii/work/kibana/x-pack/plugins/fleet/server/services/agents/crud.ts:191:32)
   │ proc [kibana]     at runMicrotasks (<anonymous>)
   │ proc [kibana]     at processTicksAndRejections (internal/process/task_queues.js:93:5)
   │ proc [kibana]     at Object.reassignAgents (/Users/jfsiii/work/kibana/x-pack/plugins/fleet/server/services/agents/reassign.ts:91:9)
   │ proc [kibana]     at postBulkAgentsReassignHandler (/Users/jfsiii/work/kibana/x-pack/plugins/fleet/server/routes/agent/handlers.ts:314:21)
   │ proc [kibana]     at Router.handle (/Users/jfsiii/work/kibana/src/core/server/http/router/router.ts:272:30)
   │ proc [kibana]     at handler (/Users/jfsiii/work/kibana/src/core/server/http/router/router.ts:227:11)
   │ proc [kibana]     at exports.Manager.execute (/Users/jfsiii/work/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
   │ proc [kibana]     at Object.internals.handler (/Users/jfsiii/work/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)
   │ proc [kibana]     at exports.execute (/Users/jfsiii/work/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)
   │ proc [kibana]     at Request._lifecycle (/Users/jfsiii/work/kibana/node_modules/@hapi/hapi/lib/request.js:370:32)
   │ proc [kibana]     at Request._execute (/Users/jfsiii/work/kibana/node_modules/@hapi/hapi/lib/request.js:279:9)
```
</details>

<details><summary>see test added in this PR fail on master</summary>

```
1)    Fleet Endpoints
       reassign agent(s)
         bulk reassign agents
           should allow to reassign multiple agents by id -- some invalid:

      Error: expected 200 "OK", got 500 "Internal Server Error"
```
</details>

## Root cause
Debugging runtime error in `searchHitToAgent` found some TS type mismatches for the ES values being returned. Perhaps from one or more of the recent changes to ES client & Fleet Server. Based on `test:jest` and `test:ftr`, it appears the possible types are `GetResponse` or `SearchResponse`, instead of only an `ESSearchHit`.

https://github.com/elastic/kibana/pull/94632/files#diff-254d0f427979efc3b442f78762302eb28fb9c8857df68ea04f8d411e052f939cL11

While a `.search` result will include return matched values, a `.get` or `.mget` will return a row for each input and a `found: boolean`. e.g. `{ _id: "does-not-exist", found: false }`. The error occurs when [`searchHitToAgent`](1702cf98f0/x-pack/plugins/fleet/server/services/agents/helpers.ts (L11)) is run on a get miss instead of a search hit.

## PR Changes
* Added a test to ensure it doesn't fail if invalid or missing IDs are given
* Moved the `bulk_reassign` tests to their own test section
* Filter out any missing results before calling `searchHitToAgent`, to match current behavior
* Consolidate repeated arguments into and code for getting agents into single [function](https://github.com/elastic/kibana/pull/94632/files#diff-f7377ed9ad56eaa8ea188b64e957e771ccc7a7652fd1eaf44251c25b930f8448R70-R87):  and [TS type](https://github.com/elastic/kibana/pull/94632/files#diff-f7377ed9ad56eaa8ea188b64e957e771ccc7a7652fd1eaf44251c25b930f8448R61-R68)
* Rename some agent service functions to be more explicit (IMO) but behavior maintained. Same API names exported.

This moves toward the "one result (success or error) per given id" approach for https://github.com/elastic/kibana/issues/90437

### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2021-03-16 20:51:57 -04:00