Commit graph

96 commits

Author SHA1 Message Date
John Schulz bb7bff5c96
[Fleet] Add UI and mappings for agent policy unenroll_timeout (#102970)
## Summary

closes https://github.com/elastic/kibana/issues/100617

UI and mappings related to ephemeral agents

- [x] Adds mapping/type/schema definition for the new field in agent policy saved object
- [x] Shows input field labelled `Unenrollment timeout` in agent policy settings that reads/writes to the new field
- [x] Same input in `Advanced options` section of create agent flyout
- [x] `unenroll_timeout` can be set using preconfigured agent policies defined in `kibana.yml`
- [x] `unenroll_timeout` can be populated if the user has a preconfigured policy that _does not_ have this field initially, but then updates their `kibana.yml` later to include it

<details><summary>Screenshot - editing an existing agent policy</summary>

<img width="1251" alt="Screen Shot 2021-06-22 at 1 42 50 PM" src="https://user-images.githubusercontent.com/57655/122974924-d479b800-d360-11eb-8d37-306a6860f140.png">

</details>


<details><summary>Screenshots - adding a new agent policy</summary>
<img width="1258" alt="Screen Shot 2021-06-22 at 1 45 01 PM" src="https://user-images.githubusercontent.com/57655/122975141-0be86480-d361-11eb-9ccb-dde0cc44e438.png">
<img width="1257" alt="Screen Shot 2021-06-22 at 1 45 35 PM" src="https://user-images.githubusercontent.com/57655/122975139-0be86480-d361-11eb-90ae-3541dbbd68e6.png">
<img width="1258" alt="Screen Shot 2021-06-22 at 1 45 44 PM" src="https://user-images.githubusercontent.com/57655/122975138-0b4fce00-d361-11eb-8347-d6f0cb43ab91.png">
<img width="1254" alt="Screen Shot 2021-06-22 at 1 45 56 PM" src="https://user-images.githubusercontent.com/57655/122975136-0b4fce00-d361-11eb-9c51-ec7a9e52ea2b.png">
</details>

<details><summary>Using <code>kibana.dev.yml</code></summary>

<h3>No <code>unenroll_timeout</code></h3>

```yml
xpack.fleet.agentPolicies:
  - name: Preconfigured Policy From Config
    description: From kibana.dev.yml (no timeout given)
    id: 1
    namespace: test
    package_policies:
      - package:
          name: system
        name: System Integration
        inputs:
          - type: system/metrics
            enabled: true
            vars:
              - name: system.hostfs
                value: home/test
            streams:
              - data_stream:
                  dataset: system.core
                enabled: true
                vars:
                  - name: period
                    value: 20s
          - type: winlog
            enabled: false
```

<h4>UI (saved object)</h4>
<img width="1072" alt="Screen Shot 2021-06-23 at 10 28 03 AM" src="https://user-images.githubusercontent.com/57655/123117442-bf0f9700-d40f-11eb-8e8f-685fee3ccb4b.png">

<h4><code>fleet-policies</code>index</h4>
<img width="1690" alt="Screen Shot 2021-06-23 at 10 52 39 AM" src="https://user-images.githubusercontent.com/57655/123119430-883a8080-d411-11eb-990f-4953845a76b5.png">


<h4>Updated <code>kibana.dev.yml</code> to include <code>unenroll_timeout</code></h4>

```yml
xpack.fleet.agentPolicies:
  - name: Preconfigured Policy From Config
    description: From kibana.dev.yml (updated with timeout)
    id: 1
    namespace: test
    unenroll_timeout: 234
    package_policies:
      - package:
          name: system
        name: System Integration
        inputs:
          - type: system/metrics
            enabled: true
            vars:
              - name: system.hostfs
                value: home/test
            streams:
              - data_stream:
                  dataset: system.core
                enabled: true
                vars:
                  - name: period
                    value: 20s
          - type: winlog
            enabled: false
```

<h4>UI (saved object)</h4>
<img width="1073" alt="Screen Shot 2021-06-23 at 10 35 17 AM" src="https://user-images.githubusercontent.com/57655/123117444-bfa82d80-d40f-11eb-94a9-99d4ca4bed2a.png">

<h4><code>fleet-policies</code>index</h4>
<img width="1690" alt="Screen Shot 2021-06-23 at 10 35 41 AM" src="https://user-images.githubusercontent.com/57655/123117446-bfa82d80-d40f-11eb-9563-1add80f43f0e.png">

</details>

### Checklist
- [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2021-06-23 14:46:04 -04:00
Nicolas Chaulet 3864fe1559
[Fleet] Add global component template to all fleet index templates (#102225) 2021-06-23 13:18:37 -04:00
John Schulz 868ae59c93
[Fleet] Support user overrides in composable templates (#101769)
## Summary
Closes #90454
Closes https://github.com/elastic/kibana/issues/72959

 * Rename the component templates which are [installed for some packages](https://github.com/elastic/kibana/blob/master/x-pack/plugins/fleet/server/services/epm/elasticsearch/template/install.ts#L197-L213) from `${templateName}-mappings` and `${templateName}-settings` to `${templateName}@mappings` and `${templateName}@settings`
 * When any package is installed, add a component template named `${templateName}@custom`
 * Any of above templates also include a `_meta` property with `{ package: { name: packageName } }`
 * On package installation, add any installed component templates to the `installed_es` property of the `epm-packages` saved object
 * On package removal, remove any installed component templates from the `installed_es` property of the `epm-packages` saved object

<details><summary>Kibana logs showing component templates added for package</summary>

```
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.file@mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.registry@mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [.logs-endpoint.diagnostic.collection@mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.library@mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.security@mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.network@mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.alerts@mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.metrics@mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.process@mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.policy@mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.metadata@mappings]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.registry@custom]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [.logs-endpoint.diagnostic.collection@custom]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.security@custom]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.file@custom]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.library@custom]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.network@custom]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.alerts@custom]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.metrics@custom]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.policy@custom]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [logs-endpoint.events.process@custom]
   │ info [o.e.c.m.MetadataIndexTemplateService] [JFSIII.local] adding component template [metrics-endpoint.metadata@custom]
```

</details>

<details><summary>screenshot - component templates are editable in the Stack Management UI</summary>
<img width="1342" alt="Screen Shot 2021-06-17 at 4 06 24 PM" src="https://user-images.githubusercontent.com/57655/122465421-1502bb80-cf86-11eb-94f4-9880cb3ea844.png">
</details>


### Checklist

Delete any items that are not applicable to this PR.

- [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2021-06-23 06:57:39 -04:00
Nicolas Chaulet d792e02e85
[Fleet] Correctly check for degraded status in agent healthbar (#102821) 2021-06-21 21:20:41 -04:00
Andrew Kroh 2e3d527696
[Fleet] Update final pipeline based on ECS event.agent_id_status (#102805)
This updates the Fleet final pipeline added in #100973 to match the specification of
`event.agent_id_status` field as defined in ECS. The field was added to ECS in
https://github.com/elastic/ecs/pull/1454. Basically the values of the field were simplified
from what was originally proposed and implemented.
2021-06-21 18:48:19 -04:00
Alejandro Fernández Gómez cd5cd65fb2
[Fleet] Improve performance of Fleet setup (#102219)
* Remove endpoint from the default packages

* Change the default spinner for the initial load

* Export fleet endpoint package as a constant

* Use constants for special packages

* Simplify type signature of `isRequiredPackage`

* Remove unused types

* Simplify required and default package definitions

* Treat REQUIRED_PACKAGES as independent from DEFAULT_PACKAGES

We want to keep the assumption that the lists contain the same packages
only in `epm/constants.ts`

* Install all default packages, not only the required ones

* Document the purpose of each package list

* Handle auto-update for non-default packages

* Make `endpoint` non-removable

* Make endpoint package be installed by default in tests

* Rename requiredPackages to unremovablePackages

* Fix type check

* Add Endpoint to be installed by default on Fleet tests too

Co-authored-by: Jen Huang <its.jenetic@gmail.com>
2021-06-17 11:58:41 -07:00
John Schulz db34f0c6af
[Fleet] Update docker image of registry used in integration tests (#101911)
## Summary

 * Use a more recent Docker image for package registry used in integration/API tests. 
    * [New image](https://container-library.elastic.co/r/package-registry/distribution:fc104ac437370d80518e24da6d0b84370edf0c0c) is from Jun-14 [old image](https://container-library.elastic.co/r/package-registry/distribution:b6a53ac9300333a4a45f3f7d350c9aed72061a66) is from Apr-14
 * Add support for using a custom Docker registry using `FLEET_PACKAGE_REGISTRY_DOCKER_IMAGE` env var
    * e.g. `production` tag
    ```
    FLEET_PACKAGE_REGISTRY_DOCKER_IMAGE='docker.elastic.co/package-registry/distribution:production' FLEET_PACKAGE_REGISTRY_PORT=12345 yarn test:ftr:runner
    ```
    or a personal one
    ```
    FLEET_PACKAGE_REGISTRY_DOCKER_IMAGE='docker.elastic.co/employees/jfsiii/package-registry-distribution:kb90454' FLEET_PACKAGE_REGISTRY_PORT=12345 yarn test:ftr:runner

    ```

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2021-06-17 14:55:17 -04:00
Tyler Smalley c225aaaaa5 [FTR] Updates esArchive paths
PR's merged after #101345

Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
2021-06-08 21:10:36 -07:00
Nicolas Chaulet 8c2c83cb7d
[Fleet] Add fleet server telemetry (#101400) 2021-06-08 18:55:48 -04:00
Spencer f466ebf1a3
[esArchiver] drop support for --dir, use repo-relative paths instead (#101345)
Co-authored-by: spalger <spalger@users.noreply.github.com>
2021-06-08 17:37:42 -04:00
Mikhail Shustov d920682e4e
Update @elastic/elasticsearch to 8.0.0-canary13 (#98266)
* bump @elastic/elasticsearch to canary.7

* address errors in core

* address errors in data plugin

* address errors in Alerting team plugins

* remove outdated messages in Lens

* remove unnecessary comments in ML

* address errors in Observability plugin

* address errors in reporting plugin

* address errors in Rule registry plugin

* fix errors in Security plugins

* fix errors in ES-UI plugin

* remove unnecessary union.

* update core tests

* fix kbn-es-archiver

* update to canary 8

* bump to v9

* use new typings

* fix new errors in core

* fix errors in core typeings

* fix type errors in data plugin

* fix type errors in telemetray plugin

* fix data plugin tests

* fix search examples type error

* fix errors in discover plugin

* fix errors in index_pattern_management

* fix type errors in vis_type_*

* fix errors in typings/elasticsearch

* fix type errors in actions plugin

* fix type errors in alerting and apm plugins

* fix type errors in canvas and cases

* fix errors in event_log

* fix type errors in ILM and ingest_pipelines

* fix errors in lens plugin

* fix errors in lists plugin

* fix errors in logstash

* fix errors in metrics_entities

* fix errors in o11y

* fix errors in watcher

* fix errors in uptime

* fix errors in upgrade_assistant

* fix errors in task_manager

* fix errors in stack_alerts

* fix errors in security_solution

* fix errors in rule_registry

* fix errors in snapshot_restore

* fix remaining errors

* fix search intergration tests

* adjust assetion

* bump version to canary.10

* adapt code to new naming schema

* use mapping types provided by the client library

* Revert "adjust assetion"

This reverts commit 19b8fe0464.

* fix so intergration tests

* fix http integration tests

* bump version to canary 11

* fix login test

* fix http integration test

* fix apm test

* update docs

* fixing some ml types

* fix new errors in data plugin

* fix new errors in alerting plugin

* fix new errors in lists plugin

* fix new errors in reporting

* fix or mute errors in rule_registry plugin

* more ML type fixes

* bump to canary 12

* fix errors after merge conflict

* additional ML fixes

* bump to canary 13

* fix errors in apm plugin

* fix errors in fleet plugin

* fix errors in infra plugin

* fix errors in monitoring plugin

* fix errors in osquery plugin

* fix errors in security solution plugins

* fix errors in transform plugin

* Update type imports for ES

* fix errors in x-pack plugins

* fix errors in tests

* update docs

* fix errors in x-pack/test

* update error description

* fix errors after master merge

* update comment in infra plugin

* fix new errors on xpack tests/

Co-authored-by: James Gowdy <jgowdy@elastic.co>
Co-authored-by: Dario Gieselaar <dario.gieselaar@elastic.co>
2021-06-08 15:06:06 +02:00
Spencer be9fcad655
[fix] import from the root of @kbn/expect (#101321)
Co-authored-by: spalger <spalger@users.noreply.github.com>
2021-06-03 18:13:11 -04:00
Tiago Costa f69d63e8be
fix(NA): windows ts_project outside sandbox compilation (#100947)
* fix(NA): windows ts_project outside sandbox compilation adding tsconfig paths for packages

* chore(NA): missing @kbn paths for node_modules so types can work

* chore(NA): missing @kbn paths for node_modules so types can work

* chore(NA): organizing deps on non ts_project packages

* chore(NA): change order to find @kbn packages on node_modules first

* chore(NA): add @kbn/expect typings setting on package.json

* chore(NA): fix typechecking

* chore(NA): add missing change on tsconfig file

* chore(NA): unblock windows build by not depending on the pkg_npm rule symlink in the package.json

* chore(NA): add missing depedencies on BUILD.bazel file for io-ts-list-types

* chore(NA): remove rootDirs configs

* chore(NA): change kbn/monaco targets order

* chore(NA): update kbn-monaco build

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-06-03 17:53:39 +01:00
Nicolas Chaulet 3b1e8b03f1
[Fleet] Install final pipeline (#100973) 2021-06-03 12:27:29 -04:00
Spencer 808b44f2c1
[kbn/test] move types/ftr into src (#99555)
* [kbn/test] move types/ftr into src

* Apply eslint updates

* fix import of Lifecycle type

Co-authored-by: spalger <spalger@users.noreply.github.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-05-17 09:51:53 -07:00
John Schulz 0364e8f5aa
[Fleet] Fix error when searching for keys whose names have spaces (#100056)
## Summary
fixes #99895

Can reproduce #99895 with something like
```shell
curl 'http://localhost:5601/api/fleet/enrollment-api-keys' \
  -H 'content-type: application/json' \
  -H 'kbn-version: 8.0.0' \
  -u elastic:changeme \
  --data-raw '{"name":"with spaces","policy_id":"d6a93200-b1bd-11eb-90ac-052b474d74cd"}'
```

Kibana logs this stack trace

```
server    log   [10:57:07.863] [error][fleet][plugins] KQLSyntaxError: Expected AND, OR, end of input but "\" found.
policy_id:"d6a93200-b1bd-11eb-90ac-052b474d74cd" AND name:with\ spaces*
--------------------------------------------------------------^
    at Object.fromKueryExpression (/Users/jfsiii/work/kibana/src/plugins/data/common/es_query/kuery/ast/ast.ts:52:13)
    at listEnrollmentApiKeys (/Users/jfsiii/work/kibana/x-pack/plugins/fleet/server/services/api_keys/enrollment_api_key.ts:37:69)
    at Object.generateEnrollmentAPIKey (/Users/jfsiii/work/kibana/x-pack/plugins/fleet/server/services/api_keys/enrollment_api_key.ts:160:31)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at postEnrollmentApiKeyHandler (/Users/jfsiii/work/kibana/x-pack/plugins/fleet/server/routes/enrollment_api_key/handler.ts:53:20)
    at Router.handle (/Users/jfsiii/work/kibana/src/core/server/http/router/router.ts:273:30)
    at handler (/Users/jfsiii/work/kibana/src/core/server/http/router/router.ts:228:11)
    at exports.Manager.execute (/Users/jfsiii/work/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
    at Object.internals.handler (/Users/jfsiii/work/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)
    at exports.execute (/Users/jfsiii/work/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)
    at Request._lifecycle (/Users/jfsiii/work/kibana/node_modules/@hapi/hapi/lib/request.js:370:32)
    at Request._execute (/Users/jfsiii/work/kibana/node_modules/@hapi/hapi/lib/request.js:279:9) {
  shortMessage: 'Expected AND, OR, end of input but "\\" found.'
```

the `kuery` value which causes the `KQLSyntaxError` is
```
policy_id:\"d6a93200-b1bd-11eb-90ac-052b474d74cd\" AND name:with\\ spaces*
``` 

a value without spaces, e.g. `no_spaces` 

```
policy_id:\"d6a93200-b1bd-11eb-90ac-052b474d74cd\" AND name:no_spaces*
```

is converted to this query object

```
{
  "bool": {
    "filter": [
      {
        "bool": {
          "should": [
            {
              "match_phrase": {
                "policy_id": "d6a93200-b1bd-11eb-90ac-052b474d74cd"
              }
            }
          ],
          "minimum_should_match": 1
        }
      },
      {
        "bool": {
          "should": [
            {
              "query_string": {
                "fields": [
                  "name"
                ],
                "query": "no_spaces*"
              }
            }
          ],
          "minimum_should_match": 1
        }
      }
    ]
  }
```

I tried some other approaches for handling the spaces based on what I saw in the docs like `name:"\"with spaces\"` and `name:(with spaces)*`but they all failed as well, like

```
KQLSyntaxError: Expected AND, OR, end of input but "*" found.
policy_id:"d6a93200-b1bd-11eb-90ac-052b474d74cd" AND name:(with spaces)*
-----------------------------------------------------------------------^
    at Object.fromKueryExpression (/Users/jfsiii/work/kibana/src/plugins/data/common/es_query/kuery/ast/ast.ts:52:13)
    at listEnrollmentApiKeys (/Users/jfsiii/work/kibana/x-pack/plugins/fleet/server/services/api_keys/enrollment_api_key.ts:37:69)
    at Object.generateEnrollmentAPIKey (/Users/jfsiii/work/kibana/x-pack/plugins/fleet/server/services/api_keys/enrollment_api_key.ts:166:31)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at postEnrollmentApiKeyHandler (/Users/jfsiii/work/kibana/x-pack/plugins/fleet/server/routes/enrollment_api_key/handler.ts:53:20)
    at Router.handle (/Users/jfsiii/work/kibana/src/core/server/http/router/router.ts:273:30)
    at handler (/Users/jfsiii/work/kibana/src/core/server/http/router/router.ts:228:11)
    at exports.Manager.execute (/Users/jfsiii/work/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
    at Object.internals.handler (/Users/jfsiii/work/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)
    at exports.execute (/Users/jfsiii/work/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)
    at Request._lifecycle (/Users/jfsiii/work/kibana/node_modules/@hapi/hapi/lib/request.js:370:32)
    at Request._execute (/Users/jfsiii/work/kibana/node_modules/@hapi/hapi/lib/request.js:279:9) {
  shortMessage: 'Expected AND, OR, end of input but "*" found.'
```

So I logged out the query object for a successful request, and put that in a function

```
{
  "query": {
    "bool": {
      "filter": [
        {
          "bool": {
            "should": [
              {
                "match_phrase": {
                  "policy_id": "d6a93200-b1bd-11eb-90ac-052b474d74cd"
                }
              }
            ],
            "minimum_should_match": 1
          }
        },
        {
          "bool": {
            "should": [
              {
                "query_string": {
                  "fields": [
                    "name"
                  ],
                  "query": "(with spaces) *"
                }
              }
            ],
            "minimum_should_match": 1
          }
        }
      ]
    }
  }
}
```


### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2021-05-13 17:32:14 -04:00
Kyle Pollich baf11e8749
[Fleet] Remove timestamp field from component template (#99619)
* Remove timestamp field from component template

Elasticsearch bug fixed upstream allows us to remove this field from the
template: elastic/elasticsearch/#58956

Closes #71095

* Remove timestamp field from functional test
2021-05-10 14:11:23 -04:00
Spencer 21c54d38c3
[ftr] auto assign ciGroupDocker to suites with dockerServers (#99393)
Co-authored-by: spalger <spalger@users.noreply.github.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-05-06 12:42:29 -07:00
Zacqary Adam Xeper a72b0393f3
[Fleet] Fix frozen key definition (#99232)
* [Fleet] Fix frozen key definition

* Add integration test for frozen vars
2021-05-04 17:24:11 -05:00
Nicolas Chaulet ed8dc62f77
[Fleet] Add default http|https port to ES hosts (#99240) 2021-05-04 18:13:06 -04:00
Nicolas Chaulet b2a20fc9af
[Fleet] Fix fleet server host port during settings creation (#99084) 2021-05-03 17:42:07 -04:00
Brian Seeders f404305eaa
[CI] Create a ciGroup just for tests that use Docker (#98535) 2021-05-03 13:53:22 -04:00
Ross Wolf 0ddea04544
Make security rules optional (revert #97191) (#98854) 2021-04-29 19:59:52 -06:00
Zacqary Adam Xeper 57f84f8593
[Fleet] Refactor setup to load default packages/policies with preconfiguration (#97328)
Co-authored-by: Nicolas Chaulet <nicolas.chaulet@elastic.co>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-04-22 11:25:10 -05:00
Nicolas Chaulet ee7ba71b43
[Fleet] Always create agent policy as active (#97874) 2021-04-21 17:13:48 -04:00
John Schulz 45255425f4
Add the more explicit & actionable text listed in ticket (#97857)
## Summary
closes #92591

Create a new `HostedAgentPolicyRestrictionRelatedError` which ensures the additional text from #92591 is included in all the appropriate places, but only specified once.

#### Some current examples
 * `Cannot update integrations of hosted agent policy ${id}`
 * `Cannot remove integrations of hosted agent policy ${id}`
 * `Cannot reassign an agent to hosted agent policy ${newAgentPolicy.id}`


#### In this PR are now
 * `Cannot update integrations of hosted agent policy ${id}  in Fleet because the agent policy is managed by an external orchestration solution, such as Elastic Cloud, Kubernetes, etc. Please make changes using your orchestration solution.`
 * `Cannot remove integrations of hosted agent policy ${id} in Fleet because the agent policy is managed by an external orchestration solution, such as Elastic Cloud, Kubernetes, etc. Please make changes using your orchestration solution.`
 * `Cannot reassign an agent to hosted agent policy ${newAgentPolicy.id} in Fleet because the agent policy is managed by an external orchestration solution, such as Elastic Cloud, Kubernetes, etc. Please make changes using your orchestration solution.`


### Checklist

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2021-04-21 15:52:02 -04:00
Jen Huang 10e52bb582
[Fleet] Add instructions and generation of a service token for Fleet Server onboarding (#97585)
* Don't block standalone agent instructions when not using Fleet server yet

* Add service token instructions - UI only

* Add route for regenerating fleet server service token

* generate tokens instead of regenerate and add error catching and tests

* fix i18n typo

* i18n fix, add sudo, copy edits

* Fix commands

* Add missing test file

Co-authored-by: Nicolas Chaulet <nicolas.chaulet@elastic.co>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-04-20 13:53:18 -04:00
Ross Wolf 1925cea9a2
[Security][Fleet] Install the security_detection_engine package automatically (#97191)
* Automatically install the security_detection_engine package via fleet
* Update dockerImage to include the security_detection_engine package
* Update api/fleet/setup install test
* Update test data for Endpoint package
* Fix JSON token
* Update firis json entry in destination_index
* Update destination_index structure
* Update destination_index structure
* Change KQL query to unblock testing
* Restore KQL and fix JSON instead
* update timestamps to pass tests
2021-04-20 13:07:11 -04:00
Sonja Krause-Harder 05bd1c0cdb
[Fleet] Finer-grained error information from install/upgrade API (#95649)
* Intercept installation errors and add meta info.

* Adjust mock.

* Catch errors in all steps of install/upgrade.

* Adjust handler for direct package upload.

* Don't throw not-found errors on assets during rollback.

* Correctly catch errors from _installPackage()

* Propagate error from installResult in bulk install case.

* Add tests for rollback.

* Remove unused code.

* Skipping test that doesn't test what it says.

* Fix and reenable test.
2021-04-18 16:47:24 +02:00
John Schulz 6e5c9278ba
[Fleet] Use 'hosted agent policy' naming in UI & API messages (#97287)
## Summary

### What this PR _does_ change
 - [x] Replace all UI & API instances of "managed policy" with "hosted agent policy" #93504
 - [x] Replace all UI & API instances of "unmanaged policy" with "regular agent policy" #93504
 - [x] Update most variable names containing `managed` to `hosted`

<details><summary>screenshots</summary>

<img width="489" alt="Screen Shot 2021-04-15 at 10 54 01 AM" src="https://user-images.githubusercontent.com/57655/114898051-cac35980-9ddf-11eb-9c55-3fb5c0211c63.png">
<img width="339" alt="Screen Shot 2021-04-15 at 11 40 47 AM" src="https://user-images.githubusercontent.com/57655/114898053-cac35980-9ddf-11eb-8c60-3459aed3db6c.png">
<img width="714" alt="Screen Shot 2021-04-15 at 11 40 59 AM" src="https://user-images.githubusercontent.com/57655/114898054-cac35980-9ddf-11eb-912a-623eba9c3860.png">
</details>

### What this PR _does not_ change
  - [ ] The `is_managed` property of the agent policy saved object*
  - [ ] The updated error messages in https://github.com/elastic/kibana/issues/92591

<details><summary>*why not change the <code>is_managed</code> property?</summary>

<p>Changing the property will require a migration from 7.12. That's not a problem, but we haven't decided on <a href="https://github.com/elastic/kibana/issues/93504#issuecomment-790060997">a new name/approach</a> yet. Rather than update to <code>is_hosted: boolean</code> now and potentially something like <code>managed_by: enum</code> later, we'll update it when we have a decision.

<p><strong>TL;DR:</strong> It's a trivial update to make later and a minuscule amount of debt to pay until then. Less than cost of doing it twice.
 
</details>

### Checklist

- [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2021-04-15 21:01:28 -04:00
Nicolas Chaulet 78937e3bb4
[Fleet] Remove fleet agent routes and related services (#97206) 2021-04-15 17:15:43 -04:00
Nicolas Chaulet b96172e27c
[Fleet] Fleet server onboarding UI (#96867) 2021-04-15 01:22:32 -04:00
John Schulz de4bcdb9d9
[Fleet] Rename force to revoke agent unenroll APIs (#97041)
## Summary

 - fcbc9d9 Rename `force` param to `revoke` for `/agents/{agent_id}/unenroll` & `/agents/bulk_unenroll`
 - 03b9b90 Add new `force` param

See https://github.com/elastic/kibana/issues/96873 for background

<table>
  <thead>
    <tr>
      <td rowspan="2"></td><td colspan="2">Unenroll Agent</td><td rowspan="2">Revoke API Keys</td>
    </tr>
    <tr>
      <td>Regular</td><td>Hosted</td></td>
    </tr>
  </thead>
  <tr><td colspan="4"><strong>Rename <code>force</code> to <code>revoke</code></strong></td></tr>
  <tr><td>Current <code>force=false|undefined</code></td><td></td><td></td><td></td></tr>
  <tr><td>Proposed <code>revoke=false|undefined</code></td><td></td><td></td><td></td></tr>
  <tr><td>Current <code>force=true</code></td><td></td><td></td><td></td></tr>
  <tr><td>Proposed <code>revoke=true</code></td><td></td><td></td><td></td></tr>
  <tr><td colspan="4"><strong>Change <code>force</code> param </strong></td></tr>
  <tr><td>Proposed <code>force=false|undefined</code></td><td></td><td></td><td></td></tr>
  <tr><td>Proposed <code>force=true</code></td><td></td><td></td><td></td></tr>
  <tr><td>Proposed <code>force=true</code> & <code>revoke=true</code></td><td></td><td></td><td></td></tr>
</table>

### Checklist

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

### Changes required for consumers
Any call to `/agents/{agent_id}/unenroll` & `/agents/bulk_unenroll` which passes the `force` param should change to `revoke` to maintain the current behavior.
2021-04-14 19:27:06 -04:00
John Schulz 58b1d10f0b
Copy esArchiver commands from ./reassign.ts to fix tests (#97012)
## Summary

Seeing failures like this locally for `x-pack/test/fleet_api_integration/apis/agents/unenroll.ts` tests
<details><summary>screenshot of error</summary>
<img width="1121" alt="Screen Shot 2021-04-13 at 10 06 51 AM" src="https://user-images.githubusercontent.com/57655/114592495-fec04280-9c58-11eb-939c-b84aa7469c2a.png">
</details>
Copied the `esArchiver` patterns from `x-pack/test/fleet_api_integration/apis/agents/reassign.ts` in https://github.com/elastic/kibana/pull/96837 and the error is gone

### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2021-04-13 15:42:36 -04:00
Nicolas Chaulet 0e7612dd1a
[Fleet] Fix Fleet API integration tests (#96837) 2021-04-13 11:16:32 -04:00
John Schulz cb3c4e3a21
[Fleet] support force flag to add/remove package_policies (#96713)
## Summary

Can now pass a `force=true` parameter to add & remove integrations on hosted policies as originally intended [1] & [2]

 * Add `force` param for `POST` `/api/fleet/package_policies` & `/api/fleet/package_policies/delete` to a policy. Update tests to confirm
 * Not strictly required, but "while I was in there"
   * Updated a few places to throw `IngestManagerError` vs `Error` for `400` response vs `500`. Updated tests.
   * removed a few unnecessary `await`s of sync function

[1] https://github.com/elastic/kibana/issues/92426#issuecomment-785092670
[2] https://github.com/elastic/kibana/issues/90445

### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-04-12 14:05:39 -04:00
Rudolf Meijering f544d8d458
Migrations v2 ignore fleet agent events (#96690)
* migrationsv2: ignore fleet agent events and tsvb telemetry

* migrationsv1: ignore tsvb-validation-telemetry

* Skip fleet test that depends on fleet-agent-events

* Fix typescript errors

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-04-12 18:19:42 +02:00
John Schulz d338f1c3de
Allow integrations of hosted policies to be updated (#96705)
## Summary

Remove the restriction against updating integrations on hosted policies.

I described the current behavior and asked if it should change in [1]. Based on the responses in [2] & [3] and looking back at prior discussion around hosted policies, I don't think updates should be restricted.

Adding or removing integrations is still blocked for hosted policies. Updated API tests to confirm behavior. 


[1] https://github.com/elastic/kibana/issues/76843#issuecomment-816096760
[2] https://github.com/elastic/kibana/issues/76843#issuecomment-816153871
[3] https://github.com/elastic/kibana/issues/76843#issuecomment-816538672

## Screenshots
<details><summary>Current behavior</summary>

<h3>Error about updating integrations of a managed policy</h3>

<img width="1208" alt="Screen Shot 2021-04-08 at 3 23 37 PM" src="https://user-images.githubusercontent.com/57655/114084750-87686880-987e-11eb-91a9-081c45dbe871.png">

<details><summary>via flow A</summary>
<img width="1223" alt="Screen Shot 2021-04-08 at 3 01 32 PM" src="https://user-images.githubusercontent.com/57655/114082826-3a839280-987c-11eb-94d0-151ae93ab523.png">

<img width="1205" alt="Screen Shot 2021-04-08 at 3 13 24 PM" src="https://user-images.githubusercontent.com/57655/114083728-5c314980-987d-11eb-92be-195d7d44c037.png">
</details>

<details><summary>via flow B</summary>
<img width="1221" alt="Screen Shot 2021-04-08 at 3 19 52 PM" src="https://user-images.githubusercontent.com/57655/114084502-3fe1dc80-987e-11eb-8879-57718586ac95.png">
<img width="1198" alt="Screen Shot 2021-04-08 at 3 20 06 PM" src="https://user-images.githubusercontent.com/57655/114084503-3fe1dc80-987e-11eb-9fa9-512210b938cd.png">
</details>

</details>

<details><summary>This PR</summary>
<h3>Successful updates using either form</h3>
<img width="1301" alt="Screen Shot 2021-04-09 at 1 21 02 PM" src="https://user-images.githubusercontent.com/57655/114219370-8f84de80-9938-11eb-9b94-dfbeb18535b2.png">
<img width="1320" alt="Screen Shot 2021-04-09 at 1 05 10 PM" src="https://user-images.githubusercontent.com/57655/114219408-9f9cbe00-9938-11eb-96d2-2918332d1539.png">

</details>


### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-04-12 11:01:38 -04:00
Nicolas Chaulet 1de77ccb4e
[Fleet] Create enrollment API keys as current user (#96464) 2021-04-12 08:27:54 -04:00
spalger afc1fd022e skip entire fleet_api_integration suite to unblock es promotion (#96515) 2021-04-08 08:14:01 -07:00
spalger f9317281d1 skip suite blocking es promotion (#96515) 2021-04-08 02:09:29 -07:00
Ross Wolf 93965343e5
[Fleet] Install security_rule assets as saved objects (#95885)
* [Fleet] Install security_rule assets as saved objects

* Add security-rule to update_assets.ts

* Update UUIDs for security_rule asset

* Change .type to match the saved object type not the asset type

* Add saved object mapping for security-rule

* Make SO non-hidden

* Fix SO mapping for security-rule

* Make security-rule a non-hidden asset
2021-04-08 00:52:10 -04:00
spalger 0cf31ae22f skip suite block es promotion (#96515)
(cherry picked from commit f06be93a40)
2021-04-07 18:59:01 -07:00
spalger fc9f97e03b skip suites failing es promotion (#96515)
(cherry picked from commit 7fdf7e1d79)
2021-04-07 16:10:19 -07:00
Zacqary Adam Xeper f7caf44876
[Fleet] Match telemetry key names to UI agent states (#95567) 2021-03-31 15:10:46 -04:00
Zacqary Adam Xeper 28410539b5
[Fleet] Configure Fleet packages and integrations through endpoint (#94509) 2021-03-29 21:41:27 -04:00
James Gowdy 478ad3bad5
[Fleet] Adding ML module asset type (#94950)
* [Fleet] Adding ML module asset type

* adding test

* guessing asset ids

* better guess at IDs

* renaming asset ids

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-03-29 16:21:49 +01:00
John Schulz 7f15931725
[Fleet] /bulk_unenroll response matches other bulk action APIs (#95571)
## Summary
`/agents/bulk_unenroll` should return a response with a result for each agent given; including invalid or missing ids. It currently returns an empty object. https://github.com/elastic/kibana/issues/90437


[TS type diff for response](dd34e4c5ef/x-pack/plugins/fleet/common/types/rest_spec/agent.ts (L124-L130))

```diff
- // eslint-disable-next-line @typescript-eslint/no-empty-interface
- export interface PostBulkAgentUnenrollResponse {}
+ export type PostBulkAgentUnenrollResponse = Record<
+   Agent['id'],
+   {
+     success: boolean;
+     error?: string;
+   }
+ >;
```

### Checklist

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2021-03-26 17:36:39 -04:00
John Schulz ab33df898a
[Fleet] Bulk upgrade api response change (#95236)
## Summary
`/agents/bulk_upgrade` should return a response with a result for each agent given; including invalid or missing ids. It currently returns an empty object.

This PR includes commits from open PR https://github.com/elastic/kibana/pull/95024. The additions from this PR are https://github.com/jfsiii/kibana/compare/bulk-reassign-response-should-include-all-given-agents..871ebcb

[TS type diff for response](https://github.com/jfsiii/kibana/compare/bulk-reassign-response-should-include-all-given-agents..871ebcb#diff-7006a6c170a608c8c7211fc218c0a6f4bc8ff642c170ea264db4b1b5545fb728)

```diff
- // eslint-disable-next-line @typescript-eslint/no-empty-interface
- export interface PostBulkAgentUpgradeResponse {}

+ export type PostBulkAgentUpgradeResponse = Record<
+   Agent['id'],
+   {
+     success: boolean;
+     error?: string;
+   }
+ >;
```

### Checklist

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
2021-03-26 11:40:38 -05:00
Tomas Della Vedova 238791b942
ES client : use the new type definitions (#83808)
* Use client from branch

* Get type checking working in core

* Fix types in other plugins

* Update client types + remove type errors from core

* migrate Task Manager Elasticsearch typing from legacy library to client library

* use SortOrder instead o string in alerts

* Update client types + fix core type issues

* fix maps ts errors

* Update Lens types

* Convert Search Profiler body from a string to an object to conform to SearchRequest type.

* Fix SOT types

* Fix/mute Security/Spaces plugins type errors.

* Fix bootstrap types

* Fix painless_lab

* corrected es typing in Event Log

* Use new types from client for inferred search responses

* Latest type defs

* Integrate latest type defs for APM/UX

* fix core errors

* fix telemetry errors

* fix canvas errors

* fix data_enhanced errors

* fix event_log errors

* mute lens errors

* fix or mute maps errors

* fix reporting errors

* fix security errors

* mute errors in task_manager

* fix errors in telemetry_collection_xpack

* fix errors in data plugins

* fix errors in alerts

* mute errors in index_management

* fix task_manager errors

* mute or fix lens errors

* fix upgrade_assistant errors

* fix or mute errors in index_lifecycle_management

* fix discover errors

* fix core tests

* ML changes

* fix core type errors

* mute error in kbn-es-archiver

* fix error in data plugin

* fix error in telemetry plugin

* fix error in discover

* fix discover errors

* fix errors in task_manager

* fix security errors

* fix wrong conflict resolution

* address errors with upstream code

* update deps to the last commit

* remove outdated comments

* fix core errors

* fix errors after update

* adding more expect errors to ML

* pull the lastest changes

* fix core errors

* fix errors in infra plugin

* fix errors in uptime plugin

* fix errors in ml

* fix errors in xpack telemetry

* fix or mute errors in transform

* fix errors in upgrade assistant

* fix or mute fleet errors

* start fixing apm errors

* fix errors in osquery

* fix telemetry tests

* core cleanup

* fix asMutableArray imports

* cleanup

* data_enhanced cleanup

* cleanup events_log

* cleaup

* fix error in kbn-es-archiver

* fix errors in kbn-es-archiver

* fix errors in kbn-es-archiver

* fix ES typings for Hit

* fix SO

* fix actions plugin

* fix fleet

* fix maps

* fix stack_alerts

* fix eslint problems

* fix event_log unit tests

* fix failures in data_enhanced tests

* fix test failure in kbn-es-archiver

* fix test failures in index_pattern_management

* fixing ML test

* remove outdated comment in kbn-es-archiver

* fix error type in ml

* fix eslint errors in osquery plugin

* fix runtime error in infra plugin

* revert changes to event_log cluser exist check

* fix eslint error in osquery

* fixing ML endpoint argument types

* fx types

* Update api-extractor docs

* attempt fix for ese test

* Fix lint error

* Fix types for ts refs

* Fix data_enhanced unit test

* fix lens types

* generate docs

* Fix a number of type issues in monitoring and ml

* fix triggers_actions_ui

* Fix ILM functional test

* Put search.d.ts typings back

* fix data plugin

* Update typings in typings/elasticsearch

* Update snapshots

* mute errors in task_manager

* mute fleet errors

* lens. remove unnecessary ts-expect-errors

* fix errors in stack_alerts

* mute errors in osquery

* fix errors in security_solution

* fix errors in lists

* fix errors in cases

* mute errors in search_examples

* use KibanaClient to enforce promise-based API

* fix errors in test/ folder

* update comment

* fix errors in x-pack/test folder

* fix errors in ml plugin

* fix optional fields in ml api_integartoon tests

* fix another casting problem in ml tests

* fix another ml test failure

* fix fleet problem after conflict resolution

* rollback changes in security_solution. trying to fix test

* Update type for discover rows

* uncomment runtime_mappings as its outdated

* address comments from Wylie

* remove eslint error due to any

* mute error due to incompatibility

* Apply suggestions from code review

Co-authored-by: John Schulz <github.com@jfsiii.org>

* fix type error in lens tests

* Update x-pack/plugins/upgrade_assistant/server/lib/reindexing/reindex_service.ts

Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com>

* Update x-pack/plugins/upgrade_assistant/server/lib/reindexing/reindex_service.test.ts

Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com>

* update deps

* fix errors in core types

* fix errors for the new elastic/elasticsearch version

* remove unused type

* remove unnecessary manual type cast and put optional chaining back

* ML: mute Datafeed is missing indices_options

* Apply suggestions from code review

Co-authored-by: Josh Dover <1813008+joshdover@users.noreply.github.com>

* use canary pacakge instead of git commit

Co-authored-by: Josh Dover <me@joshdover.com>
Co-authored-by: Josh Dover <1813008+joshdover@users.noreply.github.com>
Co-authored-by: Gidi Meir Morris <github@gidi.io>
Co-authored-by: Nathan Reese <reese.nathan@gmail.com>
Co-authored-by: Wylie Conlon <wylieconlon@gmail.com>
Co-authored-by: CJ Cenizal <cj@cenizal.com>
Co-authored-by: Aleh Zasypkin <aleh.zasypkin@gmail.com>
Co-authored-by: Dario Gieselaar <dario.gieselaar@elastic.co>
Co-authored-by: restrry <restrry@gmail.com>
Co-authored-by: James Gowdy <jgowdy@elastic.co>
Co-authored-by: John Schulz <github.com@jfsiii.org>
Co-authored-by: Alison Goryachev <alisonmllr20@gmail.com>
2021-03-25 04:47:16 -04:00