kibana/x-pack/plugins/fleet
Mikhail Shustov 3c8fa527a7
[ES] Upgrade client to v8.0 (#113950)
* bump to a pre-8.0 version

* export KibanaClient from /lib sub-folder

* workaround the problem of the absence of estypes

* update es client usage in pacakges

* export estypes from another path

* import errors from root

* import errors from root 2

* update transport import

* update import path for /api/types

* update import path for /api/types

* import errors from top export

* use TransportResult instead if ApiResponse

* fix errors in client_config

* fix src/core/server/saved_objects/migrationsv2/actions/integration_tests/actions.test.ts

* use KibanaClient in mock. we dont export the original Client

* fix client mocks

* fix errors on SO

* fix remaining core errors

* update estype import path

* fix errors in data plugin

* fix data_views

* fix es_ui_shared

* fix errors in interactive_setup

* fix errors in ./test folder

* add @elastic/transport to the runtime deps

* fix errors in packages

* fix erros in src/core

* fix errors in test/

* fix an error in actions plugin

* woraround and fix errors in APM plugin

* fix errors in canvas

* fix errors in event_log

* fix errors in fleet

* fix errors in ILM

* fix errors in infra

* fix errors in ingest_pipeline

* fix errors in lens

* fix errors in license_management

* fix errors in licensing

* fix errors in logstash

* fix errors in ml

* fix errors in monitoring

* fix errors in observability

* fix errors in rule_registry

* fix errors in reporting

* fix errors in rule_registry

* fix errors in security

* fix errors in security_solution

* fix errors in snapshot_restore

* fix errors in transform

* fix errors in UA

* fix errors in uptime

* fix errors in x-pack/test

* fix eslint errors

* fix new errors

* use default HTTP Connection. Undici does not support agent config options keepAlive and maxSockets

* create does not accept require_alias option

* update deps

* use transport types exported from ES client package

* fix ErrorCause | string errors

* do not use enum

* fix errors in data plugin

* update x-pack code

* fix transport

* fix apm search request

* do not crash on reporting

* fix kbn-test build

* mute reporting error to start

* fix ftr build

* another attempt

* update import path

* address or mute new errors

* REMOVE me. pin transport version temporarily.

* remove deep imports from transport package

* fix jest crash

* fix product check tests

* remove unnecessary ts-expect-error

* fix a few failed unit tests

* bump to canary 24

* remove unnecessary ts-expect-error

* remove dependency on transport

* fix types in tests

* mute errors in xpack tests

* product check doesn;t  spam in logs anymore

* filterPath --> filter_path

* ignoreUnavailable --> ignore_unavailable

* ignoreUnavailable --> ignore_unavailable

* trackScores --> track_scores

* trackTotalHits --> track_total_hits

* fix es-arcives

* fix data plugin crashes

* fix watcher test utils

* rollback unnecessary changes

* fix another problem in es-archiver

* fix scroll. for whatever reason scroll fails when request scroll_id in body

* add meta: true in kbn-securitysolution-es-utils

* bump client to canary 25

* fix errors in accordance with the es client spec

* update securityscolution-es-utils

* unify scroll api in reporting and fix tests

* fix unit tests in watcher

* refactor APM to abort request with AbortController API

* fix missing es client calls in tests

* fix missing meta in detection engine FTR tests

* fix another bunch of errors in js tests

* fix wrong coercion

* remove test-grep pattern

* fix apm unit test

* rename terminateAfter to terminate_after in infra plugin

* rename terminateAfter to terminate_after in uptime plugin

* rename terminateAfter to terminate_after in apm plugin

* fix security roles FTR tests

* fix reference

* fix post_privilidges test

* fix post_privilidges

* bump client to 26

* add meta for index_management test helpers

* remove ts-expect-error caused by bad type in reason

* bump client to 27

* REMOVE me. workaround until fixed in the es client

* fix incorrect type casting

* swtich from camelCase params

* use `HttpConnection` for FTR-related clients

* bump client to 29

* Revert "REMOVE me. workaround until fixed in the es client"

This reverts commit c038850c09.

* fix new util

* revert repository changes

* do not crash if cannot store event_loop data

* fix new estypes imports

* fix more types

* fix security test types and add ts-ignore for custom ES client

* fix more estypes imports

* yet more ts violations

* line by line fixing is hard

* adapt `evaluateAlert` from infra as it's also used from FTR tests

* use convertToKibanaClient in FTR test instead of meta:true in plugin code

* migrate from deprecated API in fleet

* fix intergration tests

* fix fleet tests

* fix another fleet test

* fix more tests

* let's call it a day

* Removes custom header check on 404 responses, includes es client ProductNotSupportedError in EsUnavailableError conditional (#116029)

* Removes custom header check on 404 responses, includes es client ProductNotSupportedError in EsUnavailableError conditional

* Updates proxy response integration test

* disable APM until compatible with client v8

* skip async_search FTR test

* use kbnClient in integration tests

* bump version to 29

* bump to 30

* have configureClient return a KibanaClient instead of Client, remove resolved violations.

* bump to 31

* bump to 31

* Revert "bump to 31"

This reverts commit 5ac713e640.

* trigger stop to unusubscribe

* update generated docs

* remove obsolete test

* put "as" back

* cleanup

* skip test

* remove new type errors in apm package

* remove ErrorCause casting

* update a comment

* bump version to 32

* remove unnecessary ts-expect-error in apm code

* update comments

* update to client v33

* remove outdated type definition

* bump to 34 without params mutation

* unskip the test that should not fail anymore

* remove unnecessary ts-expect-error comments

* update to v35. body can be string

* move `sort` to body and use body friendly syntax

* fix a failing test. maps register the same SO that has been already registered by home

Co-authored-by: pgayvallet <pierre.gayvallet@gmail.com>
Co-authored-by: Christiane (Tina) Heiligers <christiane.heiligers@elastic.co>
2021-10-26 14:08:22 +02:00
..
common [ES] Upgrade client to v8.0 (#113950) 2021-10-26 14:08:22 +02:00
dev_docs Add dev_docs for Fleet data model (#109476) 2021-09-07 15:45:31 +02:00
public [Fleet] Add link to integration data retention documentation (#115353) 2021-10-25 19:54:11 +01:00
server [ES] Upgrade client to v8.0 (#113950) 2021-10-26 14:08:22 +02:00
storybook [FLEET] Adding support for installing ML models (#107710) 2021-10-15 13:06:57 -04:00
CHANGELOG.md
jest.config.js [fleet] Add component story smoke test (#113634) 2021-10-04 14:49:44 -07:00
kibana.json [Fleet] Add telemetry for integration cards (#115413) 2021-10-19 09:59:29 -04:00
package.json
README.md [Fleet] Add containerized fleet server instructions to Fleet README (#105669) 2021-07-14 18:59:38 -04:00
tsconfig.json [fleet] Introduce Storybook to Fleet (#112611) 2021-09-21 08:23:13 -05:00

Fleet

Plugin

  • The plugin is enabled by default. See the TypeScript type for the the available plugin configuration options
  • Adding xpack.fleet.enabled=false will disable the plugin including the EPM and Fleet features. It will also remove the PACKAGE_POLICY_API_ROUTES and AGENT_POLICY_API_ROUTES values in common/constants/routes.ts
  • Adding --xpack.fleet.agents.enabled=false will disable the Fleet API & UI
  • Both EPM and Fleet require ingestManager be enabled. They are not standalone features.
  • For Enterprise license, a custom package registry URL can be used by setting xpack.fleet.registryUrl=http://localhost:8080
    • This property is currently only for internal Elastic development and is unsupported

Fleet Requirements

Fleet needs to have Elasticsearch API keys enabled, and also to have TLS enabled on kibana, (if you want to run Kibana without TLS you can provide the following config flag --xpack.fleet.agents.tlsCheckDisabled=false)

Also you need to configure the hosts your agent is going to use to comunication with Elasticsearch and Kibana (Not needed if you use Elastic cloud). You can use the following flags:

--xpack.fleet.agents.elasticsearch.host=http://localhost:9200
--xpack.fleet.agents.kibana.host=http://localhost:5601

Development

Getting started

See the Kibana docs for how to set up your dev environment, run Elasticsearch, and start Kibana

One common development workflow is:

  • Bootstrap Kibana
    yarn kbn bootstrap
    
  • Start Elasticsearch in one shell
    yarn es snapshot -E xpack.security.authc.api_key.enabled=true
    
  • Start Kibana in another shell
    yarn start --xpack.fleet.enabled=true --no-base-path
    

This plugin follows the common, server, public structure from the Architecture Style Guide . We also follow the pattern of developing feature branches under your personal fork of Kibana.

Note: The plugin was previously named Ingest Manager it's possible that some variables are still named with that old plugin name.

Running Fleet Server Locally in a Container

It can be useful to run Fleet Server in a container on your local machine in order to free up your actual "bare metal" machine to run Elastic Agent for testing purposes. Otherwise, you'll only be able to a single instance of Elastic Agent dedicated to Fleet Server on your local machine, and this can make testing integrations and policies difficult.

The following is adapted from the Fleet Server README

  1. Add the following configuration to your kibana.dev.yml
server.host: 0.0.0.0
  1. Append the following option to the command you use to start Elasticsearch
-E http.host=0.0.0.0

This command should look something like this:

yarn es snapshot --license trial -E xpack.security.authc.api_key.enabled=true -E path.data=/tmp/es-data -E http.host=0.0.0.0
  1. Run the Fleet Server Docker container. Make sure you include a BASE-PATH value if your local Kibana instance is using one. YOUR-IP should correspond to the IP address used by your Docker network to represent the host. For Windows and Mac machines, this should be 192.168.65.2. If you're not sure what this IP should be, run the following to look it up:
docker run -it --rm alpine nslookup host.docker.internal

To run the Fleet Server Docker container:

docker run -e KIBANA_HOST=http://{YOUR-IP}:5601/{BASE-PATH} -e KIBANA_USERNAME=elastic -e KIBANA_PASSWORD=changeme -e ELASTICSEARCH_HOST=http://{YOUR-IP}:9200 -e ELASTICSEARCH_USERNAME=elastic -e ELASTICSEARCH_PASSWORD=changeme -e KIBANA_FLEET_SETUP=1 -e FLEET_SERVER_ENABLE=1 -e FLEET_SERVER_INSECURE_HTTP=1 -p 8220:8220 docker.elastic.co/beats/elastic-agent:{VERSION}

Ensure you provide the -p 8220:8220 port mapping to map the Fleet Server container's port 8220 to your local machine's port 8220 in order for Fleet to communicate with Fleet Server.

For the latest version, use 8.0.0-SNAPSHOT. Otherwise, you can explore the available versions at https://www.docker.elastic.co/r/beats/elastic-agent.

Once the Fleet Server container is running, you should be able to treat it as if it were a local process running on http://localhost:8220 when configuring Fleet via the UI. You can then run elastic-agent on your local machine directly for testing purposes.

Tests

API integration tests

You need to have docker to run ingest manager api integration tests

  1. In one terminal, run the tests from the Kibana root directory with

    FLEET_PACKAGE_REGISTRY_PORT=12345 yarn test:ftr:server --config x-pack/test/fleet_api_integration/config.ts
    
  2. in a second terminal, run the tests from the Kibana root directory with

    FLEET_PACKAGE_REGISTRY_PORT=12345 yarn test:ftr:runner --config x-pack/test/fleet_api_integration/config.ts
    

    Optionally you can filter which tests you want to run using --grep

    FLEET_PACKAGE_REGISTRY_PORT=12345 yarn test:ftr:runner --config x-pack/test/fleet_api_integration/config.ts --grep='fleet'
    

Note you can also supply which docker image to use for the package registry via the FLEET_PACKAGE_REGISTRY_DOCKER_IMAGE env variable. For example,

FLEET_PACKAGE_REGISTRY_DOCKER_IMAGE='docker.elastic.co/package-registry/distribution:production' FLEET_PACKAGE_REGISTRY_PORT=12345 yarn test:ftr:runner