3c8fa527a7
* bump to a pre-8.0 version * export KibanaClient from /lib sub-folder * workaround the problem of the absence of estypes * update es client usage in pacakges * export estypes from another path * import errors from root * import errors from root 2 * update transport import * update import path for /api/types * update import path for /api/types * import errors from top export * use TransportResult instead if ApiResponse * fix errors in client_config * fix src/core/server/saved_objects/migrationsv2/actions/integration_tests/actions.test.ts * use KibanaClient in mock. we dont export the original Client * fix client mocks * fix errors on SO * fix remaining core errors * update estype import path * fix errors in data plugin * fix data_views * fix es_ui_shared * fix errors in interactive_setup * fix errors in ./test folder * add @elastic/transport to the runtime deps * fix errors in packages * fix erros in src/core * fix errors in test/ * fix an error in actions plugin * woraround and fix errors in APM plugin * fix errors in canvas * fix errors in event_log * fix errors in fleet * fix errors in ILM * fix errors in infra * fix errors in ingest_pipeline * fix errors in lens * fix errors in license_management * fix errors in licensing * fix errors in logstash * fix errors in ml * fix errors in monitoring * fix errors in observability * fix errors in rule_registry * fix errors in reporting * fix errors in rule_registry * fix errors in security * fix errors in security_solution * fix errors in snapshot_restore * fix errors in transform * fix errors in UA * fix errors in uptime * fix errors in x-pack/test * fix eslint errors * fix new errors * use default HTTP Connection. Undici does not support agent config options keepAlive and maxSockets * create does not accept require_alias option * update deps * use transport types exported from ES client package * fix ErrorCause | string errors * do not use enum * fix errors in data plugin * update x-pack code * fix transport * fix apm search request * do not crash on reporting * fix kbn-test build * mute reporting error to start * fix ftr build * another attempt * update import path * address or mute new errors * REMOVE me. pin transport version temporarily. * remove deep imports from transport package * fix jest crash * fix product check tests * remove unnecessary ts-expect-error * fix a few failed unit tests * bump to canary 24 * remove unnecessary ts-expect-error * remove dependency on transport * fix types in tests * mute errors in xpack tests * product check doesn;t spam in logs anymore * filterPath --> filter_path * ignoreUnavailable --> ignore_unavailable * ignoreUnavailable --> ignore_unavailable * trackScores --> track_scores * trackTotalHits --> track_total_hits * fix es-arcives * fix data plugin crashes * fix watcher test utils * rollback unnecessary changes * fix another problem in es-archiver * fix scroll. for whatever reason scroll fails when request scroll_id in body * add meta: true in kbn-securitysolution-es-utils * bump client to canary 25 * fix errors in accordance with the es client spec * update securityscolution-es-utils * unify scroll api in reporting and fix tests * fix unit tests in watcher * refactor APM to abort request with AbortController API * fix missing es client calls in tests * fix missing meta in detection engine FTR tests * fix another bunch of errors in js tests * fix wrong coercion * remove test-grep pattern * fix apm unit test * rename terminateAfter to terminate_after in infra plugin * rename terminateAfter to terminate_after in uptime plugin * rename terminateAfter to terminate_after in apm plugin * fix security roles FTR tests * fix reference * fix post_privilidges test * fix post_privilidges * bump client to 26 * add meta for index_management test helpers * remove ts-expect-error caused by bad type in reason * bump client to 27 * REMOVE me. workaround until fixed in the es client * fix incorrect type casting * swtich from camelCase params * use `HttpConnection` for FTR-related clients * bump client to 29 * Revert "REMOVE me. workaround until fixed in the es client" This reverts commit |
||
---|---|---|
.. | ||
src | ||
BUILD.bazel | ||
jest.config.js | ||
package.json | ||
README.md | ||
tsconfig.json |
@elastic/apm-synthtrace
@elastic/apm-synthtrace
is an experimental tool to generate synthetic APM data. It is intended to be used for development and testing of the Elastic APM app in Kibana.
At a high-level, the module works by modeling APM events/metricsets with a fluent API. The models can then be serialized and converted to Elasticsearch documents. In the future we might support APM Server as an output as well.
Usage
This section assumes that you've installed Kibana's dependencies by running yarn kbn bootstrap
in the repository's root folder.
This library can currently be used in two ways:
- Imported as a Node.js module, for instance to be used in Kibana's functional test suite.
- With a command line interface, to index data based on a specified scenario.
Using the Node.js module
Concepts
Service
: a logical grouping for a monitored service. AService
object contains fields likeservice.name
,service.environment
andagent.name
.Instance
: a single instance of a monitored service. E.g., the workload for a monitored service might be spread across multiple containers. AnInstance
object contains fields likeservice.node.name
andcontainer.id
.Timerange
: an object that will return an array of timestamps based on an interval and a rate. These timestamps can be used to generate events/metricsets.Transaction
,Span
,APMError
andMetricset
: events/metricsets that occur on an instance. For more background, see the explanation of the APM data model
Example
import { service, timerange, toElasticsearchOutput } from '@elastic/apm-synthtrace';
const instance = service('synth-go', 'production', 'go').instance('instance-a');
const from = new Date('2021-01-01T12:00:00.000Z').getTime();
const to = new Date('2021-01-01T12:00:00.000Z').getTime();
const traceEvents = timerange(from, to)
.interval('1m')
.rate(10)
.flatMap((timestamp) =>
instance
.transaction('GET /api/product/list')
.timestamp(timestamp)
.duration(1000)
.success()
.children(
instance
.span('GET apm-*/_search', 'db', 'elasticsearch')
.timestamp(timestamp + 50)
.duration(900)
.destination('elasticsearch')
.success()
)
.serialize()
);
const metricsets = timerange(from, to)
.interval('30s')
.rate(1)
.flatMap((timestamp) =>
instance
.appMetrics({
'system.memory.actual.free': 800,
'system.memory.total': 1000,
'system.cpu.total.norm.pct': 0.6,
'system.process.cpu.total.norm.pct': 0.7,
})
.timestamp(timestamp)
.serialize()
);
const esEvents = toElasticsearchOutput(traceEvents.concat(metricsets));
Generating metricsets
@elastic/apm-synthtrace
can also automatically generate transaction metrics, span destination metrics and transaction breakdown metrics based on the generated trace events. If we expand on the previous example:
import {
getTransactionMetrics,
getSpanDestinationMetrics,
getBreakdownMetrics,
} from '@elastic/apm-synthtrace';
const esEvents = toElasticsearchOutput([
...traceEvents,
...getTransactionMetrics(traceEvents),
...getSpanDestinationMetrics(traceEvents),
...getBreakdownMetrics(traceEvents),
]);
CLI
Via the CLI, you can upload scenarios, either using a fixed time range or continuously generating data. Some examples are available in in src/scripts/examples
. Here's an example for live data:
$ node packages/elastic-apm-synthtrace/src/scripts/run packages/elastic-apm-generator/src/examples/01_simple_trace.ts --target=http://admin:changeme@localhost:9200 --live
For a fixed time window:
$ node packages/elastic-apm-synthtrace/src/scripts/run packages/elastic-apm-generator/src/examples/01_simple_trace.ts --target=http://admin:changeme@localhost:9200 --from=now-24h --to=now
The script will try to automatically find bootstrapped APM indices. If these indices do not exist, the script will exit with an error. It will not bootstrap the indices itself.
The following options are supported:
Option | Description | Default |
---|---|---|
--target |
Elasticsearch target, including username/password. | Required |
--from |
The start of the time window. | now - 15m |
--to |
The end of the time window. | now |
--live |
Continously ingest data | false |
--clean |
Clean APM indices before indexing new data. | false |
--workers |
Amount of Node.js worker threads | 5 |
--bucketSize |
Size of bucket for which to generate data. | 15m |
--interval |
The interval at which to index data. | 10s |
--clientWorkers |
Number of simultaneously connected ES clients | 5 |
--batchSize |
Number of documents per bulk index request | 1000 |
--logLevel |
Log level. | info |