2019-07-16 17:53:51 +02:00
[[timelion]]
2019-08-08 23:17:01 +02:00
== Timelion
2019-07-16 17:53:51 +02:00
Timelion is a time series data visualizer that enables you to combine totally
independent data sources within a single visualization. It's driven by a simple
expression language you use to retrieve time series data, perform calculations
to tease out the answers to complex questions, and visualize the results.
For example, Timelion enables you to easily get the answers to questions like:
* <<time-series-intro, What is the real-time percentage of CPU time spent in user space to the results offset by one hour?>>
* <<mathematical-functions-intro, What does my inbound and outbound network traffic look like?>>
* <<timelion-conditional-intro, How much memory is my system actually using?>>
[float]
[[time-series-before-you-begin]]
=== Before you begin
In this tutorial, you'll use the time series data from https://www.elastic.co/guide/en/beats/metricbeat/current/index.html[Metricbeat]. To ingest the data locally, link:https://www.elastic.co/downloads/beats/metricbeat[download Metricbeat].
2019-07-22 17:07:11 +02:00
[float]
[[time-series-intro]]
2019-08-08 23:17:01 +02:00
=== Create time series visualizations
2019-07-22 17:07:11 +02:00
To compare the real-time percentage of CPU time spent in user space to the results offset by one hour, create a time series visualization.
2019-07-16 17:53:51 +02:00
[float]
[[time-series-define-functions]]
2019-08-08 23:17:01 +02:00
==== Define the functions
2019-07-16 17:53:51 +02:00
To start tracking the real-time percentage of CPU, enter the following in the *Timelion Expression* field:
[source,text]
----------------------------------
.es(index=metricbeat-*, timefield='@timestamp', metric='avg:system.cpu.user.pct')
----------------------------------
2019-08-08 23:17:01 +02:00
[role="screenshot"]
2019-07-16 17:53:51 +02:00
image::images/timelion-create01.png[]
{nbsp}
[float]
[[time-series-compare-data]]
2019-08-08 23:17:01 +02:00
==== Compare the data
2019-07-16 17:53:51 +02:00
To compare the two data sets, add another series with data from the previous hour, separated by a comma:
[source,text]
----------------------------------
2019-08-22 20:55:41 +02:00
.es(index=metricbeat-*,
timefield='@timestamp',
metric='avg:system.cpu.user.pct'),
.es(offset=-1h, <1>
index=metricbeat-*,
timefield='@timestamp',
metric='avg:system.cpu.user.pct')
2019-07-16 17:53:51 +02:00
----------------------------------
<1> `offset` offsets the data retrieval by a date expression. In this example, `-1h` offsets the data back by one hour.
2019-08-08 23:17:01 +02:00
[role="screenshot"]
2019-07-16 17:53:51 +02:00
image::images/timelion-create02.png[]
{nbsp}
[float]
[[time-series-add-labels]]
2019-08-08 23:17:01 +02:00
==== Add label names
2019-07-16 17:53:51 +02:00
To easily distinguish between the two data sets, add the label names:
[source,text]
----------------------------------
.es(offset=-1h,index=metricbeat-*, timefield='@timestamp', metric='avg:system.cpu.user.pct').label('last hour'), .es(index=metricbeat-*, timefield='@timestamp', metric='avg:system.cpu.user.pct').label('current hour') <1>
----------------------------------
<1> `.label()` adds custom labels to the visualization.
2019-08-08 23:17:01 +02:00
[role="screenshot"]
2019-07-16 17:53:51 +02:00
image::images/timelion-create03.png[]
{nbsp}
[float]
[[time-series-title]]
2019-08-08 23:17:01 +02:00
==== Add a title
2019-07-16 17:53:51 +02:00
Add a meaningful title:
[source,text]
----------------------------------
2019-08-22 20:55:41 +02:00
.es(offset=-1h,
index=metricbeat-*,
timefield='@timestamp',
metric='avg:system.cpu.user.pct')
.label('last hour'),
.es(index=metricbeat-*,
timefield='@timestamp',
metric='avg:system.cpu.user.pct')
.label('current hour')
.title('CPU usage over time') <1>
2019-07-16 17:53:51 +02:00
----------------------------------
<1> `.title()` adds a title with a meaningful name. Titles make is easier for unfamiliar users to understand the purpose of the visualization.
2019-08-08 23:17:01 +02:00
[role="screenshot"]
2019-07-16 17:53:51 +02:00
image::images/timelion-customize01.png[]
{nbsp}
[float]
[[time-series-change-chart-type]]
2019-08-08 23:17:01 +02:00
==== Change the chart type
2019-07-16 17:53:51 +02:00
To differentiate between the current hour data and the last hour data, change the chart type:
[source,text]
----------------------------------
2019-08-22 20:55:41 +02:00
.es(offset=-1h,
index=metricbeat-*,
timefield='@timestamp',
metric='avg:system.cpu.user.pct')
.label('last hour')
.lines(fill=1,width=0.5), <1>
.es(index=metricbeat-*,
timefield='@timestamp',
metric='avg:system.cpu.user.pct')
.label('current hour')
.title('CPU usage over time')
2019-07-16 17:53:51 +02:00
----------------------------------
<1> `.lines()` changes the appearance of the chart lines. In this example, `.lines(fill=1,width=0.5)` sets the fill level to `1`, and the border width to `0.5`.
2019-08-08 23:17:01 +02:00
[role="screenshot"]
2019-07-16 17:53:51 +02:00
image::images/timelion-customize02.png[]
{nbsp}
[float]
[[time-series-change-color]]
2019-08-08 23:17:01 +02:00
==== Change the line colors
2019-07-16 17:53:51 +02:00
To make the current hour data stand out, change the line colors:
[source,text]
----------------------------------
2019-08-22 20:55:41 +02:00
.es(offset=-1h,
index=metricbeat-*,
timefield='@timestamp',
metric='avg:system.cpu.user.pct')
.label('last hour')
.lines(fill=1,width=0.5)
.color(gray), <1>
.es(index=metricbeat-*,
timefield='@timestamp',
metric='avg:system.cpu.user.pct')
.label('current hour')
.title('CPU usage over time')
.color(#1E90FF)
2019-07-16 17:53:51 +02:00
----------------------------------
<1> `.color()` changes the color of the data. Supported color types include standard color names, hexadecimal values, or a color schema for grouped data. In this example, `.color(gray)` represents the last hour, and `.color(#1E90FF)` represents the current hour.
2019-08-08 23:17:01 +02:00
[role="screenshot"]
2019-07-16 17:53:51 +02:00
image::images/timelion-customize03.png[]
{nbsp}
[float]
[[time-series-adjust-legend]]
2019-08-08 23:17:01 +02:00
==== Make adjustments to the legend
2019-07-16 17:53:51 +02:00
Change the position and style of the legend:
[source,text]
----------------------------------
.es(offset=-1h,index=metricbeat-*, timefield='@timestamp', metric='avg:system.cpu.user.pct').label('last hour').lines(fill=1,width=0.5).color(gray), .es(index=metricbeat-*, timefield='@timestamp', metric='avg:system.cpu.user.pct').label('current hour').title('CPU usage over time').color(#1E90FF).legend(columns=2, position=nw) <1>
----------------------------------
<1> `.legend()` sets the position and style of the legend. In this example, `.legend(columns=2, position=nw)` places the legend in the north west position of the visualization with two columns.
2019-08-08 23:17:01 +02:00
[role="screenshot"]
2019-07-16 17:53:51 +02:00
image::images/timelion-customize04.png[]
{nbsp}
[float]
[[mathematical-functions-intro]]
2019-08-08 23:17:01 +02:00
=== Create visualizations with mathematical functions
2019-07-16 17:53:51 +02:00
To create a visualization for inbound and outbound network traffic, use mathematical functions.
[float]
[[mathematical-functions-define-functions]]
2019-08-08 23:17:01 +02:00
==== Define the functions
2019-07-16 17:53:51 +02:00
To start tracking the inbound and outbound network traffic, enter the following in the *Timelion Expression* field:
[source,text]
----------------------------------
.es(index=metricbeat*, timefield=@timestamp, metric=max:system.network.in.bytes)
----------------------------------
2019-08-08 23:17:01 +02:00
[role="screenshot"]
2019-07-16 17:53:51 +02:00
image::images/timelion-math01.png[]
{nbsp}
[float]
[[mathematical-functions-plot-change]]
2019-08-08 23:17:01 +02:00
==== Plot the rate of change
2019-07-16 17:53:51 +02:00
Change how the data is displayed so that you can easily monitor the inbound traffic:
[source,text]
----------------------------------
.es(index=metricbeat*, timefield=@timestamp, metric=max:system.network.in.bytes).derivative() <1>
----------------------------------
<1> `.derivative` plots the change in values over time.
2019-08-08 23:17:01 +02:00
[role="screenshot"]
2019-07-16 17:53:51 +02:00
image::images/timelion-math02.png[]
{nbsp}
Add a similar calculation for outbound traffic:
[source,text]
----------------------------------
.es(index=metricbeat*, timefield=@timestamp, metric=max:system.network.in.bytes).derivative(), .es(index=metricbeat*, timefield=@timestamp, metric=max:system.network.out.bytes).derivative().multiply(-1) <1>
----------------------------------
<1> `.multiply()` multiplies the data series by a number, the result of a data series, or a list of data series. For this example, `.multiply(-1)` converts the outbound network traffic to a negative value since the outbound network traffic is leaving your machine.
2019-08-08 23:17:01 +02:00
[role="screenshot"]
2019-07-16 17:53:51 +02:00
image::images/timelion-math03.png[]
{nbsp}
[float]
[[mathematical-functions-convert-data]]
2019-08-08 23:17:01 +02:00
==== Change the data metric
2019-07-16 17:53:51 +02:00
To make the visualization easier to analyze, change the data metric from bytes to megabytes:
[source,text]
----------------------------------
.es(index=metricbeat*, timefield=@timestamp, metric=max:system.network.in.bytes).derivative().divide(1048576), .es(index=metricbeat*, timefield=@timestamp, metric=max:system.network.out.bytes).derivative().multiply(-1).divide(1048576) <1>
----------------------------------
<1> `.divide()` accepts the same input as `.multiply()`, then divides the data series by the defined divisor.
2019-08-08 23:17:01 +02:00
[role="screenshot"]
2019-07-16 17:53:51 +02:00
image::images/timelion-math04.png[]
{nbsp}
[float]
[[mathematical-functions-add-labels]]
2019-08-08 23:17:01 +02:00
==== Customize and format the visualization
2019-07-16 17:53:51 +02:00
Customize and format the visualization using functions:
[source,text]
----------------------------------
2019-08-22 20:55:41 +02:00
.es(index=metricbeat*,
timefield=@timestamp,
metric=max:system.network.in.byte)
.derivative()
.divide(1048576)
.lines(fill=2, width=1)
.color(green)
.label("Inbound traffic") <1>
.title("Network traffic (MB/s)"), <2>
.es(index=metricbeat*,
timefield=@timestamp,
metric=max:system.network.out.bytes)
.derivative()
.multiply(-1)
.divide(1048576)
.lines(fill=2, width=1) <3>
.color(blue) < <4>
.label("Outbound traffic")
.legend(columns=2, position=nw) <5>
2019-07-16 17:53:51 +02:00
----------------------------------
<1> `.label()` adds custom labels to the visualization.
<2> `.title()` adds a title with a meaningful name.
<3> `.lines()` changes the appearance of the chart lines. In this example, `.lines(fill=2, width=1)` sets the fill level to `2`, and the border width to `1`.
<4> `.color()` changes the color of the data. Supported color types include standard color names, hexadecimal values, or a color schema for grouped data. In this example, `.color(green)` represents the inbound network traffic, and `.color(blue)` represents the outbound network traffic.
<5> `.legend()` sets the position and style of the legend. For this example, `legend(columns=2, position=nw)` places the legend in the north west position of the visualization with two columns.
2019-08-08 23:17:01 +02:00
[role="screenshot"]
2019-07-16 17:53:51 +02:00
image::images/timelion-math05.png[]
{nbsp}
[float]
[[timelion-conditional-intro]]
2019-08-08 23:17:01 +02:00
=== Create visualizations with conditional logic and tracking trends
2019-07-16 17:53:51 +02:00
To easily detect outliers and discover patterns over time, modify time series data with conditional logic and create a trend with a moving average.
With Timelion conditional logic, you can use the following operator values to compare your data:
[horizontal]
`eq`:: equal
`ne`:: not equal
`lt`:: less than
`lte`:: less than or equal to
`gt`:: greater than
`gte`:: greater than or equal to
[float]
[[conditional-define-functions]]
2019-08-08 23:17:01 +02:00
==== Define the functions
2019-07-16 17:53:51 +02:00
To chart the maximum value of `system.memory.actual.used.bytes`, enter the following in the *Timelion Expression* field:
[source,text]
----------------------------------
.es(index=metricbeat-*, timefield='@timestamp', metric='max:system.memory.actual.used.bytes')
----------------------------------
2019-08-08 23:17:01 +02:00
[role="screenshot"]
2019-07-16 17:53:51 +02:00
image::images/timelion-conditional01.png[]
{nbsp}
[float]
[[conditional-track-memory]]
2019-08-08 23:17:01 +02:00
==== Track used memory
2019-07-16 17:53:51 +02:00
To track the amount of memory used, create two thresholds:
[source,text]
----------------------------------
2019-08-22 20:55:41 +02:00
.es(index=metricbeat-*,
timefield='@timestamp',
metric='max:system.memory.actual.used.bytes'),
.es(index=metricbeat-*,
timefield='@timestamp',
metric='max:system.memory.actual.used.bytes')
.if(gt, <1>
11300000000, <2>
.es(index=metricbeat-*,
timefield='@timestamp',
metric='max:system.memory.actual.used.bytes'),
null)
.label('warning')
.color('#FFCC11'),
.es(index=metricbeat-*,
timefield='@timestamp',
metric='max:system.memory.actual.used.bytes')
.if(gt,
11375000000,
.es(index=metricbeat-*,
timefield='@timestamp',
metric='max:system.memory.actual.used.bytes'),
null)
.label('severe')
.color('red')
2019-07-16 17:53:51 +02:00
----------------------------------
<1> Timelion conditional logic for the _greater than_ operator. In this example, the warning threshold is 11.3GB (`11300000000`), and the severe threshold is 11.375GB (`11375000000`). If the threshold values are too high or low for your machine, adjust the values accordingly.
<2> `if()` compares each point to a number. If the condition evaluates to `true`, adjust the styling. If the condition evaluates to `false`, use the default styling.
2019-08-08 23:17:01 +02:00
[role="screenshot"]
2019-07-16 17:53:51 +02:00
image::images/timelion-conditional02.png[]
{nbsp}
[float]
[[conditional-determine-trend]]
2019-08-08 23:17:01 +02:00
==== Determine the trend
2019-07-16 17:53:51 +02:00
To determine the trend, create a new data series:
[source,text]
----------------------------------
.es(index=metricbeat-*, timefield='@timestamp', metric='max:system.memory.actual.used.bytes'), .es(index=metricbeat-*, timefield='@timestamp', metric='max:system.memory.actual.used.bytes').if(gt,11300000000,.es(index=metricbeat-*, timefield='@timestamp', metric='max:system.memory.actual.used.bytes'),null).label('warning').color('#FFCC11'), .es(index=metricbeat-*, timefield='@timestamp', metric='max:system.memory.actual.used.bytes').if(gt,11375000000,.es(index=metricbeat-*, timefield='@timestamp', metric='max:system.memory.actual.used.bytes'),null).label('severe').color('red'), .es(index=metricbeat-*, timefield='@timestamp', metric='max:system.memory.actual.used.bytes').mvavg(10) <1>
----------------------------------
<1> `mvavg()` calculates the moving average over a specified period of time. In this example, `.mvavg(10)` creates a moving average with a window of 10 data points.
2019-08-08 23:17:01 +02:00
[role="screenshot"]
2019-07-16 17:53:51 +02:00
image::images/timelion-conditional03.png[]
{nbsp}
[float]
[[conditional-format-visualization]]
2019-08-08 23:17:01 +02:00
==== Customize and format the visualization
2019-07-16 17:53:51 +02:00
Customize and format the visualization using functions:
[source,text]
----------------------------------
2019-08-22 20:55:41 +02:00
.es(index=metricbeat-*,
timefield='@timestamp',
metric='max:system.memory.actual.used.bytes')
.label('max memory') <1>
.title('Memory consumption over time'), <2>
.es(index=metricbeat-*,
timefield='@timestamp',
metric='max:system.memory.actual.used.bytes')
.if(gt,
11300000000,
.es(index=metricbeat-*,
timefield='@timestamp',
metric='max:system.memory.actual.used.bytes'),
null)
.label('warning')
.color('#FFCC11') <3>
.lines(width=5), <4>
.es(index=metricbeat-*,
timefield='@timestamp',
metric='max:system.memory.actual.used.bytes')
.if(gt,
11375000000,
.es(index=metricbeat-*,
timefield='@timestamp',
metric='max:system.memory.actual.used.bytes'),
null)
.label('severe')
.color('red')
.lines(width=5),
.es(index=metricbeat-*,
timefield='@timestamp',
metric='max:system.memory.actual.used.bytes')
.mvavg(10)
.label('mvavg')
.lines(width=2)
.color(#5E5E5E)
.legend(columns=4, position=nw) <5>
2019-07-16 17:53:51 +02:00
----------------------------------
<1> `.label()` adds custom labels to the visualization.
<2> `.title()` adds a title with a meaningful name.
<3> `.color()` changes the color of the data. Supported color types include standard color names, hexadecimal values, or a color schema for grouped data.
<4> `.lines()` changes the appearance of the chart lines. In this example, .lines(width=5) sets border width to `5`.
<5> `.legend()` sets the position and style of the legend. For this example, `(columns=4, position=nw)` places the legend in the north west position of the visualization with four columns.
2019-08-08 23:17:01 +02:00
[role="screenshot"]
2019-07-16 17:53:51 +02:00
image::images/timelion-conditional04.png[]
{nbsp}
For additional information on Timelion conditional capabilities, go to https://www.elastic.co/blog/timeseries-if-then-else-with-timelion[I have but one .condition()].