kibana/docs/visualize/vega.asciidoc
Alexey Antonov d158692bf5
Vega is now GA (#75157)
* Bringing Vega out of experimental

Closes: #21721

* fix CI group 12

* remove experimental flag from doc

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-08-20 12:19:21 +03:00

1640 lines
41 KiB
Plaintext

[[vega-graph]]
== Vega
Build custom visualizations using Vega and Vega-Lite, backed by one or more
data sources including {es}, Elastic Map Service, URL,
or static data. Use the {kib} extensions to Vega to embed Vega into
your dashboard, and to add interactivity to the visualizations.
Vega and Vega-Lite are both declarative formats to create visualizations
using JSON. Both use a different syntax for declaring visualizations,
and are not fully interchangeable.
[float]
[[when-to-vega]]
=== When to use Vega
Vega and Vega-Lite are capable of building most of the visualizations
that {kib} provides, but with higher complexity. The most common reason
to use Vega in {kib} is that {kib} is missing support for the query or
visualization, for example:
* Aggregations using the `nested` or `parent/child` mapping
* Aggregations without a {kib} index pattern
* Queries using custom time filters
* Complex calculations
* Extracting data from _source instead of aggregation
* Scatter charts
* Sankey charts
* Custom maps
* Using a visual theme that {kib} does not provide
[[vega-lite-tutorial]]
=== Tutorial: First visualization in Vega-Lite
In this tutorial, you will learn about how to edit Vega-Lite in {kib} to create
a stacked area chart from an {es} search query. It will give you a starting point
for a more comprehensive
https://vega.github.io/vega-lite/tutorials/getting_started.html[introduction to Vega-Lite],
while only covering the basics.
In this tutorial, you will build a stacked area chart from one of the {kib} sample data
sets.
[role="screenshot"]
image::visualize/images/vega_lite_tutorial_1.png[]
Before beginning this tutorial, install the <<add-sample-data, eCommerce sample data>>
set.
When you first open the Vega editor in {kib}, you will see a pre-populated
line chart which shows the total number of documents across all your indices
within the time range.
[role="screenshot"]
image::visualize/images/vega_lite_default.png[]
The text editor contains a Vega-Lite spec written in https://hjson.github.io/[HJSON],
which is similar to JSON but optimized for human editing. HJSON supports:
* Comments using // or /* syntax
* Object keys without quotes
* String values without quotes
* Optional commas
* Double or single quotes
* Multiline strings
[float]
==== Small steps
Always work on Vega in the smallest steps possible, and save your work frequently.
Small changes will cause unexpected results. Click the "Save" button now.
The first step is to change the index to one of the <<add-sample-data, sample data>>
sets. Change
```yaml
index: _all
```
to:
```yaml
index: kibana_sample_data_ecommerce
```
Click "Update". The result is probably not what you expect. You should see a flat
line with 0 results.
You've only changed the index, so the difference must be the query is returning
no results. You can try the <<vega-browser-debugging-console, Vega debugging process>>,
but intuition may be faster for this particular problem.
In this case, the problem is that you are querying the field `@timestamp`,
which does not exist in the `kibana_sample_data_ecommerce` data. Find and replace
`@timestamp` with `order_date`. This fixes the problem, leaving you with this spec:
.Expand Vega-Lite spec
[%collapsible%closed]
====
[source,yaml]
----
{
$schema: https://vega.github.io/schema/vega-lite/v4.json
title: Event counts from ecommerce
data: {
url: {
%context%: true
%timefield%: order_date
index: kibana_sample_data_ecommerce
body: {
aggs: {
time_buckets: {
date_histogram: {
field: order_date
interval: {%autointerval%: true}
extended_bounds: {
min: {%timefilter%: "min"}
max: {%timefilter%: "max"}
}
min_doc_count: 0
}
}
}
size: 0
}
}
format: {property: "aggregations.time_buckets.buckets" }
}
mark: line
encoding: {
x: {
field: key
type: temporal
axis: { title: null }
}
y: {
field: doc_count
type: quantitative
axis: { title: "Document count" }
}
}
}
----
====
Now, let's make the visualization more interesting by adding another aggregation
to create a stacked area chart. To verify that you have constructed the right
query, it is easiest to use the {kib} Dev Tools in a separate tab from the
Vega editor. Open the Dev Tools from the Management section of the navigation.
This query is roughly equivalent to the one that is used in the default
Vega-Lite spec. Copy it into the Dev Tools:
```js
POST kibana_sample_data_ecommerce/_search
{
"query": {
"range": {
"order_date": {
"gte": "now-7d"
}
}
},
"aggs": {
"time_buckets": {
"date_histogram": {
"field": "order_date",
"fixed_interval": "1d",
"extended_bounds": {
"min": "now-7d"
},
"min_doc_count": 0
}
}
},
"size": 0
}
```
There's not enough data to create a stacked bar in the original query, so we
will add a new
{ref}/search-aggregations-bucket-terms-aggregation.html[terms aggregation]:
```js
POST kibana_sample_data_ecommerce/_search
{
"query": {
"range": {
"order_date": {
"gte": "now-7d"
}
}
},
"aggs": {
"categories": {
"terms": { "field": "category.keyword" },
"aggs": {
"time_buckets": {
"date_histogram": {
"field": "order_date",
"fixed_interval": "1d",
"extended_bounds": {
"min": "now-7d"
},
"min_doc_count": 0
}
}
}
}
},
"size": 0
}
```
You'll see that the response format looks different from the previous query:
```json
{
"aggregations" : {
"categories" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [{
"key" : "Men's Clothing",
"doc_count" : 1661,
"time_buckets" : {
"buckets" : [{
"key_as_string" : "2020-06-30T00:00:00.000Z",
"key" : 1593475200000,
"doc_count" : 19
}, {
"key_as_string" : "2020-07-01T00:00:00.000Z",
"key" : 1593561600000,
"doc_count" : 71
}]
}
}]
}
}
}
```
Now that we have data that we're happy with, it's time to convert from an
isolated {es} query into a query with {kib} integration. Looking at the
<<vega-queries, reference for writing {es} queries in Vega>>, you will
see the full list of special tokens that are used in this query, such
as `%context: true`. This query has also replaced `"fixed_interval": "1d"`
with `interval: {%autointerval%: true}`. Copy the final query into
your spec:
```yaml
data: {
url: {
%context%: true
%timefield%: order_date
index: kibana_sample_data_ecommerce
body: {
aggs: {
categories: {
terms: { field: "category.keyword" }
aggs: {
time_buckets: {
date_histogram: {
field: order_date
interval: {%autointerval%: true}
extended_bounds: {
min: {%timefilter%: "min"}
max: {%timefilter%: "max"}
}
min_doc_count: 0
}
}
}
}
}
size: 0
}
}
format: {property: "aggregations.categories.buckets" }
}
```
If you copy and paste that into your Vega-Lite spec, and click "Update",
you will see a warning saying `Infinite extent for field "key": [Infinity, -Infinity]`.
Let's use our <<vega-browser-debugging-console, Vega debugging skills>> to understand why.
Vega-Lite generates data using the names `source_0` and `data_0`. `source_0` contains
the results from the {es} query, and `data_0` contains the visually encoded results
which are shown in the chart. To debug this problem, you need to compare both.
To look at the source, open the browser dev tools console and type
`VEGA_DEBUG.view.data('source_0')`. You will see:
```js
[{
doc_count: 454
key: "Men's Clothing"
time_buckets: {buckets: Array(57)}
Symbol(vega_id): 12822
}, ...]
```
To compare to the visually encoded data, open the browser dev tools console and type
`VEGA_DEBUG.view.data('data_0')`. You will see:
```js
[{
doc_count: 454
key: NaN
time_buckets: {buckets: Array(57)}
Symbol(vega_id): 13879
}]
```
The issue seems to be that the `key` property is not being converted the right way,
which makes sense because the `key` is now `Men's Clothing` instead of a timestamp.
To fix this, try updating the `encoding` of your Vega-Lite spec to:
```yaml
encoding: {
x: {
field: time_buckets.buckets.key
type: temporal
axis: { title: null }
}
y: {
field: time_buckets.buckets.doc_count
type: quantitative
axis: { title: "Document count" }
}
}
```
This will show more errors, and you can inspect `VEGA_DEBUG.view.data('data_0')` to
understand why. This now shows:
```js
[{
doc_count: 454
key: "Men's Clothing"
time_buckets: {buckets: Array(57)}
time_buckets.buckets.doc_count: undefined
time_buckets.buckets.key: null
Symbol(vega_id): 14094
}]
```
It looks like the problem is that the `time_buckets` inner array is not being
extracted by Vega. The solution is to use a Vega-lite
https://vega.github.io/vega-lite/docs/flatten.html[flatten transformation], available in {kib} 7.9 and later.
If using an older version of Kibana, the flatten transformation is available in Vega
but not Vega-Lite.
Add this section in between the `data` and `encoding` section:
```yaml
transform: [{
flatten: ["time_buckets.buckets"]
}]
```
This does not yet produce the results you expect. Inspect the transformed data
by typing `VEGA_DEBUG.view.data('data_0')` into the console again:
```js
[{
doc_count: 453
key: "Men's Clothing"
time_bucket.buckets.doc_count: undefined
time_buckets: {buckets: Array(57)}
time_buckets.buckets: {
key_as_string: "2020-06-30T15:00:00.000Z",
key: 1593529200000,
doc_count: 2
}
time_buckets.buckets.key: null
Symbol(vega_id): 21564
}]
```
The debug view shows `undefined` values where you would expect to see numbers, and
the cause is that there are duplicate names which are confusing Vega-Lite. This can
be fixed by making this change to the `transform` and `encoding` blocks:
```yaml
transform: [{
flatten: ["time_buckets.buckets"],
as: ["buckets"]
}]
mark: area
encoding: {
x: {
field: buckets.key
type: temporal
axis: { title: null }
}
y: {
field: buckets.doc_count
type: quantitative
axis: { title: "Document count" }
}
color: {
field: key
type: nominal
}
}
```
At this point, you have a stacked area chart that shows the top categories,
but the chart is still missing some common features that we expect from a {kib}
visualization. Let's add hover states and tooltips next.
Hover states are handled differently in Vega-Lite and Vega. In Vega-Lite this is
done using a concept called `selection`, which has many permutations that are not
covered in this tutorial. We will be adding a simple tooltip and hover state.
Because {kib} has enabled the https://vega.github.io/vega-lite/docs/tooltip.html[Vega tooltip plugin],
tooltips can be defined in several ways:
* Automatic tooltip based on the data, via `{ content: "data" }`
* Array of fields, like `[{ field: "key", type: "nominal" }]`
* Defining a custom Javascript object using the `calculate` transform
For the simple tooltip, add this to your encoding:
```yaml
encoding: {
tooltip: [{
field: buckets.key
type: temporal
title: "Date"
}, {
field: key
type: nominal
title: "Category"
}, {
field: buckets.doc_count
type: quantitative
title: "Count"
}]
}
```
As you hover over the area series in your chart, a multi-line tooltip will
appear, but it won't indicate the nearest point that it's pointing to. To
indicate the nearest point, we need to add a second layer.
The first step is to remove the `mark: area` from your visualization.
Once you've removed the previous mark, add a composite mark at the end of
the Vega-Lite spec:
```yaml
layer: [{
mark: area
}, {
mark: point
}]
```
You'll see that the points are not appearing to line up with the area chart,
and the reason is that the points are not being stacked. Change your Y encoding
to this:
```yaml
y: {
field: buckets.doc_count
type: quantitative
axis: { title: "Document count" }
stack: true
}
```
Now, we will add a `selection` block inside the point mark:
```yaml
layer: [{
mark: area
}, {
mark: point
selection: {
pointhover: {
type: single
on: mouseover
clear: mouseout
empty: none
fields: ["buckets.key", "key"]
nearest: true
}
}
encoding: {
size: {
condition: {
selection: pointhover
value: 100
}
value: 5
}
fill: {
condition: {
selection: pointhover
value: white
}
}
}
}]
```
Now that you've enabled a selection, try moving the mouse around the visualization
and seeing the points respond to the nearest position:
[role="screenshot"]
image::visualize/images/vega_lite_tutorial_2.png[]
The final result of this tutorial is this spec:
.Expand final Vega-Lite spec
[%collapsible%closed]
====
[source,yaml]
----
{
$schema: https://vega.github.io/schema/vega-lite/v4.json
title: Event counts from ecommerce
data: {
url: {
%context%: true
%timefield%: order_date
index: kibana_sample_data_ecommerce
body: {
aggs: {
categories: {
terms: { field: "category.keyword" }
aggs: {
time_buckets: {
date_histogram: {
field: order_date
interval: {%autointerval%: true}
extended_bounds: {
min: {%timefilter%: "min"}
max: {%timefilter%: "max"}
}
min_doc_count: 0
}
}
}
}
}
size: 0
}
}
format: {property: "aggregations.categories.buckets" }
}
transform: [{
flatten: ["time_buckets.buckets"]
as: ["buckets"]
}]
encoding: {
x: {
field: buckets.key
type: temporal
axis: { title: null }
}
y: {
field: buckets.doc_count
type: quantitative
axis: { title: "Document count" }
stack: true
}
color: {
field: key
type: nominal
title: "Category"
}
tooltip: [{
field: buckets.key
type: temporal
title: "Date"
}, {
field: key
type: nominal
title: "Category"
}, {
field: buckets.doc_count
type: quantitative
title: "Count"
}]
}
layer: [{
mark: area
}, {
mark: point
selection: {
pointhover: {
type: single
on: mouseover
clear: mouseout
empty: none
fields: ["buckets.key", "key"]
nearest: true
}
}
encoding: {
size: {
condition: {
selection: pointhover
value: 100
}
value: 5
}
fill: {
condition: {
selection: pointhover
value: white
}
}
}
}]
}
----
====
[[vega-tutorial]]
=== Tutorial: Updating {kib} filters from Vega
In this tutorial you will build an area chart in Vega using an {es} search query,
and add a click handler and drag handler to update {kib} filters.
This tutorial is not a full https://vega.github.io/vega/tutorials/[Vega tutorial],
but will cover the basics of creating Vega visualizations into {kib}.
First, create an almost-blank Vega chart by pasting this into the editor:
```yaml
{
$schema: "https://vega.github.io/schema/vega/v5.json"
data: [{
name: source_0
}]
scales: [{
name: x
type: time
range: width
}, {
name: y
type: linear
range: height
}]
axes: [{
orient: bottom
scale: x
}, {
orient: left
scale: y
}]
marks: [
{
type: area
from: {
data: source_0
}
encode: {
update: {
}
}
}
]
}
```
Despite being almost blank, this Vega spec still contains the minimum requirements:
* Data
* Scales
* Marks
* (optional) Axes
Next, add a valid {es} search query in the `data` block:
```yaml
data: [
{
name: source_0
url: {
%context%: true
%timefield%: order_date
index: kibana_sample_data_ecommerce
body: {
aggs: {
time_buckets: {
date_histogram: {
field: order_date
fixed_interval: "3h"
extended_bounds: {
min: {%timefilter%: "min"}
max: {%timefilter%: "max"}
}
min_doc_count: 0
}
}
}
size: 0
}
}
format: { property: "aggregations.time_buckets.buckets" }
}
]
```
Click "Update", and nothing will change in the visualization. The first step
is to change the X and Y scales based on the data:
```yaml
scales: [{
name: x
type: time
range: width
domain: {
data: source_0
field: key
}
}, {
name: y
type: linear
range: height
domain: {
data: source_0
field: doc_count
}
}]
```
Click "Update", and you will see that the X and Y axes are now showing labels based
on the real data.
Next, encode the fields `key` and `doc_count` as the X and Y values:
```yaml
marks: [
{
type: area
from: {
data: source_0
}
encode: {
update: {
x: {
scale: x
field: key
}
y: {
scale: y
value: 0
}
y2: {
scale: y
field: doc_count
}
}
}
}
]
```
Click "Update" and you will get a basic area chart:
[role="screenshot"]
image::visualize/images/vega_tutorial_3.png[]
Next, add a new block to the `marks` section. This will show clickable points to filter for a specific
date:
```yaml
{
name: point
type: symbol
style: ["point"]
from: {
data: source_0
}
encode: {
update: {
x: {
scale: x
field: key
}
y: {
scale: y
field: doc_count
}
size: {
value: 100
}
fill: {
value: black
}
}
}
}
```
Next, we will create a Vega signal to make the points clickable. You can access
the clicked `datum` in the expression used to update. In this case, you want
clicks on points to add a time filter with the 3-hour interval defined above.
```yaml
signals: [
{
name: point_click
on: [{
events: {
source: scope
type: click
markname: point
}
update: '''kibanaSetTimeFilter(datum.key, datum.key + 3 * 60 * 60 * 1000)'''
}]
}
]
```
This event is using the {kib} custom function `kibanaSetTimeFilter` to generate a filter that
gets applied to the entire dashboard on click.
The mouse cursor does not currently indicate that the chart is interactive. Find the `marks` section,
and update the mark named `point` by adding `cursor: { value: "pointer" }` to
the `encoding` section like this:
```yaml
{
name: point
type: symbol
style: ["point"]
from: {
data: source_0
}
encode: {
update: {
...
cursor: { value: "pointer" }
}
}
}
```
Next, we will add a drag interaction which will allow the user to narrow into
a specific time range in the visualization. This will require adding more signals, and
adding a rectangle overlay:
[role="screenshot"]
image::visualize/images/vega_tutorial_4.png[]
The first step is to add a new `signal` to track the X position of the cursor:
```yaml
{
name: currentX
value: -1
on: [{
events: {
type: mousemove
source: view
},
update: "clamp(x(), 0, width)"
}, {
events: {
type: mouseout
source: view
}
update: "-1"
}]
}
```
Now add a new `mark` to indicate the current cursor position:
```yaml
{
type: rule
interactive: false
encode: {
update: {
y: {value: 0}
y2: {signal: "height"}
stroke: {value: "gray"}
strokeDash: {
value: [2, 1]
}
x: {signal: "max(currentX,0)"}
defined: {signal: "currentX > 0"}
}
}
}
```
Next, add a signal to track the current selected range, which will update
until the user releases the mouse button or uses the escape key:
```yaml
{
name: selected
value: [0, 0]
on: [{
events: {
type: mousedown
source: view
}
update: "[clamp(x(), 0, width), clamp(x(), 0, width)]"
}, {
events: {
type: mousemove
source: window
consume: true
between: [{
type: mousedown
source: view
}, {
merge: [{
type: mouseup
source: window
}, {
type: keydown
source: window
filter: "event.key === 'Escape'"
}]
}]
}
update: "[selected[0], clamp(x(), 0, width)]"
}, {
events: {
type: keydown
source: window
filter: "event.key === 'Escape'"
}
update: "[0, 0]"
}]
}
```
Now that there is a signal which tracks the time range from the user, we need to indicate
the range visually by adding a new mark which only appears conditionally:
```yaml
{
type: rect
name: selectedRect
encode: {
update: {
height: {signal: "height"}
fill: {value: "#333"}
fillOpacity: {value: 0.2}
x: {signal: "selected[0]"}
x2: {signal: "selected[1]"}
defined: {signal: "selected[0] !== selected[1]"}
}
}
}
```
Finally, add a new signal which will update the {kib} time filter when the mouse is released while
dragging:
```yaml
{
name: applyTimeFilter
value: null
on: [{
events: {
type: mouseup
source: view
}
update: '''selected[0] !== selected[1] ? kibanaSetTimeFilter(
invert('x',selected[0]),
invert('x',selected[1])) : null'''
}]
}
```
Putting this all together, your visualization now supports the main features of
standard visualizations in {kib}, but with the potential to add even more control.
The final Vega spec for this tutorial is here:
.Expand final Vega spec
[%collapsible%closed]
====
[source,yaml]
----
{
$schema: "https://vega.github.io/schema/vega/v5.json"
data: [
{
name: source_0
url: {
%context%: true
%timefield%: order_date
index: kibana_sample_data_ecommerce
body: {
aggs: {
time_buckets: {
date_histogram: {
field: order_date
fixed_interval: "3h"
extended_bounds: {
min: {%timefilter%: "min"}
max: {%timefilter%: "max"}
}
min_doc_count: 0
}
}
}
size: 0
}
}
format: { property: "aggregations.time_buckets.buckets" }
}
]
scales: [{
name: x
type: time
range: width
domain: {
data: source_0
field: key
}
}, {
name: y
type: linear
range: height
domain: {
data: source_0
field: doc_count
}
}]
axes: [{
orient: bottom
scale: x
}, {
orient: left
scale: y
}]
marks: [
{
type: area
from: {
data: source_0
}
encode: {
update: {
x: {
scale: x
field: key
}
y: {
scale: y
value: 0
}
y2: {
scale: y
field: doc_count
}
}
}
},
{
name: point
type: symbol
style: ["point"]
from: {
data: source_0
}
encode: {
update: {
x: {
scale: x
field: key
}
y: {
scale: y
field: doc_count
}
size: {
value: 100
}
fill: {
value: black
}
cursor: { value: "pointer" }
}
}
},
{
type: rule
interactive: false
encode: {
update: {
y: {value: 0}
y2: {signal: "height"}
stroke: {value: "gray"}
strokeDash: {
value: [2, 1]
}
x: {signal: "max(currentX,0)"}
defined: {signal: "currentX > 0"}
}
}
},
{
type: rect
name: selectedRect
encode: {
update: {
height: {signal: "height"}
fill: {value: "#333"}
fillOpacity: {value: 0.2}
x: {signal: "selected[0]"}
x2: {signal: "selected[1]"}
defined: {signal: "selected[0] !== selected[1]"}
}
}
}
]
signals: [
{
name: point_click
on: [{
events: {
source: scope
type: click
markname: point
}
update: '''kibanaSetTimeFilter(datum.key, datum.key + 3 * 60 * 60 * 1000)'''
}]
}
{
name: currentX
value: -1
on: [{
events: {
type: mousemove
source: view
},
update: "clamp(x(), 0, width)"
}, {
events: {
type: mouseout
source: view
}
update: "-1"
}]
}
{
name: selected
value: [0, 0]
on: [{
events: {
type: mousedown
source: view
}
update: "[clamp(x(), 0, width), clamp(x(), 0, width)]"
}, {
events: {
type: mousemove
source: window
consume: true
between: [{
type: mousedown
source: view
}, {
merge: [{
type: mouseup
source: window
}, {
type: keydown
source: window
filter: "event.key === 'Escape'"
}]
}]
}
update: "[selected[0], clamp(x(), 0, width)]"
}, {
events: {
type: keydown
source: window
filter: "event.key === 'Escape'"
}
update: "[0, 0]"
}]
}
{
name: applyTimeFilter
value: null
on: [{
events: {
type: mouseup
source: view
}
update: '''selected[0] !== selected[1] ? kibanaSetTimeFilter(
invert('x',selected[0]),
invert('x',selected[1])) : null'''
}]
}
]
}
----
====
[[vega-reference]]
=== Reference for {kib} extensions
{kib} has extended Vega and Vega-Lite with extensions that support:
* Default height and width
* Default theme to match {kib}
* Writing {es} queries using the time range and filters from dashboards
* Using the Elastic Map Service in Vega maps
* Additional tooltip styling
* Advanced setting to enable URL loading from any domain
* Limited debugging support using the browser dev tools
* (Vega only) Expression functions which can update the time range and dashboard filters
[[vega-sizing-and-positioning]]
==== Default height and width
By default, Vega visualizations use the `autosize = { type: 'fit', contains: 'padding' }` layout.
`fit` uses all available space, ignores `width` and `height` values,
and respects the padding values. To override this behavior, change the
`autosize` value.
[[vega-theme]]
==== Default theme to match {kib}
{kib} registers a default https://vega.github.io/vega/docs/schemes/[Vega color scheme]
with the id `elastic`, and sets a default color for each `mark` type.
Override it by providing a different `stroke`, `fill`, or `color` (Vega-Lite) value.
[[vega-queries]]
==== Writing {es} queries in Vega
{kib} extends the Vega https://vega.github.io/vega/docs/data/[data] elements
with support for direct {es} queries specified as a `url`.
Because of this, {kib} is **unable to support dynamically loaded data**,
which would otherwise work in Vega. All data is fetched before it's passed to
the Vega renderer.
To define an {es} query in Vega, set the `url` to an object. {kib} will parse
the object looking for special tokens that allow your query to integrate with {kib}.
These tokens are:
* `%context%: true`: Set at the top level, and replaces the `query` section with filters from dashboard
* `%timefield%: <name>`: Set at the top level, integrates the query with the dashboard time filter
* `{%timefilter%: true}`: Replaced by an {es} range query with upper and lower bounds
* `{%timefilter%: "min" | "max"}`: Replaced only by the upper or lower bounds
* `{%timefilter: true, shift: -1, unit: 'hour'}`: Generates a time range query one hour in the past
* `{%autointerval%: true}`: Replaced by the string which contains the automatic {kib} time interval, such as `1h`
* `{%autointerval%: 10}`: Replaced by a string which is approximately dividing the time into 10 ranges, allowing
you to influence the automatic interval
* `"%dashboard_context-must_clause%"`: String replaced by object containing filters
* `"%dashboard_context-filter_clause%"`: String replaced by an object containing filters
* `"%dashboard_context-must_not_clause%"`: String replaced by an object containing filters
Putting this together, an example query that counts the number of documents in
a specific index:
[source,yaml]
----
// An object instead of a string for the URL value
// is treated as a context-aware Elasticsearch query.
url: {
// Specify the time filter.
%timefield%: @timestamp
// Apply dashboard context filters when set
%context%: true
// Which indexes to search
index: kibana_sample_data_logs
// The body element may contain "aggs" and "query" keys
body: {
aggs: {
time_buckets: {
date_histogram: {
// Use date histogram aggregation on @timestamp field
field: @timestamp <1>
// interval value will depend on the time filter
// Use an integer to set approximate bucket count
interval: { %autointerval%: true }
// Make sure we get an entire range, even if it has no data
extended_bounds: {
min: { %timefilter%: "min" }
max: { %timefilter%: "max" }
}
// Use this for linear (e.g. line, area) graphs
// Without it, empty buckets will not show up
min_doc_count: 0
}
}
}
// Speed up the response by only including aggregation results
size: 0
}
}
----
<1> `@timestamp` &mdash; Filters the time range and breaks it into histogram
buckets.
The full result includes the following structure:
[source,yaml]
----
{
"aggregations": {
"time_buckets": {
"buckets": [{
"key_as_string": "2015-11-30T22:00:00.000Z",
"key": 1448920800000,<1>
"doc_count": 28
}, {
"key_as_string": "2015-11-30T23:00:00.000Z",
"key": 1448924400000, <1>
"doc_count": 330
}, ...
----
<1> `"key"` &mdash; The unix timestamp you can use without conversions by the
Vega date expressions.
For most visualizations, you only need the list of bucket values. To focus on
only the data you need, use `format: {property: "aggregations.time_buckets.buckets"}`.
Specify a query with individual range and dashboard context. The query is
equivalent to `"%context%": true, "%timefield%": "@timestamp"`,
except that the time range is shifted back by 10 minutes:
[source,yaml]
----
{
body: {
query: {
bool: {
must: [
// This string will be replaced
// with the auto-generated "MUST" clause
"%dashboard_context-must_clause%"
{
range: {
// apply timefilter (upper right corner)
// to the @timestamp variable
@timestamp: {
// "%timefilter%" will be replaced with
// the current values of the time filter
// (from the upper right corner)
"%timefilter%": true
// Only work with %timefilter%
// Shift current timefilter by 10 units back
shift: 10
// week, day (default), hour, minute, second
unit: minute
}
}
}
]
must_not: [
// This string will be replaced with
// the auto-generated "MUST-NOT" clause
"%dashboard_context-must_not_clause%"
]
filter: [
// This string will be replaced
// with the auto-generated "FILTER" clause
"%dashboard_context-filter_clause%"
]
}
}
}
}
----
NOTE: When using `"%context%": true` or defining a value for `"%timefield%"` the body cannot contain a query. To customize the query within the VEGA specification (e.g. add an additional filter, or shift the timefilter), define your query and use the placeholders as in the example above. The placeholders will be replaced by the actual context of the dashboard or visualization once parsed.
The `"%timefilter%"` can also be used to specify a single min or max
value. The date_histogram's `extended_bounds` can be set
with two values - min and max. Instead of hardcoding a value, you may
use `"min": {"%timefilter%": "min"}`, which will be replaced with the
beginning of the current time range. The `shift` and `unit` values are
also supported. The `"interval"` can also be set dynamically, depending
on the currently picked range: `"interval": {"%autointerval%": 10}` will
try to get about 10-15 data points (buckets).
[float]
[[vega-esmfiles]]
=== Access Elastic Map Service files
Access the Elastic Map Service files via the same mechanism:
[source,yaml]
----
url: {
// "type" defaults to "elasticsearch" otherwise
type: emsfile
// Name of the file, exactly as in the Region map visualization
name: World Countries
}
// The result is a geojson file, get its features to use
// this data source with the "shape" marks
// https://vega.github.io/vega/docs/marks/shape/
format: {property: "features"}
----
To enable Maps, the graph must specify `type=map` in the host
configuration:
[source,yaml]
----
{
"config": {
"kibana": {
"type": "map",
// Initial map position
"latitude": 40.7, // default 0
"longitude": -74, // default 0
"zoom": 7, // default 2
// defaults to "default". Use false to disable base layer.
"mapStyle": false,
// default 0
"minZoom": 5,
// defaults to the maximum for the given style,
// or 25 when base is disabled
"maxZoom": 13,
// defaults to true, shows +/- buttons to zoom in/out
"zoomControl": false,
// Defaults to 'false', disables mouse wheel zoom. If set to
// 'true', map may zoom unexpectedly while scrolling dashboard
"scrollWheelZoom": false,
// When false, repaints on each move frame.
// Makes the graph slower when moving the map
"delayRepaint": true, // default true
}
},
/* the rest of Vega JSON */
}
----
The visualization automatically injects a `"projection"`, which you can use to
calculate the position of all geo-aware marks.
Additionally, you can use `latitude`, `longitude`, and `zoom` signals.
These signals can be used in the graph, or can be updated to modify the
position of the map.
[float]
[[vega-tooltip]]
==== Additional tooltip styling
{kib} has installed the https://vega.github.io/vega-lite/docs/tooltip.html[Vega tooltip plugin],
so tooltips can be defined in the ways documented there. Beyond that, {kib} also supports
a configuration option for changing the tooltip position and padding:
```js
{
config: {
kibana: {
tooltips: {
position: 'top',
padding: 15
}
}
}
}
```
[[vega-url-loading]]
==== Advanced setting to enable URL loading from any domain
Vega can load data from any URL, but this is disabled by default in {kib}.
To change this, set `vis_type_vega.enableExternalUrls: true` in `kibana.yml`,
then restart {kib}.
[[vega-inspector]]
==== Vega Inspector
Use the contextual *Inspect* tool to gain insights into different elements.
For Vega visualizations, there are two different views: *Request* and *Vega debug*.
===== Inspect Elasticsearch requests
Vega uses the {ref}/search-search.html[{es} search API] to get documents and aggregation
results from {es}. To troubleshoot these requests, click *Inspect*, which shows the most recent requests.
In case your specification has more than one request, you can switch between the views using the *View* dropdown.
[role="screenshot"]
image::visualize/images/vega_tutorial_inspect_requests.png[]
===== Vega debugging
With the *Vega debug* view, you can inspect the *Data sets* and *Signal Values* runtime data.
The runtime data is read from the
https://vega.github.io/vega/docs/api/debugging/#scope[runtime scope].
[role="screenshot"]
image::visualize/images/vega_tutorial_inspect_data_sets.png[]
To debug more complex specs, access to the `view` variable. For more information, refer to
the <<vega-browser-debugging-console, Vega browser debugging process>>.
===== Asking for help with a Vega spec
Because of the dynamic nature of the data in {es}, it is hard to help you with
Vega specs unless you can share a dataset. To do this, click *Inspect*, select the *Vega debug* view,
then select the *Spec* tab:
[role="screenshot"]
image::visualize/images/vega_tutorial_getting_help.png[]
To copy the response, click *Copy to clipboard*. Paste the copied data to
https://gist.github.com/[gist.github.com], possibly with a .json extension. Use the [raw] button,
and share that when asking for help.
[[vega-browser-debugging-console]]
==== Browser debugging console
experimental[] Use browser debugging tools (for example, F12 or Ctrl+Shift+J in Chrome) to
inspect the `VEGA_DEBUG` variable:
* `view` &mdash; Access to the Vega View object. See https://vega.github.io/vega/docs/api/debugging/[Vega Debugging Guide]
on how to inspect data and signals at runtime. For Vega-Lite,
`VEGA_DEBUG.view.data('source_0')` gets the pre-transformed data, and `VEGA_DEBUG.view.data('data_0')`
gets the encoded data. For Vega, it uses the data name as defined in your Vega spec.
* `vega_spec` &mdash; Vega JSON graph specification after some modifications by {kib}. In case
of Vega-Lite, this is the output of the Vega-Lite compiler.
* `vegalite_spec` &mdash; If this is a Vega-Lite graph, JSON specification of the graph before
Vega-Lite compilation.
[float]
[[vega-expression-functions]]
==== (Vega only) Expression functions which can update the time range and dashboard filters
{kib} has extended the Vega expression language with these functions:
```js
/**
* @param {object} query Elastic Query DSL snippet, as used in the query DSL editor
* @param {string} [index] as defined in Kibana, or default if missing
*/
kibanaAddFilter(query, index)
/**
* @param {object} query Elastic Query DSL snippet, as used in the query DSL editor
* @param {string} [index] as defined in Kibana, or default if missing
*/
kibanaRemoveFilter(query, index)
kibanaRemoveAllFilters()
/**
* Update dashboard time filter to the new values
* @param {number|string|Date} start
* @param {number|string|Date} end
*/
kibanaSetTimeFilter(start, end)
```
[float]
[[vega-additional-configuration-options]]
==== Additional configuration options
[source,yaml]
----
{
config: {
kibana: {
// Placement of the Vega-defined signal bindings.
// Can be `left`, `right`, `top`, or `bottom` (default).
controlsLocation: top
// Can be `vertical` or `horizontal` (default).
controlsDirection: vertical
// If true, hides most of Vega and Vega-Lite warnings
hideWarnings: true
// Vega renderer to use: `svg` or `canvas` (default)
renderer: canvas
}
}
}
----
[[vega-notes]]
[[vega-useful-links]]
=== Resources and examples
To learn more about Vega and Vega-Lite, refer to the resources and examples.
==== Vega editor
The https://vega.github.io/editor/[Vega Editor] includes examples for Vega & Vega-Lite, but does not support any
{kib}-specific features like {es} requests and interactive base maps.
==== Vega-Lite resources
* https://vega.github.io/vega-lite/tutorials/getting_started.html[Tutorials]
* https://vega.github.io/vega-lite/docs/[Docs]
* https://vega.github.io/vega-lite/examples/[Examples]
==== Vega resources
* https://vega.github.io/vega/tutorials/[Tutorials]
* https://vega.github.io/vega/docs/[Docs]
* https://vega.github.io/vega/examples/[Examples]
TIP: When you use the examples in {kib}, you may
need to modify the "data" section to use absolute URL. For example,
replace `"url": "data/world-110m.json"` with
`"url": "https://vega.github.io/editor/data/world-110m.json"`.