Developer guide - index patterns key concepts (#96017)
* add index patterns key concepts
This commit is contained in:
Matthew Kime
GitHub
parent
2ee11db1fa
commit
2e68ce1a05
BIN
dev_docs/assets/data_view_diagram.png
Normal file
BIN
dev_docs/assets/data_view_diagram.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 68 KiB |
32
dev_docs/key_concepts/data_views.mdx
Normal file
32
dev_docs/key_concepts/data_views.mdx
Normal file
|
|
@ -0,0 +1,32 @@
|
|||
---
|
||||
id: kibDataViewsKeyConcepts
|
||||
slug: /kibana-dev-docs/data-view-intro
|
||||
title: Data Views
|
||||
summary: Data views are the central method of defining queryable data sets in Kibana
|
||||
date: 2021-08-11
|
||||
tags: ['kibana','dev', 'contributor', 'api docs']
|
||||
---
|
||||
|
||||
*Note: Kibana index patterns are currently being renamed to data views. There will be some naming inconsistencies until the transition is complete.*
|
||||
|
||||
Data views (formerly Kibana index patterns or KIPs) are the central method of describing sets of indices for queries. Usage is strongly recommended
|
||||
as a number of high level <DocLink id="kibBuildingBlocks" text="building blocks"/> rely on them. Further, they provide a consistent view of data across
|
||||
a variety Kibana apps.
|
||||
|
||||
Data views are defined by a wildcard string (an index pattern) which matches indices, data streams, and index aliases, optionally specify a
|
||||
timestamp field for time series data, and are stored as a <DocLink id="kibDevDocsSavedObjectsIntro"
|
||||
text="saved object"/>. They have a field list which comprises all the fields in matching indices plus fields defined specifically
|
||||
on the data view via runtime fields. Schema-on-read functionality is provided by data view defined runtime fields.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
The data view API is made available via the data plugin (`data.indexPatterns`, soon to be renamed) and most commonly used with <DocLink id="kibDevTutorialDataSearchAndSessions" section="high-level-search" text="SearchSource" />
|
||||
(`data.search.search.SearchSource`) to perform queries. SearchSource will apply existing filters and queries from the search bar UI.
|
||||
|
||||
Users can create data views via [Data view management](https://www.elastic.co/guide/en/kibana/current/index-patterns.html).
|
||||
Additionally, they can be created through the data view API.
|
||||
|
||||
Data views also allow formatters and custom labels to be defined for fields.
|
||||
|
||||
86
dev_docs/tutorials/data_views.mdx
Normal file
86
dev_docs/tutorials/data_views.mdx
Normal file
|
|
@ -0,0 +1,86 @@
|
|||
---
|
||||
id: kibDevTutorialDataViews
|
||||
slug: /kibana-dev-docs/tutorials/data-views
|
||||
title: Data views API
|
||||
summary: Data views API
|
||||
date: 2021-08-11
|
||||
tags: ['kibana', 'onboarding', 'dev', 'architecture']
|
||||
---
|
||||
|
||||
*Note: Kibana index patterns are currently being renamed to data views. There will be some naming inconsistencies until the transition is complete.*
|
||||
|
||||
### Data views API
|
||||
|
||||
- Get list of data views
|
||||
- Get default data view and examine fields
|
||||
- Get data view by id
|
||||
- Find data view by title
|
||||
- Create data view
|
||||
- Create data view and save it
|
||||
- Modify data view and save it
|
||||
- Delete data view
|
||||
|
||||
#### Get list of data view titles and ids
|
||||
|
||||
```
|
||||
const idsAndTitles = await data.indexPatterns.getIdsWithTitle();
|
||||
idsAndTitles.forEach(({id, title}) => console.log(`Data view id: ${id} title: ${title}`));
|
||||
```
|
||||
|
||||
#### Get default data view and examine fields
|
||||
|
||||
```
|
||||
const defaultDataView = await data.indexPatterns.getDefault();
|
||||
defaultDataView.fields.forEach(({name}) => { console.log(name); })
|
||||
```
|
||||
|
||||
#### Get data view by id
|
||||
|
||||
```
|
||||
const id = 'xxxxxx-xxx-xxxxxx';
|
||||
const dataView = await data.indexPatterns.get(id);
|
||||
```
|
||||
|
||||
#### Find data view by title
|
||||
|
||||
```
|
||||
const title = 'kibana-*';
|
||||
const [dataView] = await data.indexPatterns.find(title);
|
||||
```
|
||||
|
||||
#### Create data view
|
||||
|
||||
```
|
||||
const dataView = await data.indexPatterns.create({ title: 'kibana-*' });
|
||||
```
|
||||
|
||||
#### Create data view and save it immediately
|
||||
|
||||
```
|
||||
const dataView = await data.indexPatterns.createAndSave({ title: 'kibana-*' });
|
||||
```
|
||||
|
||||
#### Create data view, modify, and save
|
||||
|
||||
```
|
||||
const dataView = await data.indexPatterns.create({ title: 'kibana-*' });
|
||||
dataView.setFieldCustomLabel('customer_name', 'Customer Name');
|
||||
data.indexPatterns.createSavedObject(dataView);
|
||||
```
|
||||
|
||||
#### Modify data view and save it
|
||||
|
||||
```
|
||||
dataView.setFieldCustomLabel('customer_name', 'Customer Name');
|
||||
await data.indexPatterns.updateSavedObject(dataView);
|
||||
```
|
||||
|
||||
#### Delete index pattern
|
||||
|
||||
```
|
||||
await data.indexPatterns.delete(dataViewId);
|
||||
```
|
||||
|
||||
### Data view HTTP API
|
||||
|
||||
Rest-like HTTP CRUD+ API - [docs](https://www.elastic.co/guide/en/kibana/master/index-patterns-api.html)
|
||||
|
|
@ -49,471 +49,14 @@ This is helpful when you want to provide a user with options, for example when c
|
|||
|
||||
```
|
||||
|
||||
## Index Patterns
|
||||
## Data Views
|
||||
|
||||
The Index Patterns API provides a consistent method of structuring and formatting documents
|
||||
The data views API provides a consistent method of structuring and formatting documents
|
||||
and field lists across the various Kibana apps. Its typically used in conjunction with
|
||||
SearchSource for composing queries.
|
||||
<DocLink id="kibDevTutorialDataSearchAndSessions" section="high-level-search" text="SearchSource" /> for composing queries.
|
||||
|
||||
### Index Patterns API
|
||||
*Note: Kibana index patterns are currently being renamed to data views. There will be some naming inconsistencies until the transition is complete.*
|
||||
|
||||
- Get list of index patterns
|
||||
- Get default index pattern and examine fields
|
||||
- Get index pattern by id
|
||||
- Find index pattern by title
|
||||
- Create index pattern
|
||||
- Create index pattern and save it
|
||||
- Modify index pattern and save it
|
||||
- Delete index pattern
|
||||
|
||||
#### Get list of index pattern titles and ids
|
||||
|
||||
```
|
||||
const idsAndTitles = await data.indexPatterns.getIdsWithTitle();
|
||||
idsAndTitles.forEach(({id, title}) => console.log(`Index pattern id: ${id} title: ${title}`));
|
||||
```
|
||||
|
||||
#### Get default index pattern and examine fields
|
||||
|
||||
```
|
||||
const defaultIndexPattern = await data.indexPatterns.getDefault();
|
||||
defaultIndexPattern.fields.forEach(({name}) => { console.log(name); })
|
||||
```
|
||||
|
||||
#### Get index pattern by id
|
||||
|
||||
```
|
||||
const id = 'xxxxxx-xxx-xxxxxx';
|
||||
const indexPattern = await data.indexPatterns.get(id);
|
||||
```
|
||||
|
||||
#### Find index pattern by title
|
||||
|
||||
```
|
||||
const title = 'kibana-*';
|
||||
const [indexPattern] = await data.indexPatterns.find(title);
|
||||
```
|
||||
|
||||
#### Create index pattern
|
||||
|
||||
```
|
||||
const indexPattern = await data.indexPatterns.create({ title: 'kibana-*' });
|
||||
```
|
||||
|
||||
#### Create index pattern and save it immediately
|
||||
|
||||
```
|
||||
const indexPattern = await data.indexPatterns.createAndSave({ title: 'kibana-*' });
|
||||
```
|
||||
|
||||
#### Create index pattern, modify, and save
|
||||
|
||||
```
|
||||
const indexPattern = await data.indexPatterns.create({ title: 'kibana-*' });
|
||||
indexPattern.setFieldCustomLabel('customer_name', 'Customer Name');
|
||||
data.indexPatterns.createSavedObject(indexPattern);
|
||||
```
|
||||
|
||||
#### Modify index pattern and save it
|
||||
|
||||
```
|
||||
indexPattern.setFieldCustomLabel('customer_name', 'Customer Name');
|
||||
await data.indexPatterns.updateSavedObject(indexPattern);
|
||||
```
|
||||
|
||||
#### Delete index pattern
|
||||
|
||||
```
|
||||
await data.indexPatterns.delete(indexPatternId);
|
||||
```
|
||||
|
||||
### Index Patterns HTTP API
|
||||
|
||||
Index patterns provide Rest-like HTTP CRUD+ API with the following endpoints:
|
||||
|
||||
- Index Patterns API
|
||||
- Create an index pattern — `POST /api/index_patterns/index_pattern`
|
||||
- Fetch an index pattern by `{id}` — `GET /api/index_patterns/index_pattern/{id}`
|
||||
- Delete an index pattern by `{id}` — `DELETE /api/index_patterns/index_pattern/{id}`
|
||||
- Partially update an index pattern by `{id}` — `POST /api/index_patterns/index_pattern/{id}`
|
||||
- Fields API
|
||||
- Update field — `POST /api/index_patterns/index_pattern/{id}/fields`
|
||||
- Scripted Fields API
|
||||
- Create a scripted field — `POST /api/index_patterns/index_pattern/{id}/scripted_field`
|
||||
- Upsert a scripted field — `PUT /api/index_patterns/index_pattern/{id}/scripted_field`
|
||||
- Fetch a scripted field — `GET /api/index_patterns/index_pattern/{id}/scripted_field/{name}`
|
||||
- Remove a scripted field — `DELETE /api/index_patterns/index_pattern/{id}/scripted_field/{name}`
|
||||
- Update a scripted field — `POST /api/index_patterns/index_pattern/{id}/scripted_field/{name}`
|
||||
|
||||
### Index Patterns API
|
||||
|
||||
Index Patterns REST API allows you to create, retrieve and delete index patterns. I also
|
||||
exposes an update endpoint which allows you to update specific fields without changing
|
||||
the rest of the index pattern object.
|
||||
|
||||
#### Create an index pattern
|
||||
|
||||
Create an index pattern with a custom title.
|
||||
|
||||
```
|
||||
POST /api/index_patterns/index_pattern
|
||||
{
|
||||
"index_pattern": {
|
||||
"title": "hello"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Customize creation behavior with:
|
||||
|
||||
- `override` — if set to `true`, replaces an existing index pattern if an
|
||||
index pattern with the provided title already exists. Defaults to `false`.
|
||||
- `refresh_fields` — if set to `true` reloads index pattern fields after
|
||||
the index pattern is stored. Defaults to `false`.
|
||||
|
||||
```
|
||||
POST /api/index_patterns/index_pattern
|
||||
{
|
||||
"override": false,
|
||||
"refresh_fields": true,
|
||||
"index_pattern": {
|
||||
"title": "hello"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
At creation all index pattern fields are option and you can provide them.
|
||||
|
||||
```
|
||||
POST /api/index_patterns/index_pattern
|
||||
{
|
||||
"index_pattern": {
|
||||
"id": "...",
|
||||
"version": "...",
|
||||
"title": "...",
|
||||
"type": "...",
|
||||
"intervalName": "...",
|
||||
"timeFieldName": "...",
|
||||
"sourceFilters": [],
|
||||
"fields": {},
|
||||
"typeMeta": {},
|
||||
"fieldFormats": {},
|
||||
"fieldAttrs": {}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The endpoint returns the created index pattern object.
|
||||
|
||||
```json
|
||||
{
|
||||
"index_pattern": {}
|
||||
}
|
||||
```
|
||||
|
||||
#### Fetch an index pattern by ID
|
||||
|
||||
Retrieve an index pattern by its ID.
|
||||
|
||||
```
|
||||
GET /api/index_patterns/index_pattern/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
|
||||
```
|
||||
|
||||
Returns an index pattern object.
|
||||
|
||||
```json
|
||||
{
|
||||
"index_pattern": {
|
||||
"id": "...",
|
||||
"version": "...",
|
||||
"title": "...",
|
||||
"type": "...",
|
||||
"intervalName": "...",
|
||||
"timeFieldName": "...",
|
||||
"sourceFilters": [],
|
||||
"fields": {},
|
||||
"typeMeta": {},
|
||||
"fieldFormats": {},
|
||||
"fieldAttrs": {}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### Delete an index pattern by ID
|
||||
|
||||
Delete and index pattern by its ID.
|
||||
|
||||
```
|
||||
DELETE /api/index_patterns/index_pattern/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
|
||||
```
|
||||
|
||||
Returns an '200 OK` response with empty body on success.
|
||||
|
||||
#### Partially update an index pattern by ID
|
||||
|
||||
Update part of an index pattern. Only provided fields will be updated on the
|
||||
index pattern, missing fields will stay as they are persisted.
|
||||
|
||||
These fields can be update partially:
|
||||
|
||||
- `title`
|
||||
- `timeFieldName`
|
||||
- `intervalName`
|
||||
- `fields` (optionally refresh fields)
|
||||
- `sourceFilters`
|
||||
- `fieldFormatMap`
|
||||
- `type`
|
||||
- `typeMeta`
|
||||
|
||||
Update a title of an index pattern.
|
||||
|
||||
```
|
||||
POST /api/index_patterns/index_pattern/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
|
||||
{
|
||||
"index_pattern": {
|
||||
"title": "new_title"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
All update fields are optional, you can specify the following fields.
|
||||
|
||||
```
|
||||
POST /api/index_patterns/index_pattern
|
||||
{
|
||||
"index_pattern": {
|
||||
"title": "...",
|
||||
"timeFieldName": "...",
|
||||
"intervalName": "...",
|
||||
"sourceFilters": [],
|
||||
"fieldFormats": {},
|
||||
"type": "...",
|
||||
"typeMeta": {},
|
||||
"fields": {}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- `refresh_fields` — if set to `true` reloads index pattern fields after
|
||||
the index pattern is stored. Defaults to `false`.
|
||||
|
||||
```
|
||||
POST /api/index_patterns/index_pattern
|
||||
{
|
||||
"refresh_fields": true,
|
||||
"index_pattern": {
|
||||
"fields": {}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
This endpoint returns the updated index pattern object.
|
||||
|
||||
```json
|
||||
{
|
||||
"index_pattern": {}
|
||||
}
|
||||
```
|
||||
|
||||
### Fields API
|
||||
|
||||
Fields API allows to change field metadata, such as `count`, `customLabel`, and `format`.
|
||||
|
||||
#### Update fields
|
||||
|
||||
Update endpoint allows you to update fields presentation metadata, such as `count`,
|
||||
`customLabel`, and `format`. You can update multiple fields in one request. Updates
|
||||
are merges with persisted metadata. To remove existing metadata specify `null` as value.
|
||||
|
||||
Set popularity `count` for field `foo`:
|
||||
|
||||
```
|
||||
POST /api/index_patterns/index_pattern/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/fields
|
||||
{
|
||||
"fields": {
|
||||
"foo": {
|
||||
"count": 123
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Update multiple metadata values and fields in one request:
|
||||
|
||||
```
|
||||
POST /api/index_patterns/index_pattern/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/fields
|
||||
{
|
||||
"fields": {
|
||||
"foo": {
|
||||
"count": 123,
|
||||
"customLabel": "Foo"
|
||||
},
|
||||
"bar": {
|
||||
"customLabel": "Bar"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Use `null` value to delete metadata:
|
||||
|
||||
```
|
||||
POST /api/index_patterns/index_pattern/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/fields
|
||||
{
|
||||
"fields": {
|
||||
"foo": {
|
||||
"customLabel": null
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
This endpoint returns the updated index pattern object.
|
||||
|
||||
```json
|
||||
{
|
||||
"index_pattern": {}
|
||||
}
|
||||
```
|
||||
|
||||
### Scripted Fields API
|
||||
|
||||
Scripted Fields API provides CRUD API for scripted fields of an index pattern.
|
||||
|
||||
#### Create a scripted field
|
||||
|
||||
Create a field by simply specifying its name, will default to `string` type. Returns
|
||||
an error if a field with the provided name already exists.
|
||||
|
||||
```
|
||||
POST /api/index_patterns/index_pattern/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/scripted_field
|
||||
{
|
||||
"field": {
|
||||
"name": "my_field"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Create a field by specifying all field properties.
|
||||
|
||||
```
|
||||
POST /api/index_patterns/index_pattern/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/scripted_field
|
||||
{
|
||||
"field": {
|
||||
"name": "",
|
||||
"type": "",
|
||||
"searchable": false,
|
||||
"aggregatable": false,
|
||||
"count": 0,
|
||||
"script": "",
|
||||
"scripted": false,
|
||||
"lang": "",
|
||||
"conflictDescriptions": {},
|
||||
"format": {},
|
||||
"esTypes": [],
|
||||
"readFromDocValues": false,
|
||||
"subType": {},
|
||||
"indexed": false,
|
||||
"customLabel": "",
|
||||
"shortDotsEnable": false
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### Upsert a scripted field
|
||||
|
||||
Creates a new field or updates an existing one, if one already exists with the same name.
|
||||
|
||||
Create a field by simply specifying its name.
|
||||
|
||||
```
|
||||
PUT /api/index_patterns/index_pattern/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/scripted_field
|
||||
{
|
||||
"field": {
|
||||
"name": "my_field"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Create a field by specifying all field properties.
|
||||
|
||||
```
|
||||
PUT /api/index_patterns/index_pattern/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/scripted_field
|
||||
{
|
||||
"field": {
|
||||
"name": "",
|
||||
"type": "",
|
||||
"searchable": false,
|
||||
"aggregatable": false,
|
||||
"count": 0,
|
||||
"script": "",
|
||||
"scripted": false,
|
||||
"lang": "",
|
||||
"conflictDescriptions": {},
|
||||
"format": {},
|
||||
"esTypes": [],
|
||||
"readFromDocValues": false,
|
||||
"subType": {},
|
||||
"indexed": false,
|
||||
"customLabel": "",
|
||||
"shortDotsEnable": false
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### Fetch a scripted field
|
||||
|
||||
Fetch an existing index pattern field by field name.
|
||||
|
||||
```
|
||||
GET /api/index_patterns/index_pattern/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/scripted_field/<name>
|
||||
```
|
||||
|
||||
Returns the field object.
|
||||
|
||||
```json
|
||||
{
|
||||
"field": {}
|
||||
}
|
||||
```
|
||||
|
||||
#### Delete a scripted field
|
||||
|
||||
Delete a field of an index pattern.
|
||||
|
||||
```
|
||||
DELETE /api/index_patterns/index_pattern/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/scripted_field/<name>
|
||||
```
|
||||
|
||||
#### Update a an existing scripted field
|
||||
|
||||
Updates an exiting field by mergin provided properties with the existing ones. If
|
||||
there is no existing field with the specified name, returns a `404 Not Found` error.
|
||||
|
||||
You can specify any field properties, except `name` which is specified in the URL path.
|
||||
|
||||
```
|
||||
POST /api/index_patterns/index_pattern/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/scripted_field/<name>
|
||||
{
|
||||
"field": {
|
||||
"type": "",
|
||||
"searchable": false,
|
||||
"aggregatable": false,
|
||||
"count": 0,
|
||||
"script": "",
|
||||
"scripted": false,
|
||||
"lang": "",
|
||||
"conflictDescriptions": {},
|
||||
"format": {},
|
||||
"esTypes": [],
|
||||
"readFromDocValues": false,
|
||||
"subType": {},
|
||||
"indexed": false,
|
||||
"customLabel": "",
|
||||
"shortDotsEnable": false
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Query
|
||||
|
||||
|
|
|
|||
Loading…
Reference in a new issue