2016-10-25 03:41:32 +02:00
[[tutorial-load-dataset]]
== Loading Sample Data
The tutorials in this section rely on the following data sets:
* The complete works of William Shakespeare, suitably parsed into fields. Download this data set by clicking here:
2017-05-16 19:49:54 +02:00
https://download.elastic.co/demos/kibana/gettingstarted/shakespeare_6.0.json[shakespeare.json].
2016-10-25 03:41:32 +02:00
* A set of fictitious accounts with randomly generated data. Download this data set by clicking here:
2016-11-08 17:31:37 +01:00
https://download.elastic.co/demos/kibana/gettingstarted/accounts.zip[accounts.zip]
2016-10-25 03:41:32 +02:00
* A set of randomly generated log files. Download this data set by clicking here:
https://download.elastic.co/demos/kibana/gettingstarted/logs.jsonl.gz[logs.jsonl.gz]
Two of the data sets are compressed. Use the following commands to extract the files:
[source,shell]
unzip accounts.zip
gunzip logs.jsonl.gz
The Shakespeare data set is organized in the following schema:
[source,json]
{
"line_id": INT,
"play_name": "String",
"speech_number": INT,
"line_number": "String",
"speaker": "String",
"text_entry": "String",
}
The accounts data set is organized in the following schema:
[source,json]
{
"account_number": INT,
"balance": INT,
"firstname": "String",
"lastname": "String",
"age": INT,
"gender": "M or F",
"address": "String",
"employer": "String",
"email": "String",
"city": "String",
"state": "String"
}
The schema for the logs data set has dozens of different fields, but the notable ones used in this tutorial are:
[source,json]
{
"memory": INT,
"geo.coordinates": "geo_point"
"@timestamp": "date"
}
2017-06-27 19:13:42 +02:00
Before we load the Shakespeare and logs data sets, we need to set up {ref}/mapping.html[_mappings_] for the fields.
2016-10-25 03:41:32 +02:00
Mapping divides the documents in the index into logical groups and specifies a field's characteristics, such as the
field's searchability or whether or not it's _tokenized_, or broken up into separate words.
2017-02-11 15:24:04 +01:00
Use the following command in a terminal (eg `bash`) to set up a mapping for the Shakespeare data set:
2016-10-25 03:41:32 +02:00
2017-05-12 14:44:46 +02:00
[source,js]
PUT /shakespeare
2016-10-25 03:41:32 +02:00
{
2017-05-16 19:49:54 +02:00
"mappings": {
"doc": {
"properties": {
"speaker": {"type": "keyword"},
"play_name": {"type": "keyword"},
"line_id": {"type": "integer"},
"speech_number": {"type": "integer"}
2016-10-25 03:41:32 +02:00
}
}
}
}
2017-05-12 14:44:46 +02:00
//CONSOLE
2016-10-25 03:41:32 +02:00
This mapping specifies the following qualities for the data set:
2017-05-12 14:44:46 +02:00
* Because the _speaker_ and _play_name_ fields are keyword fields, they are not analyzed. The strings are treated as a single unit even if they contain multiple words.
2016-10-25 03:41:32 +02:00
* The _line_id_ and _speech_number_ fields are integers.
The logs data set requires a mapping to label the latitude/longitude pairs in the logs as geographic locations by
applying the `geo_point` type to those fields.
Use the following commands to establish `geo_point` mapping for the logs:
2017-05-12 14:44:46 +02:00
[source,js]
PUT /logstash-2015.05.18
2016-10-25 03:41:32 +02:00
{
"mappings": {
"log": {
"properties": {
"geo": {
"properties": {
"coordinates": {
"type": "geo_point"
}
}
}
}
}
}
}
2017-05-12 14:44:46 +02:00
//CONSOLE
[source,js]
PUT /logstash-2015.05.19
2016-10-25 03:41:32 +02:00
{
"mappings": {
"log": {
"properties": {
"geo": {
"properties": {
"coordinates": {
"type": "geo_point"
}
}
}
}
}
}
}
2017-05-12 14:44:46 +02:00
//CONSOLE
[source,js]
PUT /logstash-2015.05.20
2016-10-25 03:41:32 +02:00
{
"mappings": {
"log": {
"properties": {
"geo": {
"properties": {
"coordinates": {
"type": "geo_point"
}
}
}
}
}
}
}
2017-05-12 14:44:46 +02:00
//CONSOLE
2016-10-25 03:41:32 +02:00
The accounts data set doesn't require any mappings, so at this point we're ready to use the Elasticsearch
2017-06-27 19:13:42 +02:00
{ref}/docs-bulk.html[`bulk`] API to load the data sets with the following commands:
2016-10-25 03:41:32 +02:00
[source,shell]
2017-04-11 17:36:42 +02:00
curl -H 'Content-Type: application/x-ndjson' -XPOST 'localhost:9200/bank/account/_bulk?pretty' --data-binary @accounts.json
2017-05-16 19:49:54 +02:00
curl -H 'Content-Type: application/x-ndjson' -XPOST 'localhost:9200/shakespeare/doc/_bulk?pretty' --data-binary @shakespeare_6.0.json
2017-04-11 17:36:42 +02:00
curl -H 'Content-Type: application/x-ndjson' -XPOST 'localhost:9200/_bulk?pretty' --data-binary @logs.jsonl
2016-10-25 03:41:32 +02:00
These commands may take some time to execute, depending on the computing resources available.
Verify successful loading with the following command:
2017-05-12 14:44:46 +02:00
[source,js]
GET /_cat/indices?v
//CONSOLE
2016-10-25 03:41:32 +02:00
You should see output similar to the following:
[source,shell]
health status index pri rep docs.count docs.deleted store.size pri.store.size
yellow open bank 5 1 1000 0 418.2kb 418.2kb
yellow open shakespeare 5 1 111396 0 17.6mb 17.6mb
yellow open logstash-2015.05.18 5 1 4631 0 15.6mb 15.6mb
yellow open logstash-2015.05.19 5 1 4624 0 15.7mb 15.7mb
yellow open logstash-2015.05.20 5 1 4750 0 16.4mb 16.4mb