kibana/scripts/README.md

69 lines
3.4 KiB
Markdown
Raw Normal View History

# Kibana Dev Scripts
[esArchiver] combine elasticDump and ScenarioManager (#10359) * As a part of bringing functional testing to plugins, esArchiver gives these plugins a way to capture and reload es indexes without needing to write a bunch of custom code. It works similarly to the elasticDump and ScenarioManager tools that it replaces. Differences: - Streaming implementation allows for much larger archives - CLI for creating and using archives - Configurable archive location - Stores the data in gzipped files (better for source control, searching, large archives) - Automatically identifies and upgrades Kibana config documents Methods: - `#load(name)`: import an archive - `#loadIfNeeded(name)`: import an archive, but skip the documents what belong to any existing index - `#unload(name)`: delete the indexes stored in an archive CLI operations: - `./bin/es_archiver save <name> [index patterns...]`: save the mapping and documents in one or more indexes that match the wild-card patterns into an the `<name>` archive - `./bin/es_archiver load <name>`: load the mapping and documents from the `<name>` archive * [functional_tests/common/nagivate] check for statusPage * [es_archiver] move bins into new scripts dir * [functional_tests/apps/context] use esArchiver * [esArchiver] general improvements after showing to a few folks - remove auto-upgrading config doc logic (until we have better access to kibana version info) - export unload command - remove preemptive checks in favor of reacting to errors - use type "doc" vs "hit" for doc records (consistency) - wrote a bunch of pending tests to think though and plan * [esArchiver] make log a stream that writes to itself * [esArchiver] fill in stats and archive format tests * [esArchiver] splitup action logic * [esArchiver/cli] fix cli --help output and comment * [esArchiver] remove type-based param coercion * [esArchiver/log] use strings for log levels * [esArchvier] remove unused var * [esArchiver/indexDocRecordsStream] add tests * [esArchive] fill in remaining tests * [esArchiver] fix dem tests * [eslint] remove unused vars * [esArchiver/loadIfNeeded] fix call to load() * [esArchiver] remove loadDumpData helpers
2017-03-27 21:29:14 +02:00
This directory contains scripts useful for interacting with Kibana tools in development. Use the node executable and `--help` flag to learn about how they work:
```sh
node scripts/{{script name}} --help
```
## For Developers
[esArchiver] combine elasticDump and ScenarioManager (#10359) * As a part of bringing functional testing to plugins, esArchiver gives these plugins a way to capture and reload es indexes without needing to write a bunch of custom code. It works similarly to the elasticDump and ScenarioManager tools that it replaces. Differences: - Streaming implementation allows for much larger archives - CLI for creating and using archives - Configurable archive location - Stores the data in gzipped files (better for source control, searching, large archives) - Automatically identifies and upgrades Kibana config documents Methods: - `#load(name)`: import an archive - `#loadIfNeeded(name)`: import an archive, but skip the documents what belong to any existing index - `#unload(name)`: delete the indexes stored in an archive CLI operations: - `./bin/es_archiver save <name> [index patterns...]`: save the mapping and documents in one or more indexes that match the wild-card patterns into an the `<name>` archive - `./bin/es_archiver load <name>`: load the mapping and documents from the `<name>` archive * [functional_tests/common/nagivate] check for statusPage * [es_archiver] move bins into new scripts dir * [functional_tests/apps/context] use esArchiver * [esArchiver] general improvements after showing to a few folks - remove auto-upgrading config doc logic (until we have better access to kibana version info) - export unload command - remove preemptive checks in favor of reacting to errors - use type "doc" vs "hit" for doc records (consistency) - wrote a bunch of pending tests to think though and plan * [esArchiver] make log a stream that writes to itself * [esArchiver] fill in stats and archive format tests * [esArchiver] splitup action logic * [esArchiver/cli] fix cli --help output and comment * [esArchiver] remove type-based param coercion * [esArchiver/log] use strings for log levels * [esArchvier] remove unused var * [esArchiver/indexDocRecordsStream] add tests * [esArchive] fill in remaining tests * [esArchiver] fix dem tests * [eslint] remove unused vars * [esArchiver/loadIfNeeded] fix call to load() * [esArchiver] remove loadDumpData helpers
2017-03-27 21:29:14 +02:00
This directory is excluded from the build and tools within it should help users discover their capabilities. Each script in this directory must:
Validate current node version (#19154) * feat(12976): node version validation at runtime. * refact(12976): move the code into a static utilities class. * test(12976): added first test case using jest. * test(12976): added test cases for node_version. * feat(12976): create setup env node to bootstrap babel, ts-node and node version validator. * refact(12976): migrated node version code from es6 to es5. * feat(12976): node version validation at runtime. * refact(12976): move the code into a static utilities class. * test(12976): added first test case using jest. * test(12976): added test cases for node_version. * feat(12976): create setup env node to bootstrap babel, ts-node and node version validator. * refact(12976): migrated node version code from es6 to es5. * fix(12976): remove one level from ts node register cache directory link. * chore(12976): added caret to semver dependecy in order to support minor versions. * refact(12976): small change from named import to default import on node version validator. * refact(12976): removed ts_node_register and add the code to babel_register. * feat(12976): split eslint config in order to properly support files built to run before and after node version validator. refact(12976): convert script files to es5 code. refact(12976): delete inline eslint configs from node version check related files. * refact(12976): remove ts node register file. * refact(12976): completely port setup_node_env to es5. * refact(12976): remove babel_register invokation from external dependencies in scripts. * refact(12976): move node_version code directly into node_version_validator inside setup_node_env folder. * refact(12976): only node version validator for kbn script.
2018-05-25 19:43:01 +02:00
- require `src/setup_node_env` to bootstrap NodeJS environment
- call out to source code in the [`src`](../src) or [`packages`](../packages) directories
[esArchiver] combine elasticDump and ScenarioManager (#10359) * As a part of bringing functional testing to plugins, esArchiver gives these plugins a way to capture and reload es indexes without needing to write a bunch of custom code. It works similarly to the elasticDump and ScenarioManager tools that it replaces. Differences: - Streaming implementation allows for much larger archives - CLI for creating and using archives - Configurable archive location - Stores the data in gzipped files (better for source control, searching, large archives) - Automatically identifies and upgrades Kibana config documents Methods: - `#load(name)`: import an archive - `#loadIfNeeded(name)`: import an archive, but skip the documents what belong to any existing index - `#unload(name)`: delete the indexes stored in an archive CLI operations: - `./bin/es_archiver save <name> [index patterns...]`: save the mapping and documents in one or more indexes that match the wild-card patterns into an the `<name>` archive - `./bin/es_archiver load <name>`: load the mapping and documents from the `<name>` archive * [functional_tests/common/nagivate] check for statusPage * [es_archiver] move bins into new scripts dir * [functional_tests/apps/context] use esArchiver * [esArchiver] general improvements after showing to a few folks - remove auto-upgrading config doc logic (until we have better access to kibana version info) - export unload command - remove preemptive checks in favor of reacting to errors - use type "doc" vs "hit" for doc records (consistency) - wrote a bunch of pending tests to think though and plan * [esArchiver] make log a stream that writes to itself * [esArchiver] fill in stats and archive format tests * [esArchiver] splitup action logic * [esArchiver/cli] fix cli --help output and comment * [esArchiver] remove type-based param coercion * [esArchiver/log] use strings for log levels * [esArchvier] remove unused var * [esArchiver/indexDocRecordsStream] add tests * [esArchive] fill in remaining tests * [esArchiver] fix dem tests * [eslint] remove unused vars * [esArchiver/loadIfNeeded] fix call to load() * [esArchiver] remove loadDumpData helpers
2017-03-27 21:29:14 +02:00
- react to the `--help` flag
- run everywhere OR check and fail fast when a required OS or toolchain is not available
## Functional Test Scripts
**`node scripts/functional_tests [--config test/functional/config.js --config test/api_integration/config.js]`**
Runs all the functional tests: selenium tests and api integration tests. List configs with multiple `--config` arguments. Uses the [@kbn/test](../packages/kbn-test) library to run Elasticsearch and Kibana servers and tests against those servers, for multiple server+test setups. In particular, calls out to [`runTests()`](../packages/kbn-test/src/functional_tests/tasks.js). Can be run on a single config.
**`node scripts/functional_tests_server [--config test/functional/config.js]`**
Starts just the Elasticsearch and Kibana servers given a single config, i.e. via `--config test/functional/config.js` or `--config test/api_integration/config`. Allows the user to start just the servers with this script, and keep them running while running tests against these servers. The idea is that the same config file configures both Elasticsearch and Kibana servers. Uses the [`startServers()`](../packages/kbn-test/src/functional_tests/tasks.js#L52-L80) method from [@kbn/test](../packages/kbn-test) library.
Example. Start servers _and_ run tests, separately, but using the same config:
```sh
# Just the servers
node scripts/functional_tests_server --config path/to/config
```
In another terminal:
```sh
# Just the tests--against the running servers
node scripts/functional_test_runner --config path/to/config
```
For details on how the internal methods work, [read this readme](../packages/kbn-test/README.md).
### ES archiver
#### Loading data
If you wish to load up specific es archived data for your test, you can do so via:
```
node scripts/es_archiver.js load <archive> [--es-url=http://username:password@localhost:9200] [--kibana-url=http://username:password@localhost:5601/{basepath?}]
```
That will load the specified archive located in the archive directory specified by the default functional config file, located in `test/functional/config.js`. To load archives from other function config files you can pass `--config path/to/config.js`.
*Note:* The `--es-url` and `--kibana-url` options may or may not be neccessary depending on your current Kibana configuration settings, and their values
may also change based on those settings (for example if you are not running with security you will not need the `username:password` portion).
#### Saving data
You can save existing data into an archive by using the `save` command:
```
node scripts/es_archiver.js save <archive name for kibana data> [space separated list of index patterns to include]
```
You may want to store the .kibana index separate from data. Since adding a lot of data will bloat our repo size, we have many tests that reuse the same
data indices but use their own `.kibana` index.