433c324016
This updated the `kbn:boostrap` scripts in the packages to use `--quiet` when bootstrapping. This prevents babel from logging every file it builds, but still logs errors if any occur. To make things a little simpler I also used the [`npm-run-all`](https://github.com/mysticatea/npm-run-all) package in `@kbn/i18n` to execute all the builds there in parallel and pass `--quiet`/`--watch` to all builds without duplicating the tasks or making them incompatible with windows. Before: ``` running `kbn:bootstrap` scripts @kbn/config-schema: $ yarn build @kbn/datemath: $ yarn build @kbn/i18n: $ yarn build @kbn/config-schema: $ tsc @kbn/datemath: $ babel src --out-dir target --copy-files @kbn/i18n: $ yarn build:web && yarn build:node && yarn build:types @kbn/i18n: $ cross-env BABEL_ENV=web babel src --config-file ./babel.config.js --out-dir target/web --extensions ".ts,.js,.tsx" @kbn/datemath: src/index.js -> target/index.js @kbn/i18n: Successfully compiled 18 files with Babel. @kbn/i18n: $ cross-env BABEL_ENV=node babel src --config-file ./babel.config.js --out-dir target/node --extensions ".ts,.js,.tsx" @kbn/i18n: Successfully compiled 18 files with Babel. @kbn/i18n: $ tsc --emitDeclarationOnly @kbn/dev-utils: $ yarn build @kbn/es-query: $ yarn build @kbn/dev-utils: $ babel src --out-dir target @kbn/es-query: $ babel src --out-dir target @kbn/dev-utils: src/index.js -> target/index.js @kbn/dev-utils: src/proc_runner/errors.js -> target/proc_runner/errors.js @kbn/dev-utils: src/proc_runner/index.js -> target/proc_runner/index.js @kbn/dev-utils: src/proc_runner/observe_lines.js -> target/proc_runner/observe_lines.js @kbn/dev-utils: src/proc_runner/observe_readable.js -> target/proc_runner/observe_readable.js @kbn/es-query: src/es_query/__tests__/_migrate_filter.js -> target/es_query/__tests__/_migrate_filter.js @kbn/dev-utils: src/proc_runner/observe_signals.js -> target/proc_runner/observe_signals.js @kbn/es-query: src/es_query/__tests__/build_es_query.js -> target/es_query/__tests__/build_es_query.js @kbn/es-query: src/es_query/__tests__/decorate_query.js -> target/es_query/__tests__/decorate_query.js @kbn/dev-utils: src/proc_runner/proc.js -> target/proc_runner/proc.js @kbn/es-query: src/es_query/__tests__/from_filters.js -> target/es_query/__tests__/from_filters.js @kbn/es-query: src/es_query/__tests__/from_kuery.js -> target/es_query/__tests__/from_kuery.js @kbn/es-query: src/es_query/__tests__/from_lucene.js -> target/es_query/__tests__/from_lucene.js @kbn/es-query: src/es_query/__tests__/lucene_string_to_dsl.js -> target/es_query/__tests__/lucene_string_to_dsl.js @kbn/dev-utils: src/proc_runner/proc_runner.js -> target/proc_runner/proc_runner.js @kbn/dev-utils: src/proc_runner/with_proc_runner.js -> target/proc_runner/with_proc_runner.js @kbn/dev-utils: src/proc_runner/with_proc_runner.test.js -> target/proc_runner/with_proc_runner.test.js @kbn/es-query: src/es_query/build_es_query.js -> target/es_query/build_es_query.js @kbn/dev-utils: src/serializers/absolute_path_serializer.js -> target/serializers/absolute_path_serializer.js @kbn/es-query: src/es_query/decorate_query.js -> target/es_query/decorate_query.js @kbn/dev-utils: src/serializers/index.js -> target/serializers/index.js @kbn/es-query: src/es_query/from_filters.js -> target/es_query/from_filters.js @kbn/dev-utils: src/streams/concat_stream.js -> target/streams/concat_stream.js @kbn/dev-utils: src/streams/index.js -> target/streams/index.js @kbn/dev-utils: src/streams/promise_from_streams.js -> target/streams/promise_from_streams.js @kbn/es-query: src/es_query/from_kuery.js -> target/es_query/from_kuery.js @kbn/dev-utils: src/streams/reduce_stream.js -> target/streams/reduce_stream.js @kbn/es-query: src/es_query/from_lucene.js -> target/es_query/from_lucene.js @kbn/dev-utils: src/tooling_log/index.js -> target/tooling_log/index.js @kbn/es-query: src/es_query/index.js -> target/es_query/index.js @kbn/dev-utils: src/tooling_log/log_levels.js -> target/tooling_log/log_levels.js @kbn/es-query: src/es_query/lucene_string_to_dsl.js -> target/es_query/lucene_string_to_dsl.js @kbn/dev-utils: src/tooling_log/log_levels.test.js -> target/tooling_log/log_levels.test.js @kbn/es-query: src/es_query/migrate_filter.js -> target/es_query/migrate_filter.js @kbn/dev-utils: src/tooling_log/tooling_log.js -> target/tooling_log/tooling_log.js @kbn/es-query: src/filters/__tests__/phrase.js -> target/filters/__tests__/phrase.js @kbn/es-query: src/filters/__tests__/query.js -> target/filters/__tests__/query.js @kbn/dev-utils: src/tooling_log/tooling_log.test.js -> target/tooling_log/tooling_log.test.js @kbn/dev-utils: src/tooling_log/tooling_log_text_writer.js -> target/tooling_log/tooling_log_text_writer.js @kbn/dev-utils: src/tooling_log/tooling_log_text_writer.test.js -> target/tooling_log/tooling_log_text_writer.test.js @kbn/es-query: src/filters/__tests__/range.js -> target/filters/__tests__/range.js @kbn/es-query: src/filters/exists.js -> target/filters/exists.js @kbn/es-query: src/filters/index.js -> target/filters/index.js @kbn/es-query: src/filters/phrase.js -> target/filters/phrase.js @kbn/es-query: src/filters/phrases.js -> target/filters/phrases.js @kbn/es-query: src/filters/query.js -> target/filters/query.js @kbn/es-query: src/filters/range.js -> target/filters/range.js @kbn/es-query: src/index.js -> target/index.js @kbn/es-query: src/kuery/ast/__tests__/ast.js -> target/kuery/ast/__tests__/ast.js @kbn/es-query: src/kuery/ast/ast.js -> target/kuery/ast/ast.js @kbn/es-query: src/kuery/ast/index.js -> target/kuery/ast/index.js @kbn/es-query: src/kuery/ast/kuery.js -> target/kuery/ast/kuery.js @kbn/es-query: src/kuery/ast/legacy_kuery.js -> target/kuery/ast/legacy_kuery.js @kbn/es-query: src/kuery/filter_migration/__tests__/exists.js -> target/kuery/filter_migration/__tests__/exists.js @kbn/es-query: src/kuery/filter_migration/__tests__/filter_to_kuery.js -> target/kuery/filter_migration/__tests__/filter_to_kuery.js @kbn/es-query: src/kuery/filter_migration/__tests__/geo_bounding_box.js -> target/kuery/filter_migration/__tests__/geo_bounding_box.js @kbn/es-query: src/kuery/filter_migration/__tests__/geo_polygon.js -> target/kuery/filter_migration/__tests__/geo_polygon.js @kbn/es-query: src/kuery/filter_migration/__tests__/phrase.js -> target/kuery/filter_migration/__tests__/phrase.js @kbn/es-query: src/kuery/filter_migration/__tests__/range.js -> target/kuery/filter_migration/__tests__/range.js @kbn/es-query: src/kuery/filter_migration/exists.js -> target/kuery/filter_migration/exists.js @kbn/es-query: src/kuery/filter_migration/filter_to_kuery.js -> target/kuery/filter_migration/filter_to_kuery.js @kbn/es-query: src/kuery/filter_migration/geo_bounding_box.js -> target/kuery/filter_migration/geo_bounding_box.js @kbn/es-query: src/kuery/filter_migration/geo_polygon.js -> target/kuery/filter_migration/geo_polygon.js @kbn/es-query: src/kuery/filter_migration/index.js -> target/kuery/filter_migration/index.js @kbn/es-query: src/kuery/filter_migration/phrase.js -> target/kuery/filter_migration/phrase.js @kbn/es-query: src/kuery/filter_migration/range.js -> target/kuery/filter_migration/range.js @kbn/es-query: src/kuery/functions/__tests__/and.js -> target/kuery/functions/__tests__/and.js @kbn/es-query: src/kuery/functions/__tests__/exists.js -> target/kuery/functions/__tests__/exists.js @kbn/es-query: src/kuery/functions/__tests__/geo_bounding_box.js -> target/kuery/functions/__tests__/geo_bounding_box.js @kbn/es-query: src/kuery/functions/__tests__/geo_polygon.js -> target/kuery/functions/__tests__/geo_polygon.js @kbn/es-query: src/kuery/functions/__tests__/is.js -> target/kuery/functions/__tests__/is.js @kbn/es-query: src/kuery/functions/__tests__/not.js -> target/kuery/functions/__tests__/not.js @kbn/es-query: src/kuery/functions/__tests__/or.js -> target/kuery/functions/__tests__/or.js @kbn/es-query: src/kuery/functions/__tests__/range.js -> target/kuery/functions/__tests__/range.js @kbn/es-query: src/kuery/functions/__tests__/utils/get_fields.js -> target/kuery/functions/__tests__/utils/get_fields.js @kbn/es-query: src/kuery/functions/and.js -> target/kuery/functions/and.js @kbn/es-query: src/kuery/functions/exists.js -> target/kuery/functions/exists.js @kbn/es-query: src/kuery/functions/geo_bounding_box.js -> target/kuery/functions/geo_bounding_box.js @kbn/es-query: src/kuery/functions/geo_polygon.js -> target/kuery/functions/geo_polygon.js @kbn/es-query: src/kuery/functions/index.js -> target/kuery/functions/index.js @kbn/es-query: src/kuery/functions/is.js -> target/kuery/functions/is.js @kbn/es-query: src/kuery/functions/not.js -> target/kuery/functions/not.js @kbn/es-query: src/kuery/functions/or.js -> target/kuery/functions/or.js @kbn/es-query: src/kuery/functions/range.js -> target/kuery/functions/range.js @kbn/es-query: src/kuery/functions/utils/get_fields.js -> target/kuery/functions/utils/get_fields.js @kbn/es-query: src/kuery/index.js -> target/kuery/index.js @kbn/es-query: src/kuery/node_types/__tests__/function.js -> target/kuery/node_types/__tests__/function.js @kbn/es-query: src/kuery/node_types/__tests__/literal.js -> target/kuery/node_types/__tests__/literal.js @kbn/es-query: src/kuery/node_types/__tests__/named_arg.js -> target/kuery/node_types/__tests__/named_arg.js @kbn/es-query: src/kuery/node_types/__tests__/wildcard.js -> target/kuery/node_types/__tests__/wildcard.js @kbn/es-query: src/kuery/node_types/function.js -> target/kuery/node_types/function.js @kbn/es-query: src/kuery/node_types/index.js -> target/kuery/node_types/index.js @kbn/es-query: src/kuery/node_types/literal.js -> target/kuery/node_types/literal.js @kbn/es-query: src/kuery/node_types/named_arg.js -> target/kuery/node_types/named_arg.js @kbn/es-query: src/kuery/node_types/wildcard.js -> target/kuery/node_types/wildcard.js @kbn/test: $ yarn build @kbn/test: $ babel src --out-dir target @kbn/test: src/es/es_test_cluster.js -> target/es/es_test_cluster.js @kbn/test: src/es/es_test_config.js -> target/es/es_test_config.js @kbn/test: src/es/index.js -> target/es/index.js @kbn/test: src/functional_tests/cli/index.js -> target/functional_tests/cli/index.js @kbn/test: src/functional_tests/cli/run_tests/args.js -> target/functional_tests/cli/run_tests/args.js @kbn/test: src/functional_tests/cli/run_tests/args.test.js -> target/functional_tests/cli/run_tests/args.test.js @kbn/test: src/functional_tests/cli/run_tests/cli.js -> target/functional_tests/cli/run_tests/cli.js @kbn/test: src/functional_tests/cli/run_tests/cli.test.js -> target/functional_tests/cli/run_tests/cli.test.js @kbn/test: src/functional_tests/cli/start_servers/args.js -> target/functional_tests/cli/start_servers/args.js @kbn/test: src/functional_tests/cli/start_servers/args.test.js -> target/functional_tests/cli/start_servers/args.test.js @kbn/test: src/functional_tests/cli/start_servers/cli.js -> target/functional_tests/cli/start_servers/cli.js @kbn/test: src/functional_tests/cli/start_servers/cli.test.js -> target/functional_tests/cli/start_servers/cli.test.js @kbn/test: src/functional_tests/lib/auth.js -> target/functional_tests/lib/auth.js @kbn/test: src/functional_tests/lib/index.js -> target/functional_tests/lib/index.js @kbn/test: src/functional_tests/lib/paths.js -> target/functional_tests/lib/paths.js @kbn/test: src/functional_tests/lib/run_cli.js -> target/functional_tests/lib/run_cli.js @kbn/test: src/functional_tests/lib/run_cli.test.js -> target/functional_tests/lib/run_cli.test.js @kbn/test: src/functional_tests/lib/run_elasticsearch.js -> target/functional_tests/lib/run_elasticsearch.js @kbn/test: src/functional_tests/lib/run_ftr.js -> target/functional_tests/lib/run_ftr.js @kbn/test: src/functional_tests/lib/run_kibana_server.js -> target/functional_tests/lib/run_kibana_server.js @kbn/test: src/functional_tests/tasks.js -> target/functional_tests/tasks.js @kbn/test: src/functional_tests/test_helpers.js -> target/functional_tests/test_helpers.js @kbn/test: src/index.js -> target/index.js @kbn/test: src/kbn/index.js -> target/kbn/index.js @kbn/test: src/kbn/kbn_test_config.js -> target/kbn/kbn_test_config.js @kbn/test: src/kbn/users.js -> target/kbn/users.js x-pack: $ gulp canvas:plugins:build x-pack: [13:22:34] /Users/spalger/kbn-dev/master/kibana/x-pack/plugins/canvas/canvas_plugin x-pack: [13:22:34] Using gulpfile ~/kbn-dev/master/kibana/x-pack/gulpfile.js x-pack: [13:22:34] Starting 'canvas:plugins:build'... x-pack: [13:22:50] canvas:plugins Plugins built in 15.75 seconds x-pack: [13:22:50] Finished 'canvas:plugins:build' after 16 s kibana: $ yarn build:types && node scripts/register_git_hook kibana: $ tsc --p tsconfig.types.json kibana: Registering Kibana pre-commit git hook... kibana: success Kibana pre-commit git hook was installed successfully. ``` (this will be even worse when we merge #26095) After: ``` running `kbn:bootstrap` scripts @kbn/config-schema: $ yarn build @kbn/datemath: $ yarn build --quiet @kbn/i18n: $ run-p 'build:babel:** --quiet' build:tsc @kbn/datemath: $ babel src --out-dir target --copy-files --quiet @kbn/config-schema: $ tsc @kbn/i18n: $ cross-env BABEL_ENV=node babel src --config-file ./babel.config.js --out-dir target/node --extensions ".ts,.js,.tsx" --quiet @kbn/i18n: $ cross-env BABEL_ENV=web babel src --config-file ./babel.config.js --out-dir target/web --extensions ".ts,.js,.tsx" --quiet @kbn/i18n: $ tsc --emitDeclarationOnly @kbn/i18n: Successfully compiled 18 files with Babel. @kbn/i18n: Successfully compiled 18 files with Babel. @kbn/dev-utils: $ yarn build --quiet @kbn/es-query: $ yarn build --quiet @kbn/dev-utils: $ babel src --out-dir target --quiet @kbn/es-query: $ babel src --out-dir target --quiet @kbn/test: $ yarn build --quiet @kbn/test: $ babel src --out-dir target --quiet x-pack: $ gulp canvas:plugins:build x-pack: [13:16:56] /Users/spalger/kbn-dev/master/kibana/x-pack/plugins/canvas/canvas_plugin x-pack: [13:16:56] Using gulpfile ~/kbn-dev/master/kibana/x-pack/gulpfile.js x-pack: [13:16:56] Starting 'canvas:plugins:build'... x-pack: [13:17:11] canvas:plugins Plugins built in 13.98 seconds x-pack: [13:17:11] Finished 'canvas:plugins:build' after 14 s kibana: $ yarn build:types && node scripts/register_git_hook kibana: $ tsc --p tsconfig.types.json kibana: Registering Kibana pre-commit git hook... kibana: success Kibana pre-commit git hook was installed successfully. ``` |
||
|---|---|---|
| .. | ||
| src | ||
| .babelrc | ||
| package.json | ||
| README.md | ||
Kibana Testing Library
The @kbn/test package provides ways to run tests. Currently only functional testing is provided by this library, with unit and other testing possibly added here.
Functional Testing
Dependencies
Functional testing methods exist in the src/functional_tests directory. They depend on the Functional Test Runner, which is found in {KIBANA_ROOT}/src/functional_test_runner. Ideally libraries provided by kibana packages such as this one should not depend on kibana source code that lives in {KIBANA_ROOT}/src. The goal is to start pulling test and development utilities out into packages so they can be used across Kibana and plugins. Accordingly the Functional Test Runner itself will be pulled out into a package (or part of a package), and this package's dependence on it will not be an issue.
Exposed methods
runTests(configPaths: Array)
For each config file specified in configPaths, starts Elasticsearch and Kibana once, runs tests specified in that config file, and shuts down Elasticsearch and Kibana once completed. (Repeats for every config file.)
configPaths: array of strings, each an absolute path to a config file that looks like this, following the config schema specified here.
Internally the method that starts Elasticsearch comes from kbn-es.
startServers(configPath: string)
Starts Elasticsearch and Kibana servers given a specified config.
configPath: absolute path to a config file that looks like this, following the config schema specified here.
Allows users to start another process to run just the tests while keeping the servers running with this method. Start servers and run tests using the same config file (see how).
Rationale
Single config per setup
We think it makes sense to specify the tests to run along with the particular server configuration for Elasticsearch and Kibana servers, because the tests expect a particular configuration. For example, saml api integration tests expect certain xml files to exist in Elasticsearch's config directory, and certain saml specific options to be passed in via the command line (or alternatively via the .yml config file) to both Elasticsearch and Kibana. It makes sense to keep all these config options together with the list of test files.
Multiple configs running in succession
We also think it makes sense to have a test runner intelligently (but simply) start servers, run tests, tear down servers, and repeat for each config, uninterrupted. There's nothing special about each kind of config that specifies running some set of functional tests against some kind of Elasticsearch/Kibana servers. There doesn't need to be a separate job to run each kind of setup/test/teardown. These can all be orchestrated sequentially via the current runTests implementation. This is how we envision tests to run on CI.
This inherently means that grouping test files in configs matters, such that a group of test files that depends on a particular server config appears together in that config's testFiles list. Given how quickly and easily we can start servers using @kbn/es, it should not impact performance to logically group tests by domain even if multiple groups of tests share the same server config. We can think about how to group test files together across domains when that time comes.