Over the last few years we've increased the number of test runners.
Entry points by test type have become too lage of a category for unique
names, so this moves top level test scripts under yarn and replaces test
types with the runner name.
e.g. `yarn test:browser` -> `yarn test:karma`
Closes#41133
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* Generate core API docs from TSDoc comments
Uses api-extractor and api-documenter to generate documentation for
the Kibana core API from TSDoc comments in the source code.
Documentation can be generated using `npm run docs:api`.
I used --no-verify to ignore the following pre-commit hook errors:
1. Filenames MUST use snake_case - api-extractor.json
It's possible to specify a different config file, but I prefer to keep the "standard" config file name.
2. UNHANDLED ERROR: Unable to find tsconfig.json file selecting "common/core_api_review/kibana.api.ts". Ensure one exists and it is listed in "src/dev/typescript/projects.ts"
This is not a source file, so safe to ignore.
* Flesh out API docs a little bit
* Ignore snake_case check for api-extractor.json
* Ignore api-extractor's review file from pre-commit check
* Try to fix build failing by using masters yarn.lock
* I'm being stupid
* Found a better home for ignoring common/core_api_review/kibana.api.ts
* Node script for detecting core API changes
I initially wanted to include this as a precommit hook, but it takes
quite long to execute (~12s) so might be better suited as a test or
as part of the release process.
The script currently fails because api-extractor uses an older version
of typescript.
* Fix tslint precommit hook ignore condition
* Write tsdoc-metadata.json into ./build
* Add LogMeta and ElasticSearch to exported types & docs
* Suppress logging when running api-extractor from script
* Improve check_core_api_changes script and run as test
* Inline api-extractor.json config
* Fix check_core_api_changes --help flag
* LogMeta TSDoc comments
* check_core_api_changes: fail if api-extractor produces warnings or errors
And print more useful messages to the console
* Move ignored ts files list into dev/file
* Add back build:types since api-exporter cannot operate on source files
* Upgrade api-exporter/documenter
* api-extractor: independantly analyze core/public and core/server
Becasue of https://github.com/Microsoft/web-build-tools/issues/1029
api-extractor can't use core/index.ts as a single entry point for
analyzing the public and server API's as isolated namespaces.
Instead we analyze these projects separately. This introduces other
problems like the api review files and documentation always being
called "kibana." from the package.json filename.
* Build types as part of build task
* Include types in typescript browser compilation
* Force inclusion of core/public for building types
* Fix api review filename in api-exporter errors
* Update docs and API review files
* Fix api-extractor warnings
* Remove ts file ignored list since it's no longer necessary
* Rename exported api package name
* Review comments
* Export other missing types
* Upgrade api-documenter to latest beta
* Export more missing types
* Fix warnings and add api-exporter to Jenkins tests
* Correctly handle runBuildTypes() exceptions
* Fix another swallowed exception
* Fix api-extractor warnings after master merge
Based on #32705
We currently have TypeScript code that was backported to 7.0, which was backported without issue because it falls outside of any TypeScript projects in 7.0. This means that the pre-commit hooks break on changes to these files, and that they are not getting type checked by the type_check script. To fix this we need to verify that every typescript file in the repository is covered by a tsconfig.json file as part of CI.
* Prefer third-party plugin development in plugins instead of kibana-extra
* Fix failing recursive directory creation and removal
* Add new built version of kbn-pm
* Replace test:api with @kbn/test runTests
* Improve CLI help menu 🆘
* Use --es-from
* Replace jenkins:selenium with kbn-test
* Validate cli args, fixing test in the process
* Clean up some stuff
* Code review fixes
* Explanation for collectCliArgs
* Remove exit codes, they're useless anyway.
* Make markdown vis test pass with dev_mode setting
* Tests
* Remove unneeded export
* Code review: move console logging up to cli.js
* Code review: refactor startServers and runTests to take single options collection
* Code review: Remove all things I am sure we do not use
* Improve tests
* Code review fixes
* Pass created log to runFtr, runElasticsearch, runKibanaServer
* Update --es-from option to --esFrom
In order to make the license that applies to each file as clear as possible, and to be consistent with elasticsearch, we are adding Apache 2.0 license headers to the top of each file.
Existence of this header is enforced by eslint and tslint and missing headers were automatically added in the last commit by running:
```
node scripts/eslint --fix && node scripts/tslint --fix
```
* [tslint] lint typescript code
* [tslint] filter projects when running specific project
* [dev/ts] use more explicit types
* [dev/ts] add note about why using glob
* [dev/ts] rely on ts, use fewer getters
* [mocha] run tests with mocha script, remove grunt-simple-mocha
* [ci] force colored output
* [ci] disable color when reading the yarn bin
* [dev/build/exec] support tooling log having its own chalk
* [dev/mocha] avoid changing the cwd of the process
* [dev/mocha] only match files, speed up negative matches
* [cli/serve/test] strip ansi control characters from logs
* [ci] disable color in all `yarn bin` invocations
* [dev/mocha] cleanup old runInBand check
* [dev/mocha] enable gloabl leak checks
* [build] don't copy unwanted files
* [build] condense mkdir/touch tasks
* [i18n] verifyTranslations task has nothing to do with the build, just run in tests
* [es/tests] remove unused module
* [testUtil/es] add utility for starting es nodes in tests
* [ftr/tests] use esTestCluster util to start es
* [es/tests/routes] use esTestCluster to start own es
* [testUtils/kbnServer] disable uiSettings unless plugins are enabled
* [testUtils/esTestCluster] use standard test es port by default
* [server/http/tests] add esTestCluster to setup
* [test/config] unify es test config into a single module
* [testUtils/esTestCluster] directory is no longer configurable
* [testUtils/esTestCluster] throw when es.start() is called again without es.stop()
* [testUtils/esTestCluster] is* checks should not mutate state
* use es6 exports in fixtures
* use es6 exports in test_utils
* use es6 exports in src/ui
* use es6 exports in src/utils
* use es6 exports in src/server
* use es6 exports in timelion
* use es6 exports in core plugins, but not console
* use es6 exports in console
* use es6 exports in src/cli
* use eslint --fix for no-extra-semi rule
* selectively disable kibana-custom/no-default-export
* replace define() with exports
* clean up some object exports
in these cases, named exports are the better replacement
* use es6 exports in src/optimize, where possible
* fix uses of named exports
* fix some imports in console
* revert postcss optimizer to module.exports
* [timelion] put shared code in common directory
* fix url module export
don't export as default, rely on the named export
* convert define modules in src to export
* convert ui-bootstrap to cjs module
* lint webpackShims
no reason not to, they pass when the no-var rule is disabled
* fix new code not using es6 exports
* fix some straggling exports
* [indexPatterns] support cross cluster patterns
* [vis] remove unused `hasTimeField` param
* [indexPatterns/create] fix method name in view
* [indexPatterns/create] disallow expanding with ccs
* [indexPatterns/create] field fetching is cheaper, react faster
* [indexPatterns/resolveTimePattern/tests] increase readability
* [tests/apiIntegration/indexPatterns] test conflict field output
* [indexPatterns/fieldCaps/readFieldCapsResponse] add unit tests
* [test/apiIntegration] ensure random word will not be valid
* [indexPatterns/ui/client] remove unused import
* remove use of auto-release-sinon
* [indexPatterns/create] don't allow expand when cross cluster
* [indexPatternsApiClient/stub] use angular promises
* [indexPatterns/create] add tests for base create ui behaviors
* [tests/functional] move screenshots to their own service
* [ftr] add testFailure and testHookFailure lifecycle hooks
* [tests/functional/screenshots] cleanup old screenshots at startup
* [test/functional/screenshots] take screenshots when tests fail
* [cli_plugin/install] fix test
* [ui/scanner] fix test
* [ftr] remove digdug, use chromedriver directly
why?
- digdug is meant to be used by intern, and expects a certain lifecycle that the FTR has tried and continuously fails to mimic
- rather than continue trying to force digdug into the stack, just spawn chromedriver ourselves. We know how to handle it
- cleans up verbose remote logging while we're in there, since selenium-standalone-server went with digdug, which was previously doing the verbose logging
- deprecate config.servers.webdriver
- add config.chromedriver.url
- if url at config.chromedriver.url points to a server that responds to http requests use it as the chromedriver instance, enables running chrome in a VM or container
- if pings to config.chromedriver.url fail a local chromedriver is started
* address review requests
* build before running selenium
* change directory into built snapshot
* WIP: what is in the build dir
* WIP: skip unit tests
* run selenium on build
* [functional tests on build] Derive version
* [functional tests on build] Missing folder name prefix
* WIP: debug on ubuntu VM
* WIP: double kibana stabilize timeout
* WIP: remove ubuntu VM debug change
* WIP: add logs for docParams
* WIP: add logs for es
* build release version without -SNAPSHOT-
* [functional test release] Cleanup
* [functional_test_runner] replace functional testing tools with custom/pluggable solution
* [functional_test_runner] Convert unit tests to commonjs format
* [functional_test_runner] Fix dashboard test in wrong mode
* [functional_test_runner] Add dashboardLandingPage test subject
* [functional_test_runner] Get Visualize page object
* [functional_test_runner] Fix outdated references
* [functional_test_runner] Fix more outdated refs
* [functional_test_runner] Remove duplicate tests
* [functional_test_runner] Improve test readability
* [functional_test_runner] 😞 So many duplicate methods
* [functional_test_runner] Move mgmt `before` outside toplevel describe
* [functional_test_runner] Settings page obj missing methods
* [functional_test_runner] Add improvements from @gammon
* [functional_test_runner] Fix return statements in async funcs
* [functional_test_runner] Move before() to correct scope
* [functional_test_runner] Add after() hooks to remove index patterns
* [functional_test_runner] Attempt to fix vertical bar chart tests
* [functional_test_runner] Clean up
* [functional_test_runner] Reinstate unit tests
* [functional_test_runner] Set default loglevel back to info
* [functional_test_runner] Replace `context`s with `describe`s
* [functional_test_runner] Better error handling
* [functional_test_runner] Add in new Tile Map tests
* Incorporate changes from master
* [functional_test_runner] validate that every test file has a single top-level suite
* Update contributing doc with link to full doc
* [docs] Spelling and grammar fixes
* docs: writing and running functional tests
* [docs] Move plugin doc to plugin area
* [docs] Housekeeping. Doc in wrong place
* [docs] Remove dup doc file
* [grunt] Only run mocha_setup when running tests, not every grunt task
* Create KuiButton, KuiLinkButton, KuiSubmitButton, and KuiButtonIcon React components in UI Framework.
* Add Jest test coverage for UI Framework, generate report in ui_framework/jest/report.
* Add UI Framework to linting task.
* Update UI Framework README with instructions on creating and testing React components.
* Add both React and HTML examples.
* Add UI Framework Jest tests to npm test script. Create separate scripts for watching and generating coverage reports.
* Fix appearance of kuiButtons with icons throughout Kibana, by adding a flexbox wrapper.
* Improve accessibility of kuiButtonIcon.
* Remove disabled attribute from KuiLinkButton.
* Remove kuiButton-isDisabled class from KuiButton and KuiSubmitButton.
* Update PageObjects for Visualize, Dashboard, and Discover to use data-test-subj selectors for menu items.
* Run checkPlugins task as part of test:ui:runner task.
While working on some functional tests I had to re-run the `test:ui:server` task several times to rebuild the front-end assets. I'm not sure why that should be necessary, so this updates the server used in that specific task to auto-rebuild the assets.
The tests in master are currently failing regularly because our current browser tests are serious memory hogs. Investigation reveals that nearly every test is retaining all of the memory it causes to be allocated. We have made some progress to being able to diagnose the problems, but we expect that problem to take some serious work to fix. We need a short-term solution though, and this is it.
Rather than modify the bundling process, we will shard the top-level test suites by name. For now, we've created 4 shards, but adding new shards is trivial if we need to.
Sharding is accomplished by creating a murmur3 hash of the top level suite names, then bucketing based on the hash output. If a test suite resolves to shard2, but we are running shard1, we simply never pass the function to `mocha.describe()`. Rather than redefine every describe statement, we have shimmed the global `window.describe()` function to accomplish this.
- Add 'test:visualRegression' grunt task.
- Run 'test:visualRegression' as part of npm script 'test'.
- Add 'clean:screenshots task'.
- Clean screenshots/session when funtional tests are run.
