Since this is no longer part of a reporting task in Jenkins, we no
longer need to re-run kbn bootstrap when generating a report.
We use the PR_SOURCE_BRANCH environment variable to prevent genererating
Github issues when we are on a PR.
Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
* Reporting phantom and chromium tests should run in parallel.
* Chromium tests should be done in group 9.
* Attempting to use group 2 in ci for chromium tests.
* X-pack CI jobs should have 7 groups.
* Phantom tests should be in group 7.
* [ci][ftr][kbn/test] split up CI jobs
* [ci] run --assert-none-excluded in ci groups, before builds
* [ftr] improve error message when excluded tests found
* [ci] document other places the ciGroups live
* chore(19674): added dev flag when calling jenkins:unit grunt task so we can check dev licenses on ours ci jobs.
* chore(NA): update yarn lock files with integrity info.
* Replace xvfb with headless browser
* Update karma to run in headless chrome
Update Karma tests for an option to run chrome in headless.
* Fix lint issues
* Fix spaces
Closes#22840
In the functional tests we want a better way to include/exclude certain tests, especially as we move forward with #22359. This PR allows us to decorate suite objects with "tags", which won't clutter up the test names and can be used to filter out specific tests within a single test config. The functional test runner supports defining `--include-tag` and `--exclude-tag` CLI arguments, and multiple can be defined.
The method of filtering out tests for running against cloud has been updated to use this approach and I plan to do the same to #22359 once this is merged.
For instances where we would otherwise be running from source in a tracked elastic/elasticsearch branch, let's use a snapshot instead. This will eliminate some gradle issues we are experiencing in master.
* Turn reporting snapshot tests back on with new baselines
* Use snapshots generated from chromium. Bump the threshold so phantom comparisons pass
* Increase timeouts because of chromium functional tests.
* remove accidentally checked in session snapshots
* add a better vis baseline, run once through at 0 to check expected threshold
* turn on verbose logs
* Use threshold, turn debug mode on in jenkins
* Wrap both convert pages in the retry
* Update baselines
Preserve layout one got messed up - it was actually a print layout.
* bump threshold even more
* Replace test:api with @kbn/test runTests
* Improve CLI help menu 🆘
* Use --es-from
* Replace jenkins:selenium with kbn-test
* Validate cli args, fixing test in the process
* Clean up some stuff
* Code review fixes
* Explanation for collectCliArgs
* Remove exit codes, they're useless anyway.
* Make markdown vis test pass with dev_mode setting
* Tests
* Remove unneeded export
* Code review: move console logging up to cli.js
* Code review: refactor startServers and runTests to take single options collection
* Code review: Remove all things I am sure we do not use
* Improve tests
* Code review fixes
* Pass created log to runFtr, runElasticsearch, runKibanaServer
* Update --es-from option to --esFrom
* [mocha] run tests with mocha script, remove grunt-simple-mocha
* [ci] force colored output
* [ci] disable color when reading the yarn bin
* [dev/build/exec] support tooling log having its own chalk
* [dev/mocha] avoid changing the cwd of the process
* [dev/mocha] only match files, speed up negative matches
* [cli/serve/test] strip ansi control characters from logs
* [ci] disable color in all `yarn bin` invocations
* [dev/mocha] cleanup old runInBand check
* [dev/mocha] enable gloabl leak checks
Restructure testing with kbn-test package
- Run with multiple configs, move cli options to config
- Package-ify kbn-test
- Eventually we'll have functional_test_runner live in a package
of its own, and then this kbn-test will use that as a dependency,
probably still as a devDependency.
- Implement functional_tests_server
- Collapse single and multiple config apis into one command
Use kbn-es
Replace es_test_cluster + es_test_config with kbn/test utils
Implement new createEsTestCluster
Improve scripts, jsdocs, cli top-level tools
Lift error handling to the top level
* Add test-subj-selector module to packages dir.
* Rename to @kbn/test-subj-selector.
* Simplify rejectRejFiles Grunt task and ignore nested node_modules.
* Update jenkins_unit script to use yarn kbn CLI.
* switch to yarn
* cleanup misc references to npm
* [yarn] loosen dependency ranges so yarn will merge more deps
* fix linting error now that moment uses ESM
* [licenses] font-awesome changed the format of its license id
* Use local yarn
* Misc fixes
* eslintignore built yarn file
* Remove mkdir which doesn't do what it should do
* Check build without upgrading lots of versions
* Fix license check
* too many moments
* Better description
* Review fixes
* Lock to angular@1.6.5
* More specific version locks
* Revert "More specific version locks"
This reverts commit 11ef81102e.
* Revert "Lock to angular@1.6.5"
This reverts commit 3ade68c14c.
* rm yarn.lock; yarn
* Forcing a specific version of React, Angular, Moment
* Using vendored version of yarn in ci
* Use --frozen-lockfile
* fixes
* [dev/ci_setup] generalize jenkins_setup script for other CI environments
* [dev/ci_setup] accept more general `CACHE_DIR` env arg
* [dev/ci_setup] node.js install might not be invalid
* build before running selenium
* change directory into built snapshot
* WIP: what is in the build dir
* WIP: skip unit tests
* run selenium on build
* [functional tests on build] Derive version
* [functional tests on build] Missing folder name prefix
* WIP: debug on ubuntu VM
* WIP: double kibana stabilize timeout
* WIP: remove ubuntu VM debug change
* WIP: add logs for docParams
* WIP: add logs for es
* build release version without -SNAPSHOT-
* [functional test release] Cleanup
We always run the build task now in jenkins jobs rather than only in the
core builds. This ensures that the build task runs in PR builds even
though the build artifacts are not used for anything.
Ideally all the selenium tests would run against this build rather than
it just being something that gets ran and forgotten about, but this is a
first step.
Until now, the ci setup tasks were stored in the Jenkins job configuration. This moves the setup into source control so that jenkins can be told to execute `./test/scripts/jenkins_{JOB_TYPE}.sh` no matter what branch of kibana it is dealing with.
