* Revert "Revert "Add pipeline for flaky test runner job (#46740)""
This reverts commit 7d96a13fad.
Also reconcile changes to Jenkinsfile since original revert happened
* Fix param parsing and add missed change
* Add missing variable
* refactor failed_tests_reporter to use TS, no octokit
* update renovate config
* ensure that all kbn-test files are in ts project
* fix some type errors
* add some more tests
* [kbn-test/githubapi] cleanup and document
* collect log messages as strings instead of message objects
* ensure issue is open when updating body
* improve readability of getKibanaIssues
* expose axios helpers from dev-utils
* fix request params for fetching github issues and validate locally
* include a README for failed_tests_reporter
* improve axios error helpers
# Conflicts:
# packages/kbn-test/src/index.ts
# renovate.json5
* Re-split ciGroups after pipeline rollout
Revert "Revert "Revert "Revert "Revert "[ci] compress jobs for CI stab… (#45454)"
This reverts commit 9a109f2170.
Revert "set IS_PIPELINE_JOB in intake jobs (#45850)"
This reverts commit b1a01effa8.
* Split one of the slow test suites up to try to make overall CI faster
* Disable visualRegression groups, they are being handled in other work
* Revert "Split one of the slow test suites up to try to make overall CI faster"
This reverts commit 1213239545.
* Move some different xpack ciGroup8 suites around
* Revert hiding Kibana icon, test Percy rendering
* Update jobs.yml
* Use percy token for kibana dev
* Update Percy token to read from kibana-issues
* Read correct field
* increase timeout
* Run each visual job 10 times
* Revert last commit
* Increase timeout
* Disable visual tests and revert percy token
* Fix jest tests
* Another jest test fix
* set IS_PIPELINE_JOB in intake jobs
* firefox smoke doesn't get an initial build
* firefox smoke doesn't get an initial build (oss edition)
* re-enable all jobs
* Revert "Revert "Revert "Revert "[ci] compress jobs for CI stability" (#44584)"""
This reverts commit 148b8c0f90.
* sync changes with Jenkinsfile
(cherry picked from commit 9a109f2170)
* Pipeline
* WIP some work for parallelization with ciGroups
* Fix xpack kibana install dir, and add some debugging
* Attempt to quick fix a few tests
* Revert "Revert "Revert "[ci] compress jobs for CI stability" (#44584)""
This reverts commit 078ac2897f.
* Recombine test groups, and try runbld again
* Mostly cleanup, and fix failed_tests reporting to hopefully work for both pipeline and non-pipeline
* Fix typo in shell script
* Remove some debug code
* Add support for changing es transport.port during testing via TEST_ES_TRANSPORT_PORT
* Fix test that uses hard-coded es transport port and add it back in to parallel groups
* Disable checks reporter again for now
* Set env var for TEST_ES_TRANSPORT_PORT in pipeline
* Update Jenkinsfile for shorter testrunner labels
* Fix another hard-coded transport port
* Fix a new test with hard-coded URLs
* Jenkinsfile cleanup and fix one of the groups
* Fix double slash
* Testing vault credentials on jenkins server
* Add a non-existent credential
* Revert "Add a non-existent credential"
This reverts commit 0dc234c465a5483b1a994cb510a182fef766e9cc.
* Try github-checks-reporter again
* github-checks-reporter should only run for elastic/kibana, forks won't work
* Clean up some debug code
* Changing names around to try to make BlueOcean UI a little better
* Add more stages
* Make some changes to stage structure to mirror a nested example from CloudBees
* Handle TODOs, and some cleanup in Jenkinsfile
* Pass GIT_BRANCH when started without GHPRB, fix branch check
* Fix mailer problem and add code that ensures all tests are in cigroups back in
* Test adding worker/job name to junit report paths
* Remove some duplication from ci_setup scripts
* Fix unit test that uses junit path
* Don't reinstall node every time setup_env is run
* Fix yarn install logic
* Fix another unit test that uses junit output dir
* Download latest ES snapshot after kibana builds
* Make sure junit reports are always processed
* Add two failing tests for testing purposes
* Add support to Jenkinsfile for kibana build e-mails
* Remove some debug code for email sending
* Change JOB env handling in junit paths and move it to a sub-directory
* Revert "Add two failing tests for testing purposes"
This reverts commit 5715203e26922a93483feb0ebb8bb3fdcc3daf8c.
* Fix junit report path in test
* Don't send kibana emails on build abort
* Address PR feedback, formatting and use built-in url formatting library
* Fix path formatting for functional test
* Add email sending back in to Jenkinsfile
* Fix another unit test with path problem
(cherry picked from commit 27d23c4184)
# Conflicts:
# src/dev/ci_setup/setup.sh
# src/dev/failed_tests/cli.js
* [ftr/percy] integrate percy with functional test runner
* execute ftr within `percy exec`
* share PERCY_BIN and GRUNT_BIN vars in setup
* [license] whitelist `(Unlicense OR Apache-2.0)`
* [percy] include pr/parallel info in env
* echo the actual variable value
* destructure required modules
* [ci] calculate percy env in a script
* remove outdated eslint overrides
* oops, nonce should be the same for each build
* take snapshots in context tests
* add percy snapshots to some dashboard tests
* account for extra execution of PERCY_BIN
* remove usage of percy service
* rename service to `visualTesting`
* write a simple test
* switch left-over percy test to visualTesting
* set log level for percy agent
* trigger ci
* try using the system chrome install instead of downloading one
* fix export definition
* Don't skip chromium download
* Add Spencer's fix to point to chrome installation
* Attempting to split tests into own ci job and test files
* Renumber job
* Add tag to decorations file
* Try new files with existing CI job
* Try again to create a new CI job
* Fix eslint problem with space
* eslint updates
* fix console test
* make test names unique
* Update ci group names
* Try old name again
* save
* add separate visual_regression ci job
* add visual regression ci job
* add new job to jobs.yml
* update path to take_percy_snapshot script
* use fixed percy job total
* fix common services import
* move visual_create_and_add_embeddables to visual_regression project
* finish moving create_and_add_embeddables
* add x-pack visual regression job
* migrate dashboard_snapshots tests to visual_regression
* remove references to removed test files
* restore some unnecessary changes
* reimplement the login page tests
* yarn.lock update
* fix test
* remove old CI_GROUP 100 block
* update failed_tests trap
* reduce yarn.lock changes
* disable debug logging
* disable visual-regression tests for now, we can enable them in specific prs
* skip and adjust functional tests for Firefox
* downgrade geckodriver to 0.22.0
* [ftr] add firefox specific config file
* remove hard coded firefox adjustments
* remove firefox specific screenshot directory
* run functional tests in firefox in x-pack
* pass the logger to readConfigFile()
* rename local var
* skip xpack firefox config
* run xpack firefox functional tests in separate command
* update report name for XPack firefox results
* adjust viz shared item test
* skip shared item test
* [test/feature_controls] increase timeout to 20 sec & use forceLogout
* FF timeout on TSVB, refresh page on failure for url navigation
* [feature_controls/visualize_security] increase timeout to 20 sec
* skip dev_tools_spaces tests on FF
* run all groups 3 times
* skip more func tests
* skip more tests
* skip all feature controls tests for Firefox
* revert back changes in feature controls tests
* rename xpack report for chrome
* skip tile map for FF, wait for render before saving
* Revert "run all groups 3 times"
This reverts commit 12b4f69627.
* revert refresh on login
* apply feedback
* Split ciGroup3 and ciGroup5 (#36928)
* split large x-pack ciGroup5 into 3 groups
* split large x-pack ciGroup3 into 2 groups
* Add groups to the jenkins xpack ci group file
* Re-level some more work
* move es_search_source test to new describe block
* move es_search_source to first in ciGroup7
* move to x-pack-ciGroup6
* Typescript sample panel action
* Update EUI version to match main cabana version
* update yarn.lock
* add back typings include
* use correct relative path
* [ci/es] base default es version on pkg.branch value
* [ci/env] initialize $TEST_ES_FROM in checkout_sibling_es.sh
* [UA] remove TEST_ES_SNAPSHOT_VERSION override
* [ci/es] call checkout_sibling_es.sh from one location
* [ci] remove unused `--from` param
* [ci/env] always default to snapshots, switch to source when necessary
* [kbn/test] default esFrom to $TEST_ES_FROM
* [ci/setup] fix define order
* [ci/grunt] don't pass --esFrom let env pass through
* [ci/env] use branch, not version
* [ci] use same indent style as following lines
* [kbn/test] apply default values when processing args
* [kbn/test] simplify defaults, read default on each process
* Use hosts instead of url for elasticsearch.
* Adds jest integration tests to run in xpack intake.
* Reverting the exclusion of throwing errors in tests.
* Strange test that seems to fail without console error exception being thrown.
* Fix issues with wrong props for EuiInMemoryTable.
* Removes the masking of console errors in tests.
Since this is no longer part of a reporting task in Jenkins, we no
longer need to re-run kbn bootstrap when generating a report.
We use the PR_SOURCE_BRANCH environment variable to prevent genererating
Github issues when we are on a PR.
Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
* Reporting phantom and chromium tests should run in parallel.
* Chromium tests should be done in group 9.
* Attempting to use group 2 in ci for chromium tests.
* X-pack CI jobs should have 7 groups.
* Phantom tests should be in group 7.
* [ci][ftr][kbn/test] split up CI jobs
* [ci] run --assert-none-excluded in ci groups, before builds
* [ftr] improve error message when excluded tests found
* [ci] document other places the ciGroups live
* chore(19674): added dev flag when calling jenkins:unit grunt task so we can check dev licenses on ours ci jobs.
* chore(NA): update yarn lock files with integrity info.
* Replace xvfb with headless browser
* Update karma to run in headless chrome
Update Karma tests for an option to run chrome in headless.
* Fix lint issues
* Fix spaces
Closes#22840
In the functional tests we want a better way to include/exclude certain tests, especially as we move forward with #22359. This PR allows us to decorate suite objects with "tags", which won't clutter up the test names and can be used to filter out specific tests within a single test config. The functional test runner supports defining `--include-tag` and `--exclude-tag` CLI arguments, and multiple can be defined.
The method of filtering out tests for running against cloud has been updated to use this approach and I plan to do the same to #22359 once this is merged.
For instances where we would otherwise be running from source in a tracked elastic/elasticsearch branch, let's use a snapshot instead. This will eliminate some gradle issues we are experiencing in master.
* Turn reporting snapshot tests back on with new baselines
* Use snapshots generated from chromium. Bump the threshold so phantom comparisons pass
* Increase timeouts because of chromium functional tests.
* remove accidentally checked in session snapshots
* add a better vis baseline, run once through at 0 to check expected threshold
* turn on verbose logs
* Use threshold, turn debug mode on in jenkins
* Wrap both convert pages in the retry
* Update baselines
Preserve layout one got messed up - it was actually a print layout.
* bump threshold even more
* Replace test:api with @kbn/test runTests
* Improve CLI help menu 🆘
* Use --es-from
* Replace jenkins:selenium with kbn-test
* Validate cli args, fixing test in the process
* Clean up some stuff
* Code review fixes
* Explanation for collectCliArgs
* Remove exit codes, they're useless anyway.
* Make markdown vis test pass with dev_mode setting
* Tests
* Remove unneeded export
* Code review: move console logging up to cli.js
* Code review: refactor startServers and runTests to take single options collection
* Code review: Remove all things I am sure we do not use
* Improve tests
* Code review fixes
* Pass created log to runFtr, runElasticsearch, runKibanaServer
* Update --es-from option to --esFrom
* [mocha] run tests with mocha script, remove grunt-simple-mocha
* [ci] force colored output
* [ci] disable color when reading the yarn bin
* [dev/build/exec] support tooling log having its own chalk
* [dev/mocha] avoid changing the cwd of the process
* [dev/mocha] only match files, speed up negative matches
* [cli/serve/test] strip ansi control characters from logs
* [ci] disable color in all `yarn bin` invocations
* [dev/mocha] cleanup old runInBand check
* [dev/mocha] enable gloabl leak checks
Restructure testing with kbn-test package
- Run with multiple configs, move cli options to config
- Package-ify kbn-test
- Eventually we'll have functional_test_runner live in a package
of its own, and then this kbn-test will use that as a dependency,
probably still as a devDependency.
- Implement functional_tests_server
- Collapse single and multiple config apis into one command
Use kbn-es
Replace es_test_cluster + es_test_config with kbn/test utils
Implement new createEsTestCluster
Improve scripts, jsdocs, cli top-level tools
Lift error handling to the top level
* Add test-subj-selector module to packages dir.
* Rename to @kbn/test-subj-selector.
* Simplify rejectRejFiles Grunt task and ignore nested node_modules.
* Update jenkins_unit script to use yarn kbn CLI.
* switch to yarn
* cleanup misc references to npm
* [yarn] loosen dependency ranges so yarn will merge more deps
* fix linting error now that moment uses ESM
* [licenses] font-awesome changed the format of its license id
* Use local yarn
* Misc fixes
* eslintignore built yarn file
* Remove mkdir which doesn't do what it should do
* Check build without upgrading lots of versions
* Fix license check
* too many moments
* Better description
* Review fixes
* Lock to angular@1.6.5
* More specific version locks
* Revert "More specific version locks"
This reverts commit 11ef81102e.
* Revert "Lock to angular@1.6.5"
This reverts commit 3ade68c14c.
* rm yarn.lock; yarn
* Forcing a specific version of React, Angular, Moment
* Using vendored version of yarn in ci
* Use --frozen-lockfile
* fixes
* [dev/ci_setup] generalize jenkins_setup script for other CI environments
* [dev/ci_setup] accept more general `CACHE_DIR` env arg
* [dev/ci_setup] node.js install might not be invalid
* build before running selenium
* change directory into built snapshot
* WIP: what is in the build dir
* WIP: skip unit tests
* run selenium on build
* [functional tests on build] Derive version
* [functional tests on build] Missing folder name prefix
* WIP: debug on ubuntu VM
* WIP: double kibana stabilize timeout
* WIP: remove ubuntu VM debug change
* WIP: add logs for docParams
* WIP: add logs for es
* build release version without -SNAPSHOT-
* [functional test release] Cleanup
We always run the build task now in jenkins jobs rather than only in the
core builds. This ensures that the build task runs in PR builds even
though the build artifacts are not used for anything.
Ideally all the selenium tests would run against this build rather than
it just being something that gets ran and forgotten about, but this is a
first step.
Until now, the ci setup tasks were stored in the Jenkins job configuration. This moves the setup into source control so that jenkins can be told to execute `./test/scripts/jenkins_{JOB_TYPE}.sh` no matter what branch of kibana it is dealing with.
