Commit graph

108 commits

Author SHA1 Message Date
Spencer 7b4278dc49
[kbn/pm] add caching to bootstrap (#53622)
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-01-03 09:35:38 -07:00
Spencer e5c562b5c1
associate metadata with runnables (#53245) 2019-12-17 12:26:50 -07:00
spalger a926e3419d Revert "Update babel related packages (#52754)"
This reverts commit 9a115f078f.
2019-12-17 11:32:49 -07:00
renovate[bot] 9a115f078f Update babel related packages (#52754)
* Update babel related packages

* update kbn/pm dist
2019-12-14 20:14:47 -07:00
Spencer 81a30d98bd
[failedTests] fix reportMessageIter and add unit test (#52886)
* [failedTests] fix reportMessageIter and add unit test

* update related snapshots
2019-12-13 08:28:17 -07:00
Dmitry Lemeshko 7658e9c631
FTR: add 'throttle' option to cli (#33241)
* [ftr/cli] add throttling option

* [ftr/cli] add headless option, fix test
2019-12-11 19:41:25 +01:00
Spencer ab1fe3f14e
[kbnClient] Retry uiSettings.replace() calls up to 5 times (#52601)
* [kbn/dev-utils] target ES2019 to transpile ??

* Retry uiSettings.replace() calls up to 5 times

* share logic for selecting junit report name to ensure they are unique

* convert to junit report path helper
2019-12-11 09:50:03 -07:00
Spencer a25bf49eb8
Add failure screenshot links to JUnit failures (#52449) 2019-12-11 09:42:43 -07:00
Spencer 6ea1b2ccee
[ftr/lifecycle] refactor to be typesafe (#52453)
* [ftr/lifecycle] refactor to be typesafe

* update test fixture
2019-12-09 18:46:45 -07:00
Spencer 8e8571bae0
[ci/reportFailures] --dry-run is overloaded, split it up (#52314)
* [ci/reportFailures] --dry-run is overloaded, split it up

* force some failures to verify the fix

* Revert "force some failures to verify the fix"

This reverts commit cf2a58e139.

* update readme to mention new flags

* remove unnecessary commas
2019-12-05 13:25:37 -07:00
Spencer 3c55e55b7a
[failed_tests_cli] update reports with links to github issues (#52048)
* [failed_tests_cli] update reports with links to github issues

* reorder test report hooks so that published Junit includes modified reports

* force failures and enable dry-run mode for debugging

* auto-switch to --dry-run when running in non-tracked branches/prs

* add --skip-junit-update flag to skip mutating the reports

* remove comma after URL to support auto-linking in Jenkins

* Revert "force failures and enable dry-run mode for debugging"

This reverts commit ac0c287a3f.

* fix method call

* extend TestResult to include relevence flag rather than wrapping

* fix createFailureIssue() tests

* make report messages more consistent, append when not dry-run

* rename module

* update snapshots to not contain valid xml

* don't send authorization header if no token defined

* merge with master modified fixtures
2019-12-05 08:46:27 -07:00
Spencer 312821f644
filter out downstream failures with updated error message (#51623) 2019-12-03 13:42:38 -07:00
Spencer 03dad2827e
run mocha tests from x-pack with root mocha script (#51352)
* run mocha tests from x-pack with root mocha script

* Only run Karma tests in xpack intake job

* disable failing suites

* fix typo

* skip correct suite (there are multiple root suites)

* support disabling junit reporting with $DISABLE_JUNIT_REPORTER

* don't generate junit in ispec_plugin tests
2019-11-25 11:39:20 -07:00
Dmitry Lemeshko 6bc43bb9fd
[page_objects/common_page] convert to ts (#50771)
* [page_objects/common_page] convert to ts

* fix lint errors

* descrease navigation timeouts

* use template literal for log messages
2019-11-18 20:08:22 +01:00
renovate[bot] 73ebd567ad Update dependency rxjs to ^6.5.3 (#49837)
* Update dependency rxjs to ^6.5.3

* move argument type def up for better coverage

* adapt to possibly undefined helpExtension

* complete definition of deprecation$ type

* define types that bindNodeCallback can no longer infer

* define more types that bindNodeCallback can't infer

* be more explicit and accurate about the types for a subject/observable pair

* fix interface error, TS now identifies it

* ignore a return type flaw because types are not being managed properly

* avoid overspecifying types

* allow types to be inferred where possible

* remove unnecessary withLatestFrom()

* reduce number of rxjs versions installed

* update kbn/pm dist
2019-11-01 14:44:21 -07:00
Spencer a859577b40
Remove rimraf (#48985)
* Update dependency del to v5

* unify del/rimraf usage with del v5

* update yarn.lock

* update kbn-pm distributable

* remove poorly transformed code

* force some deletes from tmp

* mock less of the fs module

* force tmp deletion in a few more suites

* please make this the last force
2019-10-24 06:29:43 -07:00
Brian Seeders d68eba6d96
Ensure es/kibana cleanup always happens when using scripts/functional_tests (#48986) 2019-10-23 16:13:15 -04:00
Spencer 158d86f842
standardize notice-comment usage (#48677)
* standardize notice-comment usage

* regenerate notice.txt
2019-10-21 11:54:45 -07:00
Spencer 0a85478d41
[failedTestsReporter] load github issues on demand (#47914) 2019-10-11 09:56:47 -07:00
Spencer 23dada5722
fix indentation of failure issue bodies (#47864) 2019-10-10 12:52:35 -07:00
Spencer 36b55ab67e
[ci/failed_tests_reporter] when Github 500s, retry up to 5 tim… (#47740)
* [ci/failed_tests_reporter] when Github 500s, retry up to 5 times

* remove unused import

* properly handle non-string headers
2019-10-10 08:52:34 -07:00
Dmitry Lemeshko 3495f86c45
Collecting code coverage from functional tests (#40512)
* get coverage for console functional test

* instrument kibana

* collect coverage stats after each test, on url navigation, refresh and back

* switch to babel-plugin-istanbul

* switch to babel-plugin-istanbul

* [coverage service] basic version

* trigger final coverage flush when ftr finishes, wait for all logs before exiting

* add coverage report generation

* increase optimizer timeout, re-run idx/babel plugin before istanbul

* increase memory usage for node

* put istanbul preset in the beginning to run as the last one

* bump babel-plugin-istanbul up to 5.2.0

* cleanup

* save unique json files with coverage

* [functional test coverage] update coverage.json path

* run code coverage on CI

* increase max-old-space-size with env variable

* fix coverage folder creation

* use env variable to configure code coverage

* revert ci config changes

* remove duplicate plugin

* remove comments

* fixes
2019-10-10 07:56:49 +02:00
Spencer 9b34d6b76f
refactor failed_tests_reporter to use TS, no octokit (#46993)
* refactor failed_tests_reporter to use TS, no octokit

* update renovate config

* ensure that all kbn-test files are in ts project

* fix some type errors

* add some more tests

* [kbn-test/githubapi] cleanup and document

* collect log messages as strings instead of message objects

* ensure issue is open when updating body

* improve readability of getKibanaIssues

* expose axios helpers from dev-utils

* fix request params for fetching github issues and validate locally

* include a README for failed_tests_reporter

* improve axios error helpers
2019-10-08 13:56:07 -07:00
Tre 1947608378
[FTR] Support for new and old es clients (#47377)
* Add dep for new es client,
refactor all mentions of the old client
to the new.
2019-10-08 09:53:24 -06:00
renovate[bot] 445fa434d5 Update gulp related packages (major) (#46665)
* Update gulp related packages

* ts-ify and gulp4-ify x-pack tasks, remove unused canvas tasks

* remove unnecessary gulp.TaskFunction usage

* fix old references

* update renovate config

* move constants into helpers directory

* typo

* compact tasks a bit, remove unnecessary paths

* fix build directories

* deprecate testonly task

* rather than justifying an unjustifiable ts-ignore, ts-ify the imported module

* update renovate config

* update browser download tests to mock axios

* add root index.d.ts to tsconfig

* export BrowserType

* remove unnecessary `@ts-ignore`

* use consistent casing
2019-10-05 07:53:47 -07:00
renovate[bot] fc53d4efb3 Update dependency tar-fs to ^1.16.3 (#47341) 2019-10-05 05:29:50 -07:00
Lee Drengenberg 700a7ef576
Add option for Internet Explorer IE11 to FTR (#42967)
* Add option for Internet Explorer IE11

* WIP trying to figure out IE options/capabilities

* more attempts to get IE tests working

* more work, but still can't set font size in Dev Tools Console

* two changes for IE

* revert skip in console test

* this change *appears* to set absolute times but they don't stick

* Shakespeare passing on all 3 browsers

* add config.ie.js, change 'internet explorer' to 'ie'

* increase notifications:lifetime so ew have a chance to see the message

* fix lint error remove unused Options

* updates from PR review, remove console.log

* cleanup + reuse existing code

* make iedriver path work for x-pack tests

* try to avoid changing notification timeout

* resolve path and add correct delimiter

* remove unused code, make timepicker work on IE11

* work-around 45333 for IE11

* [WIP] removing some work-arounds

* revert work-arounds

* cleanup

* remove debugging code

* remove debugging code

* removed unused code

* reverting 2 files that didn't have any material changes
2019-10-04 11:13:58 -05:00
Spencer 3bc803e371
[mkdirp] remove in favor of recursive fs.mkdir (#47251)
* [mkdirp] remove in favor of recursive fs.mkdir

* add eslint rule to educate future contributors

* reword the eslint error message to prevent copy-pasting callback code

* Commit updated kbn-pm dist

Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>

* Fix typo

Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
2019-10-04 08:15:06 -07:00
renovate[bot] 0262f6184c Update mocha related packages (major) (#43915)
* Update mocha related packages

* update snapshots

* upgrade gulp-mocha so that it uses new mocha too

* fix async/cb overspecification
2019-10-01 00:07:13 -07:00
Brian Seeders 91090dbca3
Use unique junit report filenames again (#45897)
Fix disappearing test results in pipelines
2019-09-17 15:23:27 -04:00
Spencer fc186e57d5
[ftr/savedObjects] add simple saved object api client to ftr s… (#45856)
* [ftr/savedObjects] add simple saved object api client to ftr services

* fix typo

* use consistent spacing

* fix types and TS-ify the rest of kibanaServer service

* expose server urls with better API

* tweak status api response types

* fix http body param name

* second arg to axios post is the post data

* use standardized error handling behavior

* Revert "use standardized error handling behavior"

This reverts commit 7e9a7f8dc5.

* revert unnecessary changes
2019-09-17 12:07:28 -07:00
Brian Seeders 27d23c4184 Jenkins pipeline with parallel cigroups (#45285)
* Pipeline

* WIP some work for parallelization with ciGroups

* Fix xpack kibana install dir, and add some debugging

* Attempt to quick fix a few tests

* Revert "Revert "Revert "[ci] compress jobs for CI stability" (#44584)""

This reverts commit 078ac2897f.

* Recombine test groups, and try runbld again

* Mostly cleanup, and fix failed_tests reporting to hopefully work for both pipeline and non-pipeline

* Fix typo in shell script

* Remove some debug code

* Add support for changing es transport.port during testing via TEST_ES_TRANSPORT_PORT

* Fix test that uses hard-coded es transport port and add it back in to parallel groups

* Disable checks reporter again for now

* Set env var for TEST_ES_TRANSPORT_PORT in pipeline

* Update Jenkinsfile for shorter testrunner labels

* Fix another hard-coded transport port

* Fix a new test with hard-coded URLs

* Jenkinsfile cleanup and fix one of the groups

* Fix double slash

* Testing vault credentials on jenkins server

* Add a non-existent credential

* Revert "Add a non-existent credential"

This reverts commit 0dc234c465a5483b1a994cb510a182fef766e9cc.

* Try github-checks-reporter again

* github-checks-reporter should only run for elastic/kibana, forks won't work

* Clean up some debug code

* Changing names around to try to make BlueOcean UI a little better

* Add more stages

* Make some changes to stage structure to mirror a nested example from CloudBees

* Handle TODOs, and some cleanup in Jenkinsfile

* Pass GIT_BRANCH when started without GHPRB, fix branch check

* Fix mailer problem and add code that ensures all tests are in cigroups back in

* Test adding worker/job name to junit report paths

* Remove some duplication from ci_setup scripts

* Fix unit test that uses junit path

* Don't reinstall node every time setup_env is run

* Fix yarn install logic

* Fix another unit test that uses junit output dir

* Download latest ES snapshot after kibana builds

* Make sure junit reports are always processed

* Add two failing tests for testing purposes

* Add support to Jenkinsfile for kibana build e-mails

* Remove some debug code for email sending

* Change JOB env handling in junit paths and move it to a sub-directory

* Revert "Add two failing tests for testing purposes"

This reverts commit 5715203e26922a93483feb0ebb8bb3fdcc3daf8c.

* Fix junit report path in test

* Don't send kibana emails on build abort

* Address PR feedback, formatting and use built-in url formatting library

* Fix path formatting for functional test

* Add email sending back in to Jenkinsfile

* Fix another unit test with path problem
2019-09-11 11:58:28 -07:00
Spencer 16eb033a24
[ftr] support --kibana-install-dir flag (#44552)
* [ftr] support --kibana-install-dir flag

* improve spacing/style of success message

* Remove unused var
2019-09-05 13:06:10 -07:00
renovate[bot] 0001c74370 Update dependency del to ^4.1.1 (#44806) 2019-09-04 13:15:59 -07:00
Jonathan Budzenski bad947d7d1
move optimize into data folder (#25944)
* move optimize into data folder

* path

* replace optimize with data in notice glob

* update docs

* more updates

* update snapshots

* re-add folder references

* prettier

* save changes

* rm optimize from .eslintignore

* rm extra file

* fix merge
2019-08-28 10:09:58 -05:00
renovate[bot] dfddf3dbdd Update dependency chalk to ^2.4.2 (#44032)
* Update dependency chalk to ^2.4.2

* update kbn-pm dist
2019-08-27 21:58:53 -07:00
Dmitry Lemeshko d859b79a98
fix logging 'undefined' on tests completion (#43983) 2019-08-26 19:44:43 +02:00
Spencer 90de95822b
[mocha] fix test selectors to get everything in packages (#43797)
* [mocha] fix test selectors to get everything in packages

* fix paths in tests

* ignore test files that make it into target directories

* [@kbn/test] don't build test files

* move tests from datemath and test-subj pkgs into __tests__ dirs

* [kbn/test-subj] remove test script

* update yarn.lock
2019-08-22 18:49:01 -07:00
renovate[bot] 572274b779 Update babel related packages (#43595)
* upgrade all babel related modules, and bump lodash minors

* update kbn-pm dist

* update fetch-mock, necessary to use core-js 3

* use regenerator transform in jest tests, as required by EUI

* disable useBuiltIns rather than using regenerator plugin

* remove extra regenerator-runtime import
2019-08-22 18:40:57 -07:00
Spencer e8409c0160
Move src/dev/run to @kbn/dev-utils (#43387)
* move src/dev/run into `@kbn/dev-utils`

* forgot to save a replacement
2019-08-16 07:13:54 -07:00
Tre 5d8ad40c5c
[FTR] Refactor mocha under @kbn/test (#42862)
* Run prettier on the files that will be refactored under @kbn/test, in a follow-up pr.

* Fixup all paths to mocha,
under kbn-test, with needed
exports/imports.

* Fixed borked path (bad refactor).

* Fixup one more borked path.

* Fixup tsconfig for xml.ts

* Drop setup_node_env as it's already
in KIBANA/scripts/mocha.js.
Also, fixup cwd for globby as we are
exec-ing from a different directory.
2019-08-15 15:48:39 -06:00
Spencer b2515c780d
[watcher] add missing await (#41351)
* [watcher] add missing await

* run x-pack-firefoxSmoke ciGroup 40 times, run watcher tests 20 times per job

* Fix failing watcher test

* Revert "run x-pack-firefoxSmoke ciGroup 40 times, run watcher tests 20 times per job"

This reverts commit c5d891d44d.

* Clean up watches via api

* fix linting errors

* don't swallow errors, wait for completion

* strip trailing numbers in suite tags

* run x-pack-firefoxSmoke group 40 times

* Revert "run x-pack-firefoxSmoke group 40 times"

This reverts commit cc4eb6ce54.

* run x-pack-firefoxSmoke group 40 times

# Conflicts:
#	.ci/jobs.yml

* run tests 80 times

* Revert "run tests 80 times"

This reverts commit 4320c9488a.

* Revert "run x-pack-firefoxSmoke group 40 times"

This reverts commit cf0d4056d7.
2019-08-15 10:49:36 -07:00
Mike Côté cb0cabf0cc
Add Elasticsearch SSL support for integration tests (#41765)
* Allow tests to use SSL between Kibana and Elasticsearch

* Fix cert path

* By default no tests will use the new ssl feature
2019-08-08 18:38:16 -04:00
Mike Côté aaea5679d0
Add SSL support when starting Elasticsearch from yarn es (#42527)
* Initial work

* Add integration tests

* Use constants

* Fix broken code

* Handle scenario where esArgs is a string

* Remove || []

* Apply PR feedback

* Use const format
2019-08-07 10:34:44 -04:00
Tre e62ec7c5ec
[FTR] Refactor FTR to live under KBN-TEST (#42547)
Refactor the FTR to live under KBN-TEST . 

Next, refactor any and all files that the
FTR needs to live under KBN-TEST,
as needed.
2019-08-06 15:35:16 -06:00
Aleh Zasypkin 1a103c02e4
[@kbn/es] Add a predefined list of Elasticsearch secure settings to be added into keystore. Re-enable OIDC tests. (#42239) 2019-08-01 09:14:51 +02:00
Spencer a95ae8c037
[prettier] upgrade to 1.18.2 (#40229)
* [prettier] upgrade to 1.18.2

* autofix prettier violations
2019-07-03 12:36:44 -07:00
Mikhail Shustov a75d777bea
New Platform and Legacy platform servers integration (#39047)
* New and Legacy platforms share http server instance.

Required to use a common security interceptor for incoming http requests

* generate docs

* remove excessive contract method

* add test for New platform compatibility

* address comments part #1

* log server running only for http server

* fix test. mutate hapi request headers for BWC with legacy

* return 503 on start

* address @eli comments

* address @joshdover comments
2019-06-19 16:32:37 +02:00
liza-mae 0003a32f56
[WIP] FTR Percy changes (#36897)
* [ftr/percy] integrate percy with functional test runner

* execute ftr within `percy exec`

* share PERCY_BIN and GRUNT_BIN vars in setup

* [license] whitelist `(Unlicense OR Apache-2.0)`

* [percy] include pr/parallel info in env

* echo the actual variable value

* destructure required modules

* [ci] calculate percy env in a script

* remove outdated eslint overrides

* oops, nonce should be the same for each build

* take snapshots in context tests

* add percy snapshots to some dashboard tests

* account for extra execution of PERCY_BIN

* remove usage of percy service

* rename service to `visualTesting`

* write a simple test

* switch left-over percy test to visualTesting

* set log level for percy agent

* trigger ci

* try using the system chrome install instead of downloading one

* fix export definition

* Don't skip chromium download

* Add Spencer's fix to point to chrome installation

* Attempting to split tests into own ci job and test files

* Renumber job

* Add tag to decorations file

* Try new files with existing CI job

* Try again to create a new CI job

* Fix eslint problem with space

* eslint updates

* fix console test

* make test names unique

* Update ci group names

* Try old name again

* save

* add separate visual_regression ci job

* add visual regression ci job

* add new job to jobs.yml

* update path to take_percy_snapshot script

* use fixed percy job total

* fix common services import

* move visual_create_and_add_embeddables to visual_regression project

* finish moving create_and_add_embeddables

* add x-pack visual regression job

* migrate dashboard_snapshots tests to visual_regression

* remove references to removed test files

* restore some unnecessary changes

* reimplement the login page tests

* yarn.lock update

* fix test

* remove old CI_GROUP 100 block

* update failed_tests trap

* reduce yarn.lock changes

* disable debug logging

* disable visual-regression tests for now, we can enable them in specific prs
2019-06-14 10:35:23 -06:00
renovate[bot] 4b3915ea74 Update dependency tmp to v0.1.0 (#38590) 2019-06-10 15:59:18 -07:00