Commit graph

120 commits

Author SHA1 Message Date
Spencer 18f973ea61
[junit] only include stdout in report for failures (#60530)
* [junit] only include stdout in report for failures

* fix assertion

Co-authored-by: spalger <spalger@users.noreply.github.com>
2020-03-18 17:49:40 -07:00
Thomas Watson 59a522b4ef
Upgrade @types/node to match Node.js runtime (#60368)
Kibana uses Node.js v10.19.0. The closest version of @types/node to this
version is currently v10.17.17.

This commit updates the resolutions field in package.json to ensure that
the latest version less than 10.20.0 is always used.
2020-03-18 14:27:56 +01:00
Rashmi Kulkarni 89f9260da2
FTR configurable test users (#52431)
* initial implementation of configurable test users

* user superuser by default to match master

* referenced the configs in reporting and api integration

* setting the minimum number of default roles

* looking for x-pack tests with users and roles

* add testUserService in dashboard mode tests

* running only ciGroup7

* uncommenting - addign visualization

* re-enabling all CI groups to run on CI

* reinstating Jenkinsfile

* disable Test user for OIDC config

* improved logging and added Roles for OSS tests to get better info on the runs.

* disable test_user for auth tests

* don't fetch enabledPlugins when testuser disabled

* fix es-lint

* running oss tests with x-pack enabled

* [revertme] build default dist for oss tests

* updating NOTICE.txt file as it complained in the kibana intake tests

* changed to pick OSS builds

* trying a license change to trial

* switch back to xpack builds

* created a new sample data role and used it in homepage tests

* revert test/scripts/jenkins_ci_group.sh

* only refresh browser and wait for chrome if we are already on Kibana page

* fix large_string test to use minimum set of roles and privileges

* fix for date nanos custom timestamp with a configured role

* changes to the files with addition of new roles for the test_user

* reverting to OSS changes and few additions to the time_zone test to run as a test_user

* changes to security

* changes to the x-pack test to use elastic superuser

* fix for chart_types test

* fixes to area chart , input control test

* fix for dashboard filtering test and a new config role

* changes to handle the x-pack tests

* additional role for date nanos mixed

* added the logstash role to the accessibility tests

* removed telemetry setting

* docs+few changes to the tests

* removed Page navigation

* removed pageNavigation which was unused

* test/accessibility/apps/management.ts

* update management.ts

* aria label, and other changes

* accidentally checked in a piped file with results.

* accidentally checked in a piped file with results.

* accidentally checked in a piped file with results.

* accidentally checked in a piped file with results.

* accidentally checked in a piped file with results.

* accidentally checked in a piped file with results.

* accidentally checked in a piped file with results.

* accidentally checked in a piped file with results.

* reverted

* unloading of logstash data, fixing aria label

* aria-label

* added the required role

* fix for tsvb chart

* fix for sample data test reverted home_page pageobject file

* changes to sample data test and visualize index file to incorporate OSS changes

* changes to describe() and some more changes to incorporate in settings_page

* re-adding the after()

* removed unwanted roles

* replaced kibana_user with kibana_admin

* added the check of deprecated kibana_user

* testing with kibana_admin  role

* fix for discover test

* incorporated the review comments

* incorporated the review comments

* incorporate review comments and added restoreDefaults()

* removed describe.only

* reverted the OSS logic change I had here- pulled into seperate PR

* incorporated the review comments

* incorporated review changes

* adding hidden=true to find hidden kibanaChrome

* change field.test.tsx to be same as that of master branch

Co-authored-by: spalger <spalger@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-03-17 10:41:23 -07:00
Brian Seeders ce64895b2e
[FTR] Add support for --include and --exclude files via tags (#60123) 2020-03-17 10:53:59 -04:00
Spencer 992e223700
[test-es-cluster] remove random cluster name (#58762)
* [test-es-cluster] remove random cluster name

* make cluster names unique in parallel test groups

* specify default customEsArgs

* share ci parallel prefix logic to avoid use of wrong env var

* remove - postfix from tag, add docs with explaination

* fix file name

* ci parallel process prefix doesn't need to be exported so don't

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-03-03 14:47:23 -07:00
Spencer 6cacfd0c15
[failed-test-report] if one test fails twice don't create two… (#58778)
* [failed-test-report] if one test fails twice don't create two issues

* fix type check error

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-03-03 08:58:50 -07:00
Brian Seeders c90cfe208d
[CI] Pipeline refactoring (#56447) 2020-03-02 15:04:27 -05:00
Spencer 4122705bc2
[kbn/test/ftr] add support for custom runners (#58610)
* [kbn/test/ftr] add support for custom runners

* remove irrelevant tests
2020-02-26 13:32:51 -07:00
Spencer 15e4e6c632
Fix failed test reporter for SIEM Cypress use (#57240)
* [@kbn/test] build with source-maps

* [@kbn/test] the jUnit Reporter being used for SIEM Cypress tests sometimes doesn't include testcases in XML

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-02-14 18:47:32 -07:00
Spencer 4e8ab56497
build immutable bundles for new platform plugins (#53976)
* build immutable bundles for new platform plugins

* only inspect workers if configured to do so

* [navigation] use an index.scss file

* add yarn.lock symlink

* set pluginScanDirs in test so fixtures stay consistent

* cleanup helpers a little

* fix type error

* support KBN_OPTIMIZER_MAX_WORKERS for limiting workers via env

* test support for KBN_OPTIMIZER_MAX_WORKERS

* expand the available memory for workers when only running one or two

* add docs about KBN_OPTIMIZER_MAX_WORKERS environment variable

* fix README link

* update kbn/pm dist

* implement bundle caching/reuse

* update kbn/pm dist

* don't check for cache if --no-cache is passed

* update renovate config

* standardize on index.scss, move console styles over

* add support for --no-cache to cli

* include worker config vars in optimizer version

* ignore concatenated modules

* update integration test

* add safari to browserslist to avoid user-agent warnings in dev

* update docs, clean up optimizer message/misc naming

* always handle initialized messages, don't ignore states that are attached to specific events

* reword caching docs, add environment var to disable caching

* tweak logging and don't use optimizer.useBundleCache as that's disabled in dev

* handle change notifications

* batch changes for 1 second

* rename CompilerState type to CompilerMsg

* getChanges() no longer needs to assign changes to dirs

* remove unused deps

* split up run_worker.ts and share cacheKey generation logic

* add a couple docs

* update tests and remove unused imports

* specify files when creating bundle cache key

* remove one more unused import

* match existing dev cli output more closely

* update kbn/pm dist

* set KBN_NP_PLUGINS_BUILT to avoid warning in CI

* avoid extending global window type

* add note to keep pluginScanDirs in sync

* pass browserslistEnv in workerConfig so it is used for cache key

* load commons.bundle.js in parallel too

* emit initialized+success states if all bundles are cached

* load bootstraps as quickly as possible

* skip flaky suite

* bump

* update jest snapshots

* remove hashing from cache key generation

* remove unnecessary non-null assertion

* improve docs and break up Optimizer#run()

* remove unused import

* refactor kbn/optimizer to break up observable logic, implement more helpful cache invalidation logic with logging

* fix tests

* add initializing phase

* avoid rxjs observable constructor

* remove unnecessary rxjs helper, add tests for bundle cache

* update consumers of optimizer

* update readme with new call style

* replace "new platform" with "kibana platform"

* fix a couple more renames

* add support for several plain-text file formats

* fix naming of OptimizerMsg => OptimizerUpdate, use "store" naming too

* one more OptimizerMsg update

* ensure bundles are not cached when cache config is false

* test for initializing states and bundle cache events

* remove unnecessary timeout change

* Remove unnecessary helpers

* Add tests for BundleCache class

* Add tests for Bundle class

* test summarizeEvent$

* missing paths are no longer listed in mtimes map

* add tests for optimizer/cache_keys

* Add some extra docs

* Remove labeled loop

* add integration test for kbn-optimizer watcher components

* querystring-browser removed

* tweak logging a smidge, improve info and final message

* remove unused imports

* remove duplication of getModuleCount() method

* move type annotation that validates things

* clear up the build completion message

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-02-12 19:42:42 -07:00
Pierre Gayvallet 38dc1cbd3c
Migrate es-archiver to typescript (#56008)
* migrate lib/archives and lib/docs

* migrate lib/indices

* migrate end of /lib

* migrate /actions

* migrate es_archiver

* migrate cli

* migrate tests

* use proper log stub

* add Record typing

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-02-03 12:03:50 +01:00
Joe Portner 13593344a0
Add lockfile symlinks (#55440)
This is to enable dependency scanning tools to correctly resolve
the dependencies in each package.json file that is used in
production.
2020-01-27 11:38:20 -05:00
Spencer 7b4278dc49
[kbn/pm] add caching to bootstrap (#53622)
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-01-03 09:35:38 -07:00
Spencer e5c562b5c1
associate metadata with runnables (#53245) 2019-12-17 12:26:50 -07:00
spalger a926e3419d Revert "Update babel related packages (#52754)"
This reverts commit 9a115f078f.
2019-12-17 11:32:49 -07:00
renovate[bot] 9a115f078f Update babel related packages (#52754)
* Update babel related packages

* update kbn/pm dist
2019-12-14 20:14:47 -07:00
Spencer 81a30d98bd
[failedTests] fix reportMessageIter and add unit test (#52886)
* [failedTests] fix reportMessageIter and add unit test

* update related snapshots
2019-12-13 08:28:17 -07:00
Dmitry Lemeshko 7658e9c631
FTR: add 'throttle' option to cli (#33241)
* [ftr/cli] add throttling option

* [ftr/cli] add headless option, fix test
2019-12-11 19:41:25 +01:00
Spencer ab1fe3f14e
[kbnClient] Retry uiSettings.replace() calls up to 5 times (#52601)
* [kbn/dev-utils] target ES2019 to transpile ??

* Retry uiSettings.replace() calls up to 5 times

* share logic for selecting junit report name to ensure they are unique

* convert to junit report path helper
2019-12-11 09:50:03 -07:00
Spencer a25bf49eb8
Add failure screenshot links to JUnit failures (#52449) 2019-12-11 09:42:43 -07:00
Spencer 6ea1b2ccee
[ftr/lifecycle] refactor to be typesafe (#52453)
* [ftr/lifecycle] refactor to be typesafe

* update test fixture
2019-12-09 18:46:45 -07:00
Spencer 8e8571bae0
[ci/reportFailures] --dry-run is overloaded, split it up (#52314)
* [ci/reportFailures] --dry-run is overloaded, split it up

* force some failures to verify the fix

* Revert "force some failures to verify the fix"

This reverts commit cf2a58e139.

* update readme to mention new flags

* remove unnecessary commas
2019-12-05 13:25:37 -07:00
Spencer 3c55e55b7a
[failed_tests_cli] update reports with links to github issues (#52048)
* [failed_tests_cli] update reports with links to github issues

* reorder test report hooks so that published Junit includes modified reports

* force failures and enable dry-run mode for debugging

* auto-switch to --dry-run when running in non-tracked branches/prs

* add --skip-junit-update flag to skip mutating the reports

* remove comma after URL to support auto-linking in Jenkins

* Revert "force failures and enable dry-run mode for debugging"

This reverts commit ac0c287a3f.

* fix method call

* extend TestResult to include relevence flag rather than wrapping

* fix createFailureIssue() tests

* make report messages more consistent, append when not dry-run

* rename module

* update snapshots to not contain valid xml

* don't send authorization header if no token defined

* merge with master modified fixtures
2019-12-05 08:46:27 -07:00
Spencer 312821f644
filter out downstream failures with updated error message (#51623) 2019-12-03 13:42:38 -07:00
Spencer 03dad2827e
run mocha tests from x-pack with root mocha script (#51352)
* run mocha tests from x-pack with root mocha script

* Only run Karma tests in xpack intake job

* disable failing suites

* fix typo

* skip correct suite (there are multiple root suites)

* support disabling junit reporting with $DISABLE_JUNIT_REPORTER

* don't generate junit in ispec_plugin tests
2019-11-25 11:39:20 -07:00
Dmitry Lemeshko 6bc43bb9fd
[page_objects/common_page] convert to ts (#50771)
* [page_objects/common_page] convert to ts

* fix lint errors

* descrease navigation timeouts

* use template literal for log messages
2019-11-18 20:08:22 +01:00
renovate[bot] 73ebd567ad Update dependency rxjs to ^6.5.3 (#49837)
* Update dependency rxjs to ^6.5.3

* move argument type def up for better coverage

* adapt to possibly undefined helpExtension

* complete definition of deprecation$ type

* define types that bindNodeCallback can no longer infer

* define more types that bindNodeCallback can't infer

* be more explicit and accurate about the types for a subject/observable pair

* fix interface error, TS now identifies it

* ignore a return type flaw because types are not being managed properly

* avoid overspecifying types

* allow types to be inferred where possible

* remove unnecessary withLatestFrom()

* reduce number of rxjs versions installed

* update kbn/pm dist
2019-11-01 14:44:21 -07:00
Spencer a859577b40
Remove rimraf (#48985)
* Update dependency del to v5

* unify del/rimraf usage with del v5

* update yarn.lock

* update kbn-pm distributable

* remove poorly transformed code

* force some deletes from tmp

* mock less of the fs module

* force tmp deletion in a few more suites

* please make this the last force
2019-10-24 06:29:43 -07:00
Brian Seeders d68eba6d96
Ensure es/kibana cleanup always happens when using scripts/functional_tests (#48986) 2019-10-23 16:13:15 -04:00
Spencer 158d86f842
standardize notice-comment usage (#48677)
* standardize notice-comment usage

* regenerate notice.txt
2019-10-21 11:54:45 -07:00
Spencer 0a85478d41
[failedTestsReporter] load github issues on demand (#47914) 2019-10-11 09:56:47 -07:00
Spencer 23dada5722
fix indentation of failure issue bodies (#47864) 2019-10-10 12:52:35 -07:00
Spencer 36b55ab67e
[ci/failed_tests_reporter] when Github 500s, retry up to 5 tim… (#47740)
* [ci/failed_tests_reporter] when Github 500s, retry up to 5 times

* remove unused import

* properly handle non-string headers
2019-10-10 08:52:34 -07:00
Dmitry Lemeshko 3495f86c45
Collecting code coverage from functional tests (#40512)
* get coverage for console functional test

* instrument kibana

* collect coverage stats after each test, on url navigation, refresh and back

* switch to babel-plugin-istanbul

* switch to babel-plugin-istanbul

* [coverage service] basic version

* trigger final coverage flush when ftr finishes, wait for all logs before exiting

* add coverage report generation

* increase optimizer timeout, re-run idx/babel plugin before istanbul

* increase memory usage for node

* put istanbul preset in the beginning to run as the last one

* bump babel-plugin-istanbul up to 5.2.0

* cleanup

* save unique json files with coverage

* [functional test coverage] update coverage.json path

* run code coverage on CI

* increase max-old-space-size with env variable

* fix coverage folder creation

* use env variable to configure code coverage

* revert ci config changes

* remove duplicate plugin

* remove comments

* fixes
2019-10-10 07:56:49 +02:00
Spencer 9b34d6b76f
refactor failed_tests_reporter to use TS, no octokit (#46993)
* refactor failed_tests_reporter to use TS, no octokit

* update renovate config

* ensure that all kbn-test files are in ts project

* fix some type errors

* add some more tests

* [kbn-test/githubapi] cleanup and document

* collect log messages as strings instead of message objects

* ensure issue is open when updating body

* improve readability of getKibanaIssues

* expose axios helpers from dev-utils

* fix request params for fetching github issues and validate locally

* include a README for failed_tests_reporter

* improve axios error helpers
2019-10-08 13:56:07 -07:00
Tre 1947608378
[FTR] Support for new and old es clients (#47377)
* Add dep for new es client,
refactor all mentions of the old client
to the new.
2019-10-08 09:53:24 -06:00
renovate[bot] 445fa434d5 Update gulp related packages (major) (#46665)
* Update gulp related packages

* ts-ify and gulp4-ify x-pack tasks, remove unused canvas tasks

* remove unnecessary gulp.TaskFunction usage

* fix old references

* update renovate config

* move constants into helpers directory

* typo

* compact tasks a bit, remove unnecessary paths

* fix build directories

* deprecate testonly task

* rather than justifying an unjustifiable ts-ignore, ts-ify the imported module

* update renovate config

* update browser download tests to mock axios

* add root index.d.ts to tsconfig

* export BrowserType

* remove unnecessary `@ts-ignore`

* use consistent casing
2019-10-05 07:53:47 -07:00
renovate[bot] fc53d4efb3 Update dependency tar-fs to ^1.16.3 (#47341) 2019-10-05 05:29:50 -07:00
Lee Drengenberg 700a7ef576
Add option for Internet Explorer IE11 to FTR (#42967)
* Add option for Internet Explorer IE11

* WIP trying to figure out IE options/capabilities

* more attempts to get IE tests working

* more work, but still can't set font size in Dev Tools Console

* two changes for IE

* revert skip in console test

* this change *appears* to set absolute times but they don't stick

* Shakespeare passing on all 3 browsers

* add config.ie.js, change 'internet explorer' to 'ie'

* increase notifications:lifetime so ew have a chance to see the message

* fix lint error remove unused Options

* updates from PR review, remove console.log

* cleanup + reuse existing code

* make iedriver path work for x-pack tests

* try to avoid changing notification timeout

* resolve path and add correct delimiter

* remove unused code, make timepicker work on IE11

* work-around 45333 for IE11

* [WIP] removing some work-arounds

* revert work-arounds

* cleanup

* remove debugging code

* remove debugging code

* removed unused code

* reverting 2 files that didn't have any material changes
2019-10-04 11:13:58 -05:00
Spencer 3bc803e371
[mkdirp] remove in favor of recursive fs.mkdir (#47251)
* [mkdirp] remove in favor of recursive fs.mkdir

* add eslint rule to educate future contributors

* reword the eslint error message to prevent copy-pasting callback code

* Commit updated kbn-pm dist

Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>

* Fix typo

Signed-off-by: Tyler Smalley <tyler.smalley@elastic.co>
2019-10-04 08:15:06 -07:00
renovate[bot] 0262f6184c Update mocha related packages (major) (#43915)
* Update mocha related packages

* update snapshots

* upgrade gulp-mocha so that it uses new mocha too

* fix async/cb overspecification
2019-10-01 00:07:13 -07:00
Brian Seeders 91090dbca3
Use unique junit report filenames again (#45897)
Fix disappearing test results in pipelines
2019-09-17 15:23:27 -04:00
Spencer fc186e57d5
[ftr/savedObjects] add simple saved object api client to ftr s… (#45856)
* [ftr/savedObjects] add simple saved object api client to ftr services

* fix typo

* use consistent spacing

* fix types and TS-ify the rest of kibanaServer service

* expose server urls with better API

* tweak status api response types

* fix http body param name

* second arg to axios post is the post data

* use standardized error handling behavior

* Revert "use standardized error handling behavior"

This reverts commit 7e9a7f8dc5.

* revert unnecessary changes
2019-09-17 12:07:28 -07:00
Brian Seeders 27d23c4184 Jenkins pipeline with parallel cigroups (#45285)
* Pipeline

* WIP some work for parallelization with ciGroups

* Fix xpack kibana install dir, and add some debugging

* Attempt to quick fix a few tests

* Revert "Revert "Revert "[ci] compress jobs for CI stability" (#44584)""

This reverts commit 078ac2897f.

* Recombine test groups, and try runbld again

* Mostly cleanup, and fix failed_tests reporting to hopefully work for both pipeline and non-pipeline

* Fix typo in shell script

* Remove some debug code

* Add support for changing es transport.port during testing via TEST_ES_TRANSPORT_PORT

* Fix test that uses hard-coded es transport port and add it back in to parallel groups

* Disable checks reporter again for now

* Set env var for TEST_ES_TRANSPORT_PORT in pipeline

* Update Jenkinsfile for shorter testrunner labels

* Fix another hard-coded transport port

* Fix a new test with hard-coded URLs

* Jenkinsfile cleanup and fix one of the groups

* Fix double slash

* Testing vault credentials on jenkins server

* Add a non-existent credential

* Revert "Add a non-existent credential"

This reverts commit 0dc234c465a5483b1a994cb510a182fef766e9cc.

* Try github-checks-reporter again

* github-checks-reporter should only run for elastic/kibana, forks won't work

* Clean up some debug code

* Changing names around to try to make BlueOcean UI a little better

* Add more stages

* Make some changes to stage structure to mirror a nested example from CloudBees

* Handle TODOs, and some cleanup in Jenkinsfile

* Pass GIT_BRANCH when started without GHPRB, fix branch check

* Fix mailer problem and add code that ensures all tests are in cigroups back in

* Test adding worker/job name to junit report paths

* Remove some duplication from ci_setup scripts

* Fix unit test that uses junit path

* Don't reinstall node every time setup_env is run

* Fix yarn install logic

* Fix another unit test that uses junit output dir

* Download latest ES snapshot after kibana builds

* Make sure junit reports are always processed

* Add two failing tests for testing purposes

* Add support to Jenkinsfile for kibana build e-mails

* Remove some debug code for email sending

* Change JOB env handling in junit paths and move it to a sub-directory

* Revert "Add two failing tests for testing purposes"

This reverts commit 5715203e26922a93483feb0ebb8bb3fdcc3daf8c.

* Fix junit report path in test

* Don't send kibana emails on build abort

* Address PR feedback, formatting and use built-in url formatting library

* Fix path formatting for functional test

* Add email sending back in to Jenkinsfile

* Fix another unit test with path problem
2019-09-11 11:58:28 -07:00
Spencer 16eb033a24
[ftr] support --kibana-install-dir flag (#44552)
* [ftr] support --kibana-install-dir flag

* improve spacing/style of success message

* Remove unused var
2019-09-05 13:06:10 -07:00
renovate[bot] 0001c74370 Update dependency del to ^4.1.1 (#44806) 2019-09-04 13:15:59 -07:00
Jonathan Budzenski bad947d7d1
move optimize into data folder (#25944)
* move optimize into data folder

* path

* replace optimize with data in notice glob

* update docs

* more updates

* update snapshots

* re-add folder references

* prettier

* save changes

* rm optimize from .eslintignore

* rm extra file

* fix merge
2019-08-28 10:09:58 -05:00
renovate[bot] dfddf3dbdd Update dependency chalk to ^2.4.2 (#44032)
* Update dependency chalk to ^2.4.2

* update kbn-pm dist
2019-08-27 21:58:53 -07:00
Dmitry Lemeshko d859b79a98
fix logging 'undefined' on tests completion (#43983) 2019-08-26 19:44:43 +02:00
Spencer 90de95822b
[mocha] fix test selectors to get everything in packages (#43797)
* [mocha] fix test selectors to get everything in packages

* fix paths in tests

* ignore test files that make it into target directories

* [@kbn/test] don't build test files

* move tests from datemath and test-subj pkgs into __tests__ dirs

* [kbn/test-subj] remove test script

* update yarn.lock
2019-08-22 18:49:01 -07:00