Right now the functional tests are run against a distributable of Kibana in CI, but that distributable is running with `--env.name=development`. That causes the optimizer to run again before the tests can start and prevents the functional tests from running against the actual version of the application users will end up getting. This seems necessary for some tests, but not all of them, but I would like to get all of the tests running against the production version of Kibana soon.
This PR implements a second ftr config, `test/functional_production` that uses basically a copy of the `test/functional` config but with a few minor adjustments, removing the `--env.name=development` kbnServerArg and using a unique junit report name. To accomplish this I needed to modify the `@kbn/test` module to only pass the `--dev` flag to the Kibana server if it is being run in development mode, which it currently does by testing the args for `--env.name=development` or two args next to each other: `'--env.name'` and `'development'`. It does this by converting the `extraKbnOpts` option into an `addExtraKbnArgs` function, which is called with the final args just before passing them to the proc runner and given a chance to modify then after all other args are resolved (pulling from different places in config based on the build type, etc.)
Over the next couple weeks I'll push up PRs for individual test suites, migrating them over to the new production config, hopefully in a short period of time we will have all the function test suites back under that `test/function` config and can run them all against the Kibana server in production mode.
This PR implements the `uiSettings.overrides` setting which [when stored in kibana.yml or passed as config args when starting Kibana] allows forcing some uiSettings to always have a specific value. This setting accepts a map of uiSetting keys to values that will always be used to override whatever is stored in the config saved object.
When users view the settings in the advanced settings UI they are disabled and describe why they can't be changed.
Attempting to change these values from the uiSettings client/service/api is also prevented, causing a 400 error to be thrown and/or sent as the response.
We currently use the `retry` service to call a function over and over in a loop, waiting for it to run without throwing an error, and ultimately failing if it does not succeed before a timeout is exceeded. This is the easiest way to get certain interactions to work, either because we don't know when we should be able to execute the interaction successfully, or because the timing is just too tricky to plan out correctly. Another place where we are using the `retry` service, which I don't think is appropriate, is when we need to wait for a certain condition to be met. This is where `retry.waitFor()` comes in:
```js
await retry.waitFor('dashboard search to be enabled', async () => {
const searchInput = await testSubjects.find('savedObjectFinderSearchInput');
return await searchInput.isEnabled();
})
```
The `retry.waitFor()` method behaves much like the `retry.try()` method behind the scenes, calling a function over and over, but instead of waiting for it to run without throwing an error it waits for it to return a "truthy" value. It also requires a description string that is used to make rather nice log output and a more descriptive error message than something like https://github.com/elastic/kibana/blob/master/test/functional/apps/dashboard/_data_shared_attributes.js#L61-L67
* [kbn-test] convert kibana-install-dir flag to installDir option
* [kbn-test] replicate kibana-install-dir handling to startServers
* [ftr] try running functional tests in production in CI
* Revert "[ftr] try running functional tests in production in CI"
This reverts commit e5b94aa024.
* [core/public/legacyPlatform] exclude ui/test_harness from the distributable
* [optimizer] fix `process.env.IS_KIBANA_DISTRIBUTABLE` definition
* [optimizer] only define `process.env.IS_KIBANA_DISTRIBUTABLE` when needed
Adding a `webpack.DefinePlugin` slows down the optimizer a small amount,
so only apply it when it is necessary, and skip it if it is going to
be defined as "false".
* [kbn-test/startServer] don't run in --dev mode if running from dist
* [ftr/kibanaServer/version] attach `-SNAPSHOT` suffix to version if running build_snapshot
* Replace test:api with @kbn/test runTests
* Improve CLI help menu 🆘
* Use --es-from
* Replace jenkins:selenium with kbn-test
* Validate cli args, fixing test in the process
* Clean up some stuff
* Code review fixes
* Explanation for collectCliArgs
* Remove exit codes, they're useless anyway.
* Make markdown vis test pass with dev_mode setting
* Tests
* Remove unneeded export
* Code review: move console logging up to cli.js
* Code review: refactor startServers and runTests to take single options collection
* Code review: Remove all things I am sure we do not use
* Improve tests
* Code review fixes
* Pass created log to runFtr, runElasticsearch, runKibanaServer
* Update --es-from option to --esFrom
In order to make the license that applies to each file as clear as possible, and to be consistent with elasticsearch, we are adding Apache 2.0 license headers to the top of each file.
Existence of this header is enforced by eslint and tslint and missing headers were automatically added in the last commit by running:
```
node scripts/eslint --fix && node scripts/tslint --fix
```
* Revert "[DOCS] Removes redundant index.asciidoc files (#19192)"
This reverts commit d11b5aae9a.
* Revert "[typescript] add typescript support for the server and browser (#19104)"
This reverts commit c6112067fc.
* Revert "Option to run kibana from build for CI (#19125)"
This reverts commit 5969860303.
Restructure testing with kbn-test package
- Run with multiple configs, move cli options to config
- Package-ify kbn-test
- Eventually we'll have functional_test_runner live in a package
of its own, and then this kbn-test will use that as a dependency,
probably still as a devDependency.
- Implement functional_tests_server
- Collapse single and multiple config apis into one command
Use kbn-es
Replace es_test_cluster + es_test_config with kbn/test utils
Implement new createEsTestCluster
Improve scripts, jsdocs, cli top-level tools
Lift error handling to the top level
* [es][savedObjects/index] put template on each savedObject write
The elasticsearch plugin currently checks for the Kibana index on each iteration of the healthCheck, and creates it if it does not exist. This removes that step from the healthCheck and instead, before each savedObject is written to elasticsearch, ensures that Elasticsearch has the necessary index template should the write result in index creation.
The healthCheck still has the `patchKibanaIndex()` logic, which checks the type in the Kibana index and adds any missing types. This step now does nothing when the Kibana index does not exist, and does what it has always done when it does.
* [ftr] remove unused kibanaIndex service
(cherry picked from commit b1ef897dafeb6d43fe279776e44a9d793a389dc3)
* [savedObjects/integration] create now creates kibana index
* [es/healthCheck] remove use of format()
* [es/healthCheck/tests] use sinon assertions
* [es/patchKibanaIndex] test for kibana index missing behavior
* [savedObjects/errors] add tests for EsAutoCreateIndexError
* [savedObjects/config] deprecate and remove savedObjects.indexCheckTimeout config
* use dangling commas consistently
* [ui/error_auto_create_index] fix class names
* [ui/savedObjectsClient] no need to specify basePath
* [eslint] fix linting issue
* [eslint] upgrade to 4.10.0
* [eslint-config-kibana] limit jest config to jest test files
* [ui_framework] remove trailing comma from rest-spreads
* [dashboard/tests] tag jest helpers with .test.js suffix
* explicitly import expect.js where used
* [eslint] apply auto-fixes
* [eslint] manually add/wrap some parens for compliance
* [npm] point to local packages for testing/review
* [jest] remove .test extension from jest helpers
* [ui_framework] fix trailing comma removal from 3bc661a1c8
* [packages] upgrade eslint packages
* [uiSettings] auto upgrade savedConfig doc when missing
* naming tweaks
* fix comments
* ensure that rcVersions are not found within a version
* add some tests for non-single digit versions/rcs/betas
* return the condition, rather than using an if()
* assert that getUpgradeableConfig() is always called once
* [uiSettingsService] remove excess space
* [savedObjectsClient] only attempt to createOrUpgradeSavedConfig once
* [uiSettings/routes/tests] remove unused assert helper
* [functional/console] correct test title
* [ftr/kibanaServer/uiSettings] fix disableToastAutohide timeout
* [SavedObjectClient] emit detectable errors
* [uiSettingsService] consume new SavedObjectsClient errors
* [SavedObjectsClient] expose errorTypeHelpers as such
* [elasticsearch/tests] recreate error for each test
* [http] wait for elasticsearch plugin to be ready
* [shortUrl/tests] ensure that create request responds with 200
* [shortUrl] use errorTypeHelpers to filter errors
* [uiSettings/savedObjectsClientStub] stub errorTypeHelpers
* [SavedObjectsClient/errors] expose error module so tests can make errors
* [shortUrl/tests] use actual SavedObjectsClient errors
* [uiSettings/savedObjectsClientStub] use actual errors lib
* [SavedObjectsClient] use decorate instead of "wrap"
* [server/routes/uiSettings] refactor routes to forward Boom errors from uiSettings
* [uiSettings] colocate routes and service
* [testUtils/esTestCluster] use more standard api style
* [testUtils/es] add createCallCluster util
* [testUtils/esTestCluster] add getters for client/callCluster
* [es/healthcheck] ensure that healtcheck stops when server is stopped
* [uiSettings/routes] add param/payload validation
* [uiSettings/routes] add tests that verify error behaviors
* [es/tests] remove unused module
* [testUtil/es] add utility for starting es nodes in tests
* [ftr/tests] use esTestCluster util to start es
* [es/tests/routes] use esTestCluster to start own es
* [testUtils/kbnServer] disable uiSettings unless plugins are enabled
* [testUtils/esTestCluster] use standard test es port by default
* [server/http/tests] add esTestCluster to setup
* [test/config] unify es test config into a single module
* [testUtils/esTestCluster] directory is no longer configurable
* [testUtils/esTestCluster] throw when es.start() is called again without es.stop()
* [testUtils/esTestCluster] is* checks should not mutate state
* ensure timezone attribute is adhered to
Two issues - one the archiver was loading the visualize mapping after
the config setting so it was bring overwritten, and two,
uiSettings.replace wasn’t working correctly to begin with.
* extra line to ensure server is stabilized to prevent transitory failures
