.. | ||
api | ||
configs | ||
es_archives | ||
functional | ||
services | ||
README.md |
The Reporting Tests
Overview
Reporting tests have their own top level test folder because:
- We run the same tests with different kibana.yml settings for your browser choice - Chromium or Phantom.
- Current API tests run with
optimize.enabled=false
flag for performance reasons, but reporting actually requires UI assets. - Reporting tests take a lot longer than other test types. This separation allows developers to run them in isolation, or to run other functional or API tests without them.
Starting in 7.0 Phantom support will be removed and we can remove the phantom test versions.
Running the tests
There is more information on running x-pack tests here: https://github.com/elastic/kibana/blob/master/x-pack/README.md#running-functional-tests. Similar to running the API tests, you need to specify a reporting configuration file. Reporting currently has four configuration files you can point to:
- test/reporting/configs/chromium_api.js
- test/reporting/configs/phantom_api.js
- test/reporting/configs/chromium_functional.js
- test/reporting/configs/phantom_functional.js
The api
versions hit the reporting api and ensure report generation completes successfully, but does not verify the output of the reports. This is done in the functional
test versions, which does a snapshot comparison of the generated URL against a baseline to determine success.
To run the tests in a single command. :
- cd into x-pack directory.
- run:
node scripts/functional_tests --config test/reporting/configs/[config_file_name_here].js
You can also run the test server seperately from the runner. This is beneficial when debugging as Kibana and Elasticsearch will remain up and running throughout multiple test runs. To do this:
- cd into x-pack directory.
- In one terminal window, run:
node scripts/functional_tests_server.js --config test/reporting/configs/[test_config_name_here].js
- In another terminal window, cd into x-pack dir and run:
node ../scripts/functional_test_runner.js --config test/reporting/configs/[test_config_name_here].js
Prerequisites
The reporting functional tests use pdf-image to convert PDF's pages to png files for image comparisons between generated reports and baseline reports.
pdf-image requires the system commands convert
, gs
, and pdfinfo
to function. Those can be set up by running the following.
//OSX
brew install imagemagick ghostscript poppler
//Ubutnu
sudo apt-get install imagemagick ghostscript poppler-utils
For windows:
ImageMagick-6.9.9-37-Q16-HDRI-x64-dll.exe from https://sourceforge.net/projects/imagemagick/postdownload Install with all default options
gs925w64.exe from https://github.com/ArtifexSoftware/ghostpdl-downloads/releases/download/gs925/gs925w64.exe Install with all default options
Note: Configurations from kibana.dev.yml
are picked up when running the tests. Ensure that kibana.dev.yml
does not contain any xpack.reporting
configurations.
Reporting baseline snapshots
The functional version of the reporting tests create a few pdf reports and do a snapshot comparison against a couple baselines. The baseline images are stored in ./functional/reports/baseline
.
Note: The snapshot comparisons use a threshold due to expected visual difference when running on different browsers and different Operating Systems. This threshold is currently very high because of differences between Chromium and Phantom versions, and also because of some slight bugs with the Phantom version. The bug is here, the issue tracking this very high threshold is here. Once we remove Phantom support in 7.0, we can drop this threshold and hopefully catch more bugs!
Updating the baselines
Every now and then visual changes will be made that will require the snapshots to be updated. This is how you go about updating it. I will discuss generating snapshots from chromium since that is the way of the future.
- Run the test server for chromium.
node scripts/functional_tests_server.js --config test/reporting/configs/chromium_functional.js
- Run the test runner
node ../scripts/functional_test_runner.js --config test/reporting/configs/chromium_functional.js
- This will create new report snapshots in
./functional/reports/session/
. - After manually verifying the new reports in the
session
folder, copy them into ./functional/reports/baseline/ - Create a new PR with the new snapshots in the baseline folder.
Note: Dashboard has some snapshot testing too, in _dashboard_snapshots.js
. This test watches for a command line flag --updateBaselines
which automates updating the baselines. Probably worthwhile to do some similar here in the long run.
Adding a new BWC test
We have tests that ensure the latest version of Kibana will continue to generate reports from URLs generated in previous versions, to ensure backward compatibility. These tests are in api/bwc_generation_urls.js
. It's important to update these every now and then and add new ones, especially if anything in the URL changed in a release.
To add test coverage for a specific minor release,:
- Checkout previous branch, e.g.
git checkout upstream/6.4
- Sync your environment via
yarn kbn bootstrap
(Note, if you run into problems you may want to first clean viayarn kbn clean
) - Start up kibana and Elasticsearch (
yarn es snapshot --license trial
in one terminal, andyarn start
in another) - Load the reporting test data that is used in the tests. Ensure you are in the
x-pack
directory and run:
node ../scripts/es_archiver.js --es-url http://elastic:changeme@localhost:9200 load ../../../../test/functional/fixtures/es_archiver/dashboard/current/kibana
^^ That loads the .kibana index.
node ../scripts/es_archiver.js --es-url http://elastic:changeme@localhost:9200 load ../../../../test/functional/fixtures/es_archiver/dashboard/current/data
^^ That loads the data indices.
Note: Depending on your kibana.yml configuration, you may need to adjust the username, pw, and port in the urls above.
- Navigate to Kibana in the browser (
http://localhost:5601
) - Log in, pick any index to be the default to get page the management screen (doesn’t matter)
- Generate some reporting URLs
- Use a mixture of Visualize, Discover (CSV), Dashboard
- Can view the current test coverage by checkout out api/generation_urls.js. You can use different ones for better test coverage (e.g. different dashboards, different visualizations).
- Don’t generate urls from huge dashboards since this is time consuming.
- Use dashboards that have time saved with them if you wish to have data included.
- Save these reporting urls.
- Navigate back to the main branch via
git checkout master
. Then create, or work off your branch as usual to add the extra test coverage. - Copy the urls into
api/generation_urls.js
, stripping out the domain, port and base path (if they have it). - Write your new tests in api/bwc_generation_urls.js
- Run tests via the mechanism above.
Note: We may at some point wish to push most of these tests into an integration suite, as testing BWC of urls generated in every single minor, especially if there were not notable changes, may be overkill, especially given the time they add to the ci.
Expanding test coverage by including more data
As Kibana development progresses, our existing data indices often fail to cover new situations, such as:
- Reports with new visualization types
- Canvas reports
- Reports with visualizations referencing new index types (e.g. visualizations referencing rolled up indices)
- etc
Every now and then we should expand our test coverage to include new features. This is how you go about doing that in the context of reporting:
-
Checkout the
[version].x
branch, whereversion
is the currently working minor version (e.g. 6.x, notmaster
). This is because we don't want to run tests from data generated from a master version. The opposite works fine however - tests on master can run against data generated from the last minor. At least generally, though major version upgrades may require updating archived data (or be run through the appropriate migration scripts). -
Load the current archives via:
node ../scripts/es_archiver.js --es-url http://elastic:changeme@localhost:9200 load ../../../../test/functional/fixtures/es_archiver/dashboard/current/kibana
^^ That loads the .kibana index.
node ../scripts/es_archiver.js --es-url http://elastic:changeme@localhost:9200 load ../../../../test/functional/fixtures/es_archiver/dashboard/current/data
^^ That loads the data indices.
Note: Your es-url parameter might be different, but those are the default ports if running via yarn start
and yarn es snapshot --license trial
.
- Now generate the new data, create new index patterns, create new visualizations, and create new dashboards using those visualizations. All the fun stuff that you may want to use in your tests.
Note: This data is used in open source dashboard testing. All visualizations and saved searches that have Rendering Test
in their name are dynamically added to a new dashboard and their rendering is confirmed in https://github.com/elastic/kibana/tree/master/test/functional/apps/dashboard/_embedddable_rendering.js. You may need to adjust the expectations if you add new tests (which will be a good thing anyway, help extend the basic rendering tests - this way issues are caught before it gets to reporting tests!). Similarly all visualizations and saved searches that have Filter Bytes Test
in their name are tested in https://github.com/elastic/kibana/tree/master/test/functional/apps/dashboard/_dashboard_filtering.js
Note: The current reporting tests add visualizations from what is in PageObjects.dashboard.getTestVisualizationNames
. We should probably instead use a saved dashboard we generate this report from. Then you can add any new visualizations, re-save the dashboard, and re-generate the snapshot above.
- After adding more visualizations to a test dashboard, update tests if necessary, update snapshots, then save the new archives!
node ../scripts/es_archiver.js --es-url http://elastic:changeme@localhost:9200 save ../../../../test/functional/fixtures/es_archiver/dashboard/current/kibana
^^ That saves the .kibana index.
node ../scripts/es_archiver.js --es-url http://elastic:changeme@localhost:9200 save ../../../../test/functional/fixtures/es_archiver/dashboard/current/data log* animal* dog* [ANY OTHER NEW INDEX NAME YOU ADDED]*
^^ That saves the data indices. The last parameter is a list of indices you want archived. You don't want to include the .kibana
one in there (this way you can use a custom .kibana
, but can reuse the data archive, for tests, and .kibana
archive is small, but the data archives are larger and should be reused).
- Create your PR with test updates, and the larger archive.