* chore(NA): upgrade to lodash@4 (#69868) * chore(NA): upgrade oss to lodash4 chore(NA): migrate cli, cli_plugin, cli_keystore, dev, test_utils and apm src script to lodash4 chore(NA): missing file for cli plugin chore(NA): add src core chore(NA): es archiver and fixtures chore(NA): try to fix functional test failure chore(NA): migrate src/legacy entirely to lodash4 except src/legacy/core_plugins chore(NA): move legacy core plugins to lodash4 chore(NA): upgrade optimize to lodash4 chore(NA): upgrade to lodash4 on advanced_settings, charts, console and dashboard chore(NA): migrate to lodash4 on dev_tools, discover, embeddable, es_ui)shared, expressions, home plugins chore(NA): upgrade data plugin to lodash4 chore(NA): upgrade usage_collection, ui_actions, tile_map, telemtry, share, saved_objects, saved_objects_management, region_map and navigation to lodash4 chore(NA): missing data upgrades to lodash4 Revert "chore(NA): upgrade usage_collection, ui_actions, tile_map, telemtry, share, saved_objects, saved_objects_management, region_map and navigation to lodash4" This reverts commit 137055c5fed2fc52bb26547e0bc1ad2e3d4fe309. Revert "Revert "chore(NA): upgrade usage_collection, ui_actions, tile_map, telemtry, share, saved_objects, saved_objects_management, region_map and navigation to lodash4"" This reverts commit f7e73688782998513d9fb6d7e8f0765e9beb28d1. Revert "chore(NA): missing data upgrades to lodash4" This reverts commit 92b85bf947a89bfc70cc4052738a6b2128ffb076. Revert "chore(NA): upgrade data plugin to lodash4" This reverts commit 88fdb075ee1e26c4ac979b6681d8a2b002df74c6. chore(NA): upgrade idx_pattern_mgt, input_control_vis, inspector, kbn_legacy, kbn_react, kbn_usage_collections, kbn_utils, management and maps_legacy to lodash4 chore(NA): map src plugin data to lodash3 chore(NA): missing lodash.clonedeep dep chore(NA): change packages kbn-config-schema deps chore(NA): update renovate config chore(NA): upgrade vis_type plugins to lodash4 chore(NA): move vis_type_vislib to lodash3 chore(NA): update visualizations and visualize to lodash4 chore(NA): remove lodash 3 types from src and move test to lodash4 chore(NA): move home, usage_collection and management to lodash 3 Revert "chore(NA): move home, usage_collection and management to lodash 3" This reverts commit f86e8585f02d21550746569af54215b076a79a3d. chore(NA): move kibana_legacy, saved_objects saved_objects_management into lodash3 chore(NA): update x-pack test to mock lodash4 Revert "chore(NA): move kibana_legacy, saved_objects saved_objects_management into lodash3" This reverts commit 2d10fe450533e1b36db21d99cfae3ce996a244e0. * chore(NA): move x-pack and packages to lodash 4 * chore(NA): remove mention to lodash from main package.json * chore(NA): remove helper alias for lodash4 and make it the default lodash * chore(NA): fix last failing types in the repo * chore(NA): fix public api * chore(NA): fix types for agg_row.tsx * chore(NA): fix increment of optimizer modules in the rollup plugin * chore(NA): migrate `src/core/public/http/fetch.ts` (#5) * omit undefined query props * just remove merge usage * fix types * chore(NA): fixes for feedback from apm team * chore(NA): recover old behaviour on apm LoadingIndeicatorContext.tsx * chore(NA): fixes for feedback from watson * Platform lodash4 tweaks (#6) * chore(NA): fix types and behaviour on src/core/server/elasticsearch/errors.ts * Canvas fixes for lodash upgrade * [APM] Adds unit test for APM service maps transform (#7) * Adds a snapshot unit test for getConnections and rearranges some code to make testing easier * reverts `ArrayList` back to `String[]` in the painless script within `fetch_service_paths_from_trace_ids.ts` * chore(NA): update yarn.lock * chore(NA): remove any and use a real type for alerts task runner Co-authored-by: Gidi Meir Morris <github@gidi.io> * chore(NA): used named import for triggers_actions_ui file * chore(NA): fix eslint * chore(NA): fix types * Delete most uptime lodash references. * Simplify. Clean up types. * [Uptime] Delete most uptime lodash references (#8) * Delete most uptime lodash references. * Simplify. Clean up types. * chore(NA): add eslint rule to avoid using lodash3 * chore(NA): apply changes on feedback from es-ui team * fix some types (#9) * Clean up some expressions types. * chore(NA): missing ts-expect-error statements * Upgrade lodash 4 vislib (#11) * replace lodash 3 with lodash 4 on vislib plugin * Further changes * further replacement of lodash3 to 4 * further work on upgrading to lodash 4 * final changes to update lodash * chore(NA): upgrade data plugin to lodash4 chore(NA): upgrade data plugin public to lodash4 chore(NA): fix typecheck task chore(NA): fix agg_config with hasIn chore(NA): assign to assignIn and has to hasIn chore(NA): upgrade data plugin server to lodash4 chore(NA): new signature for core api fix(NA): match behaviour between lodash3 and lodash4 for set in search_source * chore(NA): remove lodash3 completely from the repo * chore(NA): fix x-pack/test/api_integration/apis/metrics_ui/snapshot.ts missing content * chore(NA): fix lodash usage on apm * chore(NA): fix typecheck for maps * Patch lodash template (#12) * Applying changes from https://github.com/elastic/kibana/pull/64985 * Using isIterateeCall, because it seems less brittle * Also patching `lodash/template` and `lodash/fp/template` * Reorganizing some files... * Revising comment * Ends up `_` is a function also... I hate JavaScript Co-authored-by: Pierre Gayvallet <pierre.gayvallet@gmail.com> Co-authored-by: Josh Dover <me@joshdover.com> Co-authored-by: Clint Andrew Hall <clint.hall@elastic.co> Co-authored-by: Oliver Gupte <ogupte@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> Co-authored-by: Gidi Meir Morris <github@gidi.io> Co-authored-by: Justin Kambic <justin.kambic@elastic.co> Co-authored-by: Stratoula Kalafateli <stratoula1@gmail.com> Co-authored-by: Luke Elmers <luke.elmers@elastic.co> Co-authored-by: Brandon Kobel <brandon.kobel@gmail.com> Co-authored-by: kobelb <brandon.kobel@elastic.co> # Conflicts: # renovate.json5 # x-pack/plugins/apm/public/services/rest/observability_dashboard.ts # x-pack/plugins/canvas/common/lib/pivot_object_array.ts # x-pack/plugins/canvas/public/components/workpad_templates/workpad_templates.js # x-pack/plugins/canvas/server/collectors/custom_element_collector.ts # x-pack/plugins/canvas/server/collectors/workpad_collector.ts * chore(NA): updated kbn-pm dist file |
||
---|---|---|
.. | ||
common | ||
fixtures | ||
public | ||
server | ||
kibana.json | ||
README.md |
Rollup
Summary
Welcome to the Kibana rollup plugin! This plugin provides Kibana support for Elasticsearch's rollup feature. Please refer to the Elasticsearch documentation to understand rollup indices and how to create rollup jobs.
This plugin allows Kibana to:
- Create and manage rollup jobs
- Create rollup index patterns
- Create visualizations from rollup index patterns
- Identify rollup indices in Index Management
The rest of this doc dives into the implementation details of each of the above functionality.
Quick steps for testing
The pattern for creating a rollup job and rollup index pattern is:
- Install sample data (web logs is a good one).
- Create a rollup job with an index pattern that captures this index (e.g.
k*
). - Set frequency to "minute". Clear the latency buffer field.
- Select the time field which is the same time field selected in the installed index pattern (
timestamp
without an@
in the case of web logs). - Specify a time bucket size (
10m
will do). - Select a few terms, histogram, and metrics fields.
- Create and start the rollup job. Wait a minute for the job to run. You should see the numbers for documents and pages processed change in the detail panel.
- Create a rollup index pattern in the Index Patterns app.
- Now you can create visualizations using this index pattern.
Create and manage rollup jobs
The most straight forward part of this plugin! A new app called Rollup Jobs is registered in the Management section and follows a typical CRUD UI pattern. This app allows users to create, start, stop, clone, and delete rollup jobs. There is no way to edit an existing rollup job; instead, the UI offers a cloning ability. The client-side portion of this app lives in public/crud_app and uses endpoints registered in server/routes/api/jobs.
Refer to the Elasticsearch documentation to understand rollup indices and how to create rollup jobs.
Create rollup index patterns
Kibana uses index patterns to consume and visualize rollup indices. Typically, Kibana can inspect the indices captured by an index pattern, identify its aggregations and fields, and determine how to consume the data. Rollup indices don't contain this type of information, so we predefine how to consume a rollup index pattern with the type and typeMeta fields on the index pattern saved object. All rollup index patterns have type
defined as "rollup" and typeMeta
defined as an object of the index pattern's capabilities.
In the Index Pattern app, the "Create index pattern" button includes a context menu when a rollup index is detected. This menu offers items for creating a standard index pattern and a rollup index pattern. A rollup config is registered to index pattern creation extension point. The context menu behavior in particular uses the getIndexPatternCreationOption()
method. When the user chooses to create a rollup index pattern, this config changes the behavior of the index pattern creation wizard:
- Adds a
Rollup
badge to rollup indices usinggetIndexTags()
. - Enforces index pattern rules using
checkIndicesForErrors()
. Rollup index patterns must match one rollup index, and optionally, any number of regular indices. A rollup index pattern configured with one or more regular indices is known as a "hybrid" index pattern. This allows the user to visualize historical (rollup) data and live (regular) data in the same visualization. - Routes to this plugin's rollup
_fields_for_wildcard
endpoint, instead of the standard one, usinggetFetchForWildcardOptions()
, so that the internal rollup data field names are mapped to the original field names. - Writes additional information about aggregations, fields, histogram interval, and date histogram interval and timezone to the rollup index pattern saved object using
getIndexPatternMappings()
. This collection of information is referred to as its "capabilities".
Once a rollup index pattern is created, it is tagged with Rollup
in the list of index patterns, and its details page displays capabilities information. This is done by registering yet another config for the index pattern list extension points.
Create visualizations from rollup index patterns
This plugin enables the user to create visualizations from rollup data using the Visualize app, excluding TSVB, Vega, and Timelion. When Visualize sends search requests, this plugin routes the requests to the Elasticsearch rollup search endpoint, which searches the special document structure within rollup indices. The visualization options available to users are based on the capabilities of the rollup index pattern they're visualizing.
Routing to the Elasticsearch rollup search endpoint is done by creating an extension point in Courier, effectively allowing multiple "search strategies" to be registered. A rollup search strategy is registered by this plugin that queries this plugin's rollup search endpoint.
Limiting visualization editor options is done by registering configs to various vis extension points. These configs use information stored on the rollup index pattern to limit:
- Available aggregation types
- Available fields for a particular aggregation
- Default and base interval for histogram aggregation
- Default and base interval, and time zone, for date histogram aggregation
Identify rollup indices in Index Management
In Index Management, similar to system indices, rollup indices are hidden by default. A toggle is provided to show rollup indices and add a badge to the table rows. This is done by using Index Management's extension points.
The toggle and badge are registered on the client-side in public/extend_index_management.
Additional data needed to filter rollup indices in Index Management is provided with a data enricher.