Commit graph

216 commits

Author SHA1 Message Date
Walter Rafelsberger 262e0754ff
[ML] Kibana API endpoint for histogram chart data (#70976)
- Introduces dedicated Kibana API endpoints as part of ML and transform plugin API endpoints and moves the logic to query and transform the required data from client to server.
- Adds support for sampling to retrieve the data for the field histograms. For now this is not configurable by the end user and is hard coded to 5000. This is to have a first iteration of this functionality in for 7.9 and protect users when querying large clusters. The button to enable the histogram charts now includes a tooltip that mentions the sampler.
2020-07-14 13:37:36 +02:00
Melissa Alvarez 1b1962f18c
[ML] DF Analytics creation and update: adds max_num_threads (#71318)
* add max_num_threads to edit flyout

* add maxNumThreads setting to job wizard

* add maxNumThreads to cloning
2020-07-13 16:51:22 -04:00
Michael Hirsch c5729b87d6
[ML] Adds siem_cloudtrail Module (#71323)
* adds siem_cloudtrail module

* updates logo to logoSecurity

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-07-13 12:35:04 -04:00
Quynh Nguyen cf3133a5df
[ML] Add switch to enable model plot annotations independently (#70678) 2020-07-09 09:15:57 -05:00
James Gowdy a86110488b
[ML] Fixing missing daily_model_snapshot_retention_after_days in job update schema (#71086) 2020-07-08 16:19:12 +01:00
Melissa Alvarez a4340f0ece
[ML] DF Analytics: add ability to edit job for fields supported by API (#70489)
* wip: add edit action to dfanalytics table

* add update endpoint and edit flyout

* show success and error toasts. close flyout and refresh on success

* show permission message in edit action

* update types

* disable update button if mml not valid

* show error in toast, init values are config values

* fix undefined check for allow lazy start

* prevent update if mml is empty
2020-07-06 15:10:01 -04:00
James Gowdy 97ad58c529
[ML] Changing shared module setup function parameters (#70589)
* [ML] Changing shared module setup function parameters

* changes based on review

* adding comments for documentation

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-07-03 08:30:17 +01:00
Tiago Costa 6c62c686cf
chore(NA): upgrade to lodash@4 (#69868)
* chore(NA): upgrade oss to lodash4

chore(NA): migrate cli, cli_plugin, cli_keystore,  dev, test_utils and apm src script to lodash4

chore(NA): missing file for cli plugin

chore(NA): add src core

chore(NA): es archiver and fixtures

chore(NA): try to fix functional test failure

chore(NA): migrate src/legacy entirely to lodash4 except src/legacy/core_plugins

chore(NA): move legacy core plugins to lodash4

chore(NA): upgrade optimize to lodash4

chore(NA): upgrade to lodash4 on advanced_settings, charts, console and dashboard

chore(NA): migrate to lodash4 on dev_tools, discover, embeddable, es_ui)shared, expressions, home plugins

chore(NA): upgrade data plugin to lodash4

chore(NA): upgrade usage_collection, ui_actions, tile_map, telemtry, share, saved_objects, saved_objects_management, region_map and navigation to lodash4

chore(NA): missing data upgrades to lodash4

Revert "chore(NA): upgrade usage_collection, ui_actions, tile_map, telemtry, share, saved_objects, saved_objects_management, region_map and navigation to lodash4"

This reverts commit 137055c5fed2fc52bb26547e0bc1ad2e3d4fe309.

Revert "Revert "chore(NA): upgrade usage_collection, ui_actions, tile_map, telemtry, share, saved_objects, saved_objects_management, region_map and navigation to lodash4""

This reverts commit f7e73688782998513d9fb6d7e8f0765e9beb28d1.

Revert "chore(NA): missing data upgrades to lodash4"

This reverts commit 92b85bf947a89bfc70cc4052738a6b2128ffb076.

Revert "chore(NA): upgrade data plugin to lodash4"

This reverts commit 88fdb075ee1e26c4ac979b6681d8a2b002df74c6.

chore(NA): upgrade idx_pattern_mgt, input_control_vis, inspector, kbn_legacy, kbn_react, kbn_usage_collections, kbn_utils, management and maps_legacy to lodash4

chore(NA): map src plugin data to lodash3

chore(NA): missing lodash.clonedeep dep

chore(NA): change packages kbn-config-schema deps

chore(NA): update renovate config

chore(NA): upgrade vis_type plugins to lodash4

chore(NA): move vis_type_vislib to lodash3

chore(NA): update visualizations and visualize to lodash4

chore(NA): remove lodash 3 types from src and move test to lodash4

chore(NA): move home, usage_collection and management to lodash 3

Revert "chore(NA): move home, usage_collection and management to lodash 3"

This reverts commit f86e8585f02d21550746569af54215b076a79a3d.

chore(NA): move kibana_legacy, saved_objects saved_objects_management into lodash3

chore(NA): update x-pack test to mock lodash4

Revert "chore(NA): move kibana_legacy, saved_objects saved_objects_management into lodash3"

This reverts commit 2d10fe450533e1b36db21d99cfae3ce996a244e0.

* chore(NA): move x-pack and packages to lodash 4

* chore(NA): remove mention to lodash from main package.json

* chore(NA): remove helper alias for lodash4 and make it the default lodash

* chore(NA): fix last failing types in the repo

* chore(NA): fix public api

* chore(NA): fix types for agg_row.tsx

* chore(NA): fix increment of optimizer modules in the rollup plugin

* chore(NA): migrate `src/core/public/http/fetch.ts` (#5)

* omit undefined query props

* just remove merge usage

* fix types

* chore(NA): fixes for feedback from apm team

* chore(NA): recover old behaviour on apm LoadingIndeicatorContext.tsx

* chore(NA): fixes for feedback from watson

* Platform lodash4 tweaks (#6)

* chore(NA): fix types and behaviour on src/core/server/elasticsearch/errors.ts

* Canvas fixes for lodash upgrade

* [APM] Adds unit test for APM service maps transform (#7)

* Adds a snapshot unit test for getConnections and rearranges some code to make testing easier

* reverts `ArrayList` back to `String[]` in the painless script within `fetch_service_paths_from_trace_ids.ts`

* chore(NA): update yarn.lock

* chore(NA): remove any and use a real type for alerts task runner

Co-authored-by: Gidi Meir Morris <github@gidi.io>

* chore(NA): used named import for triggers_actions_ui file

* chore(NA): fix eslint

* chore(NA): fix types

* Delete most uptime lodash references.

* Simplify. Clean up types.

* [Uptime] Delete most uptime lodash references (#8)

* Delete most uptime lodash references.

* Simplify. Clean up types.

* chore(NA): add eslint rule to avoid using lodash3

* chore(NA): apply changes on feedback from es-ui team

* fix some types (#9)

* Clean up some expressions types.

* chore(NA): missing ts-expect-error statements

* Upgrade lodash 4 vislib (#11)

* replace lodash 3 with lodash 4 on vislib plugin

* Further changes

* further replacement of lodash3 to 4

* further work on upgrading to lodash 4

* final changes to update lodash

* chore(NA): upgrade data plugin to lodash4

chore(NA): upgrade data plugin public to lodash4

chore(NA): fix typecheck task

chore(NA): fix agg_config with hasIn

chore(NA): assign to assignIn and has to hasIn

chore(NA): upgrade data plugin server to lodash4

chore(NA): new signature for core api

fix(NA): match behaviour between lodash3 and lodash4 for set in search_source

* chore(NA): remove lodash3 completely from the repo

* chore(NA): fix x-pack/test/api_integration/apis/metrics_ui/snapshot.ts missing content

* chore(NA): fix lodash usage on apm

* chore(NA): fix typecheck for maps

* Patch lodash template (#12)

* Applying changes from https://github.com/elastic/kibana/pull/64985

* Using isIterateeCall, because it seems less brittle

* Also patching `lodash/template` and `lodash/fp/template`

* Reorganizing some files...

* Revising comment

* Ends up `_` is a function also... I hate JavaScript

Co-authored-by: Pierre Gayvallet <pierre.gayvallet@gmail.com>
Co-authored-by: Josh Dover <me@joshdover.com>
Co-authored-by: Clint Andrew Hall <clint.hall@elastic.co>
Co-authored-by: Oliver Gupte <ogupte@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Gidi Meir Morris <github@gidi.io>
Co-authored-by: Justin Kambic <justin.kambic@elastic.co>
Co-authored-by: Stratoula Kalafateli <stratoula1@gmail.com>
Co-authored-by: Luke Elmers <luke.elmers@elastic.co>
Co-authored-by: Brandon Kobel <brandon.kobel@gmail.com>
Co-authored-by: kobelb <brandon.kobel@elastic.co>
2020-07-03 01:30:13 +01:00
Michael Hirsch 8a09f247e3
[ML] Updates APM Module to Work with Service Maps (#70361)
* updates apm integration job to work with service maps

* rename apm job in setup_module test

* modifies detector description

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-07-02 11:35:40 -04:00
James Gowdy eafd2af6aa
[ML] Adding capabilities checks to shared functions (#70069)
* [ML] Adding capabilities checks to shared functions

* small refactor

* disabling capabilities checks for functions called by SIEM alerting

* testing git

* removing comment

* using constant for ml app id

* tiny type clean up

* removing check in ml_capabilities

* fixing types

* removing capabilities checks from ml_capabilities endpoint

* updating types

* better error handling

* improving capabilities check

* adding custom errors

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-07-01 17:45:36 +01:00
Pierre Gayvallet fe1c508d8d
Move and rename legacy elasticsearch client (#69797)
* move last snapshot to inline

* move legacy files to legacy subfolder

* move request types out of legacy

* export Headers from http instead of elasticsearch

* renaming - first pass

* renaming - second pass

* fix core mocks

* adapt new calls

* update generated doc

* fix IT test mocks

* fix new usages
2020-06-29 17:43:31 +02:00
Ahmad Bamieh 266f853b0b
[Telemetry] Collector Schema (#64942)
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-06-27 02:52:26 +03:00
Pete Harverson b270321ff3
[ML] Fixes anomaly chart and validation for one week bucket span (#69671)
* [ML] Fixes anomaly chart and validation for one week bucket span

* [ML] Fix interval Jest tests
2020-06-24 09:20:38 +01:00
James Gowdy 7a60f18ef9
[ML] Model snapshot management (#68182)
* [ML] Model snapshot management

* more updates

* adding calendar range

* updating layout

* multiple calendars

* moving calendar creator

* fixing chart issues

* fixing chart issues

* improving calendar rendering

* adding capabilities checks

* code clean up

* fixing end time argument type

* fix translations

* code clean up

* comments based on review

* changes based on review

* fixing include

* adding useMemo to theme function
2020-06-16 14:03:17 +01:00
Alexey Antonov c98845c7c7
Update vega version (#68639)
* update vega version

* a11y skip test

* a11y skip test attempt 2

* adding back all a11y tests

* fix failed karma test

* remove extra  0BSD license

* coalesce yarn.lock versions a little

* update kbn/pm dist

* fix CI

* fix Vega View for ML

Co-authored-by: Michail Yasonik <michail.yasonik@elastic.co>
Co-authored-by: spalger <spalger@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-06-16 12:35:34 +03:00
James Gowdy b540ada036
[ML] Allow editing of model_snapshot_retention_days (#69003)
* [ML] Allow editing of model_snapshot_retention_days

* removing commented code

* flooring number input

* adding daily_model_snapshot_retention_after_days

* updating default values

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-06-15 21:45:51 +01:00
Oliver Gupte cac9018f81
[APM] Service maps - Fix missing ML status for services with jobs but no anomalies (#68486)
* Closes #68485 by:
  - use the ML plugin to query for all APM jobs
  - inspect the ml job groups to find all jobs related to a particular service
  - use the mlAnomalySearch client to get ml job buckets with the max anomaly score
  - query for the model_plot buckets to obtain actual/median values for the ML description
  - return the relevant ML job with the max anomaly score for a service
  - indicate to the user that no anomalies were found for a service with an ml job

* - Use the anomalyDetectorsProvider jobs API rather than the search endpoint directly
- Defines a specific return type for the ml jobs api
- Update the empty anomaly data message

* Code and types cleanup

* Return to using record result type on anomaly queries. These are the
same values used in the anomaly explorer and it includes actual &
typical values which greatly improve performance of the previous query.

* - If anomaly data is missing show a gray border around node

* - moved AnomalyDetection out of service map Contents into own component

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-06-09 20:58:46 -07:00
Robert Oskamp 004df98c87
[ML] Sample data modules - use event.dataset instead of index name (#68538)
This PR adds event.dataset to the Kibana ecommerce sample data and makes the ML modules for ecommerce and weblog sample data use event.dataset to recognize the modules and also as the corresponding datafeed query instead of looking for the index name.
2020-06-09 11:30:15 +02:00
Melissa Alvarez edc4d58e12
[ML] DF Analytics: Creation wizard part 1 (#67564)
* create newJob route and start of wizard

* wip: create configStep component

* finish configStep form and details

* wip: create andvanced step components

* create details step component

* createStep component

* ensure advanced options are correct for each job type

* add validation to each step

* use custom table for excludes

* move customSelectionTable to shared components

* form validation for advanced fields

* wip: source index selection modal

* add source index preview

* update details

* ensure advanced parameters added to config on creation

* can create job from savedSearch. can set source query in ui

* validate source object has supported fields

* eslint updates

* update tests. comment out clone action for now

* add create button to advanced editor

* remove deprecated test helper functions

* fix translation errors

* update help text. read only once job created.

* fix functional tests

* add nextStepNav to df service for tests

* fix excludes table page jump and hyperParameter not showing in details

* fix checkbox width for custom table
2020-06-04 13:39:43 -04:00
James Gowdy 0897418795
[ML] Adding per_partition_categorization to job interface and schema (#67953)
* [ML] Adding per_partition_categorization to job interface and schema

* adding new fields to Category interface
2020-06-03 15:23:54 +01:00
Quynh Nguyen f31330a01b
[ML] Add ability to delete target index & index pattern when deleting DFA job (#66934)
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-06-01 10:53:33 -05:00
Mikhail Shustov 53b95424fe
Deprecate es API exposed from setup contract (#67596)
* move elasticsearch client under legacy namespace

* update mocks and tests

* update platform code

* update legacy code

* update plugins using elasticsearch setup API

* update request handler context

* update docs

* rename remaining places

* address comments

* fix merge conflict error
2020-06-01 16:16:44 +02:00
James Gowdy cae6b9e35a
[ML] Using mlClient callAsInternalUser in job validation (#67609) 2020-05-28 18:04:16 +01:00
James Gowdy 321a10d1df
[ML] Renaming mlSearch to mlAnomalySearch (#67602) 2020-05-28 18:02:50 +01:00
James Gowdy dcca06bad2
[ML] Update job service id schema (#67462) 2020-05-28 11:50:56 +01:00
Mikhail Shustov 4040c3090b
Mark elasticsearch client exposed via request context as deprecated (#67319)
* add legacy prefix for es client exposed via request handler context

* update src/plugins

* update core mocks and tests

* update test plugins

* update xpack plugins

* include x-pack/mocks.ts

* update after master merge

* update docs

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-05-27 19:00:16 +02:00
restrry bf04235dae apply prettier styles 2020-05-22 09:08:58 +02:00
Pete Harverson 3e6b3ae74d
[ML] Fixes capabilities check for evaluate analytics endpoint (#66983) 2020-05-19 15:46:18 +01:00
Pete Harverson 1c61ca2c6d
[ML] Enhances api docs for modules endpoints (#66738)
* [ML] Enhances api docs for modules endpoints

* [ML] Edits to modules schema following review

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-05-18 18:13:43 +01:00
Dima Arnautov f7a89601c7
[ML] Custom template for apiDoc markdown (#66567)
* [ML] custom template for apiDoc markdown

* [ML] update apiDoc description
2020-05-15 11:41:29 +02:00
Dima Arnautov d0b9840041
[ML] API integration tests for job validation (#66265)
* [ML] TS refactoring

* [ML] fix page overflow

* [ML] tests

* [ML] fix typing issues and unit tests

* [ML] remove string conversion

* [ML] indexOf checks

* [ML] fix tooltip overflow

* [ML] fix i18n

* [ML] fix unit tests

* [ML] use MlJobAggregation

* [ML] use enums

* Revert "[ML] fix tooltip overflow"

This reverts commit 103c36bc

* Revert "[ML] fix page overflow"

This reverts commit 3c869228
2020-05-13 18:52:49 +02:00
Joe Reuter b7e87c2c8b
Switch to core application service (#63443) 2020-05-13 09:19:24 +02:00
Dima Arnautov d1e4d37c46
[ML] Cardinality validation API integration tests (#65971)
* [ML] refactor validate_cardinality to TS

* [ML] cardinality api integration tests

* [ML] resolve PR comments, validateJobObject as TS guard
2020-05-11 15:57:12 +02:00
Walter Rafelsberger e148eb4437
[ML] Anomaly Detection: Fix test to reflect model memory limit change. (#65967)
Adapt jest test mocks to consider update introduced in #65652.
2020-05-11 11:41:01 +02:00
Quynh Nguyen d3b155f843
[ML] Add job timing stats to anomaly jobs (#65696)
* [ML] Add anomaly job timing stats to Counts & JSON

* [ML] Remove roundTo3DecimalPlace and clean up

* [ML] Fix format_values to round decimals for time values

* [ML] Remove timing_stats and forecast_stats from cloneJob

* [ML] Remove timing_stats & forecasts in job_service instead of utils
2020-05-08 12:06:59 -05:00
Brian Seeders d40387161e
Skipping failing tests. #65867 #65866 #65865 2020-05-08 11:35:17 -04:00
James Gowdy 75e12db4ba
[ML] Show warning when the model memory limit is higher than the memory available in the ML node (#65652)
* [ML] Show warning when the model memory limit is higher than the memory available in the ML node

* reverting UI check

* removing from UI job validator

* adding cap to estimate mml

* adding mml value to message

* fixing translations

* updating translations

* fixing translation ids
2020-05-08 13:13:17 +01:00
Dima Arnautov 46fc73a213
[ML] add missing single quotes for index ids in modules custom urls (#65707) 2020-05-07 19:43:53 +02:00
Walter Rafelsberger 6a6b3edd7f
[ML] Migrate server side Mocha tests to Jest. (#65651)
Migrates job validation related server side tests from Mocha to Jest.
2020-05-07 15:58:16 +02:00
James Gowdy 55e4c7f9a7
[ML] Consolidating shared types and util functions (#65247)
* [ML] Consolidating shared types and util functions

* including formatter

* adding missing includes

* removing unused export

* ignoring numeral type error

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-05-07 12:23:42 +01:00
James Gowdy 8944dbaac4
[ML] Adding daily_model_snapshot_retention_after_days to types and schemas (#65417) 2020-05-06 10:22:40 +01:00
James Gowdy 943f8b9df1
[ML] Fix packetbeat module query (#65241) 2020-05-05 20:44:52 +01:00
Lisa Cawley 415d33b756
[ML] Edits create anomaly job intro text (#64607)
* Edits create anomaly job intro text

* [DOCS] More edits

* [DOCS] Removes obsolete translations

* [DOCS] Addresses feedback

* [DOCS] Update create job breadcrumb

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-05-05 11:38:20 -07:00
James Gowdy 33b2b5c92c
[ML] Enabling mml estimation in data recognizer module setup (#64900)
* [ML] Enabling mml estimation in data recognizer module setup

* small refactor

* adding functional tests

* increasing uptime test timeout

* tiny refactor

* checking for default setting

* testng flakey uptime test

* catching erros in mml estimation

* lowering timeout

* ensuing data is present for ML tests

* adding await

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-05-04 19:30:36 +01:00
Felix Stürmer 39e31d6123
[Logs UI] Add dataset filter to ML module setup screen (#64470)
This adds the ability to filter the datasets to be processed by the ML jobs on the setup screen.
2020-05-04 13:29:28 +02:00
James Gowdy 497c5da763
[ML] Moving get filters capability to admin (#64879)
* [ML] Moving get filters capability to admin

* updating test
2020-04-30 14:07:32 +01:00
James Gowdy d8c15f5ad3
[ML] Adding endpoint capability checks (#64662)
* [ML] Adding endpoint capability checks

* adding missing capability checks

* fixing test

* removing commented code

* fixing functional test

* fixing functional tests

* changes based on review

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-04-29 18:25:48 +01:00
James Gowdy b5fb78bfce
[ML] Fix new job wizard with multiple indices (#64567)
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-04-28 20:45:57 +01:00
James Gowdy f9c81a30cb
[ML] Add kibana setting for file data visualizer max file size (#64427)
* [ML] Add kibana setting for file data visualizers max file size

* adding failsafe for setting

* fixing id

* [DOCS] Updates Data Visualizer setting

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: lcawl <lcawley@elastic.co>
2020-04-28 09:22:46 +01:00
James Gowdy b3c7002799
[ML] Moving to kibana capabilities (#64057)
* [ML] Moving to kibana capabilities

* fixing types

* renaming privilges

* renaming privileges to capabilities

* renaming resolvers

* correcting admin capabilities

* fixing includes

* removing any types

* renaming type

* readding spaces

* adding capabilities switcher

* updating comment

* removing unnecessary failing tests

* adding error to log
2020-04-24 12:08:19 +01:00