Commit graph

155 commits

Author SHA1 Message Date
Walter Rafelsberger
262e0754ff
[ML] Kibana API endpoint for histogram chart data (#70976)
- Introduces dedicated Kibana API endpoints as part of ML and transform plugin API endpoints and moves the logic to query and transform the required data from client to server.
- Adds support for sampling to retrieve the data for the field histograms. For now this is not configurable by the end user and is hard coded to 5000. This is to have a first iteration of this functionality in for 7.9 and protect users when querying large clusters. The button to enable the histogram charts now includes a tooltip that mentions the sampler.
2020-07-14 13:37:36 +02:00
Michael Hirsch
c5729b87d6
[ML] Adds siem_cloudtrail Module (#71323)
* adds siem_cloudtrail module

* updates logo to logoSecurity

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-07-13 12:35:04 -04:00
James Gowdy
97ad58c529
[ML] Changing shared module setup function parameters (#70589)
* [ML] Changing shared module setup function parameters

* changes based on review

* adding comments for documentation

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-07-03 08:30:17 +01:00
Tiago Costa
6c62c686cf
chore(NA): upgrade to lodash@4 (#69868)
* chore(NA): upgrade oss to lodash4

chore(NA): migrate cli, cli_plugin, cli_keystore,  dev, test_utils and apm src script to lodash4

chore(NA): missing file for cli plugin

chore(NA): add src core

chore(NA): es archiver and fixtures

chore(NA): try to fix functional test failure

chore(NA): migrate src/legacy entirely to lodash4 except src/legacy/core_plugins

chore(NA): move legacy core plugins to lodash4

chore(NA): upgrade optimize to lodash4

chore(NA): upgrade to lodash4 on advanced_settings, charts, console and dashboard

chore(NA): migrate to lodash4 on dev_tools, discover, embeddable, es_ui)shared, expressions, home plugins

chore(NA): upgrade data plugin to lodash4

chore(NA): upgrade usage_collection, ui_actions, tile_map, telemtry, share, saved_objects, saved_objects_management, region_map and navigation to lodash4

chore(NA): missing data upgrades to lodash4

Revert "chore(NA): upgrade usage_collection, ui_actions, tile_map, telemtry, share, saved_objects, saved_objects_management, region_map and navigation to lodash4"

This reverts commit 137055c5fed2fc52bb26547e0bc1ad2e3d4fe309.

Revert "Revert "chore(NA): upgrade usage_collection, ui_actions, tile_map, telemtry, share, saved_objects, saved_objects_management, region_map and navigation to lodash4""

This reverts commit f7e73688782998513d9fb6d7e8f0765e9beb28d1.

Revert "chore(NA): missing data upgrades to lodash4"

This reverts commit 92b85bf947a89bfc70cc4052738a6b2128ffb076.

Revert "chore(NA): upgrade data plugin to lodash4"

This reverts commit 88fdb075ee1e26c4ac979b6681d8a2b002df74c6.

chore(NA): upgrade idx_pattern_mgt, input_control_vis, inspector, kbn_legacy, kbn_react, kbn_usage_collections, kbn_utils, management and maps_legacy to lodash4

chore(NA): map src plugin data to lodash3

chore(NA): missing lodash.clonedeep dep

chore(NA): change packages kbn-config-schema deps

chore(NA): update renovate config

chore(NA): upgrade vis_type plugins to lodash4

chore(NA): move vis_type_vislib to lodash3

chore(NA): update visualizations and visualize to lodash4

chore(NA): remove lodash 3 types from src and move test to lodash4

chore(NA): move home, usage_collection and management to lodash 3

Revert "chore(NA): move home, usage_collection and management to lodash 3"

This reverts commit f86e8585f02d21550746569af54215b076a79a3d.

chore(NA): move kibana_legacy, saved_objects saved_objects_management into lodash3

chore(NA): update x-pack test to mock lodash4

Revert "chore(NA): move kibana_legacy, saved_objects saved_objects_management into lodash3"

This reverts commit 2d10fe450533e1b36db21d99cfae3ce996a244e0.

* chore(NA): move x-pack and packages to lodash 4

* chore(NA): remove mention to lodash from main package.json

* chore(NA): remove helper alias for lodash4 and make it the default lodash

* chore(NA): fix last failing types in the repo

* chore(NA): fix public api

* chore(NA): fix types for agg_row.tsx

* chore(NA): fix increment of optimizer modules in the rollup plugin

* chore(NA): migrate `src/core/public/http/fetch.ts` (#5)

* omit undefined query props

* just remove merge usage

* fix types

* chore(NA): fixes for feedback from apm team

* chore(NA): recover old behaviour on apm LoadingIndeicatorContext.tsx

* chore(NA): fixes for feedback from watson

* Platform lodash4 tweaks (#6)

* chore(NA): fix types and behaviour on src/core/server/elasticsearch/errors.ts

* Canvas fixes for lodash upgrade

* [APM] Adds unit test for APM service maps transform (#7)

* Adds a snapshot unit test for getConnections and rearranges some code to make testing easier

* reverts `ArrayList` back to `String[]` in the painless script within `fetch_service_paths_from_trace_ids.ts`

* chore(NA): update yarn.lock

* chore(NA): remove any and use a real type for alerts task runner

Co-authored-by: Gidi Meir Morris <github@gidi.io>

* chore(NA): used named import for triggers_actions_ui file

* chore(NA): fix eslint

* chore(NA): fix types

* Delete most uptime lodash references.

* Simplify. Clean up types.

* [Uptime] Delete most uptime lodash references (#8)

* Delete most uptime lodash references.

* Simplify. Clean up types.

* chore(NA): add eslint rule to avoid using lodash3

* chore(NA): apply changes on feedback from es-ui team

* fix some types (#9)

* Clean up some expressions types.

* chore(NA): missing ts-expect-error statements

* Upgrade lodash 4 vislib (#11)

* replace lodash 3 with lodash 4 on vislib plugin

* Further changes

* further replacement of lodash3 to 4

* further work on upgrading to lodash 4

* final changes to update lodash

* chore(NA): upgrade data plugin to lodash4

chore(NA): upgrade data plugin public to lodash4

chore(NA): fix typecheck task

chore(NA): fix agg_config with hasIn

chore(NA): assign to assignIn and has to hasIn

chore(NA): upgrade data plugin server to lodash4

chore(NA): new signature for core api

fix(NA): match behaviour between lodash3 and lodash4 for set in search_source

* chore(NA): remove lodash3 completely from the repo

* chore(NA): fix x-pack/test/api_integration/apis/metrics_ui/snapshot.ts missing content

* chore(NA): fix lodash usage on apm

* chore(NA): fix typecheck for maps

* Patch lodash template (#12)

* Applying changes from https://github.com/elastic/kibana/pull/64985

* Using isIterateeCall, because it seems less brittle

* Also patching `lodash/template` and `lodash/fp/template`

* Reorganizing some files...

* Revising comment

* Ends up `_` is a function also... I hate JavaScript

Co-authored-by: Pierre Gayvallet <pierre.gayvallet@gmail.com>
Co-authored-by: Josh Dover <me@joshdover.com>
Co-authored-by: Clint Andrew Hall <clint.hall@elastic.co>
Co-authored-by: Oliver Gupte <ogupte@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Gidi Meir Morris <github@gidi.io>
Co-authored-by: Justin Kambic <justin.kambic@elastic.co>
Co-authored-by: Stratoula Kalafateli <stratoula1@gmail.com>
Co-authored-by: Luke Elmers <luke.elmers@elastic.co>
Co-authored-by: Brandon Kobel <brandon.kobel@gmail.com>
Co-authored-by: kobelb <brandon.kobel@elastic.co>
2020-07-03 01:30:13 +01:00
Michael Hirsch
8a09f247e3
[ML] Updates APM Module to Work with Service Maps (#70361)
* updates apm integration job to work with service maps

* rename apm job in setup_module test

* modifies detector description

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-07-02 11:35:40 -04:00
James Gowdy
eafd2af6aa
[ML] Adding capabilities checks to shared functions (#70069)
* [ML] Adding capabilities checks to shared functions

* small refactor

* disabling capabilities checks for functions called by SIEM alerting

* testing git

* removing comment

* using constant for ml app id

* tiny type clean up

* removing check in ml_capabilities

* fixing types

* removing capabilities checks from ml_capabilities endpoint

* updating types

* better error handling

* improving capabilities check

* adding custom errors

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-07-01 17:45:36 +01:00
Pierre Gayvallet
fe1c508d8d
Move and rename legacy elasticsearch client (#69797)
* move last snapshot to inline

* move legacy files to legacy subfolder

* move request types out of legacy

* export Headers from http instead of elasticsearch

* renaming - first pass

* renaming - second pass

* fix core mocks

* adapt new calls

* update generated doc

* fix IT test mocks

* fix new usages
2020-06-29 17:43:31 +02:00
Pete Harverson
b270321ff3
[ML] Fixes anomaly chart and validation for one week bucket span (#69671)
* [ML] Fixes anomaly chart and validation for one week bucket span

* [ML] Fix interval Jest tests
2020-06-24 09:20:38 +01:00
James Gowdy
7a60f18ef9
[ML] Model snapshot management (#68182)
* [ML] Model snapshot management

* more updates

* adding calendar range

* updating layout

* multiple calendars

* moving calendar creator

* fixing chart issues

* fixing chart issues

* improving calendar rendering

* adding capabilities checks

* code clean up

* fixing end time argument type

* fix translations

* code clean up

* comments based on review

* changes based on review

* fixing include

* adding useMemo to theme function
2020-06-16 14:03:17 +01:00
Alexey Antonov
c98845c7c7
Update vega version (#68639)
* update vega version

* a11y skip test

* a11y skip test attempt 2

* adding back all a11y tests

* fix failed karma test

* remove extra  0BSD license

* coalesce yarn.lock versions a little

* update kbn/pm dist

* fix CI

* fix Vega View for ML

Co-authored-by: Michail Yasonik <michail.yasonik@elastic.co>
Co-authored-by: spalger <spalger@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-06-16 12:35:34 +03:00
Robert Oskamp
004df98c87
[ML] Sample data modules - use event.dataset instead of index name (#68538)
This PR adds event.dataset to the Kibana ecommerce sample data and makes the ML modules for ecommerce and weblog sample data use event.dataset to recognize the modules and also as the corresponding datafeed query instead of looking for the index name.
2020-06-09 11:30:15 +02:00
Quynh Nguyen
f31330a01b
[ML] Add ability to delete target index & index pattern when deleting DFA job (#66934)
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-06-01 10:53:33 -05:00
restrry
bf04235dae apply prettier styles 2020-05-22 09:08:58 +02:00
Dima Arnautov
d0b9840041
[ML] API integration tests for job validation (#66265)
* [ML] TS refactoring

* [ML] fix page overflow

* [ML] tests

* [ML] fix typing issues and unit tests

* [ML] remove string conversion

* [ML] indexOf checks

* [ML] fix tooltip overflow

* [ML] fix i18n

* [ML] fix unit tests

* [ML] use MlJobAggregation

* [ML] use enums

* Revert "[ML] fix tooltip overflow"

This reverts commit 103c36bc

* Revert "[ML] fix page overflow"

This reverts commit 3c869228
2020-05-13 18:52:49 +02:00
Joe Reuter
b7e87c2c8b
Switch to core application service (#63443) 2020-05-13 09:19:24 +02:00
Dima Arnautov
d1e4d37c46
[ML] Cardinality validation API integration tests (#65971)
* [ML] refactor validate_cardinality to TS

* [ML] cardinality api integration tests

* [ML] resolve PR comments, validateJobObject as TS guard
2020-05-11 15:57:12 +02:00
Walter Rafelsberger
e148eb4437
[ML] Anomaly Detection: Fix test to reflect model memory limit change. (#65967)
Adapt jest test mocks to consider update introduced in #65652.
2020-05-11 11:41:01 +02:00
Quynh Nguyen
d3b155f843
[ML] Add job timing stats to anomaly jobs (#65696)
* [ML] Add anomaly job timing stats to Counts & JSON

* [ML] Remove roundTo3DecimalPlace and clean up

* [ML] Fix format_values to round decimals for time values

* [ML] Remove timing_stats and forecast_stats from cloneJob

* [ML] Remove timing_stats & forecasts in job_service instead of utils
2020-05-08 12:06:59 -05:00
Brian Seeders
d40387161e
Skipping failing tests. #65867 #65866 #65865 2020-05-08 11:35:17 -04:00
James Gowdy
75e12db4ba
[ML] Show warning when the model memory limit is higher than the memory available in the ML node (#65652)
* [ML] Show warning when the model memory limit is higher than the memory available in the ML node

* reverting UI check

* removing from UI job validator

* adding cap to estimate mml

* adding mml value to message

* fixing translations

* updating translations

* fixing translation ids
2020-05-08 13:13:17 +01:00
Dima Arnautov
46fc73a213
[ML] add missing single quotes for index ids in modules custom urls (#65707) 2020-05-07 19:43:53 +02:00
Walter Rafelsberger
6a6b3edd7f
[ML] Migrate server side Mocha tests to Jest. (#65651)
Migrates job validation related server side tests from Mocha to Jest.
2020-05-07 15:58:16 +02:00
James Gowdy
943f8b9df1
[ML] Fix packetbeat module query (#65241) 2020-05-05 20:44:52 +01:00
Lisa Cawley
415d33b756
[ML] Edits create anomaly job intro text (#64607)
* Edits create anomaly job intro text

* [DOCS] More edits

* [DOCS] Removes obsolete translations

* [DOCS] Addresses feedback

* [DOCS] Update create job breadcrumb

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-05-05 11:38:20 -07:00
James Gowdy
33b2b5c92c
[ML] Enabling mml estimation in data recognizer module setup (#64900)
* [ML] Enabling mml estimation in data recognizer module setup

* small refactor

* adding functional tests

* increasing uptime test timeout

* tiny refactor

* checking for default setting

* testng flakey uptime test

* catching erros in mml estimation

* lowering timeout

* ensuing data is present for ML tests

* adding await

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-05-04 19:30:36 +01:00
Felix Stürmer
39e31d6123
[Logs UI] Add dataset filter to ML module setup screen (#64470)
This adds the ability to filter the datasets to be processed by the ML jobs on the setup screen.
2020-05-04 13:29:28 +02:00
James Gowdy
b5fb78bfce
[ML] Fix new job wizard with multiple indices (#64567)
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-04-28 20:45:57 +01:00
Pete Harverson
09c2727d78
[ML] Adds API integration tests for data viz and fields endpoints (#64165)
* [ML] Adds API integration tests for data viz and fields endpoints

* [ML] Fix review comments and errors from settings endpoints

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-04-23 18:06:33 +01:00
James Gowdy
959a0e5d3d
[ML] Listing global calendars on the job management page (#63124)
* [ML] Listing global calendars on the job management page

* tiny refactor

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-04-14 18:57:26 +01:00
James Gowdy
64f27ca34e
[ML] Show better file structure finder explanations (#62316)
* [ML] Show better file structure finder explanations

* more typescript changes

* changing function format

* fixing some types

* fixing translation id

* fix boom error reporting

* changes based on review

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-04-07 08:47:39 +01:00
Dima Arnautov
cca0502ed3
[ML] call job validation endpoint with complete payload (#62331) 2020-04-03 10:22:49 +02:00
Dima Arnautov
aeba1b7bb0
[ML] Handle Empty Partition Field Values in Single Metric Viewer (#61649)
* [ML] WIP support empty partition fields values

* [ML] support empty field in anomaly table

* [ML] remove comments

* [ML] fix context chart

* [ML] rename empty field label, render as italic

* [ML] rename empty field label

* [ML] fix focus chart

* [ML] add time range capping for fields_service.ts

* [ML] empty string labels in anomaly explorer
2020-03-31 16:29:00 +02:00
Robert Oskamp
f5e94c4ca3
[ML] Move get_module Jest tests to functional API test (#61613)
This PR removes the flaky listModules and getModule Jest tests and adds them as functional API tests.
2020-03-27 16:12:15 +01:00
Robert Oskamp
4f863c627a
[ML] Add uptime_heartbeat module to recognizer test (#61235)
This PR adds the new uptime_heartbeat module to the recognizer list modules test.
2020-03-25 13:17:32 +01:00
Robert Oskamp
db0a4e8ca0
[ML] Jest tests - re-activate data recognizer tests (#60525)
This PR re-activates the data recognizer Jest tests.
2020-03-25 09:57:08 +01:00
Shahzad
d31e5f524f
[Uptime] Ml detection of duration anomalies (#59785)
* add flyout

* add state

* update state

* ad job

* update

* updat

* add ml analyze button

* update api

* use differential colors for duration chart

* remove duration chart gql

* update type

* type fix

* fix tyoe

* update translation

* update test

* update conflicts

* update anomaly record

* chart

* added annotations

* update error handling

* update

* update types

* fixed types

* fix types

* update types

* update

* update

* remove unnecessary change

* remove unnecessary change

* fix type

* update

* save

* update pr

* update tets

* update job deletion

* update

* update tets

* upadte tests

* fix types

* update title text

* update types

* fixed tests

* update tests and types

* updated types

* fix PR feedback

* unit test

* update more types

* update test and manage  job

* resolve conflicts

* types

* remove unnecessary change

* revert ml code

* revert ml code

* fixed formatting issues pointed by pr feedback
2020-03-24 12:34:43 +01:00
Dima Arnautov
66b5efd084
[ML] Module setup with dynamic model memory estimation (#60656)
* [ML] add estimateModelMemory to the setup endpoint

* [ML] wip caching cardinality checks

* [ML] refactor

* [ML] fix a fallback time range

* [ML] fix typing issue

* [ML] fields_aggs_cache.ts as part of fields_service

* [ML] fix types, add comments

* [ML] check for MML overrides

* [ML] disable estimateModelMemory

* [ML] fix typing

* [ML] check for empty max mml

* [ML] refactor, update types, fix jobsForModelMemoryEstimation

* [ML] fix override lookup

* [ML] resolve nit comments

* [ML] init jobsForModelMemoryEstimation
2020-03-24 08:46:11 +01:00
James Gowdy
f18c571ed6
[ML] Listing all categorization wizard checks (#60502)
* [ML] Listing all categorization wizard checks

* fixing translation

* changes based on review

* moving check

* adding real values to messages

* reordering checks enum

* fixing types

* updating tests

* updating id
2020-03-20 12:07:54 +00:00
Dima Arnautov
7aa4651292
[ML] Use a new ML endpoint to estimate a model memory (#60376)
* [ML] refactor calculate_model_memory_limit route, use estimateModelMemory endpoint

* [ML] refactor validate_model_memory_limit, migrate tests to jest

* [ML] fix typing issue

* [ML] start estimateModelMemory url with /

* [ML] fix typo, filter mlcategory

* [ML] extract getCardinalities function

* [ML] fields_service.ts

* [ML] wip getMaxBucketCardinality

* [ML] refactor and comments

* [ML] fix aggs keys with special characters, fix integration tests

* [ML] use pre-defined job types

* [ML] fallback to 0 in case max bucket cardinality receives null

* [ML] calculateModelMemoryLimit on influencers change

* [ML] fix maxModelMemoryLimit

* [ML] cap aggregation to max 1000 buckets

* [ML] rename intervalDuration
2020-03-19 16:45:40 +01:00
James Gowdy
35302ed273
[ML] Client side cut over (#60100)
* [ML] Client side cut over

* updating paths and commented code

* changes based on review

* disabling telemetry tests

* fixing start job stylesheets

* fixing everything that is broken

* fixing types and ml icon order

* using icon constant
2020-03-13 19:16:41 +00:00
spalger
410019ab5a skip flaky suite (#59541) 2020-03-12 13:17:23 -07:00
James Gowdy
d5c092811b
[ML] Adding shared services to ml setup contract (#59730)
* [ML] Adding shared services to ml setup contract

* adding data recognizer

* typescripting js client

* adding results service

* code clean up

* adding generic ml index search

* making cloud optional
2020-03-12 10:04:40 +00:00
James Gowdy
8a4bb61f05
[ML] Fixes bucket span estimators loading of max_buckets setting (#59639)
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-03-09 14:43:23 +00:00
James Gowdy
92cc04c7df
[ML] Refactoring anomaly detector job types (#59556)
* [ML] Refactoring anomaly detector job types

* removing calendar type

* update validateCardinality

* fixing test

* updating  types in functional tests

* using state constants
2020-03-09 13:51:07 +00:00
Pete Harverson
b104980b88
[ML] Add support for date_nanos time field in anomaly job wizard (#59017)
* [ML] Add support for date_nanos time field in anomaly job wizard

* [ML] Edits following review

* [ML] Add functional test for creating job off date_nanos data
2020-03-05 09:28:31 +00:00
Melissa Alvarez
8c3d71b370
[ML] NP: migrate server (#58680)
* remove obsolete legacy server deps

* licensePreRoutingFactory uses licensing plugin rather than legacy xpack

* move schemas to dir in routes

* use NP license check method for license check

* store license data in plugin for passing to check

* create server plugin files in NP plugin dir

* remove dependency on legacy xpack plugin

* add sample data links first step

* move all server dirs from legacy to np dir

* fix requiredPlugin spaces name and update import routes

* delete unnecessary files and add sample data links

* update license and privilege check tests

* add routeInit types
2020-02-27 21:15:08 -05:00
Court Ewing
7ac8e4d9cc Move x-pack plugins/server/common to x-pack/legacy 2019-06-20 13:34:48 -04:00
James Gowdy
89892ba097
[ML] Fixing server side logger after new platform change (#39154) 2019-06-18 15:18:44 +01:00
Melissa Alvarez
c47a0d5cdf
[ML] New Platform Migration - server (#38360)
* wip: pull out all server dependencies

* create new platform plugin

* shim newPlatform plugin into legacy init

* update server tests for migration refactor

* cleanup interfaces

* Only add ML links for sample data sets if full license from - 38120

* update test and fix typescript errors

* Remove unused types
2019-06-17 16:08:13 -04:00
Pete Harverson
41d74498e8
[ML] Edit job_service jobsExist call to query ML Get Jobs endpoint (#38649) 2019-06-11 15:02:35 +01:00