Commit graph

147 commits

Author SHA1 Message Date
James Gowdy
7a60f18ef9
[ML] Model snapshot management (#68182)
* [ML] Model snapshot management

* more updates

* adding calendar range

* updating layout

* multiple calendars

* moving calendar creator

* fixing chart issues

* fixing chart issues

* improving calendar rendering

* adding capabilities checks

* code clean up

* fixing end time argument type

* fix translations

* code clean up

* comments based on review

* changes based on review

* fixing include

* adding useMemo to theme function
2020-06-16 14:03:17 +01:00
Alexey Antonov
c98845c7c7
Update vega version (#68639)
* update vega version

* a11y skip test

* a11y skip test attempt 2

* adding back all a11y tests

* fix failed karma test

* remove extra  0BSD license

* coalesce yarn.lock versions a little

* update kbn/pm dist

* fix CI

* fix Vega View for ML

Co-authored-by: Michail Yasonik <michail.yasonik@elastic.co>
Co-authored-by: spalger <spalger@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-06-16 12:35:34 +03:00
Robert Oskamp
004df98c87
[ML] Sample data modules - use event.dataset instead of index name (#68538)
This PR adds event.dataset to the Kibana ecommerce sample data and makes the ML modules for ecommerce and weblog sample data use event.dataset to recognize the modules and also as the corresponding datafeed query instead of looking for the index name.
2020-06-09 11:30:15 +02:00
Quynh Nguyen
f31330a01b
[ML] Add ability to delete target index & index pattern when deleting DFA job (#66934)
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-06-01 10:53:33 -05:00
restrry
bf04235dae apply prettier styles 2020-05-22 09:08:58 +02:00
Dima Arnautov
d0b9840041
[ML] API integration tests for job validation (#66265)
* [ML] TS refactoring

* [ML] fix page overflow

* [ML] tests

* [ML] fix typing issues and unit tests

* [ML] remove string conversion

* [ML] indexOf checks

* [ML] fix tooltip overflow

* [ML] fix i18n

* [ML] fix unit tests

* [ML] use MlJobAggregation

* [ML] use enums

* Revert "[ML] fix tooltip overflow"

This reverts commit 103c36bc

* Revert "[ML] fix page overflow"

This reverts commit 3c869228
2020-05-13 18:52:49 +02:00
Joe Reuter
b7e87c2c8b
Switch to core application service (#63443) 2020-05-13 09:19:24 +02:00
Dima Arnautov
d1e4d37c46
[ML] Cardinality validation API integration tests (#65971)
* [ML] refactor validate_cardinality to TS

* [ML] cardinality api integration tests

* [ML] resolve PR comments, validateJobObject as TS guard
2020-05-11 15:57:12 +02:00
Walter Rafelsberger
e148eb4437
[ML] Anomaly Detection: Fix test to reflect model memory limit change. (#65967)
Adapt jest test mocks to consider update introduced in #65652.
2020-05-11 11:41:01 +02:00
Quynh Nguyen
d3b155f843
[ML] Add job timing stats to anomaly jobs (#65696)
* [ML] Add anomaly job timing stats to Counts & JSON

* [ML] Remove roundTo3DecimalPlace and clean up

* [ML] Fix format_values to round decimals for time values

* [ML] Remove timing_stats and forecast_stats from cloneJob

* [ML] Remove timing_stats & forecasts in job_service instead of utils
2020-05-08 12:06:59 -05:00
Brian Seeders
d40387161e
Skipping failing tests. #65867 #65866 #65865 2020-05-08 11:35:17 -04:00
James Gowdy
75e12db4ba
[ML] Show warning when the model memory limit is higher than the memory available in the ML node (#65652)
* [ML] Show warning when the model memory limit is higher than the memory available in the ML node

* reverting UI check

* removing from UI job validator

* adding cap to estimate mml

* adding mml value to message

* fixing translations

* updating translations

* fixing translation ids
2020-05-08 13:13:17 +01:00
Dima Arnautov
46fc73a213
[ML] add missing single quotes for index ids in modules custom urls (#65707) 2020-05-07 19:43:53 +02:00
Walter Rafelsberger
6a6b3edd7f
[ML] Migrate server side Mocha tests to Jest. (#65651)
Migrates job validation related server side tests from Mocha to Jest.
2020-05-07 15:58:16 +02:00
James Gowdy
943f8b9df1
[ML] Fix packetbeat module query (#65241) 2020-05-05 20:44:52 +01:00
Lisa Cawley
415d33b756
[ML] Edits create anomaly job intro text (#64607)
* Edits create anomaly job intro text

* [DOCS] More edits

* [DOCS] Removes obsolete translations

* [DOCS] Addresses feedback

* [DOCS] Update create job breadcrumb

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-05-05 11:38:20 -07:00
James Gowdy
33b2b5c92c
[ML] Enabling mml estimation in data recognizer module setup (#64900)
* [ML] Enabling mml estimation in data recognizer module setup

* small refactor

* adding functional tests

* increasing uptime test timeout

* tiny refactor

* checking for default setting

* testng flakey uptime test

* catching erros in mml estimation

* lowering timeout

* ensuing data is present for ML tests

* adding await

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-05-04 19:30:36 +01:00
Felix Stürmer
39e31d6123
[Logs UI] Add dataset filter to ML module setup screen (#64470)
This adds the ability to filter the datasets to be processed by the ML jobs on the setup screen.
2020-05-04 13:29:28 +02:00
James Gowdy
b5fb78bfce
[ML] Fix new job wizard with multiple indices (#64567)
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-04-28 20:45:57 +01:00
Pete Harverson
09c2727d78
[ML] Adds API integration tests for data viz and fields endpoints (#64165)
* [ML] Adds API integration tests for data viz and fields endpoints

* [ML] Fix review comments and errors from settings endpoints

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-04-23 18:06:33 +01:00
James Gowdy
959a0e5d3d
[ML] Listing global calendars on the job management page (#63124)
* [ML] Listing global calendars on the job management page

* tiny refactor

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-04-14 18:57:26 +01:00
James Gowdy
64f27ca34e
[ML] Show better file structure finder explanations (#62316)
* [ML] Show better file structure finder explanations

* more typescript changes

* changing function format

* fixing some types

* fixing translation id

* fix boom error reporting

* changes based on review

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-04-07 08:47:39 +01:00
Dima Arnautov
cca0502ed3
[ML] call job validation endpoint with complete payload (#62331) 2020-04-03 10:22:49 +02:00
Dima Arnautov
aeba1b7bb0
[ML] Handle Empty Partition Field Values in Single Metric Viewer (#61649)
* [ML] WIP support empty partition fields values

* [ML] support empty field in anomaly table

* [ML] remove comments

* [ML] fix context chart

* [ML] rename empty field label, render as italic

* [ML] rename empty field label

* [ML] fix focus chart

* [ML] add time range capping for fields_service.ts

* [ML] empty string labels in anomaly explorer
2020-03-31 16:29:00 +02:00
Robert Oskamp
f5e94c4ca3
[ML] Move get_module Jest tests to functional API test (#61613)
This PR removes the flaky listModules and getModule Jest tests and adds them as functional API tests.
2020-03-27 16:12:15 +01:00
Robert Oskamp
4f863c627a
[ML] Add uptime_heartbeat module to recognizer test (#61235)
This PR adds the new uptime_heartbeat module to the recognizer list modules test.
2020-03-25 13:17:32 +01:00
Robert Oskamp
db0a4e8ca0
[ML] Jest tests - re-activate data recognizer tests (#60525)
This PR re-activates the data recognizer Jest tests.
2020-03-25 09:57:08 +01:00
Shahzad
d31e5f524f
[Uptime] Ml detection of duration anomalies (#59785)
* add flyout

* add state

* update state

* ad job

* update

* updat

* add ml analyze button

* update api

* use differential colors for duration chart

* remove duration chart gql

* update type

* type fix

* fix tyoe

* update translation

* update test

* update conflicts

* update anomaly record

* chart

* added annotations

* update error handling

* update

* update types

* fixed types

* fix types

* update types

* update

* update

* remove unnecessary change

* remove unnecessary change

* fix type

* update

* save

* update pr

* update tets

* update job deletion

* update

* update tets

* upadte tests

* fix types

* update title text

* update types

* fixed tests

* update tests and types

* updated types

* fix PR feedback

* unit test

* update more types

* update test and manage  job

* resolve conflicts

* types

* remove unnecessary change

* revert ml code

* revert ml code

* fixed formatting issues pointed by pr feedback
2020-03-24 12:34:43 +01:00
Dima Arnautov
66b5efd084
[ML] Module setup with dynamic model memory estimation (#60656)
* [ML] add estimateModelMemory to the setup endpoint

* [ML] wip caching cardinality checks

* [ML] refactor

* [ML] fix a fallback time range

* [ML] fix typing issue

* [ML] fields_aggs_cache.ts as part of fields_service

* [ML] fix types, add comments

* [ML] check for MML overrides

* [ML] disable estimateModelMemory

* [ML] fix typing

* [ML] check for empty max mml

* [ML] refactor, update types, fix jobsForModelMemoryEstimation

* [ML] fix override lookup

* [ML] resolve nit comments

* [ML] init jobsForModelMemoryEstimation
2020-03-24 08:46:11 +01:00
James Gowdy
f18c571ed6
[ML] Listing all categorization wizard checks (#60502)
* [ML] Listing all categorization wizard checks

* fixing translation

* changes based on review

* moving check

* adding real values to messages

* reordering checks enum

* fixing types

* updating tests

* updating id
2020-03-20 12:07:54 +00:00
Dima Arnautov
7aa4651292
[ML] Use a new ML endpoint to estimate a model memory (#60376)
* [ML] refactor calculate_model_memory_limit route, use estimateModelMemory endpoint

* [ML] refactor validate_model_memory_limit, migrate tests to jest

* [ML] fix typing issue

* [ML] start estimateModelMemory url with /

* [ML] fix typo, filter mlcategory

* [ML] extract getCardinalities function

* [ML] fields_service.ts

* [ML] wip getMaxBucketCardinality

* [ML] refactor and comments

* [ML] fix aggs keys with special characters, fix integration tests

* [ML] use pre-defined job types

* [ML] fallback to 0 in case max bucket cardinality receives null

* [ML] calculateModelMemoryLimit on influencers change

* [ML] fix maxModelMemoryLimit

* [ML] cap aggregation to max 1000 buckets

* [ML] rename intervalDuration
2020-03-19 16:45:40 +01:00
James Gowdy
35302ed273
[ML] Client side cut over (#60100)
* [ML] Client side cut over

* updating paths and commented code

* changes based on review

* disabling telemetry tests

* fixing start job stylesheets

* fixing everything that is broken

* fixing types and ml icon order

* using icon constant
2020-03-13 19:16:41 +00:00
spalger
410019ab5a skip flaky suite (#59541) 2020-03-12 13:17:23 -07:00
James Gowdy
d5c092811b
[ML] Adding shared services to ml setup contract (#59730)
* [ML] Adding shared services to ml setup contract

* adding data recognizer

* typescripting js client

* adding results service

* code clean up

* adding generic ml index search

* making cloud optional
2020-03-12 10:04:40 +00:00
James Gowdy
8a4bb61f05
[ML] Fixes bucket span estimators loading of max_buckets setting (#59639)
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-03-09 14:43:23 +00:00
James Gowdy
92cc04c7df
[ML] Refactoring anomaly detector job types (#59556)
* [ML] Refactoring anomaly detector job types

* removing calendar type

* update validateCardinality

* fixing test

* updating  types in functional tests

* using state constants
2020-03-09 13:51:07 +00:00
Pete Harverson
b104980b88
[ML] Add support for date_nanos time field in anomaly job wizard (#59017)
* [ML] Add support for date_nanos time field in anomaly job wizard

* [ML] Edits following review

* [ML] Add functional test for creating job off date_nanos data
2020-03-05 09:28:31 +00:00
Melissa Alvarez
8c3d71b370
[ML] NP: migrate server (#58680)
* remove obsolete legacy server deps

* licensePreRoutingFactory uses licensing plugin rather than legacy xpack

* move schemas to dir in routes

* use NP license check method for license check

* store license data in plugin for passing to check

* create server plugin files in NP plugin dir

* remove dependency on legacy xpack plugin

* add sample data links first step

* move all server dirs from legacy to np dir

* fix requiredPlugin spaces name and update import routes

* delete unnecessary files and add sample data links

* update license and privilege check tests

* add routeInit types
2020-02-27 21:15:08 -05:00
Court Ewing
7ac8e4d9cc Move x-pack plugins/server/common to x-pack/legacy 2019-06-20 13:34:48 -04:00
James Gowdy
89892ba097
[ML] Fixing server side logger after new platform change (#39154) 2019-06-18 15:18:44 +01:00
Melissa Alvarez
c47a0d5cdf
[ML] New Platform Migration - server (#38360)
* wip: pull out all server dependencies

* create new platform plugin

* shim newPlatform plugin into legacy init

* update server tests for migration refactor

* cleanup interfaces

* Only add ML links for sample data sets if full license from - 38120

* update test and fix typescript errors

* Remove unused types
2019-06-17 16:08:13 -04:00
Pete Harverson
41d74498e8
[ML] Edit job_service jobsExist call to query ML Get Jobs endpoint (#38649) 2019-06-11 15:02:35 +01:00
Pete Harverson
4a3034de49
[ML] Fixes loading of Single Metric Viewer if partition field is text (#37975)
* [ML] Fixes loading of Single Metric Viewer if partition field is text

* [ML] Fix failing test and edit translations for cardinality check

* [ML] Edits to fields_service.js following review
2019-06-05 12:02:03 +01:00
Matthew Kime
964f9e2d41
remove unneeded eslint-disable directives (#37922)
* remove unneeded eslint-disable directives
2019-06-04 07:53:22 -05:00
Melissa Alvarez
ca2a66dd26
return false for each jobId if it does not exist (#37116) 2019-05-28 11:58:43 -04:00
Matthew Kime
e390be2f11
No unused vars cleanup set two (#37051)
* no unused vars cleanup
2019-05-24 14:35:21 -05:00
stevedodson
9463fd8778
[ML] Improve anomalies in Kibana sample data (#36982)
* [ML] Sample ML jobs: Updates to make anomalies clearer

Minor updates to web logs data and ML job
definitions to make anomalous activity more
distinct.

* [ML] Sample ML jobs: Further updates to jobs and data

Adding missing files from previous commit and
changing job naming (removing weblog_* prefix etc.)

* [ML] Resolving PR review comments.

* [ML] Improve anomalies in Kibana sample data

Minor fixes:
- Rename job group to `kibana_sample_data` throughout
- Fix index pattern link in `sample_data_sets.ts`
- Fix log.json.gz count test
2019-05-24 15:32:06 +02:00
James Gowdy
0babbf150d
[ML] Changes interval to fixed_interval in aggregations in created jobs (#36759)
* [ML] Changes interval to fixed_interval in aggregations in created jobs

* updates following review
2019-05-22 09:57:41 +01:00
Pete Harverson
cc7a6ec2c1
[ML] Add links to ML jobs to the Kibana sample dataset view data menu (#36702)
* [ML] Add links to ML jobs to the Kibana sample dataset view data menu

* [ML] Remove redundant TODO comment from check_module.js

* [ML] Edits to sample data ML links following review

* [ML] Edits to comments about use of reject in check_module.js
2019-05-21 15:34:41 +01:00
Camilla
e997ffe0e9
[ML] Kibana Sample Dataset data recognizer modules (#35138)
* Initial job configuration files for Kibana Sample Ecommerce data recognizer

* Initial configuration files for Kibana Sample Data for web logs

* Initial configuration for Kibana Sample Data recognizer for Flights data

* Correcting ID, title and description of Flights sample data recognizer

* Adding Custom URLs to link to Discover from Kibana Sample Dataset ML jobs

* Adding custom URLs to Ecommerce Data Recognizer results

* Adding new custom URL to Kibana Sample Ecommerce data recognizer

* Adding custom URL for Sample Flights dataset dashboard

* Adding custom URLs to sample Web Logs data recogniser config

* Fixing single quotes and url names in custom URLs for flights and ecommerce datasets

* Adding Kibana Sample Data modules to list of modules that need testing

* Renaming Kibana Sample Data recognizer modules, fixing IDs and adding modules to test framework

* Fixing data recognizer titles to sentence case

* Fixing sentence case issues in Kibana eCommerce sample data

* Fixing sentence case in Kibana sample flights data

* Fixing sentence case in Kibana sample weblogs data

* Changing job configuration for Kibana sample flights data

* Making changes to job descriptions

* Changing job configurations and renaming files for Kibana eCommerce sample data recognizer

* Removing mean bytes by file extension from Web Logs sample jobs and adding three new configurations

* Fixing minor issues with Kibana Sample Data recognizer configurations

* Adding jobs to job groups

* Changing job group names to use underscores instead of hyphens

* Adding filtering to Kibana Sample Data dashboards

* Fixing linting issues in JSON configs

* Finetuning data recognizer module descriptions

* Renaming Kibana Sample Data flights jobs

* Removing plural from Kibana Sample Flights dataset description

* Modifying custom URLs for Kibana Flights jobs

* Fixing custom URLs for Kibana Sample eCommerce dataset jobs

* Adding new influencers and fixing Discover custom URLs for Kibana Weblogs sample dataset job

* Fixing space before colon issue in job configurations

* Converting tabs to spaces

* Fixing description of detector in sample dataset eCommerce job
2019-05-21 16:32:57 +02:00