Commit graph

205 commits

Author SHA1 Message Date
Ahmad Bamieh 266f853b0b
[Telemetry] Collector Schema (#64942)
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-06-27 02:52:26 +03:00
Pete Harverson b270321ff3
[ML] Fixes anomaly chart and validation for one week bucket span (#69671)
* [ML] Fixes anomaly chart and validation for one week bucket span

* [ML] Fix interval Jest tests
2020-06-24 09:20:38 +01:00
James Gowdy 7a60f18ef9
[ML] Model snapshot management (#68182)
* [ML] Model snapshot management

* more updates

* adding calendar range

* updating layout

* multiple calendars

* moving calendar creator

* fixing chart issues

* fixing chart issues

* improving calendar rendering

* adding capabilities checks

* code clean up

* fixing end time argument type

* fix translations

* code clean up

* comments based on review

* changes based on review

* fixing include

* adding useMemo to theme function
2020-06-16 14:03:17 +01:00
Alexey Antonov c98845c7c7
Update vega version (#68639)
* update vega version

* a11y skip test

* a11y skip test attempt 2

* adding back all a11y tests

* fix failed karma test

* remove extra  0BSD license

* coalesce yarn.lock versions a little

* update kbn/pm dist

* fix CI

* fix Vega View for ML

Co-authored-by: Michail Yasonik <michail.yasonik@elastic.co>
Co-authored-by: spalger <spalger@users.noreply.github.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-06-16 12:35:34 +03:00
James Gowdy b540ada036
[ML] Allow editing of model_snapshot_retention_days (#69003)
* [ML] Allow editing of model_snapshot_retention_days

* removing commented code

* flooring number input

* adding daily_model_snapshot_retention_after_days

* updating default values

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-06-15 21:45:51 +01:00
Oliver Gupte cac9018f81
[APM] Service maps - Fix missing ML status for services with jobs but no anomalies (#68486)
* Closes #68485 by:
  - use the ML plugin to query for all APM jobs
  - inspect the ml job groups to find all jobs related to a particular service
  - use the mlAnomalySearch client to get ml job buckets with the max anomaly score
  - query for the model_plot buckets to obtain actual/median values for the ML description
  - return the relevant ML job with the max anomaly score for a service
  - indicate to the user that no anomalies were found for a service with an ml job

* - Use the anomalyDetectorsProvider jobs API rather than the search endpoint directly
- Defines a specific return type for the ml jobs api
- Update the empty anomaly data message

* Code and types cleanup

* Return to using record result type on anomaly queries. These are the
same values used in the anomaly explorer and it includes actual &
typical values which greatly improve performance of the previous query.

* - If anomaly data is missing show a gray border around node

* - moved AnomalyDetection out of service map Contents into own component

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-06-09 20:58:46 -07:00
Robert Oskamp 004df98c87
[ML] Sample data modules - use event.dataset instead of index name (#68538)
This PR adds event.dataset to the Kibana ecommerce sample data and makes the ML modules for ecommerce and weblog sample data use event.dataset to recognize the modules and also as the corresponding datafeed query instead of looking for the index name.
2020-06-09 11:30:15 +02:00
Melissa Alvarez edc4d58e12
[ML] DF Analytics: Creation wizard part 1 (#67564)
* create newJob route and start of wizard

* wip: create configStep component

* finish configStep form and details

* wip: create andvanced step components

* create details step component

* createStep component

* ensure advanced options are correct for each job type

* add validation to each step

* use custom table for excludes

* move customSelectionTable to shared components

* form validation for advanced fields

* wip: source index selection modal

* add source index preview

* update details

* ensure advanced parameters added to config on creation

* can create job from savedSearch. can set source query in ui

* validate source object has supported fields

* eslint updates

* update tests. comment out clone action for now

* add create button to advanced editor

* remove deprecated test helper functions

* fix translation errors

* update help text. read only once job created.

* fix functional tests

* add nextStepNav to df service for tests

* fix excludes table page jump and hyperParameter not showing in details

* fix checkbox width for custom table
2020-06-04 13:39:43 -04:00
James Gowdy 0897418795
[ML] Adding per_partition_categorization to job interface and schema (#67953)
* [ML] Adding per_partition_categorization to job interface and schema

* adding new fields to Category interface
2020-06-03 15:23:54 +01:00
Quynh Nguyen f31330a01b
[ML] Add ability to delete target index & index pattern when deleting DFA job (#66934)
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-06-01 10:53:33 -05:00
Mikhail Shustov 53b95424fe
Deprecate es API exposed from setup contract (#67596)
* move elasticsearch client under legacy namespace

* update mocks and tests

* update platform code

* update legacy code

* update plugins using elasticsearch setup API

* update request handler context

* update docs

* rename remaining places

* address comments

* fix merge conflict error
2020-06-01 16:16:44 +02:00
James Gowdy cae6b9e35a
[ML] Using mlClient callAsInternalUser in job validation (#67609) 2020-05-28 18:04:16 +01:00
James Gowdy 321a10d1df
[ML] Renaming mlSearch to mlAnomalySearch (#67602) 2020-05-28 18:02:50 +01:00
James Gowdy dcca06bad2
[ML] Update job service id schema (#67462) 2020-05-28 11:50:56 +01:00
Mikhail Shustov 4040c3090b
Mark elasticsearch client exposed via request context as deprecated (#67319)
* add legacy prefix for es client exposed via request handler context

* update src/plugins

* update core mocks and tests

* update test plugins

* update xpack plugins

* include x-pack/mocks.ts

* update after master merge

* update docs

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-05-27 19:00:16 +02:00
restrry bf04235dae apply prettier styles 2020-05-22 09:08:58 +02:00
Pete Harverson 3e6b3ae74d
[ML] Fixes capabilities check for evaluate analytics endpoint (#66983) 2020-05-19 15:46:18 +01:00
Pete Harverson 1c61ca2c6d
[ML] Enhances api docs for modules endpoints (#66738)
* [ML] Enhances api docs for modules endpoints

* [ML] Edits to modules schema following review

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-05-18 18:13:43 +01:00
Dima Arnautov f7a89601c7
[ML] Custom template for apiDoc markdown (#66567)
* [ML] custom template for apiDoc markdown

* [ML] update apiDoc description
2020-05-15 11:41:29 +02:00
Dima Arnautov d0b9840041
[ML] API integration tests for job validation (#66265)
* [ML] TS refactoring

* [ML] fix page overflow

* [ML] tests

* [ML] fix typing issues and unit tests

* [ML] remove string conversion

* [ML] indexOf checks

* [ML] fix tooltip overflow

* [ML] fix i18n

* [ML] fix unit tests

* [ML] use MlJobAggregation

* [ML] use enums

* Revert "[ML] fix tooltip overflow"

This reverts commit 103c36bc

* Revert "[ML] fix page overflow"

This reverts commit 3c869228
2020-05-13 18:52:49 +02:00
Joe Reuter b7e87c2c8b
Switch to core application service (#63443) 2020-05-13 09:19:24 +02:00
Dima Arnautov d1e4d37c46
[ML] Cardinality validation API integration tests (#65971)
* [ML] refactor validate_cardinality to TS

* [ML] cardinality api integration tests

* [ML] resolve PR comments, validateJobObject as TS guard
2020-05-11 15:57:12 +02:00
Walter Rafelsberger e148eb4437
[ML] Anomaly Detection: Fix test to reflect model memory limit change. (#65967)
Adapt jest test mocks to consider update introduced in #65652.
2020-05-11 11:41:01 +02:00
Quynh Nguyen d3b155f843
[ML] Add job timing stats to anomaly jobs (#65696)
* [ML] Add anomaly job timing stats to Counts & JSON

* [ML] Remove roundTo3DecimalPlace and clean up

* [ML] Fix format_values to round decimals for time values

* [ML] Remove timing_stats and forecast_stats from cloneJob

* [ML] Remove timing_stats & forecasts in job_service instead of utils
2020-05-08 12:06:59 -05:00
Brian Seeders d40387161e
Skipping failing tests. #65867 #65866 #65865 2020-05-08 11:35:17 -04:00
James Gowdy 75e12db4ba
[ML] Show warning when the model memory limit is higher than the memory available in the ML node (#65652)
* [ML] Show warning when the model memory limit is higher than the memory available in the ML node

* reverting UI check

* removing from UI job validator

* adding cap to estimate mml

* adding mml value to message

* fixing translations

* updating translations

* fixing translation ids
2020-05-08 13:13:17 +01:00
Dima Arnautov 46fc73a213
[ML] add missing single quotes for index ids in modules custom urls (#65707) 2020-05-07 19:43:53 +02:00
Walter Rafelsberger 6a6b3edd7f
[ML] Migrate server side Mocha tests to Jest. (#65651)
Migrates job validation related server side tests from Mocha to Jest.
2020-05-07 15:58:16 +02:00
James Gowdy 55e4c7f9a7
[ML] Consolidating shared types and util functions (#65247)
* [ML] Consolidating shared types and util functions

* including formatter

* adding missing includes

* removing unused export

* ignoring numeral type error

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-05-07 12:23:42 +01:00
James Gowdy 8944dbaac4
[ML] Adding daily_model_snapshot_retention_after_days to types and schemas (#65417) 2020-05-06 10:22:40 +01:00
James Gowdy 943f8b9df1
[ML] Fix packetbeat module query (#65241) 2020-05-05 20:44:52 +01:00
Lisa Cawley 415d33b756
[ML] Edits create anomaly job intro text (#64607)
* Edits create anomaly job intro text

* [DOCS] More edits

* [DOCS] Removes obsolete translations

* [DOCS] Addresses feedback

* [DOCS] Update create job breadcrumb

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-05-05 11:38:20 -07:00
James Gowdy 33b2b5c92c
[ML] Enabling mml estimation in data recognizer module setup (#64900)
* [ML] Enabling mml estimation in data recognizer module setup

* small refactor

* adding functional tests

* increasing uptime test timeout

* tiny refactor

* checking for default setting

* testng flakey uptime test

* catching erros in mml estimation

* lowering timeout

* ensuing data is present for ML tests

* adding await

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-05-04 19:30:36 +01:00
Felix Stürmer 39e31d6123
[Logs UI] Add dataset filter to ML module setup screen (#64470)
This adds the ability to filter the datasets to be processed by the ML jobs on the setup screen.
2020-05-04 13:29:28 +02:00
James Gowdy 497c5da763
[ML] Moving get filters capability to admin (#64879)
* [ML] Moving get filters capability to admin

* updating test
2020-04-30 14:07:32 +01:00
James Gowdy d8c15f5ad3
[ML] Adding endpoint capability checks (#64662)
* [ML] Adding endpoint capability checks

* adding missing capability checks

* fixing test

* removing commented code

* fixing functional test

* fixing functional tests

* changes based on review

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-04-29 18:25:48 +01:00
James Gowdy b5fb78bfce
[ML] Fix new job wizard with multiple indices (#64567)
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-04-28 20:45:57 +01:00
James Gowdy f9c81a30cb
[ML] Add kibana setting for file data visualizer max file size (#64427)
* [ML] Add kibana setting for file data visualizers max file size

* adding failsafe for setting

* fixing id

* [DOCS] Updates Data Visualizer setting

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: lcawl <lcawley@elastic.co>
2020-04-28 09:22:46 +01:00
James Gowdy b3c7002799
[ML] Moving to kibana capabilities (#64057)
* [ML] Moving to kibana capabilities

* fixing types

* renaming privilges

* renaming privileges to capabilities

* renaming resolvers

* correcting admin capabilities

* fixing includes

* removing any types

* renaming type

* readding spaces

* adding capabilities switcher

* updating comment

* removing unnecessary failing tests

* adding error to log
2020-04-24 12:08:19 +01:00
Pete Harverson 09c2727d78
[ML] Adds API integration tests for data viz and fields endpoints (#64165)
* [ML] Adds API integration tests for data viz and fields endpoints

* [ML] Fix review comments and errors from settings endpoints

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-04-23 18:06:33 +01:00
Melissa Alvarez dc5c2f0e3f
[ML] Migrate Mocha unit tests to Jest: migrate job utils and query utils tests (#63775)
* migrate mocha tests to jest for query utils

* update jobUtils test to jest from mocha

* update tests to use jest syntax
2020-04-17 13:55:46 -04:00
Joe Portner d72de0ea16
Refactor saved object types to use namespaceType (#63217)
Remove `namespaceAgnostic` field that was deprecated in #54605,
use `namespaceType` instead.
2020-04-16 14:37:55 -04:00
Dima Arnautov 7058070e15
[ML] Extract apiDoc params from the schema definitions (#62933)
* [ML] WIP apiDoc schema extractor

* [ML] extract actual type

* [ML] refactor schema definitions

* [ML] Update README.md

* [ML] extract nested

* [ML] call job validation endpoint with complete payload

* [ML] escape special chars and fix line breaks

* [ML] clean up extractDocEntries

* [ML] serializeWithType

* [ML] add missing annotations

* [ML] fix parent schema assigment

* [ML] support object composition

* [ML] support multiple schemas per block

* [ML] fix for collections

* [ML] fix calendarIdsSchema

* [ML] add ml package.json with apidoc commands

* [ML] use the single output markdown file

* [ML] fix typo

* [ML] change the Calendars order

* [ML] adjust the order in adidoc.json

* [ML] update api version

* [ML] update tsconfig.json include

* [ML] update packages/kbn-pm/dist/index.js

* [ML] update ML overrides in .eslintrc.js

* [ML] yarn.lock symlink

* Revert "[ML] yarn.lock symlink"

This reverts commit 07f06801

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-04-15 18:40:34 +02:00
James Gowdy 6fc4840bae
[ML] Changing file data visualizer max upload setting to string (#63502) 2020-04-15 15:48:04 +01:00
James Gowdy 959a0e5d3d
[ML] Listing global calendars on the job management page (#63124)
* [ML] Listing global calendars on the job management page

* tiny refactor

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-04-14 18:57:26 +01:00
Larry Gregory 9d89a4fb49
Support multiple reserved feature privileges (#61980)
* support multiple reserved feature privileges

* update reserved privilege ids

* additional testing

* Add ml_user and ml_admin reserved privileges

* prrevent reserved privilege ids from sttarting with 'reserved_'

* address pr feedback: dedicated reserved privilege type

* re-enable ML test suites

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-04-08 13:03:15 -04:00
James Gowdy 6b52ce7341
[ML] Adding configurable file size to file data visualizer (#62752)
* [ML] Adding configurable file size to file data visualizer

* updating translated strings
2020-04-08 15:09:56 +01:00
James Gowdy 64f27ca34e
[ML] Show better file structure finder explanations (#62316)
* [ML] Show better file structure finder explanations

* more typescript changes

* changing function format

* fixing some types

* fixing translation id

* fix boom error reporting

* changes based on review

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-04-07 08:47:39 +01:00
Dima Arnautov cca0502ed3
[ML] call job validation endpoint with complete payload (#62331) 2020-04-03 10:22:49 +02:00
Dima Arnautov aeba1b7bb0
[ML] Handle Empty Partition Field Values in Single Metric Viewer (#61649)
* [ML] WIP support empty partition fields values

* [ML] support empty field in anomaly table

* [ML] remove comments

* [ML] fix context chart

* [ML] rename empty field label, render as italic

* [ML] rename empty field label

* [ML] fix focus chart

* [ML] add time range capping for fields_service.ts

* [ML] empty string labels in anomaly explorer
2020-03-31 16:29:00 +02:00