Commit graph

198 commits

Author SHA1 Message Date
Melissa Alvarez edc4d58e12
[ML] DF Analytics: Creation wizard part 1 (#67564)
* create newJob route and start of wizard

* wip: create configStep component

* finish configStep form and details

* wip: create andvanced step components

* create details step component

* createStep component

* ensure advanced options are correct for each job type

* add validation to each step

* use custom table for excludes

* move customSelectionTable to shared components

* form validation for advanced fields

* wip: source index selection modal

* add source index preview

* update details

* ensure advanced parameters added to config on creation

* can create job from savedSearch. can set source query in ui

* validate source object has supported fields

* eslint updates

* update tests. comment out clone action for now

* add create button to advanced editor

* remove deprecated test helper functions

* fix translation errors

* update help text. read only once job created.

* fix functional tests

* add nextStepNav to df service for tests

* fix excludes table page jump and hyperParameter not showing in details

* fix checkbox width for custom table
2020-06-04 13:39:43 -04:00
James Gowdy 0897418795
[ML] Adding per_partition_categorization to job interface and schema (#67953)
* [ML] Adding per_partition_categorization to job interface and schema

* adding new fields to Category interface
2020-06-03 15:23:54 +01:00
Quynh Nguyen f31330a01b
[ML] Add ability to delete target index & index pattern when deleting DFA job (#66934)
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-06-01 10:53:33 -05:00
Mikhail Shustov 53b95424fe
Deprecate es API exposed from setup contract (#67596)
* move elasticsearch client under legacy namespace

* update mocks and tests

* update platform code

* update legacy code

* update plugins using elasticsearch setup API

* update request handler context

* update docs

* rename remaining places

* address comments

* fix merge conflict error
2020-06-01 16:16:44 +02:00
James Gowdy cae6b9e35a
[ML] Using mlClient callAsInternalUser in job validation (#67609) 2020-05-28 18:04:16 +01:00
James Gowdy 321a10d1df
[ML] Renaming mlSearch to mlAnomalySearch (#67602) 2020-05-28 18:02:50 +01:00
James Gowdy dcca06bad2
[ML] Update job service id schema (#67462) 2020-05-28 11:50:56 +01:00
Mikhail Shustov 4040c3090b
Mark elasticsearch client exposed via request context as deprecated (#67319)
* add legacy prefix for es client exposed via request handler context

* update src/plugins

* update core mocks and tests

* update test plugins

* update xpack plugins

* include x-pack/mocks.ts

* update after master merge

* update docs

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-05-27 19:00:16 +02:00
restrry bf04235dae apply prettier styles 2020-05-22 09:08:58 +02:00
Pete Harverson 3e6b3ae74d
[ML] Fixes capabilities check for evaluate analytics endpoint (#66983) 2020-05-19 15:46:18 +01:00
Pete Harverson 1c61ca2c6d
[ML] Enhances api docs for modules endpoints (#66738)
* [ML] Enhances api docs for modules endpoints

* [ML] Edits to modules schema following review

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-05-18 18:13:43 +01:00
Dima Arnautov f7a89601c7
[ML] Custom template for apiDoc markdown (#66567)
* [ML] custom template for apiDoc markdown

* [ML] update apiDoc description
2020-05-15 11:41:29 +02:00
Dima Arnautov d0b9840041
[ML] API integration tests for job validation (#66265)
* [ML] TS refactoring

* [ML] fix page overflow

* [ML] tests

* [ML] fix typing issues and unit tests

* [ML] remove string conversion

* [ML] indexOf checks

* [ML] fix tooltip overflow

* [ML] fix i18n

* [ML] fix unit tests

* [ML] use MlJobAggregation

* [ML] use enums

* Revert "[ML] fix tooltip overflow"

This reverts commit 103c36bc

* Revert "[ML] fix page overflow"

This reverts commit 3c869228
2020-05-13 18:52:49 +02:00
Joe Reuter b7e87c2c8b
Switch to core application service (#63443) 2020-05-13 09:19:24 +02:00
Dima Arnautov d1e4d37c46
[ML] Cardinality validation API integration tests (#65971)
* [ML] refactor validate_cardinality to TS

* [ML] cardinality api integration tests

* [ML] resolve PR comments, validateJobObject as TS guard
2020-05-11 15:57:12 +02:00
Walter Rafelsberger e148eb4437
[ML] Anomaly Detection: Fix test to reflect model memory limit change. (#65967)
Adapt jest test mocks to consider update introduced in #65652.
2020-05-11 11:41:01 +02:00
Quynh Nguyen d3b155f843
[ML] Add job timing stats to anomaly jobs (#65696)
* [ML] Add anomaly job timing stats to Counts & JSON

* [ML] Remove roundTo3DecimalPlace and clean up

* [ML] Fix format_values to round decimals for time values

* [ML] Remove timing_stats and forecast_stats from cloneJob

* [ML] Remove timing_stats & forecasts in job_service instead of utils
2020-05-08 12:06:59 -05:00
Brian Seeders d40387161e
Skipping failing tests. #65867 #65866 #65865 2020-05-08 11:35:17 -04:00
James Gowdy 75e12db4ba
[ML] Show warning when the model memory limit is higher than the memory available in the ML node (#65652)
* [ML] Show warning when the model memory limit is higher than the memory available in the ML node

* reverting UI check

* removing from UI job validator

* adding cap to estimate mml

* adding mml value to message

* fixing translations

* updating translations

* fixing translation ids
2020-05-08 13:13:17 +01:00
Dima Arnautov 46fc73a213
[ML] add missing single quotes for index ids in modules custom urls (#65707) 2020-05-07 19:43:53 +02:00
Walter Rafelsberger 6a6b3edd7f
[ML] Migrate server side Mocha tests to Jest. (#65651)
Migrates job validation related server side tests from Mocha to Jest.
2020-05-07 15:58:16 +02:00
James Gowdy 55e4c7f9a7
[ML] Consolidating shared types and util functions (#65247)
* [ML] Consolidating shared types and util functions

* including formatter

* adding missing includes

* removing unused export

* ignoring numeral type error

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-05-07 12:23:42 +01:00
James Gowdy 8944dbaac4
[ML] Adding daily_model_snapshot_retention_after_days to types and schemas (#65417) 2020-05-06 10:22:40 +01:00
James Gowdy 943f8b9df1
[ML] Fix packetbeat module query (#65241) 2020-05-05 20:44:52 +01:00
Lisa Cawley 415d33b756
[ML] Edits create anomaly job intro text (#64607)
* Edits create anomaly job intro text

* [DOCS] More edits

* [DOCS] Removes obsolete translations

* [DOCS] Addresses feedback

* [DOCS] Update create job breadcrumb

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-05-05 11:38:20 -07:00
James Gowdy 33b2b5c92c
[ML] Enabling mml estimation in data recognizer module setup (#64900)
* [ML] Enabling mml estimation in data recognizer module setup

* small refactor

* adding functional tests

* increasing uptime test timeout

* tiny refactor

* checking for default setting

* testng flakey uptime test

* catching erros in mml estimation

* lowering timeout

* ensuing data is present for ML tests

* adding await

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-05-04 19:30:36 +01:00
Felix Stürmer 39e31d6123
[Logs UI] Add dataset filter to ML module setup screen (#64470)
This adds the ability to filter the datasets to be processed by the ML jobs on the setup screen.
2020-05-04 13:29:28 +02:00
James Gowdy 497c5da763
[ML] Moving get filters capability to admin (#64879)
* [ML] Moving get filters capability to admin

* updating test
2020-04-30 14:07:32 +01:00
James Gowdy d8c15f5ad3
[ML] Adding endpoint capability checks (#64662)
* [ML] Adding endpoint capability checks

* adding missing capability checks

* fixing test

* removing commented code

* fixing functional test

* fixing functional tests

* changes based on review

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-04-29 18:25:48 +01:00
James Gowdy b5fb78bfce
[ML] Fix new job wizard with multiple indices (#64567)
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-04-28 20:45:57 +01:00
James Gowdy f9c81a30cb
[ML] Add kibana setting for file data visualizer max file size (#64427)
* [ML] Add kibana setting for file data visualizers max file size

* adding failsafe for setting

* fixing id

* [DOCS] Updates Data Visualizer setting

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: lcawl <lcawley@elastic.co>
2020-04-28 09:22:46 +01:00
James Gowdy b3c7002799
[ML] Moving to kibana capabilities (#64057)
* [ML] Moving to kibana capabilities

* fixing types

* renaming privilges

* renaming privileges to capabilities

* renaming resolvers

* correcting admin capabilities

* fixing includes

* removing any types

* renaming type

* readding spaces

* adding capabilities switcher

* updating comment

* removing unnecessary failing tests

* adding error to log
2020-04-24 12:08:19 +01:00
Pete Harverson 09c2727d78
[ML] Adds API integration tests for data viz and fields endpoints (#64165)
* [ML] Adds API integration tests for data viz and fields endpoints

* [ML] Fix review comments and errors from settings endpoints

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-04-23 18:06:33 +01:00
Melissa Alvarez dc5c2f0e3f
[ML] Migrate Mocha unit tests to Jest: migrate job utils and query utils tests (#63775)
* migrate mocha tests to jest for query utils

* update jobUtils test to jest from mocha

* update tests to use jest syntax
2020-04-17 13:55:46 -04:00
Joe Portner d72de0ea16
Refactor saved object types to use namespaceType (#63217)
Remove `namespaceAgnostic` field that was deprecated in #54605,
use `namespaceType` instead.
2020-04-16 14:37:55 -04:00
Dima Arnautov 7058070e15
[ML] Extract apiDoc params from the schema definitions (#62933)
* [ML] WIP apiDoc schema extractor

* [ML] extract actual type

* [ML] refactor schema definitions

* [ML] Update README.md

* [ML] extract nested

* [ML] call job validation endpoint with complete payload

* [ML] escape special chars and fix line breaks

* [ML] clean up extractDocEntries

* [ML] serializeWithType

* [ML] add missing annotations

* [ML] fix parent schema assigment

* [ML] support object composition

* [ML] support multiple schemas per block

* [ML] fix for collections

* [ML] fix calendarIdsSchema

* [ML] add ml package.json with apidoc commands

* [ML] use the single output markdown file

* [ML] fix typo

* [ML] change the Calendars order

* [ML] adjust the order in adidoc.json

* [ML] update api version

* [ML] update tsconfig.json include

* [ML] update packages/kbn-pm/dist/index.js

* [ML] update ML overrides in .eslintrc.js

* [ML] yarn.lock symlink

* Revert "[ML] yarn.lock symlink"

This reverts commit 07f06801

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-04-15 18:40:34 +02:00
James Gowdy 6fc4840bae
[ML] Changing file data visualizer max upload setting to string (#63502) 2020-04-15 15:48:04 +01:00
James Gowdy 959a0e5d3d
[ML] Listing global calendars on the job management page (#63124)
* [ML] Listing global calendars on the job management page

* tiny refactor

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-04-14 18:57:26 +01:00
Larry Gregory 9d89a4fb49
Support multiple reserved feature privileges (#61980)
* support multiple reserved feature privileges

* update reserved privilege ids

* additional testing

* Add ml_user and ml_admin reserved privileges

* prrevent reserved privilege ids from sttarting with 'reserved_'

* address pr feedback: dedicated reserved privilege type

* re-enable ML test suites

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-04-08 13:03:15 -04:00
James Gowdy 6b52ce7341
[ML] Adding configurable file size to file data visualizer (#62752)
* [ML] Adding configurable file size to file data visualizer

* updating translated strings
2020-04-08 15:09:56 +01:00
James Gowdy 64f27ca34e
[ML] Show better file structure finder explanations (#62316)
* [ML] Show better file structure finder explanations

* more typescript changes

* changing function format

* fixing some types

* fixing translation id

* fix boom error reporting

* changes based on review

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-04-07 08:47:39 +01:00
Dima Arnautov cca0502ed3
[ML] call job validation endpoint with complete payload (#62331) 2020-04-03 10:22:49 +02:00
Dima Arnautov aeba1b7bb0
[ML] Handle Empty Partition Field Values in Single Metric Viewer (#61649)
* [ML] WIP support empty partition fields values

* [ML] support empty field in anomaly table

* [ML] remove comments

* [ML] fix context chart

* [ML] rename empty field label, render as italic

* [ML] rename empty field label

* [ML] fix focus chart

* [ML] add time range capping for fields_service.ts

* [ML] empty string labels in anomaly explorer
2020-03-31 16:29:00 +02:00
Melissa Alvarez f1866b331e
[ML] Settings: Increase number of items that can be paged in calendars and filters lists (#61842)
* set size limit to 1000 for calendars/filters get routes

* update comment

* update default size comment
2020-03-31 09:14:56 -04:00
Ryland Herrick 8b31ce0a89
[SIEM] Check ML Job status on ML Rule execution (#61715)
* Move isMlRule helper to a more general location

And use it during rule execution as well.

* Add error message back to rule error status

This was unintentionally removed in a previous merge commit.

* Expose mlClient as part of ML's Setup contract

This allows dependent plugins to leverage the exposed services without
having to define their own ml paths, e.g. "ml.jobs"

* Move ML Job predicates to common folder

These are pure functions and used on both the client and server.

* WIP: Check ML Job status on ML Rule execution

This works, but unfortunately it pushes this executor function to a
complexity of 25. We're gonna refactor this next.

* Move isMlRule and RuleType to common

These are used on both the frontend and the backend, and can be shared.

* Refactor Signal Rule executor to use RuleStatusService

RuleStatusService holds the logic for updating the current status as
well as adding an error status. It leverages a simple
RuleStatusSavedObjectClient to handle the communication with
SavedObjects.

This removes the need for our specialized 'writeError', 'writeGap', and
'writeSuccess' functions, which duplicated much of the rule status
logic and code. It also fixes a bug with gap failures, with should have
been treated the same as other failures.

NB that an error does not necessarily prevent the rule from running, as
in the case of a gap or an ML Job not running.

This also adds a buildRuleMessage helper to reduce the noise of
generating logs/messages, and to make them more consistent.

* Remove unneeded 'async' keywords

We're not awaiting here, so we can just return the promise.

* Make buildRuleStatusAttributes synchronous

We weren't doing anything async here, and in fact the returning of a
promise was causing a bug when we tried to spread it into our attributes
object.

* Fix incorrectly-named RuleStatus attributes

This mapping could be done within the ruleStatusService, but it
lives outside it for now.

Also renames the object holding these values to the more general
'result,' as creationSuccess implies it always succeeds.

* Move our rule message helpers to a separate file

Adds some tests, as well.

* Refactor how rule status objects interact

Only ruleStatusSavedObjectsClient receives a savedObjectsClient, the
other functions receive the ruleStatusSavedObjectsClient

* pluralizes savedObjects in ruleStatusSavedObjectsClient
* Backfills tests

* Handle adding multiple errors during a single rule execution

We were storing state in our RuleStatusClient, and consequently could
get into a situation where that state did not reflect reality, and we
would incorrectly try to delete a SavedObject that had already been
deleted.

Rather than try to store the _correct_ state in the service, we remove
state entirely and just fetch our statuses on each action.

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-03-30 16:35:38 -05:00
Robert Oskamp f5e94c4ca3
[ML] Move get_module Jest tests to functional API test (#61613)
This PR removes the flaky listModules and getModule Jest tests and adds them as functional API tests.
2020-03-27 16:12:15 +01:00
Robert Oskamp 4f863c627a
[ML] Add uptime_heartbeat module to recognizer test (#61235)
This PR adds the new uptime_heartbeat module to the recognizer list modules test.
2020-03-25 13:17:32 +01:00
Robert Oskamp db0a4e8ca0
[ML] Jest tests - re-activate data recognizer tests (#60525)
This PR re-activates the data recognizer Jest tests.
2020-03-25 09:57:08 +01:00
Pete Harverson a68eaf22c9
[ML] Allow edits to jobs with additional custom_settings props (#61067)
* [ML] Allow edits to jobs with additional custom_settings props

* [ML] Minor text edit

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-03-24 20:18:53 +00:00
Larry Gregory b82cc6ed4a
Support for sub-feature privileges (#60563)
* initial server-side support for sub-feature privileges (#57507)

* initial server-side support for sub-feature privileges

* start addressing PR feedback

* renaming interfaces

* move privilege id collision check to security plugin

* additional testing

* change featurePrivilegeIterator import location

* fix link assertions following rebase from master

* Initial UI support for sub-feature privileges (#59198)

* Initial UI support for sub-feature privileges

* Address PR feedback

* display deleted spaces correctly in the privilege summary

* additional testing

* update snapshot

* Enables sub-feature privileges for gold+ licenses (#59750)

* enables sub-feature privileges for gold+ licenses

* Address PR feedback

* address platform review feedback
2020-03-24 11:12:49 -04:00