Commit graph

87 commits

Author SHA1 Message Date
James Gowdy
18df677da7
[ML] Fixing file import, module creation and results viewing permission checks (#72825)
* [ML] Fixing file import and module creation permission checks

* correcting searches on results index

* fixing test

* removing unnecessary index

* updating apidoc

* fixing test

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-07-23 16:11:15 +01:00
Melissa Alvarez
d0d271c07d
[ML] DF Analytics creation: ensure monitor cluster privilege not required to create job (#71934)
* add checkIndexExists endpoint wrapping field_caps

* replace indexNames with checkIndexExists check

* update translations

* show error toast on index check fail

* add new route to api doc
2020-07-16 13:22:25 -04:00
Quynh Nguyen
c24f180391
[ML] Anomaly Detection: Annotations enhancements (#70198)
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-07-14 12:36:01 -05:00
James Gowdy
a1e511a727
[ML] Changing all calls to ML endpoints to use internal user (#70487)
* [ML] Changing all calls to ML endpoints to use internal user

* updating alerting

* updating documentation

* [ML] Changing all calls to ML endpoints to use internal user

* updating alerting

* updating documentation

* fixing missed types

* adding authorization headers to endpoint calls

* correcting has privileges call

* updating security tests

* odd eslint error

* adding auth header to module setup

* fixing missing auth argument

* fixing delete DFA job permission checks

* removing debug test tag

* removing additional ml privilege checks

* adding authorization header to _evaluate

* updating alerting cluster client name

* code clean up

* changing authorizationHeader name

* updating alterting documentation

* fixing secondary credentials

* adding management links

* updating SIEM telemetry

* fixing merge conflicts

* granting access to index patterns

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-07-14 15:48:24 +01:00
Walter Rafelsberger
262e0754ff
[ML] Kibana API endpoint for histogram chart data (#70976)
- Introduces dedicated Kibana API endpoints as part of ML and transform plugin API endpoints and moves the logic to query and transform the required data from client to server.
- Adds support for sampling to retrieve the data for the field histograms. For now this is not configurable by the end user and is hard coded to 5000. This is to have a first iteration of this functionality in for 7.9 and protect users when querying large clusters. The button to enable the histogram charts now includes a tooltip that mentions the sampler.
2020-07-14 13:37:36 +02:00
Melissa Alvarez
1b1962f18c
[ML] DF Analytics creation and update: adds max_num_threads (#71318)
* add max_num_threads to edit flyout

* add maxNumThreads setting to job wizard

* add maxNumThreads to cloning
2020-07-13 16:51:22 -04:00
Quynh Nguyen
cf3133a5df
[ML] Add switch to enable model plot annotations independently (#70678) 2020-07-09 09:15:57 -05:00
James Gowdy
a86110488b
[ML] Fixing missing daily_model_snapshot_retention_after_days in job update schema (#71086) 2020-07-08 16:19:12 +01:00
Melissa Alvarez
a4340f0ece
[ML] DF Analytics: add ability to edit job for fields supported by API (#70489)
* wip: add edit action to dfanalytics table

* add update endpoint and edit flyout

* show success and error toasts. close flyout and refresh on success

* show permission message in edit action

* update types

* disable update button if mml not valid

* show error in toast, init values are config values

* fix undefined check for allow lazy start

* prevent update if mml is empty
2020-07-06 15:10:01 -04:00
James Gowdy
97ad58c529
[ML] Changing shared module setup function parameters (#70589)
* [ML] Changing shared module setup function parameters

* changes based on review

* adding comments for documentation

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-07-03 08:30:17 +01:00
James Gowdy
eafd2af6aa
[ML] Adding capabilities checks to shared functions (#70069)
* [ML] Adding capabilities checks to shared functions

* small refactor

* disabling capabilities checks for functions called by SIEM alerting

* testing git

* removing comment

* using constant for ml app id

* tiny type clean up

* removing check in ml_capabilities

* fixing types

* removing capabilities checks from ml_capabilities endpoint

* updating types

* better error handling

* improving capabilities check

* adding custom errors

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-07-01 17:45:36 +01:00
James Gowdy
7a60f18ef9
[ML] Model snapshot management (#68182)
* [ML] Model snapshot management

* more updates

* adding calendar range

* updating layout

* multiple calendars

* moving calendar creator

* fixing chart issues

* fixing chart issues

* improving calendar rendering

* adding capabilities checks

* code clean up

* fixing end time argument type

* fix translations

* code clean up

* comments based on review

* changes based on review

* fixing include

* adding useMemo to theme function
2020-06-16 14:03:17 +01:00
James Gowdy
b540ada036
[ML] Allow editing of model_snapshot_retention_days (#69003)
* [ML] Allow editing of model_snapshot_retention_days

* removing commented code

* flooring number input

* adding daily_model_snapshot_retention_after_days

* updating default values

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-06-15 21:45:51 +01:00
Melissa Alvarez
edc4d58e12
[ML] DF Analytics: Creation wizard part 1 (#67564)
* create newJob route and start of wizard

* wip: create configStep component

* finish configStep form and details

* wip: create andvanced step components

* create details step component

* createStep component

* ensure advanced options are correct for each job type

* add validation to each step

* use custom table for excludes

* move customSelectionTable to shared components

* form validation for advanced fields

* wip: source index selection modal

* add source index preview

* update details

* ensure advanced parameters added to config on creation

* can create job from savedSearch. can set source query in ui

* validate source object has supported fields

* eslint updates

* update tests. comment out clone action for now

* add create button to advanced editor

* remove deprecated test helper functions

* fix translation errors

* update help text. read only once job created.

* fix functional tests

* add nextStepNav to df service for tests

* fix excludes table page jump and hyperParameter not showing in details

* fix checkbox width for custom table
2020-06-04 13:39:43 -04:00
James Gowdy
0897418795
[ML] Adding per_partition_categorization to job interface and schema (#67953)
* [ML] Adding per_partition_categorization to job interface and schema

* adding new fields to Category interface
2020-06-03 15:23:54 +01:00
Quynh Nguyen
f31330a01b
[ML] Add ability to delete target index & index pattern when deleting DFA job (#66934)
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-06-01 10:53:33 -05:00
James Gowdy
cae6b9e35a
[ML] Using mlClient callAsInternalUser in job validation (#67609) 2020-05-28 18:04:16 +01:00
James Gowdy
dcca06bad2
[ML] Update job service id schema (#67462) 2020-05-28 11:50:56 +01:00
Mikhail Shustov
4040c3090b
Mark elasticsearch client exposed via request context as deprecated (#67319)
* add legacy prefix for es client exposed via request handler context

* update src/plugins

* update core mocks and tests

* update test plugins

* update xpack plugins

* include x-pack/mocks.ts

* update after master merge

* update docs

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-05-27 19:00:16 +02:00
restrry
bf04235dae apply prettier styles 2020-05-22 09:08:58 +02:00
Pete Harverson
3e6b3ae74d
[ML] Fixes capabilities check for evaluate analytics endpoint (#66983) 2020-05-19 15:46:18 +01:00
Pete Harverson
1c61ca2c6d
[ML] Enhances api docs for modules endpoints (#66738)
* [ML] Enhances api docs for modules endpoints

* [ML] Edits to modules schema following review

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-05-18 18:13:43 +01:00
Dima Arnautov
f7a89601c7
[ML] Custom template for apiDoc markdown (#66567)
* [ML] custom template for apiDoc markdown

* [ML] update apiDoc description
2020-05-15 11:41:29 +02:00
Dima Arnautov
d0b9840041
[ML] API integration tests for job validation (#66265)
* [ML] TS refactoring

* [ML] fix page overflow

* [ML] tests

* [ML] fix typing issues and unit tests

* [ML] remove string conversion

* [ML] indexOf checks

* [ML] fix tooltip overflow

* [ML] fix i18n

* [ML] fix unit tests

* [ML] use MlJobAggregation

* [ML] use enums

* Revert "[ML] fix tooltip overflow"

This reverts commit 103c36bc

* Revert "[ML] fix page overflow"

This reverts commit 3c869228
2020-05-13 18:52:49 +02:00
James Gowdy
8944dbaac4
[ML] Adding daily_model_snapshot_retention_after_days to types and schemas (#65417) 2020-05-06 10:22:40 +01:00
James Gowdy
d8c15f5ad3
[ML] Adding endpoint capability checks (#64662)
* [ML] Adding endpoint capability checks

* adding missing capability checks

* fixing test

* removing commented code

* fixing functional test

* fixing functional tests

* changes based on review

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-04-29 18:25:48 +01:00
James Gowdy
b3c7002799
[ML] Moving to kibana capabilities (#64057)
* [ML] Moving to kibana capabilities

* fixing types

* renaming privilges

* renaming privileges to capabilities

* renaming resolvers

* correcting admin capabilities

* fixing includes

* removing any types

* renaming type

* readding spaces

* adding capabilities switcher

* updating comment

* removing unnecessary failing tests

* adding error to log
2020-04-24 12:08:19 +01:00
Pete Harverson
09c2727d78
[ML] Adds API integration tests for data viz and fields endpoints (#64165)
* [ML] Adds API integration tests for data viz and fields endpoints

* [ML] Fix review comments and errors from settings endpoints

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-04-23 18:06:33 +01:00
Dima Arnautov
7058070e15
[ML] Extract apiDoc params from the schema definitions (#62933)
* [ML] WIP apiDoc schema extractor

* [ML] extract actual type

* [ML] refactor schema definitions

* [ML] Update README.md

* [ML] extract nested

* [ML] call job validation endpoint with complete payload

* [ML] escape special chars and fix line breaks

* [ML] clean up extractDocEntries

* [ML] serializeWithType

* [ML] add missing annotations

* [ML] fix parent schema assigment

* [ML] support object composition

* [ML] support multiple schemas per block

* [ML] fix for collections

* [ML] fix calendarIdsSchema

* [ML] add ml package.json with apidoc commands

* [ML] use the single output markdown file

* [ML] fix typo

* [ML] change the Calendars order

* [ML] adjust the order in adidoc.json

* [ML] update api version

* [ML] update tsconfig.json include

* [ML] update packages/kbn-pm/dist/index.js

* [ML] update ML overrides in .eslintrc.js

* [ML] yarn.lock symlink

* Revert "[ML] yarn.lock symlink"

This reverts commit 07f06801

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-04-15 18:40:34 +02:00
James Gowdy
6fc4840bae
[ML] Changing file data visualizer max upload setting to string (#63502) 2020-04-15 15:48:04 +01:00
James Gowdy
64f27ca34e
[ML] Show better file structure finder explanations (#62316)
* [ML] Show better file structure finder explanations

* more typescript changes

* changing function format

* fixing some types

* fixing translation id

* fix boom error reporting

* changes based on review

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-04-07 08:47:39 +01:00
Melissa Alvarez
f1866b331e
[ML] Settings: Increase number of items that can be paged in calendars and filters lists (#61842)
* set size limit to 1000 for calendars/filters get routes

* update comment

* update default size comment
2020-03-31 09:14:56 -04:00
Pete Harverson
a68eaf22c9
[ML] Allow edits to jobs with additional custom_settings props (#61067)
* [ML] Allow edits to jobs with additional custom_settings props

* [ML] Minor text edit

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-03-24 20:18:53 +00:00
Dima Arnautov
66b5efd084
[ML] Module setup with dynamic model memory estimation (#60656)
* [ML] add estimateModelMemory to the setup endpoint

* [ML] wip caching cardinality checks

* [ML] refactor

* [ML] fix a fallback time range

* [ML] fix typing issue

* [ML] fields_aggs_cache.ts as part of fields_service

* [ML] fix types, add comments

* [ML] check for MML overrides

* [ML] disable estimateModelMemory

* [ML] fix typing

* [ML] check for empty max mml

* [ML] refactor, update types, fix jobsForModelMemoryEstimation

* [ML] fix override lookup

* [ML] resolve nit comments

* [ML] init jobsForModelMemoryEstimation
2020-03-24 08:46:11 +01:00
James Gowdy
de7151e204
[ML] Disabling datafeed editing when job is running (#60751)
* [ML] Disabling datafeed editing when job is running

* changing variable

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-03-23 16:40:56 +00:00
Dima Arnautov
7aa4651292
[ML] Use a new ML endpoint to estimate a model memory (#60376)
* [ML] refactor calculate_model_memory_limit route, use estimateModelMemory endpoint

* [ML] refactor validate_model_memory_limit, migrate tests to jest

* [ML] fix typing issue

* [ML] start estimateModelMemory url with /

* [ML] fix typo, filter mlcategory

* [ML] extract getCardinalities function

* [ML] fields_service.ts

* [ML] wip getMaxBucketCardinality

* [ML] refactor and comments

* [ML] fix aggs keys with special characters, fix integration tests

* [ML] use pre-defined job types

* [ML] fallback to 0 in case max bucket cardinality receives null

* [ML] calculateModelMemoryLimit on influencers change

* [ML] fix maxModelMemoryLimit

* [ML] cap aggregation to max 1000 buckets

* [ML] rename intervalDuration
2020-03-19 16:45:40 +01:00
James Gowdy
fd16c46128
[ML] Re-enabling file upload telemetry (#60418)
* [ML] Re-enabling file upload telemetry

* small refactor

* removing exported function

* removing commented out code

* removing commented out include

* cleaning up types
2020-03-18 08:33:53 +00:00
James Gowdy
35302ed273
[ML] Client side cut over (#60100)
* [ML] Client side cut over

* updating paths and commented code

* changes based on review

* disabling telemetry tests

* fixing start job stylesheets

* fixing everything that is broken

* fixing types and ml icon order

* using icon constant
2020-03-13 19:16:41 +00:00
James Gowdy
d5c092811b
[ML] Adding shared services to ml setup contract (#59730)
* [ML] Adding shared services to ml setup contract

* adding data recognizer

* typescripting js client

* adding results service

* code clean up

* adding generic ml index search

* making cloud optional
2020-03-12 10:04:40 +00:00
James Gowdy
92cc04c7df
[ML] Refactoring anomaly detector job types (#59556)
* [ML] Refactoring anomaly detector job types

* removing calendar type

* update validateCardinality

* fixing test

* updating  types in functional tests

* using state constants
2020-03-09 13:51:07 +00:00
Melissa Alvarez
d55524c338
[ML] NP Server: make security and spaces plugins optional (#59156)
* make security and spaces plugins optional

* update spacesPlugin name. update current user check
2020-03-05 10:06:36 -05:00
James Gowdy
a6d2b57777
[ML] Fixing records and buckets results endpoints (#59313) 2020-03-04 17:04:47 +00:00
James Gowdy
eb40046b93
[ML] Fixing licensing after server NP cutover (#59275) 2020-03-04 10:14:11 +00:00
Jonathan Budzenski
ff23fa3849
Revert "[ML] Fixing licensing after server NP cutover (#58853)" (#59244)
This reverts commit fd5c6c16cd.
2020-03-03 17:09:29 -06:00
James Gowdy
fd5c6c16cd
[ML] Fixing licensing after server NP cutover (#58853)
* [ML] Fixing licensing after server NP cutover

* client side refactor

* management refactor

* moving management license check

* adding server side license class

* added common license class

* client sides using license class

* updating server api guards

* fix privileges tests

* updating translations

* fixing disabled tabs on basic license

* refactoring client side license checks

* setting license initialized flag

* fixing license check on index data viz

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
2020-03-03 18:57:40 +00:00
James Gowdy
4ef594c208
[ML] Adding indices_options to datafeed (#59119)
* [ML] Adding indices_options to datafeed

* adding extra checks to the schema

* updating expand_wildcards
2020-03-03 11:59:35 +00:00
Melissa Alvarez
8c3d71b370
[ML] NP: migrate server (#58680)
* remove obsolete legacy server deps

* licensePreRoutingFactory uses licensing plugin rather than legacy xpack

* move schemas to dir in routes

* use NP license check method for license check

* store license data in plugin for passing to check

* create server plugin files in NP plugin dir

* remove dependency on legacy xpack plugin

* add sample data links first step

* move all server dirs from legacy to np dir

* fix requiredPlugin spaces name and update import routes

* delete unnecessary files and add sample data links

* update license and privilege check tests

* add routeInit types
2020-02-27 21:15:08 -05:00
Court Ewing
7ac8e4d9cc Move x-pack plugins/server/common to x-pack/legacy 2019-06-20 13:34:48 -04:00
James Gowdy
89892ba097
[ML] Fixing server side logger after new platform change (#39154) 2019-06-18 15:18:44 +01:00
Melissa Alvarez
c47a0d5cdf
[ML] New Platform Migration - server (#38360)
* wip: pull out all server dependencies

* create new platform plugin

* shim newPlatform plugin into legacy init

* update server tests for migration refactor

* cleanup interfaces

* Only add ML links for sample data sets if full license from - 38120

* update test and fix typescript errors

* Remove unused types
2019-06-17 16:08:13 -04:00