Commit graph

90 commits

Author SHA1 Message Date
Pete Harverson
0e5730c991
[ML] Removes non ECS apache, nginx and auditbeat data recognizer modules (#30537) 2019-02-11 12:21:21 +00:00
Walter Rafelsberger
e312daecfc
[ML] Fix Job Audit Messages filter. (#30490)
The way job audit messages were fetched didn't retrieve the expected results if there were deleted jobs with messages still present for these jobs.
This fix allows to specify a list of job IDs to filter the audit messages on. For the jobs list UI, the currently existing job IDs will be passed on to ignore messages from deleted jobs.
2019-02-08 15:13:17 +01:00
Pete Harverson
aed16cf27b
[ML] Fixes bucket spans and detector descriptions in ECS auditbeat modules (#30362) 2019-02-07 12:50:24 +00:00
Walter Rafelsberger
3d4dc093f3
[ML] Adds jest tests for annotation_service. (#30193)
This adds tests to `server/models/annotation_service`. The tests include a check if the `.key` attribute of an annotation is properly removed from an annotation before indexing it.
2019-02-06 14:43:04 +01:00
Walter Rafelsberger
0942332b0a
[ML] Remove key attribute from annotation before indexing. (#30183)
The UI adds a `key` attribute to annotation objects to store the letter used for labels in the chart and tables. When editing and saving an annotation that `key` could end up being saved to the annotations index. This isn't necesseary since the `key` attribute is just a dynamic label used within the UI. This fixes it by deleting an eventual `key` attribute from the annotations object before saving it to the index.
2019-02-06 11:23:29 +01:00
Sophie Chang
9ebddcc8a4 [ML] Update auditbeat modules for ECS (#29934)
* [ML] Initial commit for auditbeat hosts ECS

Rename fields for ECS
Rework dashboards due to bwc

* [ML] Further auditbeat tidy up and consistency changes

Custom urls should link to saved search, not discover
Ensure savedSearchId is used for visualizations
Ensure filter terms are consistent

TODO Decide if we should rename to auditd module
TODO Fix for new saved object format

* [ML] Refinements for auditbeat host module

Remove duplicated title from visState
Shrink panel heights in row 1

* [ML] Refinements to auditbeat module

Update module name from auditd to auditbeat
Add useMargins true for dashboards
Add filter to custom url for
   exists auditd.data.syscall
   not exists container.runtime
   event.module: auditd

* [ML] Initial commit for auditbeat_process_docker_ecs

Update for ECS using
  container.name (instead of container.id)
  container.runtime: docker
  process.executable
  event.module: auditd
  auditd.data.syscall exists

TODOs
Use auditd.message_type: syscall (instead of auditd.data.syscall)
Possibly combine with auditbeat hosts saved objects (depending on host.name being shared)
Possibly combine to single dashboard
Test against live auditbeat data collection
2019-02-05 19:06:25 +00:00
Walter Rafelsberger
fd8434a823
[ML] Fixes annotations integrity check. (#30102)
With security enabled, the internal user wouldn't have enough permissions to run the integrity check. This changes the check to use the currently logged in user. Also fixes some typos in messages.
2019-02-05 19:45:14 +01:00
Pete Harverson
d0f3b2a068
[ML] Fix filter params in HTTP Access job custom URLs (#30066) 2019-02-05 15:41:55 +00:00
James Gowdy
2850fd6735
[ML] Job deleting optimisations (#29848)
* [ML] Job deleting optimisations

* fixing force=true

* updating deleting jobs check
2019-02-05 12:29:01 +00:00
James Gowdy
fda6efed1a
[ML] Adding index migration warnings (#28938)
* [ML] Adding index migration warnings

* small refactor

* correctlng comment

* adding upgrade service to manage upgradeInProgress state

* removing missing function
2019-02-05 12:27:32 +00:00
Walter Rafelsberger
f2e3aff333
[ML] Adds an integrity check to creating, updating and deleting annotations. (#29969)
Adds an integrity check to make sure the required index and aliases are present before creating, updating or deleting annotations.
2019-02-05 10:14:45 +01:00
Pete Harverson
6afcc28c3b
[ML] Edits to labels in the Apache and Nginx ECS data recognizer modules (#29935) 2019-02-04 17:19:25 +00:00
Walter Rafelsberger
7966b2ff78
[ML] adds telemetry to ML (#29121)
Adds telemetry to collect the amount of indices created using File Data Visualizer.
2019-02-04 13:14:25 +01:00
Pete Harverson
0c86822e40
[ML] Automatically migrate saved objects created by data recognizer (#29736)
* [ML] Automatically migrate saved objects created by data recognizer

* [ML] Remove TODO from saved object import step
2019-01-31 19:54:22 +00:00
Sophie Chang
413d37d5ed [ML] Update http access modules for ECS (#29383)
* [ML] Initial commit for apache ecs module

* [ML] Update apache2 module for ECS

Rename following fields
event.module:apache
event.dataset:access
source.address
url.original
http.response.status_code
source.geo.location

Rationalise to only use one set of kibana saved objects for all http web access logs
Rename files from apache
Combined URL explorer into Count explorer dashboard as there was a lot of duplication
Add filter to custom url
Rename custom urls to Investigate Source IP and Status Code
Add chart to show overall event rate split by event.module - can tell if multiple datasets are included
Increase limit for top source ips from 5 to 50
Add created_by to custom setting for telemetry
Rename jobs and saved objects to include ecs tag

Tested side by side against v6 jobs

* [ML] Rename apache files from hyphen to underscores

* [ML] Further apache renames

Also change custom URLs to lower case to match "View series"
Change created_by to ml-module-apache-access

* [ML] Initial commit of nginx ml module

* [ML] Rename dashboard to generic explorer

* [ML] Further refinement for apache

Rename http_status_code to status_code_rate
Update custom url to use filters instead of lucene query bar

* [ML] Convert apache module to nginx

Copy files, keeping nginx logo
Multiple renames to nginx

* [ML] Make chart legend visible by default
2019-01-31 11:39:18 +00:00
Mike Côté
1b0f595f01
Add new "references" attribute to saved objects for relationships (#28199)
* Add new references attribute to saved objects

* Add dual support for dashboard export API

* Use new relationships API supporting legacy relationships extraction

* Code cleanup

* Fix style and CI error

* Add missing spaces test for findRelationships

* Convert collect_references_deep to typescript

* Add missing trailing commas

* Fix broken test by making saved object API consistently return references

* Fix broken api integration tests

* Add comment about the two TS types for saved object

* Only return title from the attributes returned in findRelationships

* Fix broken test

* Add missing security tests

* Drop filterTypes support

* Implement references to search, dashboard, visualization, graph

* Add index pattern migration to dashboards

* Add references mapping to dashboard mppings.json

* Remove findRelationships from repository and into it's own function / file

* Apply PR feedback pt1

* Fix some failing tests

* Remove error throwing in migrations

* Add references to edit saved object screen

* Pass types to findRelationships

* [ftr] restore snapshots from master, rely on migrations to add references

* [security] remove `find_relationships` action

* remove data set modifications

* [security/savedObjectsClient] remove _getAuthorizedTypes method

* fix security & spaces tests to consider references and migrationVersion

* Add space id prefixes to es_archiver/saved_objects/spaces/data.json

* Rename referenced attributes to have a suffix of RefName

* Fix length check in scenario references doesn't exist

* Add test for inject references to not be called when references array is empty or missing

* some code cleanup

* Make migrations run on machine learning data files, fix rollup filterPath for savedSearchRefName

* fix broken test

* Fix collector.js to include references in elasticsearch response

* code cleanup pt2

* add some more tests

* fix broken tests

* updated documentation on referencedBy option for saved object client find function

* Move visualization migrations into kibana plugin

* Update docs with better description on references

* Apply PR feedback

* Fix merge

* fix tests I broke adressing PR feedback

* PR feedback pt2
2019-01-30 15:53:03 -05:00
James Gowdy
ec607e26b9
[ML] Better handling of recognizer module manifest parsing errors (#29322) 2019-01-28 10:32:15 +00:00
James Gowdy
45b8ff99f0
[ML] Adding ability to override number of sample lines in File Data Visualizer (#29214)
* [ML] Adding ability to override number of sample lines in file data viz

* tiny tweak

* updating tests
2019-01-24 10:03:09 +00:00
Spencer
c052613962
[dashboard+gis] remove dark mode options (#29017)
* [dashboard+gis] remove dark mode options

* [reporting/extract] restore fixtures

* remove mentions of old `.theme-dark` class

* import panel styles from panel/_index.scss
2019-01-23 14:29:35 -08:00
James Gowdy
c91ec0d1ef
[ML] Adding privilege checks to File Data Visualizer (#29109)
* [ML] Adding privilege checks to File Data Visualizer

* fixing typo in comment
2019-01-22 17:07:53 +00:00
Pete Harverson
ed9a599b0d
[ML] Removes use of types from ES index and search calls (#28987) 2019-01-18 12:32:31 +00:00
Melissa Alvarez
7f8203715d
[ML] Add support to Data Visualizer for index patterns without a time field (#28511)
* Prevent docCount fetch and remove sidebar if no timeField set.

* Don't show metrics section if no metrics cards

* Add parens to conditional statement as per styleguide

* Don't create docCount card if not timeseries based
2019-01-14 12:31:46 -05:00
pavel06081991
19b84f0bc2
[ML] translate job validation messages (#28235)
[ML] translate job validation messages
2019-01-14 14:04:39 +03:00
pavel06081991
09ea8e35a3
[ML] translate models (#27888)
[ML] translate models
2019-01-14 12:24:30 +03:00
Nox911
2dfe4136b9 Translate ml -> server -> lib (#27974) 2019-01-08 11:13:17 +03:00
Pete Harverson
ef5e7a596e
[ML] Update REST endpoint calls to remove _xpack (#28069) 2019-01-04 16:48:05 +00:00
James Gowdy
0232b9b69e
[ML] Fix for audit message search size being 0 when jobs size is 0 (#27955) 2019-01-03 11:06:50 +00:00
Dimitris Athanasiou
b2f44622fe
[ML] Remove use of types in datafeeds (#27572)
This commit removes the use of types in datafeeds
and a few more places where it was possible.

Relates #26968
2019-01-02 09:34:38 +02:00
Walter Rafelsberger
7971b15cec
[ML] Fix audit messages. (#27575) 2018-12-21 10:59:52 +01:00
Pete Harverson
9b4a5a07d8
[ML] Handle removal of backend ml.enabled node attribute (#27604) 2018-12-21 09:13:30 +00:00
Walter Rafelsberger
5a7ab185f9
[ML] Adds a check if the current user can access the indexpattern required to clone a job. (#27496)
Adds a check if all index patterns necessary to clone a job are available for the current user. The check will only allow cloning of a job if the user has the right permissions and can still access the indexPattern the job was created for. An indexPattern could either have been deleted since the the job was created or the current user doesn't have the required permissions to access the indexPattern. The "Clone Job" action in the jobs list action menu will be greyed out and unavailable for that job.
2018-12-20 10:34:27 +01:00
Pete Harverson
bfa80411bb
[ML] Fix data visualizer to obtain total hit count as int (#27490) 2018-12-19 13:42:04 +00:00
Walter Rafelsberger
59d078e31c
[ML] Fix annotations feature startup behavior. (#27424)
* [ML] Fix annotations feature startup behavior.
* [ML] Move strings to constants.
2018-12-18 21:46:16 +01:00
Walter Rafelsberger
c6dc1a1fec
[ML] List all annotations in jobs list annotations table. (#27300)
* [ML] List all annotations in jobs list annotations table even outside analysis time range.
* [ML] Adjust the links time range if annotation is outside analysis time range.
2018-12-18 09:24:24 +01:00
Pete Harverson
6cc9bc2fc6
[ML] Adds created_by meta data for file data viz import indexes (#27303) 2018-12-17 15:28:04 +00:00
Pete Harverson
c334c40453
[ML] Updates to jobs and dashboards in auditbeat process modules (#27233) 2018-12-17 11:35:16 +00:00
Walter Rafelsberger
26c77eb25e
[ML] User Annotations (#26034)
Allows users to add/edit/delete annotations in the Single Series Viewer.
2018-12-14 14:09:52 +01:00
Pete Harverson
ae27e83712
[ML] Add rest_total_hit_as_int where total hits is required (#26421) 2018-12-04 09:25:24 +00:00
Pete Harverson
d97609eaeb
[ML] Aggregate anomalies table data using configured Kibana timezone (#26192)
* [ML] Aggregate anomalies table data using configured Kibana timezone

* [ML] Move dataFormatTz prop out of controller scope
2018-11-27 11:49:47 +00:00
Pete Harverson
cc07aa29d5
[ML] Edits to fields used in auditbeat module configurations (#25866) 2018-11-19 16:22:42 +00:00
Melissa Alvarez
354d7cc431
[ML] Add checkbox to enable model plot in Advanced job wizard (#25468)
* Move cardinality success check to utils

* enableModelPlot checkbox base added

* Run cardinality check on add/update fields

* Handle changes made via json

* only run cardinality check if model plot enabled

* Handle model plot enabled via EditJSON tab

* show message on cardinality check error

* multi-metric + pop: show message on cardinality check error

* add test for callout component

* Fix flexitem overflow in IE11
2018-11-16 13:41:08 -08:00
Pete Harverson
b52ddd206a
[ML] Adds auditbeat process data recognizer modules (#25716)
* [ML] Adds auditbeat process data recognizer modules

* [ML] Sorts Kibana objects by title in recognizer job wizard

* [ML] Rename auditbeat modules Kibana objects to snake_case

* [ML] Remove auditbeat docker module kibana files

* [ML] Add auditbeat docker kibana objects with lowercase names

* [ML] Remove auditbeat host module kibana files

* [ML] Add auditbeat host module files with lowercase filenames
2018-11-16 15:33:42 +00:00
Pete Harverson
1a3f0cac4b
[ML] Fixes recognizer job wizard to use dedicated index setting (#25706) 2018-11-15 14:40:41 +00:00
Pete Harverson
df0d62f6ed
[ML] Updates job type and APM module icon to new designs (#25380)
* [ML] Updates job type and APM module icon to new designs

* [ML] Use self closing tags for icons

* [ML] Removes unnecessary icon from jobs list
2018-11-08 21:13:59 +00:00
Walter Rafelsberger
3b94e3cece
[ML] Removes deprecated angularjs based jobs list and related code. (#25216) 2018-11-06 20:05:45 +01:00
Melissa Alvarez
830e149787
[ML] MultiMetric/Population Job creation: Allow model plot enablement via checkbox (#24914)
* Add route/api-mapping for validateCardinality

* Create directive for enableModelPlot checkbox

* Ensure model plot enabled prior to cardinality check

* Add callout when cardinality high

* ensure correct cardinality success check

* Population wizard: add enableModelPlot checkbox

* Update with suggested changes from review

* Remove warning when invalid. Add tests.

* Ensure checkbox updated on uncheck
2018-11-06 11:52:26 +00:00
Walter Rafelsberger
e5ebc62b16
[ML] Fixes typo. (#25130)
Fixes "Dectector" Typo.
2018-11-05 18:13:56 +01:00
James Gowdy
62f82c96f6
[ML] Fixing issues when trying to import non-timestamp data (#24894)
* [ML] Fixing issues when trying to import non-timestamp data

* making results link time range optional

* small cleanup after fixing merge conflict
2018-10-31 18:06:49 +00:00
Josh Dover
27e5406d7a
Upgrade Hapi in legacy platform to v17 (#21707)
* Disable even-better monitoring

* Upgrade to Hapi v15

* Upgrade to Hapi v16

* Handle optional req params correctly

* Update http and kbnServer

* Get mocha tests passing

* Convert `reply` usages [wip]

* Fix Joi and Plugin incompatibilities

* Get server up and running

* Get basic logging working

* Fix optimizer

* Fix recent route handlers

* Various fixes

* Fix recent routes

* Upgrade wreck for async/await

* Fix mocha tests

* Fix joi issues

* Fix xpack jest tests

* Fix recent routes

* Fix tests

* Fix index setup

* Decouple monitoring stats collection from good plugin

* Update reload logging test to work

* Reimplement logging with updated good plugin

* Fix unit tests

* Fix getConnections back

* Make LegacyLoggingServer compatible with Hapi v17

* Update joi types

* Fix x-pack unit tests

* Remove stray debugger

* Remove hapi-compat

* Fix API integrations

* Upgrade boom

* Fix security plugin

* Misc fixes

* bump

* Fix licensePreRoutingFactory

* Fix failing integration tests

* Remove unnecessary test change

* Remove hapi-latest package

* fx

* Various cleanup

* Fix race condition in oppsy events

* Use elastic/good fork

* Fix boom.wrap and hapi-latest changes

* Simplify LegacyLoggingServer updates

* package.json cleanup + test fix

* yarn.lock cleanup

* Change good tag

* Fixes

* Change return err -> throw err in routes

* Fix await returns

* Fix new load_data test

* Make cookie security flags consistent

* tmp doc

* Fix types

* Fix tests

* Upgrade canvas plugin

* Move good package to published @elastic/good one

* Fix SO test

* Fix logging reloading

* Update APM apis

* Fix error logging

* Fix logging test

* Convert spaces plugin

* Add validation error shim

* Remove 7.0 release notes

* Await renderApp

* Fix ccr routes

* Prevent header popovers from scrolling with page content (#23850)

* Fix spaces test

* new yarn.lock-s

* Fix spaces tests

* Remove h2o2-latest

* Fix @types/hapi

* Upgrade InfraOps plugin

* Fix package.json

* Add back isSameSite: false

* Upgrade beats_management plugin

* Update snapshot

* Fix InfraOps

* Upgrade kql_telemetry

* Merge upstream/master

* Upgrade apm and ml

* Put snapshot test back

* Fx beats

* Upgrade rollups

* Update boom usages in new plugins
2018-10-25 16:01:12 -05:00
Pete Harverson
e900879cc3
[ML] Fixes labels in datafeed time range selector modal (#24489) 2018-10-24 15:50:56 +01:00