* [ML] Re-enabling file upload telemetry
* small refactor
* removing exported function
* removing commented out code
* removing commented out include
* cleaning up types
* [ML] Client side cut over
* updating paths and commented code
* changes based on review
* disabling telemetry tests
* fixing start job stylesheets
* fixing everything that is broken
* fixing types and ml icon order
* using icon constant
* [ML] Adding shared services to ml setup contract
* adding data recognizer
* typescripting js client
* adding results service
* code clean up
* adding generic ml index search
* making cloud optional
* [ML] Add support for date_nanos time field in anomaly job wizard
* [ML] Edits following review
* [ML] Add functional test for creating job off date_nanos data
* [ML] Fixing licensing after server NP cutover
* client side refactor
* management refactor
* moving management license check
* adding server side license class
* added common license class
* client sides using license class
* updating server api guards
* fix privileges tests
* updating translations
* fixing disabled tabs on basic license
* refactoring client side license checks
* setting license initialized flag
* fixing license check on index data viz
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
* remove obsolete legacy server deps
* licensePreRoutingFactory uses licensing plugin rather than legacy xpack
* move schemas to dir in routes
* use NP license check method for license check
* store license data in plugin for passing to check
* create server plugin files in NP plugin dir
* remove dependency on legacy xpack plugin
* add sample data links first step
* move all server dirs from legacy to np dir
* fix requiredPlugin spaces name and update import routes
* delete unnecessary files and add sample data links
* update license and privilege check tests
* add routeInit types
* wip: pull out all server dependencies
* create new platform plugin
* shim newPlatform plugin into legacy init
* update server tests for migration refactor
* cleanup interfaces
* Only add ML links for sample data sets if full license from - 38120
* update test and fix typescript errors
* Remove unused types
* [ML] Fixes loading of Single Metric Viewer if partition field is text
* [ML] Fix failing test and edit translations for cardinality check
* [ML] Edits to fields_service.js following review
* [ML] Sample ML jobs: Updates to make anomalies clearer
Minor updates to web logs data and ML job
definitions to make anomalous activity more
distinct.
* [ML] Sample ML jobs: Further updates to jobs and data
Adding missing files from previous commit and
changing job naming (removing weblog_* prefix etc.)
* [ML] Resolving PR review comments.
* [ML] Improve anomalies in Kibana sample data
Minor fixes:
- Rename job group to `kibana_sample_data` throughout
- Fix index pattern link in `sample_data_sets.ts`
- Fix log.json.gz count test
- Adds a confirm modal to the Start button of the data frame jobs list. image
- Adds a snippet with a warning about possibly cluster load to wizard. image
- Uses _stop?force=true to stop jobs.
- Adds a check to disable the start button in the jobs list if the job is a completed batch job.
* [ML] Add links to ML jobs to the Kibana sample dataset view data menu
* [ML] Remove redundant TODO comment from check_module.js
* [ML] Edits to sample data ML links following review
* [ML] Edits to comments about use of reject in check_module.js
* Initial job configuration files for Kibana Sample Ecommerce data recognizer
* Initial configuration files for Kibana Sample Data for web logs
* Initial configuration for Kibana Sample Data recognizer for Flights data
* Correcting ID, title and description of Flights sample data recognizer
* Adding Custom URLs to link to Discover from Kibana Sample Dataset ML jobs
* Adding custom URLs to Ecommerce Data Recognizer results
* Adding new custom URL to Kibana Sample Ecommerce data recognizer
* Adding custom URL for Sample Flights dataset dashboard
* Adding custom URLs to sample Web Logs data recogniser config
* Fixing single quotes and url names in custom URLs for flights and ecommerce datasets
* Adding Kibana Sample Data modules to list of modules that need testing
* Renaming Kibana Sample Data recognizer modules, fixing IDs and adding modules to test framework
* Fixing data recognizer titles to sentence case
* Fixing sentence case issues in Kibana eCommerce sample data
* Fixing sentence case in Kibana sample flights data
* Fixing sentence case in Kibana sample weblogs data
* Changing job configuration for Kibana sample flights data
* Making changes to job descriptions
* Changing job configurations and renaming files for Kibana eCommerce sample data recognizer
* Removing mean bytes by file extension from Web Logs sample jobs and adding three new configurations
* Fixing minor issues with Kibana Sample Data recognizer configurations
* Adding jobs to job groups
* Changing job group names to use underscores instead of hyphens
* Adding filtering to Kibana Sample Data dashboards
* Fixing linting issues in JSON configs
* Finetuning data recognizer module descriptions
* Renaming Kibana Sample Data flights jobs
* Removing plural from Kibana Sample Flights dataset description
* Modifying custom URLs for Kibana Flights jobs
* Fixing custom URLs for Kibana Sample eCommerce dataset jobs
* Adding new influencers and fixing Discover custom URLs for Kibana Weblogs sample dataset job
* Fixing space before colon issue in job configurations
* Converting tabs to spaces
* Fixing description of detector in sample dataset eCommerce job
* Initial code to force collectors to indicate when they are ready
* Add and fix tests
* Remove debug
* Add ready check in api call
* Fix prettier complaints
* Return 503 if not all collectors are ready
* PR feedback
* Add retry logic for usage collection in the reporting tests
* Fix incorrect boomify usage
* Fix more issues with the tests
* Just add debug I guess
* More debug
* Try and handle this exception
* Try and make the tests more defensive and remove console logs
* Retry logic here too
* Debug for the reporting tests failure
* I don't like this, but lets see if it works
* Move the retry logic into the collector set directly
* Add support for this new collector
* Localize this
* This shouldn't be static on the class, but rather static for the entire runtime
Revamps the job creation step of the data frame wizard.
- Create & start buttons now have additional descriptions
- Adds a "copy to clipboard" button to copy a Kibana Dev Console statement of the job creation command to the clipboard
- Adds a progress bar once the job is started
- Adds a card which links to Kibana Discover if an index pattern was created for the job
- Hides the wizard's Previous button once the job is successfully created.
* wip: create react jobSelector wrapper + main component
* Load jobs and select first if none selected via url
* wip: create flyout content
* Add endpoint for fetching jobs with timerange for table
* display selected ids in flyout
* Add custom table allowing external selection
* add groups table in groups tab
* Get groups and jobs in initial api call
* add ability to select groups
* Hook jobSelector into SingleMetricView
* Show selected group badges with count
* Organize jobSelector component directories
* Move timerange logic to server
* Move group color selection to utils
* hide/show badges and add localization
* fetch jobs in route to enable selector jobid validation
* upate globalState on setting jobId in SingleMetricView
* Add pager options.Retain search query on tab change
* Ensure gantBar timeRanges correct
* cleanup old commented code. tweak flyout header/footer style
* running gantt bar and remove unnecessary api call
* GanttBar running style. Pass timezone to server.
* Running gantt bar limited to timerange. Clean up comments.
* Refactor jobSelector endpoint to use fullJobs
* Retain group selection in globalState
* Recalculate ganttbars on resize
* add test for JobSelectorTable
enable security on file dataviz and import (ML plugin)
update unit tests
add api test coverage for security in basic
move audit logging to standard+ license level
* [@kbn/expect] "fork" expect.js into repo
* [eslint] autofix references to expect.js
* [tslint] autofix all expect.js imports
* now that expect.js is in strict mode, avoid reassigning fn.length
* wip: add autocomplete kueryFilterBar.update endpoints with query.
* Create indexPattern on job change. Value is autocompleted
* Mask unrelated swimlanes on filter apply
* Only show relevant viewBy swimlane
* reload viewBy swimlane when view By change is relevant to filter
* handle various query types when building filteredFields list
* remove deprecated code from old filter bar
* Show error callout when invalid filter syntax
* remove dependency on deprecated import var
* remove all left over apm dependencies
* persist filter on refresh
* Create initial filter bar placeholder dynamically
* add description text when filter active
* switch to relevant viewBy fieldname on filter
* recalculate placeholder on job change
* Create tests for all components
* View by dropdown only contains relevant fields for filter
* fix localization message noInfluencers component
* Move indexPattern to state and viewBy option filtering to utils
* remove unnecessary setState when setting viewBy for filter
* Use preloaded jobs from mlJobService to get influencers
* Only filter viewByOptions if valid field
* use kql and move to components dir
* move filter bar to public/components/
- This adds more thorough error handling to retrieving annotations from the server to avoid breaking Anomaly Explorer when there's a problem retrieving annotations. The UI will silently fail and hide the annotations table if for whatever reasons annotations cannot be retrieved for the selected jobs (this could be permissions or setup related if e.g. the annotations feature index/aliases are not present). This is now the same behavior like in Single Metric Viewer. The browser console will still output an error message. This fixes an error scenario where Anomaly Explorer would fail to load if a user upgraded to 6.6 because the backend task wasn't run yet to create the annotation index/aliases.
- Disables annotations feature in Single Metric Viewer if annotations fail to load.
- Use componentDidUpdate and an update check instead of deprecated componentWillUpdate for updating the annotations table in jobs management.
The way job audit messages were fetched didn't retrieve the expected results if there were deleted jobs with messages still present for these jobs.
This fix allows to specify a list of job IDs to filter the audit messages on. For the jobs list UI, the currently existing job IDs will be passed on to ignore messages from deleted jobs.
This adds tests to `server/models/annotation_service`. The tests include a check if the `.key` attribute of an annotation is properly removed from an annotation before indexing it.
The UI adds a `key` attribute to annotation objects to store the letter used for labels in the chart and tables. When editing and saving an annotation that `key` could end up being saved to the annotations index. This isn't necesseary since the `key` attribute is just a dynamic label used within the UI. This fixes it by deleting an eventual `key` attribute from the annotations object before saving it to the index.
* [ML] Initial commit for auditbeat hosts ECS
Rename fields for ECS
Rework dashboards due to bwc
* [ML] Further auditbeat tidy up and consistency changes
Custom urls should link to saved search, not discover
Ensure savedSearchId is used for visualizations
Ensure filter terms are consistent
TODO Decide if we should rename to auditd module
TODO Fix for new saved object format
* [ML] Refinements for auditbeat host module
Remove duplicated title from visState
Shrink panel heights in row 1
* [ML] Refinements to auditbeat module
Update module name from auditd to auditbeat
Add useMargins true for dashboards
Add filter to custom url for
exists auditd.data.syscall
not exists container.runtime
event.module: auditd
* [ML] Initial commit for auditbeat_process_docker_ecs
Update for ECS using
container.name (instead of container.id)
container.runtime: docker
process.executable
event.module: auditd
auditd.data.syscall exists
TODOs
Use auditd.message_type: syscall (instead of auditd.data.syscall)
Possibly combine with auditbeat hosts saved objects (depending on host.name being shared)
Possibly combine to single dashboard
Test against live auditbeat data collection
With security enabled, the internal user wouldn't have enough permissions to run the integrity check. This changes the check to use the currently logged in user. Also fixes some typos in messages.
* [ML] Adding index migration warnings
* small refactor
* correctlng comment
* adding upgrade service to manage upgradeInProgress state
* removing missing function
* [ML] Initial commit for apache ecs module
* [ML] Update apache2 module for ECS
Rename following fields
event.module:apache
event.dataset:access
source.address
url.original
http.response.status_code
source.geo.location
Rationalise to only use one set of kibana saved objects for all http web access logs
Rename files from apache
Combined URL explorer into Count explorer dashboard as there was a lot of duplication
Add filter to custom url
Rename custom urls to Investigate Source IP and Status Code
Add chart to show overall event rate split by event.module - can tell if multiple datasets are included
Increase limit for top source ips from 5 to 50
Add created_by to custom setting for telemetry
Rename jobs and saved objects to include ecs tag
Tested side by side against v6 jobs
* [ML] Rename apache files from hyphen to underscores
* [ML] Further apache renames
Also change custom URLs to lower case to match "View series"
Change created_by to ml-module-apache-access
* [ML] Initial commit of nginx ml module
* [ML] Rename dashboard to generic explorer
* [ML] Further refinement for apache
Rename http_status_code to status_code_rate
Update custom url to use filters instead of lucene query bar
* [ML] Convert apache module to nginx
Copy files, keeping nginx logo
Multiple renames to nginx
* [ML] Make chart legend visible by default
* Add new references attribute to saved objects
* Add dual support for dashboard export API
* Use new relationships API supporting legacy relationships extraction
* Code cleanup
* Fix style and CI error
* Add missing spaces test for findRelationships
* Convert collect_references_deep to typescript
* Add missing trailing commas
* Fix broken test by making saved object API consistently return references
* Fix broken api integration tests
* Add comment about the two TS types for saved object
* Only return title from the attributes returned in findRelationships
* Fix broken test
* Add missing security tests
* Drop filterTypes support
* Implement references to search, dashboard, visualization, graph
* Add index pattern migration to dashboards
* Add references mapping to dashboard mppings.json
* Remove findRelationships from repository and into it's own function / file
* Apply PR feedback pt1
* Fix some failing tests
* Remove error throwing in migrations
* Add references to edit saved object screen
* Pass types to findRelationships
* [ftr] restore snapshots from master, rely on migrations to add references
* [security] remove `find_relationships` action
* remove data set modifications
* [security/savedObjectsClient] remove _getAuthorizedTypes method
* fix security & spaces tests to consider references and migrationVersion
* Add space id prefixes to es_archiver/saved_objects/spaces/data.json
* Rename referenced attributes to have a suffix of RefName
* Fix length check in scenario references doesn't exist
* Add test for inject references to not be called when references array is empty or missing
* some code cleanup
* Make migrations run on machine learning data files, fix rollup filterPath for savedSearchRefName
* fix broken test
* Fix collector.js to include references in elasticsearch response
* code cleanup pt2
* add some more tests
* fix broken tests
* updated documentation on referencedBy option for saved object client find function
* Move visualization migrations into kibana plugin
* Update docs with better description on references
* Apply PR feedback
* Fix merge
* fix tests I broke adressing PR feedback
* PR feedback pt2
* [dashboard+gis] remove dark mode options
* [reporting/extract] restore fixtures
* remove mentions of old `.theme-dark` class
* import panel styles from panel/_index.scss
* Prevent docCount fetch and remove sidebar if no timeField set.
* Don't show metrics section if no metrics cards
* Add parens to conditional statement as per styleguide
* Don't create docCount card if not timeseries based
Adds a check if all index patterns necessary to clone a job are available for the current user. The check will only allow cloning of a job if the user has the right permissions and can still access the indexPattern the job was created for. An indexPattern could either have been deleted since the the job was created or the current user doesn't have the required permissions to access the indexPattern. The "Clone Job" action in the jobs list action menu will be greyed out and unavailable for that job.
* [ML] List all annotations in jobs list annotations table even outside analysis time range.
* [ML] Adjust the links time range if annotation is outside analysis time range.
* Move cardinality success check to utils
* enableModelPlot checkbox base added
* Run cardinality check on add/update fields
* Handle changes made via json
* only run cardinality check if model plot enabled
* Handle model plot enabled via EditJSON tab
* show message on cardinality check error
* multi-metric + pop: show message on cardinality check error
* add test for callout component
* Fix flexitem overflow in IE11
* [ML] File datavisualizer initial commit (#22828)
* [ML] File datavisualizer initial commit
* removing mocked data and adding initial stats
* adding card styling to fields
* Revert "". accidentally added with no commit message
This reverts commit d762d20b706e6a770e631f863b9e7d8879bb7ee6.
* adding date type to timestamp field
* renaming FileStats to FieldsStats
* code clean up
* changes based on review
* changes to error handling
* [ML] Adding file datavisualizer overrides (#23194)
* [ML] Adding file datavisualizer overrides
* improvements to overrides
* removing comment
* small refactor
* removing accidentally added file
* updates based on review
* fixing broken test
* adding missing grok pattern override
* fixing test
* [ML] Refactoring override option lists (#23424)
* [ML] Refactoring override option lists
* moving lists out of class
* updating test snapshot
* [ML] Fixing field editing (#23500)
* [ML] Changes to timestamp formats (#23498)
* [ML] Changes to timestamp formats
* updating test snapshot
* [ML] Allow Datavisualizer use on basic license (#23748)
* [ML] Allow ML use on basic license
* removing timeout change
* adding permission checks
* updating tests
* removing unnecessary checks
* [ML] Adds new page for choosing file or index based data visualizer (#23763)
* [ML] Adding license check to datavisualizer landing page (#23809)
* [ML] Adding license check to datavisualizer landing page
* removing comments
* updating redirect to landing page
* [ML] Adding ability to upload data to elasticsearch from datavisualizer (#24042)
* [ML] Initial work for delimited file upload
* adding results links cards
* adding nav menu
* removing accidental debugger
* initial work for importing semi structured text
* using ingest pipeline for import
* adding json importer and better error reporting
* better progress steps
* time range added to results links
* first import only creates index and pipeline
* adding status constants
* using status constants
* adding explanation comment
* updating yarn.lock
* changes based on review
* fixing space
* fixing space again, stort it out git
* removing oversized background container causing constant scrollbar
* [ML] Adding basic license check when loading privileges (#24173)
* [ML] Adding basic license check
* missing import
* [ML] Adds an About panel to the file data visualizer landing page (#24260)
* [ML] Adds an About panel to the file data visualizer landing page
* [ML] Remove unnecessary style from file data visualizer scss
* [ML] Adding better error reporting for reading and importing data (#24269)
* [ML] Adding better error reporting for reading and importing data
* changes to endpoint errors
* displaying errors
* step logic refactor
* removing log statements
* [ML] Switch file data visualizer to use Papa Parse for CSV parsing (#24329)
* [ML] Fixes layout of Data Visualizer selector page for IE (#24387)
* [ML] Adding ability to override various settings when importing data (#24346)
* [ML] Adding ability to override various settings when importing data
* second commit with most of the outstanding code
* improving index pattern name validation
* better index pattern matching
* adding comments
* adding empty index pattern check
* changes based on review
* fixing test
- Fixes the job validation for the lower bound of the model memory limit. Previously the check was against zero, now it's again less than 1MB, which is the same as the backend expects.
- If the user entered model memory limit is less than half the value of the estimated model memory limit, a warning type message gets triggered. If the user entered model memory limit is more than half the value but less then the actual value of the estimated model memory limit, then the already existing info type message is shown. The unit tests have been updated to reflect that behavior.
* [ML] Fixing issue with incorrect timezones in jobs list
* refactoring min and max calculation
* changes based on review
* changing TimeStamp to Timestamp
This fixes bucket span estimation for the machine learning admin user when security is enabled. The original issue was that the machine learning admin by default doesn't have the necessary permissions to call cluster.getSettings to get the settings search.max_buckets. This is now fixed by using callWithInternalUser() and additional permissions checks.
* [ML] [WIP] Adding group selector to jobs management
* adding group name validation
* removing comment
* adding keyboard events
* moving new group input to its own component
* changes based on review
* adding tooltip
* adding better error reporting
- Fixes the bucket span estimator when median is selected as a detector function. agg.type.name is median and therefor not usable for an Elasticsearch aggregation. agg.type.dslName is percentile and is the correct mapping. .dslName is also used for the aggregations used for the preview charts.
- 7.0 will introduce a search.max_buckets setting which defaults to 10000. This could lead to failing bucket estimations because the values used for creating the required aggregations could result in more buckets. This PR fixes it by taking search.max_buckets into account when calculating the time range used for the bucket estimation. (Since 6.2 that setting is available so backporting this to current unreleased minor releases 6.4 and 6.5)
Improvements to job validation messages:
- Changed cardinality_over_field_low "might be less suitable" to "might not be suitable".
- Fixed documentation URLs.
- Extended categorization_filters_invalid message + documentation URL.
- If a job configuration's datafeed contains script_fields, they are now available in the field dropdowns of the advanced wizard's detector modal.
- Additionally, job validation now considers script_fields and doesn't report them as non aggregatable anymore.
This provides more helpful texts for job validation success messages. Previously only a list of checks was shown to the user without any further explanation. This PR addresses this issue in the following way:
- At the bottom of the modal an introductory brief text about job validation including a link to documentation is inserted.
- The success messages in the list now provide a more helpful text including (where applicable) a deep link to documentation
- The messages now support a richer Callout layout including a header and additional text.
While the output in the UI was fine, certain job configurations containing non-aggregatable fields could trigger errors on the Elasticsearch side.
This PR fixes it by adding an additional query for fieldCaps to check first which fields are actually aggregatable.
For categorization jobs, job validation would report that mlcategory isn't an aggregatable field. This fix checks the job configuration and only reports the error if the job config isn't using categorization_field_name and the detector field isn't set to mlcategory.
* [ML] Adding close jobs menu item
* changing icon
* updates based on review
* adding extra check for close failure
* adding extra guard against missing response
* add _bulk_create rest endpoint
* document bulk_create api
* provide 409 status code when conflicts exist
* add overwrite and version to documenation
* clean up assert statements and 2 places where bulkCreate is getting used with new output
* properly stub bulkCreate return
* remove version of documenation example
* [ML] Adding jobs stats bar to top of jobs list
* unsetting update function
* changing node variable name
* small refactor
* using props copy of `updateJobStats`
* adding proptypes
* adding fixed height
* [ML] Adding filter bar to jobs list
* fixing page index when filtering
* refreshing job selection after actions have happened
* adding job counts to groups
* catching multi-select start datafeed errors
* style tweaks
* more style tweaks
* changes based on review
* refactoring search logic
* [ML] Removing angular from services
* removing commented out code
* fix include path for tests
* adding flag to initPromise
* moving job service for jest tests to pass
* moving initPromise to its own file
This adds an additional check and corresponding error message to the validation of the model memory limit. If the parsing of the memory model limit results in 0 bytes the message will be triggered. This behaviour catches both correctly formatted but invalid strings (like 0mb) and invalid strings (like asdf).
Updates the behaviour of the job validation's model memory limit checks: The success messages success_mml gets only returned if the checks were actually run. This omits the success message for example in the single metric and population wizard.
