Adds Capacity Estimation to the Task Manager Health Endpoint.
Below is a diagram depicting what information we use to estimate the varying capacity variables.
Please use the user facing docs to understand how it fits together. If the docs aren't clear enough - make a review comment and I'll clarify in the docs.
* bump @elastic/elasticsearch to canary.7
* address errors in core
* address errors in data plugin
* address errors in Alerting team plugins
* remove outdated messages in Lens
* remove unnecessary comments in ML
* address errors in Observability plugin
* address errors in reporting plugin
* address errors in Rule registry plugin
* fix errors in Security plugins
* fix errors in ES-UI plugin
* remove unnecessary union.
* update core tests
* fix kbn-es-archiver
* update to canary 8
* bump to v9
* use new typings
* fix new errors in core
* fix errors in core typeings
* fix type errors in data plugin
* fix type errors in telemetray plugin
* fix data plugin tests
* fix search examples type error
* fix errors in discover plugin
* fix errors in index_pattern_management
* fix type errors in vis_type_*
* fix errors in typings/elasticsearch
* fix type errors in actions plugin
* fix type errors in alerting and apm plugins
* fix type errors in canvas and cases
* fix errors in event_log
* fix type errors in ILM and ingest_pipelines
* fix errors in lens plugin
* fix errors in lists plugin
* fix errors in logstash
* fix errors in metrics_entities
* fix errors in o11y
* fix errors in watcher
* fix errors in uptime
* fix errors in upgrade_assistant
* fix errors in task_manager
* fix errors in stack_alerts
* fix errors in security_solution
* fix errors in rule_registry
* fix errors in snapshot_restore
* fix remaining errors
* fix search intergration tests
* adjust assetion
* bump version to canary.10
* adapt code to new naming schema
* use mapping types provided by the client library
* Revert "adjust assetion"
This reverts commit 19b8fe0464.
* fix so intergration tests
* fix http integration tests
* bump version to canary 11
* fix login test
* fix http integration test
* fix apm test
* update docs
* fixing some ml types
* fix new errors in data plugin
* fix new errors in alerting plugin
* fix new errors in lists plugin
* fix new errors in reporting
* fix or mute errors in rule_registry plugin
* more ML type fixes
* bump to canary 12
* fix errors after merge conflict
* additional ML fixes
* bump to canary 13
* fix errors in apm plugin
* fix errors in fleet plugin
* fix errors in infra plugin
* fix errors in monitoring plugin
* fix errors in osquery plugin
* fix errors in security solution plugins
* fix errors in transform plugin
* Update type imports for ES
* fix errors in x-pack plugins
* fix errors in tests
* update docs
* fix errors in x-pack/test
* update error description
* fix errors after master merge
* update comment in infra plugin
* fix new errors on xpack tests/
Co-authored-by: James Gowdy <jgowdy@elastic.co>
Co-authored-by: Dario Gieselaar <dario.gieselaar@elastic.co>
When something causes an exception in `TaskRunner.markTaskAsRunning()` its execution fails, but this happens before we update the SO, which means that this failure does not count towards the `attempts` on the task. Task Manager will continue to try running this task for ever.
This PR increments the `attempts` when a failure occurs during `TaskRunner.markTaskAsRunning()` to ensure such a task doesn't continue to run to infinity.
Note that this fix will not affect `scheduled` tasks, as they are designed to _ignore_ their `attempts` and run for ever. In such a case this task will continue to consume Task Manager resources until canceled, but these failures will be logged and could be identified when needed.
* chore(NA): create new x-pack cigroups and rebalancing them all
* chore(NA): better cigroups balancing
* chore(NA): push rollup tests back into ciGroup1
* chore(NA): move some functional ml tests from cigroup3 into cigroup13
* chore(NA): move some more tests into ciGroup13
* chore(NA): use a single top level describe at x-pack/test/functional/apps/ml
* chore(NA): move settings into ciGroup13
* temporary test for es snapshots env
* Revert "temporary test for es snapshots env"
This reverts commit 789ebe7b9c.
* docs(NA): add missing documentation on the function tests describe split
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
This PR Introduces a `pollingDelay` which is applied to the polling interval whenever the average percentage of tasks experiencing a version conflict is higher than a preconfigured threshold (default to 80%).
Adds additional polling stats to Task Manager monitoring:
- **duration**: Running average of polling duration measuring the time from the scheduled polling cycle start until all claimed tasks are marked as running
- **claim_conflicts**: Running average of number of version clashes caused by the markAvailableTasksAsClaimed stage of the polling cycle
- **claim_mismatches**: Running average of mismatch between the number of tasks updated by the markAvailableTasksAsClaimed stage of the polling cycle and the number of docs found by the sweepForClaimedTasks stage
- **load** - Running average of the percentage of workers in use at the end of each polling cycle.
Added the following values to the Polling stats:
- **NoAvailableWorkers**: This tells us when a polling cycle resulted in no tasks being claimed due to there being no available workers
- **RunningAtCapacity**: This tells us when a polling cycle resulted in tasks being claimed at 100% capacity of the available workers
- **Failed**: This tells us when the poller failed to claim
This addresses a bug in Task Manager in the task timeout behaviour. When a recurring task's `retryAt` field is set (which happens at task run), it is currently scheduled to the task definition's `timeout` value, but the original intention was for these tasks to retry on their next scheduled run (originally identified as part of https://github.com/elastic/kibana/issues/39349).
In this PR we ensure recurring task retries are scheduled according to their recurring schedule, rather than the default `timeout` of the task type.
This PR adds an an internal monitoring mechanism in Task Manager which keep track of a variety of metrics and a health api endpoint which makes the monitored statistics accessible.
* wip
* Adding updateFieldsAndMarkAsFailed function
* Updating UBQ
* Only updating retryAt if marking as claiming
* Updating query
* Updating query to only fail one time tasks that have exceeded max attempts
* Fixing tests
* Fixing tests
* Handling claiming tasks by id
* Removing unused function
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Fixes flaky tests in Task Manager and Alerting.
The fix in #73244 was correct, but it missed an edge case which causes the already running task to be rescheduled over and over.
This prevents that edge case which was effecting both TM in general and Alerting specifically.
This PR addresses two issues which caused several tests to be flaky in TM.
When `runNow` was introduced to TM we added a pinned query which returned specific tasks by ID.
This query does not have the filter applied to it which causes task to return when they're already marked as `running` but we didn't address these correctly which caused flakyness in the tests.
This didn't cause a broken beahviour, but it did cause beahviour that was hard to reason about - we now address them correctly.
It seems that sometimes, especially if the ES queue is overworked, it can take some time for the update to the underlying task to be visible (we don't user `refresh:true` on purpose), so adding a wait for the index to refresh to make sure the task is updated in time for the next stage of the test.
Completes the migration of all Alerting Services plugins onto the Kibana Platform
It includes:
1. Actions plugin
2. Alerting plugin
3. Task Manager plugin
4. Triggers UI plugin
And touches the Uptime and Siem plugins as their use of the Task Manager relied on some of the legacy lifecycle to work (registering AlertTypes and Telemetry tasks after the Start stage has already began). The fix was simply to moves these registrations to the Setup stage.
Enables access to the Alert State, which allows us to see which current Alert Instances are active.
This includes:
1. Addition of a `get` api on Task Manager
2. Typing and validation on Serialisation & Deserialisation of the State of an Alert's underlying Task
3. Addition of the `getAlertState` api on AlertsClient
As of Elasticsearch 8.0.0 it will no longer be possible to use the _id field on documents.
This PR removes the usage that Task Manager makes of this field and switches to pinned queries to achieve a similar effect.
Migrates the existing TaskManager plugin from Legacy to Kibana Platform.
We retain the Legacy API to prevent a breaking change, but under the hood, the legacy plugin is now using the Kibana Platform plugin.
Another reason we retain the Legacy plugin to support several features that the Platform team has yet to migrate to Kibana Platform (mapping, SO schema and migrations).
This moves the interval field under a generic schedule object field in preparation for the introduction of richer scheduling options (such as cron).
It includes a migration for existing tasks, and we've ensured no existing Task Type Definitions exist in Kibana that rely on Interval.
This includes support for the deprecated interval field (which gets mapped to schedule) but that support will be removed in 8.0.0, as it's a breaking change.
Adds a `runNow` api to Task Manager, allowing us to force the refresh of a recurring task.
This PR includes a couple of sustainability changes as well as the feature itself.
1. **Declarative query composition.** At the moment the queries in the TaskStore are huge JSON objects that are hard to maintain and understand. This PR introduces a pattern where the different parts of the query are composed out of type-checked functions, making it easier to maintain and to construct dynamically as needs change. _This was included in this PR as the **markAvailableTasksAsClaimed** query needs different query clauses depending on whether there are specific Tasks we wish to claim first.
2. **Refactoring of the Task Poller** As the `runNow` api is introduced we find Task Manager's lifecycle in a weird state where it has both a _pull_ model, where timeouts & callbacks interact without having to responsd to any external requests, and a _push_ model where requests are made to the new `runNow` api. Balancing these two proved error prone, hard to maintain and had the potential of _lossy_ behaviour where requests are dropped accidentally. To address this TaskPoller has been refactored using Rxjs observables, remodelling the existing _pull_ mechanism as a _push_ mechanism so Task Manager can _respond_ to both _polling_ calls and _runNow_ in a similar fashion.
And ofcourse the main feature of this PR:
3. **runNow api** An api on TaskManager that takes a _task ID_ and attempts to run the task. The call returns a promise which resolves with a result which notifies the caller when the task has either completed successfully, or result in an error.
This PR adds a test that ensures Task Manager is capable of picking up new tasks in parallel to a long running tasks that might otherwise hold up task execution.
This doesn't add functionality - just a missing test case.
* Additional validation for elasticsearch username
If "elastic" user is set in config:
* In dev mode, throws an error
* In prod mode, logs a deprecation warning
* Fix user for functional tests
* Revert last two commits
Revert "Fix user for functional tests" and
"Fix user for plugin functional tests in Jenkinsfile"
* Change elasticsearch creds for test server
Now uses "kibana" user instead of "elastic" user
* Fix plugin API functional tests
* Fix PKI API integration test
* Change log messages, now conditional on `dist: false` not `dev: true`
Adds an ensureScheduling api to Task Manager which allow safer rescheduling of existing tasks by handling the case where a Task with a known ID is scheduled and clashes with an existing schedule of that same task.
Task Manager now uses updateByQuery to identify tasks it needs to pick up and claim ownership of them in a single call to Elasticsearch.
This reduces the version conflicts that often plague Task Manager when running multiple Kibana against a single ES cluster.
This has already shown small perf gains locally, and lays the ground work to a second phase (bulk processing the markTaskAsRunning step) which should allow to paralelise the task running feature further.
* Allow mtask definitions to overwrite default setting maxAttemps
* Leverage scheduledAt from task manager
* Treat maxAttempts like attempts and not retries
* Add support for second intervals
* Min 1 attempt
* Reverse relying on scheduledAt
* Add new startedAt attribute in task manager that keeps track when task started running
* Don't extend runAt when claiming a task
* Remove startedAt from state
* Attempt trying to define custom getBackpressureDelay function
* Pass error object to getBackpressureDelay
* Cleanup processResultForRecurringTask code
* Add backpressure to timed out tasks
* Change default timeout backpressure calculation
* getBackpressureDelay to return seconds instead of milliseconds
* Add comment for task store query
* Compress query
* Revert alert / actions specific code
* Add more interval tests
* Fix failing jest tests
* Fix test
* Add more unit tests
* Fix integration tests
* Fix sorting of tasks to process
* WIP
* Always provide error when getBackpressureDelay is called
* Rename getBackpressureDelay to getRetryDelay
* retryAt to be calculated from timeout time by default
* Remove invalid test
* Add unit tests
* Consider timeout before scheduling a retryAt
* Remove backpressure terminology
* Remove support for 0 based intervals and timeouts
* Apply PR feedback
* Fix last place using Math.abs
* Modify migrations to allow running a script when converting an index to an alias
* Convert task manager to use saved objects
* Fix broken test
* Fix broken tests pt1
* Remove index from task manager config schema
* Accept platform changes
* PR feedback
* Apply PR feedback
* Apply PR feedback pt2
* Apply PR feedback pt3
* Apply PR feedback pt4
* Fix feedback pt3
* Rename RawSavedObjectDoc to SavedObjectsRawDoc
* [@kbn/expect] "fork" expect.js into repo
* [eslint] autofix references to expect.js
* [tslint] autofix all expect.js imports
* now that expect.js is in strict mode, avoid reassigning fn.length
* Core task manager (#23555)
* Initial stab at core task manager logic
* Update task_manager readme
* Update task manager readme
* Add cancelable helper package, OSS, and general purpose, but
will be useful for writing cancelable x-pack tasks.
* Make the cancellable package promise-compliant
* Update task manager readme w/ reference to cancellable
* Change pool from lazy to eager, add support for sub-pools per task type.
* Move cancellable to task_manager, and typescriptify it.
* Working proof of concept for task manager. Still have lots to do:
clean up, tests, comments, validations, assertions, etc.
* Add pagination support to task manager fetch
* Move task manager to OSS
* Remove task manager reference from x-pack
* Make task_manager a valid core plugin
* Modify how task resource allocation works
* Remove the special case 'max' workers occupied value
* Remove x-pack package.json changes
* Make taskDefinitions a part of uiExports
* Make task docs saved-object compliant.
* Add kbnServer to the task context.
* Allow tasks to have a void / undefined return type
* revert x-pack change
* move cancellable to src/utils
* move to src/server
* use afterPluginsInit hook
* task_manager.ts rename
* add a wrapper with a setClient method
* Add tests for task runner
* Break task_pool into smaller, testable pieces
* return raw task doc fields for calling code
* remove todo comment
* helper module for default client
- setClient takes a callback fn
* fix misidentified field name
* fix rest args warning
* flatten task_pool
* remove cancellable
* return raw task doc fields for calling code
* remove todo comment
* helper module for default client
- setClient takes a callback fn
* fix rest args warning
* typescript fixes
* roll back setClient takes a callback
* createTaskRunner returns an object with run/cancel functions
* Test task store, tweak task poller tests
* Rename num_workers to max_workers
* Tweak task pool method names
* Fix cancellable test, and ts errors
* Rename doc to _doc
* Fix the doc type in the task manager index mappings
* Make task costs configurable in kibana.yml
* fix a tslint warning
* TaskManager.afterPluginsInit replaces circuitous stuff
* addMiddleware, implement beforeSave
* wip
* comment
* run context stuff
* pretty fix
* comments
* lint/pretty fixes
* safeties in case they don't define both beforeRun and beforeSave
* task runner test constructor opts fix
* Add task_manager integration tests
* FIx readme mistakes, fix schedule signature
* Fix integration tests
* Add blurb about middleware
* make a libs directory for the small things
* test for extractTaskDefinitions
* unit test for middleware
* Comment, refactor, and test task_manager
* Touch up comments
* Make task store throw error if given an unknown task type
* Fix sample task definition in readme
* Make task manager toggle on / off based on Elasticsearch connectivity
* Prevent task manager from crashing Kibana if
elasticsearch plugin is disabled
* Fix task manager signature
* Move the task manager into the task manager mixin,
fix tests. It's an uber integration object anyway, so
it seemed to make sense to keep it in one place.
* Fix task manager integration tests
* Update the task manager index creation to use auto_expand_replicas
* Fix task manager integration tests
* Fix task manager integration tests
* Fix the overzealous deletion of .kibana during tests
* Core task manager
* Allow hardcoded ID passed in
* comments
* don't deconstruct type and task for payload
* remove uiExport support
* move task_manager to x-pack/plugins
* expose as client on the plugin namespace
* fix tests
* move plugin api integration tests
* roll back readme sample task definition example
* fix sample plugin link
* server.taskManager
* sanitizeTaskDefinitions
* fix integration tests
* sanitize rename
* assert unintialized and check for duplicates in registerTaskDefinitions
* Remove unnecessary defaults from task_manager.test.ts
* Remove task manager from OSS
* Remove task manager from src
* Tidy up task_manager unit tests around elasticsearch status changes
* Integration test for specifying ID in scheduling
* Task_manager init -> preInit to allow other plugins to consume
taskManager in their init.
* Remove task manager integration tests from OSS
* spelling
* readme fix
* fix test code impacted by hapi upgrade
* Task Manager index creation changes (#24542)
* Uses putTemplate for task manager mappings and index settings.
* Removes create option.
* Opposite day, the test catches up to the code changes.
* Ignores index unavailable during searches.
* Improve taskRunner's processResult (#24880)
* Set task state to idle after run, and add failed state
* fix tests
* Test alerting demo (#25136)
* Ensures that task store init gets called before schedule.
* Removes unused option for debugging purposes.
* Fix unit tests because a second callCluster was made.
* Task manager starts sanely.
Does not preInit
Will not allow definitions after initialization
Creates store immediately.
Modifies store after all plugins have initialized
Adds static tasks that will be defined by plugins.
* Task manager should not allow operations before initialization.
* Attempts to fix runner tests.
* Fixes unit test contract with APIs.
* Removes unused type definitions.
* Removes unused package json.
* Removes unused import type.
* Removes unnecessary async applied to a function.
* Returns diferently if task store has already initialized.
* Explains how to add static task instances to task definitions. (#25880)
* Tasks cannot be scheduled until task manager is initialized.
* Adds task manager api integration tests to ci group4.
* Context of describe test must be the test framework object.
* Update src/es_archiver/lib/indices/kibana_index.js
regex update to actually disallow non-kibana system indices
Co-Authored-By: tsullivan <tsullivan@users.noreply.github.com>
* verify fillPool exception passing
* readme update about max_workers/numWorkers
* change forEach to reduce
* use public interfaces in internal method
* replace getMaxAttempts with public readonly maxAttempts
* Update x-pack/plugins/task_manager/task_store.ts
`throw new Error` and initializattion spell fix
Co-Authored-By: tsullivan <tsullivan@users.noreply.github.com>
* min = 1 for max_workers
Co-Authored-By: tsullivan <tsullivan@users.noreply.github.com>
* timeOut => timeout
* min 1
* scope as an array
* no retries
Co-Authored-By: tsullivan <tsullivan@users.noreply.github.com>
* ConcreteTaskInstance is a TaskInstance
* remove guard per joi logic
* more return types for functions
* more comments around incremental back-off
Co-Authored-By: tsullivan <tsullivan@users.noreply.github.com>
* throw error instead of return undefined
* poll_interval min 1000
* avoid handling err.stack directly
* break up processResult
* fix a few runtime issues
* only fetch idle tasks
* remove check for status idle
* always return a state, and when there is an error return the state we had at beforeRun
* check isStarted before attemptWork
Co-Authored-By: tsullivan <tsullivan@users.noreply.github.com>
* ts fix
* Initial stab at core task manager logic
* Update task_manager readme
* Update task manager readme
* Add cancelable helper package, OSS, and general purpose, but
will be useful for writing cancelable x-pack tasks.
* Make the cancellable package promise-compliant
* Update task manager readme w/ reference to cancellable
* Change pool from lazy to eager, add support for sub-pools per task type.
* Move cancellable to task_manager, and typescriptify it.
* Working proof of concept for task manager. Still have lots to do:
clean up, tests, comments, validations, assertions, etc.
* Add pagination support to task manager fetch
* Move task manager to OSS
* Remove task manager reference from x-pack
* Make task_manager a valid core plugin
* Modify how task resource allocation works
* Remove the special case 'max' workers occupied value
* Remove x-pack package.json changes
* Make taskDefinitions a part of uiExports
* Make task docs saved-object compliant.
* Add kbnServer to the task context.
* Allow tasks to have a void / undefined return type
* revert x-pack change
* move cancellable to src/utils
* move to src/server
* use afterPluginsInit hook
* task_manager.ts rename
* add a wrapper with a setClient method
* Add tests for task runner
* Break task_pool into smaller, testable pieces
* return raw task doc fields for calling code
* remove todo comment
* helper module for default client
- setClient takes a callback fn
* fix misidentified field name
* fix rest args warning
* flatten task_pool
* remove cancellable
* return raw task doc fields for calling code
* remove todo comment
* helper module for default client
- setClient takes a callback fn
* fix rest args warning
* typescript fixes
* roll back setClient takes a callback
* createTaskRunner returns an object with run/cancel functions
* Test task store, tweak task poller tests
* Rename num_workers to max_workers
* Tweak task pool method names
* Fix cancellable test, and ts errors
* Rename doc to _doc
* Fix the doc type in the task manager index mappings
* Make task costs configurable in kibana.yml
* fix a tslint warning
* TaskManager.afterPluginsInit replaces circuitous stuff
* addMiddleware, implement beforeSave
* wip
* comment
* run context stuff
* pretty fix
* comments
* lint/pretty fixes
* safeties in case they don't define both beforeRun and beforeSave
* task runner test constructor opts fix
* Add task_manager integration tests
* FIx readme mistakes, fix schedule signature
* Fix integration tests
* Add blurb about middleware
* make a libs directory for the small things
* test for extractTaskDefinitions
* unit test for middleware
* Comment, refactor, and test task_manager
* Touch up comments
* Make task store throw error if given an unknown task type
* Fix sample task definition in readme
* Make task manager toggle on / off based on Elasticsearch connectivity
* Prevent task manager from crashing Kibana if
elasticsearch plugin is disabled
* Fix task manager signature
* Move the task manager into the task manager mixin,
fix tests. It's an uber integration object anyway, so
it seemed to make sense to keep it in one place.
* Fix task manager integration tests
* Update the task manager index creation to use auto_expand_replicas
* Fix task manager integration tests
* Fix task manager integration tests
* Fix the overzealous deletion of .kibana during tests
* Core task manager
* Allow hardcoded ID passed in
* comments
* don't deconstruct type and task for payload
* remove uiExport support
* move task_manager to x-pack/plugins
* expose as client on the plugin namespace
* fix tests
* move plugin api integration tests
* roll back readme sample task definition example
* fix sample plugin link
* server.taskManager
* sanitizeTaskDefinitions
* fix integration tests
* sanitize rename
* assert unintialized and check for duplicates in registerTaskDefinitions
* Remove unnecessary defaults from task_manager.test.ts
* Remove task manager from OSS
* Remove task manager from src
* Tidy up task_manager unit tests around elasticsearch status changes
* Integration test for specifying ID in scheduling
* Task_manager init -> preInit to allow other plugins to consume
taskManager in their init.
* Remove task manager integration tests from OSS
