When something causes an exception in `TaskRunner.markTaskAsRunning()` its execution fails, but this happens before we update the SO, which means that this failure does not count towards the `attempts` on the task. Task Manager will continue to try running this task for ever.
This PR increments the `attempts` when a failure occurs during `TaskRunner.markTaskAsRunning()` to ensure such a task doesn't continue to run to infinity.
Note that this fix will not affect `scheduled` tasks, as they are designed to _ignore_ their `attempts` and run for ever. In such a case this task will continue to consume Task Manager resources until canceled, but these failures will be logged and could be identified when needed.
* chore(NA): create new x-pack cigroups and rebalancing them all
* chore(NA): better cigroups balancing
* chore(NA): push rollup tests back into ciGroup1
* chore(NA): move some functional ml tests from cigroup3 into cigroup13
* chore(NA): move some more tests into ciGroup13
* chore(NA): use a single top level describe at x-pack/test/functional/apps/ml
* chore(NA): move settings into ciGroup13
* temporary test for es snapshots env
* Revert "temporary test for es snapshots env"
This reverts commit 789ebe7b9c.
* docs(NA): add missing documentation on the function tests describe split
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
This PR Introduces a `pollingDelay` which is applied to the polling interval whenever the average percentage of tasks experiencing a version conflict is higher than a preconfigured threshold (default to 80%).
Adds additional polling stats to Task Manager monitoring:
- **duration**: Running average of polling duration measuring the time from the scheduled polling cycle start until all claimed tasks are marked as running
- **claim_conflicts**: Running average of number of version clashes caused by the markAvailableTasksAsClaimed stage of the polling cycle
- **claim_mismatches**: Running average of mismatch between the number of tasks updated by the markAvailableTasksAsClaimed stage of the polling cycle and the number of docs found by the sweepForClaimedTasks stage
- **load** - Running average of the percentage of workers in use at the end of each polling cycle.
Added the following values to the Polling stats:
- **NoAvailableWorkers**: This tells us when a polling cycle resulted in no tasks being claimed due to there being no available workers
- **RunningAtCapacity**: This tells us when a polling cycle resulted in tasks being claimed at 100% capacity of the available workers
- **Failed**: This tells us when the poller failed to claim
resolves#55634resolves#65746
Buffers event docs being written for a fixed interval / buffer size,
and indexes those docs via a bulk ES call.
Also now flushing those buffers at plugin stop() time, which
we couldn't do before with the single index calls, which were
run via `setImmediate()`.
This is a redo of PR https://github.com/elastic/kibana/pull/80941 which
had to be reverted.
resolves https://github.com/elastic/kibana/issues/55634
resolves https://github.com/elastic/kibana/issues/65746
Buffers event docs being written for a fixed interval / buffer size,
and indexes those docs via a bulk ES call.
Also now flushing those buffers at plugin stop() time, which
we couldn't do before with the single index calls, which were
run via `setImmediate()`.
This addresses a bug in Task Manager in the task timeout behaviour. When a recurring task's `retryAt` field is set (which happens at task run), it is currently scheduled to the task definition's `timeout` value, but the original intention was for these tasks to retry on their next scheduled run (originally identified as part of https://github.com/elastic/kibana/issues/39349).
In this PR we ensure recurring task retries are scheduled according to their recurring schedule, rather than the default `timeout` of the task type.
* chore(NA): update gitignore to include first changes from moving into a single package.json
* chore(NA): update gitignore
* chore(NA): move all the dependencies into the single package.json and apply changes to bootstrap
* chore(NA): fix types problems after the single package json
* chore(NA): include code to find the dependencies used across the code
* chore(NA): introduce pure lockfile for install dependencies on build
* chore(NA): update clean task to not delete anything from xpack node_modules
* chore(NA): update gitignore to remove development temporary rules
* chore(NA): update notice file
* chore(NA): update jest snapshots
* chore(NA): fix whitelisted licenses to include a new specify form of an already included one
* chore(NA): remove check lockfile symlinks from child projects
* chore(NA): fix eslint and add missing declared deps on single pkg json
* chore(NA): correctly update notice
* chore(NA): fix failing jest test for storyshots.test.tsx
* chore(NA): fix cypress multi reporter path
* chore(NA): fix Project tests check
* chore(NA): fix problem with logic to detect used dependes on oss build
* chore(NA): include correct x-pack plugins dep discovery
* chore(NA): discover entries under dynamic requires on vis_type_timelion
* chore(NA): remove canvas
* test(NA): fix jest unit tests
* chore(NA): remove double react declaration from storyshot test file
* chore(NA): try removing isOSS check
* chore(NA): support for plugin development
* chore(NA): update logic to fix unit tests and typechecking
* chore(NA): support to run npm scripts in child kbn projects across all envs
* chore(NA): support github checks reporter on x-pack and remove cpy types as the package correctly provides them
* chore(NA): update cpy version
* chore(NA): include last kbn pm changes
* chore(NA): update style on build_production_projects.ts
* chore(NA): remove any cast fom telemetry opt in stats
* chore(NA): remove del and re-use rm -rf again
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
This PR adds an an internal monitoring mechanism in Task Manager which keep track of a variety of metrics and a health api endpoint which makes the monitored statistics accessible.
* wip
* Adding updateFieldsAndMarkAsFailed function
* Updating UBQ
* Only updating retryAt if marking as claiming
* Updating query
* Updating query to only fail one time tasks that have exceeded max attempts
* Fixing tests
* Fixing tests
* Handling claiming tasks by id
* Removing unused function
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
This PR addresses a list of legacy code debt the plugin has incurred over the past year due to extensive changes in its internals and the adoption of the Kibana Platform.
It includes:
1. The `TaskManager` class has been split into several independent components: `TaskTypeDictionary`, `TaskPollingLifecycle`, `TaskScheduling`, `Middleware`. This has made it easier to understand the roles of the different parts and makes it easier to plug them into the observability work.
2. The exposed `mocks` have been corrected to correctly express the Kibana Platform api
3. The lifecycle has been corrected to remove the need for intermediary streames/promises which we're needed when we first introduced the `setup`/`start` lifecycle to support legacy.
4. The Logger mocks have been replaced with the platform's `coreMocks` implementation
5. The integration tests now test the plugin's actual public api (instead of the internals).
6. The Legacy Elasticsearch client has been replaced with the typed client in response to the deprecation notice.
7. Typing has been narrowed to prevent the `type` field from conflicting with the key in the `TaskDictionary`. This could have caused the displayed `type` on a task to differ from the `type` used in the Dictionary itself (this broke a test during refactoring and could have caused a bug in production code if left).
* bump ts to v4
* MOAR RAM
* fix type errors for OSS
* first pass on x-pack errors
* second pass on x-pack type errors
* 3rd pass on x-pack type-errors
* mute errors if complex cases
* don't delete if spread suffices
* mute other complex cases
* make User fields optional
* fix optional types
* fix tests
* fix typings for time_range
* fix type errors in x-pack/tests
* rebuild kbn-pm
* remove leftovers from master update
* fix alert tests
* [Telemetry Checker] TS4 Fixes
* bump to 4.0.1-rc
* fix new errors in master
* bump typescript-eslint to version supporting TS v4 syntax
* fix merge commit errors
* update to the stable TS version 4.0.2
* bump ts-eslint to version supporting ts v4
* fix typo
* fix type errors after merge
* update ts in another new package.json
* TEMP: remove me
* Revert "TEMP: remove me"
This reverts commit dc0fc3bae6.
* [Telemetry] Update snapshot for new TS4 SyntaxKind
* bump prettier to support TS v4 syntax
* fix prettier rules
* last style change
* fix new type errors
Co-authored-by: Alejandro Fernández Haro <alejandro.haro@elastic.co>
Fixes flaky tests in Task Manager and Alerting.
The fix in #73244 was correct, but it missed an edge case which causes the already running task to be rescheduled over and over.
This prevents that edge case which was effecting both TM in general and Alerting specifically.
This PR addresses two issues which caused several tests to be flaky in TM.
When `runNow` was introduced to TM we added a pinned query which returned specific tasks by ID.
This query does not have the filter applied to it which causes task to return when they're already marked as `running` but we didn't address these correctly which caused flakyness in the tests.
This didn't cause a broken beahviour, but it did cause beahviour that was hard to reason about - we now address them correctly.
It seems that sometimes, especially if the ES queue is overworked, it can take some time for the update to the underlying task to be visible (we don't user `refresh:true` on purpose), so adding a wait for the index to refresh to make sure the task is updated in time for the next stage of the test.
* mark legacy ES client types as deprecated
* expose es client to plugins and update mocks
* ElasticSearchClientMock --> ElasticsearchClientMock
* expose es client mocks
* expose es client via RequestHandlerContext
* convert test/plugin_functional/config into ts
* convert top_nav test into ts
* add an integration test for the es client
* update comments to refer to the new es client
* fix import paths. do not use extensions
temp
* update docs
* fix other refs
* add test for a custom client
* fix context
* add test for scoped client
* update docs
resolves https://github.com/elastic/kibana/issues/70086
Configures the saved object client for the event log to access the recently
hidden action and alert saved objects.
We didn't have tests for action/alert event log activity, so added some now.
Also found a buglet that was preventing access to event log data from actions
and alerts in non-default spaces.
* add babel support for export type
* bump ts version to 3.9.3
* rebuild kbn-pm
* bump typescript-eslint
* fix error in security plugin UI
* check export as works
* fix app migration type
* use correct test subj attribute
* fix errors from the old PR
* embeddable is already passed in props
* explicitly define type of fetch
* add some types for viz
* fix fetch type p.2
* add null to allow spreading without type errors due to override
* add type guard to fix type error
* cast to any, since cannot assign unknown
* add timestamp to known types
* fix type error in fetch
* fix type error. id is always defined in attibutes
* declare a type
* move ts-ignore to the lines with errors
* declare tuple type explicitly
* mute type error. cannot assign unknown
* fix errors. id is always defined
* fix error type
* fix override errors. id is always defined
* fix error. extends any doesn't work anymore
* fix type error. type is always defined
* env doesn't always contain values
* fix type error
* cast to string
* add: logs is already declared in getNodeLogsUrl
* state is already passed in props
* fix some errors in timelion
* number of fragments is always defined
* 'absolute' is not just string, but value
* TEMP: option is always defined
* always true if cast to promise manually
* both props are always defined
* explicitly define returned SO type
* workaround type
* bump tslib to be compatible with ts v3.9
* test private property
* rebuild kbn-pm
* Fix ts errors for beats management
* Fix type inference broken by the TS 3.9 upgrade
* Fix ingest manager saved object attributes typings
* Fix TS errors in cross_cluster_replication and index_management.
* Fix TS error in Watcher.
* roll back colorRange wrong type
* fix security plugin types
* TypeScript 3.9 fixes for APM
* Fix ColorRange types.
* fix actions & alerts errors. ByGidi
* fix lists error
* More APM fixes
* Remove paramaterization from `removeEmpty in agent config SettingsPage component (it's only used there and doesn't need to be parameterized.)
* Add option chain for case in registerTransactionDurationAlertType
* Cast `overallValue` in transform_metrics_chart
* Use more specific type for custom link filters
* Add more option chaining for local UI filters buckets response
* Remove unused parameters from routes
* Fix getProjection type parameter
* Use destructuring in serviceNodesLocalFiltersRoute to hide `never` error
* Revert `UnionToIntersection` change in `AggregationResponseMap`
Fixes#67804.
* fix platform type error
* Fix visualizations types.
* Fix data plugin types.
* bump TS version to 3.9.5
* Fix telemetry TS errors
* Fix dashboard code
* Adding Canvas Fixes for TS 3.9
* Fix case and security_solution types
* roll back to the old export syntax. new one might cause problems in api-extractor
* update docs
* Fix timelion code
* Fix meta
* Fix types
* fix type errors om ingest_manager
* bump babel deps
* enable private props & methods syntax
* update kbn-pm dist
* whitelist 0BSD license
* use @babel/plugin-proposal-private-methods in default set as well
* disable new babel plugins
* Revert "disable new babel plugins"
This reverts commit 04d959431d.
* cleanup security_solution types
* Fixes type error for newer TypeScript
* update docs
Co-authored-by: Nicolas Chaulet <nicolas.chaulet@elastic.co>
Co-authored-by: Felix Stürmer <stuermer@weltenwort.de>
Co-authored-by: CJ Cenizal <cj@cenizal.com>
Co-authored-by: Larry Gregory <larry.gregory@elastic.co>
Co-authored-by: Nathan L Smith <smith@nlsmith.com>
Co-authored-by: Walter Rafelsberger <walter@elastic.co>
Co-authored-by: Luke Elmers <luke.elmers@elastic.co>
Co-authored-by: Alejandro Fernández Haro <alejandro.haro@elastic.co>
Co-authored-by: Tim Roes <tim.roes@elastic.co>
Co-authored-by: Clint Andrew Hall <clint.hall@elastic.co>
Co-authored-by: Patryk Kopycinski <contact@patrykkopycinski.com>
Co-authored-by: FrankHassanabad <frank.hassanabad@elastic.co>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Creating events in parallel may be causing a slight flakyness, this change staggers creation to ensure this doesn't happen.
In addition it turned out the `event.end` field was missing in certain cases, causing the test that sorts by `end` to fail.
resolves https://github.com/elastic/kibana/issues/62668
Adds a property named `rel` to the nested saved objects in the event
documents, whose value should not be set, or set to `primary`.
The query by saved object function changes to only match event documents
with that saved objects if it has the `rel: primary` value.
This is used to limit searching alerting's executeAction event document
with only the alert saved object, and not the action saved object (this
document has an alert and action saved object). The alert saved object
has the `rel: primary` field set, and the action does not. Previously,
those documents were returned with a query of the action saved object.
Completes the migration of all Alerting Services plugins onto the Kibana Platform
It includes:
1. Actions plugin
2. Alerting plugin
3. Task Manager plugin
4. Triggers UI plugin
And touches the Uptime and Siem plugins as their use of the Task Manager relied on some of the legacy lifecycle to work (registering AlertTypes and Telemetry tasks after the Start stage has already began). The fix was simply to moves these registrations to the Setup stage.
resolves https://github.com/elastic/kibana/issues/64275
Changes the fields used to query the event log by time range to use the
`@timestamp` field.
Also allow `@timestamp` as a sort option, and make it the default sort option.
* Added server api tests for event log service
* fixed tests
* fixed type check issue
* Fixed failing tests
* fixed jest tests
* Fixed due to comments
* Removed flackiness tests
* fixed type check error
* Fixed func test
Adds a namespace attribute to the saved object object within the Event Log so that each Saved Object can have its own. This change also removes the existing kibana.namespace field.
As Event Log is not yet in use, this does not include a migration.
Enables access to the Alert State, which allows us to see which current Alert Instances are active.
This includes:
1. Addition of a `get` api on Task Manager
2. Typing and validation on Serialisation & Deserialisation of the State of an Alert's underlying Task
3. Addition of the `getAlertState` api on AlertsClient
As of Elasticsearch 8.0.0 it will no longer be possible to use the _id field on documents.
This PR removes the usage that Task Manager makes of this field and switches to pinned queries to achieve a similar effect.
Migrates the existing TaskManager plugin from Legacy to Kibana Platform.
We retain the Legacy API to prevent a breaking change, but under the hood, the legacy plugin is now using the Kibana Platform plugin.
Another reason we retain the Legacy plugin to support several features that the Platform team has yet to migrate to Kibana Platform (mapping, SO schema and migrations).
