When something causes an exception in `TaskRunner.markTaskAsRunning()` its execution fails, but this happens before we update the SO, which means that this failure does not count towards the `attempts` on the task. Task Manager will continue to try running this task for ever.
This PR increments the `attempts` when a failure occurs during `TaskRunner.markTaskAsRunning()` to ensure such a task doesn't continue to run to infinity.
Note that this fix will not affect `scheduled` tasks, as they are designed to _ignore_ their `attempts` and run for ever. In such a case this task will continue to consume Task Manager resources until canceled, but these failures will be logged and could be identified when needed.
resolves#55634resolves#65746
Buffers event docs being written for a fixed interval / buffer size,
and indexes those docs via a bulk ES call.
Also now flushing those buffers at plugin stop() time, which
we couldn't do before with the single index calls, which were
run via `setImmediate()`.
This is a redo of PR https://github.com/elastic/kibana/pull/80941 which
had to be reverted.
resolves https://github.com/elastic/kibana/issues/55634
resolves https://github.com/elastic/kibana/issues/65746
Buffers event docs being written for a fixed interval / buffer size,
and indexes those docs via a bulk ES call.
Also now flushing those buffers at plugin stop() time, which
we couldn't do before with the single index calls, which were
run via `setImmediate()`.
This addresses a bug in Task Manager in the task timeout behaviour. When a recurring task's `retryAt` field is set (which happens at task run), it is currently scheduled to the task definition's `timeout` value, but the original intention was for these tasks to retry on their next scheduled run (originally identified as part of https://github.com/elastic/kibana/issues/39349).
In this PR we ensure recurring task retries are scheduled according to their recurring schedule, rather than the default `timeout` of the task type.
* chore(NA): update gitignore to include first changes from moving into a single package.json
* chore(NA): update gitignore
* chore(NA): move all the dependencies into the single package.json and apply changes to bootstrap
* chore(NA): fix types problems after the single package json
* chore(NA): include code to find the dependencies used across the code
* chore(NA): introduce pure lockfile for install dependencies on build
* chore(NA): update clean task to not delete anything from xpack node_modules
* chore(NA): update gitignore to remove development temporary rules
* chore(NA): update notice file
* chore(NA): update jest snapshots
* chore(NA): fix whitelisted licenses to include a new specify form of an already included one
* chore(NA): remove check lockfile symlinks from child projects
* chore(NA): fix eslint and add missing declared deps on single pkg json
* chore(NA): correctly update notice
* chore(NA): fix failing jest test for storyshots.test.tsx
* chore(NA): fix cypress multi reporter path
* chore(NA): fix Project tests check
* chore(NA): fix problem with logic to detect used dependes on oss build
* chore(NA): include correct x-pack plugins dep discovery
* chore(NA): discover entries under dynamic requires on vis_type_timelion
* chore(NA): remove canvas
* test(NA): fix jest unit tests
* chore(NA): remove double react declaration from storyshot test file
* chore(NA): try removing isOSS check
* chore(NA): support for plugin development
* chore(NA): update logic to fix unit tests and typechecking
* chore(NA): support to run npm scripts in child kbn projects across all envs
* chore(NA): support github checks reporter on x-pack and remove cpy types as the package correctly provides them
* chore(NA): update cpy version
* chore(NA): include last kbn pm changes
* chore(NA): update style on build_production_projects.ts
* chore(NA): remove any cast fom telemetry opt in stats
* chore(NA): remove del and re-use rm -rf again
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* wip
* Adding updateFieldsAndMarkAsFailed function
* Updating UBQ
* Only updating retryAt if marking as claiming
* Updating query
* Updating query to only fail one time tasks that have exceeded max attempts
* Fixing tests
* Fixing tests
* Handling claiming tasks by id
* Removing unused function
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
This PR addresses a list of legacy code debt the plugin has incurred over the past year due to extensive changes in its internals and the adoption of the Kibana Platform.
It includes:
1. The `TaskManager` class has been split into several independent components: `TaskTypeDictionary`, `TaskPollingLifecycle`, `TaskScheduling`, `Middleware`. This has made it easier to understand the roles of the different parts and makes it easier to plug them into the observability work.
2. The exposed `mocks` have been corrected to correctly express the Kibana Platform api
3. The lifecycle has been corrected to remove the need for intermediary streames/promises which we're needed when we first introduced the `setup`/`start` lifecycle to support legacy.
4. The Logger mocks have been replaced with the platform's `coreMocks` implementation
5. The integration tests now test the plugin's actual public api (instead of the internals).
6. The Legacy Elasticsearch client has been replaced with the typed client in response to the deprecation notice.
7. Typing has been narrowed to prevent the `type` field from conflicting with the key in the `TaskDictionary`. This could have caused the displayed `type` on a task to differ from the `type` used in the Dictionary itself (this broke a test during refactoring and could have caused a bug in production code if left).
* bump ts to v4
* MOAR RAM
* fix type errors for OSS
* first pass on x-pack errors
* second pass on x-pack type errors
* 3rd pass on x-pack type-errors
* mute errors if complex cases
* don't delete if spread suffices
* mute other complex cases
* make User fields optional
* fix optional types
* fix tests
* fix typings for time_range
* fix type errors in x-pack/tests
* rebuild kbn-pm
* remove leftovers from master update
* fix alert tests
* [Telemetry Checker] TS4 Fixes
* bump to 4.0.1-rc
* fix new errors in master
* bump typescript-eslint to version supporting TS v4 syntax
* fix merge commit errors
* update to the stable TS version 4.0.2
* bump ts-eslint to version supporting ts v4
* fix typo
* fix type errors after merge
* update ts in another new package.json
* TEMP: remove me
* Revert "TEMP: remove me"
This reverts commit dc0fc3bae6.
* [Telemetry] Update snapshot for new TS4 SyntaxKind
* bump prettier to support TS v4 syntax
* fix prettier rules
* last style change
* fix new type errors
Co-authored-by: Alejandro Fernández Haro <alejandro.haro@elastic.co>
This PR addresses two issues which caused several tests to be flaky in TM.
When `runNow` was introduced to TM we added a pinned query which returned specific tasks by ID.
This query does not have the filter applied to it which causes task to return when they're already marked as `running` but we didn't address these correctly which caused flakyness in the tests.
This didn't cause a broken beahviour, but it did cause beahviour that was hard to reason about - we now address them correctly.
It seems that sometimes, especially if the ES queue is overworked, it can take some time for the update to the underlying task to be visible (we don't user `refresh:true` on purpose), so adding a wait for the index to refresh to make sure the task is updated in time for the next stage of the test.
* mark legacy ES client types as deprecated
* expose es client to plugins and update mocks
* ElasticSearchClientMock --> ElasticsearchClientMock
* expose es client mocks
* expose es client via RequestHandlerContext
* convert test/plugin_functional/config into ts
* convert top_nav test into ts
* add an integration test for the es client
* update comments to refer to the new es client
* fix import paths. do not use extensions
temp
* update docs
* fix other refs
* add test for a custom client
* fix context
* add test for scoped client
* update docs
resolves https://github.com/elastic/kibana/issues/70086
Configures the saved object client for the event log to access the recently
hidden action and alert saved objects.
We didn't have tests for action/alert event log activity, so added some now.
Also found a buglet that was preventing access to event log data from actions
and alerts in non-default spaces.
* add babel support for export type
* bump ts version to 3.9.3
* rebuild kbn-pm
* bump typescript-eslint
* fix error in security plugin UI
* check export as works
* fix app migration type
* use correct test subj attribute
* fix errors from the old PR
* embeddable is already passed in props
* explicitly define type of fetch
* add some types for viz
* fix fetch type p.2
* add null to allow spreading without type errors due to override
* add type guard to fix type error
* cast to any, since cannot assign unknown
* add timestamp to known types
* fix type error in fetch
* fix type error. id is always defined in attibutes
* declare a type
* move ts-ignore to the lines with errors
* declare tuple type explicitly
* mute type error. cannot assign unknown
* fix errors. id is always defined
* fix error type
* fix override errors. id is always defined
* fix error. extends any doesn't work anymore
* fix type error. type is always defined
* env doesn't always contain values
* fix type error
* cast to string
* add: logs is already declared in getNodeLogsUrl
* state is already passed in props
* fix some errors in timelion
* number of fragments is always defined
* 'absolute' is not just string, but value
* TEMP: option is always defined
* always true if cast to promise manually
* both props are always defined
* explicitly define returned SO type
* workaround type
* bump tslib to be compatible with ts v3.9
* test private property
* rebuild kbn-pm
* Fix ts errors for beats management
* Fix type inference broken by the TS 3.9 upgrade
* Fix ingest manager saved object attributes typings
* Fix TS errors in cross_cluster_replication and index_management.
* Fix TS error in Watcher.
* roll back colorRange wrong type
* fix security plugin types
* TypeScript 3.9 fixes for APM
* Fix ColorRange types.
* fix actions & alerts errors. ByGidi
* fix lists error
* More APM fixes
* Remove paramaterization from `removeEmpty in agent config SettingsPage component (it's only used there and doesn't need to be parameterized.)
* Add option chain for case in registerTransactionDurationAlertType
* Cast `overallValue` in transform_metrics_chart
* Use more specific type for custom link filters
* Add more option chaining for local UI filters buckets response
* Remove unused parameters from routes
* Fix getProjection type parameter
* Use destructuring in serviceNodesLocalFiltersRoute to hide `never` error
* Revert `UnionToIntersection` change in `AggregationResponseMap`
Fixes#67804.
* fix platform type error
* Fix visualizations types.
* Fix data plugin types.
* bump TS version to 3.9.5
* Fix telemetry TS errors
* Fix dashboard code
* Adding Canvas Fixes for TS 3.9
* Fix case and security_solution types
* roll back to the old export syntax. new one might cause problems in api-extractor
* update docs
* Fix timelion code
* Fix meta
* Fix types
* fix type errors om ingest_manager
* bump babel deps
* enable private props & methods syntax
* update kbn-pm dist
* whitelist 0BSD license
* use @babel/plugin-proposal-private-methods in default set as well
* disable new babel plugins
* Revert "disable new babel plugins"
This reverts commit 04d959431d.
* cleanup security_solution types
* Fixes type error for newer TypeScript
* update docs
Co-authored-by: Nicolas Chaulet <nicolas.chaulet@elastic.co>
Co-authored-by: Felix Stürmer <stuermer@weltenwort.de>
Co-authored-by: CJ Cenizal <cj@cenizal.com>
Co-authored-by: Larry Gregory <larry.gregory@elastic.co>
Co-authored-by: Nathan L Smith <smith@nlsmith.com>
Co-authored-by: Walter Rafelsberger <walter@elastic.co>
Co-authored-by: Luke Elmers <luke.elmers@elastic.co>
Co-authored-by: Alejandro Fernández Haro <alejandro.haro@elastic.co>
Co-authored-by: Tim Roes <tim.roes@elastic.co>
Co-authored-by: Clint Andrew Hall <clint.hall@elastic.co>
Co-authored-by: Patryk Kopycinski <contact@patrykkopycinski.com>
Co-authored-by: FrankHassanabad <frank.hassanabad@elastic.co>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Creating events in parallel may be causing a slight flakyness, this change staggers creation to ensure this doesn't happen.
In addition it turned out the `event.end` field was missing in certain cases, causing the test that sorts by `end` to fail.
resolves https://github.com/elastic/kibana/issues/62668
Adds a property named `rel` to the nested saved objects in the event
documents, whose value should not be set, or set to `primary`.
The query by saved object function changes to only match event documents
with that saved objects if it has the `rel: primary` value.
This is used to limit searching alerting's executeAction event document
with only the alert saved object, and not the action saved object (this
document has an alert and action saved object). The alert saved object
has the `rel: primary` field set, and the action does not. Previously,
those documents were returned with a query of the action saved object.
Completes the migration of all Alerting Services plugins onto the Kibana Platform
It includes:
1. Actions plugin
2. Alerting plugin
3. Task Manager plugin
4. Triggers UI plugin
And touches the Uptime and Siem plugins as their use of the Task Manager relied on some of the legacy lifecycle to work (registering AlertTypes and Telemetry tasks after the Start stage has already began). The fix was simply to moves these registrations to the Setup stage.
* Added server api tests for event log service
* fixed tests
* fixed type check issue
* Fixed failing tests
* fixed jest tests
* Fixed due to comments
* Removed flackiness tests
* fixed type check error
* Fixed func test
Enables access to the Alert State, which allows us to see which current Alert Instances are active.
This includes:
1. Addition of a `get` api on Task Manager
2. Typing and validation on Serialisation & Deserialisation of the State of an Alert's underlying Task
3. Addition of the `getAlertState` api on AlertsClient
As of Elasticsearch 8.0.0 it will no longer be possible to use the _id field on documents.
This PR removes the usage that Task Manager makes of this field and switches to pinned queries to achieve a similar effect.
Migrates the existing TaskManager plugin from Legacy to Kibana Platform.
We retain the Legacy API to prevent a breaking change, but under the hood, the legacy plugin is now using the Kibana Platform plugin.
Another reason we retain the Legacy plugin to support several features that the Platform team has yet to migrate to Kibana Platform (mapping, SO schema and migrations).
This moves the interval field under a generic schedule object field in preparation for the introduction of richer scheduling options (such as cron).
It includes a migration for existing tasks, and we've ensured no existing Task Type Definitions exist in Kibana that rely on Interval.
This includes support for the deprecated interval field (which gets mapped to schedule) but that support will be removed in 8.0.0, as it's a breaking change.
Adds a `runNow` api to Task Manager, allowing us to force the refresh of a recurring task.
This PR includes a couple of sustainability changes as well as the feature itself.
1. **Declarative query composition.** At the moment the queries in the TaskStore are huge JSON objects that are hard to maintain and understand. This PR introduces a pattern where the different parts of the query are composed out of type-checked functions, making it easier to maintain and to construct dynamically as needs change. _This was included in this PR as the **markAvailableTasksAsClaimed** query needs different query clauses depending on whether there are specific Tasks we wish to claim first.
2. **Refactoring of the Task Poller** As the `runNow` api is introduced we find Task Manager's lifecycle in a weird state where it has both a _pull_ model, where timeouts & callbacks interact without having to responsd to any external requests, and a _push_ model where requests are made to the new `runNow` api. Balancing these two proved error prone, hard to maintain and had the potential of _lossy_ behaviour where requests are dropped accidentally. To address this TaskPoller has been refactored using Rxjs observables, remodelling the existing _pull_ mechanism as a _push_ mechanism so Task Manager can _respond_ to both _polling_ calls and _runNow_ in a similar fashion.
And ofcourse the main feature of this PR:
3. **runNow api** An api on TaskManager that takes a _task ID_ and attempts to run the task. The call returns a promise which resolves with a result which notifies the caller when the task has either completed successfully, or result in an error.
This PR adds a test that ensures Task Manager is capable of picking up new tasks in parallel to a long running tasks that might otherwise hold up task execution.
This doesn't add functionality - just a missing test case.
* Additional validation for elasticsearch username
If "elastic" user is set in config:
* In dev mode, throws an error
* In prod mode, logs a deprecation warning
* Fix user for functional tests
* Revert last two commits
Revert "Fix user for functional tests" and
"Fix user for plugin functional tests in Jenkinsfile"
* Change elasticsearch creds for test server
Now uses "kibana" user instead of "elastic" user
* Fix plugin API functional tests
* Fix PKI API integration test
* Change log messages, now conditional on `dist: false` not `dev: true`
Adds an ensureScheduling api to Task Manager which allow safer rescheduling of existing tasks by handling the case where a Task with a known ID is scheduled and clashes with an existing schedule of that same task.
* Create REST API to fire actions
* Add more tests to the fire API
* Remove dead file reference
* Fix broken tests
* Fix broken test
* Update docs
* Apply PR feedback
* Create actions plugin (#35679)
* Basic alerting plugin with actions
* Remove relative imports
* Code cleanup
* Split service into 3 parts, change connector structure
* Ability to disable plugin, ability to get actions
* Add slack connector
* Add email connector
* Ability to validate params and connector options
* Remove connectorOptionsSecrets for now
* Fix plugin config validation
* Add tests for slack connector
* Default connectors register on plugin init, console renamed to log, slack to message_slack
* Add remaining API endpoints for action CRUD
* Add list connectors API
* Change actions CRUD APIs to be closer with saved objects structure
* WIP
* Fix broken tests
* Add encrypted attribute support
* Add params and connectorOptions for email
* WIP
* Remove action's ability to have custom ids
* Remove ts-ignore
* Fix broken test
* Remove default connectors from this branch
* Fix API integration tests to use fixture connector
* Rename connector terminology to action type
* Rename actionTypeOptions to actionTypeConfig
* Code cleanup
* Fix broken tests
* Rename alerting plugin to actions
* Some code cleanup and add API unit tests
* Change signature of action type service execute function
* Add some plugin api integration tests
* Fix type check failure
* Code cleanup
* Create an actions client instead of an action service
* Apply Bill's PR feedback
* Fix broken test
* Find function to have destructured params
* Add tests to ensure encrypted attributes are not returned
* Fix broken test
* Add tests for validation
* Ensure actions can be updated without re-passing the config
* Remove dead code
* Test cleanup
* Fix eslint issue
* Apply Peter's PR feedback
* Code cleanup and fix broken tests
* Apply Brandon's PR feedback
* Add namespace support
* Fix broken test
* Pass services to action executors (#37194)
* Pass services to action executors
* Fix tests
* Apply PR feedback
* Apply PR feedback pt2
* Cleanup actions plugin (#37250)
* Cleanup actions, move code from alerting plugin PR
* Rename service terminology to registry
* Use static encryption key for encrypted attributes plugin inside of tests
* Empty data after create test is done running
* Fix type checks
* Fix inconsistent naming
* add server log action for alerting (#37530)
adds the first "builtin" alertType for performing a `server.log()`
* Create alerting plugin (#37043)
* WIP
* Rename fire function and remove @ts-ignore in all places
* Change naming in alerting service
* Remove alert instance class for now, support interval configuration
* Cleanup TS
* Split alerting between registry and client
* Use saved object alongside task manager instance
* Add remaining alerting APIs
* Change create structure
* Rename some variables, change actionGroups structure
* Use handlebars for templating strings at fire time
* Fix params given to alert type execute function
* Use alert instance class
* Alert instances support meta attributes
* Move alert instances deserialization
* Change interval to be ms
* Rename actions es archive
* Fix tests to use encrypted esArchive for action record
* Add create alert test to demo end to end flow
* Fix type check issue
* Alerts to use references to action objects
* Only update task manager tasks after saved objects are fully updated
* Use scope in task manager
* Fix type check
* Use task manager to execute actions
* Convert ids into references and back
* Apply PR feedback
* Fix broken test
* Fix some bugs
* Fix test errors
* Alert interval to be previous runAt + interval instead of now + interval
* Add range support
* Remove extra line
* Cleanup
* Add alert_instance.test.ts
* Add alert_type_registry.test.ts
* Move tests around
* Create generic task manager mock
* Add note about saved objects client mock
* Create alert_type_registry.mock.ts
* Add alerts_client.test.ts
* Add create_alert_instance_factory.test.ts
* Add create_fire_handler.test.ts
* WIP
* Fix get_create_task_runner_function.test.ts and make test pass
* Make get_create_task_runner_function.test.ts 100% coverage
* Add unit tests for routes
* Move files around
* Created transform_action_params.ts
* Add get_next_run_at.ts
* Add comment explaining why we copy nextRunAt
* Re-use state within alert instance
* Finalize code coverage in unit tests
* Create base api integration tests
* Add a test that ensures end to end functionality of an alert
* Fix ui capabilities test
* Fix broken plugin api integration test
* Fix jest tests with new saved objects client
* Fix broken integration tests
* Change api integration test fixture to make more sense, add functions for future tests
* Move alerts integration testing into own file, prep to add more tests
* Add tests to ensure failed task instances get retried
* Add get_create_task_runner_function.test.ts for actions, create encrypted saved objects mock
* Add action validation tests
* Ensure action type validation occurs on update
* Test 400 on unregistered alert types
* Ensure alertTypeId can't be updated
* Add validation test for alert create / update
* Fix broken checks / tests
* Skip failing test for now
* Cleanup jest tests
* Ensure action objects can be updated while keeping encrypted attributes readable
* Remove partial update sopport, remove ability to change actionTypeId, require config
* Ensure actionTypeConfig is validated on create and update
* Add alertTypeParams validation support
* Fix failing tests
* Ensure alert cleanup errors don't replace the original error
* Pass callCluster as a service to alerts and actions
* Only pass log to alerts client
* Pass savedObjectsClient as a service to alerting and actions
* Fix failing tests
* Remove range support, provide when current and previous task got scheduled
* Ensure Joi validation happens before every execute
* Remove skipped tests, to be done in future PR
* Apply self feedback pt1
* Apply self feedback pt2
* Fix broken tests
* Apply PR feedback
* PR feedback pt1
* Apply security team PR feedback
* PR feedback pt1
* PR feedback pt2
* PR feedback pt3
* Fix broken tests
* Fix callCluster to have signature
* Revert f11a6aeb06
* PR feedback pt4
* Remove __jest__ folders
* PR feedback pt5
* Fix Joi from leaking secrets
* Fire instance actions in parallel instead of series
* Fix failing jest tests
* Accept core api changes
* Fix saved objects client mock
* PR feedback pt1
* Fix eslint issues
* Throw error when alert instance already fired (#39251)
* Throw error when alert instance already fired
* shouldFire doesn't need its own boolean value
* Actions & alerting getting started user guides (#39093)
* Initial user guides
* Cleanup
* Typos, example changes
* Switch to tables, use ordered list for usage
* Start docs around alert instances and templating
* Documentation changes
* Some adjustments
* Apply PR feedback
* Apply suggestions from code review
Co-Authored-By: gchaps <33642766+gchaps@users.noreply.github.com>
* PR feedback pt2
* Provide better examples for alert types
* Apply PR feedback
* Update README locations
* chore(NA): first changes on every package.json order to support new babel 7. chore(NA): build for kbn-pm with babel 7.
* chore(NA): patch babel register to load typescrit
* chore(NA): first working version with babel 7 replacing typescript compiler.
* fix(NA): common preset declaration in order to make it work with babel-loader.
* chore(na): organizing babel preset env package json.
* chore(NA): mocha tests enabled.
* fix(NA): typo on importing
* test(NA): majority of x-pack tests ported to use babel-jest
* fix(NA): report info button test with babel-jest.
* fix(NA): polling service tests.
* test(na): fix server plugins plugin tests.
* test(NA): batch of test fixs for jest tests under babel-jest hoisting.
* chore(NA): add babel plugin to hoist mock prefixed vars on jest tests.
* chore(NA): update yarn.lock file.
* chore(NA): tests passing.
* chore(NA): remove wrong dep
* chore(NA): fix tsconfig
* chore(NA): skip babel for ts-jest.
* chore(NA): selectively apply the plugin to strip off namespace from ts files.
* chore(NA): remove not needed changes from ts tests
* chore(NA): removed ts-jest dependency. chore(NA): migrate ts tests on x-pack to use babel-jest with the new pattern.
* chore(NA): migrate kibana default distribution typescript tests to run with babel-jest and the new test mock pattern.
* chore(NA): merge and solve conflicts with master.
* chore(NA): fix problems reported by eslint
* chore(NA): fix license ovveride for babel-plugin-mock-imports
* chore(NA): update jest integration tests for kbn pm
* chore(NA): update babel jest integration tests for kbn pm.
* test(NA): update jest integration snapshot for kbn pm.
* chore(NA): apply changes according to the pull request reviews.
* chore(NA): apply changes according to the pull request reviews.
* refact(NA): migrate jest tests to the new pattern.
* fix(NA): babel 7 polyfill in the tests bundle.
* chore(NA): restore needed step in order to compile x-pack with typescript.
* chore(NA): change build to compile typescript with babel for the oss code. chore(NA): change transpile typescript task to only transpile types for x-pack. refact(NA): common preset for babel 7
* Revert "chore(NA): change build to compile typescript with babel for the oss code. chore(NA): change transpile typescript task to only transpile types for x-pack. refact(NA): common preset for babel 7"
This reverts commit 2707d538f5.
* fix(NA): import paths for tabConfigConst
* chore(NA): fix transpiling error on browser tests
* chore(NA): simplify kbn babel preset package.
* chore(NA): migrate build to use babel transpiler for typescript excluding xpack.
* fix(NA): introduced error on test quick task.
* fix(NA): fix preset for client side code on build.
* fix(NA): build with babel
* fix(NA): negated patterns in the end.
* fix(NA): kbn_tp_sample_panel_action creation.
* fix(NA): babel typescript transform plugin workaround when exporting interface name.
* refact(NA): remove not needed type cast to any on jest test.
* docs(NA): add developement documentation about jest mocks test pattern.
* chore(NA): missing unmerged path.
* chore(NA): fix jest tests for template.
* [CCR] Client integration tests (table lists) (#33525)
* Force user to re-authenticate if token refresh fails with `400` status code. (#33774)
* Improve performance of the Logstash Pipeline Viewer (#33793)
Resolves#27513.
_This PR is a combination of #31293 (the code changes) + #33570 (test updates). These two PRs were individually reviewed and merged into a feature branch. This combo PR here simply sets up the merge from the feature branch to `master`._
Summary of changes, taken from #31293:
## Before this PR
The Logstash Pipeline Viewer UI would make a single Kibana API call to fetch all the information necessary to render the Logstash pipeline. This included information necessary to render the detail drawer that opens up when a user clicks on an individual vertex in the pipeline.
Naturally, this single API call fetched _a lot_ of data, not just from the Kibana server but also, in turn, from Elasticsearch as well. The "pro" of this approach was that the user would see instantaneous results if they clicked on a vertex in a pipeline and opened the detail drawer for that vertex. The "cons" were the amount of computation Elasticsearch had to perform and the amount of data being transferred over the wire between Elasticsearch and the Kibana server as well as between the Kibana server and the browser.
## With this PR
This PR makes the Kibana API call to fetch data necessary for **initially** rendering the pipeline — that is, with the detail drawer closed — much lighter. When the user clicks on a vertex in a pipeline, a second API call is then made to fetch data necessary for the detail drawer.
## Gains, by the numbers
Based on a simple, 1-input, 1-filter, and 1-output pipeline.
* Before this PR, the Elasticsearch `logstash_stats` API responses (multiple calls were made using the `composite` aggregation over the `date_histogram` aggregation) generated a total of 1228 aggregation buckets (before any `filter_path`s were applied but across all `composite` "pages"). With this PR, the single `logstash_stats` API response (note that this is just for the initial rendering of the pipeline, with the detail drawer closed) generated 12 buckets (also before any `filter_path`s were applied). That's a **99.02% reduction** in number of buckets.
* Before this PR, the Elasticsearch `logstash_stats` API responses added up to 70319 bytes. With this PR, the single `logstash_stats` API response for the same pipeline is 746 bytes. That's a **98.93% reduction** in size.
* Before this PR, the Elasticsearch `logstash_state` API response was 7718 bytes. With this PR, the API response for the same pipeline is 2328 bytes. That's a **69.83% reduction** in size.
* Before this PR the Kibana API response was 51777 bytes. With this PR, the API response for the same pipeline is 2567 bytes (again, note that this is just for the initial rendering of the pipeline, with the detail drawer closed). That's a **95.04% reduction** in size.
* [Maps] split settings into layer and source panels (#33788)
* [Maps] split settings into layer and source panels
* fix SCSS import
* [env] exit if starting as root (#21563)
* [env] exit if starting as root
* fix windows
* s/--allow-root
* Typescript sample panel action (#33602)
* Typescript sample panel action
* Update EUI version to match main cabana version
* update yarn.lock
* add back typings include
* use correct relative path
* Home page "recent links" should communicate saved object type #21896 (#33694)
* adds object type for screen order
* adds object type for pointer hovering
* Update src/legacy/ui/public/chrome/directives/header_global_nav/components/header.tsx
Co-Authored-By: rockfield <philipp.b@ya.ru>
* Core task manager (#23555)
* Initial stab at core task manager logic
* Update task_manager readme
* Update task manager readme
* Add cancelable helper package, OSS, and general purpose, but
will be useful for writing cancelable x-pack tasks.
* Make the cancellable package promise-compliant
* Update task manager readme w/ reference to cancellable
* Change pool from lazy to eager, add support for sub-pools per task type.
* Move cancellable to task_manager, and typescriptify it.
* Working proof of concept for task manager. Still have lots to do:
clean up, tests, comments, validations, assertions, etc.
* Add pagination support to task manager fetch
* Move task manager to OSS
* Remove task manager reference from x-pack
* Make task_manager a valid core plugin
* Modify how task resource allocation works
* Remove the special case 'max' workers occupied value
* Remove x-pack package.json changes
* Make taskDefinitions a part of uiExports
* Make task docs saved-object compliant.
* Add kbnServer to the task context.
* Allow tasks to have a void / undefined return type
* revert x-pack change
* move cancellable to src/utils
* move to src/server
* use afterPluginsInit hook
* task_manager.ts rename
* add a wrapper with a setClient method
* Add tests for task runner
* Break task_pool into smaller, testable pieces
* return raw task doc fields for calling code
* remove todo comment
* helper module for default client
- setClient takes a callback fn
* fix misidentified field name
* fix rest args warning
* flatten task_pool
* remove cancellable
* return raw task doc fields for calling code
* remove todo comment
* helper module for default client
- setClient takes a callback fn
* fix rest args warning
* typescript fixes
* roll back setClient takes a callback
* createTaskRunner returns an object with run/cancel functions
* Test task store, tweak task poller tests
* Rename num_workers to max_workers
* Tweak task pool method names
* Fix cancellable test, and ts errors
* Rename doc to _doc
* Fix the doc type in the task manager index mappings
* Make task costs configurable in kibana.yml
* fix a tslint warning
* TaskManager.afterPluginsInit replaces circuitous stuff
* addMiddleware, implement beforeSave
* wip
* comment
* run context stuff
* pretty fix
* comments
* lint/pretty fixes
* safeties in case they don't define both beforeRun and beforeSave
* task runner test constructor opts fix
* Add task_manager integration tests
* FIx readme mistakes, fix schedule signature
* Fix integration tests
* Add blurb about middleware
* make a libs directory for the small things
* test for extractTaskDefinitions
* unit test for middleware
* Comment, refactor, and test task_manager
* Touch up comments
* Make task store throw error if given an unknown task type
* Fix sample task definition in readme
* Make task manager toggle on / off based on Elasticsearch connectivity
* Prevent task manager from crashing Kibana if
elasticsearch plugin is disabled
* Fix task manager signature
* Move the task manager into the task manager mixin,
fix tests. It's an uber integration object anyway, so
it seemed to make sense to keep it in one place.
* Fix task manager integration tests
* Update the task manager index creation to use auto_expand_replicas
* Fix task manager integration tests
* Fix task manager integration tests
* Fix the overzealous deletion of .kibana during tests
* Core task manager
* Allow hardcoded ID passed in
* comments
* don't deconstruct type and task for payload
* remove uiExport support
* move task_manager to x-pack/plugins
* expose as client on the plugin namespace
* fix tests
* move plugin api integration tests
* roll back readme sample task definition example
* fix sample plugin link
* server.taskManager
* sanitizeTaskDefinitions
* fix integration tests
* sanitize rename
* assert unintialized and check for duplicates in registerTaskDefinitions
* Remove unnecessary defaults from task_manager.test.ts
* Remove task manager from OSS
* Remove task manager from src
* Tidy up task_manager unit tests around elasticsearch status changes
* Integration test for specifying ID in scheduling
* Task_manager init -> preInit to allow other plugins to consume
taskManager in their init.
* Remove task manager integration tests from OSS
* spelling
* readme fix
* fix test code impacted by hapi upgrade
* Task Manager index creation changes (#24542)
* Uses putTemplate for task manager mappings and index settings.
* Removes create option.
* Opposite day, the test catches up to the code changes.
* Ignores index unavailable during searches.
* Improve taskRunner's processResult (#24880)
* Set task state to idle after run, and add failed state
* fix tests
* Test alerting demo (#25136)
* Ensures that task store init gets called before schedule.
* Removes unused option for debugging purposes.
* Fix unit tests because a second callCluster was made.
* Task manager starts sanely.
Does not preInit
Will not allow definitions after initialization
Creates store immediately.
Modifies store after all plugins have initialized
Adds static tasks that will be defined by plugins.
* Task manager should not allow operations before initialization.
* Attempts to fix runner tests.
* Fixes unit test contract with APIs.
* Removes unused type definitions.
* Removes unused package json.
* Removes unused import type.
* Removes unnecessary async applied to a function.
* Returns diferently if task store has already initialized.
* Explains how to add static task instances to task definitions. (#25880)
* Tasks cannot be scheduled until task manager is initialized.
* Adds task manager api integration tests to ci group4.
* Context of describe test must be the test framework object.
* Update src/es_archiver/lib/indices/kibana_index.js
regex update to actually disallow non-kibana system indices
Co-Authored-By: tsullivan <tsullivan@users.noreply.github.com>
* verify fillPool exception passing
* readme update about max_workers/numWorkers
* change forEach to reduce
* use public interfaces in internal method
* replace getMaxAttempts with public readonly maxAttempts
* Update x-pack/plugins/task_manager/task_store.ts
`throw new Error` and initializattion spell fix
Co-Authored-By: tsullivan <tsullivan@users.noreply.github.com>
* min = 1 for max_workers
Co-Authored-By: tsullivan <tsullivan@users.noreply.github.com>
* timeOut => timeout
* min 1
* scope as an array
* no retries
Co-Authored-By: tsullivan <tsullivan@users.noreply.github.com>
* ConcreteTaskInstance is a TaskInstance
* remove guard per joi logic
* more return types for functions
* more comments around incremental back-off
Co-Authored-By: tsullivan <tsullivan@users.noreply.github.com>
* throw error instead of return undefined
* poll_interval min 1000
* avoid handling err.stack directly
* break up processResult
* fix a few runtime issues
* only fetch idle tasks
* remove check for status idle
* always return a state, and when there is an error return the state we had at beforeRun
* check isStarted before attemptWork
Co-Authored-By: tsullivan <tsullivan@users.noreply.github.com>
* ts fix
