[DOCS] improving production documentation for task manager and alerting (#87484) (#87502)

* improving production documentation for task manager and alerting

* calling it Kibana Task Manager to be more explicit

* addressing PR feedback

docs/settings/settings-xkb.asciidoc

@@ -19,5 +19,6 @@ include::logs-ui-settings.asciidoc[]
 include::ml-settings.asciidoc[]
 include::reporting-settings.asciidoc[]
 include::spaces-settings.asciidoc[]
+include::task-manager-settings.asciidoc[]
 include::i18n-settings.asciidoc[]
 include::fleet-settings.asciidoc[]

docs/settings/task-manager-settings.asciidoc

@@ -0,0 +1,32 @@
+[role="xpack"]
+[[task-manager-settings-kb]]
+=== Task Manager settings in {kib}
+++++
+<titleabbrev>Task Manager settings</titleabbrev>
+++++
+
+Task Manager runs background tasks by polling for work on an interval.  You can configure its behavior to tune for performance and throughput.
+
+[float]
+[[task-manager-settings]]
+==== Task Manager settings 
+
+[cols="2*<"]
+|===
+| `xpack.task_manager.max_attempts`
+  | The maximum number of times a task will be attempted before being abandoned as failed.  Defaults to 3.
+
+| `xpack.task_manager.poll_interval`
+  | How often, in milliseconds, the task manager will look for more work.  Defaults to 3000 and cannot be lower than 100.
+
+| `xpack.task_manager.request_capacity`
+  | How many requests can Task Manager buffer before it rejects new requests.  Defaults to 1000.
+
+| `xpack.task_manager.index`
+  | The name of the index used to store task information.  Defaults to `.kibana_task_manager`.
+
+  | `xpack.task_manager.max_workers`
+  | The maximum number of tasks that this Kibana instance will run simultaneously.  Defaults to 10.
+  
+
+|===

docs/user/alerting/alerting-getting-started.asciidoc

@@ -57,7 +57,7 @@ Alert schedules are defined as an interval between subsequent checks, and can ra
 
 [IMPORTANT]
 ==============================================
-The intervals of alert checks in {kib} are approximate, their timing of their execution is affected by factors such as the frequency at which tasks are claimed and the task load on the system. See <<alerting-scale-performance>> for more information. 
+The intervals of alert checks in {kib} are approximate, their timing of their execution is affected by factors such as the frequency at which tasks are claimed and the task load on the system. See <<alerting-production-considerations>> for more information. 
 ==============================================
 
 [float]

docs/user/alerting/alerting-production-considerations.asciidoc

@@ -1,10 +1,10 @@
 [role="xpack"]
-[[alerting-scale-performance]]
-== Scale and performance
+[[alerting-production-considerations]]
+== Production considerations
 
-{kib} alerting run both alert checks and actions as persistent background tasks. This has two major benefits:
+{kib} alerting run both alert checks and actions as persistent background tasks managed by the Kibana Task Manager. This has two major benefits:
 
-* *Persistence*: all task state and scheduling is stored in {es}, so if {kib} is restarted, alerts and actions will pick up where they left off. 
+* *Persistence*: all task state and scheduling is stored in {es}, so if {kib} is restarted, alerts and actions will pick up where they left off.  Task definitions for alerts and actions are stored in the index specified by `xpack.task_manager.index` (defaults to `.kibana_task_manager`).  It is important to have at least 1 replica of this index for production deployments, since if you lose this index all scheduled alerts and actions are also lost.
 * *Scaling*: multiple {kib} instances can read from and update the same task queue in {es}, allowing the alerting and action load to be distributed across instances. In cases where a {kib} instance no longer has capacity to run alert checks or actions, capacity can be increased by adding additional {kib} instances.
 
 [float]
@@ -12,17 +12,19 @@
 
 {kib} background tasks are managed by:
 
-* Polling an {es} task index for overdue tasks at 3 second intervals.
+* Polling an {es} task index for overdue tasks at 3 second intervals.  This interval can be changed using the `xpack.task_manager.poll_interval` setting.
 * Tasks are then claiming them by updating them in the {es} index, using optimistic concurrency control to prevent conflicts. Each {kib} instance can run a maximum of 10 concurrent tasks, so a maximum of 10 tasks are claimed each interval. 
 * Tasks are run on the {kib} server. 
 * In the case of alerts which are recurring background checks, upon completion the task is scheduled again according to the <<defining-alerts-general-details, check interval>>.
 
 [IMPORTANT]
 ==============================================
-Because tasks are polled at 3 second intervals and only 10 tasks can run concurrently per {kib} instance, it is possible for alert and action tasks to be run late. This can happen if: 
+Because by default tasks are polled at 3 second intervals and only 10 tasks can run concurrently per {kib} instance, it is possible for alert and action tasks to be run late. This can happen if: 
 
 * Alerts use a small *check interval*. The lowest interval possible is 3 seconds, though intervals of 30 seconds or higher are recommended.
 * Many alerts or actions must be *run at once*. In this case pending tasks will queue in {es}, and be pulled 10 at a time from the queue at 3 second intervals. 
 * *Long running tasks* occupy slots for an extended time, leaving fewer slots for other tasks. 
 
+For details on the settings that can influence the performance and throughput of Task Manager, see {task-manager-settings}.
+
 ==============================================

docs/user/alerting/defining-alerts.asciidoc

@@ -26,7 +26,7 @@ image::images/alert-flyout-general-details.png[alt='All alerts have name, tags,
 
 Name::      The name of the alert. While this name does not have to be unique, the name can be referenced in actions and also appears in the searchable alert listing in the management UI. A distinctive name can help identify and find an alert.
 Tags::      A list of tag names that can be applied to an alert. Tags can help you organize and find alerts, because tags appear in the alert listing in the management UI which is searchable by tag.
-Check every::      This value determines how frequently the alert conditions below are checked. Note that the timing of background alert checks are not guaranteed, particularly for intervals of less than 10 seconds. See <<alerting-scale-performance>> for more information.
+Check every::      This value determines how frequently the alert conditions below are checked. Note that the timing of background alert checks are not guaranteed, particularly for intervals of less than 10 seconds. See <<alerting-production-considerations>> for more information.
 Notify every::      This value limits how often actions are repeated when an alert instance remains active across alert checks. See <<alerting-concepts-suppressing-duplicate-notifications>> for more information.
 
 [float]

docs/user/alerting/index.asciidoc

@@ -2,4 +2,4 @@ include::alerting-getting-started.asciidoc[]
 include::defining-alerts.asciidoc[]
 include::action-types.asciidoc[]
 include::alert-types.asciidoc[]
-include::alerting-scale-performance.asciidoc[]
+include::alerting-production-considerations.asciidoc[]