* bump to a pre-8.0 version
* export KibanaClient from /lib sub-folder
* workaround the problem of the absence of estypes
* update es client usage in pacakges
* export estypes from another path
* import errors from root
* import errors from root 2
* update transport import
* update import path for /api/types
* update import path for /api/types
* import errors from top export
* use TransportResult instead if ApiResponse
* fix errors in client_config
* fix src/core/server/saved_objects/migrationsv2/actions/integration_tests/actions.test.ts
* use KibanaClient in mock. we dont export the original Client
* fix client mocks
* fix errors on SO
* fix remaining core errors
* update estype import path
* fix errors in data plugin
* fix data_views
* fix es_ui_shared
* fix errors in interactive_setup
* fix errors in ./test folder
* add @elastic/transport to the runtime deps
* fix errors in packages
* fix erros in src/core
* fix errors in test/
* fix an error in actions plugin
* woraround and fix errors in APM plugin
* fix errors in canvas
* fix errors in event_log
* fix errors in fleet
* fix errors in ILM
* fix errors in infra
* fix errors in ingest_pipeline
* fix errors in lens
* fix errors in license_management
* fix errors in licensing
* fix errors in logstash
* fix errors in ml
* fix errors in monitoring
* fix errors in observability
* fix errors in rule_registry
* fix errors in reporting
* fix errors in rule_registry
* fix errors in security
* fix errors in security_solution
* fix errors in snapshot_restore
* fix errors in transform
* fix errors in UA
* fix errors in uptime
* fix errors in x-pack/test
* fix eslint errors
* fix new errors
* use default HTTP Connection. Undici does not support agent config options keepAlive and maxSockets
* create does not accept require_alias option
* update deps
* use transport types exported from ES client package
* fix ErrorCause | string errors
* do not use enum
* fix errors in data plugin
* update x-pack code
* fix transport
* fix apm search request
* do not crash on reporting
* fix kbn-test build
* mute reporting error to start
* fix ftr build
* another attempt
* update import path
* address or mute new errors
* REMOVE me. pin transport version temporarily.
* remove deep imports from transport package
* fix jest crash
* fix product check tests
* remove unnecessary ts-expect-error
* fix a few failed unit tests
* bump to canary 24
* remove unnecessary ts-expect-error
* remove dependency on transport
* fix types in tests
* mute errors in xpack tests
* product check doesn;t spam in logs anymore
* filterPath --> filter_path
* ignoreUnavailable --> ignore_unavailable
* ignoreUnavailable --> ignore_unavailable
* trackScores --> track_scores
* trackTotalHits --> track_total_hits
* fix es-arcives
* fix data plugin crashes
* fix watcher test utils
* rollback unnecessary changes
* fix another problem in es-archiver
* fix scroll. for whatever reason scroll fails when request scroll_id in body
* add meta: true in kbn-securitysolution-es-utils
* bump client to canary 25
* fix errors in accordance with the es client spec
* update securityscolution-es-utils
* unify scroll api in reporting and fix tests
* fix unit tests in watcher
* refactor APM to abort request with AbortController API
* fix missing es client calls in tests
* fix missing meta in detection engine FTR tests
* fix another bunch of errors in js tests
* fix wrong coercion
* remove test-grep pattern
* fix apm unit test
* rename terminateAfter to terminate_after in infra plugin
* rename terminateAfter to terminate_after in uptime plugin
* rename terminateAfter to terminate_after in apm plugin
* fix security roles FTR tests
* fix reference
* fix post_privilidges test
* fix post_privilidges
* bump client to 26
* add meta for index_management test helpers
* remove ts-expect-error caused by bad type in reason
* bump client to 27
* REMOVE me. workaround until fixed in the es client
* fix incorrect type casting
* swtich from camelCase params
* use `HttpConnection` for FTR-related clients
* bump client to 29
* Revert "REMOVE me. workaround until fixed in the es client"
This reverts commit c038850c09.
* fix new util
* revert repository changes
* do not crash if cannot store event_loop data
* fix new estypes imports
* fix more types
* fix security test types and add ts-ignore for custom ES client
* fix more estypes imports
* yet more ts violations
* line by line fixing is hard
* adapt `evaluateAlert` from infra as it's also used from FTR tests
* use convertToKibanaClient in FTR test instead of meta:true in plugin code
* migrate from deprecated API in fleet
* fix intergration tests
* fix fleet tests
* fix another fleet test
* fix more tests
* let's call it a day
* Removes custom header check on 404 responses, includes es client ProductNotSupportedError in EsUnavailableError conditional (#116029)
* Removes custom header check on 404 responses, includes es client ProductNotSupportedError in EsUnavailableError conditional
* Updates proxy response integration test
* disable APM until compatible with client v8
* skip async_search FTR test
* use kbnClient in integration tests
* bump version to 29
* bump to 30
* have configureClient return a KibanaClient instead of Client, remove resolved violations.
* bump to 31
* bump to 31
* Revert "bump to 31"
This reverts commit 5ac713e640.
* trigger stop to unusubscribe
* update generated docs
* remove obsolete test
* put "as" back
* cleanup
* skip test
* remove new type errors in apm package
* remove ErrorCause casting
* update a comment
* bump version to 32
* remove unnecessary ts-expect-error in apm code
* update comments
* update to client v33
* remove outdated type definition
* bump to 34 without params mutation
* unskip the test that should not fail anymore
* remove unnecessary ts-expect-error comments
* update to v35. body can be string
* move `sort` to body and use body friendly syntax
* fix a failing test. maps register the same SO that has been already registered by home
Co-authored-by: pgayvallet <pierre.gayvallet@gmail.com>
Co-authored-by: Christiane (Tina) Heiligers <christiane.heiligers@elastic.co>
* [Alerting] Active alerts do not recover after re-enabling a rule
* created reusable lib file for generating event log object
* comment fix
* fixed tests
* fixed tests
* fixed typecheck
* fixed due to comments
* Apply suggestions from code review
Co-authored-by: ymao1 <ying.mao@elastic.co>
* fixed due to comments
* fixed due to comments
* fixed due to comments
* fixed tests
* Update disable.ts
* Update disable.ts
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: ymao1 <ying.mao@elastic.co>
* Adding execution duration to get alert instance summary
* Showing execution duration on rule details view
* Fixing unit tests
* Updating to match new mockup
* Fixing types
* Fixing functional test
* Removing unneeded max and min and adding tests
* Removing unneeded max and min and adding tests
* Fixing functional test
* Adding left axis
* PR feedback
* Reducing chart height
* PR feedback
* PR feedback
* PR feedback
* Adding last duration to execution status and returning in alerting routes
* Fixing types
* Adding helper function to format duration
* Returning rule timeout value in list rules API
* Updating rules table to add duration column and tweaks to match mockup
* Updating rules table to add duration column and tweaks to match mockup
* i18n fix
* Only showing duration warning if duration is long
* Unit tests
* i18n fix
* Fixing functional test
* Aligning warning icon
* Reset last duration when rule is disabled then reenabled
* Fixing functional test
* Fixing functional test
* Restoring muted badge. Fixing scss
* Dont show muted badge if rule is disabled
* Moving disabled icontip to right of rule name
* Updating tooltips
* Updating last run
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
Fixes https://github.com/elastic/kibana/issues/113276
* Migrates the legacy `siem.notifications` "ruleAlertId" to be within the references array
* Adds code to serialize and de-serialize "ruleAlertId" from the saved object references array
* Adds migration code to `kibana-alerting` to migrate on startup
* Adds `legacy_saved_object_references/README.md` which describes how to test and what those files are for.
* Updates earlier similar `signals/saved_object_references/README.md` after reviewing it during my work
* Names these files the format of `legacy_foo` since this is all considered legacy work and will be removed once the legacy notification system is removed after customers have migrated.
* Adds unit tests
* Adds 2e2 tests
We only migrate if we find these conditions and cases:
* "ruleAlertId" is not `null`, `undefined` or malformed data
* The"ruleAlertId" references do not already have an exceptionItem reference already found within it.
We migrate on the common use case:
* "ruleAlertId" exists and is a string
We do these additional (mis-use) cases and steps as well. These should NOT be common things that happen but we safe guard for them here:
* If the migration is run twice we are idempotent and do NOT add duplicates or remove items.
* If the migration was partially successful but re-run a second time, we only add what is missing. Again no duplicates or removed items should occur.
* If the saved object references already exists and contains a different or foreign value, we will retain the foreign reference(s) and still migrate.
Before migration you should see data structures like this if you query:
```json
# Get the alert type of "siem-notifications" which is part of the legacy system.
GET .kibana/_search
{
"query": {
"term": {
"alert.alertTypeId": "siem.notifications"
}
}
}
```
```json
"data..omitted": "data..omitted",
"params" : {
"ruleAlertId" : "933ca720-1be1-11ec-a722-83da1c22a481" <-- Pre-migration we had this Saved Object ID which is not part of references array below
},
"actions" : [
{
"group" : "default",
"params" : {
"message" : "Hourly\nRule {{context.rule.name}} generated {{state.signals_count}} alerts"
},
"actionTypeId" : ".slack",
"actionRef" : "action_0" <-- Pre-migration this is correct as this work is already done within the alerting plugin
},
"references" : [
{
"id" : "879e8ff0-1be1-11ec-a722-83da1c22a481",
"name" : "action_0", <-- Pre-migration this is correct as this work is already done within the alerting plugin
"type" : "action"
}
]
],
"data..omitted": "data..omitted",
```
After migration you should see data structures like this:
```json
"data..omitted": "data..omitted",
"params" : {
"ruleAlertId" : "933ca720-1be1-11ec-a722-83da1c22a481" <-- Post-migration this is not used but rather the serialized version references is used instead.
},
"actions" : [
{
"group" : "default",
"params" : {
"message" : "Hourly\nRule {{context.rule.name}} generated {{state.signals_count}} alerts"
},
"actionTypeId" : ".slack",
"actionRef" : "action_0"
},
"references" : [
{
"id" : "879e8ff0-1be1-11ec-a722-83da1c22a481",
"name" : "action_0",
"type" : "action"
},
{
"id" : "933ca720-1be1-11ec-a722-83da1c22a481", <-- Our id here is preferred and used during serialization.
"name" : "param:alert_0", <-- We add the name of our reference which is param:alert_0 similar to action_0 but with "param"
"type" : "alert" <-- We add the type which is type of alert to the references
}
]
],
"data..omitted": "data..omitted",
```
## Manual testing
There are e2e and unit tests but for any manual testing or verification you can do the following:
If you have a 7.14.0 system and can migrate it forward that is the most straight forward way to ensure this does migrate correctly and forward. You should see that the legacy notification system still operates as expected.
If you are a developer off of master and want to test different scenarios then this section is for below as it is more involved and harder to do but goes into more depth:
* Create a rule and activate it normally within security_solution
* Do not add actions to the rule at this point as we are exercising the older legacy system. However, you want at least one action configured such as a slack notification.
* Within dev tools do a query for all your actions and grab one of the `_id` of them without their prefix:
```json
# See all your actions
GET .kibana/_search
{
"query": {
"term": {
"type": "action"
}
}
}
```
Mine was `"_id" : "action:879e8ff0-1be1-11ec-a722-83da1c22a481"`, so I will be copying the ID of `879e8ff0-1be1-11ec-a722-83da1c22a481`
Go to the file `detection_engine/scripts/legacy_notifications/one_action.json` and add this id to the file. Something like this:
```json
{
"name": "Legacy notification with one action",
"interval": "1m", <--- You can use whatever you want. Real values are "1h", "1d", "1w". I use "1m" for testing purposes.
"actions": [
{
"id": "879e8ff0-1be1-11ec-a722-83da1c22a481", <--- My action id
"group": "default",
"params": {
"message": "Hourly\nRule {{context.rule.name}} generated {{state.signals_count}} alerts"
},
"actionTypeId": ".slack" <--- I am a slack action id type.
}
]
}
```
Query for an alert you want to add manually add back a legacy notification to it. Such as:
```json
# See all your siem.signals alert types and choose one
GET .kibana/_search
{
"query": {
"term": {
"alert.alertTypeId": "siem.signals"
}
}
}
```
Grab the `_id` without the alert prefix. For mine this was `933ca720-1be1-11ec-a722-83da1c22a481`
Within the directory of detection_engine/scripts execute the script:
```json
./post_legacy_notification.sh 933ca720-1be1-11ec-a722-83da1c22a481
{
"ok": "acknowledged"
}
```
which is going to do a few things. See the file `detection_engine/routes/rules/legacy_create_legacy_notification.ts` for the definition of the route and what it does in full, but we should notice that we have now:
Created a legacy side car action object of type `siem-detection-engine-rule-actions` you can see in dev tools:
```json
# See the actions "side car" which are part of the legacy notification system.
GET .kibana/_search
{
"query": {
"term": {
"type": {
"value": "siem-detection-engine-rule-actions"
}
}
}
}
```
But more importantly what the saved object references are which should be this:
```json
# Get the alert type of "siem-notifications" which is part of the legacy system.
GET .kibana/_search
{
"query": {
"term": {
"alert.alertTypeId": "siem.notifications"
}
}
}
```
If you need to ad-hoc test what happens when the migration runs you can get the id of an alert and downgrade it, then
restart Kibana. The `ctx._source.references.remove(1)` removes the last element of the references array which is assumed
to have a rule. But it might not, so ensure you check your data structure and adjust accordingly.
```json
POST .kibana/_update/alert:933ca720-1be1-11ec-a722-83da1c22a481
{
"script" : {
"source": """
ctx._source.migrationVersion.alert = "7.15.0";
ctx._source.references.remove(1);
""",
"lang": "painless"
}
}
```
If you just want to remove your your "param:alert_0" and it is the second array element to test the errors within the console
then you would use
```json
POST .kibana/_update/alert:933ca720-1be1-11ec-a722-83da1c22a481
{
"script" : {
"source": """
ctx._source.references.remove(1);
""",
"lang": "painless"
}
}
```
Check your log files and should see errors about the saved object references missing until you restart Kibana. Once you restart then it will migrate forward and you will no longer see errors.
### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
* Try and add logging here
* Fix linting
* Only this test
* Better logging
* More debugging
* More debug
* Try something different
* Better way to do the test
* Get this PR ready
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Initial commit of serverType in email connector config
* Fleshing in route to get well known email service configs from nodemailer
* Adding elastic cloud to well known server type
* Cleaning up email constants and allowing for empty selection
* Showing error if user doesn't select server type
* Adding hook for setting email config based on server type
* Adding tests and making sure settings are not overwritten on edit
* Fixing functional test
* Adding migration
* Adding functional test for migration
* Repurposing service instead of adding serverType
* Cleanup
* Disabling host/port/secure form fields when settings retrieved from API
* Updating docs for service
* Filtering options based on whether cloud is enabled
* Initialize as disabled
* Fixing types
* Update docs/management/connectors/action-types/email.asciidoc
Co-authored-by: David Kilfoyle <41695641+kilfoyle@users.noreply.github.com>
Co-authored-by: David Kilfoyle <41695641+kilfoyle@users.noreply.github.com>
* Exposing preconfigured connectors through actions setup contract
* Adding stub for migration using preconfigured connectors
* Adding isPreconfigured fn to actions client
* Updating rules client logic to not extract predefined connector ids
* Functional tests
* Adding migration
* Adding functional test for migration
* Adding functional test for migration
* Adding note to docs about referenced_by_count if is_preconfigured
* Fixing functional test
* Changing to isPreconfiguredConnector fn in actions plugin setup contract
* Update docs/api/actions-and-connectors/get_all.asciidoc
Co-authored-by: Mike Côté <mikecote@users.noreply.github.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Mike Côté <mikecote@users.noreply.github.com>
* Extracting saved object references before saving action_task_params saved object
* Injecting saved object ids from references when reading action_task_param
* Adding migration
* Adding unit test for migrations
* Not differentiating between preconfigured or not
* Adding functional test for migration
* Skip extracting action id if action is preconfigured
* Only migrating action task params for non preconfigured connectors
* Simplifying related saved objects
* Fixing functional test
* Fixing migration
* Javascript is sometimes magical
* Updating functional test
* PR feedback
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Initial commit
* Rules client to set legacyId when creating a rule in < 8.0.0
* Set legacyId to null on export, change empty state of legacyId from undefined to null
* Fix broken tests
* Refactor create.test.ts to avoid increasing file size
* Fix broken test
## Summary
This is part 2 to addressing the issue seen here: #101975 (Part 1 #107064)
This adds the alerting migration scripts and unit tests for exception list containers on Kibana startup for `7.15.0`
We only migrate if we find these conditions and cases:
- `exceptionLists` are an `array` and not `null`, `undefined`, or malformed data.
- The exceptionList item is an `object` and its `id` is a `string` and not `null`, `undefined`, or malformed data
- The existing references do not already have an exceptionItem reference already found within it.
We migrate on the common use case
- The saved object references do not exist but we have exceptionList items with the id's to create the saved object references, 👍 so we migrate.
- The alert contains no exception list items, in which case we have nothing to migrate
We do these additional (mis-use) cases and steps as well. These should _NOT_ be common things that happen but we safe guard for them here:
- If the migration is run twice we are idempotent and do _NOT_ add duplicates list items or remove items.
- If the migration was partially successful but re-run a second time, we only add what is missing. Again no duplicates or removed items should occur.
- If the `exceptionLists` contains invalid data shape or not enough information to migrate, we filter it out and ignore it
- If the saved object references already exists and contains a different or foreign value, we will retain the foreign reference(s) and still migrate.
## Manual testing
There are unit tests but for any manual testing or verification you can do the following:
Create a few alerts through the `security_solution` application with exception lists
<img width="1775" alt="Screen Shot 2021-08-11 at 5 42 31 PM" src="https://user-images.githubusercontent.com/1151048/129117377-61b17fcf-ad01-4405-bbfe-42d97a6f7654.png">
Use the dev tools to de-migrate as well as to test end to end like so:
```json
# First get an "_id" with an exceptions list like below. Mine I found was: "alert:38482620-ef1b-11eb-ad71-7de7959be71c":
GET .kibana/_search
{
"query": {
"terms": {
"alert.alertTypeId": [
"siem.signals"
]
}
},
"size": 10000
}
```
With Kibana running downgrade and remove the references as a test:
```json
# Set saved object array references as empty arrays and set our migration version to be 7.14.0
POST .kibana/_update/alert:38482620-ef1b-11eb-ad71-7de7959be71c
{
"script" : {
"source": """
ctx._source.migrationVersion.alert = "7.14.0";
ctx._source.references = []
""",
"lang": "painless"
}
}
# Double check the references is empty and the version is 7.14.0
GET .kibana/_doc/alert:38482620-ef1b-11eb-ad71-7de7959be71c
```
Reload the alert in the `security_solution` and notice you get these errors until you restart Kibana to cause a migration moving forward
```sh
server log [17:35:16.914] [error][plugins][securitySolution] The saved object references were not found for our exception list when we were expecting to find it. Kibana migrations might not have run correctly or someone might have removed the saved object references manually. Returning the last known good exception list id which might not work. exceptionItem with its id being returned is: {"list_id":"endpoint_list","namespace_type":"agnostic","id":"endpoint_list","type":"endpoint"}
server log [17:35:16.914] [error][plugins][securitySolution] Cannot get a saved object reference using an index which is larger than the saved object references. Index is:1 which is larger than the savedObjectReferences:[]
server log [17:35:16.915] [error][plugins][securitySolution] The saved object references were not found for our exception list when we were expecting to find it. Kibana migrations might not have run correctly or someone might have removed the saved object references manually. Returning the last known good exception list id which might not work. exceptionItem with its id being returned is: {"list_id":"cd152d0d-3590-4a45-a478-eed04da7936b","namespace_type":"single","id":"50e3bd70-ef1b-11eb-ad71-7de7959be71c","type":"detection"}
server log [17:35:16.940] [error][plugins][securitySolution] The saved object references were not found for our exception list when we were expecting to find it. Kibana migrations might not have run correctly or someone might have removed the saved object references manually. Returning the last known good exception list id which might not work. exceptionItem with its id being returned is: {"list_id":"endpoint_list","namespace_type":"agnostic","id":"endpoint_list","type":"endpoint"}
server log [17:35:16.940] [error][plugins][securitySolution] Cannot get a saved object reference using an index which is larger than the saved object references. Index is:1 which is larger than the savedObjectReferences:[]
server log [17:35:16.940] [error][plugins][securitySolution] The saved object references were not found for our exception list when we were expecting to find it. Kibana migrations might not have run correctly or someone might have removed the saved object references manually. Returning the last known good exception list id which might not work. exceptionItem with its id being returned is: {"list_id":"cd152d0d-3590-4a45-a478-eed04da7936b","namespace_type":"single","id":"50e3bd70-ef1b-11eb-ad71-7de7959be71c","type":"detection"}
```
Restart Kibana and you should no longer have errors in the Kibana console.
If you do this query in dev tools
```json
# Check that the `migrationVersion` is `7.15.0` and that we have a `references` array filled out with the correct structure
GET .kibana/_doc/alert:38482620-ef1b-11eb-ad71-7de7959be71c
```
You should notice that you now have a `references` array filled out:
```json
"references" :
[
{
"name" : "param:exceptionsList_0",
"id" : "endpoint_list",
"type" : "exception-list-agnostic"
},
{
"name" : "param:exceptionsList_1",
"id" : "50e3bd70-ef1b-11eb-ad71-7de7959be71c",
"type" : "exception-list"
}
],
"migrationVersion" : {
"alert" : "7.15.0"
}
```
For testing [idempotentence](https://en.wikipedia.org/wiki/Idempotence)
Run just this to downgrade and restart Kibana and you should notice on a GET that we do not have anything extra in the references array:
```json
# Set our migration version to be 7.14.0 only
POST .kibana/_update/alert:38482620-ef1b-11eb-ad71-7de7959be71c
{
"script" : {
"source": """
ctx._source.migrationVersion.alert = "7.14.0";
""",
"lang": "painless"
}
}
# Double check the `references` is still there, and we do not get errors or changes to `references` after we restart Kibana
GET .kibana/_doc/alert:38482620-ef1b-11eb-ad71-7de7959be71c
```
For testing foreign keys:
```json
# Set saved object array references to foreign keys and set our migration version to be 7.14.0
POST .kibana/_update/alert:38482620-ef1b-11eb-ad71-7de7959be71c
{
"script" : {
"source": """
ctx._source.migrationVersion.alert = "7.14.0";
ctx._source.references = [["name" : "foreign", "id" : "123", "type" : "some-type"]];
""",
"lang": "painless"
}
}
```
Restart, ensure no errors in Kibana console and do a get call to ensure we have the foreign mixed with valid values:
```json
GET .kibana/_doc/alert:38482620-ef1b-11eb-ad71-7de7959be71c
```
Should return this data:
```json
"type" : "alert",
"references" :
[
{
"name" : "foreign",
"id" : "123",
"type" : "some-type"
},
{
"name" : "param:exceptionsList_0",
"id" : "endpoint_list",
"type" : "exception-list-agnostic"
},
{
"name" : "param:exceptionsList_1",
"id" : "50e3bd70-ef1b-11eb-ad71-7de7959be71c",
"type" : "exception-list"
}
]
```
### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
* added ability to run ephemeral tasks
* fixed typing
* added typing on plugin
* WIP
* Fix type issues
* Hook up the ephemeral task into the task runner for actions
* Tasks can now run independently of one another
* Use deferred language
* Refactor taskParams slightly
* Use Promise.all
* Remove deferred logic
* Add config options to limit the amount of tasks executing at once
* Add ephemeral task monitoring
* WIP
* Add single test so far
* Ensure we log after actions have executed
* Remove confusing * 1
* Add logic to ensure we fallback to default enqueueing if the total actions is above the config
* Add additional test
* Fix tests a bit, ensure we log the alerting:actions-execute right away and the tests should listen for alerts:execute
* Better tests
* If the queue is at capacity, attempt to execute the ephemeral task as a regular action
* Ensure we run ephemeral tasks before to avoid them getting stuck in the queue
* Do not handle the promise anymore
* Remove unnecessary code
* Properly handle errors from ephemeral task lifecycle
* moved acitons domain out of alerting and into actions plugin
* Remove some tests
* Fix TS and test issues
* Fix type issues
* Fix more type issues
* Fix more type issues
* Fix jest tests
* Fix more jest tests
* Off by default
* Fix jest tests
* Update config for this suite too
* Start of telemetry code
* Fix types and add missing files
* Fix telemetry schema
* Fix types
* Fix more types
* moved load event emission to pollingcycle and added health stats on Ephemeral tasks
* Add more telemetry data based on new health metrics for the ephemeral queue
* Fix tests and types
* Add separate request capacity for ephemeral queue
* Fix telemetry schema and add tests for usage collection
* track polled tasks by persistence and use in capacity estimation instead of executions
* fixed typing
* Bump default capacity
* added delay metric to ephemeral stats
* Fix bad merge
* Fix tests
* Fix tests
* Fix types
* Skip failing tests
* Exclude ephemeral stats from capacity estimation tests
* PR feedback
* More PR feedback
* PR feedback
* Fix merge conflict
* Try fixing CI
* Fix broken lock file from merge
* Match master
* Add this back
* PR feedback
* Change to queue and add test
* Disable ephemeral queue in tests
* Updated desc
* Comment out ephemeral-specific tests tha require the entire test suite to support ephemeral tasks
* Add clarifying comment
Co-authored-by: Gidi Meir Morris <github@gidi.io>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Removing feature flag changes
* Adding isExportable flag to rule type definition
* Adding isExportable flag to rule type definition
* Adding isExportable flag to rule type definition
* Filtering rule on export by rule type isExportable flag
* Fixing types
* Adding docs
* Fix condition when exportCount is 0
* Unit test for fix condition when exportCount is 0
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
resolves#98634
This adds a new object property to the event log kibana object named
task, with two properties to track the time the task was scheduled to
run, and the delay between when it was supposed to run and when it
actually started. This task property is only added to the appropriate
events.
task: schema.maybe(
schema.object({
scheduled: ecsDate(),
schedule_delay: ecsNumber(),
})
),
Note that these changes were previously merged to master in https://github.com/elastic/kibana/pull/102252 which had to be reverted - this PR contains the same commits, plus some additional ones to resolve the tests that were broken during the bad merge.
resolves#98634
This adds a new object property to the event log kibana object named
task, with two properties to track the time the task was scheduled to
run, and the delay between when it was supposed to run and when it
actually started. This task property is only added to the appropriate
events.
task: schema.maybe(
schema.object({
scheduled: ecsDate(),
schedule_delay: ecsNumber(),
})
),
* Add event log entry when a rule starts executing
* Add in alertId
* Add more logging and fix tests
* Use existing startTiming logic
* Add additional tests
* WIP
* Storing start, duration and end in alert state
* Writing to event log
* Updating unit tests
* Adding unit tests
* Fixing uuid in tests
* Updating functional test
* Adding functional test
* Removing console logs
* Fixing unit tests
* PR fixes
* Removing uuid from alert information
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* fix(NA): windows ts_project outside sandbox compilation adding tsconfig paths for packages
* chore(NA): missing @kbn paths for node_modules so types can work
* chore(NA): missing @kbn paths for node_modules so types can work
* chore(NA): organizing deps on non ts_project packages
* chore(NA): change order to find @kbn packages on node_modules first
* chore(NA): add @kbn/expect typings setting on package.json
* chore(NA): fix typechecking
* chore(NA): add missing change on tsconfig file
* chore(NA): unblock windows build by not depending on the pkg_npm rule symlink in the package.json
* chore(NA): add missing depedencies on BUILD.bazel file for io-ts-list-types
* chore(NA): remove rootDirs configs
* chore(NA): change kbn/monaco targets order
* chore(NA): update kbn-monaco build
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
resolves https://github.com/elastic/kibana/issues/100607
This fixes a problem when very large parameters (over 32K bytes) are saved with
an alert. Before this fix, an error from elasticsearch would be thrown with
the following message, and a 400 returned from create (and presumably update).
Document contains at least one immense term in field=\"alert.params\"
(whose UTF8 encoding is longer than the max length 32766), all of which
were skipped.
After the fix, alerts with immense params can be saved and executed.
Note that the immense params will not be searchable, since they won't be indexed,
but that seems both unavoidable, and not a severe issue.
resolves: https://github.com/elastic/kibana/issues/80120
Adds a new Kibana configuration key xpack.actions.customHostSettings which
allows per-host configuration of connection settings for https and smtp for
alerting actions. Initially this is just for TLS settings, expandable to other
settings in the future.
The purpose of these is to allow customers to provide server certificates for
servers accessed by actions, whose certificate authority is not available
publicly. Alternatively, a per-server rejectUnauthorized: false configuration
may be used to bypass the verification step for specific servers, but require it
for other servers that do not have per-host customization.
Support was also added to allow per-host customization of ignoreTLS and
requireTLS flags for use with the email action.
* [Connectors][API] Updated connectors with enabledAfterImport flag
* fixed functional tests
* added new field to connectors API docs
* added update unit test
* fixed test
* renamed enableAfterImport to isMissingSecrets
* removed onExport
* revert the logic of true/false for isMissingSecrets
* fixed test
* fixed tests
* added unit test
* fixed docs
* fixed import text and button labels
* fixed import text
* fixed text
* Initial commit
* Add tests and support for concurrency
* Ability to disable functionality, use bulk APIs
* Fix type check
* Fix jest tests
* Cleanup
* Cleanup pt2
* Add unit tests
* Fix type check
* Fixes
* Update test failures
* Split schedule between cleanup and idle
* Add functional tests
* Add one more test
* Cleanup repeated code
* Remove duplicate actions plugin requirement
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Move current alert HTTP APIs to legacy folder (#93943)
* Move current HTTP APIs to legacy folder
* Rename BASE_ALERT_API_PATH to LEGACY_BASE_ALERT_API_PATH
* Fix failing tests and extra files
* Create new rule HTTP APIs (#93980)
* Move current HTTP APIs to legacy folder
* Rename BASE_ALERT_API_PATH to LEGACY_BASE_ALERT_API_PATH
* Fix failing tests and extra files
* Move current alert HTTP APIs to legacy folder (#93943)
* Move current HTTP APIs to legacy folder
* Rename BASE_ALERT_API_PATH to LEGACY_BASE_ALERT_API_PATH
* Fix failing tests and extra files
* Add necessary files
* Create rule route
* Get rule API
* Update rule API
* Delete rule route
* Aggregate rules API
* Disable rule API
* Enable rule API
* Find rules API
* Fix Update API
* Get rule alert summary API
* Get rule state API
* Health API
* Rule types API
* Mute all API
* Mute alert API
* Unmute all API
* Unmute alert route
* Update API key API
* corrected tpye by making it much more complicated
* removed unneeded cocde
* Fixes
* Add back health route
* mutedInstanceIds -> mutedAlertIds
* lastRun -> last_run
* alert_type_state -> rule_type_state & alert_instances -> alerts
Co-authored-by: Gidi Meir Morris <github@gidi.io>
* Create docs for new rule HTTP APIs, deprecate old docs (#94745)
* Create docs for new APIs, deprecate old docs
* Remove connector_type_id
* Update docs
* Add link to legacy APIs from rules API docs
* Remove connector_type_id references
* [DOCS] Add legacy APIs to index.asciidoc
* Fix camel case
Co-authored-by: lcawl <lcawley@elastic.co>
* Make alerting tests use new rules APIs (#95159)
* Make API integration tests use new HTTP APIs
* Fix end to end tests
* Fix test failures
* Fix more test failures
* Rename some files
* Add tests for legacy APIs (#95333)
* Initial commit (#95457)
* Move some new alerting APIs to /internal (#95461)
* Initial commit
* Update README.md
* Use internal API
* Merge deprecated warning w/ alternative solution
* Update API docs
Co-authored-by: Gidi Meir Morris <github@gidi.io>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: lcawl <lcawley@elastic.co>
* Removing duplicate ActionVariable interface. Updating common action variables in UI
* Passing in alert values as rule variables in transform_action_params
* Fixing unit tests
* Fixing functional test
* Adding functiional test
* Updating paths
* Fixing i18n
* Fixing i18n
* Fixing api docs
* Plugin api build
* Moving spaceId and tags under rule prefix
* Using top level alert prefix
* Fixing i18n
* build api docs
* Fixing functional test
* Fixing functional test
* Rename alerts plugin to alerting
* Deprecate old config values
* Few more renames
* Update plugin list
* Rename xpack.alerts -> xpack.alerting
* Fix some ESLint rules
* Fix typecheck
* Fix some test failures
* Some more renames
* Fix ESLint
* Fix some test failures
* Fix failing jest test
* Undo exclusive test
* Fix APM deps
* Fix docs
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* moved legacy actions api to legacy folder
* introduced connector create api
* added new delete route
* added new execute and get_all
* introduced all connector APIs
* renamed action to connector in Apis
* comment on camel case type
* fixed va
* updated docs
* legacy title
* corrected APIs
* legacy links
* added linik to deprecatred APIs
* added linik to deprecatred APIs from index
* moved legacy apis down one level
* Apply suggestions from code review
Co-authored-by: ymao1 <ying.mao@elastic.co>
* renamed route file for connectorTypesRoute
* define legacy route
* Update docs/api/actions-and-connectors/legacy/index.asciidoc
Co-authored-by: Mike Côté <mikecote@users.noreply.github.com>
* api docs
Co-authored-by: ymao1 <ying.mao@elastic.co>
Co-authored-by: Mike Côté <mikecote@users.noreply.github.com>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
