A bugfix for significant terms on the ES side resulted in changes to assertions we have for api integration tests for APM correlations. This PR updates the affected tests.
Code deduplication:
- combine 3 existing client side hooks into useSearchStrategy
- combine server side multiple search strategy files into shared search_strategy_provider.ts
- Clarified naming (client/server params, strategy naming etc.), improved types.
- None of the actual deeper internal logic changed, larger chunks of code that show up as new lines is mostly just moved code + additional types (e.g. function overloads) to support the different search strategies with the same server side code and client side hook.
When the observability:enableInspectEsQueries advanced setting is enabled, show an inspector that includes all queries through useFetcher.
Remove the callout.
**Ticket:** https://github.com/elastic/kibana/issues/109293🚨 **This PR is critical for Observability 7.15** 🚨
## Summary
This PR fixes the indexing implementation in `rule_registry`. It implements the suggestions for backwards compatibility described in the ticket:
- changes the naming scheme and introduces the concept of "backing indices", so that names of the concrete ("backing") indices != names of their aliases
- adds versioning based on the current Kibana version
TODO:
- [x] Change index naming (implement the concept of backing indices)
- [x] Include Kibana version into the index template metadata
- [x] Include Kibana version into the document fields
- [x] Remove `version` from `IndexOptions` (parameters provided by solutions/plugins when initializing alerts-as-data indices)
- [x] Fix CI
### Checklist
- [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials
- [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
* removes hardcoded index from apm integration tests
* adds integration tests for rule registry's get index route
* more cleanup
* fail try-catch if response is not empty, adds comments as to why index refresh catch block is empty
- Show the same empty state in the correlations table
- Add "Correlations" title above the table
- Add EuiSpacer between the sections before and after progress
- Update the copy within the beta badge title=Failed transaction correlations description=Failed transaction correlations is not GA...
- Remove s size from the beta badge
- Move the help popover to the top of the panel (similar to the Latency correlations tab)
- Move the Cancel/Refresh option to the right of the progress bar (similar to the Latency correlations tab)
- When the correlation job is running the correlations tab should show a loading state similar to the latency correlations table
- Indicate in the table headers Score is sorted by default
- Add sortability to both Latency and failed transactions correlations table
- Refactor to prevent duplicate code/components like Log, progress bar
- Fix alignments of the tab content header (previously navigating from one Trace samples tab to Latency correlation tabs will cause a minor jump in the header, or the titles within the same tab were not the same size )
- Remove the event.outcome as a field candidate (because event.outcome: failure would always be significant in this case)
- Shrink the column width for impact (previously it was at 116px)
- Added badge for High, medium, low [APM] Correlations: Show impact levels (high, medium, low) as colored badges indicating their severity
- Fix license prompt text
- Functional tests for the new tab
- Make the p value & error rate columns visible only when esInspect mode is enabled
- Moves APM Latency Correlations from flyout to transactions page.
- Introduces a tab based navigation for `Trace samples` / `Latency correlations` / `Failed transactions correlations`
- For trace samples, the previous low detailed histogram chart gets replaced with the new log log based histogram chart. Users can drag select a range of lantency to filter trace samples.
- Removes code related to the previous distribution chart. Renames `useTransactionDistributionFetcher` to `useTransactionTraceSamplesFetcher`. Instead of a histogram with top hits, this will now just return a sample of traces for the given filters.
* Remove outdated top_alerts route and related types
* Remove tests for deleted code
* Remove test for deleted API
* Remove reference to deleted type
* Remove unused translations
* Remove unused mock from story
* Remove no-op alerts page story for now
* Remove unsafe type assertions
* Factor out alert field type
* Compile kbn-io-ts-utils for the browser as well
* Avoid deep import which doesn't work cross-platform
* Revert "Avoid deep import which doesn't work cross-platform"
This reverts commit 492378c6b5.
* Revert "Compile kbn-io-ts-utils for the browser as well"
This reverts commit a1267b139d.
* Revert "Factor out alert field type"
This reverts commit def6987498.
* Revert "Remove unsafe type assertions"
This reverts commit c88d4cd005.
* Remove unsafe type assertions (again)
* [APM] Displays callout when transaction events are used instead of aggregrated metrics (#107477)
* Apply suggestions from code review
Co-authored-by: Søren Louv-Jansen <sorenlouv@gmail.com>
* PR feedback, and isolates the logic for getting the fallback strategy
* PR feedback
Co-authored-by: Søren Louv-Jansen <sorenlouv@gmail.com>
### Summary
### Fields used moving forward
`kibana.alert.rule.consumer` will refer to the context in which a rule instance is created. Rules created in:
- stack --> `alerts`
- security solution --> `siem`
- apm --> `apm`
`kibana.alert.rule.producer` will refer to the plugin that registered a rule type. Rules registered in:
- stack --> `alerts`
- security solution --> `siem`
- apm --> `apm`
So an `apm.error_rate` rule created in stack will have:
- consumer: `alerts` and producer: `apm`
An `apm.error_rate` rule created in apm will have:
- consumer: `apm` and producer: `apm`
`kibana.alert.rule.rule_type_id` will refer to a rule's rule type id. Examples:
- `apm.error_rate`
- `siem.signals`
- `siem.threshold`
Also renamed the following because `rule.*` fields are meant to be ecs fields pulled from the source/event document, not refer to our rule fields.
`rule.name` --> `kibana.alert.rule.name` will refer to the rule's name.
`rule.category` --> `kibana.alert.rule.category` will refer to the rule's category.
`rule.id` --> `kibana.alert.rule.uuid` will refer to the rule's uuid.
* Use Serializable from package
* Rename to align with core
* fix
* more replacements
* docssss
* fix
* Move it to @kbn/utility-types and remove core export
* buildy build
* tests
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* adding comparison to inventory page
* new api to get detailed statistics
* show comparison data
* adding api test
* fixing unit test
* fixing ts issue
* adding loading to table
* refactoring
* fixing TS issue
* addressing PR comments
* fixing merge
* addressing PR comments
* fixing api test
* adding comment
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* injects bulkCreate and wrapHits to individual rule executors
* WIP create_security_rule_type_factory based on Marshall's work in #d3076ca54526ea0e61a9a99e1c1bce854806977e
* removes ruleStatusService from old rule executors, fixes executor unit tests
* fixes rebase
* Rename reference_rules to rule_types
* Fix type errors
* Fix type errors in base security rule factory
* Additional improvements to types and interfaces
* More type alignment
* Fix remaining type errors in query rule
* Add validation / inject lists plugin
* Formatting
* Improvements to typing
* Static typing on executors
* cleanup
* Hook up params for query/threshold rules... includes exceptionsList and daterange tuple
* Scaffolding for wrapHits and bulkCreate
* Add error handling / status reporting
* Fixup alert type state
* Begin threshold
* Begin work on threshold state
* Organize rule types
* Export base security rule types
* Fixup lifecycle static typing
* WrapHits / bulk changes
* Field mappings (partial)
* whoops
* Remove redundant params
* More flexibile implementation of bulkCreateFactory
* Add mappings
* Finish query rule
* Revert "Remove redundant params"
This reverts commit 87aff9c810.
* Revert "whoops"
This reverts commit a7771bd392.
* Fixup return types
* Use alertWithPersistence
* Fix import
* End-to-end rule mostly working
* Fix bulkCreate
* Bug fixes
* Bug fixes and mapping changes
* Fix indexing
* cleanup
* Fix type errors
* Test fixes
* Fix query tests
* cleanup / rename kibana.rac to kibana
* Remove eql/threshold (for now)
* Move technical fields to package
* Add indexAlias and buildRuleMessageFactory
* imports
* type errors
* Change 'kibana.rac.*' to 'kibana.*'
* Fix lifecycle tests
* Single alert instance
* fix import
* Fix type error
* Fix more type errors
* Fix query rule type test
* revert to previous ts-expect-error
* type errors again
* types / linting
* General readability improvements
* Add invariant function from Dmitrii's branch
* Use invariant and constants
* Improvements to field mappings
* More test failure fixes
* Add refresh param for bulk create
* Update more field refs
* Actually use refresh param
* cleanup
* test fixes
* changes to rule creation script
* Fix created signals count
* Use ruleId
* Updates to bulk indexing
* Mapping updates
* Cannot use 'strict' for dynamic setting
Co-authored-by: Marshall Main <marshall.main@elastic.co>
Co-authored-by: Ece Ozalp <ozale272@newschool.edu>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Adds first functional tests for APM Latency Correlations code.
- Fix: The log log chart's Y axis would only start at 1 by default which hides results for small datasets like the ones used in these tests. Starting the axis at 0 isn't supported to log based ones so we're setting it to just a small value above 0.
- Fix: Instead of the hard coded apm-* index we passed on from the client, we now correctly consider APM's custom settings.
* kind of working solution... need to fix types.. would be great if all of this could go in the authorization class but I don't think we have access to the spaceids when we generate the kibana security action strings?
* update mapping type as array:true for space_ids field, fixes types, updates jest tests, adds integration tests
* undo changes in alerting authz class
* update snapshot for apm api integration test for rules writing alerts
* fix apm integration tests
* omit version and sequence from expected outcome
* re-add space id after this code was moved in master
* add another default space id to test
* fixes bug to remove duplicate spaceids
* add space ids filter to elasticsearch query, updates detection role
* update snapshot
* update type docs for alerts client
* remove dead code
* fix type error
* renames space ids field on alert documents from kibana.rac.alert.space_ids to kibana.space_ids
* fixes kb-rule-data-utils package
* update snapshots
* remove references to kibana.rac.alert.space_ids and replace with kibana.space_ids in rule registry integration tests and apm integration tests
* fix apm functional test snapshots
* undo index name changes I made in apm integration test configs
* update typedocs references to upstream, not local repo
Adds API integration tests for APM Latency Correlations code.
Writing the tests surfaced some glitches fixed as part of this PR:
- If the applied filters don't return any docs, we won't throw an error anymore. Instead, the async search service finishes early and just returns no results.
- If for whatever reason the async search service throws an error, it will also set its state now to isRunning = false.
- If the client triggers a request with a service ID we now make sure that async search service still exists. We throw an error if that service no longer exists. This avoids re-instantiating async search services when they've already finished or failed and for whatever reason a client triggers another request with the same ID.
- Refactored requests to reuse APM's own getCorrelationsFilters(). We now require start/end to be set and it will be converted from ISO (client side) to epochmillis (server side) to be more in line with APM's existing code.
- The async search service now creates a simple internal log. This gets exposed via the API and we assert it using the API tests. In the future, we might also expose it in the UI to allow for better problem investigation for users and support.
An MVP of the RBAC work required for the "alerts as data" effort. An example of the existing implementation for alerts would be that of the security solution. The security solution stores its alerts generated from rules in a single data index - .siem-signals. In order to gain or restrict access to alerts, users do so by following the Elasticsearch privilege architecture. A user would need to go into the Kibana role access UI and give explicit read/write/manage permissions for the index itself.
Kibana as a whole is moving away from this model and instead having all user interactions run through the Kibana privilege model. When solutions use saved objects, this authentication layer is abstracted away for them. Because we have chosen to use data indices for alerts, we cannot rely on this abstracted out layer that saved objects provide - we need to provide our own RBAC! Instead of giving users explicit permission to an alerts index, users are instead given access to features. They don't need to know anything about indices, that work we do under the covers now.
Co-authored-by: Yara Tercero <yctercero@users.noreply.github.com>
Co-authored-by: Yara Tercero <yara.tercero@elastic.co>
* adding kuery bar to search transactions metrics
* addressig PR comments
* fixing api test
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
