* fixing bucket size when no data available
* fixing layout
* removing import
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
e2e tests are still seeing flake with conflicts and it looks like it _might_ be with querying and not with inserting data. Hard to tell. This PR:
* Adds more console logging when the response is not a 200
* Removes the 200 expect statement and hopes for the best but should blow up if it's not 200 in a different way and we will get the console logging statements.
* Fixes one other flake with the matrix histogram having different counts. We have encountered this before and are applying the same fix which is to just have it check > 0.
* This does fix the timeouts seen where 1 in every 1k rule runs, a rule will not fire until _after_ the 5 minute mark. The timeouts were seen when running the flake runner.
Flake failures around `conflict`:
https://github.com/elastic/kibana/issues/116926https://github.com/elastic/kibana/issues/116904https://github.com/elastic/kibana/issues/116231
Not saying this is going to fix those yet, but it's the last 200 ok's we did an expect on, so it might if we are ignoring the conflict. If it fails again I am hopeful beyond hope that we get the body message and line number within the utilities to determine where/why we are getting these from time to time. It does look to fix the timeouts when a rule misfires and slows down the rate at which we continuously query for rule results.
Failure around matrix histogram (The error messages are slightly different on CI each time):
https://github.com/elastic/kibana/issues/97365
Ran this with the flake runner across groups 11 and 12 100 times each and did not see the conflict crop up:
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/128https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/129
The 1 failure in each of those runs were due to something on startup that prevented it from running.
### Checklist
Delete any items that are not applicable to this PR.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
* added cypress setup for fleet
* updated readme
* fixed types
* fix unused
* starting up fleet server in cypress
* added more fleet tests
* added package update test
* updated readme
* trying jenkins run
* trying jenkins run
* trying jenkins run
* fix types, cypress config
* example of mocked requests
* added getBySel
* fix test after refactor
* removed duplication
* fix tests
* added to buildkite
* updated tests
* starting agent with docker
* trying to fix ip address
* ifconfig
* ip address
* ip address
* ip address
* type fix
* remove extra logging
* fixed test
* fix buildkite
* cleanup
* cleanup
* using docker for fleet_server
* fix
* fix
* trying to fix
* update config
* test
* test
* test
* test
* revert changes
* test
* test
* static ip on linux
* docker version
* try again
* fixed review comments
* fixed types
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Patryk Kopycinski <contact@patrykkopycinski.com>
* [Global Nav] Moved Overview page/link to the accordion/Solution title
* Using Cloud logo for deployment link and fixing isActive state
* Change Deployment section to dark background
* Completely remove locking ability
* Change buttonElement if overview page exists
* Snaps from button/div
* PR Feedback:
- Puts `Overview` links back for Observability and Security
- Increases hit area and scopes underline of anchor tag within the accordion header
- Uses `id`’s to find Overview pages instead of `title` text
* use event.ingested for datastream last activity
* remove ms precision from datastream dates
* add timestamp check to test
* fix type error
* split test out to be less complex and more reliable 🤞
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* WIP - need to figure out how to delete old siem-detection action SO's after each test
* WIP - adds some fixes for the update rules utility that differ from patch rules utility
* fix type checks
* cleanup
* remove commented out code
* rename const to use capital snake case
* naming integration tests, adds expect for disabled rules that get migrated, adds expect for pre-migrated rules
## Summary
Adds e2e tests for the legacy notification system for:
* Exporting rules
* Reading rules
* Finding rules
Also adds missing e2e tests for the non-legacy actions where they previously did not have e2e tests.
These tests ensure that the legacy notifications system will run for a while.
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
* Only apply field aliases to legacy .siem-signals indices
* Fix unit test mocks
* Add new function for special index existence check
* Actually add new function for special index existence check
* Undo getIndexVersion change
* Add basic integration tests for field alias logic
* Add back create_index to test list
* Add missing markdown to readme
* Revert change to delete_index_route
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* [Logs UI][Metrics UI] Remove deprecated config fields from APIs
* Fix typecheck
* Fix typecheck
* Fix typecheck
* Fix jest
* Fix functional test
* Remove extraneous timeField args
* Typecheck fix
* Consolidate log file changes to ResolvedLogSourceConfiguration
* Fix merge
* Revert additional logs files
* Revert inventory models
* Revert log_analysis api
* Fix timefield reference in process list
* Restore logs page files, fix typecheck on mock
* Fix functional test
* Restore inventory models index
* Fix typecheck on getFilteredMetrics
* Look CI if you don't tell me all the type errors at once I can't fix them all
* Maybe this is the last typecheck fix who knows
* unskip test suite
* Fix functional tests and some refactoring
* Refactor Policy Details test and centralize getting of Agent Policy combined input for endpoint
* Change approach for checking policy data in fleet pages
* Change Policy Settings displayed in Fleet to a) show a loader while retrieving settings and b) show loading errors if any
* Close any visible toasts before clicking on the save button
## Summary
Adds console logging and relaxes a few 200 checks in the end to end tests in two areas of:
* createExceptionListItem
* importFile
As recently the tests are failing around these parts and possibly we can get information on the next failures if it is from these two areas or if it's from somewhere else we are not expecting.
Also outputs the body message so if it fails in these areas we can see the message with the conflict that might help us see where the issue is arising.
### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
This PR removes the `monitor` cluster privilege from the `ft_ml_ui_extras` test role as it's no longer required by the categorization wizard and we want to stay close to the minimum set of required privileges for our test users.
* [ILM] Fixed ILM a11y test by creating a snapshot repo that is now required in the ILM API
* [ILM] Fixed functional test by creating a snapshot repository
* [ILM] Updated the params after es client update
* [ILM] Added filtering by policy name to the a11y test to find the correct ILM policy in the list
* [ILM] Added filtering by policy name to the a11y test to find the correct ILM policy in the list
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
## Summary
* For the test of `create_endpoint_exceptions.ts` code was changed to try to reduce/remove flake. I put the code back to where it was and with the sorting introduced it all should pass
* Added `await waitForSignalsToBePresent(supertest, 1, [id]);` to areas of code that were missing.
* The `.flat` should be an added layer of protection for flakes.
### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
## Summary
Fixes flake tests of:
https://github.com/elastic/kibana/issues/115918https://github.com/elastic/kibana/issues/103273https://github.com/elastic/kibana/issues/108640https://github.com/elastic/kibana/issues/109447https://github.com/elastic/kibana/issues/100630https://github.com/elastic/kibana/issues/94535https://github.com/elastic/kibana/issues/104260
Security solution has been using `bsearch` and has encountered flake in various forms. Different developers have been fixing the flake in a few odd ways (myself included) which aren't 100%. This PR introduces a once-in-for-all REST API retry service called `bsearch` which will query `bsearch` and if `bsearch` is not completed because of async occurring due to slower CI runtimes it will continuously call into the `bsearch` with the correct API to ensure it gets a complete response before returning.
## Usage
Anyone can use this service like so:
```ts
const bsearch = getService('bsearch');
const response = await bsearch.send<MyType>({
supertest,
options: {
defaultIndex: ['large_volume_dns_data'],
}
strategy: 'securitySolutionSearchStrategy',
});
```
If you're using a custom auth then you can set that beforehand like so:
```ts
const bsearch = getService('bsearch');
const supertestWithoutAuth = getService('supertestWithoutAuth');
const supertest supertestWithoutAuth.auth(username, password);
const response = await bsearch.send<MyType>({
supertest,
options: {
defaultIndex: ['large_volume_dns_data'],
}
strategy: 'securitySolutionSearchStrategy',
});
```
## Misconceptions in the tests leading to flake
* Can you just call the bsearch REST API and it will always return data first time? Not always true, as when CI slows down or data increases `bsearch` will give you back an async reference and then your test will blow up.
* Can we wrap the REST API in `retry` to fix the flake? Not always but mostly true, as when CI slows down or data increases `bsearch` could return the async version continuously which could then fail your test. It's also tedious to tell everyone in code reviews to wrap everything in `retry` instead of just fixing it with a service as well as inform new people why we are constantly wrapping these tests in `retry`.
* Can we manually parse the `bsearch` if it has `async` for each test? This is true but is error prone and I did this for one test and it's ugly and I had issues as I have to wrap 2 things in `retry` and test several conditions. Also it's harder for people to read the tests rather than just reading there is a service call. Also people in code reviews missed where I had bugs with it. Also lots of boiler plate.
* Can we just increase the timeout with `wait_for_completion_timeout` and the tests will pass for sure then? Not true today but maybe true later, as this hasn't been added as plumbing yet. See this [open ticket](https://github.com/elastic/kibana/issues/107241). Even if it is and we increase the timeout to a very large number bsearch might return with an `async` or you might want to test the `async` path. Either way, if/when we add the ability we can increase it within 1 spot which is this service for everyone rather than going to each individual test to add it. If/when it's added if people don't use the bsearch service we can remove it later if we find this is deterministic enough and no one wants to test bsearch features with their strategies down the road.
## Manual test of bsearch service
If you want to manually watch the bsearch operate as if the CI system is running slow or to cause an `async` manually you manually modify this setting here:
https://github.com/elastic/kibana/blob/master/src/plugins/data/server/search/strategies/ese_search/request_utils.ts#L61
To be of a lower number such as `1ms` and then you will see it enter the `async` code within `bsearch` consistently
## Reference PRs
We cannot set the wait_for_complete just yet
https://github.com/elastic/kibana/issues/107241 so we decided this was the best way to reduce flake for testing for now.
### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
* Initial commit
* Properly handle signal history
* Fix#95258 - cardinality sort bug
* Init threshold rule
* Create working threshold rule
* Fix threshold signal generation
* Fix tests
* Update mappings
* ALERT_TYPE_ID => RULE_TYPE_ID
* Add tests
* Fix types
* Adds RAC rule type migration
* Fix threshold tests (remove outputIndex)
* Add threshold rule type to ruleTypeMappings
* Add kbn-securitysolution-rules package for sharing with alerting framework
* Fix type errors
* Fix find_rules tests
* First round of test fixes
* Fix issues from merge conflicts
* Use ruleDataClient getReader() for reading
* Fixes to 'generating_signals' tests
* Remove more refs to legacy schema
* Linting
* Quick type fix
* Bug fixes
* Add saved query rule type
* Linting
* Fix types
* Signal generation tests
* Test updates
* Update some more refs
* build_alert tests
* Cleanup
* Ref updates
* Revert "Ref updates"
This reverts commit 4d1473d6b0.
* Update status field
* Test fixes
* Another test
* Got a little too aggressive with search/replace
* let's see where we're at
* Fix
* Test fixes
* cleanup
* Fix cases API integration test config, flaky DE tests
* Move flattenWithPrefix to package / skip signal migration tests
* Fix unit tests
* Use new schema for bulk rule creation
* event: { kind } => event.kind
* Fix signal migration API tests
* Fix ml integration test
* Fix threat match integration tests
* Fix ML rule type tests and add correct producer to all rule types
* Update threat match API integration test
* Remove dupe properties
* Type fix
* Fix ML producer in functional test
* Fix generating_signals tests
* Remove usage of RuleDataClient-based execution log client
* Don't check output index version if rule registry enabled
* Fix bulk duplicate rule
* Fix duplicate rule test
* Fix readPrivileges and timestamp check logic
* Fixes for eql and exceptions tests... disable open_close_signals
* Type fixes / keyword test fixes
* Additional test fixes
* Unit test fixes + signal -> kibana.alert
* Test fixes for exceptions
* Fix read_resolve_rules test
* Various test fixes with marshallmain
* Sort search results
* Fix create_rules tests
* Disable writer cache for integration tests
* Disable writer cache for cases integration tests
* Fix types in rule_data_plugin_service
* Fix ordering in exceptions tests
* Remove rule_registry.enabled flag
* Fix signals migration tests
* Don't check signals index before creation
* Fix cypress config
* Fix type error
* create_migrations tests
* Skip flaky test
* Helpful comment
* Fixes from merge conflicts
* Pretend that signals index exists
* Fix type errors
* Skip flaky tests
* Fix threat matching test
* Clean up
* Reverting default ruleRegistry experimental flag (breaks unit tests)
* Reenable rule registry experimental feature by default
* Execute DE rule migration in 8.0
Co-authored-by: Marshall Main <marshall.main@elastic.co>
