## Summary
Improves FTR/e2e conflict retries with exception lists and security rules.
Fixes:
https://github.com/elastic/kibana/issues/115734https://github.com/elastic/kibana/issues/115769https://github.com/elastic/kibana/issues/115715https://github.com/elastic/kibana/issues/115702https://github.com/elastic/kibana/issues/115701
This past week we have been seeing increasing flake across tests involving `exception_lists` involving a `409 conflict` on our tests. Looking at each of the tests above and the flake it looks like we were calling Elasticsearch directly within the `.kibana` index to delete the exception list and list items as a shortcut:
```
export const deleteAllExceptions = async (es: KibanaClient): Promise<void> => {
return countDownES(async () => {
return es.deleteByQuery({
index: '.kibana',
q: 'type:exception-list or type:exception-list-agnostic',
wait_for_completion: true,
refresh: true,
body: {},
});
}, 'deleteAllExceptions');
};
```
Although I think we did everything correctly `wait_for_completion: true` and `refresh: true` within the tests there might be a slight race condition where the delete by query does not immediately happen for us. Since we should prefer to use direct REST API's where we can instead of calling into `.kibana` I changed this to using the exception list API:
```
export const deleteAllExceptions = async (
supertest: SuperTest.SuperTest<SuperTest.Test>
): Promise<void> => {
await countDownTest(
async () => {
const { body } = await supertest
.get(`${EXCEPTION_LIST_URL}/_find?per_page=9999`)
.set('kbn-xsrf', 'true')
.send();
const ids: string[] = body.data.map((exception: ExceptionList) => exception.id);
for await (const id of ids) {
await supertest.delete(`${EXCEPTION_LIST_URL}?id=${id}`).set('kbn-xsrf', 'true').send();
}
const { body: finalCheck } = await supertest
.get(`${EXCEPTION_LIST_URL}/_find`)
.set('kbn-xsrf', 'true')
.send();
return finalCheck.data.length === 0;
},
'deleteAllExceptions',
50,
1000
);
};
```
The additional final check above should ensure it sees that the data has been deleted before returning. Otherwise it will loop around again and keep trying.
I also improve both the `createRules` and `createExceptionList` by introducing a one-time, "detect if in conflict" and then "remove if in conflict" within those tests. This should help safe guard against flake if the above does not fix it. I also added more logging statements in case we do encounter this again on the CI system we can further trouble shoot it and add additional retry logic/fix logic.
A good side effect is if now you kill your tests half way through and restart them, the additional "detect if conflict" will recover your test for you as a developer. So 👍 that is an added benefit.
Example error message you would get (but not test failure) if you remove one of the cleanup sections in the `afterEach` or if you kill a test half way through and then restart it as an engineer:
```
└-: "is" operator
└-> "before all" hook for "should find all the text from the data set when no exceptions are set on the rule"
└-> should find all the text from the data set when no exceptions are set on the rule
└-> "before each" hook: global before each for "should find all the text from the data set when no exceptions are set on the rule"
└-> "before each" hook for "should find all the text from the data set when no exceptions are set on the rule"
When creating a rule found an unexpected conflict (409), will attempt a cleanup and one time re-try. This usually indicates a bad cleanup or race condition within the tests: {"message":"rule_id: \"rule-1\" already exists","status_code":409}
└- ✓ pass (7.9s)
```
### Checklist
- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
* Add comparison to occurrences errors chart
* fix types
* Add errors rate chart
* Rename component and other PR review changes
* Subtitute noHits for fetch status
* Use status from correct request
* Improve types for comparisonType and comparisonEnable
* Move TimeRangeComparisonEnum to /runtime_types/comparison_type_rt
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Ported options list to use Redux. added selection management and selection clear buttons to options list. Options list clears selections when field or index pattern change. Design cleanups and updates
Co-authored-by: Andrea Del Rio <delrio.andre@gmail.com>
* Add attachment to list of synced objects for Kibana
* Add attachments to Salesforce Sandbox as well
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
* Add support for extensions using latest version of a package and forcing upgrade state for edit policy view
* Fix isUpgrade flag on integrations UI version of edit page
* Treat non-validation errors as general failures in server and UI
* Fix tests + don't call upgrade API when saving
* fix i18n
* Fix default name always appearing when editing package policies via extension UI
* Opt security solution plugin out of new extension option
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
