maxmustermann/kibana
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

[Security Solutions] end to end tests, adds more console logging, removes 200 expect statements (#116987)

Browse source 
## Summary

e2e tests are still seeing flake with conflicts and it looks like it _might_ be with querying and not with inserting data. Hard to tell. This PR:

* Adds more console logging when the response is not a 200
* Removes the 200 expect statement and hopes for the best but should blow up if it's not 200 in a different way and we will get the console logging statements.
* Fixes one other flake with the matrix histogram having different counts. We have encountered this before and are applying the same fix which is to just have it check > 0.
* This does fix the timeouts seen where 1 in every 1k rule runs, a rule will not fire until _after_ the 5 minute mark. The timeouts were seen when running the flake runner.


Flake failures around `conflict`:
https://github.com/elastic/kibana/issues/116926
https://github.com/elastic/kibana/issues/116904
https://github.com/elastic/kibana/issues/116231

Not saying this is going to fix those yet, but it's the last 200 ok's we did an expect on, so it might if we are ignoring the conflict. If it fails again I am hopeful beyond hope that we get the body message and line number within the utilities to determine where/why we are getting these from time to time. It does look to fix the timeouts when a rule misfires and slows down the rate at which we continuously query for rule results.

Failure around matrix histogram (The error messages are slightly different on CI each time):
https://github.com/elastic/kibana/issues/97365

Ran this with the flake runner across groups 11 and 12 100 times each and did not see the conflict crop up:
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/128
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/129

The 1 failure in each of those runs were due to something on startup that prevented it from running.

### Checklist

Delete any items that are not applicable to this PR.

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
This commit is contained in:
Frank Hassanabad 2021-11-02 08:28:48 -06:00 committed by GitHub
parent 157a37114f
commit 4bdc17039b
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
3 changed files with 226 additions and 74 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

6
x-pack/test/api_integration/apis/security_solution/matrix_dns_histogram.ts
View file

@ -58,7 +58,11 @@ export default function ({ getService }: FtrProviderContext) {
},
strategy: 'securitySolutionSearchStrategy',
});
expect(networkDns.rawResponse.aggregations?.dns_count).to.eql({ value: 6604 });
// This can have a odd unknown flake if we do anything more strict than this.
const dnsCount = networkDns.rawResponse.aggregations?.dns_count as unknown as {
value: number;
};
expect(dnsCount.value).to.be.above(0);
});
});
});

246
x-pack/test/detection_engine_api_integration/utils.ts
View file

@ -809,16 +809,18 @@ export const getSimpleRuleOutputWithWebHookAction = (actionId: string): Partial<
export const waitFor = async (
functionToTest: () => Promise<boolean>,
functionName: string,
maxTimeout: number = 20000,
timeoutWait: number = 10
maxTimeout: number = 800000,
timeoutWait: number = 250
): Promise<void> => {
let found = false;
let numberOfTries = 0;
while (!found && numberOfTries < Math.floor(maxTimeout / timeoutWait)) {
const maxTries = Math.floor(maxTimeout / timeoutWait);
while (!found && numberOfTries < maxTries) {
if (await functionToTest()) {
found = true;
} else {
// eslint-disable-next-line no-console
console.log(`Try number ${numberOfTries} out of ${maxTries} for function ${functionName}`);
numberOfTries++;
}
@ -940,9 +942,9 @@ export const createRule = async (
if (rule.rule_id != null) {
// eslint-disable-next-line no-console
console.log(
`When creating a rule found an unexpected conflict (409), will attempt a cleanup and one time re-try. This usually indicates a bad cleanup or race condition within the tests: ${JSON.stringify(
`Did not get an expected 200 "ok" when creating a rule (createRule). CI issues could happen. Suspect this line if you are seeing CI issues. body: ${JSON.stringify(
response.body
)}`
)}, status: ${JSON.stringify(response.status)}`
);
await deleteRule(supertest, rule.rule_id);
const secondResponseTry = await supertest
@ -986,7 +988,7 @@ export const deleteRule = async (
if (response.status !== 200) {
// eslint-disable-next-line no-console
console.log(
`Did not get an expected 200 "ok" when deleting the rule. CI issues could happen. Suspect this line if you are seeing CI issues. body: ${JSON.stringify(
`Did not get an expected 200 "ok" when deleting the rule (deleteRule). CI issues could happen. Suspect this line if you are seeing CI issues. body: ${JSON.stringify(
response.body
)}, status: ${JSON.stringify(response.status)}`
);
@ -1023,12 +1025,19 @@ export const updateRule = async (
supertest: SuperTest.SuperTest<SuperTest.Test>,
updatedRule: UpdateRulesSchema
): Promise<FullResponseSchema> => {
const { body } = await supertest
const response = await supertest
.put(DETECTION_ENGINE_RULES_URL)
.set('kbn-xsrf', 'true')
.send(updatedRule)
.expect(200);
return body;
.send(updatedRule);
if (response.status !== 200) {
// eslint-disable-next-line no-console
console.log(
`Did not get an expected 200 "ok" when updating a rule (updateRule). CI issues could happen. Suspect this line if you are seeing CI issues. body: ${JSON.stringify(
response.body
)}, status: ${JSON.stringify(response.status)}`
);
}
return response.body;
};
/**
@ -1037,12 +1046,19 @@ export const updateRule = async (
* @param supertest The supertest deps
*/
export const createNewAction = async (supertest: SuperTest.SuperTest<SuperTest.Test>) => {
const { body } = await supertest
const response = await supertest
.post('/api/actions/action')
.set('kbn-xsrf', 'true')
.send(getWebHookAction())
.expect(200);
return body;
.send(getWebHookAction());
if (response.status !== 200) {
// eslint-disable-next-line no-console
console.log(
`Did not get an expected 200 "ok" when creating a new action. CI issues could happen. Suspect this line if you are seeing CI issues. body: ${JSON.stringify(
response.body
)}, status: ${JSON.stringify(response.status)}`
);
}
return response.body;
};
/**
@ -1059,14 +1075,21 @@ export const findImmutableRuleById = async (
total: number;
data: FullResponseSchema[];
}> => {
const { body } = await supertest
const response = await supertest
.get(
`${DETECTION_ENGINE_RULES_URL}/_find?filter=alert.attributes.tags: "${INTERNAL_IMMUTABLE_KEY}:true" AND alert.attributes.tags: "${INTERNAL_RULE_ID_KEY}:${ruleId}"`
)
.set('kbn-xsrf', 'true')
.send()
.expect(200);
return body;
.send();
if (response.status !== 200) {
// eslint-disable-next-line no-console
console.log(
`Did not get an expected 200 "ok" when finding an immutable rule by id (findImmutableRuleById). CI issues could happen. Suspect this line if you are seeing CI issues. body: ${JSON.stringify(
response.body
)}, status: ${JSON.stringify(response.status)}`
);
}
return response.body;
};
/**
@ -1077,12 +1100,20 @@ export const findImmutableRuleById = async (
export const getPrePackagedRulesStatus = async (
supertest: SuperTest.SuperTest<SuperTest.Test>
): Promise<PrePackagedRulesAndTimelinesStatusSchema> => {
const { body } = await supertest
const response = await supertest
.get(`${DETECTION_ENGINE_PREPACKAGED_URL}/_status`)
.set('kbn-xsrf', 'true')
.send()
.expect(200);
return body;
.send();
if (response.status !== 200) {
// eslint-disable-next-line no-console
console.log(
`Did not get an expected 200 "ok" when getting a pre-packaged rule status. CI issues could happen. Suspect this line if you are seeing CI issues. body: ${JSON.stringify(
response.body
)}, status: ${JSON.stringify(response.status)}`
);
}
return response.body;
};
/**
@ -1104,9 +1135,9 @@ export const createExceptionList = async (
if (exceptionList.list_id != null) {
// eslint-disable-next-line no-console
console.log(
`When creating an exception list found an unexpected conflict (409), will attempt a cleanup and one time re-try. This usually indicates a bad cleanup or race condition within the tests: ${JSON.stringify(
`When creating an exception list found an unexpected conflict (409) creating an exception list (createExceptionList), will attempt a cleanup and one time re-try. This usually indicates a bad cleanup or race condition within the tests: ${JSON.stringify(
response.body
)}`
)}, status: ${JSON.stringify(response.status)}`
);
await deleteExceptionList(supertest, exceptionList.list_id);
const secondResponseTry = await supertest
@ -1152,7 +1183,7 @@ export const deleteExceptionList = async (
if (response.status !== 200) {
// eslint-disable-next-line no-console
console.log(
`Did not get an expected 200 "ok" when deleting an exception list. CI issues could happen. Suspect this line if you are seeing CI issues. body: ${JSON.stringify(
`Did not get an expected 200 "ok" when deleting an exception list (deleteExceptionList). CI issues could happen. Suspect this line if you are seeing CI issues. body: ${JSON.stringify(
response.body
)}, status: ${JSON.stringify(response.status)}`
);
@ -1179,7 +1210,7 @@ export const createExceptionListItem = async (
if (response.status !== 200) {
// eslint-disable-next-line no-console
console.log(
`Did not get an expected 200 "ok" when creating an exception list item. CI issues could happen. Suspect this line if you are seeing CI issues. body: ${JSON.stringify(
`Did not get an expected 200 "ok" when creating an exception list item (createExceptionListItem). CI issues could happen. Suspect this line if you are seeing CI issues. body: ${JSON.stringify(
response.body
)}, status: ${JSON.stringify(response.status)}`
);
@ -1197,11 +1228,19 @@ export const getRule = async (
supertest: SuperTest.SuperTest<SuperTest.Test>,
ruleId: string
): Promise<RulesSchema> => {
const { body } = await supertest
const response = await supertest
.get(`${DETECTION_ENGINE_RULES_URL}?rule_id=${ruleId}`)
.set('kbn-xsrf', 'true')
.expect(200);
return body;
.set('kbn-xsrf', 'true');
if (response.status !== 200) {
// eslint-disable-next-line no-console
console.log(
`Did not get an expected 200 "ok" when getting a rule (getRule). CI issues could happen. Suspect this line if you are seeing CI issues. body: ${JSON.stringify(
response.body
)}, status: ${JSON.stringify(response.status)}`
);
}
return response.body;
};
export const waitForAlertToComplete = async (
@ -1209,11 +1248,16 @@ export const waitForAlertToComplete = async (
id: string
): Promise<void> => {
await waitFor(async () => {
const { body: alertBody } = await supertest
.get(`/api/alerts/alert/${id}/state`)
.set('kbn-xsrf', 'true')
.expect(200);
return alertBody.previousStartedAt != null;
const response = await supertest.get(`/api/alerts/alert/${id}/state`).set('kbn-xsrf', 'true');
if (response.status !== 200) {
// eslint-disable-next-line no-console
console.log(
`Did not get an expected 200 "ok" when waiting for an alert to complete (waitForAlertToComplete). CI issues could happen. Suspect this line if you are seeing CI issues. body: ${JSON.stringify(
response.body
)}, status: ${JSON.stringify(response.status)}`
);
}
return response.body.previousStartedAt != null;
}, 'waitForAlertToComplete');
};
@ -1229,12 +1273,28 @@ export const waitForRuleSuccessOrStatus = async (
): Promise<void> => {
await waitFor(async () => {
try {
const { body } = await supertest
const response = await supertest
.post(`${DETECTION_ENGINE_RULES_URL}/_find_statuses`)
.set('kbn-xsrf', 'true')
.send({ ids: [id] })
.expect(200);
return body[id]?.current_status?.status === status;
.send({ ids: [id] });
if (response.status !== 200) {
// eslint-disable-next-line no-console
console.log(
`Did not get an expected 200 "ok" when waiting for a rule success or status (waitForRuleSuccessOrStatus). CI issues could happen. Suspect this line if you are seeing CI issues. body: ${JSON.stringify(
response.body
)}, status: ${JSON.stringify(response.status)}`
);
}
if (response.body[id]?.current_status?.status !== status) {
// eslint-disable-next-line no-console
console.log(
`Did not get an expected status of ${status} while waiting for a rule success or status for rule id ${id} (waitForRuleSuccessOrStatus). Will continue retrying until status is found. body: ${JSON.stringify(
response.body
)}, status: ${JSON.stringify(response.status)}`
);
}
return response.body[id]?.current_status?.status === status;
} catch (e) {
if ((e as Error).message.includes('got 503 "Service Unavailable"')) {
return false;
@ -1274,11 +1334,21 @@ export const getSignalsByRuleIds = async (
supertest: SuperTest.SuperTest<SuperTest.Test>,
ruleIds: string[]
): Promise<estypes.SearchResponse<RACAlert>> => {
const { body: signalsOpen }: { body: estypes.SearchResponse<RACAlert> } = await supertest
const response = await supertest
.post(DETECTION_ENGINE_QUERY_SIGNALS_URL)
.set('kbn-xsrf', 'true')
.send(getQuerySignalsRuleId(ruleIds))
.expect(200);
.send(getQuerySignalsRuleId(ruleIds));
if (response.status !== 200) {
// eslint-disable-next-line no-console
console.log(
`Did not get an expected 200 "ok" when getting a signal by rule_id (getSignalsByRuleIds). CI issues could happen. Suspect this line if you are seeing CI issues. body: ${JSON.stringify(
response.body
)}, status: ${JSON.stringify(response.status)}`
);
}
const { body: signalsOpen }: { body: estypes.SearchResponse<RACAlert> } = response;
return signalsOpen;
};
@ -1293,11 +1363,20 @@ export const getSignalsByIds = async (
ids: string[],
size?: number
): Promise<estypes.SearchResponse<RACAlert>> => {
const { body: signalsOpen }: { body: estypes.SearchResponse<RACAlert> } = await supertest
const response = await supertest
.post(DETECTION_ENGINE_QUERY_SIGNALS_URL)
.set('kbn-xsrf', 'true')
.send(getQuerySignalsId(ids, size))
.expect(200);
.send(getQuerySignalsId(ids, size));
if (response.status !== 200) {
// eslint-disable-next-line no-console
console.log(
`Did not get an expected 200 "ok" when getting a signal by id. CI issues could happen (getSignalsByIds). Suspect this line if you are seeing CI issues. body: ${JSON.stringify(
response.body
)}, status: ${JSON.stringify(response.status)}`
);
}
const { body: signalsOpen }: { body: estypes.SearchResponse<RACAlert> } = response;
return signalsOpen;
};
@ -1310,11 +1389,20 @@ export const getSignalsById = async (
supertest: SuperTest.SuperTest<SuperTest.Test>,
id: string
): Promise<estypes.SearchResponse<RACAlert>> => {
const { body: signalsOpen }: { body: estypes.SearchResponse<RACAlert> } = await supertest
const response = await supertest
.post(DETECTION_ENGINE_QUERY_SIGNALS_URL)
.set('kbn-xsrf', 'true')
.send(getQuerySignalsId([id]))
.expect(200);
.send(getQuerySignalsId([id]));
if (response.status !== 200) {
// eslint-disable-next-line no-console
console.log(
`Did not get an expected 200 "ok" when getting signals by id (getSignalsById). CI issues could happen. Suspect this line if you are seeing CI issues. body: ${JSON.stringify(
response.body
)}, status: ${JSON.stringify(response.status)}`
);
}
const { body: signalsOpen }: { body: estypes.SearchResponse<RACAlert> } = response;
return signalsOpen;
};
@ -1322,10 +1410,19 @@ export const installPrePackagedRules = async (
supertest: SuperTest.SuperTest<SuperTest.Test>
): Promise<void> => {
await countDownTest(async () => {
const { status } = await supertest
const { status, body } = await supertest
.put(DETECTION_ENGINE_PREPACKAGED_URL)
.set('kbn-xsrf', 'true')
.send();
if (status !== 200) {
// eslint-disable-next-line no-console
console.log(
`Did not get an expected 200 "ok" when installing pre-packaged rules (installPrePackagedRules) yet. Retrying until we get a 200 "ok". body: ${JSON.stringify(
body
)}, status: ${JSON.stringify(status)}`
);
}
return status === 200;
}, 'installPrePackagedRules');
};
@ -1480,12 +1577,19 @@ export const createRuleWithExceptionEntries = async (
exceptions_list: [...maybeExceptionList, ...maybeEndpointList],
};
const ruleResponse = await createRule(supertest, ruleWithException);
await supertest
const response = await supertest
.patch(DETECTION_ENGINE_RULES_URL)
.set('kbn-xsrf', 'true')
.send({ rule_id: ruleResponse.rule_id, enabled: true })
.expect(200);
.send({ rule_id: ruleResponse.rule_id, enabled: true });
if (response.status !== 200) {
// eslint-disable-next-line no-console
console.log(
`Did not get an expected 200 "ok" when patching a rule with exception entries (createRuleWithExceptionEntries). CI issues could happen. Suspect this line if you are seeing CI issues. body: ${JSON.stringify(
response.body
)}, status: ${JSON.stringify(response.status)}`
);
}
return ruleResponse;
};
@ -1542,14 +1646,22 @@ export const startSignalsMigration = async ({
supertest: SuperTest.SuperTest<SuperTest.Test>;
indices: string[];
}): Promise<CreateMigrationResponse[]> => {
const {
body: { indices: created },
}: { body: { indices: CreateMigrationResponse[] } } = await supertest
const response = await supertest
.post(DETECTION_ENGINE_SIGNALS_MIGRATION_URL)
.set('kbn-xsrf', 'true')
.send({ index: indices })
.expect(200);
.send({ index: indices });
const {
body: { indices: created },
}: { body: { indices: CreateMigrationResponse[] } } = response;
if (response.status !== 200) {
// eslint-disable-next-line no-console
console.log(
`Did not get an expected 200 "ok" when starting a signals migration (startSignalsMigration). CI issues could happen. Suspect this line if you are seeing CI issues. body: ${JSON.stringify(
response.body
)}, status: ${JSON.stringify(response.status)}`
);
}
return created;
};
@ -1566,14 +1678,22 @@ export const finalizeSignalsMigration = async ({
supertest: SuperTest.SuperTest<SuperTest.Test>;
migrationIds: string[];
}): Promise<FinalizeMigrationResponse[]> => {
const {
body: { migrations },
}: { body: { migrations: FinalizeMigrationResponse[] } } = await supertest
const response = await supertest
.post(DETECTION_ENGINE_SIGNALS_FINALIZE_MIGRATION_URL)
.set('kbn-xsrf', 'true')
.send({ migration_ids: migrationIds })
.expect(200);
.send({ migration_ids: migrationIds });
const {
body: { migrations },
}: { body: { migrations: FinalizeMigrationResponse[] } } = response;
if (response.status !== 200) {
// eslint-disable-next-line no-console
console.log(
`Did not get an expected 200 "ok" when finalizing signals migration (finalizeSignalsMigration). CI issues could happen. Suspect this line if you are seeing CI issues. body: ${JSON.stringify(
response.body
)}, status: ${JSON.stringify(response.status)}`
);
}
return migrations;
};

48
x-pack/test/lists_api_integration/utils.ts
View file

@ -116,20 +116,25 @@ export const removeExceptionListServerGeneratedProperties = (
export const waitFor = async (
functionToTest: () => Promise<boolean>,
functionName: string,
maxTimeout: number = 5000,
timeoutWait: number = 10
maxTimeout: number = 800000,
timeoutWait: number = 250
) => {
await new Promise<void>(async (resolve, reject) => {
try {
let found = false;
let numberOfTries = 0;
const maxTries = Math.floor(maxTimeout / timeoutWait);
while (!found && numberOfTries < Math.floor(maxTimeout / timeoutWait)) {
while (!found && numberOfTries < maxTries) {
const itPasses = await functionToTest();
if (itPasses) {
found = true;
} else {
// eslint-disable-next-line no-console
console.log(
`Try number ${numberOfTries} out of ${maxTries} for function ${functionName}`
);
numberOfTries++;
}
@ -219,7 +224,7 @@ export const importFile = async (
if (response.status !== 200) {
// eslint-disable-next-line no-console
console.log(
`Did not get an expected 200 "ok" When importing a file. CI issues could happen. Suspect this line if you are seeing CI issues. body: ${JSON.stringify(
`Did not get an expected 200 "ok" When importing a file (importFile). CI issues could happen. Suspect this line if you are seeing CI issues. body: ${JSON.stringify(
response.body
)}, status: ${JSON.stringify(response.status)}`
);
@ -246,12 +251,20 @@ export const importTextFile = async (
contents: string[],
fileName: string
): Promise<void> => {
await supertest
const response = await supertest
.post(`${LIST_ITEM_URL}/_import?type=${type}`)
.set('kbn-xsrf', 'true')
.attach('file', getImportListItemAsBuffer(contents), fileName)
.expect('Content-Type', 'application/json; charset=utf-8')
.expect(200);
.expect('Content-Type', 'application/json; charset=utf-8');
if (response.status !== 200) {
// eslint-disable-next-line no-console
console.log(
`Did not get an expected 200 "ok" when importing a text file (importTextFile). CI issues could happen. Suspect this line if you are seeing CI issues. body: ${JSON.stringify(
response.body
)}, status: ${JSON.stringify(response.status)}`
);
}
// although we have pushed the list and its items, it is async so we
// have to wait for the contents before continuing
@ -271,10 +284,17 @@ export const waitForListItem = async (
fileName: string
): Promise<void> => {
await waitFor(async () => {
const { status } = await supertest
const { status, body } = await supertest
.get(`${LIST_ITEM_URL}?list_id=${fileName}&value=${itemValue}`)
.send();
if (status !== 200) {
// eslint-disable-next-line no-console
console.log(
`Did not get an expected 200 "ok" when waiting for a list item (waitForListItem) yet. Retrying until we get a 200 "ok". body: ${JSON.stringify(
body
)}, status: ${JSON.stringify(status)}`
);
}
return status === 200;
}, `waitForListItem fileName: "${fileName}" itemValue: "${itemValue}"`);
};
@ -310,9 +330,17 @@ export const waitForTextListItem = async (
await waitFor(async () => {
const promises = await Promise.all(
tokens.map(async (token) => {
const { status } = await supertest
const { status, body } = await supertest
.get(`${LIST_ITEM_URL}?list_id=${fileName}&value=${token}`)
.send();
if (status !== 200) {
// eslint-disable-next-line no-console
console.log(
`Did not get an expected 200 "ok" when waiting for a text list item (waitForTextListItem) yet. Retrying until we get a 200 "ok". body: ${JSON.stringify(
body
)}, status: ${JSON.stringify(status)}`
);
}
return status === 200;
})
);