kibana/x-pack/plugins/security_solution
Frank Hassanabad e5944a3646
[Security Solutions][Detection Engine] Fixes timestamp bugs within source indexes when the formats are not ISO8601 format (#101349)
## Summary

We have a few bugs where when the source index for detections is not `"strict_date_optional_time"` it is possible that we will misinterpret the format to be epoch milliseconds when it could be epoch seconds or another ambiguous format or blow up when trying to write out the signals index. This fixes it to where we query for the source index format as an ISO8601 and when we copy the date time format we copy it back out as ISO8601 and insert it into the signal index as ISO8601.

See this [gist](https://gist.github.com/FrankHassanabad/f614ec9762d59cd1129b3269f5bae41c) for more details of how this was accidentally introduced when we added support for runtime fields and the general idea of the fix.

* Removes `docvalue_field` and we now only use `fields` in detection engine search requests
* Splits out the timestamp e2e tests into their own file for `timestamps` file
* Adds more tests to ensure we copy what we expect and we are converting to ISO8601 in the signals
* Removes `ts-expect-error` in a lot of areas including tests and then I fix the types and issues once it is removed. 

### Checklist

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
2021-06-04 16:19:59 -06:00
..
.storybook
common [Security Solution][Endpoint] Add ability to isolate the Host from the Endpoint Details flyout (#100482) 2021-06-04 10:59:53 -04:00
cypress [Security Solution] [Bug Fix] Fix flakey cypress tests (#101231) 2021-06-02 19:37:11 -04:00
public [Security Solution] Truncate long policy name in admin tab (#101105) 2021-06-04 15:29:38 -05:00
scripts [Security Solution] Utilizes constants package and deletes duplicate code (#100513) 2021-05-24 18:38:14 -06:00
server [Security Solutions][Detection Engine] Fixes timestamp bugs within source indexes when the formats are not ISO8601 format (#101349) 2021-06-04 16:19:59 -06:00
jest.config.js [jest/securitySolution] Switch to jest-environment-jsdom (#96255) 2021-04-06 12:06:14 -04:00
kibana.json [RAC][Security Solution] Register Security Detection Rules with Rule Registry (#96015) 2021-05-28 12:38:49 -06:00
package.json [SecuritySolution] remove files that are not in used (#97740) 2021-04-20 19:18:28 -04:00
README.md
tsconfig.json security solution and lists to TS incremental builds (#100447) 2021-05-24 16:34:29 +02:00

Security Solution

Welcome to the Kibana Security Solution plugin! This README will go over getting started with development and testing.

Development

Tests

The endpoint specific tests leverage the ingest manager to install the endpoint package. Before the api integration and functional tests are run the ingest manager is initialized. This initialization process includes reaching out to a package registry service to install the endpoint package. The endpoint tests support three different ways to run the tests given the constraint on an available package registry.

  1. Using Docker
  2. Running your own local package registry
  3. Using the default external package registry

These scenarios will be outlined the sections below.

Endpoint API Integration Tests Location

The endpoint api integration tests are located here

Endpoint Functional Tests Location

The endpoint functional tests are located here

Using Docker

To run the tests using the recommended docker image version you must have docker installed. The testing infrastructure will stand up a docker container using the image defined here

Make sure you're in the Kibana root directory.

Endpoint API Integration Tests

In one terminal, run:

FLEET_PACKAGE_REGISTRY_PORT=12345 yarn test:ftr:server --config x-pack/test/security_solution_endpoint_api_int/config.ts

In another terminal, run:

FLEET_PACKAGE_REGISTRY_PORT=12345 yarn test:ftr:runner --config x-pack/test/security_solution_endpoint_api_int/config.ts

Endpoint Functional Tests

In one terminal, run:

FLEET_PACKAGE_REGISTRY_PORT=12345 yarn test:ftr:server --config x-pack/test/security_solution_endpoint/config.ts

In another terminal, run:

FLEET_PACKAGE_REGISTRY_PORT=12345 yarn test:ftr:runner --config x-pack/test/security_solution_endpoint/config.ts

Running your own package registry

If you are doing endpoint package development it will be useful to run your own package registry to serve the latest package you're building. To do this use the following commands:

Make sure you're in the Kibana root directory.

Endpoint API Integration Tests

In one terminal, run:

PACKAGE_REGISTRY_URL_OVERRIDE=<url to your package registry like http://localhost:8080> yarn test:ftr:server --config x-pack/test/security_solution_endpoint_api_int/config.ts

In another terminal, run:

PACKAGE_REGISTRY_URL_OVERRIDE=<url to your package registry like http://localhost:8080>  yarn test:ftr:runner --config x-pack/test/security_solution_endpoint_api_int/config.ts

Endpoint Functional Tests

In one terminal, run:

PACKAGE_REGISTRY_URL_OVERRIDE=<url to your package registry like http://localhost:8080> yarn test:ftr:server --config x-pack/test/security_solution_endpoint/config.ts

In another terminal, run:

PACKAGE_REGISTRY_URL_OVERRIDE=<url to your package registry like http://localhost:8080>  yarn test:ftr:runner --config x-pack/test/security_solution_endpoint/config.ts

Using the default public registry

If you don't have docker installed and don't want to run your own registry, you can run the tests using the ingest manager's default public package registry. The actual package registry used is here

Make sure you're in the Kibana root directory.

Endpoint API Integration Tests

In one terminal, run:

yarn test:ftr:server --config x-pack/test/security_solution_endpoint_api_int/config.ts

In another terminal, run:

yarn test:ftr:runner --config x-pack/test/security_solution_endpoint_api_int/config.ts

Endpoint Functional Tests

In one terminal, run:

yarn test:ftr:server --config x-pack/test/security_solution_endpoint/config.ts

In another terminal, run:

yarn test:ftr:runner --config x-pack/test/security_solution_endpoint/config.ts