* WIP: Adding integration test * Replace threat.indicator mappings with threat.enrichments mappings The nested threat.indicator mappings were experimental, and replaced by threat.enrichmentsin ECS 1.10. While these fields are also experimental, they fix the conflict between CTI data's normal threat.indicator mappings. * Add threat.enrichments mappings to our signals template mappings event.* is no longer nested within here; it was determined that event fields were not relevant to enrichment. All relevant ECS fieldsets (file, pe, etc) are now nested under threat.enrichments. * Update snapshot with newest threat.enrichments mappings This test is a snapshot of the actual mappings applied by our templates. Looks good to me! * Update ECS types to match latest We now have two threat fields we care about for CTI, for legacy and official ECS. * Add a basic test for behavior of legacy enriched signals. They're still queryable by threat.indicator, meaning that any existing dashboards will still work. * WIP: First pass at a data migration for CTI signals * Defines reindex script to move things around * Adds integration tests to make sure the migration and new mappings work * Need to test a few more things and verify corner cases * Need to extract some helpers from tests * Bump our template version to ensure devs roll over Marshall bumped to 55, giving us 10 versions for 7.14.x updates. However, devs would not otherwise roll over and get my mapping updates without destroying their signals index and rebuilding (which is also not the same thing, exactly), so this trades having one higher signals version for a more streamlined dev workflow. * More robust guard against data migration We only attempt to migrate legacy enrichments if the document: * is a signal from an indicator match rule * has a `threat.indicator` field * does not have a `threat.enrichments` field * Minor reorder of operations to make logic clearer * Add more assertions around our signals data migration Tests a few more pieces of the resulting document, giving more confidence that it's the correct transformation (and mappings). This also modifies/anonymizes the data that was originally generated on a work machine. * Remove outdated note This was for when these tests were driven via the UI; the API is more responsive and now synchronization is currently needed here, beyond the 200 responses. * Fix typo in comment These fields are in ECS 1.11. * Update snapshot test We bumped the version previously, causing this test to become outdated. * Update ECS typings in timelines plugin These were copied from the security_solution plugin. I updated those, but neglected to update these. Until there's a better mechanism for deduplication here, I'm going to kick the can and update both for now. * Update enrichments logic to read/write from threat.enrichments * indicator match rule logic * we now simply copy from the specified indicator path, and place that in `threat.enrichments.indicator` * event enrichment API logic * We were previously returning fields from `indicator.*`, we now include the `indicator.*` suffix in order to be more consistent with the sibling `matched.*` fields * row renderer logic * removal of dataset * updates relevant to API changes above * Fix logical error in generating links from indicator fields We want to link the reference field, not a `first_seen` field. * Always include the indicator prefix in first-party indicator fields Prior to this change we would display e.g. `threatintel.indicator.foo` for investigation enrichment fields. Now that the structure has changed slightly and we return both `indicator.*` and `matched.*` fields for existing enrichents, we want to display investigation enrichment similarly. * Update indicator match rule integration tests Now that we've updated our enrichment logic, we need to update our enrichment tests. * Remove unused translation * Update example row renderer data for enriched alerts * Update parallel CTI constants to get our CTI row renderer working We were not requesting the necessary fields for our row renderer, since these constants (specifically CTI_ROW_RENDERER_FIELDS) now exist in both security_solution and the timelines plugin. I had updated the former, but only the latter is actually used. * Update CTI enrichment UI tests * Update prepackaged threat timeline template with new threat fields Also bumps the timelineTemplateVersion. * Update Indicator Match rule tests These needed three things: * Update to timeline template (see previous commit) * Changing expectations from `threat.indicator` to `threat.enrichments` * Update row renderer expectation to exclude dataset * Update mock data with newest CTI enrichment fields * Fix assertion on our threat details These fields are prefixed with `indicator` now because: 1. This data pertains to the indicator, not the match per se 2. The actual field is prefixed with indicator (or, it at least specifies an indicator in the case of a custom threat index (via threat_indicator_path)) * Update test data and tests for our field parsing helpers * Update more event-parsing tests Ths one involved updating a mock in another package. * Modify our helper function to support old filebeat indicators When we query indicators for enrichment matches, the current expectation is that we'll be querying 7.14 filebeat modules, which have an indicator path of 'threatintel.indicator'. The only place that matters on the UI is on the threat intel panel, where these indicators come back with such a prefix. This change has one behavior: it brings back the `provider` field on the Alert summary tab for queried enrichments from filebeat modules. * Update variable and method names to be more consistent with internal terminology Indicators come from a CTI index. Enrichments are the application of indicator data to other documents, and contain both indicator fields and matched context. Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> |
||
---|---|---|
.. | ||
elastic-datemath | ||
elastic-eslint-config-kibana | ||
elastic-safer-lodash-set | ||
kbn-ace | ||
kbn-analytics | ||
kbn-apm-config-loader | ||
kbn-apm-utils | ||
kbn-babel-code-parser | ||
kbn-babel-preset | ||
kbn-cli-dev-mode | ||
kbn-config | ||
kbn-config-schema | ||
kbn-crypto | ||
kbn-dev-utils | ||
kbn-docs-utils | ||
kbn-es | ||
kbn-es-archiver | ||
kbn-es-query | ||
kbn-eslint-import-resolver-kibana | ||
kbn-eslint-plugin-eslint | ||
kbn-expect | ||
kbn-field-types | ||
kbn-i18n | ||
kbn-interpreter | ||
kbn-io-ts-utils | ||
kbn-legacy-logging | ||
kbn-logging | ||
kbn-mapbox-gl | ||
kbn-monaco | ||
kbn-optimizer | ||
kbn-plugin-generator | ||
kbn-plugin-helpers | ||
kbn-pm | ||
kbn-rule-data-utils | ||
kbn-securitysolution-autocomplete | ||
kbn-securitysolution-es-utils | ||
kbn-securitysolution-hook-utils | ||
kbn-securitysolution-io-ts-alerting-types | ||
kbn-securitysolution-io-ts-list-types | ||
kbn-securitysolution-io-ts-types | ||
kbn-securitysolution-io-ts-utils | ||
kbn-securitysolution-list-api | ||
kbn-securitysolution-list-constants | ||
kbn-securitysolution-list-hooks | ||
kbn-securitysolution-list-utils | ||
kbn-securitysolution-t-grid | ||
kbn-securitysolution-utils | ||
kbn-server-http-tools | ||
kbn-server-route-repository | ||
kbn-spec-to-console | ||
kbn-std | ||
kbn-storybook | ||
kbn-telemetry-tools | ||
kbn-test | ||
kbn-test-subj-selector | ||
kbn-tinymath | ||
kbn-typed-react-router-config | ||
kbn-ui-framework | ||
kbn-ui-shared-deps | ||
kbn-utility-types | ||
kbn-utils | ||
BUILD.bazel | ||
README.md |
Kibana-related packages
This folder contains packages that are intended for use in Kibana and Kibana plugins.
tl;dr:
- Don't publish to npm registry
- Always use the
@kbn
namespace - Always set
"private": true
inpackage.json
Using these packages
We no longer publish these packages to the npm registry. Now, instead of specifying a version when including these packages, we rely on yarn workspaces, which sets up a symlink to the package.
For example if you want to use the @kbn/i18n
package in Kibana itself, you
can specify the dependency like this:
"@kbn/i18n": "1.0.0"
However, if you want to use this from a Kibana plugin, you need to use a link:
dependency and account for the relative location of the Kibana repo, so it would
instead be:
"@kbn/i18n": "link:../../kibana/packages/kbn-i18n"
How all of this works is described in more detail in the
@kbn/pm
docs.
Creating a new package
Create a new sub-folder. The name of the folder should mirror the name
in the
package's package.json
. E.g. if the name is @kbn/i18n
the folder name
should be kbn-i18n
.
All new packages should use the @kbn
namespace, and should be marked with
"private": true
.
Unit tests for a package
Currently there is only one tool being used in order to test packages which is Jest. Below we will explain how it should be done.
Jest
A package should follow the pattern of having .test.js
files as siblings of the source code files, and these run by Jest.
A package using the .test.js
naming convention will have those tests automatically picked up by Jest and run by the unit test runner, currently mapped to the Kibana test
script in the root package.json
.
yarn test
oryarn grunt test
runs all unit tests.yarn jest
runs all Jest tests in Kibana.
In order for the plugin or package to use Jest, a jest.config.js file must be present in it's root. However, there are safeguards for this in CI should a test file be added without a corresponding config file.
Each package can also specify its own test
script in the package's package.json
, for cases where you'd prefer to run the tests from the local package directory.