## Summary Phase 1 of a multi-phase cautious approach for adding an experimental application cache for Kibana solutions called `metric_entities` and integrates it within Security Solutions. Phase 1 is putting experimental support into the application without breaking existing features. Lots of TODO's, conversations and a possible RFC from phase 1 to phase 2 approach. Some features are missing, but for phase 1 the general idea and code is all there. To enable this first phase after checking out the branch add this to your `kibana.dev.yml` ```yml xpack.metricsEntities.enabled: true xpack.securitySolution.enableExperimental: ['metricsEntitiesEnabled'] ``` Then go into Stack Management -> Advanced Settings (Under Security Solutions) and set the enabled to true like so: <img width="1229" alt="Screen Shot 2021-04-08 at 2 21 02 PM" src="https://user-images.githubusercontent.com/1151048/114091276-b3cbb700-9875-11eb-9083-5c1d91dd20ed.png"> Next go to the security_solutions page and you will see it being activated and you will have these transforms running if you look under stack management: <img width="1710" alt="Screen Shot 2021-04-29 at 2 00 27 PM" src="https://user-images.githubusercontent.com/1151048/116611174-4a2e4e00-a8f3-11eb-9e15-55cb504dfb2a.png"> On the hosts page, network, page, etc... You can see them being activated when you have no query/filter and you click on request: <img width="1405" alt="Screen Shot 2021-04-29 at 2 01 28 PM" src="https://user-images.githubusercontent.com/1151048/116611274-6a5e0d00-a8f3-11eb-9998-9f5b3d1c5c63.png"> You will see in the request the index patterns all starting with `estc_xyz*` ### Checklist Delete any items that are not applicable to this PR. - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md) We have lots of TODO's but no concrete docs with this just yet. - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials Behind a feature flag and this isn't there yet. - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios |
||
---|---|---|
.. | ||
common | ||
server | ||
jest.config.js | ||
kibana.json | ||
README.md | ||
tsconfig.json |
metrics_entities
This is the metrics and entities plugin where you add can add transforms for your project and group those transforms into modules. You can also re-use existing transforms in your modules as well.
Turn on experimental flags
During at least phase 1 of this development, please add these to your kibana.dev.yml
file to turn on the feature:
xpack.metricsEntities.enabled: true
xpack.securitySolution.enableExperimental: ['metricsEntitiesEnabled']
Quick start on using scripts to call the API
The scripts rely on CURL and jq:
Install curl and jq
brew update
brew install curl
brew install jq
Open $HOME/.zshrc
or ${HOME}.bashrc
depending on your SHELL output from echo $SHELL
and add these environment variables:
export ELASTICSEARCH_USERNAME=${user}
export ELASTICSEARCH_PASSWORD=${password}
export ELASTICSEARCH_URL=https://${ip}:9200
export KIBANA_URL=http://localhost:5601
source $HOME/.zshrc
or ${HOME}.bashrc
to ensure variables are set:
source ~/.zshrc
Restart Kibana and ensure that you are using --no-base-path
as changing the base path is a feature but will
get in the way of the CURL scripts written as is.
Go to the scripts folder cd kibana/x-pack/plugins/metrics_entities/server/scripts
and can run some of the scripts
such as:
./post_transforms.sh ./post_examples/all.json
which will post transforms from the all.json
You can also delete them by running:
./delete_transforms.sh ./delete_examples/all.json
See the folder for other curl scripts that exercise parts of the REST API and feel free to add your own examples in the folder as well.
Quick start on how to add a transform
You will want to figure out how you want your transform from within Kibana roughly using the UI and then copy the JSON. The JSON you will want to change and paste within a folder which represents a module.
For example, for the host_entities
and a host_entities_mapping
we created a folder called host_entities
here:
sever/modules/host_entities
Then we add two files, a subset of the transform JSON and a mapping like so:
server/modules/host_entities/host_entities_mapping.json <--- this is the mappings
server/modules/host_entities/host_entities.json <--- This is a subset of the transform JSON
index.ts <--- Import/export your json here
The mappings can be normal mapping like so with host_entities_mapping.json
:
{
"mappings": {
"_meta": {
"index": "host_ent"
},
"dynamic": "strict",
"properties": {
"@timestamp": {
"type": "date"
},
"metrics": {
"properties": {
"host": {
"properties": {
"name": {
"properties": {
"value_count": {
"type": "long"
}
}
}
}
}
}
},
"host": {
"properties": {
"name": {
"type": "keyword"
},
"os": {
"properties": {
"name": {
"type": "keyword"
},
"version": {
"type": "keyword"
}
}
}
}
}
}
}
}
One caveat is that you need to add this to the meta section to tell it what the name will be:
"_meta": {
"index": "host_ent"
},
Keep the name short as there is only 65 characters for a transform job and we prepend extra information to the mapping such as:
- prefix
- name of estc
Although not required, a "dynamic": "strict"
is strongly encouraged to prevent mapping guesses from elastic and it will be better for us
to spot errors quicker in the mappings such as type-o's if this is set to strict.
Next, for the transform, you should add a subset that doesn't have any additional settings or meta associated like so for host_entities.json
:
{
"id": "host_ent",
"description": "[host.name entities] grouped by @timestamp, host.name, os.name, and os.version, and aggregated on host.name",
"pivot": {
"group_by": {
"@timestamp": {
"date_histogram": {
"field": "@timestamp",
"calendar_interval": "1h"
}
},
"host.name": {
"terms": {
"field": "host.name"
}
},
"host.os.name": {
"terms": {
"field": "host.os.name",
"missing_bucket": true
}
},
"host.os.version": {
"terms": {
"field": "host.os.version",
"missing_bucket": true
}
}
},
"aggregations": {
"metrics.host.name.value_count": {
"value_count": {
"field": "host.name"
}
}
}
}
}
Look in the server/modules
for other examples, but it should be that clear cut. The final part is to wire everything up in the code by touching a few files
to either add this to an existing module or create your own module. In server/module/host_entities
we add an index.ts
like so that does an import/export
of the JSON:
import hostEntities from './host_entities.json';
import hostEntitiesMapping from './host_entities_mapping.json';
export { hostEntities, hostEntitiesMapping };
Then in modules/index.ts
we add a new module name if we are creating a new module to the export enum ModuleNames {
like so:
// Import your host entities you just made
import { hostEntities, hostEntitiesMapping } from './host_entities';
/**
* These module names will map 1 to 1 to the REST interface.
*/
export enum ModuleNames {
hostSummaryMetrics = 'host_metrics',
hostSummaryEntities = 'host_entities', // <-- Add the entities/transform and give it a enum name and a module name
networkSummaryEntities = 'network_entities',
networkSummaryMetrics = 'network_metrics',
userSummaryEntities = 'user_entities',
userSummaryMetrics = 'user_metrics',
}
If you're not creating a new module but rather you are adding to an existing module, you can skip the above step. Next, you
just need to add your installable transform and installable mapping to the two data structures of installableTransforms
and
installableMappings
like so:
/**
* Add any new folders as modules with their names below and grouped with
* key values.
*/
export const installableTransforms: Record<ModuleNames, Transforms[]> = {
[ModuleNames.hostSummaryMetrics]: [hostMetrics],
[ModuleNames.hostSummaryEntities]: [hostEntities], // <-- Adds my new module name and transform to a new array.
[ModuleNames.networkSummaryEntities]: [
destinationIpEntities, // <-- If instead I am adding to an existing module, I just add it to the array like these show
sourceIpEntities,
destinationCountryIsoCodeEntities,
sourceCountryIsoCodeEntities,
],
[ModuleNames.networkSummaryMetrics]: [ipMetrics],
[ModuleNames.userSummaryEntities]: [userEntities],
[ModuleNames.userSummaryMetrics]: [userMetrics],
};
/**
* For all the mapping types, add each with their names below and grouped with
* key values.
*/
export const installableMappings: Record<ModuleNames, Mappings[]> = {
[ModuleNames.hostSummaryMetrics]: [hostMetricsMapping],
[ModuleNames.hostSummaryEntities]: [hostEntitiesMapping], // <-- Adds my new module name and mapping to a new array.
[ModuleNames.networkSummaryEntities]: [ // <-- If instead I am adding to an existing module, I just add it to the array like these show
sourceIpEntitiesMapping,
destinationIpEntitiesMapping,
destinationCountryIsoCodeEntitiesMapping,
sourceCountryIsoCodeEntitiesMapping,
],
[ModuleNames.networkSummaryMetrics]: [ipMetricsMapping],
[ModuleNames.userSummaryEntities]: [userEntitiesMapping],
[ModuleNames.userSummaryMetrics]: [userMetricsMapping],
};
And after that, you should check out if there are any existing e2e tests or unit tests to update here to ensure that your mapping and transform will pass ci. Create a pull request and your mapping and transform are completed.
To call into the code to activate your module and create your transforms and mappings would be the following where you substitute your ${KIBANA_URL} with your kibana URL and the ${SPACE_URL} with any space id you have. If you're using the default space then you would use an empty string:
POST ${KIBANA_URL}${SPACE_URL}/api/metrics_entities/transforms
{
"prefix": "all",
"modules": [
"host_entities",
],
"indices": [
"auditbeat-*",
"endgame-*",
"filebeat-*",
"logs-*",
"packetbeat-*",
"winlogbeat-*",
"-*elastic-cloud-logs-*"
],
"auto_start": true,
"settings": {
"max_page_search_size": 5000
},
"query": {
"range": {
"@timestamp": {
"gte": "now-1d/d",
"format": "strict_date_optional_time"
}
}
}
}
Very similar to the regular transform REST API, with the caveats that you define which modules you want to install, the prefix name you want to use, and
if you want to auto_start
it or not. The rest such as settings
, query
will be the same as the transforms API. They will also push those same setting into
each of your transforms within your module(s) as the same setting for each individual ones.
TODO List
During the phase 1, phase 2, phase N, this TODO will exist as a reminder and notes for what still needs to be developed. These are not in a priority order, but are notes during the phased approach. As we approach production and the feature flags are removed these TODO's should be removed in favor of Kibana issues or regular left over TODO's in the code base.
- Add these properties to the route which are:
- disable_transforms/exclude flag to exclude 1 or more transforms within a module,
- pipeline flag,
- Change the REST routes on post to change the indexes for whichever indexes you want
- Unit tests to ensure the data of the mapping.json includes the correct fields such as _meta, at least one alias, a mapping section, etc...
- Add text/keyword and other things to the mappings (not just keyword maybe?) ... At least review the mappings one more time
- Add a sort of @timestamp to the output destination indexes?
- Add the REST Kibana security based tags if needed and push those to any plugins using this plugin. Something like: tags: ['access:metricsEntities-read'] and ['access:metricsEntities-all'],
- Add schema validation choosing some schema library (io-ts or Kibana Schema or ... )
- Add unit tests
- Add e2e tests
- Move ui code into this plugin from security_solutions? (maybe?)
- UI code could be within
kibana/packages
instead of in here directly and I think we will be better off.
- UI code could be within