kibana/x-pack/plugins/event_log

Event Log

Overview

The purpose of this plugin is to provide a way to persist a history of events occuring in Kibana, initially just for the Make It Action project - alerts and actions.

Basic Usage - Logging Events

Follow these steps to use eventLog in your plugin:

1. Declare eventLog as a dependency in kibana.json:

{
  ...
  "requiredPlugins": ["eventLog"],
  ...
}

2. Register provider / actions, and create your plugin's logger, using service API provided in the setup stage:

...
import { IEventLogger, IEventLogService } from '../../event_log/server';
interface PluginSetupDependencies {
  eventLog: IEventLogService;
}
...
public setup(core: CoreSetup, { eventLog }: PluginSetupDependencies) {
  ...
  eventLog.registerProviderActions('my-plugin', ['action-1, action-2']);
  const eventLogger: IEventLogger = eventLog.getLogger({ event: { provider: 'my-plugin' } });
  ...
}
...

4. To log an event, call logEvent() on the eventLogger object you created:

...
  eventLogger.logEvent({ event: { action: 'action-1' }, tags: ['fe', 'fi', 'fo'] });
...

Testing

Unit tests

From kibana-root-folder/x-pack, run:

$ node node scripts/jest plugins/event_log

API Integration tests

None yet!

Background

For the Make It Action alerting / action plugins, we will need a way to persist data regarding alerts and actions, for UI and investigative purposes. We're referring to this persisted data as "events", and will be persisted to a new elasticsearch index referred to as the "event log".

Example events are actions firing, alerts running their scheduled functions, alerts scheduling actions to run, etc.

This functionality will be provided in a new NP plugin eventLog, and will provide server-side plugin APIs to write to the event log, and run limited queries against it. For now, access via HTTP will not be available, due to security concerns and lack of use cases.

The current clients for the event log are the actions and alerting plugins, however the event log currently has nothing specific to them, and is general purpose, so can be used by any plugin to "log events".

We currently assume that there may be many events logged, and that (some) customers may not be interested in "old" events, and so to keep the event log from consuming too much disk space, we'll set it up with ILM and some kind of reasonable default policy that can be customized by the user. This implies also the use of rollver, setting a write index alias upon rollover, and that searches for events will be done via an ES index pattern / alias to search across event log indices with a wildcard.

The shape of the documents indexed into the event log index is a subset of ECS properties with a few Kibana extensions. Over time the subset is of ECS and Kibana extensions will likely grow.

Basic example

When an action is executed, an event should be written to the event log.

Here's a kbn-action command to execute a "server log" action (writes a message to the Kibana log):

$ kbn-action execute 79b4c37e-ef42-4421-a0b0-b536840f930d '{level:info message:hallo}'
{
    "status": "ok"
}

Here's the event written to the event log index:

{
  "_index": ".kibana-event-log-000001",
  "_type": "_doc",
  "_id": "d2CXT20BPOpswQ8vgXp5",
  "_score": 1,
  "_source": {
    "event": {
      "provider": "actions",
      "action": "execute",
      "start": "2019-12-09T21:16:43.424Z",
      "end": "2019-12-09T21:16:43.425Z",
      "duration": 1000000
    },
    "kibana": {
      "saved_objects": [
        {
          "type": "action",
          "id": "79b4c37e-ef42-4421-a0b0-b536840f930d"
        }
      ]
    },
    "message": "action executed successfully: 79b4c37e-ef42-4421-a0b0-b536840f930d - .server-log - server-log",
    "@timestamp": "2019-12-09T21:16:43.425Z",
    "ecs": {
      "version": "1.3.1"
    }
  }
}

The shape of the document written to the index is a subset of ECS with an extended field of kibana with some Kibana-related properties contained within it.

The ES mappings for the ECS data, and the config-schema for the ECS data, are generated by a script, and available here:

• generated/mappings.json
• generated/schemas.ts

It's anticipated that these interfaces will grow over time, hopefully adding more ECS fields but adding Kibana extensions as required.

Since there are some security concerns with the data, we are currently restricting access via known saved object ids. That is, you can only query history records associated with specific saved object ids.

API

// IEvent is a TS type generated from the subset of ECS supported

// the NP plugin returns a service instance from setup() and start()
export interface IEventLogService {
  registerProviderActions(provider: string, actions: string[]): void;
  isProviderActionRegistered(provider: string, action: string): boolean;
  getProviderActions(): Map<string, Set<string>>;

  getLogger(properties: IEvent): IEventLogger;
}

export interface IEventLogger {
  logEvent(properties: IEvent): void;
  startTiming(event: IEvent): void;
  stopTiming(event: IEvent): void;
}

The plugin exposes an IEventLogService object to plugins that pre-req it. Those plugins need to call registerProviderActions() to indicate the values of the event.provider and event.action values they will be using when logging events.

The pre-registration helps in two ways:

• dealing with misspelled values
• preventing index explosion on those fields

Once the values are registered, the plugin will get an IEventLogger instance by passing in a set of default properties to be used for all it's logging, to the getLogger() method. For instance, the actions plugin creates a logger with event.provider set to actions, and provides event.action values when writing actual entries.

The IEventLogger object can be cached at the plugin level and accessed by any code in the plugin. It has a single method to write an event log entry, logEvent(), which is passed specific properties for the event.

The final data written is a combination of the data passed to getLogger() when creating the logger, and the data passed on the logEvent() call, and then that result is validated to ensure it's complete and valid. Errors will be logged to the server log.

The logEvent() method returns no values, and is itself not asynchronous. It's a "call and forget" kind of thing. The method itself will arrange to have the ultimate document written to the index asynchronously. It's designed this way because it's not clear what a client would do with a result from this method, nor what it would do if the method threw an error. All the error processing involved with getting the data into the index is handled internally, and logged to the server log as appropriate.

The startTiming() and stopTiming() methods can be used to set the timing properties start, end, and duration in the event. For example:

    const loggedEvent: IEvent = { event: { action: 'foo' } };

    // sets event.start
    eventLogger.startTiming(loggedEvent);

    longRunningFunction();
    
    // sets event.end and event.duration
    eventLogger.stopTiming(loggedEvent);

    eventLogger.logEvent(loggedEvent);

It's anticipated that more "helper" methods like this will be provided in the future.

Stored data

The elasticsearch index for the event log will have ILM and rollover support, as customers may decide to only keep recent event documents, wanting indices with older event documents deleted, turned cold, frozen, etc. We'll supply some default values, but customers will be able to tweak these.

The index template, mappings, config-schema types, etc for the index can be found in the generated directory. These files are generated from a script which takes as input the ECS properties to use, and the Kibana extensions.

See ilm rollover action docs for more info on the is_write_index, and index.lifecycle.* properties.

Of particular note in the mappings:

• all "objects" are dynamic: 'strict' implies users can't add new fields
• all the properties are indexed

We may change some of that before releasing.

ILM setup

We'll want to provide default ILM policy, this seems like a reasonable first attempt:

PUT _ilm/policy/event_log_policy   
{
  "policy": {                       
    "phases": {
      "hot": {                      
        "actions": {
          "rollover": {             
            "max_size": "50GB",
            "max_age": "30d"
          }
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

This means that ILM would "rollover" the current index, say .kibana-event-log-8.0.0-000001 by creating a new index .kibana-event-log-8.0.0-000002, which would "inherit" everything from the index template, and then ILM will set the write index of the the alias to the new index. This would happen when the original index grew past 50 GB, or was created more than 30 days ago. After rollover, the indices will be removed after 90 days to avoid disks to fill up.

For more relevant information on ILM, see: getting started with ILM doc and write index alias behavior: