120 lines
4.6 KiB
Plaintext
120 lines
4.6 KiB
Plaintext
[role="xpack"]
|
|
[[xpack-grokdebugger]]
|
|
== Debugging grok expressions
|
|
|
|
You can build and debug grok patterns in the {kib} *Grok Debugger*
|
|
before you use them in your data processing pipelines. Grok is a pattern
|
|
matching syntax that you can use to parse arbitrary text and
|
|
structure it. Grok is good for parsing syslog, apache, and other
|
|
webserver logs, mysql logs, and in general, any log format that is
|
|
written for human consumption.
|
|
|
|
Grok patterns are supported in the ingest node
|
|
{ref}/grok-processor.html[grok processor] and the Logstash
|
|
{logstash-ref}/plugins-filters-grok.html[grok filter]. See
|
|
{logstash-ref}/plugins-filters-grok.html#_grok_basics[grok basics]
|
|
for more information on the syntax for a grok pattern.
|
|
|
|
The Elastic Stack ships
|
|
with more than 120 reusable grok patterns. See
|
|
https://github.com/elastic/elasticsearch/tree/master/libs/grok/src/main/resources/patterns[Ingest node grok patterns] and https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns[Logstash grok patterns]
|
|
for the complete list of patterns.
|
|
|
|
Because
|
|
ingest node and Logstash share the same grok implementation and pattern
|
|
libraries, any grok pattern that you create in the *Grok Debugger* will work
|
|
in ingest node and Logstash.
|
|
|
|
[float]
|
|
[[grokdebugger-getting-started]]
|
|
=== Getting started with the Grok Debugger
|
|
|
|
This example walks you through using the *Grok Debugger*. This tool
|
|
is automatically enabled in {kib}.
|
|
|
|
NOTE: If you're using {security}, you must have the `manage_pipeline`
|
|
permission to use the Grok Debugger.
|
|
|
|
. In the side navigation, click *DevTools*, then open the *Grok Debugger*.
|
|
. In *Sample Data*, enter a message that is representative of the data that you
|
|
want to parse. For example:
|
|
+
|
|
[source,ruby]
|
|
-------------------------------------------------------------------------------
|
|
55.3.244.1 GET /index.html 15824 0.043
|
|
-------------------------------------------------------------------------------
|
|
|
|
. In *Grok Pattern*, enter the grok pattern that you want to apply to the data.
|
|
+
|
|
To parse the log line in this example, use:
|
|
+
|
|
[source,ruby]
|
|
-------------------------------------------------------------------------------
|
|
%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}
|
|
-------------------------------------------------------------------------------
|
|
|
|
. Click **Simulate**.
|
|
+
|
|
You'll see the simulated event that results from applying the grok
|
|
pattern.
|
|
+
|
|
[role="screenshot"]
|
|
image::dev-tools/grokdebugger/images/grok-debugger-overview.png["Grok Debugger"]
|
|
|
|
|
|
//TODO: Update LS and ingest node docs with pointers to the new grok debugger. Replace references to the Heroku app.
|
|
|
|
[float]
|
|
[[grokdebugger-custom-patterns]]
|
|
=== Testing custom patterns
|
|
|
|
If the default grok pattern dictionary doesn't contain the patterns you need,
|
|
you can define, test, and debug custom patterns using the Grok Debugger.
|
|
|
|
Custom patterns that you enter in the Grok Debugger are not saved. Custom patterns
|
|
are only available for the current debugging session and have no side effects.
|
|
|
|
Follow this example to define a custom pattern.
|
|
|
|
. In *Sample Data*, enter the following sample message:
|
|
+
|
|
[source,ruby]
|
|
-------------------------------------------------------------------------------
|
|
Jan 1 06:25:43 mailserver14 postfix/cleanup[21403]: BEF25A72965: message-id=<20130101142543.5828399CCAF@mailserver14.example.com>
|
|
-------------------------------------------------------------------------------
|
|
|
|
. Enter this grok pattern:
|
|
+
|
|
[source,ruby]
|
|
-------------------------------------------------------------------------------
|
|
%{SYSLOGBASE} %{POSTFIX_QUEUEID:queue_id}: %{MSG:syslog_message}
|
|
-------------------------------------------------------------------------------
|
|
+
|
|
Notice that the grok pattern references custom patterns called `POSTFIX_QUEUEID`
|
|
and `MSG`.
|
|
|
|
. Expand **Custom Patterns** and enter pattern definitions for the custom
|
|
patterns that you want to use in the grok expression. You must specify each pattern definition
|
|
on its own line.
|
|
+
|
|
For this example, you must specify pattern definitions
|
|
for `POSTFIX_QUEUEID` and `MSG`:
|
|
+
|
|
[source,ruby]
|
|
-------------------------------------------------------------------------------
|
|
POSTFIX_QUEUEID [0-9A-F]{10,11}
|
|
MSG message-id=<%{GREEDYDATA}>
|
|
-------------------------------------------------------------------------------
|
|
|
|
. Click **Simulate**.
|
|
+
|
|
You'll see the simulated output event that results from applying
|
|
the grok pattern that contains the custom pattern:
|
|
+
|
|
[role="screenshot"]
|
|
image::dev-tools/grokdebugger/images/grok-debugger-custom-pattern.png["Debugging a custom pattern"]
|
|
+
|
|
If an error occurs, you can continue iterating over
|
|
the custom pattern until the output matches the event
|
|
that you expect.
|