module LogStash

The Dissect filter is a kind of split operation. Unlike a regular split operation where one delimiter is applied to the whole string, this operation applies a set of delimiters # to a string value. + Dissect does not use regular expressions and is very fast. + However, if the structure of your text varies from line to line then Grok is more suitable. + There is a hybrid case where Dissect can be used to de-structure the section of the line that is reliably repeated and then Grok can be used on the remaining field values with # more regex predictability and less overall work to do. +

A set of fields and delimiters is called a dissection.

The dissection is described using a set of `%{}` sections: .… %{a} - %{b} - %{c} .…

A field is the text from `%` to `}` inclusive.

A delimiter is the text between `}` and `%` characters.

NOTE

delimiters can't contain these `}{%` characters.

The config might look like this: .…

filter {
  dissect {
    mapping => {
      "message" => "%{ts} %{+ts} %{+ts} %{src} %{} %{prog}[%{pid}]: %{msg}"
    }
  }
}

.… When dissecting a string from left to right, text is captured upto the first delimiter - this captured text is stored in the first field. This is repeated for each field/# delimiter pair thereafter until the last delimiter is reached, then *the remaining text is stored in the last field*. +

*The Key:* + The key is the text between the `%{` and `}`, exclusive of the ?, +, & prefixes and the ordinal suffix. + `%{?aaa}` - key is `aaa` + `%{+bbb/3}` - key is `bbb` + `%{&ccc}` - key is `ccc` +

*Normal field notation:* + The found value is added to the Event using the key. + `%{some_field}` - a normal field has no prefix or suffix

*Skip field notation:* + The found value is stored internally but not added to the Event. + The key, if supplied, is prefixed with a `?`.

`%{}` is an empty skip field.

`%{?foo}` is a named skip field.

*Append field notation:* + The value is appended to another value or stored if its the first field seen. + The key is prefixed with a `+`. + The final value is stored in the Event using the key. +

NOTE

¶ ↑

The delimiter found before the field is appended with the value. + If no delimiter is found before the field, a single space character is used.

¶ ↑

`%{+some_field}` is an append field. + `%{+some_field/2}` is an append field with an order modifier.

An order modifier, `/digits`, allows one to reorder the append sequence. + e.g. for a text of `1 2 3 go`, this `%{+a/2} %{+a/1} %{+a/4} %{+a/3}` will build a key/value of `a => 2 1 go 3` + Append fields without an order modifier will append in declared order. + e.g. for a text of `1 2 3 go`, this `%{a} %{b} %{+a}` will build two key/values of `a => 1 3 go, b => 2` +

*Indirect field notation:* + The found value is added to the Event using the found value of another field as the key. + The key is prefixed with a `&`. + `%{&some_field}` - an indirect field where the key is indirectly sourced from the value of `some_field`. + e.g. for a text of `error: some_error, some_description`, this `error: %{?err}, %{&err}` will build a key/value of `some_error => some_description`.

NOTE

for append and indirect field the key can refer to a field that already exists in the event before dissection.

NOTE

use a Skip field if you do not want the indirection key/value stored.

e.g. for a text of `google: 77.98`, this `%{?a}: %{&a}` will build a key/value of `google => 77.98`.

NOTE

¶ ↑

append and indirect cannot be combined and will fail validation. + `%{+&something}` - will add a value to the `&something` key, probably not the intended outcome. + `%{&+something}` will add a value to the `+something` key, again probably unintended. +

¶ ↑

*Delimiter repetition:* + In the source text if a field has variable width padded with delimiters, the padding will be ignored. + e.g. for texts of: .… 00000043 ViewReceiver I 000000b3 Peer I .… with a dissection of `%{a} %{b} %{c}`; the padding is ignored, `event.get() -> “I”`

NOTE

¶ ↑

You probably want to use this filter inside an `if` block. + This ensures that the event contains a field value with a suitable structure for the dissection.

¶ ↑

For example… .… filter {

if [type] == "syslog" or "syslog" in [tags] {
  dissect {
    mapping => {
      "message" => "%{ts} %{+ts} %{+ts} %{src} %{} %{prog}[%{pid}]: %{msg}"
    }
  }
}

} .…