fluent-plugin-record-reformer

Fluentd plugin to add or replace fields of a event record

Installation

Use RubyGems:

gem install fluent-plugin-record-reformer

Configuration

Example:

<match foo.**>
  type record_reformer
  remove_keys remove_me
  renew_record false
  enable_ruby false
  
  tag reformed.${tag_prefix[-2]}
  <record>
    hostname ${hostname}
    input_tag ${tag}
    last_tag ${tag_parts[-1]}
    message ${message}, yay!
  </record>
</match>

Assume following input is coming (indented):

foo.bar {
  "remove_me":"bar",
  "not_remove_me":"bar",
  "message":"Hello world!"
}

then output becomes as below (indented):

reformed.foo {
  "not_remove_me":"bar",
  "hostname":"YOUR_HOSTNAME",
  "input_tag":"foo.bar",
  "last_tag":"bar",
  "message":"Hello world!, yay!",
}

Configuration (Classic Style)

Example:

<match foo.**>
  type record_reformer
  remove_keys remove_me
  renew_record false
  enable_ruby false
  tag reformed.${tag_prefix[-2]}
  
  hostname ${hostname}
  input_tag ${tag}
  last_tag ${tag_parts[-1]}
  message ${message}, yay!
</match>

This results in same, but please note that following option parameters are reserved, so can not be used as a record key.

Option Parameters

• output_tag (obsolete)

  The output tag name. This option is deprecated. Use tag option instead

• tag

  The output tag name.

• remove_keys

  Specify record keys to be removed by a string separated by , (comma) like

    remove_keys message,foo
  

• renew_record bool

  renew_record true creates an output record newly without extending (merging) the input record fields. Default is false.

• keep_keys

  You may want to remain some record fields although you specify renew_record true. Then, specify record keys to be kept by a string separated by , (comma) like

    keep_keys message,foo
  

• enable_ruby bool

  Enable to use ruby codes in placeholders. See Placeholders section. Default is true (just for lower version compatibility).

• query_key string

  Specifies record that we can attempt to parse a query string params from. If given, unpacks query params found in the query_key key and adds them to the top-level record, prefixed with query_prefix (See query_prefix below)

• query_prefix string

  Specifies a prefix for record keys derived from query params found in the query_key key. (Defaults to 'q') eg. if query_prefix is 'q', and query_key is 'request' then given,

  "request" => "/path/query?this=is&another=query"

  the resulting keys on the top-level record will be

  "q:this" => "is" "q:another" => "query"

Placeholders

The keys of input json are available as placeholders. In the above example,

• ${remove_me}
• ${not_remove_me}
• ${message}

shall be available. In addition, following placeholders are reserved:

• ${hostname} Hostname of the running machine

• ${tag} Input tag

• ${time} Time of the event

• ${tags[N]} (Obsolete. Use tag_parts) Input tag splitted by '.'

• ${tag_parts[N]} Input tag splitted by '.' indexed with N such as ${tag_parts[0]}, ${tag_parts[-1]}.

• ${tag_prefix[N]} Tag parts before and on the index N. For example,

    Input tag: prefix.test.tag.suffix
    
    ${tag_prefix[0]}  => prefix
    ${tag_prefix[1]}  => prefix.test
    ${tag_prefix[-2]} => prefix.test.tag
    ${tag_prefix[-1]} => prefix.test.tag.suffix
  

• ${tag_suffix[N]} Tag parts after and on the index N. For example,

    Input tag: prefix.test.tag.suffix
  
    ${tag_suffix[0]}  => prefix.test.tag.suffix
    ${tag_suffix[1]}  => test.tag.suffix
    ${tag_suffix[-2]} => tag.suffix
    ${tag_suffix[-1]} => suffix
  

It is also possible to write a ruby code in placeholders if you set enable_ruby true option, so you may write some codes as

• ${time.strftime('%Y-%m-%dT%H:%M:%S%z')}
• ${tag_parts.last}

but, please note that enabling ruby codes is not encouraged by security reasons and also in terms of the performance.

Relatives

Following plugins look similar:

• fluent-plugin-record-reformer The upstream project
• fluent-plugin-record-modifier
• fluent-plugin-format
• fluent-plugin-add
• filter_record_transformer is a Fluentd v0.12 built-in plugin which is based on record-reformer.

ChangeLog

See CHANGELOG.md for details.

Contributing

1. Fork it
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create new Pull Request

Copyright

Copyright (c) 2013 - 2015 Naotoshi Seo. See LICENSE for details.