Fluent Bit

Introduction

Fluent Bit is a super-fast, lightweight, and highly scalable logging and metrics processor and forwarder. It is the preferred choice for cloud and containerized environments. OpsRamp allows you to export logs from Fluent Bit, which is already configured to collect logs from various sources.

Configuration for exporting logs to OpsRamp

Add the following configuration to your Fluent Bit configuration file to export logs to OpsRamp:

[FILTER]
    Name modify
    Match *
    Add source fluent-bit
    Rename time timestamp
    Rename log message
    Rename _HOSTNAME host
[OUTPUT]
    name  http
    match *
    host 
    port 443
    tls on
    uri 
    format json

[FILTER]
    Name modify
    Match *
    Add source fluent-bit
    Rename time timestamp
    Rename log message
    Rename _HOSTNAME host
[OUTPUT]
    name  http
    match *
    host 
    port 443
    tls on
    uri 
    format msgpack

Example 1: Fluent Bit configuration for exporting logs to OpsRamp

Below is a complete configuration example for Fluent Bit to export logs to OpsRamp and specify the source of the logs using the service label:

[SERVICE]
    # Flush
    # =====
    # set an interval of seconds before to flush records to a destination
    flush        1

    # Daemon
    # ======
    # instruct Fluent Bit to run in foreground or background mode.
    daemon       Off
 
    # Log_Level
    # =========
    # Set the verbosity level of the service, values can be:
    #
    # - error
    # - warning
    # - info
    # - debug
    # - trace
    #
    # by default 'info' is set, that means it includes 'error' and 'warning'.
    log_level    debug
 
    # Parsers File
    # ============
    # specify an optional 'Parsers' configuration file
    parsers_file parsers.conf
 
    # Plugins File
    # ============
    # specify an optional 'Plugins' configuration file to load external plugins.
    plugins_file plugins.conf
 
    # HTTP Server
    # ===========
    # Enable/Disable the built-in HTTP Server for metrics
    http_server  Off
    http_listen  0.0.0.0
    http_port    2020
 
    # Storage
    # =======
    # Fluent Bit can use memory and filesystem buffering based mechanisms
    #
    # - https://docs.fluentbit.io/manual/administration/buffering-and-storage
    #
    # storage metrics
    # ---------------
    # publish storage pipeline metrics in '/api/v1/storage'. The metrics are
    # exported only if the 'http_server' option is enabled.
    #
    storage.metrics on
 
    # storage.path
    # ------------
    # absolute file system path to store filesystem data buffers (chunks).
    #
    # storage.path /tmp/storage
 
    # storage.sync
    # ------------
    # configure the synchronization mode used to store the data into the
    # filesystem. It can take the values normal or full.
    #
    # storage.sync normal
 
    # storage.checksum
    # ----------------
    # enable the data integrity check when writing and reading data from the
    # filesystem. The storage layer uses the CRC32 algorithm.
    #
    # storage.checksum off
 
    # storage.backlog.mem_limit
    # -------------------------
    # if storage.path is set, Fluent Bit will look for data chunks that were
    # not delivered and are still in the storage layer, these are called
    # backlog data. This option configure a hint of maximum value of memory
    # to use when processing these records.
    #
    # storage.backlog.mem_limit 5M

[INPUT]
    name tail
    path /var/log/syslog
    key message

[FILTER]
    Name record_modifier
    Match *
    Record host ${HOSTNAME}
    Record source fluent-bit

[FILTER]
    Name modify
    Match *
    Rename date timestamp

[OUTPUT]
    name  http
    match *
    host 
    port 443
    tls on
    uri 
    format json

Example 2: Fluent Bit configuration with resource attributes

Before exporting the log events, ensure that the following attributes or fields are set using Fluent Bit configuration:

Resource attributes:

• source
• host
• level (If not set, it is considered as “Unknown”)

Parsed labels:

• message # Mandatory field to be set
• timestamp ( If not set, the time the record received at Opsramp is considered as log record time )
• level ( If not set, it’s considered as “Unknown” )

See Fluent-Bit configuration for more details.