Zum Inhalt springen

Fluent Bit: Write log data with Loki API

Zuletzt aktualisiert am

Fluent Bit is a popular agent for collecting logs and shipping them to multiple destinations. It has a built-in plugin available for Loki as an output which can be configured to target a STACKIT Logs instance using the Loki API.

To authenticate requests, STACKIT Logs supports two authentication methods:

  • STACKIT Service Account (recommended): Because Fluent Bit does not natively support OAuth2 JWT bearer token authentication, you can pair Fluent Bit with Grafana Alloy running as a sidecar proxy. Fluent Bit sends logs to Grafana Alloy, which exchanges service account credentials for short-lived JWT access tokens and exports the logs to STACKIT Logs.
  • Instance Access Token: Uses the bearer_token option directly in Fluent Bit’s loki output plugin with a static bearer token created in your Logs instance.

Section titled “Authenticate with a STACKIT Service Account (Recommended)”

Using a STACKIT Service Account is the recommended approach for automated and production workloads. Because Fluent Bit does not support the OAuth2 JWT bearer profile (urn:ietf:params:oauth:grant-type:jwt-bearer), it cannot authenticate directly with STACKIT IAM using service account keys.

To use service account authentication with Fluent Bit, deploy Grafana Alloy alongside Fluent Bit as a sidecar proxy:

  1. Fluent Bit collects logs and forwards them over HTTP to the local Grafana Alloy sidecar using the standard Loki output plugin.
  2. Grafana Alloy listens for logs using loki.source.api, automatically exchanges your STACKIT Service Account key for short-lived JWT access tokens, and pushes the logs to your STACKIT Logs instance using loki.write.

Fluent Bit configuration (fluent-bit.conf)

Section titled “Fluent Bit configuration (fluent-bit.conf)”

Configure Fluent Bit to forward logs to Grafana Alloy on port 3100:

fluent-bit.conf
# 1. Configure a Loki output to send log data to the Grafana Alloy sidecar
[OUTPUT]
name loki
match *
# Use "alloy" in Docker Compose, or "localhost" if Alloy runs on the same host or Kubernetes pod
host alloy
# Port where Grafana Alloy loki.source.api is listening
port 3100
# TLS is disabled for local sidecar communication
tls off
# 2. Example: generate dummy logs as input
[INPUT]
name dummy
dummy {"message": "Hello from Fluent Bit to STACKIT Logs"}
tag app.logs

Grafana Alloy configuration (config.alloy)

Section titled “Grafana Alloy configuration (config.alloy)”

Configure Grafana Alloy to receive logs from Fluent Bit and forward them to STACKIT Logs using service account authentication:

config.alloy
// 1. Load the STACKIT Service Account private key file
local.file "stackit_service_account_private_key" {
filename = "/etc/alloy/sa-private-key.pem"
is_secret = true
}
// 2. Listen for logs from Fluent Bit over the Loki push API
loki.source.api "fluentbit" {
http {
listen_address = "0.0.0.0"
listen_port = 3100
}
forward_to = [loki.write.stackit_logs.receiver]
}
// 3. Configure Loki write receiver with OAuth2 for STACKIT Logs
loki.write "stackit_logs" {
endpoint {
// Set here the complete 'Ingest URL' of STACKIT Logs instance
url = "<ingest-url-of-logs-instance>"
oauth2 {
grant_type = "urn:ietf:params:oauth:grant-type:jwt-bearer" // For using Service Account key authentication
audience = "https://accounts.stackit.cloud"
token_url = "https://accounts.stackit.cloud/oauth/v2/token"
client_id = "<sa-credentials-sub-claim>" // The `sub` field from service account key JSON file
iss = "<service-account-id>" // Service account ID or email
client_certificate_key = local.file.stackit_service_account_private_key.content
}
}
}

Docker Compose setup (docker-compose.yaml)

Section titled “Docker Compose setup (docker-compose.yaml)”

Run Fluent Bit and Grafana Alloy together in a shared network using Docker Compose. A similar sidecar architecture can be used in Kubernetes (running Alloy as a sidecar container in the same pod communicating over localhost) or on a Linux server or virtual machine (running both binaries directly as system services communicating over localhost):

docker-compose.yaml
services:
fluent-bit:
image: fluent/fluent-bit:latest
volumes:
- ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro # Fluent Bit configuration file
depends_on:
- alloy
alloy:
image: grafana/alloy:latest
volumes:
- ./config.alloy:/etc/alloy/config.alloy:ro # Grafana Alloy configuration file
- ./sa-private-key.pem:/etc/alloy/sa-private-key.pem:ro # STACKIT Service Account private key file
command:
- run
- /etc/alloy/config.alloy
  1. Save your STACKIT Service Account private key PEM file as sa-private-key.pem.
  2. Create the Grafana Alloy configuration file (config.alloy):
    • Add a local.file block pointing to /etc/alloy/sa-private-key.pem with is_secret = true.
    • Add a loki.source.api block listening on port 3100 that forwards logs to loki.write.stackit_logs.receiver.
    • Add a loki.write block with an endpoint configured with your STACKIT Logs Ingest URL and the oauth2 credentials for your service account.
  3. Create the Fluent Bit configuration file (fluent-bit.conf):
    • Configure your inputs according to your log sources.
    • Add an OUTPUT block with name loki and match *, setting host to alloy (or localhost if running on the same host or Kubernetes pod), port to 3100, and tls to off.
  4. Create the docker-compose.yaml file to run both services together, mounting the configuration files and private key into the respective containers. (Alternatively, configure both services in a Kubernetes pod or as local services on a Linux server or VM.)
  5. Start the containers:
    Terminal window
    docker compose up -d
    Fluent Bit forwards logs to Grafana Alloy, which authenticates with STACKIT IAM and exports them to STACKIT Logs.

Authenticate with an Instance Access Token

Section titled “Authenticate with an Instance Access Token”

You can also authenticate directly using a static access token generated from your STACKIT Logs instance. The bearer_token option in the Fluent Bit loki output plugin attaches this token to the Authorization: Bearer <token> header of every exported HTTP request.

fluent-bit.conf
[OUTPUT]
name loki
host <host-name-from-url-of-data-source-in-logs-instance>
port 443
# The token value can be injected via environment variable as well
bearer_token <access-token-value-of-logs-instance>
tls on
tls.verify on
  1. Create or edit the Fluent Bit configuration file (typically named fluent-bit.conf).
  2. Add an OUTPUT block with name loki to use the Loki output plugin.
  3. In the OUTPUT block, set the host key to the hostname from your STACKIT Logs instance Ingest URL (for example, logs.eu01.stackit.cloud).
  4. Set the port key to 443.
  5. Set the bearer_token key to the access token generated for your STACKIT Logs instance. Ensure the token has write permissions.
  6. Enable TLS by setting tls on and tls.verify on.
  7. Fluent Bit is now configured to send log data directly to your STACKIT Logs instance.