Fluent Bit: Write log data with Loki API
Zuletzt aktualisiert am
Fluent Bit is a popular agent for collecting logs and shipping them to multiple destinations. It has a built-in plugin available for Loki as an output which can be configured to target a STACKIT Logs instance using the Loki API.
To authenticate requests, STACKIT Logs supports two authentication methods:
- STACKIT Service Account (recommended): Because Fluent Bit does not natively support OAuth2 JWT bearer token authentication, you can pair Fluent Bit with Grafana Alloy running as a sidecar proxy. Fluent Bit sends logs to Grafana Alloy, which exchanges service account credentials for short-lived JWT access tokens and exports the logs to STACKIT Logs.
- Instance Access Token: Uses the
bearer_tokenoption directly in Fluent Bit’slokioutput plugin with a static bearer token created in your Logs instance.
Prerequisites
Section titled “Prerequisites”- STACKIT Logs instance: Create your first STACKIT Logs instance
- Data source URL: Retrieve API URLs for your STACKIT Logs instance
- Credentials:
- For Service Account authentication (recommended):
- A STACKIT Service Account: Create a service account
- A Service Account Key pair: Create a service account key
- An Access Policy configured on your STACKIT Logs instance granting the Service Account write permissions.
- For Access Token authentication:
- An access token with write permissions: Retrieve credentials for STACKIT Logs instance
- For Service Account authentication (recommended):
Authenticate with a STACKIT Service Account (Recommended)
Section titled “Authenticate with a STACKIT Service Account (Recommended)”Using a STACKIT Service Account is the recommended approach for automated and production workloads. Because Fluent Bit does not support the OAuth2 JWT bearer profile (urn:ietf:params:oauth:grant-type:jwt-bearer), it cannot authenticate directly with STACKIT IAM using service account keys.
To use service account authentication with Fluent Bit, deploy Grafana Alloy alongside Fluent Bit as a sidecar proxy:
- Fluent Bit collects logs and forwards them over HTTP to the local Grafana Alloy sidecar using the standard Loki output plugin.
- Grafana Alloy listens for logs using
loki.source.api, automatically exchanges your STACKIT Service Account key for short-lived JWT access tokens, and pushes the logs to your STACKIT Logs instance usingloki.write.
Configuration example
Section titled “Configuration example”Fluent Bit configuration (fluent-bit.conf)
Section titled “Fluent Bit configuration (fluent-bit.conf)”Configure Fluent Bit to forward logs to Grafana Alloy on port 3100:
# 1. Configure a Loki output to send log data to the Grafana Alloy sidecar[OUTPUT] name loki match * # Use "alloy" in Docker Compose, or "localhost" if Alloy runs on the same host or Kubernetes pod host alloy # Port where Grafana Alloy loki.source.api is listening port 3100 # TLS is disabled for local sidecar communication tls off
# 2. Example: generate dummy logs as input[INPUT] name dummy dummy {"message": "Hello from Fluent Bit to STACKIT Logs"} tag app.logsGrafana Alloy configuration (config.alloy)
Section titled “Grafana Alloy configuration (config.alloy)”Configure Grafana Alloy to receive logs from Fluent Bit and forward them to STACKIT Logs using service account authentication:
// 1. Load the STACKIT Service Account private key filelocal.file "stackit_service_account_private_key" { filename = "/etc/alloy/sa-private-key.pem" is_secret = true}
// 2. Listen for logs from Fluent Bit over the Loki push APIloki.source.api "fluentbit" { http { listen_address = "0.0.0.0" listen_port = 3100 } forward_to = [loki.write.stackit_logs.receiver]}
// 3. Configure Loki write receiver with OAuth2 for STACKIT Logsloki.write "stackit_logs" { endpoint { // Set here the complete 'Ingest URL' of STACKIT Logs instance url = "<ingest-url-of-logs-instance>" oauth2 { grant_type = "urn:ietf:params:oauth:grant-type:jwt-bearer" // For using Service Account key authentication audience = "https://accounts.stackit.cloud" token_url = "https://accounts.stackit.cloud/oauth/v2/token" client_id = "<sa-credentials-sub-claim>" // The `sub` field from service account key JSON file iss = "<service-account-id>" // Service account ID or email client_certificate_key = local.file.stackit_service_account_private_key.content } }}Docker Compose setup (docker-compose.yaml)
Section titled “Docker Compose setup (docker-compose.yaml)”Run Fluent Bit and Grafana Alloy together in a shared network using Docker Compose. A similar sidecar architecture can be used in Kubernetes (running Alloy as a sidecar container in the same pod communicating over localhost) or on a Linux server or virtual machine (running both binaries directly as system services communicating over localhost):
services: fluent-bit: image: fluent/fluent-bit:latest volumes: - ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro # Fluent Bit configuration file depends_on: - alloy
alloy: image: grafana/alloy:latest volumes: - ./config.alloy:/etc/alloy/config.alloy:ro # Grafana Alloy configuration file - ./sa-private-key.pem:/etc/alloy/sa-private-key.pem:ro # STACKIT Service Account private key file command: - run - /etc/alloy/config.alloySetup steps
Section titled “Setup steps”- Save your STACKIT Service Account private key PEM file as
sa-private-key.pem. - Create the Grafana Alloy configuration file (
config.alloy):- Add a
local.fileblock pointing to/etc/alloy/sa-private-key.pemwithis_secret = true. - Add a
loki.source.apiblock listening on port3100that forwards logs toloki.write.stackit_logs.receiver. - Add a
loki.writeblock with anendpointconfigured with your STACKIT Logs Ingest URL and theoauth2credentials for your service account.
- Add a
- Create the Fluent Bit configuration file (
fluent-bit.conf):- Configure your inputs according to your log sources.
- Add an
OUTPUTblock withname lokiandmatch *, settinghosttoalloy(orlocalhostif running on the same host or Kubernetes pod),portto3100, andtlstooff.
- Create the
docker-compose.yamlfile to run both services together, mounting the configuration files and private key into the respective containers. (Alternatively, configure both services in a Kubernetes pod or as local services on a Linux server or VM.) - Start the containers:
Fluent Bit forwards logs to Grafana Alloy, which authenticates with STACKIT IAM and exports them to STACKIT Logs.
Terminal window docker compose up -d
Authenticate with an Instance Access Token
Section titled “Authenticate with an Instance Access Token”You can also authenticate directly using a static access token generated from your STACKIT Logs instance. The bearer_token option in the Fluent Bit loki output plugin attaches this token to the Authorization: Bearer <token> header of every exported HTTP request.
Configuration example
Section titled “Configuration example”[OUTPUT] name loki host <host-name-from-url-of-data-source-in-logs-instance> port 443 # The token value can be injected via environment variable as well bearer_token <access-token-value-of-logs-instance> tls on tls.verify onSetup steps
Section titled “Setup steps”- Create or edit the Fluent Bit configuration file (typically named
fluent-bit.conf). - Add an
OUTPUTblock withname lokito use the Loki output plugin. - In the
OUTPUTblock, set thehostkey to the hostname from your STACKIT Logs instance Ingest URL (for example,logs.eu01.stackit.cloud). - Set the
portkey to443. - Set the
bearer_tokenkey to the access token generated for your STACKIT Logs instance. Ensure the token has write permissions. - Enable TLS by setting
tls onandtls.verify on. - Fluent Bit is now configured to send log data directly to your STACKIT Logs instance.