How to push audit logs from Telemetry Router

Prerequisites

Before you begin routing audit logs to your STACKIT Observability instance, ensure you have the following ready:

• STACKIT user account: Create a user account
• STACKIT customer account: Create a customer account
• STACKIT project: Create a project
• Active Observability instance: You must have a running STACKIT Observability instance.
• Technical user credentials: You need a technical <USERNAME> and <PASSWORD> for your specific instance. If you haven’t generated these yet, follow the instructions to obtain your technical user credentials.
• OTLP Ingest URL: You can copy this directly from the STACKIT Portal. Open your Observability instance, go to the API tab, and locate OTLP ingest under the Logs section.

---

Create your Telemetry Router instance

First, you need to set up the Telemetry Router and generate an access token for it. This token is required to authorize the Telemetry Link.

1. Visit the STACKIT Portal and select Telemetry Router from the sidebar.
2. Click Create Telemetry Router, enter a router name, and click Create.
3. Once the status is Active, click on Access Token in the instance sidebar.
4. Click Create access token.
5. Enter a name, select Unlimited lifetime, and click Create.
6. Copy your generated token and your Telemetry Router Instance ID to a secure location.

For getting started, an unlimited token is convenient. For production environments, we highly recommend defining a limited lifetime for security reasons.

---

Create a Telemetry Link

The Telemetry Link connects your project’s audit events to your router.

1. Visit the STACKIT Portal and select Telemetry Link from the sidebar.
2. Click Create Telemetry Link.
3. Fill in the Instance ID and the Access token you just generated from your Telemetry Router.
4. Click Create.

• Permissions: You must have Telemetry Link Admin permissions in your STACKIT project to perform this action.
• Wait time: After creating the link, wait a few moments until it becomes ready. If you proceed too quickly, you may need to trigger a new audit event manually after completing the destination setup.

---

Point the router destination to Observability

Now, tell your Telemetry Router to send the collected audit data to your Observability instance via the OpenTelemetry (OTLP) protocol using Basic Authentication.

1. Navigate back to Telemetry Router in the sidebar and select your router instance.
2. On the instance sidebar, click on Destinations.
3. Click Create destination and enter a meaningful name.
4. As Type, select OpenTelemetry and click Next.
5. For authentication, select Basic Auth.
6. In the URI field, paste the OTLP ingest URL from your Observability instance.
7. Fill in the Username and Password fields with your Observability technical user credentials.
8. Click Create.

---

View your audit data in Grafana

Creating the destination usually triggers an initial auditable event automatically. You can view your incoming audit trails directly inside Grafana.

If you did not run through this guide in the intended order (for example, if you created the Telemetry Link after the destination), the initial setup event might not have been captured.

To manually trigger a new auditable event, you can perform any simple project-level action in the STACKIT Portal, such as creating new Credentials and immediately deleting them.

Querying Raw Logs via Explore

To run custom filters on your raw log stream, you can query Loki directly.

1. Open your STACKIT Observability instance in the portal and launch the Grafana interface.
2. Navigate to Explore in the left menu.
3. Select your Loki data source from the dropdown.
4. Run a query filtering for the service name (for example, {(service_name = ~".+")}) to see your incoming audit trails.

---

Set up Alerts for Audit Logs

Staring at dashboards is not scalable for operational teams. STACKIT Observability provides a powerful alerting engine to automatically notify you when critical events occur within your project.

The following example demonstrates how to set up an alert that fires whenever an ERROR is logged in your audit trails (e.g., failed authorization attempts, critical system failures, or misconfigured API calls).

Prerequisites for Alerting

To configure alerts, you will use the Observability API. Ensure you have the following ready before executing the commands:

• STACKIT CLI: You need the CLI installed to use the stackit curl wrapper. For instructions on how to authenticate and use it, read Interacting with the Observability API.
• Alerting Concepts: If you are new to the alerting engine, we recommend reading the Alerting Overview to understand the relationship between Rules, Receivers, and Routes.
• Environment Variables: Export your identifiers in your terminal to easily copy-paste the configuration commands below:

export PROJECT_ID="<your-project-id>"
export INSTANCE_ID="<your-instance-id>"

---

Step 1: Create the Loki Alert Group and Rule

First, create an Alert Group containing the evaluation logic. Because Audit Logs are stored and indexed by Loki, we send this request specifically to the /logs-alertgroups endpoint so the Loki ruler can native-parse the LogQL expression.

Execute the following command to deploy the Alert Group:

stackit curl -X POST "https://argus.api.eu01.stackit.cloud/v1/projects/$PROJECT_ID/instances/$INSTANCE_ID/logs-alertgroups" \
  -H "accept: application/json" \
  -H "Content-Type: application/json" \
  --data '{
    "name": "AuditLogAlerts",
    "interval": "5m",
    "rules": [
      {
        "alert": "AuditLogErrorDetected",
        "expr": "sum(count_over_time({service_name=~\".+\"} | stackit_log_type=\"AUDIT\" | detected_level=\"ERROR\" [5m])) > 0",
        "for": "1m",
        "labels": {
          "severity": "critical"
        },
        "annotations": {
          "summary": "Critical Audit Log Error Detected",
          "description": "An action resulting in an ERROR severity was logged in the audit trail within the last 5 minutes. Please investigate the raw logs in Grafana immediately."
        }
      }
    ]
  }'

• interval: The system evaluates your logs every 5 minutes.
• expr: The LogQL metric expression checking for any audit entry where detected_level="ERROR". If the count is greater than zero (> 0), the alert condition is met.
• for: The condition must persist for 1 minute before transitioning from pending to firing, avoiding alerts on short transient blips.

---

Step 2: Create the Alert Receiver

Next, define where the notification should be dispatched once the rule triggers. We will establish an email notification receiver for your operations or security team.

stackit curl -X POST "https://argus.api.eu01.stackit.cloud/v1/projects/$PROJECT_ID/instances/$INSTANCE_ID/alertconfigs/receivers" \
  -H "accept: application/json" \
  -H "Content-Type: application/json" \
  --data '{
    "name": "MyEmailReceiver",
    "emailConfigs": [
      {
        "to": "your-email@example.com"
      }
    ]
  }'

Be sure to replace your-email@example.com with the actual destination email address of your monitoring personnel.

---

Step 3: Create the Alert Route

The final step is to connect the created alert to your MyEmailReceiver. The route acts as a switchboard routing traffic based on the labels attached to the alert rule.

If your instance already has an existing routing tree configured, ensure you append or nest this new path correctly instead of wiping out pre-existing roots. The command below configures a clean routing topology.

stackit curl -X POST "https://argus.api.eu01.stackit.cloud/v1/projects/$PROJECT_ID/instances/$INSTANCE_ID/alertconfigs/routes" \
  -H "accept: application/json" \
  -H "Content-Type: application/json" \
  --data '{
    "receiver": "MyEmailReceiver",
    "groupBy": ["job"],
    "groupWait": "30s",
    "groupInterval": "5m",
    "repeatInterval": "4h",
    "routes": [
      {
        "receiver": "MyEmailReceiver",
        "match": {
          "severity": "critical"
        },
        "groupBy": ["alertname"],
        "groupWait": "30s",
        "groupInterval": "5m",
        "repeatInterval": "4h"
      }
    ]
  }'

---

Test the configured alert

To ensure that your automated alerting configuration works flawlessly end-to-end, you need to trigger an alert and verify the notification.

Intentionally causing a critical ERROR audit trail can be difficult. For an easier test, temporarily modify your LogQL expression in Step 1: change detected_level="ERROR" to detected_level="INFO".

This allows any standard action (like creating or deleting a token) to trigger the alert. Remember to change it back to ERROR after your test is complete!

1. Trigger an action within your project that produces an ERROR audit trail (or an INFO trail if you are using the workaround).
2. Wait 3 to 5 minutes for the infrastructure to ingest the log, cross-evaluate the threshold, and group the notification.
3. Check your specified email inbox to verify the alert was received.