How to set up quota consumption alerts

To ensure continuous data ingestion without unexpected drops, it is highly recommended to set up early warning alerts. By configuring alerts to trigger when your resource consumption reaches 90% of your allocated limits, you have enough time to react – either by optimizing your data ingestion or upgrading your plan.

This guide provides a ready-to-use configuration to set up email alerts for the following critical quotas:

• Logs: Storage bucket size
• Traces: Storage bucket size
• Metrics: Write samples per minute

Prerequisites

• Active Observability instance: You have a running STACKIT Observability instance.

To execute the following commands, you need to set your PROJECT_ID and INSTANCE_ID as environment variables (or replace them directly in the commands).

If you need help locating these identifiers or want to learn how the stackit curl wrapper works, refer to our central guide: Interacting with the Observability API.

Step-by-step setup

Follow these steps to configure your Alertmanager receiver, routing, and Prometheus alert rules.

1. Export your variables

   To execute the commands in this guide smoothly, first export the variables from the code box in your terminal. Replace the values with your actual project ID, instance ID, and the email address that should receive the alerts:

   export PROJECT_ID="your-project-id"
   export INSTANCE_ID="your-instance-id"
   export EMAIL_ADDRESS="your-email@company.com"

2. Create the Alertmanager receiver

   First, configure an Alertmanager receiver to define where the alert should be sent. We will create an email receiver using your exported email address.

   Ensure you provide a real and valid email address. Using fake or unreachable addresses can cause delivery issues and may result in your alerts being flagged as spam or blacklisted over time.

   This guide uses an email receiver as a quick example. If you want to route your alerts to a webhook (like MS Teams or Google Chat) or Opsgenie instead, check out our guide on How to manage Alert Receivers.

   STACKIT API (CLI)

   stackit curl -X POST "https://argus.api.eu01.stackit.cloud/v1/projects/$PROJECT_ID/instances/$INSTANCE_ID/alertconfigs/receivers" \
     -H "accept: application/json" \
     -H "Content-Type: application/json" \
     --data '{
       "name":"QuotaAlertsReceiver",
       "emailConfigs":[{"to":"'$EMAIL_ADDRESS'"}]
     }'

3. Create the Alertmanager route

   Next, create an alert config route. This tells the Alertmanager to route alerts to the receiver you just created based on specific grouping and timing settings. We also apply a matcher to ensure only alerts with a warning severity are routed here.

   STACKIT API (CLI)

   stackit curl -X POST "https://argus.api.eu01.stackit.cloud/v1/projects/$PROJECT_ID/instances/$INSTANCE_ID/alertconfigs/routes" \
     -H "accept: application/json" \
     -H "Content-Type: application/json" \
     --data '{
       "receiver":"QuotaAlertsReceiver",
       "groupBy":["alertname"],
       "groupWait":"30s",
       "groupInterval":"5m",
       "repeatInterval":"4h",
       "matchers":["severity=warning"]
     }'

4. Create the Prometheus Alert Group and Rules

   Finally, define the alert conditions based on the Prometheus Query Language (expr). The following command creates an Alert Group containing three rules that trigger when your Logs, Traces, or Metrics usage hits 90%.

   The payload below includes rules for all three quotas. If you only use specific features (e.g., only Metrics), simply delete the unneeded objects from the "rules": [...] array before executing the command.

   STACKIT API (CLI)

   stackit curl -X POST "https://argus.api.eu01.stackit.cloud/v1/projects/$PROJECT_ID/instances/$INSTANCE_ID/alertgroups" \
     -H "accept: application/json" \
     -H "Content-Type: application/json" \
     --data '{
       "name": "QuotaConsumptionAlerts",
       "interval": "60s",
       "rules": [
           {
               "alert": "LogQuotaApproaching",
               "expr": "max(instance_logs_storage_size{} * 100 / instance_logs_plan_size{}) >= 90",
               "for": "5m",
               "labels": { "severity": "warning" }
           },
           {
               "alert": "TraceQuotaApproaching",
               "expr": "max(instance_traces_storage_size{} * 100 / instance_trace_plan_size{}) >= 90",
               "for": "5m",
               "labels": { "severity": "warning" }
           },
           {
               "alert": "MetricsQuotaApproaching",
               "expr": "avg by(namespace) (increase(instance_remote_write_samples_received_total[2m])) / on() group_right min(instance_remote_write_samples_max_1m) >= 0.9",
               "for": "5m",
               "labels": { "severity": "warning" }
           }
       ]
   }'