How to use recording rules in Grafana

Recording rules allow you to precompute frequently needed or computationally expensive expressions and save their result as a new set of time series.

Querying the precomputed result will then often be much faster than running the original expression every time it is needed. This is especially useful for dashboards, which need to query the same expression repeatedly every time they refresh.

Recording rules and Alerting rules are syntactically the same. Therefore, both types can be written to a configuration file. Recording and alerting rules exist in a rule group. Rules within a group are run sequentially at a regular interval, with the same evaluation time.

To execute the API commands below, you need your STACKIT identifiers ($PROJECT_ID, $INSTANCE_ID) and custom names you define yourself for your rules ($GROUP_NAME, $RECORD_NAME).

For instructions on finding your STACKIT IDs and using the stackit curl wrapper for authentication, see Interacting with the Observability API.

Common Parameters

Common parameters for recording rules

• record (string): The name of the time series to output to. Must be a valid metric name.
• expr (string): The PromQL expression to evaluate. Every evaluation cycle this is evaluated at the current time, and the result is recorded as a new set of time series with the metric name given by record.
• labels ([ labelname: label value ]): Labels to add or overwrite before storing the result.

Common parameters for recording groups

• name (string): The name of the group. Must be unique within a file.
• interval (duration | default = global.evaluation_interval): How often rules in the group are evaluated.
• limit (int | default = 0): Limit the number of alerts an alerting rule and series a recording rule can produce. 0 means no limit.
• rules: A list of the specific rules.

Metric Names and Labels

Every time series is uniquely identified by its metric name and optional key-value pairs called labels.

Metric names

• Specify the general feature of a system that is measured (e.g., http_requests_total - the total number of HTTP requests received).
• Metric names may contain ASCII letters, digits, underscores, and colons. They must match the regex [a-zA-Z_:][a-zA-Z0-9_:]*.
• Note: The colons are reserved for user-defined recording rules. They should not be used by exporters or direct instrumentation.

Metric labels

• Enable Prometheus’s dimensional data model to identify any given combination of labels for the same metric name.
• They identify a particular dimensional instantiation of that metric (for example: all HTTP requests that used the method POST to the /api/tracks handler).
• The change of any label’s value, including adding or removing labels, will create a new time series.
• Labels may contain ASCII letters, numbers, as well as underscores. They must match the regex [a-zA-Z_][a-zA-Z0-9_]*.
• Label names beginning with __ (two underscores) are reserved for internal use.
• Label values may contain any Unicode characters. Labels with an empty label value are considered equivalent to labels that do not exist.

For more details, see the Prometheus best practices for naming metrics.

---

Examples

Example rules file

A typical YAML definition for a recording rule requires the following structure:

groups:
  - name: <The group name of recording rule the>
    rules:
      - record: <The name of series the time>
        expr: <The PromQL evaluate expression to>

A simple example rules file (rules.yaml) would be:

groups:
  - name: http_resources
    interval: 5m
    rules:
      - record: code:prometheus_http_requests_total:sum
        expr: sum by (code) (prometheus_http_requests_total)
  - name: node_resources
    rules:
      - record: node_memory_MemFree_percent
        expr: 100 - (100 * node_memory_MemFree_bytes / node_memory_MemTotal_bytes)
      - record: node_filesystem_free_percent
        expr: 100 * node_filesystem_free_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}
  - name: group_name
    rules:
      - record: job:prometheus_http_requests_total:rate5m
        expr: sum without(instance, method, controller, status_code)(rate(prometheus_http_requests_total[5m]))
        labels:
          test: test

---

Managing Recording Rules via API

You can manage your recording rules directly via the STACKIT Observability API using the stackit curl wrapper. The variables $GROUP_NAME and $RECORD_NAME in the commands below are custom names you define for your resources.

Create a recording group

A recording group contains one or more recording rules and is used to combine rules with related metrics.

stackit curl -X POST "https://argus.api.eu01.stackit.cloud/v1/projects/$PROJECT_ID/instances/$INSTANCE_ID/alertgroups/$GROUP_NAME/records" \
  -H "Content-Type: application/json" \
  --data '{
  "record": "string",
  "expr": "string",
  "labels": {}
}'

Create a recording rule

To define recording rules independently of the group or add rules to an existing group, use the following PUT call.

stackit curl -X PUT "https://argus.api.eu01.stackit.cloud/v1/projects/$PROJECT_ID/instances/$INSTANCE_ID/alertgroups/$GROUP_NAME/records/$RECORD_NAME" \
  -H "Content-Type: application/json" \
  --data '{
    "expr": "freezer_room_temperature{freezer='\''lidl_hn_neckarsulmerstr_f121'\''} - freezer_temperature{freezer='\''lidl_hn_neckarsulmerstr_f121'\''}",
    "labels": {}
}'

Update a recording rule

One or more recording rules within a group can be updated with a PATCH request:

stackit curl -X PATCH "https://argus.api.eu01.stackit.cloud/v1/projects/$PROJECT_ID/instances/$INSTANCE_ID/alertgroups/$GROUP_NAME/records" \
  -H "Content-Type: application/json" \
  --data '[
  {
    "record": "string",
    "expr": "string",
    "labels": {}
  },
  {
    "record": "freezerdifftemp",
    "expr": "freezer_room_temperature{freezer='\''lidl_hn_neckarsulmerstr_f121'\''} - freezer_temperature{freezer='\''lidl_hn_neckarsulmerstr_f121'\''}",
    "labels": {
      "device": "freezer"
    }
  }
]'

Read a recording group

To verify your configuration, you can retrieve all rules within a specific recording group:

stackit curl -X GET "https://argus.api.eu01.stackit.cloud/v1/projects/$PROJECT_ID/instances/$INSTANCE_ID/alertgroups/$GROUP_NAME/records"

Read a specific recording rule

To display the details of a single, specific recording rule:

stackit curl -X GET "https://argus.api.eu01.stackit.cloud/v1/projects/$PROJECT_ID/instances/$INSTANCE_ID/alertgroups/$GROUP_NAME/records/$RECORD_NAME"

Delete a recording group

To delete an entire recording group and all its associated rules:

stackit curl -X DELETE "https://argus.api.eu01.stackit.cloud/v1/projects/$PROJECT_ID/instances/$INSTANCE_ID/alertgroups/$GROUP_NAME/records"

Delete a single recording rule

To delete only a specific recording rule from a group:

stackit curl -X DELETE "https://argus.api.eu01.stackit.cloud/v1/projects/$PROJECT_ID/instances/$INSTANCE_ID/alertgroups/$GROUP_NAME/records/$RECORD_NAME"

---

Syntax-checking recording rules

To quickly check whether a rule file is syntactically correct without starting a Prometheus server, you can use Prometheus’s promtool command-line utility:

promtool check rules /path/to/example.rules.yml

How to use recording rules within Grafana

In Grafana, you can see additional metrics in the Metrics Browser that were formed from your recording rules. The name of the new metric corresponds directly to the value you set in the record field.

Reference

• STACKIT Observability Alert Records API Specification
• Prometheus Documentation: Recording Rules
• Prometheus Documentation: Metric Names and Labels
• Prometheus Documentation: Best Practices for Naming
• How to optimize PromQL and MetricsQL queries