Skip to content

How to use recording rules in Grafana

Last updated on

Recording rules allow you to precompute frequently needed or computationally expensive expressions and save their result as a new set of time series.

Querying the precomputed result will then often be much faster than running the original expression every time it is needed. This is especially useful for dashboards, which need to query the same expression repeatedly every time they refresh.

Recording rules and Alerting rules are syntactically the same. Therefore, both types can be written to a configuration file. Recording and alerting rules exist in a rule group. Rules within a group are run sequentially at a regular interval, with the same evaluation time.

  • record (string): The name of the time series to output to. Must be a valid metric name.
  • expr (string): The PromQL expression to evaluate. Every evaluation cycle this is evaluated at the current time, and the result is recorded as a new set of time series with the metric name given by record.
  • labels ([ labelname: label value ]): Labels to add or overwrite before storing the result.
  • name (string): The name of the group. Must be unique within a file.
  • interval (duration | default = global.evaluation_interval): How often rules in the group are evaluated.
  • limit (int | default = 0): Limit the number of alerts an alerting rule and series a recording rule can produce. 0 means no limit.
  • rules: A list of the specific rules.

Every time series is uniquely identified by its metric name and optional key-value pairs called labels.

  • Specify the general feature of a system that is measured (e.g., http_requests_total - the total number of HTTP requests received).
  • Metric names may contain ASCII letters, digits, underscores, and colons. They must match the regex [a-zA-Z_:][a-zA-Z0-9_:]*.
  • Note: The colons are reserved for user-defined recording rules. They should not be used by exporters or direct instrumentation.
  • Enable Prometheus’s dimensional data model to identify any given combination of labels for the same metric name.
  • They identify a particular dimensional instantiation of that metric (for example: all HTTP requests that used the method POST to the /api/tracks handler).
  • The change of any label’s value, including adding or removing labels, will create a new time series.
  • Labels may contain ASCII letters, numbers, as well as underscores. They must match the regex [a-zA-Z_][a-zA-Z0-9_]*.
  • Label names beginning with __ (two underscores) are reserved for internal use.
  • Label values may contain any Unicode characters. Labels with an empty label value are considered equivalent to labels that do not exist.

For more details, see the Prometheus best practices for naming metrics.


A typical YAML definition for a recording rule requires the following structure:

groups:
- name: <The group name of recording rule the>
rules:
- record: <The name of series the time>
expr: <The PromQL evaluate expression to>

A simple example rules file (rules.yaml) would be:

groups:
- name: http_resources
interval: 5m
rules:
- record: code:prometheus_http_requests_total:sum
expr: sum by (code) (prometheus_http_requests_total)
- name: node_resources
rules:
- record: node_memory_MemFree_percent
expr: 100 - (100 * node_memory_MemFree_bytes / node_memory_MemTotal_bytes)
- record: node_filesystem_free_percent
expr: 100 * node_filesystem_free_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}
- name: group_name
rules:
- record: job:prometheus_http_requests_total:rate5m
expr: sum without(instance, method, controller, status_code)(rate(prometheus_http_requests_total[5m]))
labels:
test: test

You can manage your recording rules directly via the STACKIT Observability API using the stackit curl wrapper. The variables $GROUP_NAME and $RECORD_NAME in the commands below are custom names you define for your resources.

A recording group contains one or more recording rules and is used to combine rules with related metrics.

Terminal window
stackit curl -X POST "https://argus.api.eu01.stackit.cloud/v1/projects/$PROJECT_ID/instances/$INSTANCE_ID/alertgroups/$GROUP_NAME/records" \
-H "Content-Type: application/json" \
--data '{
"record": "string",
"expr": "string",
"labels": {}
}'

To define recording rules independently of the group or add rules to an existing group, use the following PUT call.

Terminal window
stackit curl -X PUT "https://argus.api.eu01.stackit.cloud/v1/projects/$PROJECT_ID/instances/$INSTANCE_ID/alertgroups/$GROUP_NAME/records/$RECORD_NAME" \
-H "Content-Type: application/json" \
--data '{
"expr": "freezer_room_temperature{freezer='\''lidl_hn_neckarsulmerstr_f121'\''} - freezer_temperature{freezer='\''lidl_hn_neckarsulmerstr_f121'\''}",
"labels": {}
}'

One or more recording rules within a group can be updated with a PATCH request:

Terminal window
stackit curl -X PATCH "https://argus.api.eu01.stackit.cloud/v1/projects/$PROJECT_ID/instances/$INSTANCE_ID/alertgroups/$GROUP_NAME/records" \
-H "Content-Type: application/json" \
--data '[
{
"record": "string",
"expr": "string",
"labels": {}
},
{
"record": "freezerdifftemp",
"expr": "freezer_room_temperature{freezer='\''lidl_hn_neckarsulmerstr_f121'\''} - freezer_temperature{freezer='\''lidl_hn_neckarsulmerstr_f121'\''}",
"labels": {
"device": "freezer"
}
}
]'

To verify your configuration, you can retrieve all rules within a specific recording group:

Terminal window
stackit curl -X GET "https://argus.api.eu01.stackit.cloud/v1/projects/$PROJECT_ID/instances/$INSTANCE_ID/alertgroups/$GROUP_NAME/records"

To display the details of a single, specific recording rule:

Terminal window
stackit curl -X GET "https://argus.api.eu01.stackit.cloud/v1/projects/$PROJECT_ID/instances/$INSTANCE_ID/alertgroups/$GROUP_NAME/records/$RECORD_NAME"

To delete an entire recording group and all its associated rules:

Terminal window
stackit curl -X DELETE "https://argus.api.eu01.stackit.cloud/v1/projects/$PROJECT_ID/instances/$INSTANCE_ID/alertgroups/$GROUP_NAME/records"

To delete only a specific recording rule from a group:

Terminal window
stackit curl -X DELETE "https://argus.api.eu01.stackit.cloud/v1/projects/$PROJECT_ID/instances/$INSTANCE_ID/alertgroups/$GROUP_NAME/records/$RECORD_NAME"

To quickly check whether a rule file is syntactically correct without starting a Prometheus server, you can use Prometheus’s promtool command-line utility:

Terminal window
promtool check rules /path/to/example.rules.yml

In Grafana, you can see additional metrics in the Metrics Browser that were formed from your recording rules. The name of the new metric corresponds directly to the value you set in the record field.