Using recording rules in Grafana

Introduction

Recording rules allow you to precompute frequently needed or computationally expensive expressions and save their result as a new set of time series.
Querying the precomputed result will then often be much faster than executing the original expression every time it is needed.
This is especially useful for dashboards, which need to query the same expression repeatedly every time they refresh.

Recording rules and Alerting rules are syntactically the same. Therefore both types can be written to a configuration file.
The API endpoint for this is:
https://argus.api.eu01.stackit.cloud/v1/projects/[project_id]/instances/[instance_id]/alertgroups/[alert_group]/records

Recording and alerting rules exist in a rule group. Rules within a group are run sequentially at a regular interval, with the same evaluation time.
The names of recording rules must be valid metric names.

Common parameters

Common parameters for recording rules

record: string The name of the time series to output to. Must be a valid metric name.

expr: string The PromQL expression to evaluate. Every evaluation cycle this is evaluated at the current time, and the result recorded as a new set of time series with the metric name as given by ‘record’.

labels: [ labelname: labelvalue ] Labels to add or overwrite before storing the result.

Common parameters for recording groups

name: string The name of the group. Must be unique within a file.

[ interval: duration | default = global.evaluation_interval ] How often rules in the group are evaluated.

[ limit: int | default = 0 ] Limit the number of alerts an alerting rule and series a recording
rule can produce. 0 is no limit.

rules:
[ - rule… ]

Metric names and labels

Every time series is uniquely identified by its metric name and optional key-value pairs called labels.

Metric names

• Specify the general feature of a system that is measured (e.g. http_requests_total - the total number of HTTP requests received).
  Metric names may contain ASCII letters, digits, underscores, and colons. It must match the regex [ a-zA-Z_: ][a-zA-Z0-9_:]*.
• Note: The colons are reserved for user defined recording rules. They should not be used by exporters or direct instrumentation.

Metric labels

• Enable Prometheus’s dimensional data model to identify any given combination of labels for the same metric name.
  It identifies a particular dimensional instantiation of that metric (for example: all HTTP requests that used the method POST to the /api/tracks handler).
  The query language allows filtering and aggregation based on these dimensions.
• The change of any label’s value, including adding or removing labels, will create a new time series.
• Labels may contain ASCII letters, numbers, as well as underscores. They must match the regex [ a-zA-Z_ ][a-zA-Z0-9_]*.
• Label names beginning with __ (two “_”) are reserved for internal use.
• Label values may contain any Unicode characters.
• Labels with an empty label value are considered equivalent to labels that do not exist.

See best practices for naming metrics: https://prometheus.io/docs/practices/naming/

Examples

Example rules file

Following is required:

groups:
 - name: <The name of the recording rule>
 rules:
 - record: <The name of the time series>
 expr: <The PromQL expression to evaluate>

A simple example rules file would be:
filename: rules.yaml

groups:
- name: http_ressources
 interval: 5m
 rules:
 - record: code:prometheus_http_requests_total:sum
 expr: sum by (code) (prometheus_http_requests_total)
- name: node_ressources
 rules:
 - record: node_memory_MemFree_percent
 expr: 100 - (100 * node_memory_MemFree_bytes / node_memory_MemTotal_bytes)
 - record: node_filesystem_free_percent
 expr: 100 * node_filesystem_free_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}
- name: group_name
 rules:
 - record: job:prometheus_http_requests_total:rate5m
    expr: sum without(instance, method, controller, status_code)(rate(prometheus_http_requests_total[5m]))
 labels:
 test: test

Create recording group

A recording group contains one or more recording rules and is used to combine rules with related metrics.
Recording rules can also be defined within the alert group.

curl -X POST --location 'https://argus.api.eu01.stackit.cloud/v1/projects/{projectId}/instances/{instanceId}/alertgroups/{groupName}/records' \
--header 'Authorization: Bearer <bearer-token_from_stackit-portal_access-token>' \
--header 'Content-Type: text/plain' \
--data '{
 "record": "string",
  "expr": "string",
 "labels": {}
}'

Create recording rule

To define recording rules independently of the group or add rules to an existing group, this works with the following call.

curl -X PUT --location 'https://argus.api.eu01.stackit.cloud/v1/projects/{projectId}/instances/{instanceId}/alertgroups/{groupName}/records/{alertRecord}' \
--header 'Authorization: Bearer <bearer-token_from_stackit-portal_access-token>' \
--header 'Content-Type: text/plain' \
--data '{
    "expr": "freezer_room_temperature{freezer='\''lidl_hn_neckarsulmerstr_f121'\''} - freezer_temperature{freezer='\''lidl_hn_neckarsulmerstr_f121'\''}",
    "labels": {}
}'

Update recording rule

One or more recording rules can be updated with the following API call.

curl -X PATCH --location 'https://argus.api.eu01.stackit.cloud/v1/projects/{projectId}/instances/{instanceId}/alertgroups/{groupName}/records' \
--header 'Authorization: Bearer <bearer-token_from_stackit-portal_access-token>' \
--header 'Content-Type: text/plain' \
--data '[
 {
 "record": "string",
 "expr": "string",
 "labels": {}
 },
 {
    "record": "freezerdifftemp",
    "expr": "freezer_room_temperature{freezer='\''lidl_hn_neckarsulmerstr_f121'\''} - freezer_temperature{freezer='\''lidl_hn_neckarsulmerstr_f121'\''}",
   "labels": {"device": "freezer"}
 }
]'

Read a recording group

To check, you can read out a recording group:

curl -X GET --location 'https://argus.api.eu01.stackit.cloud/v1/projects/{projectId}/instances/{instanceId}/alertgroups/{groupName}/records' \
--header 'Authorization: Bearer <bearer-token_from_stackit-portal_access-token>'

Read a recording rule

Display a specific recording rule:

curl -X GET --location 'https://argus.api.eu01.stackit.cloud/v1/projects/{projectId}/instances/{instanceId}/alertgroups/{groupName}/records/{alertRecord}' \
--header 'Authorization: Bearer <bearer-token_from_stackit-portal_access-token>'

Delete a group of recording rules

Deleting a recording group is also possible. The following call shows how:

curl -X DELETE --location --request DELETE 'https://argus.api.eu01.stackit.cloud/v1/projects/{projectId}/instances/{instanceId}/alertgroups/{groupName}/records' \
--header 'Authorization: Bearer <bearer-token_from_stackit-portal_access-token>'

Delete a single recording rule

Deleting a single recording rule looks like this:

curl -X DELETE --location --request DELETE 'https://argus.api.eu01.stackit.cloud/v1/projects/{projectId}/instances/{instanceId}/alertgroups/{groupName}/records/{alertRecord}' \
--header 'Authorization: Bearer <bearer-token_from_stackit-portal_access-token>'

Syntax-checking recording rules

To quickly check whether a rule file is syntactically correct without starting a Prometheus server, you can use Prometheus’s promtool command-line utility tool:

promtool check rules /path/to/example.rules.yml

How to use recording rules within Grafana

In Grafana you can see additional metrics in the Metrics Browser that were formed from the recording rules. The name of the metric corresponds to the field record.

Further Information

• https://docs.api.eu01.stackit.cloud/documentation/argus/version/v1#tag/alert-records
• https://prometheus.io/docs/prometheus/latest/configuration/recording%5Frules/
• https://prometheus.io/docs/concepts/data%5Fmodel/#metric-names-and-labels
• https://prometheus.io/docs/practices/naming/
• https://valyala.medium.com/how-to-optimize-promql-and-metricsql-queries-85a1b75bf986