Skip to content

Setup cluster audit logging

Last updated on

Kubernetes audit logging provides a security-relevant chronological record of cluster activities, allowing administrators to trace who did what, when, and where. This visibility is essential for:

  • detecting malicious activity
  • troubleshooting unexpected state changes
  • demonstrating regulatory compliance in a production environment

Audit logs track requests made to the kube-apiserver from the moment they are received until a response is sent. When a user, service account, or internal component interacts with the cluster, the API server evaluates the request against an audit policy, which dictates what data to log (ranging from simple metadata to full request and response bodies) and at what level.

For STACKIT Kubernetes Engine (SKE) clusters, you can configure your cluster to forward these logs to a STACKIT Telemetry Router. Once the logs reach the router, you can configure the appropriate telemetry links and destinations to enable long-term log storage.

Currently, audit logging integration can only be enabled via STACKIT CLI:

  1. Generate and fetch a basic cluster configuration using the STACKIT CLI.

  2. Modify the following part of the configuration to enable the audit logging feature:

    {
    "audit": {
    "enabled": true
    }
    }
  3. Use the following command to create or update a SKE cluster using audit logging:

    Terminal window
    # create new cluster
    stackit ske cluster create -p <PROJECT_ID> <CLUSTER_NAME> --payload @cluster-configuration.json
    # update existing cluster
    stackit ske cluster update -p <PROJECT_ID> <CLUSTER_NAME> --payload @cluster-configuration.json

This activates audit log forwarding to the Telemetry Router. To disable the feature, set the enabled parameter to false or omit the audit section entirely in a subsequent configuration update.

Audit logs for SKE clusters are generated based on the following audit policy:

apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
- "RequestReceived"
rules:
# Don't log creates for calico tier CRD; the healthcheck tries to create it every iteration even though it already exists, which is noisy
- level: None
verbs: ["create"]
resources:
- group: "crd.projectcalico.org"
resources: ["tiers"]
users:
- system:serviceaccount:kube-system:calico-kube-controllers
# Do not log watch operations performed by kube-proxy on endpoints and services
- level: None
users: ["system:kube-proxy"]
verbs: ["watch"]
resources:
- group: "" # core
resources: ["endpoints", "services", "services/status"]
# Do not log get operations performed by the system:nodes group (kubelet) on nodes and their statuses
- level: None
userGroups: ["system:nodes"]
verbs: ["get"]
resources:
- group: "" # core
resources: ["nodes", "nodes/status"]
# Do not log get and update operations performed by controller manager, scheduler, and endpoint-controller on endpoints in kube-system namespace
- level: None
users:
- system:kube-controller-manager
- system:kube-scheduler
- system:serviceaccount:kube-system:endpoint-controller
verbs: ["get", "update"]
namespaces: ["kube-system"]
resources:
- group: "" # core
resources: ["endpoints"]
# Do not log endpointslice updates from endpointslice service accounts because very noisy
- level: None
verbs: ["update"]
resources:
- group: "discovery.k8s.io"
resources: ["endpointslices"]
users:
- system:serviceaccount:kube-system:endpointslicemirroring-controller
- system:serviceaccount:kube-system:endpointslice-controller
# Do not log get operations performed by apiserver on namespaces and their statuses/finalizations
- level: None
users: ["system:apiserver"]
verbs: ["get"]
resources:
- group: "" # core
resources: ["namespaces", "namespaces/status", "namespaces/finalize"]
# Exclude kube-apiserver getting the 'kubernetes' endpoints/endpointslices
- level: None
users: ["system:apiserver"]
verbs: ["get"]
resources:
- group: "" # Core API group
resources: ["endpoints"]
namespaces: ["default"]
resourceNames: ["kubernetes"]
- group: "discovery.k8s.io"
resources: ["endpointslices"]
namespaces: ["default"]
resourceNames: ["kubernetes"]
# Do not log HPA fetching metrics
- level: None
users:
- system:serviceaccount:kube-system:horizontal-pod-autoscaler
verbs: ["get", "list"]
resources:
- group: "metrics.k8s.io"
# Do not log leases and events at all
- level: None
resources:
- group: "" # core
resources: ["events"]
- group: "coordination.k8s.io"
resources: ["leases"]
# Do not log non-security-relevant endpoints and resources
- level: None
nonResourceURLs:
- "/healthz*"
- "/livez*"
- "/readyz*"
- "/swagger*"
- "/version"
- "/"
# Secrets, ConfigMaps, TokenRequest and TokenReviews can contain sensitive & binary data, so only log at the Metadata level
- level: Metadata
resources:
- group: "" # core
resources: ["secrets", "configmaps", "serviceaccounts/token"]
- group: authentication.k8s.io
resources: ["tokenreviews"]
# Node and pod status calls from nodes are high-volume and can be large; do not log responses for expected updates from nodes
- level: Request
users: ["system:serviceaccount:kube-system:node-problem-detector"]
verbs: ["update", "patch"]
resources:
- group: "" # core
resources: ["nodes/status", "pods/status"]
- level: Request
userGroups: ["system:nodes"]
verbs: ["update", "patch"]
resources:
- group: "" # core
resources: ["nodes/status", "pods/status"]
# Deletecollection calls can be large; don't log responses for expected namespace deletions
- level: Request
users: ["system:serviceaccount:kube-system:namespace-controller"]
verbs: ["deletecollection"]
# Get responses can be large; don't log response
- level: Request
verbs: ["get", "list", "watch"]
resources:
- group: ""
- group: "admissionregistration.k8s.io"
- group: "apiextensions.k8s.io"
- group: "apiregistration.k8s.io"
- group: "apps"
- group: "authentication.k8s.io"
- group: "authorization.k8s.io"
- group: "autoscaling"
- group: "batch"
- group: "certificates.k8s.io"
- group: "extensions"
- group: "metrics.k8s.io"
- group: "networking.k8s.io"
- group: "policy"
- group: "rbac.authorization.k8s.io"
- group: "scheduling.k8s.io"
- group: "settings.k8s.io"
- group: "storage.k8s.io"
# Default level for known APIs
- level: RequestResponse
resources:
- group: ""
- group: "admissionregistration.k8s.io"
- group: "apiextensions.k8s.io"
- group: "apiregistration.k8s.io"
- group: "apps"
- group: "authentication.k8s.io"
- group: "authorization.k8s.io"
- group: "autoscaling"
- group: "batch"
- group: "certificates.k8s.io"
- group: "extensions"
- group: "metrics.k8s.io"
- group: "networking.k8s.io"
- group: "policy"
- group: "rbac.authorization.k8s.io"
- group: "scheduling.k8s.io"
- group: "settings.k8s.io"
- group: "storage.k8s.io"
# Catch-all rule but only metadata
- level: Metadata