How to monitor applications via STACKIT Observability

This section shows the process, how customer workloads on SKE can be monitored using the STACKIT Observability service.

Overview

Here is a high-level architecture diagram of the monitoring setup:

Prerequisites

• You have ordered the following services:
  • STACKIT Kubernetes Engine
  • STACKIT Observability
• You have basic knowledge of Prometheus, Grafana and Kubernetes.
• You have verified your Kubernetes cluster using the kubectl command. Kubectl is a command line tool which allows you to run commands against Kubernetes clusters:

  kubectl get nodes
  
  NAME STATUS ROLES AGE VERSION
  shoot--9kjnrurmk8--pawanpg-worker-jpqie-z1-767df-dsg99 Ready <none> 19d v1.21.10
  shoot--9kjnrurmk8--pawanpg-worker-jpqie-z1-767df-g2q42 Ready <none> 19d v1.21.10

Steps

1. Deploy the Prometheus operator on your SKE cluster

curl -sL https://github.com/prometheus-operator/prometheus-operator/releases/download/v0.63.0/bundle.yaml | kubectl create -f -

Create credentials for Observability Service: API Prerequisites Observability

There you get a JSON with your username, password and your pushMetricsUrl (inside the urls object).

Create a kubernetes secret with your Observability credentials:

kubectl create secret generic observabilitysecret \
 --from-literal=username='<your_observability_username>'\
 --from-literal=password='<your_observability_password>'\
 -n default

Next you need to create a prometheus instance in order to scrape your metrics and push them to Observability.

In the example below we created a PodMonitor in order to scrape the metrics of a pod. You could also create a ServiceMonitor to scrape metrics there.

Here you have to use your pushMetricsUrl which you got with your username and password.

apiVersion: v1
kind: ServiceAccount
metadata:
 name: prometheus
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
 name: prometheus
rules:
 - apiGroups: [""]
 resources:
 - nodes
 - nodes/metrics
 - services
 - endpoints
 - pods
 verbs: ["get", "list", "watch"]
 - apiGroups: [""]
 resources:
 - configmaps
 verbs: ["get"]
 - apiGroups:
 - networking.k8s.io
 resources:
 - ingresses
 verbs: ["get", "list", "watch"]
 - nonResourceURLs: ["/metrics"]
 verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
 name: prometheus
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: ClusterRole
 name: prometheus
subjects:
 - kind: ServiceAccount
 name: prometheus
 namespace: default
---
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
 name: prometheus
spec:
 serviceAccountName: prometheus
 podMonitorSelector:
 matchLabels:
 team: frontend
 resources:
 requests:
 memory: 400Mi
 enableAdminAPI: false
 remoteWrite:
 - url: "<your_pushMetrics_URL>"
 basicAuth:
 username:
 name: observabilitysecret
 key: username
 password:
 name: observabilitysecret
 key: password
---
apiVersion: v1
kind: Service
metadata:
 name: prometheus
spec:
 type: ClusterIP
 ports:
 - port: 80
 targetPort: http
 protocol: TCP
 name: http
 selector:
 prometheus: prometheus

2. Create a PodMonitor for your application

In order to monitor your application, you need to create a PodMonitor to scrape the metrics of your pod:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
 name: product-demo
 labels:
 team: frontend
spec:
 selector:
 matchLabels:
 app: product-demo
 podMetricsEndpoints:
 - port: metrics
 namespaceSelector:
 any: true

3. Visualize your metrics in Grafana

Get your grafanaUrl, grafanaAdminUsername and grafanaAdminPassword from step 2. Open the grafanaUrl in browser and use your Grafana credentials. Click the Explore symbol:

Check your metrics using simple query. Query up{job="node-exporter"} to find the SKE cluster worker nodes detail:

You can verify your metrics by comparing the IPs from your kubernetes node details using below command:

kubectl get nodes -o wide

NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
shoot--9kjnrurmk8--pawanpg-worker-jpqie-z1-767df-8xvkg Ready <none> 4h8m v1.21.10 10.250.0.151 <none> Flatcar Container Linux by Kinvolk 3033.2.3 (Oklo)
shoot--9kjnrurmk8--pawanpg-worker-jpqie-z1-767df-dsg99 Ready <none> 20d v1.21.10 10.250.3.136 <none> Flatcar Container Linux by Kinvolk 3033.2.3 (Oklo)
shoot--9kjnrurmk8--pawanpg-worker-jpqie-z1-767df-g2q42 Ready <none> 20d v1.21.10 10.250.3.144 <none> Flatcar Container Linux by Kinvolk 3033.2.3 (Oklo)