Knowledgebase

Release Notes
FAQ
Support
Release Notes FAQ Support

Getting Started with Monitoring of STACKIT SKE with Argus

This section shows the process how STACKIT SKE and customer workload on SKE service can be monitored using STACKIT Argus service. 

Overview

Prerequisites

  • You have ordered following Services
  • Basic knowledge of Prometheus, Grafana and Kubernetes.

  • Verify your Kubernetes cluster using kubectl command. Kubectl is a command line tool which allows you to run commands against Kubernetes clusters.

    $ kubectl get nodes

    NAME                                                                                         STATUS   ROLES    AGE   VERSION
    shoot--9kjnrurmk8--pawanpg-worker-jpqie-z1-767df-dsg99   Ready    <none>   19d   v1.21.10
    shoot--9kjnrurmk8--pawanpg-worker-jpqie-z1-767df-g2q42   Ready    <none>   19d   v1.21.10


Deploy Prometheus and kube-state-metric on SKE cluster (Step 1)

After clone both the repos follow the below steps:

SKE_TEST:~$  ls

StateMetric  kubernetes-prometheus                                         ########  Downloaded Repos in your local environment

SKE_TEST:~$   cd kubernetes-prometheus

SKE_TEST:~$~/kubernetes-prometheus $   ls

README.md    clusterRole.yaml  config-map.yaml  prometheus-deployment.yaml  prometheus-ingress.yaml  prometheus-service.yaml

SKE_TEST:~$~/kubernetes-prometheus $ kubectl create namespace monitoring    ######## Create a namespace monitoring, to run all the prometheus related resource in monitoring namespace

SKE_TEST:~$~/kubernetes-prometheus $ kubectl apply -f . -n monitoring       ########   Create all the kubernetes resources

SKE_TEST:~$~/kubernetes-prometheus $ kubectl get all -n monitoring         #########  Check all the resources in Running state

NAME                                        READY   STATUS    RESTARTS   AGE
pod/prometheus-deployment-bb68585c9-fjg4w   2/2     Running   5          2d23h

NAME                         TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
service/prometheus-service   NodePort   100.68.137.39   <none>        8080:30000/TCP   4d20h

NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/prometheus-deployment   1/1     1            1           4d20h

NAME                                               DESIRED   CURRENT   READY   AGE
replicaset.apps/prometheus-deployment-599bbd9457   0         0         0       4d20h
replicaset.apps/prometheus-deployment-7bbd57d657   0         0         0       3d1h
replicaset.apps/prometheus-deployment-85584d5564   0         0         0       3d1h
replicaset.apps/prometheus-deployment-bb68585c9    1         1         1       2d23h


########  Kubernetes Prometheus is installed , Now install kube-state-metric ###################################

SKE_TEST:~$ cd  kube-state-metrics-configs

SKE_TEST:~$~/kube-state-metrics-configs $  ls

README.md  cluster-role-binding.yaml  cluster-role.yaml  deployment.yaml  service-account.yaml  service.yaml

SKE_TEST:~$~/kube-state-metrics-configs $ kubectl apply -f . 

SKE_TEST:~$~/kube-state-metrics-configs $ kubectl get all --all-namespaces |grep kube-state

kube-system     pod/kube-state-metrics-5f684858d6-l5fx8                   1/1     Running   0          4d22h
kube-system     service/kube-state-metrics                   ClusterIP      None             <none>            8080/TCP,8081/TCP              4d22h
kube-system     deployment.apps/kube-state-metrics                   1/1     1            1           4d22h
kube-system     replicaset.apps/kube-state-metrics-5f684858d6                   1         1         1       4d22h

######################################  Kube-state-metric is also deployed on kubernetes ################################
  • Find the remote-write end point:

           Create API credentials for your Argus service via portal. 

 Highlighted information: apiUrlSwagger, instanceId, username and password. Username and password will be used in following step while we add remote-write configuration in Prometheus.

  • Open the apiUrlSwagger link in browser:

Click on the Authorize button and put your credentials in username and password section. Once you login with credential, then go to Instance API's (/v1/instance/{instanceId}. Provide your instanceId and some random character like "ABC" at Authorization and click on Execute.

In the response field you have two important information. First information is about Grafana grafanaAdminPassword, grafanaAdminUser and grafanUrl which will be used in last step to verify your SKE metric data in Grafana dashboard. Second information is remote-write endpoint which is called as pushMetricUrl. We need to use this endpoint in the next step while configuring Prometheus.

Configure Remote-Write endpoint in Prometheus config-map.yaml file (Step 2)

Provide pushMetricUrl and API username and Password in the remote_write section of config-map.yaml

SKE_TEST:~$~/kubernetes-prometheus $   ls

README.md    clusterRole.yaml  config-map.yaml  prometheus-deployment.yaml  prometheus-ingress.yaml  prometheus-service.yaml

#######   Edit  config-map.yaml add remote write endpoint section under scrape config

SKE_TEST:~$~/kubernetes-prometheus $  vi config-map.yaml

scrape_configs:
  
remote_write:
  - url: "https://push.metrics.testing.argus.eu01.stackit.cloud/instances/test-awesome/api/v1/receive"
      basic_auth:
        username: "testing_test-awesome_378432da-4056-45ea-8944-461dc4393a8a"
        password: "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"


SKE_TEST:~$~/kubernetes-prometheus $ kubectl apply -f config-map.yaml

####  After apply the configuartion you need to restart the deployment

SKE_TEST:~$~/kubernetes-prometheus $ kubectl rollout restart deployments.apps -n monitoring prometheus-deployment

Your SKE cluster now starts sending metrics data to your Argus instance (Thanos storage). To visualize the data, we can have a look into Grafana dashboard.


Visualize your Metrics in Grafana (Step 3)

Get your grafanaUrl ,garfanaAdminUsername and grafanaAdminPassword from Step 2. Open grafanaUrl in browser and use your Grafana credentials. Click the Explore symbol.


 


Check your metrics using simple query. Query up{job="node=exporter"} to find the SKE cluster worker nodes detail.  

You can verify your metrics by comparing the IPs from your kubernetes node details using below command:

SKE_TEST:~$~  kubectl get nodes -o wide

NAME                                                     STATUS   ROLES    AGE    VERSION    INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                                             KERNEL-VERSION     CONTAINER-RUNTIME
shoot--9kjnrurmk8--pawanpg-worker-jpqie-z1-767df-8xvkg   Ready    <none>   4h8m   v1.21.10   10.250.0.151   <none>        Flatcar Container Linux by Kinvolk 3033.2.3 (Oklo)   
shoot--9kjnrurmk8--pawanpg-worker-jpqie-z1-767df-dsg99   Ready    <none>   20d    v1.21.10   10.250.3.136   <none>        Flatcar Container Linux by Kinvolk 3033.2.3 (Oklo)  
shoot--9kjnrurmk8--pawanpg-worker-jpqie-z1-767df-g2q42   Ready    <none>   20d    v1.21.10   10.250.3.144   <none>        Flatcar Container Linux by Kinvolk 3033.2.3 (Oklo)