Troubleshooting

If you have trouble with your cluster, you can run various checks which help you to narrow down the problem and find a solution.

If you receive any errors not mentioned below and/or you are not able to find a solution by yourself, please open a ticket to Report an issue with Confidential Kubernetes. Please also provide the output of the command in the ticket description.

Issues during creation of a cluster

• Check if you are using the latest release: Image Version Table Confidential Kubernetes.
• Look for known issues for your version: Known Issues (in the Edgeless documentation).
• Check if you followed all steps on Installation and Setup.
• Check if you followed all steps on Create a Confidential Kubernetes cluster.
• Check if some information in the following files might be missing.
  If that is the case, you might receive various errors. Please read the error messages carefully, fill in the missing information and try to create the cluster again.
  • constellation-conf.yaml,
  • ~/.config/openstack/clouds.yaml or
  • ~/.stackit/credentials.json

Run the command constellation apply -y. Make sure that you wait until it is finished and then check its output for any information that might help you.

• If you encounter errors like the following, check the Project quota of your project and ensure that you have enough free resources to create all requested control planes, workers and load balancer.

  "Error: Error waiting for instance (xxxxxxx) to become ready: unexpected state 'ERROR', wanted target 'ACTIVE'."

• If you encounter errors like the following, check the name of your cluster in the constellation-config.yml file. Only lowercase letters, numbers or a dash can be part of the name.

  "Calling API: 400 Bad Request, status code 400, Body: {"code":3,"message":"one or more fields are invalid","details":[{"@type":"type.googleapis.com/google.rpc.BadRequest.FieldViolation","field":"loadBalancer.name","description":"Field name \"{your_cluster_name}-a6cad6d8-lb\" is not DNS conform (1-63 characters [a-z0-9-] and no '-' at either start or end)."}]}

• If you encounter errors like the following, there is a problem with the OpenStack User Access Token (UAT). Please use the infrastructure API to double check whether the OpenStack credentials are correctly created and included in the config files.

  Error: Error creating OpenStack identity client: Authentication failed

  For further information please follow the steps on:
  Installation and Setup

• If you encounter errors like the following, check whether the token stored in ~/.stackit/credentials.json (%USERPROFILE%\.stackit\credentials.json in Windows) is valid and the Service Account has the neccessary permissions.

  Error: Error getting status of load balancer functionality
  
    with module.stackit_loadbalancer[0].stackit_loadbalancer.loadbalancer,
    on modules/stackit_loadbalancer/main.tf line 10, in resource "stackit_loadbalancer" "loadbalancer":
    10: resource "stackit_loadbalancer" "loadbalancer" {
  
  Calling API: 403 Forbidden, status code 403, Body:
  {"status":403,"message":"authentication failed"}

  For further information please follow step 3 and onwards of this guideline:
  Installation and Setup

Issues after creation of the cluster

• Check the log files of nodes, pods, and other resources in the cluster for any irregularities. Most logs are available through Kubernetes’ standard logging interfaces.
  See: Logging (in the Kubernetes documentation).
• To debug issues occurring during boot time of the nodes, use the web console while the machine boots up to get a read-only view of the boot logs.
  See: How to access your server using web console.
• Try Debugging via a shell on a node.
  See also: Debug Running Pods (in the Kubernetes documentation).
  1. Figure out which node to connect to:

     kubectl get nodes
     # or to see more information, such as IPs:
     kubectl get nodes -o wide

  2. Connect to the node:

     kubectl debug node/constell-worker-xksa0-000000 -it --image=busybox

     You will be presented with a prompt. The nodes file system is mounted at /host.
  3. Once finished, clean up the debug pod:

     kubectl delete pod node-debugger-constell-worker-xksa