How to rotate credentials

New credentials handling in SKE brings new behavior and deprecations

Cluster administrators: Please read this document carefully.

We improved the rotation of credentials for your clusters. In this process, two SKE API endpoints will be deprecated, while three new ones are introduced. Read on to find out why we decided to implement these improvements and how you can transition smoothly, both in the STACKIT Cloud Portal and the SKE API.

Summary

We’re introducing some changes that will greatly improve your cluster security, namely

• more comprehensive cluster credentials rotation
• short-lived admin kubeconfigs

These changes come with the addition of new and deprecation of old API endpoints, which affects the portal workflows as well as the API usage.

General changes:

• Each time a kubeconfig is requested a new one will be generated which expires after a specified time (one hour to six months). The previously generated kubeconfigs remain valid until they expire.
• Rotating your cluster’s credentials involves multiple steps instead of one. In-between the steps, you should issue new credentials to all out-of-cluster clients.
• After the first step (Start credentials rotation), a rolling update is applied to replace all cluster nodes.
• Using the old credential rotation feature or retrieving the old token based kubeconfigs is impossible for clusters
  • with Kubernetes versions 1.27 or greater
  • that have already used the new credentials rotation or requested short-lived kubeconfigs

Portal

• Kubeconfigs can still be easily downloaded, but an expiration time can be specified.
• The Rotate credentials button now indicates if a credentials rotation can be started or completed.
• Details about the current credentials rotation phase can be found in the cluster details view.

API

• Kubeconfigs can be obtained using a new endpoint for kubeconfig creation, with an optional request body to specify the expiration time.
• Rotating credentials now involves two endpoints - one for starting the rotation, the other one to complete the rotation.
• The existing endpoint to get information about your cluster has been extended with a section that reports the current phase of the credentials rotation.

Next, we explain how these changes affect cluster administration on a high level.

Implications for cluster administrators

Before you initiate a cluster credentials rotation, make sure to obtain a short-lived admin kubeconfig. The static token kubeconfig will no longer be valid after the first step of the new rotation process.

The process of rotating your cluster’s credentials now involves four steps:

1. If you have never obtained a short-lived admin kubeconfig before (i.e., if you have only used static token kubeconfigs so far), make sure to first download a short-lived admin kubeconfig. Static token kubeconfigs will stop working after the second step (Start credentials rotation).
2. Start the Credentials Rotation (including a complete node roll).
3. Manually update all out-of-cluster clients with new credentials.
4. Complete the Credentials Rotation.

This ensures that external clients (developers or applications that access your cluster from the outside) don’t experience authorization issues. The previously created short-lived admin kubeconfig from the first step will still be valid after the second step. In contrast, static token kubeconfigs will stop working after the rotation has been started.

Additionally, you can obtain a new short-lived admin kubeconfig after step 2. Restarting applications with the new credentials or updating the developers’ kubeconfigs will suffice. After completing the credentials rotation in step 4, the old credentials (short-lived, obtained before step 2; old static token) are no longer valid.

No changes will be necessary to in-cluster components. By performing an automatic rolling node update, we make sure that all Pods get an up-to-date ServiceAccount token using the new credentials.

Next, let’s take a look at how this change in process affects the STACKIT Cloud Portal, as well as your workflow using the API.

Portal

How the changes affect portal usage

Obtaining an admin kubeconfig

If you previously used the STACKIT Cloud Portal to obtain kubeconfigs, you can continue to do so. Instead of being able to download one directly (without an expiration time), you need to specify an expiration duration. This duration can be in a range from one hour to six months. After that, you can download and use the generated kubeconfig as before, until its expiration.

Rotating cluster credentials

Before you initiate a cluster credentials rotation, make sure to obtain a short-lived admin kubeconfig. The static token kubeconfig will no longer be valid after the first step of the new rotation process.

Previously, clicking Rotate credentials only invalidated the existing static token kubeconfig. As described above, the rotation of credentials is now a multi-step process. Therefore, you will now find a Start credentials rotation button instead.

1. Click this button to initiate a credentials rotation, including a rolling update of your nodes. Depending on the number of nodes in your cluster’s node pools and the configured maximum surge, this can take some time. During this phase, you won’t be able to trigger a new credentials rotation.
2. Track the status of the ongoing preparation for the rotation in the cluster overview. Once the first step is completed, the cluster status will be “Active” indicating that the rotation is prepared.
3. Now, update all out-of-cluster clients with new credentials. This means equipping all developers and external applications with new kubeconfigs or ServiceAccount tokens, respectively.
4. Once this is done, note that the button for the credentials rotation has changed to Complete credentials rotation. Click it to complete the process. This removes the old credentials from the cluster. As long as you don’t complete the process, both old and new short-lived admin kubeconfigs will continue to be valid until they expire.

API

How the changes affect the SKE API usage

This section describes new, changed and deprecated endpoints. You can also check the

SKE API spec for a comprehensive overview of all available endpoints.

Obtaining an admin kubeconfig

Previously, you could use the following endpoint to get the existing static token admin kubeconfig:

Get credentials - DEPRECATED

GET /v1/projects/{projectId}/clusters/{clusterName}/credentials

This endpoint is deprecated. It will not be usable anymore for clusters

• with Kubernetes version 1.27 or greater or
• that have already used the new credentials rotation or requested short-lived kubeconfigs

To obtain a kubeconfig going forward, use the new endpoint:

Create a kubeconfig - NEW

POST /v1/projects/{projectId}/clusters/{clusterName}/kubeconfig

Note that this is a POST request, and a new kubeconfig will be created every time the endpoint is called. There is no way to GET existing kubeconfigs.

Kubeconfigs generated via this endpoint have an expiration time, which is part of the response:

Create a kubeconfig - NEW

{
  "kubeconfig": "<kubeconfig string>", "expirationTimestamp": "<UTC timestamp of the expiration>"
}

The default expiration time is 1 hour. You can configure the expiration time by providing a request body, specifying the expiration time in seconds:

expirationSeconds

{
  "expirationSeconds": 15552000
}

This example request body will generate a kubeconfig that is valid for six months (60 * 60 * 24 * 30 * 6). “15552000” is also the highest possible value.

Rotating cluster credentials

You could previously rotate the static token kubeconfig using the endpoint:

Rotate cluster credentials - DEPRECATED

POST /v1/projects/{projectId}/clusters/{clusterName}/rotate-credentials

This endpoint is deprecated. It will not be usable anymore for clusters

• with Kubernetes version 1.27 or greater or
• that have already used the new credentials rotation or requested short-lived kubeconfigs

Because the credentials rotation is now a multi-step process with two distinct commands (“start” and “complete”), we introduced two new endpoints.

Starting the Credentials Rotation

Before you initiate a cluster credentials rotation, make sure to obtain a short-lived admin kubeconfig. The static token kubeconfig will no longer be valid after the first step of the new rotation process.

Start credentials rotation - NEW

POST /v1/projects/{projectId}/clusters/{clusterName}/start-credentials-rotation

Sending a POST request to this endpoint will initiate a credentials rotation, including a rolling update of your nodes. Depending on the number of nodes in your cluster’s node pools and the configured maximum surge, this can take some time. During this phase, you won’t be able to trigger a new credentials rotation.

Checking the phase of the Credentials Rotation

You can track the status of the first step for the rotation using the existing cluster endpoint.

Get a cluster - NEW

GET /v1/projects/{projectId}/clusters/{clusterName}

The status section has been extended by an object called credentialsRotation:

Get a cluster - NEW

{
  "name": "my-cluster",
  "kubernetes": {...},
  "nodepools": [...],
  "hibernation": {...},
  "maintenance": {...},
  "extensions": {...},
  "status": {
    "aggregated": "STATE_HEALTHY",
    "error": nil,
    "hibernated": false,
    "credentialsRotation": {
      "phase": "PREPARING",
      "lastInitiationTime": "<timestamp>",
      "lastCompletionTime": "<timestamp>"
    }
  }
}

The phase field contains the string **PREPARING** as long as the cluster is in the process of setting up the new credentials and rolling all cluster nodes. For clusters whose credentials have not yet been rotated, the phase will be **NEVER** and both timestamps will show the cluster’s creation timestamp.

Once the first step is completed, the phase field will contain **PREPARED**. You can now execute the next step:
Updating all out-of-cluster clients with new credentials. Equip all developers and external applications with new kubeconfigs or Service Account tokens, respectively.

Completing the Credentials Rotation

Once this step is finished, you can complete the credentials rotation:

Complete crendentials rotation - NEW

POST /v1/projects/{projectId}/clusters/{clusterName}/complete-credentials-rotation

This removes the old credentials from the cluster. As long as you don’t complete the process, both old and new short-lived admin kubeconfigs will continue to be valid until they expire.

Why we’re changing the credentials rotation process

The changes described above provide you with better control concerning your cluster’s security in two ways: We give you more control over the credentials in your cluster (For example: Certificate Authorities or token signing key), and we discontinue the support for static token kubeconfigs with Kubernetes v1.27.

More control over your cluster’s credentials

Your SKE cluster uses several credentials to ensure you and the cluster’s internal components can communicate securely with each other. These credentials include:

• Kubeconfigs (which you use to access your cluster)
• A Certificate Authority (CA), which is used to sign the certificate provided by your cluster’s Kubernetes API server, and also relevant for kubeconfigs.
• An ETCD Encryption Key, which is used to encrypt the content of Secret resources inside etcd.
• A Token Signing Key, used to sign the tokens for Service Accounts.

So far, the only credentials that you could roll using the API or the portal were the static token kubeconfig. Now, we enable you to roll all the credentials mentioned above. This increases the security for your clusters even further.

Ending support for static token kubeconfigs with Kubernetes v1.27

In addition to the extended credentials rotation, we are changing how cluster access is managed.

Introducing short-lived Kubeconfigs

We are replacing static token kubeconfigs with a more secure method: short-lived kubeconfigs.

This change further increases security by allowing you to limit the validity of any specific kubeconfig to a set time frame, from one hour up to six months. It also provides more flexibility, as you can now generate multiple admin kubeconfigs with different expiration times. As always, you retain full control and can still invalidate all issued kubeconfigs instantly by rotating your cluster’s CA.

Timeline and availability

• For Kubernetes 1.27 and greater: Short-lived kubeconfigs will be mandatory. Static token kubeconfigs will no longer be available for these versions.
• For Kubernetes < 1.27: You can begin using the new short-lived kubeconfigs immediately.

Important: Action for clusters < 1.27

You can use the new short-lived kubeconfigs or the new credentials rotation process on your older clusters right away.

However, please be aware that this is a one-way process. Once you use either of these new features on a specific cluster, the old method of retrieving static token kubeconfigs will be permanently disabled for that cluster.

To summarize

• We give you more capabilities and control concerning your cluster’s security by rotating not only the kubeconfig, but also the cluster’s CA, ETCD encryption key, and token signing key.
• Using static token kubeconfigs is a deprecated way to access your cluster. Instead, we provide a new possibility to generate short-lived kubeconfigs for increased security. This works for all Kubernetes versions right away. Static token kubeconfigs will no longer be available starting with Kubernetes version 1.27.

As always, if you have questions, feedback or trouble concerning these changes, please don’t hesitate to open a support ticket.