Topologies

The following article describes what topologies are and how they can be used in SKE to improve your application’s resilience.

Topology is a widely-used concept in datacenters, public clouds and Kubernetes itself. It refers to the failure-resistent architecture of compute resources by subdividing their location into regions and zones (also called Availability Zones or AZs). STACKIT works with topologies, too. You can learn more about the STACKIT-specific topology layout here.

As an SKE customer, you can use this topology to your advantage: By distributing your Kubernetes workloads across multiple AZs, you can improve resilience and protect your services against unforeseen failures. In the following, we will have a look at how exactly this works in SKE.

Topologies in SKE

Of course, Kubernetes has some built-in mechanisms to make workloads failure-resistant, think of Deployments that start many instances of the same application and Loadbalancers that route traffic to all of them (that are healthy).

On top of that, you can leverage STACKIT Regions and Availability Zones to further improve the resilience of your applications. You achieve this by spanning your cluster over multiple AZs. For example, in our region “Heilbronn/Germany”, there are the following zones:

• eu01-1
• eu01-2
• eu01-3
• eu01-m

These AZs can be used in SKE when creating node pools for your cluster.

While a single Node Pool (consisting of one or more machines) can only reside in a specific AZ, it is possible to create multiple Node Pools in different AZs.

It is not possible to change the AZ of existing Node Pools. However, you can create a new Node Pool in the desired AZ and then move your workloads over. This is achieved by draining the old nodes. Pay close attention to the migration of Volumes during this process: How to migrate your storage to another AZ.

Single AZs and Metro AZ

The Metro AZ is a special STACKIT offering that is actually a High Availability Zone spanning the other three AZs (eu01-1 to eu01-03, called Single AZs). It is designed for applications that cannot achieve resilience on their own - in case of a failure of one of the Single AZs an application placed in the Metro AZ is started in one of the other Single AZs. You can learn more about this concept here: Basic Concepts Compute Engine.

As described above, the built-in Kubernetes mechanisms and its ability to span multiple AZs make it very failure-resistant already. You do not have to use the Metro AZ in SKE to achieve High Availability if the node pools are configured correctly to run in multiple AZs.

Working with Multiple AZs in Kubernetes

Suppose you have created an SKE cluster and configured two Node Pools, one in AZ eu01-1 and one in AZ eu01-2. Both Node Pools contain one machine each. After connecting to the cluster with the .kubeconfig file, you can see that each node is labelled according to its placement in the respective AZs:

kubectl get nodes --show-labels

NAME STATUS ROLES AGE VERSION LABELS
node-1 Ready <none> 1h v1.30.3 failure-domain.beta.kubernetes.io/region=RegionOne,failure-domain.beta.kubernetes.io/zone=eu01-1,topology.cinder.csi.openstack.org/zone=eu01-1,topology.kubernetes.io/region=RegionOne,topology.kubernetes.io/zone=eu01-1
node-2 Ready <none> 1h v1.30.3 failure-domain.beta.kubernetes.io/region=RegionOne,failure-domain.beta.kubernetes.io/zone=eu01-2,topology.cinder.csi.openstack.org/zone=eu01-2,topology.kubernetes.io/region=RegionOne,topology.kubernetes.io/zone=eu01-2

(The node names and the list of labels have been shortened for brevity.)

As the Kubernetes documentation states:

Kubernetes automatically spreads the Pods for workload resources (such as Deployment or StatefulSet) across different nodes in a cluster. This spreading helps reduce the impact of failures.

As Kubernetes distributes the Pods of a Deployment across different nodes automatically, we already have a working failure-resistant setup across two AZs in this example case. Learn more about how Kubernetes supports multiple AZs: Multiple Zones.

However, if there are several nodes in the same Node Pool, it could be that Pods get scheduled within the same AZ. You can fine-tune the behaviour of Multi-AZ clusters manually. The mechanisms to do that are:

• Node Selector Constraints
• Pod Topology Spread Constraints

Example for Node Selector Constraints:

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
    - name: nginx
      image: nginx
      imagePullPolicy: IfNotPresent
  nodeSelector:
    topology.kubernetes.io/zone: eu01-1

The example Pod running nginx would be placed on a node in the eu01-1 AZ because of the node selector constraint. The node selector field in the Pod specification is a simple key-value list, where the key must match the node’s label name and the value must match the label’s value. This means you could use any of the labels shown above (kubectl get nodes —show-labels) to constrain node selection to specific AZs.

Example for Pod Topology Spread Constraints:

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  labels:
    app: example
spec:
  containers:
    - name: nginx
      image: nginx
      imagePullPolicy: IfNotPresent
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: DoNotSchedule
      labelSelector:
        matchLabels:
          app: nginx

This approach will take into account all nodes that have the topology.kubernetes.io/zone label present. The constraint is applied for all Pods that have the label “app: example”. Kubernetes will make sure that these pods are scheduled across all considered nodes in such a way that the maxSkew value is not violated. (The maxium skew describes how big a difference between the number of pods on each considered node is tolerated.)

Using Persistent Volumes with Availability Zones

When your applications require Persistent Volumes by requesting Persistent Volume Claims, you need to pay attention to some details in regard to AZs.

If you create a Persistent Volume Claim that is not yet used by any Pod, it will not create a bound Persistent Volume right away. That means the volume is not placed in a specififc AZ of your cluster because Kubernetes does not yet know where the mounting Pod will be placed. Instead, the placement of this Volume occurs when a Pod is created that references the Persistent Volume Claim. The needed Persistent Volume then gets placed in the same AZ as the Pod requesting it. This late-binding mode is called WaitForFirstConsumer.

Example for Dynamic Volume Creation

Suppose you have the following PersistentVolumeClaim in a file called pvc.yaml:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: example-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
  storageClassName: "premium-perf1-stackit"

You can apply this by running kubectl apply -f pvc.yaml. Check the status of this PVC:

kubectl describe pvc example-pvc

Name:            example-pvc
Namespace:       default
StorageClass:    premium-perf1-stackit
Status:          Pending
Volume:
Labels:          <none>
Annotations:     <none>
Finalizers:      [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:      Filesystem
Used By:         <none>
Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Normal WaitForFirstConsumer 7s (x3 over 26s) persistentvolume-controller waiting for first consumer to be created before binding

As you can see, no Volume is bound to the Claim yet. The PVC is waiting for the first consumer. Now, let’s create a pod using this PVC (using a YAML file called pod.yaml):

apiVersion: v1
kind: Pod
metadata:
  name: example
spec:
  volumes:
    - name: pv-storage
      persistentVolumeClaim:
        claimName: example-pvc
  containers:
    - name: exmaple-container
      image: busybox
      volumeMounts:
        - mountPath: "/tmp/data"
          name: pv-storage

Now, the PVC is bound. The name of the bound Persistent Volume can be retrieved as follows:

kubectl get pvc example-pvc

NAME          STATUS   VOLUME        CAPACITY   ACCESS MODES   STORAGECLASS            AGE
example-pvc   Bound    <volume-name> 2Gi        RWO            premium-perf1-stackit   10

It is now possible to verify the location of the Volume in the AZ:

kubectl get pv <volume-name> -o yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  ...
  name: <volume-name>
spec:
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 2Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: example-pvc
    namespace: default
    ...
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: topology.cinder.csi.openstack.org/zone
              operator: In
              values:
                - eu01-2
  persistentVolumeReclaimPolicy: Delete
  storageClassName: premium-perf1-stackit
  volumeMode: Filesystem
status:
  phase: Bound

The nodeAffinity field shows that this particular Volume is required to be scheduled

in the eu01-2 zone (this can be different for you). It can therefore not be mounted in another AZ. See below for a migration guide.

For more information on binding modes, see the official Kubernetes documentation.

Migrating Volumes between AZs

In some cases, you might want to migrate Persistent Volumes between AZs. We have put together a tutorial on how to do that here: How to migrate your storage to another AZ.