How to resize Kubernetes Statefulset Storage

How to resize Kubernetes Statefulset Storage

So you want to know about Kubernetes resizing statefulset storage? Well, statefulsets are a type of Kubernetes resource used for managing stateful applications. Unlike deployments, they always restart with their data intact. By contrast, deployments always throw out all data accumulated in the controller during restart.

Applications that require a level of ordering and predictability use Statefulsets.For example databases and message brokers. They provide several key features, such as stable network identifiers, persistent storage, and ordered deployment and scaling. Overall, statefulsets offer a powerful way to manage stateful applications on Kubernetes. While not a common task in daily Kubernetes administration; resizing statefulset volumes can be a good opportunity to learn more about managing stateful applications on k8s. 

They provide several key features, the main are:

  • Stable, unique network identifiers: Each pod in a StatefulSetis assigned a stable, unique DNS name, which allows applications to refer to the pods by name. Also, it ensures that the network identifiers for the pods remain consistent even if the pods are rescheduled or restarted.
  • Persistent storage: StatefulSets support persistent volumes, which allows the pods to retain data even if they are rescheduled or restarted. This is important for applications that need to store data in a durable manner, such as databases.
  • Ordered deployment and scaling: StatefulSets support ordered deployment and scaling, which means that the pods are created and scaled in a predictable order. This is important for applications that require a certain sequence of operations, such as performing a rolling upgrade or backing up data before scaling down.

One of the current limitations of Kubernetes statefulSets is that you can only edit a few statefulset values. This means that you won’t be able to actually directly change the storage size of its pods. But there is a way to achieve this. And it’s interesting because it shows us some interesting repercussions of the design decisions behind kubernetes. 

Why was the Kubernetes pod unhappy? Because it was in a state-full set.

Cocreated with ChatGPT
Souce: Dall-E: “cloud computing funny cartoon”

How to change the size of kubernetes statefulset volumes

The first thing your volumes need to use is a storageClass hat supports resizing. Simply kubectl describe storageclass <<name>> to find out. If it doesn’t you will have to create a copy of the statefulSet with larger volumes. Then you will mount both statefulSets to a temporary pod and copy all the data to the new volume. But if your storage class allows resizing you can do something much more elegant.

  1. Save current statefulset using kubectl get statefulset <<name>> -o yaml statefulset.yaml
  2. edit the document and remove resource id and creation timestamp. Also, edit the volume claim template to the desired storage size
  3. Now get and edit persistent volume claims to the desired size. kubectl get pvc followed by kubectl edit pvc <<name>> (you can edit all PVCs in a namespace by omitting name).
  4. Your containers and cloud provider probably don’t allow the fly resizing of storage, so you will have to restart pods one by one (delete the statefulset pods one by one).
  5. Now your pods show the correct storage size but your stateful set still has the old values. To update it you have to delete and recreate it without disturbing your pods. To do this you simply need to delete it while orphaning its pods: kubectl delete statefulset --cascade=orphan <<name>>
  6. Now reapply the statefuset file you edited in steps 1 and 2: kubectl apply -f statefulset.yaml

This allows us to very quickly and safely resize kubernetes statefulSet volumes. In some cases, this operation can happen without downtime. It also shows us a few interesting insights into how Kubernetes works under the hood.

Kubernetes Resources can survive  parent resource destruction

In Kubernetes, even if the parent resource is destroyed, resources created by other resources can continue to exist. For example, if a deployment or statefulSet creates a set of pods, the pods will continue to exist even if the deployment is deleted. This can be useful for ensuring the availability of an application. Even when the original resource that created it is no longer present. 

Also, it allows us to avoid cases where AdmissionControles would reject our edit or creation of kubernetes resource.  For example, editing the size of statefulset volumes (on how look above!).

Kubernetes doesn’t always care about all values in a resource definition

Usually, we assume that kubernetes will simply try to maintain all child resource configurations. So if a deployment created a pod, and we change an env var in said pod definition, kubernetes should complain. It should change it back to the previous value using the deployment definition as a source of truth.

In our case, this does not happen. Kubernetes seems to be more than happy to accept different storage sizes in its pods and in the persistent volume claim template. But why does this happen? As soon as we look at what happens in the background, the reason becomes very clear.

Kubernetes sees a volume claim template and forwards it to the storage class controller. The volume is now not its problem, it will simply wait for it to finish creating the persistent volume and persistent volume claim. Once that’s done it attaches the pvc to the pod. 

Next time the pod restarts, it will again simply attach the pvc to the pod. This means that as long as the PVC exists, the statefulset controller doesn’t care what happens to it and will attach it to the pod.

In conclusion, while not a common task in daily Kubernetes administration, resizing statefulset storage can be a valuable skill to have in your Kubernetes toolkit. By using a storageClass that supports resizing and following the steps outlined in the article, you can safely and efficiently change the storage size of your statefulset volumes. Understanding the limitations and capabilities of statefulsets can also provide insight into the design decisions behind Kubernetes and how it manages replicated pods.

AI disclaimer:

This post was assisted by OpenAI’s GPT AI chatbot. I did this because I was interested to explore using AI to assist with some up-to recently human-only labor. Also I am lazy. Here are some of the prompt’s whose output was used in this article:

  • I fed it the first two paragraphs from my Calico to Flannel – changing CNI article. I asked it to reply to all the following prompts mimicking that style of writing.
  • How to change the volume size of kubernetes statefulset – the answer wasn’t really useful
  • write an introduction to a technical article on kubernetes – spot on, only a few tweaks in text above
  • what are kubernetes statefulsets – also spot on, used half of it
  • rewrite: Kubernetes resources created by other resources can survive its parent resource’s destruction – more than half of it got used in the paragraph
  • provide a title for the previous answer – ‘Kubernetes Resources can survive  parent resource destruction’
  • tell me a pun about kubernetes – the source of the pun in this article. I substituted deployment with pun as it was more on point.
  • please write a conclusion for the following article: + text of this article – result: “Resizing Kubernetes Statefulset Storage”
  • Then I tweaked the article and did a few stylistic prompts: “split into more sentences + paragraph” and “rewrite using active voice + sentence” eg:
  • rewrite using active voice: Statefulsets are used for applications where a level of ordering and predictability is required – the first sentence of the 2nd paragraph.

For now, I’m fascinated with the future prospects of AI-assisted work and I will for sure follow up on this topic withan article on AI-assisted writing. Bdv, the heading image is also generated by openAI’s Dall-e AI image generator using prompt: penguing fixing a network switch in the cloud and cloud computing funny cartoon.

Back to top