How to renew Kubeadm expired certificates

System-wide kubeadm Certificates in Kubernetes

In Kubernetes, system-wide certificates generated by kubeadm are critical for securing communication between various components of the cluster. They are used to enable mutual authentication, encryption, and ensuring integrity among Kubernetes control plane components and worker nodes.

When you create a cluster using kubeadm or join nodes to the cluster, certificates are automatically generated and stored in the control plane node(s). These certificates have a default expiration period of 365 days (1 year). If they are not renewed before expiration, the cluster may stop functioning properly as secure communication between components will fail.

Role of kubeadm Certificates in Kubernetes

Certificates generated by kubeadm adhere to the TLS (Transport Layer Security) protocol and are stored in the cluster in /etc/kubernetes/pki. They are primarily used for:

1. Securing Communication Between Control Plane Components

• Control plane components such as the kube-apiserver, kube-controller-manager, kube-scheduler, and etcd use certificates to securely communicate with each other.

  • For example, kube-apiserver and etcd require a certificate to encrypt their communication.

2. Authenticating Worker Nodes to the Kubernetes API Server

• Worker nodes (using the kubelet) need client authentication certificates to connect securely with the kube-apiserver for tasks like pod scheduling, retrieving configurations, and updating status.

3. Enabling Secure Connections Between Users and the Cluster

• Users (via tools like kubectl) interact with the Kubernetes API Server securely using client certificates and keys.

4. Serving Kubernetes Webhooks and Extensions

• Kubernetes webhooks, admission controllers, and any custom extensions often use certificates to interact with the cluster securely.

5. Enabling Kubernetes API Server to Use TLS for Secure Access

• The Kubernetes API Server uses certificates to handle TLS for incoming network requests (from nodes, users, or third-party apps)

Checking and Manually Renewing Certificates

By default, the kubeadm certificates expire every 365 days. When certificates in your cluster expire, it can lead to connectivity issues and errors such as:

These errors can affect the core controllers’ ability to establish TLS connections with the Kubernetes API server. To ensure a smooth operation, promptly address certificate expiration.

Checking the kubeadm cert expiration

 
$ sudo kubeadm certs check-expiration 

Backing up the old certs and configs 

$ mkdir -p $HOME/k8s-old-certs/pki 
$ sudo /bin/cp -p /etc/kubernetes/pki/*.* $HOME/k8s-old-certs/pki  $ mkdir -p $HOME/k8s-old-certs/pki/etcd  $ sudo /bin/cp -p /etc/kubernetes/pki/etcd/*.* $HOME/k8s-old-certs/pki/etcd  $ sudo /bin/cp -p /etc/kubernetes/*.conf $HOME/k8s-old-certs  $ mkdir -p $HOME/k8s-old-certs/.kube  $ sudo /bin/cp -p ~/.kube/config $HOME/k8s-old-certs/.kube/. 

Renewing the certificates 

$ sudo kubeadm certs renew all

View full size

The certificate used by kubelet 

You'll find four files /var/lib/kubelet/pki/. One of them is kubelet.crt. This file has also expired if we check with openssl: 

$ sudo ls /var/lib/kubelet/pki 
kubelet-client-2023-09-28-23-51-54.pem  kubelet-client-2024-08-09-20-31-07.pem  kubelet-client-current.pem  kubelet.crt  kubelet.key  $ sudo cat /var/lib/kubelet/pki/kubelet.crt | openssl x509 -noout -enddate  notAfter=Sep 27 22:51:54 2024 GMT

Stop the Service and Deleting old certificates

$ sudo systemctl stop kubelet  $ sudo rm /etc/kubernetes/kubelet.conf  $ sudo ls /var/lib/kubelet/pki  $ sudo rm /var/lib/kubelet/pki/kubelet-client-current.pem  $ sudo rm /var/lib/kubelet/pki/kubelet.crt  $ sudo rm /var/lib/kubelet/pki/kubelet.key 

Fixing Kubelet Service 

$ sudo kubeadm init phase kubeconfig kubelet $ sudo systemctl start kubelet 

Updating the client data configs

$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config  $ sudo chown $(id -u):$(id -g) $HOME/.kube/config 

What If Certificates Expire

If your certificates expire before being renewed, you must re-install linkerd and LumenVox in the order shown below, after renewing them:

Uninstall LumenVox 

$ helm uninstall lumenvox -n lumenvox 

Uninstall linkerd 

$ export PATH=$PATH:$HOME/.linkerd2/bin 
$ linkerd viz uninstall | kubectl delete -f -  $ linkerd jaeger uninstall | kubectl delete -f -  $ linkerd uninstall | kubectl delete -f - 

Install linkerd

$ linkerd check --pre  $ linkerd install --crds | kubectl apply -f -   $ linkerd install --set proxyInit.runAsRoot=true --set proxyInit.iptablesMode=nft | kubectl apply -f -   $ linkerd check  $ linkerd viz install | kubectl apply -f -   $ linkerd jaeger install | kubectl apply -f - 

Install LumenVox 

$ helm install lumenvox lumenvox/lumenvox -n lumenvox -f values.yaml

Monitoring Expiration

If you are using some monitoring framework, such as Prometheus, you can configure a metric, such as this to keep track of the amount of time remaining before the certificates expire:

sum(kubelet_certificate_manager_client_ttl_seconds/86400)

Which should indicate the number of days remaining before the certificate expiration.

Best Practices to Maintain Certificates

To ensure stability, follow these best practices:

1. Automate Certificate Monitoring Use monitoring scripts or tools like Prometheus & Grafana to periodically assess expiration and alert you 30-60 days before expiration.

2. Renew Certificates Before Expiry Schedule regular maintenance windows to renew certificates proactively (e.g., every 9 months).

3. Use Long-Lived Certificates for CA if Possible When bootstrapping a cluster with kubeadm, you can manually extend the CA certificate's expiration period (the CA is critical for signing all other certificates). for example:

   sudo kubeadm init --certificate-expiration=87600h 

4. Version Control and Backup Certificates Always back up your certificates in /etc/kubernetes/pki, especially the CA certificates. If something goes wrong during renewal, you can restore from the backup.

5. Rotate kubeconfig Files Update the kubeconfig files (e.g., ~/.kube/config, admin.conf) after certificate renewal using the following command: 

   sudo kubeadm init --certificate-expiration=87600h 

Automate with Managed Tools (Optional)

To reduce operational overhead, consider adopting Kubernetes tools like:

• Cert-Manager: Automates TLS certificate management with dynamic renewal for custom certificates (e.g., for Ingress).

• Managed Kubernetes (e.g., EKS, GKE, AKS): If using a managed Kubernetes solution, the cloud provider often handles control-plane certificates automatically.

Disaster Recovery

If certificates expire before being renewed, you'll need to manually restore or renew them:

1. SSH into a control plane node.

2. Use kubeadm certs renew to regenerate certificates.

3. Restart the necessary Kubernetes components.

To avoid this, build alerting and monitoring mechanisms, and always test your renewal process in staging environments first.

By combining proactive monitoring, scheduled renewals, and robust backups, you can avoid certificate expiration issues and maintain a robust Kubernetes cluster.