The mystery of Kubernetes cluster certificates renewal

This note based on Kubernetes 1.15

Check current certificates expiration date.

Run: kubeadm alpha certs check-expiration

# kubeadm alpha certs  check-expiration
CERTIFICATE                EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
admin.conf                 Apr 18, 2021 12:01 UTC   364d            no      
apiserver                  Apr 18, 2021 12:01 UTC   364d            no      
apiserver-etcd-client      Apr 18, 2021 12:01 UTC   364d            no      
apiserver-kubelet-client   Apr 18, 2021 12:01 UTC   364d            no      
controller-manager.conf    Apr 18, 2021 12:01 UTC   364d            no      
etcd-healthcheck-client    Apr 18, 2021 12:01 UTC   364d            no      
etcd-peer                  Apr 18, 2021 12:01 UTC   364d            no      
etcd-server                Apr 18, 2021 12:01 UTC   364d            no      
front-proxy-client         Apr 18, 2021 12:01 UTC   364d            no      
scheduler.conf             Apr 18, 2021 12:01 UTC   364d            no

Renew all available certificates

Run: kubeadm alpha certs renew all

# kubeadm alpha certs renew all                  
certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healtcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed
[root@s139 ~]# 
[root@s139 ~]# kubeadm alpha certs  check-expiration
CERTIFICATE                EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
admin.conf                 Apr 18, 2021 12:48 UTC   364d            no      
apiserver                  Apr 18, 2021 12:48 UTC   364d            no      
apiserver-etcd-client      Apr 18, 2021 12:48 UTC   364d            no      
apiserver-kubelet-client   Apr 18, 2021 12:48 UTC   364d            no      
controller-manager.conf    Apr 18, 2021 12:48 UTC   364d            no      
etcd-healthcheck-client    Apr 18, 2021 12:48 UTC   364d            no      
etcd-peer                  Apr 18, 2021 12:48 UTC   364d            no      
etcd-server                Apr 18, 2021 12:48 UTC   364d            no      
front-proxy-client         Apr 18, 2021 12:48 UTC   364d            no      
scheduler.conf             Apr 18, 2021 12:48 UTC   364d            no

,but is it really updated ? How to verify ?

check current api-server validity. “Not After : Apr 18 12:01:25 2021 GMT”

[root@s139 ~]# openssl s_client -host master-lb -port 6443 | openssl x509 -text
depth=0 CN = kube-apiserver
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = kube-apiserver
verify error:num=21:unable to verify the first certificate
verify return:1
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 3600651870215984932 (0x31f815cc8411d724)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: xxx
        Validity
            Not Before: Apr 18 11:56:00 2020 GMT
            Not After : Apr 18 12:01:25 2021 GMT
        Subject: CN=kube-apiserver

Why ? Oh, Have to to restart api-server first.

crictl stop $(crictl ps |grep kube-apiserver | awk {'print $1'})

Run the same openssl s_client again

[root@s139 ~]# openssl s_client -host master-lb -port 6443 | openssl x509 -text
depth=0 CN = kube-apiserver
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = kube-apiserver
verify error:num=21:unable to verify the first certificate
verify return:1
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 5538339693505838144 (0x4cdc229b9cdc7c40)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: xxx
        Validity
            Not Before: Apr 18 11:56:00 2020 GMT
            Not After : Apr 18 12:48:39 2021 GMT
        Subject: CN=kube-apiserver

OK, it now updated. Easy, Right ?

Let see the tricky part. The Kubelet configuration file:

/etc/kubernetes/kubelet.conf

add the last two line, kubeadm have put based64 data into client-certificate-data and client-key-data.

client-certificate-data: LS0tLS1CR...
client-key-data: LS0tLS1CRUdJTiBSU0EgU...

“kubeadm alpha certs renew all” did t not touch kubelet.conf. so you have to update it manually to these certificates instead.

client-certificate: /etc/kubernetes/pki/apiserver-kubelet-client.crt
client-key: /etc/kubernetes/pki/apiserver-kubelet-client.key

How to verify this file have been renewed ?

cat /etc/kubernetes/pki/apiserver-kubelet-client.crt  | openssl x509 -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 2687675835970532787 (0x254c8aeed26cb5b3)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: xxx
        Validity
            Not Before: Apr 18 11:56:00 2020 GMT
            Not After : Apr 18 12:48:39 2021 GMT

OK? then just restart kubelet.

PS. Some document say use this setting

client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
client-key: /var/lib/kubelet/pki/kubelet-client-current.pem

It is wrong, kubeadm renew did not change on these files