K3S Certificates, Access and Dashboards

[mhl787156]

This issue describes a non-common certification problem related to K3S.
According to this documentation (https://rancher.com/docs/k3s/latest/en/advanced/#certificate-rotation), certificates automatically expire every 12 months. Also that 'If the certificates are expired or have fewer than 90 days remaining before they expire, the certificates are rotated when K3s is restarted.'. However the rotation doesnt work very well for us it would seem.

In order to get rotation to work, you need to delete (or at least backup) your //var/lib/rancher/k3s/server/tls directory. Then you need to restart k3s. This will generate a new set of keys for you. See this tutorial: https://www.ibm.com/support/pages/node/6444205 (resolving the problem).

Then you will need to copy the new k3s.yaml file (/etc/rancher/k3s/j3s.yaml) to all of the desktop machines. Also have to change all the URLs to point to the flyingserver. A script can be written to do this.

Then! It would seem that the dashboard does not like this. Either all pods need to be manually restarted, or you can kubectl delete the dashboard deployment file. Downside to that is that it also attempts to delete the kubernetes-dashboard namespace. This may break as the dashboard gets stuck on terminating and you cannot redeploy the dashboard. Therefore to delete the dashboard, you will need to run kubectl proxy in one terminal, then follow the isntructions in this comment: kubernetes/kubernetes#19317 (comment)

[mhl787156]

Also need to re-add all of the vehicles manually. This will involve going onto the flyingserver as admin and running:

K3S_SERVER=https://192.168.10.80:6443 ./start_k3s_agent.sh pi 192.168.10.106 clover13

The script will update the admin token given to the vehicle itself.
This script is in the Murmuration project.

[mhl787156]

Then! It would seem that the dashboard does not like this. Either all pods need to be manually restarted, or you can kubectl delete the dashboard deployment file. Downside to that is that it also attempts to delete the kubernetes-dashboard namespace. This may break as the dashboard gets stuck on terminating and you cannot redeploy the dashboard. Therefore to delete the dashboard, you will need to run kubectl proxy in one terminal, then follow the isntructions in this comment: kubernetes/kubernetes#19317 (comment)

Just to clarify for this, we want to run the following locally:

1. Set namespace to kubernetes-dashboard
2. The kubeconfig file is under $KUBECONFIG at `~/.kube/config/k3s.yaml
3. In a seperate terminal run kubectl proxy

So:

kubeconfig=$KUBECONFIG

# Ask which namespace will be delete
echo "Enter Namespace you want to delete:"
read namespace

echo "Get Namespace $namespace"
kubectl --kubeconfig $kubeconfig get ns $namespace -o json > $namespace.json

# Removes the whole "Spec" block of the namespace
echo "Removing spec block"
sed -i -e '/\"spec\"/,/}/ d; /^$/d' $namespace.json

# Push namespace back, will be deleted immediately if already dangling
echo "Send edited json file back to rancher"
curl -k -H "Content-Type: application/json" -X PUT --data-binary @$namespace.json http://127.0.0.1:8001/api/v1/namespaces/$namespace/finalize