Use one proxy per VM

[jodh-intel]

Modify runtime to launch one proxy per VM. This will improve our HA story since there will no longer be a single point of failure on the system (the system-level instance of cc-proxy).

Plan (ordered steps)

• Update the proxy to allow 1 proxy / pod, alongside the system-level systemd cc-proxy service (One proxy per vm proxy#167).
  At this point:
  • multiple instances of cc-proxy will be able to co-exist with the system-level cc-proxy systemd service.
• Update virtcontainers to spawn 1 proxy / pod (One proxy per vm containers/virtcontainers#483).
• Re-vendor the virtcontainers changes into the runtime (proxy: Use one proxy instance per VM #835).
  At this point:
  • the system-level cc-proxy systemd instance won't be being used (although it can continue to run without interfering with the pod-specific cc-proxy instances).
  • we will lose the KSM feature of cc-proxy because since all the proxy requests will be handled by the pod-specific cc-proxy instances and since we don't want multiple proxies fighting over KSM kernel settings, the KSM code will never be run.
• Create a daemon to replace the KSM functionality in cc-proxy ([PROPOSAL] [RFC] KSM throttler proxy#168)
  which is now implemented by https://github.com/kata-containers/ksm-throttler.
• Remove the following features from the proxy:
  • KSM.
  • socket activation.
  • handling code for >1 pods.
    (Remove unused features (socket activation and KSM) proxy#177 for KSM + socket activation).
• Update packaging for the proxy (Package ksm-throttler packaging#198) to do something like:

   $ sudo systemctl stop cc-proxy.socket
   $ sudo systemctl stop cc-proxy.service
   $ sudo rm /lib/systemd/system/cc-proxy.socket
   $ sudo rm /lib/systemd/system/cc-proxy.service
   $ sudo systemctl daemon-reload
  

• Release updated packages for proxy, runtime and the new KSM daemon.
  At this point, there will no longer be a system-level cc-proxy instance.

[jodh-intel]

/cc @sameo, @grahamwhaley, @sboeuf, @dvoytik.

[dvoytik]

Hi @jodh-intel,

Overall IMHO this makes sense.

Btw, I guess clearcontainers/proxy#107 still could be useful with the above plan. Although the PR should be refactored a bit.

EDIT: orthography

[sboeuf]

@dvoytik yes this would still make sense, but with a lower priority. Having one proxy per VM would solve most of our issues. In case of a proxy crash we would only loose one pod.