Add documentation for k8s executor with privileged container

[eseliger]

Today, we do not have an optimal way of deploying executors to Kubernetes. We have work arounds for customers to be able to deploy executors.

Done

• Document how to deploy executors to Kubernetes
  • When working this Issue, take Runtime Interface Spec #46175 into consideration (may make things easier)
• Document security expectations

[eseliger]

Potentially also do that for docker compose.

[eseliger]

https://github.com/sourcegraph/sourcegraph/issues/36839
https://github.com/sourcegraph/sourcegraph/issues/36838
sourcegraph/deploy-sourcegraph-docker#826
sourcegraph/deploy-sourcegraph#4140

One thing to find out: K8s 1.24+ doesn’t support docker anymore. So in my understanding it’s not docker-in-docker, it’s containerd that you could access as a privileged container .. and executor and src-cli don’t support contained today.
So.. maybe this would only work on k8s clusters < 1.24, and beyond that we would have to support containerd as a runtime.

[eseliger]

I think this could also be useful for an initial E2E test.

[davejrt]

@eseliger do we actually have the workaround published (without current docs)? I have a rough deployment for an executor working in k8s now we could use that I publish and document

[eseliger]

The only stuff we have on this topics lives in the comments and PRs in these tickets https://github.com/sourcegraph/sourcegraph/issues/42469#issuecomment-1268160825

I want to reiterate that we rely on being able to speak to the underlying docker daemon on the k8s node machines, and that kubernetes by default (and in a bunch of standard deployments, too) does not actually use docker but containerd, which is not a supported backend today afaik.

Hope that helps, and we should be careful how we document this as it is relatively brittle and relies on a bunch of assumptions to the deployment of k8s!

[Piszmog]

Closing as the PR has been merged