|
| 1 | +# ECS with Kubernetes |
| 2 | + |
| 3 | +> ⚠️ **Warning** |
| 4 | +> All Kubernetes work done is in a stage of prototype. |
| 5 | +
|
| 6 | +## Kubernetes Cluster |
| 7 | + |
| 8 | +While prototyping we used many Kubernetes clusters, namely [`kind`](https://kind.sigs.k8s.io/), [`minikube`](https://minikube.sigs.k8s.io/docs/) and [`k3s`](https://k3s.io/) |
| 9 | +in both local and remote cluster deployment. We used Openstack for remote deployment. |
| 10 | +Follow the guides at the individual distributions in order to create the desired cluster setup. |
| 11 | +For now we chose `k3s` for most of the activities performed because it is lightweight |
| 12 | +and easily installed distribution which is also [`CNCF`](https://www.cncf.io/training/certification/) certified. |
| 13 | + |
| 14 | +All settings of `k3s` were used as default except one: locked-in-memory size. Use `ulimit -l` to learn |
| 15 | +what is the limit for the current user and `LimitMEMLOCK` inside the k3s systemd service config |
| 16 | +to set it for correct value. Right now the `flp` user has unlimited size (`LimitMEMLOCK=infinity`). |
| 17 | +This config is necessary because even if you are running PODs with the privileged security context |
| 18 | +under user flp, Kubernetes still sets limits according to its internal settings and doesn't |
| 19 | +respect linux settings. |
| 20 | + |
| 21 | +Another setup we expect at this moment to be present at the target nodes |
| 22 | +is ability to run PODs with privileged permissions and also under user `flp`. |
| 23 | +This means that the machine has to have `flp` user setup the same way as |
| 24 | +if you would do the installation with [`o2-flp-setup`](https://alice-flp.docs.cern.ch/Operations/Experts/system-configuration/utils/o2-flp-setup/). |
| 25 | + |
| 26 | +## Running tasks (`KubectlTask`) |
| 27 | + |
| 28 | +ECS is setup to run tasks through Mesos on all required hosts baremetal with active |
| 29 | +task management (see [`ControllableTask`](/executor/executable/controllabletask.go)) |
| 30 | +and OCC gRPC communication. When running docker task through ECS we could easily |
| 31 | +wrap command to be run into the docker container with proper settings |
| 32 | +([see](/docs/running_docker.md)). This is however not possible for Kubernetes |
| 33 | +workloads as the PODs are "hidden" inside the cluster. So we plan |
| 34 | +to deploy our own Task Controller which will connect to and guide |
| 35 | +OCC state machine of required tasks. Thus we need to create custom |
| 36 | +POC way to communicate with Kubernetes cluster from Mesos executor. |
| 37 | + |
| 38 | +The reason why we don't call Kubernetes cluster directly from ECS core |
| 39 | +is that ECS does a lot of heavy lifting while deploying workloads, |
| 40 | +monitoring workloads and by generating a lot of configuration which |
| 41 | +is not trivial to replicate manually. However, if we create some class |
| 42 | +that would be able to deploy one task into the Kubernetes and monitor its |
| 43 | +state we could replicate `ControllableTask` workflow and leave ECS |
| 44 | +mostly intact for now, save a lot of work and focus on prototyping |
| 45 | +Kubernetes operator pattern. |
| 46 | + |
| 47 | +Thus [`KubectlTask`](/executor/executable/kubectltask.go) was created. This class |
| 48 | +is written as a wrapper around `kubectl` utility to manage Kubernetes cluster. |
| 49 | +It is based on following `kubectl` commands: |
| 50 | + |
| 51 | +* `apply` => `kubectl apply -f manifest.yaml` - deploys resource described inside given manifest |
| 52 | +* `delete` => `kubectl delete -f manifest.yaml` - deletes resource from cluster |
| 53 | +* `patch` => `kubectl patch -f exampletask.yaml --type='json' -p='[{"op": "replace", "path": "/spec/state", "value": "running"}]` - changes the state of resource inside cluster |
| 54 | +* `get` => `kubectl get -f manifest.yaml -o jsonpath='{.spec.state}'` - queries exact field of resource (`state` in the example) inside cluster. |
| 55 | + |
| 56 | +These four commands allow us to deploy and monitor status of the deployed |
| 57 | +resource without necessity to interact with it directly. However `KubectlTask` |
| 58 | +expects that resource is the CRD [Task](/control-operator/api/v1alpha1/task_types.go). |
| 59 | + |
| 60 | +In order to activate `KubectlTask` you need to change yaml template |
| 61 | +inside the `ControlWorkflows` directory. Namely: |
| 62 | + |
| 63 | +* add path to the kubectl manifest as the first argument in `.command.arguments` field |
| 64 | +* change `.control.mode` to either `kubectl_direct` or `kubectl_fairmq` |
| 65 | +You can find working template inside `control-operator/ecs-manifests/control-workflows/*_kube.yaml` |
| 66 | + |
| 67 | +Working kubectl manifests are to be found in `control-operator/ecs-manifests/kubernetes-manifests`. |
| 68 | +You can see `*test.yaml` for concrete deployable manifests by `kubectl apply`, the rest |
| 69 | +are the templates with variables to be filled in in a `${var}` format. `KubectlTask` |
| 70 | +fills these variables from env vars. |
0 commit comments