At the surface the GitLab runner and the GitHub runner look quite different, but there may be a way to add GitLab support to Forrest without too many hacks and without sacrificing features like Docker support and without having to run the runner client outside of the VMs (if these two details were not relevant to us we could build something using the custom executor).
The GitLab runner polls for jobs to run by sending POST requests to the (undocumented) /api/v4/jobs/request GitLab API endpoint:
$ gitlab-runner run-single --url "http://localhost:8080" --token "glrt-GNma-AAAAAAAAAAAAAAA" --executor "shell"
…
$ nc -l 8080
POST /api/v4/jobs/request HTTP/1.1
Host: localhost:8080
User-Agent: gitlab-runner development version (HEAD; go1.22.6; linux/arm)
Content-Length: 847
Accept: application/json
Content-Type: application/json
Private-Token: glrt-GNma-AAAAAAAAAAAAAAA
Accept-Encoding: gzip
{"info":{"name":"gitlab-runner","version":"development version","revision":"HEAD","platform":"linux","architecture":"arm","executor":"shell","shell":"bash","features":{"variables":true,"image":false,"services":false,"artifacts":true,"cache":true,"fallback_cache_keys":true,"shared":true,"upload_multiple_artifacts":true,"upload_raw_artifacts":true,"session":true,"terminal":true,"refspecs":true,"masking":true,"proxy":false,"raw_variables":true,"artifacts_exclude":true,"multi_build_steps":true,"trace_reset":true,"trace_checksum":true,"trace_size":true,"vault_secrets":true,"cancelable":true,"return_exit_code":true,"service_variables":false,"service_multiple_aliases":false,"image_executor_opts":false,"service_executor_opts":false,"cancel_gracefully":true},"config":{"gpus":""}},"token":"glrt-GNma-AAAAAAAAAAAAAAA","system_id":"s_f7b6304c4e01"}
When no job is pending the API will respond with 204 No Content, but if a job is pending it will assign the job to the runner and will send back a json-formatted job definition. The definition also contains a job-token that will be used for all further communication.
This means we can:
- Pretend to be a GitLab runner and poll the API endpoint mentioned above (when we have the capacity to run another job of the configured type).
- Spawn a VM if a job is available.
- Provide a HTTP server that will respond to
POST requests to /api/v4/jobs/request with the job definition in 1). It will also have to proxy some other API requests to the actual GitLab instance, since job status updates are sent that way (git clone etc. are however not. The correct URLs are included in the job definition).
This proxy can run either inside the VM or as part of the Forrest process.
I've stumbled upon this approach thanks to the collabora/gitlab-runner-rs project, which allows building custom GitLab runners like the collabora/lava-gitlab-runner. Most of the features there are however out of scope for us, since we actually want to use the official GitLab runner inside of the VMs.
At the surface the GitLab runner and the GitHub runner look quite different, but there may be a way to add GitLab support to Forrest without too many hacks and without sacrificing features like Docker support and without having to run the runner client outside of the VMs (if these two details were not relevant to us we could build something using the custom executor).
The GitLab runner polls for jobs to run by sending
POSTrequests to the (undocumented)/api/v4/jobs/requestGitLab API endpoint:When no job is pending the API will respond with
204 No Content, but if a job is pending it will assign the job to the runner and will send back a json-formatted job definition. The definition also contains a job-token that will be used for all further communication.This means we can:
POSTrequests to/api/v4/jobs/requestwith the job definition in 1). It will also have to proxy some other API requests to the actual GitLab instance, since job status updates are sent that way (git clone etc. are however not. The correct URLs are included in the job definition).This proxy can run either inside the VM or as part of the Forrest process.
I've stumbled upon this approach thanks to the collabora/gitlab-runner-rs project, which allows building custom GitLab runners like the collabora/lava-gitlab-runner. Most of the features there are however out of scope for us, since we actually want to use the official GitLab runner inside of the VMs.