e2e test reliability

[tijmenbruggeman]

Improving the robustness of the mock api and with that the reliability of the e2e tests.

Added caching for the mock image
I've bumped into the occassional 502 error when fetching the php5.6 image from docker.

#4 [internal] load metadata for docker.io/library/php:5.6-apache
#4 ERROR: failed to authorize: failed to fetch oauth token: unexpected status from POST request to https://auth.docker.io/token: 502 Bad Gateway: error code: 502

There was no healthcheck on the image so the process would just continue and the e2e failed because the mock api was not available. Every push will run 7 versions of the integration test and rebuild the same image 7 times over while nothing had changed. Caching the image would speed up the tests by 20s~ and reduce the chance of running into the 502 hiccup.

Before:

After:

(additional reading: https://www.blacksmith.sh/blog/cache-is-king-a-guide-for-docker-layer-caching-in-github-actions)

Added wait and exit on error
Caches can expire or get evicted so the 502 hiccup can still happen. If that happens we want to know before running the integration tests. I've added a health-check on the docker compose for the mock image and will wait for a succesful exit, otherwise exit on error.

Remove e2e for S3 Offload
Having full e2e tests is nice but this had a lot of dependencies:

• localstack had to be used to mock s3
• additional plugin needed to be installed
• updates or changes could block the pipeline
  I could cache localstack as well but all felt quite a lot. In the end the user base for the plug-in is not high enough to justify the cost of having these dependencies.