WIP test: optional traefik installation

[shreddedbacon]

General Checklist

• Affected Issues have been mentioned in the Closing issues section
• Documentation has been written/updated
• PR title is ready for inclusion in changelog

Database Migrations

• If your PR contains a database migation, it MUST be the latest in date order alphabetically

Description

This is an experimental branch to see which tests pass/fail when using Traefik instead of ingress-nginx, requires https://github.com/uselagoon/lagoon-charts/tree/traefik. This is not an indication that Traefik is our supported ingress-controller, just verifying alternatives.

Traefik has some support for a fair few ingress-nginx annotations, described here https://doc.traefik.io/traefik/master/reference/routing-configuration/kubernetes/ingress-nginx/

This branch is not using the ingress-nginx provider type, it is using a build-deploy-tool image that has support for traefik built in, and leverages middleware to reproduce some of the basics that Lagoon leveraged from ingress-nginx annotations. It runs a version of aergia-controller that has limited support for idling with traefik too.

What doesn't work in Lagoon (so far)

HSTS Configuration

The build-deploy-tool sets a nginx.ingress.kubernetes.io/configuration-snippet annotation that contains the HSTS header, the Traefik ingress-nginx provider doesn't support this.

Dev route X-Robots-Tag

The build-deploy-tool templating sets a nginx.ingress.kubernetes.io/server-snippet annotation, this is not supported in Traefik. Our nginx images do set this, however an alternative would need to be found for anything else.

Route annotations

Lagoon allows users to set annotations on their routes. If you've set annotations that are specific to ingress-nginx, these will likely not work unless they're listed as supported in Traefik.

Redirect annotations described in the docs are also unsupported.

See Traefik unsupported ingress-nginx annotations for all annotations without support. Edit: only applies if using the provider, which this PR is not.

Idling

Out of the box, aergia-controller does not support Traefik. It is possible to extend it to support Traefik using middlewares, however it does require a few things:

• configuring Traefik with --providers.kubernetescrd.allowcrossnamespace=true to allow the aergia backend to be available
• changes to the build-deploy-tool to provision a middleware CR for errors.

Notes

Traefik ingress-nginx provider

Traefik needs to be configured with the following flags to enable the ingress-nginx support

--experimental.kubernetesingressnginx
--providers.kubernetesingressnginx

However, the ingressClass on existing ingress still needs to be left pointing to whatever ingress-nginx would have created (usually nginx). This means the remote-controller needs to have LAGOON_FEATURE_FLAG_DEFAULT_INGRESS_CLASS=nginx set.

Using this provider though with the nginx ingress class, you aren't able to leverage any other Traefik features like middlewares.

Middlewares

Traefik middlewares offer a potential path out. Things like adding headers etc can mostly be done with middlewares.

Supporting middlewares would likely mean we should create specific lagoon yaml entries for things. A lot of people use the annotations to set permanent redirects, or IP whitelists. This would give us a structure that could potentially apply to other ingress controllers.

Linked

• feat: traefik support build-deploy-tool#472
• uselagoon/lagoon-charts@main...traefik
• feat: traefik support aergia-controller#65

[shreddedbacon]

https://traefik.io/blog/transition-from-ingress-nginx-to-traefik and https://doc.traefik.io/traefik/v3.5/reference/routing-configuration/kubernetes/ingress-nginx/

Update: There is a weird bug with traefik and the way ssl-redirect annotations, a community user has submit a PR to fix waiting for review. There is an image override in the traefik branch of lagoon-charts with an image built from that PR which does indeed solve the issue.

This only matters if using the ingress-nginx provider in traefik

[shreddedbacon]

New charts branch that can install different controllers.