Generate per-backend directors for multi-endpoint backends

[jensens]

Problem

When a backend spec has multiple endpoints (e.g. a Kubernetes Service with multiple pods), the generator creates individual backend definitions like thumbor_0, thumbor_1, etc. But there is no director that groups them under the backend name.

VCL snippets cannot reference thumbor.backend() because no such director exists. Users must hardcode thumbor_0 which is fragile and breaks with endpoint changes.

Current behavior

backend thumbor_0 { .host = "10.42.5.171"; .port = "8888"; }
backend thumbor_1 { .host = "10.42.12.89"; .port = "8888"; }

No director for thumbor.

Expected behavior

backend thumbor_0 { .host = "10.42.5.171"; .port = "8888"; }
backend thumbor_1 { .host = "10.42.12.89"; .port = "8888"; }

sub vcl_init {
    new thumbor = directors.round_robin();
    thumbor.add_backend(thumbor_0);
    thumbor.add_backend(thumbor_1);
}

VCL snippets can then use thumbor.backend().

Architecture reference

This aligns with the existing shard director pattern in vcl_init.vcl.tmpl but extends it to per-backend directors.

[jensens]

Observed on operator 0.3.2 in a prod setup: serviceRef: {name: …} currently generates a single backend with the Service DNS as .host, not one backend per endpoint. Concretely, a Plone backend with 6 pods behind plone-backend-service renders as:

backend plone_backend_0 {
    .host = "aaf-deployment-plone-backend-service-c89174ef";
    .port = "8080";
    .probe = { .url = "/ok"; .interval = 5s; .timeout = 2s; .window = 5; .threshold = 3; }
}

Only one _0 backend, Service DNS as host. Load balancing is delegated to kube-proxy under the covers, which has two downsides vs. per-endpoint backends:

1. Per-pod health is invisible to Varnish. The probe hits whichever backend kube-proxy picks that moment, so a single sick pod is averaged out and Varnish keeps considering plone_backend_0 healthy. Requests that get routed to the sick pod by kube-proxy see connection errors / timeouts / 5xx that Varnish can't fail-over because from its POV there's only one backend to retry.

2. Rolling deploys produce brief 503 windows. During a rollout there's a race between a pod being marked Terminating and kube-proxy removing its IP from the Service's endpoint set. In that window, new connections can land on a pod that's already gone, and Varnish has no alternate backend to route to.

My read of this issue's title ("Generate per-backend directors for multi-endpoint backends") is that it's about the second step — creating the director that groups plone_backend_0 … _N once they exist. But in our case step 1 (expanding serviceRef into per-pod backends at all) doesn't happen yet — only Service DNS is used.

Is the endpoint-expansion step in scope of this issue, or would a separate issue / feature request cover that? Happy to contribute either way if the direction is agreed.

For reference, Varnish backend spec already has everything needed to make the per-pod model work well (probes are per-backend, maxConnections is per-backend), and a shard director with healthy = CHOSEN gracefully skips individual sick pods.