Helm: add Liveness/Readiness for stac-proxy deployment

[thenav56]

Reference

• https://github.com/IFRCGo/go-deploy/tree/develop/applications/argocd/staging/applications/montandon-eoapi

Problem

If an error happens during application startup or runtime, the container logs the error but the pod is not restarted automatically.
This leaves the pod in a broken state, and a manual restart is required.

Error log

INFO:     Will watch for changes in these directories: ['/app']
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
INFO:     Started reloader process [1] using StatReload
DEBUG:    Mounted app at /stac
DEBUG:    CORS: credentials enabled with wildcard origins, using origin reflection
INFO:     CORS: handling locally (allow_origins=['*'], allow_methods=['*'], allow_credentials=True)
INFO:     Started server process [8]
INFO:     Waiting for application startup.
DEBUG:    Appending required conformance for collections filter
DEBUG:    Appending required conformance for items filter
INFO:     Running upstream server health checks...
INFO:     Upstream API 'http://montandon-eoapi-stac:8080/' is healthy
ERROR:    Traceback (most recent call last):
  File "/usr/local/lib/python3.13/site-packages/starlette/routing.py", line 694, in lifespan
    async with self.lifespan_context(app) as maybe_state:
               ~~~~~~~~~~~~~~~~~~~~~^^^^^
  File "/usr/local/lib/python3.13/contextlib.py", line 214, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/fastapi/routing.py", line 201, in merged_lifespan
    async with original_context(app) as maybe_original_state:
               ~~~~~~~~~~~~~~~~^^^^^
  File "/usr/local/lib/python3.13/contextlib.py", line 214, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/app/src/stac_auth_proxy/lifespan.py", line 141, in lifespan
    await check_server_healths(
        settings.upstream_url, settings.oidc_discovery_internal_url
    )
  File "/app/src/stac_auth_proxy/lifespan.py", line 24, in check_server_healths
    await check_server_health(url)
  File "/app/src/stac_auth_proxy/lifespan.py", line 47, in check_server_health
    response.raise_for_status()
  File "/usr/local/lib/python3.13/site-packages/httpx/_models.py", line 829, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)

httpx.HTTPStatusError: Server error '502 Bad Gateway' for url 'https://goadmin-stage.ifrc.org/o/.well-known/openid-configuration'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/502

ERROR:    Application startup failed. Exiting.

Note

The logs show "ERROR: Application startup failed. Exiting.", but the container does not actually exit. The pod continues running and appears healthy when checked through the Kubernetes API.

Scenario

stac-auth-proxy runs in Azure AKS. When a node scales up or down, both the stac-auth-proxy pod and its dependency go-api pod are recreated simultaneously if they were running on the node that was scaled down.

As the dependent service goadmin-stage.ifrc.org is temporarily unavailable during that time, the startup health check fails and stac-auth-proxy throws error but doesn't exit.

The pod then stays in this failed state and does not restart automatically. Recovery only happens if:

• the node scales again, or
• someone manually deletes the pod.

Possible Solution

• Exit the container when a startup error occurs so Kubernetes can restart the pod automatically.
• Add proper liveness and readiness probes to ensure Kubernetes can detect unhealthy pods and restart them when needed.

---

@batpad @alukach @pantierra

[alukach]

Thanks for this issue @thenav56! I appreciate the well thought out description. There are a few things that stand out to me:

1. The point of the server health checks is to not start up the auth proxy until upstream services are available. The goal is for the health check to try a few times before giving up, however that doesn't seem to be happening. It seems like we handle connection errors in our server health checks but don't properly handle 502s. That should probably be changed.

   stac-auth-proxy/src/stac_auth_proxy/lifespan.py

   Lines 47 to 50 in 158f507

   	response.raise_for_status()
   	logger.info(f"Upstream API {url!r} is healthy")
   	return
   	except httpx.ConnectError as e:

   We will track this in Health checks don't handle 502s #141
2. I'm surprised to hear that the pod persists when the application exits. It looks like Uvicorn only exits if reload is disabled (code). I think it's reasonable for us to make reload disabled by default and something that could be enabled for local dev.

   stac-auth-proxy/src/stac_auth_proxy/__main__.py

   Line 22 in 158f507

   reload=True,

   We will track this in Disable reload in production #142
3. The proxy ships with /healthz and /healthz/upstream endpoints intended to serve as liveliness and readiness endpoints, respectively. However, it seems like they're not utilized in our helm chart. https://github.com/developmentseed/stac-auth-proxy/blob/158f50756ffb44086eb872b427c46e8101518c50/helm/templates/deployment.yaml We can track that shortcoming in this issue

[thenav56]

Thanks, @alukach, for the quick response.

All points make sense, and I'm glad most of them are already fixed 🙌🏽.
Even with the implementation of points 1 and 2, it should resolve the current issue we have in IFRC cluster.

Let me know when we have a new release, and we can test it out in the ifrcgo staging.

---

Not urgent, but point 3 would address cases where the stac-proxy stops working after startup, which has happened only once for us.

[alukach]

@thenav56 I've manually published v1.0.3-rc2 (after mistakenly making v1.0.3-rc1 off of main). This release is built off of #143. I honestly know very little about Helm/K8s and use Claude Code to put it together, so please review. If all looks good, I'll merge the fix and release as v1.0.3