Fix API server memory leak: bound DBDagBag version cache with LRU eviction

[dheerajturaga]

DBDagBag._dags is an unbounded in-memory cache causing steady memory
growth in the API server.

DBDagBag was designed for the scheduler, which works with a bounded set
of currently-active DAG versions. As an API server singleton, it is exposed to
the full history of DAG versions in the database with no bound on how
many it will cache

Replace the plain dict in DBDagBag._dags with a bounded OrderedDict-based
LRU cache. In long-running API server processes, every unique dag_version_id
accessed is inserted and never evicted, causing unbounded RSS growth (observed:
9.4 GiB after 7 days with ~70k DAG versions in DB).

The cache is now capped at 512 entries by default (configurable via
core.max_dag_version_cache_size). Cache hits promote the entry to MRU so
frequently-accessed versions are retained over stale historical ones.

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)
  ClaudeCode

[shivaam]

Nice. Seems like a real production bug. A few thoughts:

1. Default of 512 may be too low. The scheduler processes all active DAGs every cycle. With 1000+ DAGs, a 512 cache means constant eviction and re-fetching from the DB on every loop. The API server's Execution API also serves worker requests for every task state transition, so it can accumulate entries fast too. Consider starting higher (2048+) and letting people tune down — it's easier to reduce a known number than to discover you need to increase one you didn't know existed.
2. A single config for both scheduler and API server may not be ideal. The scheduler's working set is bounded (latest version per active DAG) and performance-sensitive — it needs a cache big enough to hold all active DAGs. There are no metrics for the cache which will also cause problems in debugging