Doris 3.0.8 on Kubernetes: File cache memory not released quickly, causing MEM_ALLOC_FAILED and query starvation

[shashankk90]

Description

I am running Apache Doris 3.0.8 on Kubernetes using the operator, and encountering an issue where backend (BE) pods stop executing new queries due to memory pressure caused by OS file cache.

During query execution, Doris loads data into the file cache. However, this cache is not released quickly enough, leading to high memory usage at the OS level. As a result, Doris refuses to execute new queries even though the internal memory tracker shows low usage.

This appears related to how the OS file cache is accounted for in Kubernetes environments (possibly similar to: kubernetes/kubernetes#43916).

---

Error Message

[MEM_ALLOC_FAILED] Create Expr failed because [E11] Allocator sys memory check failed:
Cannot alloc:64, consuming tracker:<Load#Id=cc827453e9894a89-94be3f0322fd9413>,
peak used 0, current used 0, exec node:<>,
process memory used 18.58 GB exceed limit 198.00 GB
or sys available memory 3.55 GB less than low water mark 12.00 GB.

---

Environment

• Doris version: 3.0.8
• Deployment: Kubernetes (via Doris Operator)
• Affected components: Backend (BE) pods

---

Observed Behavior

• Queries trigger file cache growth on BE nodes
• OS-level memory (page cache) increases significantly
• Doris internal memory tracker does not reflect this usage
• sys available memory drops below low water mark
• New queries fail with MEM_ALLOC_FAILED
• Cache is not reclaimed fast enough by the OS

---

Expected Behavior

• File cache should either:
  • Be released fast enough by the OS, or
  • Be accounted for in Doris memory checks appropriately
• Doris should continue executing queries as long as actual process memory usage is within limits

---

Questions

1. Is there a recommended way to configure Doris to ignore or better handle OS page cache pressure in Kubernetes?
2. Are there tunables for:
   • file cache size limits
   • cache eviction aggressiveness
3. Should sys available memory be used in containerized environments, or can this check be relaxed?
4. Any best practices for running Doris BE nodes with large file cache workloads in K8s?

[lsy3993]

we think we need be.INFO and master fe.log if you can reproduce this issue. btw, you can join our Discord(https://discord.gg/JemTbMKj ), we are always online at Discord.