From 31b3f32a1bee0e1f893e11d9c586e3e28025e6f7 Mon Sep 17 00:00:00 2001 From: Samy OUBOUAZIZ Date: Wed, 3 Dec 2025 14:58:20 +0100 Subject: [PATCH 1/6] docs(srv): add tb on tmpfs-related oom issues --- .../reference-content/containers-sandbox.mdx | 23 ++++++++++++++++ .../troubleshooting/container-oom.mdx | 27 +++++++++++++++++++ .../troubleshooting/index.mdx | 1 + 3 files changed, 51 insertions(+) create mode 100644 pages/serverless-containers/troubleshooting/container-oom.mdx diff --git a/pages/serverless-containers/reference-content/containers-sandbox.mdx b/pages/serverless-containers/reference-content/containers-sandbox.mdx index 858b8a0dd9..83aa26c0ca 100644 --- a/pages/serverless-containers/reference-content/containers-sandbox.mdx +++ b/pages/serverless-containers/reference-content/containers-sandbox.mdx @@ -25,4 +25,27 @@ Sandbox v1 is known to experience clock drift over time. A difference of approxi ## Sandbox v2 Sandbox v2 is a modern isolation environment that relies on [gVisor](https://gvisor.dev/). This option offers faster [cold starts](/serverless-containers/concepts/#cold-start), but only implements a selection of Linux syscalls. + Refer to the [official gVisor documentation](https://gvisor.dev/docs/user_guide/compatibility/linux/amd64/) for a comprehensive list of supported syscalls. + +## Sandbox special directories + +In sandbox v1 and v2, some directories are mounted as `tmpfs` for temporary in-memory storage: + +| Sandbox v1 |Sandbox v2 | +|---|---| +| /dev (max size: 65 536 KB) | /dev (no max size)| +| | /tmp (no max size) | + +In Sandbox v1, the size is limited, and its impact on memory usage is negligible. + +In Sandbox v2, directories such as `/tmp` and `/dev` are stored in memory. This means that any data written to `/tmp` consumes RAM directly. For example, writing a 100 MB file to `/tmp` will increase the container’s memory usage by the same amount. + +Since this usage counts toward the container’s memory limit, excessive writes to `/tmp` can lead to out-of-memory (OOM) errors, potentially triggering a container restart. + +To avoid this, we recommend being cautious when writing temporary files in `/tmp`, especially in long-running or high-throughput workloads. + +Applications that rely heavily on temporary file storage should either: +- Use alternative storage paths within your container +- Increase the container’s memory allocation to accommodate for temporary file usage +- Use ephemeral storage diff --git a/pages/serverless-containers/troubleshooting/container-oom.mdx b/pages/serverless-containers/troubleshooting/container-oom.mdx new file mode 100644 index 0000000000..a32f5e72f2 --- /dev/null +++ b/pages/serverless-containers/troubleshooting/container-oom.mdx @@ -0,0 +1,27 @@ +--- +title: My container runs out of memory (OOM) +description: Troubleshoot issues encountered with Scaleway Serverless Containers running out of memory and restarting. +tags: serverless containers troubleshooting issue error memory ram oom restart crash freeze bsod +dates: + validation: 2025-12-03 + posted: 2025-12-03 +--- + +## Problem + +My container runs out of memory (OOM) and crashes or restarts. + +## Possible cause + +In sandbox v1 and v2, some directories are mounted as `tmpfs` for temporary in-memory storage. In Sandbox v2, directories such as `/tmp` and `/dev` are stored in memory. + +Excessive writes to `/tmp` can fill up the provisioned memory, leading to out-of-memory (OOM) errors, potentially triggering a container restart. + +## Solutions + +- Avoid writing temporary files in `/tmp`. +- Use alternative storage paths within your container +- Increase the container’s memory allocation to accommodate for temporary file usage +- Use ephemeral storage + +Refer to the [dedicated documentation](/serverless-containers/reference-content/containers-sandbox/) for more information on Sandboxes. \ No newline at end of file diff --git a/pages/serverless-containers/troubleshooting/index.mdx b/pages/serverless-containers/troubleshooting/index.mdx index 0604cab83c..d9ca11333b 100644 --- a/pages/serverless-containers/troubleshooting/index.mdx +++ b/pages/serverless-containers/troubleshooting/index.mdx @@ -71,4 +71,5 @@ productIcon: ContainersProductIcon - [I cannot access my container](/serverless-containers/troubleshooting/cannot-access-container/) - [No data show up on Grafana dashboards](/serverless-containers/troubleshooting/no-data-grafana-dashboards/) - [I get errors when using HTTP/1.0](/serverless-containers/troubleshooting/http1-errors/) + - [My container runs out of memory](/serverless-containers/troubleshooting/container-oom/) From c34db40071214b128eb06b410c9258b0bb5197fb Mon Sep 17 00:00:00 2001 From: Samy OUBOUAZIZ Date: Thu, 4 Dec 2025 10:43:48 +0100 Subject: [PATCH 2/6] docs(srv): update --- .../reference-content/containers-sandbox.mdx | 7 +++--- .../troubleshooting/container-oom.mdx | 3 +-- .../reference-content/functions-sandbox.mdx | 23 ++++++++++++++++++- 3 files changed, 26 insertions(+), 7 deletions(-) diff --git a/pages/serverless-containers/reference-content/containers-sandbox.mdx b/pages/serverless-containers/reference-content/containers-sandbox.mdx index 83aa26c0ca..22011e7d6c 100644 --- a/pages/serverless-containers/reference-content/containers-sandbox.mdx +++ b/pages/serverless-containers/reference-content/containers-sandbox.mdx @@ -39,13 +39,12 @@ In sandbox v1 and v2, some directories are mounted as `tmpfs` for temporary in-m In Sandbox v1, the size is limited, and its impact on memory usage is negligible. -In Sandbox v2, directories such as `/tmp` and `/dev` are stored in memory. This means that any data written to `/tmp` consumes RAM directly. For example, writing a 100 MB file to `/tmp` will increase the container’s memory usage by the same amount. +In Sandbox v2, directories such as `/tmp` and `/dev` are stored in memory. This means that any data written to `/tmp` consumes RAM directly. For example, writing a 100 MB file to `/tmp` will increase the container's memory usage by the same amount. -Since this usage counts toward the container’s memory limit, excessive writes to `/tmp` can lead to out-of-memory (OOM) errors, potentially triggering a container restart. +Since this usage counts toward the container's memory limit, excessive writes to `/tmp` can lead to out-of-memory (OOM) errors, potentially triggering a container restart. To avoid this, we recommend being cautious when writing temporary files in `/tmp`, especially in long-running or high-throughput workloads. Applications that rely heavily on temporary file storage should either: - Use alternative storage paths within your container -- Increase the container’s memory allocation to accommodate for temporary file usage -- Use ephemeral storage +- Increase the container's memory allocation to accommodate for temporary file usage diff --git a/pages/serverless-containers/troubleshooting/container-oom.mdx b/pages/serverless-containers/troubleshooting/container-oom.mdx index a32f5e72f2..17ee553910 100644 --- a/pages/serverless-containers/troubleshooting/container-oom.mdx +++ b/pages/serverless-containers/troubleshooting/container-oom.mdx @@ -21,7 +21,6 @@ Excessive writes to `/tmp` can fill up the provisioned memory, leading to out-of - Avoid writing temporary files in `/tmp`. - Use alternative storage paths within your container -- Increase the container’s memory allocation to accommodate for temporary file usage -- Use ephemeral storage +- Increase the container's memory allocation to accommodate for temporary file usage Refer to the [dedicated documentation](/serverless-containers/reference-content/containers-sandbox/) for more information on Sandboxes. \ No newline at end of file diff --git a/pages/serverless-functions/reference-content/functions-sandbox.mdx b/pages/serverless-functions/reference-content/functions-sandbox.mdx index b1d637ce91..b745280b6d 100644 --- a/pages/serverless-functions/reference-content/functions-sandbox.mdx +++ b/pages/serverless-functions/reference-content/functions-sandbox.mdx @@ -25,4 +25,25 @@ Sandbox v1 is known to experience clock drift over time. A difference of approxi ## Sandbox v2 Sandbox v2 is a modern isolation environment that relies on [gVisor](https://gvisor.dev/). This option offers faster [cold starts](/serverless-functions/concepts/#cold-start), but only implements a selection of Linux syscalls. -Refer to the [official gVisor documentation](https://gvisor.dev/docs/user_guide/compatibility/linux/amd64/) for a comprehensive list of supported syscalls. \ No newline at end of file +Refer to the [official gVisor documentation](https://gvisor.dev/docs/user_guide/compatibility/linux/amd64/) for a comprehensive list of supported syscalls. + +## Sandbox special directories + +In sandbox v1 and v2, some directories are mounted as `tmpfs` for temporary in-memory storage: + +| Sandbox v1 |Sandbox v2 | +|---|---| +| /dev (max size: 65 536 KB) | /dev (no max size)| +| | /tmp (no max size) | + +In Sandbox v1, the size is limited, and its impact on memory usage is negligible. + +In Sandbox v2, directories such as `/tmp` and `/dev` are stored in memory. This means that any data written to `/tmp` consumes RAM directly. For example, writing a 100 MB file to `/tmp` will increase the function's memory usage by the same amount. + +Since this usage counts toward the function's memory limit, excessive writes to `/tmp` can lead to out-of-memory (OOM) errors, potentially triggering a function restart. + +To avoid this, we recommend being cautious when writing temporary files in `/tmp`, especially in long-running or high-throughput workloads. + +Applications that rely heavily on temporary file storage should either: +- Use alternative storage paths within your function +- Increase the function's memory allocation to accommodate for temporary file usage From d21d2c0b74362564696265a653c3b3531b9554c9 Mon Sep 17 00:00:00 2001 From: Samy OUBOUAZIZ Date: Thu, 4 Dec 2025 10:44:53 +0100 Subject: [PATCH 3/6] docs(srv): update --- pages/serverless-containers/troubleshooting/container-oom.mdx | 2 +- .../reference-content/functions-sandbox.mdx | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/pages/serverless-containers/troubleshooting/container-oom.mdx b/pages/serverless-containers/troubleshooting/container-oom.mdx index 17ee553910..d9406776eb 100644 --- a/pages/serverless-containers/troubleshooting/container-oom.mdx +++ b/pages/serverless-containers/troubleshooting/container-oom.mdx @@ -13,7 +13,7 @@ My container runs out of memory (OOM) and crashes or restarts. ## Possible cause -In sandbox v1 and v2, some directories are mounted as `tmpfs` for temporary in-memory storage. In Sandbox v2, directories such as `/tmp` and `/dev` are stored in memory. +In Sandbox v1 and v2, some directories are mounted as `tmpfs` for temporary in-memory storage. In Sandbox v2, directories such as `/tmp` and `/dev` are stored in memory. Excessive writes to `/tmp` can fill up the provisioned memory, leading to out-of-memory (OOM) errors, potentially triggering a container restart. diff --git a/pages/serverless-functions/reference-content/functions-sandbox.mdx b/pages/serverless-functions/reference-content/functions-sandbox.mdx index b745280b6d..2097ad7dd5 100644 --- a/pages/serverless-functions/reference-content/functions-sandbox.mdx +++ b/pages/serverless-functions/reference-content/functions-sandbox.mdx @@ -29,7 +29,7 @@ Refer to the [official gVisor documentation](https://gvisor.dev/docs/user_guide/ ## Sandbox special directories -In sandbox v1 and v2, some directories are mounted as `tmpfs` for temporary in-memory storage: +In Sandbox v1 and v2, some directories are mounted as `tmpfs` for temporary in-memory storage: | Sandbox v1 |Sandbox v2 | |---|---| From 4745a155721b375ff2d3f7c8372d1664bcba8965 Mon Sep 17 00:00:00 2001 From: Samy OUBOUAZIZ Date: Thu, 4 Dec 2025 13:33:06 +0100 Subject: [PATCH 4/6] docs(srv): update --- .../troubleshooting/index.mdx | 2 ++ .../reference-content/jobs-limitations.mdx | 12 ++++++++++ .../serverless-jobs/troubleshooting/index.mdx | 9 +++---- .../troubleshooting/job-oom.mdx | 24 +++++++++++++++++++ 4 files changed, 43 insertions(+), 4 deletions(-) create mode 100644 pages/serverless-jobs/troubleshooting/job-oom.mdx diff --git a/pages/serverless-functions/troubleshooting/index.mdx b/pages/serverless-functions/troubleshooting/index.mdx index 0d93a615f6..2d433c7ffd 100644 --- a/pages/serverless-functions/troubleshooting/index.mdx +++ b/pages/serverless-functions/troubleshooting/index.mdx @@ -70,4 +70,6 @@ productIcon: FunctionsProductIcon - [My function fails after trying too many times](/serverless-functions/troubleshooting/too-many-retries) - [I am experiencing clock drift with my Serverless Functions](/serverless-functions/troubleshooting/function-clock-drift) - [I get errors when using HTTP/1.0](/serverless-functions/troubleshooting/http1-errors/) + - [My function runs out of memory](/serverless-containers/troubleshooting/function-oom/) + diff --git a/pages/serverless-jobs/reference-content/jobs-limitations.mdx b/pages/serverless-jobs/reference-content/jobs-limitations.mdx index dbe708beb1..fc8252ab38 100644 --- a/pages/serverless-jobs/reference-content/jobs-limitations.mdx +++ b/pages/serverless-jobs/reference-content/jobs-limitations.mdx @@ -70,3 +70,15 @@ We recommend keeping your image size **below 2 GB** to ensure faster deployment. * Minimize layers and unnecessary dependencies * Clean up temp files and cache (e.g., `rm -rf /var/lib/apt/lists/*` in Debian-based images) * Use multi-stage builds to keep only runtime essentials. [Example](https://github.com/scaleway/serverless-examples/blob/a7a3b5fc3427ec0f1391aaa78957f91af266867c/containers/rust-hello-world/Dockerfile#L14) + +## gVisor sandbox limitation + +Jobs uses [gVisor](https://gvisor.dev/) as a sandboxing solution to isolate the underlying client containers mutualized on the same host. Directories such as `/tmp` and `/dev` are stored in memory. This means that any data written to `/tmp` consumes memory directly. For example, writing a 100 MB file to `/tmp` will increase the container's memory usage by the same amount. + +Since this usage counts toward the job's memory limit, excessive writes to `/tmp` can lead to out-of-memory (OOM) errors, potentially triggering a container restart. + +To avoid this, we recommend being cautious when writing temporary files in `/tmp`, especially in long-running or high-throughput workloads. + +Jobs that rely heavily on temporary in-memory storage should either: +- Use alternative storage paths within the job's file system; +- Increase the jobs's memory limit if it is necessary to use the temporary in-memory storage in the client use case. \ No newline at end of file diff --git a/pages/serverless-jobs/troubleshooting/index.mdx b/pages/serverless-jobs/troubleshooting/index.mdx index 4bc66a033c..22f49887d7 100644 --- a/pages/serverless-jobs/troubleshooting/index.mdx +++ b/pages/serverless-jobs/troubleshooting/index.mdx @@ -49,7 +49,8 @@ productIcon: ServerlessJobsProductIcon - [My secrets and variables are not injected properly in my job](/serverless-jobs/troubleshooting/cannot-inject-secret-variable) - [I cannot retrieve an external image for my job](/serverless-jobs/troubleshooting/cannot-retrieve-external-image) -- [My Job run is in an error state](/serverless-jobs/troubleshooting/job-in-error-state) -- [Missing metrics for my Job Run](/serverless-jobs/troubleshooting/missing-metrics) - -- [Job startup command fails](/serverless-jobs/troubleshooting/job-startup-command-fails/) \ No newline at end of file +- [My Job run is in an error state](/serverless-jobs/troubleshooting/job-in-error-state/) +- [Missing metrics for my Job Run](/serverless-jobs/troubleshooting/missing-metrics/) +- [Job startup command fails](/serverless-jobs/troubleshooting/job-startup-command-fails/) +- [Job runs out of memory](/serverless-jobs/troubleshooting/job-oom/) + \ No newline at end of file diff --git a/pages/serverless-jobs/troubleshooting/job-oom.mdx b/pages/serverless-jobs/troubleshooting/job-oom.mdx new file mode 100644 index 0000000000..0b56e5d052 --- /dev/null +++ b/pages/serverless-jobs/troubleshooting/job-oom.mdx @@ -0,0 +1,24 @@ +--- +title: My job runs out of memory (OOM) +description: Troubleshoot issues encountered with Scaleway Serverless Jobs running out of memory and restarting. +tags: serverless jobs troubleshooting issue error memory ram oom restart crash freeze bsod +dates: + validation: 2025-12-03 + posted: 2025-12-03 +--- + +## Problem + +My job runs out of memory (OOM) and crashes or restarts. + +## Possible cause + +Jobs uses [gVisor](https://gvisor.dev/) as a sandboxing solution to isolate the underlying client containers mutualized on the same host. Directories such as `/tmp` and `/dev` are stored in memory. This means that any data written to `/tmp` consumes memory directly. For example, writing a 100 MB file to `/tmp` will increase the container's memory usage by the same amount. + +Excessive writes to `/tmp` can fill up the provisioned memory, leading to out-of-memory (OOM) errors, potentially triggering a job restart. + +## Solutions + +- Avoid writing temporary files in `/tmp`. +- Use alternative storage paths within your job +- Increase the job's memory allocation to accommodate for temporary file usage From 7337d3ae8697ec7a0ecdf64231c4dc8a5f0935d3 Mon Sep 17 00:00:00 2001 From: Samy OUBOUAZIZ Date: Thu, 4 Dec 2025 14:02:57 +0100 Subject: [PATCH 5/6] docs(srv): update --- .../troubleshooting/function-oom.mdx | 26 +++++++++++++++++++ 1 file changed, 26 insertions(+) create mode 100644 pages/serverless-functions/troubleshooting/function-oom.mdx diff --git a/pages/serverless-functions/troubleshooting/function-oom.mdx b/pages/serverless-functions/troubleshooting/function-oom.mdx new file mode 100644 index 0000000000..c4ca754002 --- /dev/null +++ b/pages/serverless-functions/troubleshooting/function-oom.mdx @@ -0,0 +1,26 @@ +--- +title: My function runs out of memory (OOM) +description: Troubleshoot issues encountered with Scaleway Serverless functions running out of memory and restarting. +tags: serverless functions troubleshooting issue error memory ram oom restart crash freeze bsod +dates: + validation: 2025-12-03 + posted: 2025-12-03 +--- + +## Problem + +My function runs out of memory (OOM) and crashes or restarts. + +## Possible cause + +In sandbox v1 and v2, some directories are mounted as `tmpfs` for temporary in-memory storage. In Sandbox v2, directories such as `/tmp` and `/dev` are stored in memory. + +Excessive writes to `/tmp` can fill up the provisioned memory, leading to out-of-memory (OOM) errors, potentially triggering a function restart. + +## Solutions + +- Avoid writing temporary files in `/tmp`. +- Use alternative storage paths within your function +- Increase the function's memory allocation to accommodate for temporary file usage + +Refer to the [dedicated documentation](/serverless-functions/reference-content/functions-sandbox/) for more information on Sandboxes. \ No newline at end of file From 024b28ee9b69c7eebbbc410fcc8bd7668fc3e3fc Mon Sep 17 00:00:00 2001 From: Samy OUBOUAZIZ Date: Thu, 4 Dec 2025 14:03:45 +0100 Subject: [PATCH 6/6] docs(srv): update --- pages/serverless-jobs/troubleshooting/index.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pages/serverless-jobs/troubleshooting/index.mdx b/pages/serverless-jobs/troubleshooting/index.mdx index 22f49887d7..4451227fa9 100644 --- a/pages/serverless-jobs/troubleshooting/index.mdx +++ b/pages/serverless-jobs/troubleshooting/index.mdx @@ -52,5 +52,5 @@ productIcon: ServerlessJobsProductIcon - [My Job run is in an error state](/serverless-jobs/troubleshooting/job-in-error-state/) - [Missing metrics for my Job Run](/serverless-jobs/troubleshooting/missing-metrics/) - [Job startup command fails](/serverless-jobs/troubleshooting/job-startup-command-fails/) -- [Job runs out of memory](/serverless-jobs/troubleshooting/job-oom/) +- [My job runs out of memory](/serverless-jobs/troubleshooting/job-oom/) \ No newline at end of file