diff --git a/pages/serverless-containers/reference-content/containers-sandbox.mdx b/pages/serverless-containers/reference-content/containers-sandbox.mdx index 858b8a0dd9..22011e7d6c 100644 --- a/pages/serverless-containers/reference-content/containers-sandbox.mdx +++ b/pages/serverless-containers/reference-content/containers-sandbox.mdx @@ -25,4 +25,26 @@ Sandbox v1 is known to experience clock drift over time. A difference of approxi ## Sandbox v2 Sandbox v2 is a modern isolation environment that relies on [gVisor](https://gvisor.dev/). This option offers faster [cold starts](/serverless-containers/concepts/#cold-start), but only implements a selection of Linux syscalls. + Refer to the [official gVisor documentation](https://gvisor.dev/docs/user_guide/compatibility/linux/amd64/) for a comprehensive list of supported syscalls. + +## Sandbox special directories + +In sandbox v1 and v2, some directories are mounted as `tmpfs` for temporary in-memory storage: + +| Sandbox v1 |Sandbox v2 | +|---|---| +| /dev (max size: 65 536 KB) | /dev (no max size)| +| | /tmp (no max size) | + +In Sandbox v1, the size is limited, and its impact on memory usage is negligible. + +In Sandbox v2, directories such as `/tmp` and `/dev` are stored in memory. This means that any data written to `/tmp` consumes RAM directly. For example, writing a 100 MB file to `/tmp` will increase the container's memory usage by the same amount. + +Since this usage counts toward the container's memory limit, excessive writes to `/tmp` can lead to out-of-memory (OOM) errors, potentially triggering a container restart. + +To avoid this, we recommend being cautious when writing temporary files in `/tmp`, especially in long-running or high-throughput workloads. + +Applications that rely heavily on temporary file storage should either: +- Use alternative storage paths within your container +- Increase the container's memory allocation to accommodate for temporary file usage diff --git a/pages/serverless-containers/troubleshooting/container-oom.mdx b/pages/serverless-containers/troubleshooting/container-oom.mdx new file mode 100644 index 0000000000..d9406776eb --- /dev/null +++ b/pages/serverless-containers/troubleshooting/container-oom.mdx @@ -0,0 +1,26 @@ +--- +title: My container runs out of memory (OOM) +description: Troubleshoot issues encountered with Scaleway Serverless Containers running out of memory and restarting. +tags: serverless containers troubleshooting issue error memory ram oom restart crash freeze bsod +dates: + validation: 2025-12-03 + posted: 2025-12-03 +--- + +## Problem + +My container runs out of memory (OOM) and crashes or restarts. + +## Possible cause + +In Sandbox v1 and v2, some directories are mounted as `tmpfs` for temporary in-memory storage. In Sandbox v2, directories such as `/tmp` and `/dev` are stored in memory. + +Excessive writes to `/tmp` can fill up the provisioned memory, leading to out-of-memory (OOM) errors, potentially triggering a container restart. + +## Solutions + +- Avoid writing temporary files in `/tmp`. +- Use alternative storage paths within your container +- Increase the container's memory allocation to accommodate for temporary file usage + +Refer to the [dedicated documentation](/serverless-containers/reference-content/containers-sandbox/) for more information on Sandboxes. \ No newline at end of file diff --git a/pages/serverless-containers/troubleshooting/index.mdx b/pages/serverless-containers/troubleshooting/index.mdx index 0604cab83c..d9ca11333b 100644 --- a/pages/serverless-containers/troubleshooting/index.mdx +++ b/pages/serverless-containers/troubleshooting/index.mdx @@ -71,4 +71,5 @@ productIcon: ContainersProductIcon - [I cannot access my container](/serverless-containers/troubleshooting/cannot-access-container/) - [No data show up on Grafana dashboards](/serverless-containers/troubleshooting/no-data-grafana-dashboards/) - [I get errors when using HTTP/1.0](/serverless-containers/troubleshooting/http1-errors/) + - [My container runs out of memory](/serverless-containers/troubleshooting/container-oom/) diff --git a/pages/serverless-functions/reference-content/functions-sandbox.mdx b/pages/serverless-functions/reference-content/functions-sandbox.mdx index b1d637ce91..2097ad7dd5 100644 --- a/pages/serverless-functions/reference-content/functions-sandbox.mdx +++ b/pages/serverless-functions/reference-content/functions-sandbox.mdx @@ -25,4 +25,25 @@ Sandbox v1 is known to experience clock drift over time. A difference of approxi ## Sandbox v2 Sandbox v2 is a modern isolation environment that relies on [gVisor](https://gvisor.dev/). This option offers faster [cold starts](/serverless-functions/concepts/#cold-start), but only implements a selection of Linux syscalls. -Refer to the [official gVisor documentation](https://gvisor.dev/docs/user_guide/compatibility/linux/amd64/) for a comprehensive list of supported syscalls. \ No newline at end of file +Refer to the [official gVisor documentation](https://gvisor.dev/docs/user_guide/compatibility/linux/amd64/) for a comprehensive list of supported syscalls. + +## Sandbox special directories + +In Sandbox v1 and v2, some directories are mounted as `tmpfs` for temporary in-memory storage: + +| Sandbox v1 |Sandbox v2 | +|---|---| +| /dev (max size: 65 536 KB) | /dev (no max size)| +| | /tmp (no max size) | + +In Sandbox v1, the size is limited, and its impact on memory usage is negligible. + +In Sandbox v2, directories such as `/tmp` and `/dev` are stored in memory. This means that any data written to `/tmp` consumes RAM directly. For example, writing a 100 MB file to `/tmp` will increase the function's memory usage by the same amount. + +Since this usage counts toward the function's memory limit, excessive writes to `/tmp` can lead to out-of-memory (OOM) errors, potentially triggering a function restart. + +To avoid this, we recommend being cautious when writing temporary files in `/tmp`, especially in long-running or high-throughput workloads. + +Applications that rely heavily on temporary file storage should either: +- Use alternative storage paths within your function +- Increase the function's memory allocation to accommodate for temporary file usage diff --git a/pages/serverless-functions/troubleshooting/function-oom.mdx b/pages/serverless-functions/troubleshooting/function-oom.mdx new file mode 100644 index 0000000000..c4ca754002 --- /dev/null +++ b/pages/serverless-functions/troubleshooting/function-oom.mdx @@ -0,0 +1,26 @@ +--- +title: My function runs out of memory (OOM) +description: Troubleshoot issues encountered with Scaleway Serverless functions running out of memory and restarting. +tags: serverless functions troubleshooting issue error memory ram oom restart crash freeze bsod +dates: + validation: 2025-12-03 + posted: 2025-12-03 +--- + +## Problem + +My function runs out of memory (OOM) and crashes or restarts. + +## Possible cause + +In sandbox v1 and v2, some directories are mounted as `tmpfs` for temporary in-memory storage. In Sandbox v2, directories such as `/tmp` and `/dev` are stored in memory. + +Excessive writes to `/tmp` can fill up the provisioned memory, leading to out-of-memory (OOM) errors, potentially triggering a function restart. + +## Solutions + +- Avoid writing temporary files in `/tmp`. +- Use alternative storage paths within your function +- Increase the function's memory allocation to accommodate for temporary file usage + +Refer to the [dedicated documentation](/serverless-functions/reference-content/functions-sandbox/) for more information on Sandboxes. \ No newline at end of file diff --git a/pages/serverless-functions/troubleshooting/index.mdx b/pages/serverless-functions/troubleshooting/index.mdx index 0d93a615f6..2d433c7ffd 100644 --- a/pages/serverless-functions/troubleshooting/index.mdx +++ b/pages/serverless-functions/troubleshooting/index.mdx @@ -70,4 +70,6 @@ productIcon: FunctionsProductIcon - [My function fails after trying too many times](/serverless-functions/troubleshooting/too-many-retries) - [I am experiencing clock drift with my Serverless Functions](/serverless-functions/troubleshooting/function-clock-drift) - [I get errors when using HTTP/1.0](/serverless-functions/troubleshooting/http1-errors/) + - [My function runs out of memory](/serverless-containers/troubleshooting/function-oom/) + diff --git a/pages/serverless-jobs/reference-content/jobs-limitations.mdx b/pages/serverless-jobs/reference-content/jobs-limitations.mdx index dbe708beb1..fc8252ab38 100644 --- a/pages/serverless-jobs/reference-content/jobs-limitations.mdx +++ b/pages/serverless-jobs/reference-content/jobs-limitations.mdx @@ -70,3 +70,15 @@ We recommend keeping your image size **below 2 GB** to ensure faster deployment. * Minimize layers and unnecessary dependencies * Clean up temp files and cache (e.g., `rm -rf /var/lib/apt/lists/*` in Debian-based images) * Use multi-stage builds to keep only runtime essentials. [Example](https://github.com/scaleway/serverless-examples/blob/a7a3b5fc3427ec0f1391aaa78957f91af266867c/containers/rust-hello-world/Dockerfile#L14) + +## gVisor sandbox limitation + +Jobs uses [gVisor](https://gvisor.dev/) as a sandboxing solution to isolate the underlying client containers mutualized on the same host. Directories such as `/tmp` and `/dev` are stored in memory. This means that any data written to `/tmp` consumes memory directly. For example, writing a 100 MB file to `/tmp` will increase the container's memory usage by the same amount. + +Since this usage counts toward the job's memory limit, excessive writes to `/tmp` can lead to out-of-memory (OOM) errors, potentially triggering a container restart. + +To avoid this, we recommend being cautious when writing temporary files in `/tmp`, especially in long-running or high-throughput workloads. + +Jobs that rely heavily on temporary in-memory storage should either: +- Use alternative storage paths within the job's file system; +- Increase the jobs's memory limit if it is necessary to use the temporary in-memory storage in the client use case. \ No newline at end of file diff --git a/pages/serverless-jobs/troubleshooting/index.mdx b/pages/serverless-jobs/troubleshooting/index.mdx index 4bc66a033c..4451227fa9 100644 --- a/pages/serverless-jobs/troubleshooting/index.mdx +++ b/pages/serverless-jobs/troubleshooting/index.mdx @@ -49,7 +49,8 @@ productIcon: ServerlessJobsProductIcon - [My secrets and variables are not injected properly in my job](/serverless-jobs/troubleshooting/cannot-inject-secret-variable) - [I cannot retrieve an external image for my job](/serverless-jobs/troubleshooting/cannot-retrieve-external-image) -- [My Job run is in an error state](/serverless-jobs/troubleshooting/job-in-error-state) -- [Missing metrics for my Job Run](/serverless-jobs/troubleshooting/missing-metrics) - -- [Job startup command fails](/serverless-jobs/troubleshooting/job-startup-command-fails/) \ No newline at end of file +- [My Job run is in an error state](/serverless-jobs/troubleshooting/job-in-error-state/) +- [Missing metrics for my Job Run](/serverless-jobs/troubleshooting/missing-metrics/) +- [Job startup command fails](/serverless-jobs/troubleshooting/job-startup-command-fails/) +- [My job runs out of memory](/serverless-jobs/troubleshooting/job-oom/) + \ No newline at end of file diff --git a/pages/serverless-jobs/troubleshooting/job-oom.mdx b/pages/serverless-jobs/troubleshooting/job-oom.mdx new file mode 100644 index 0000000000..0b56e5d052 --- /dev/null +++ b/pages/serverless-jobs/troubleshooting/job-oom.mdx @@ -0,0 +1,24 @@ +--- +title: My job runs out of memory (OOM) +description: Troubleshoot issues encountered with Scaleway Serverless Jobs running out of memory and restarting. +tags: serverless jobs troubleshooting issue error memory ram oom restart crash freeze bsod +dates: + validation: 2025-12-03 + posted: 2025-12-03 +--- + +## Problem + +My job runs out of memory (OOM) and crashes or restarts. + +## Possible cause + +Jobs uses [gVisor](https://gvisor.dev/) as a sandboxing solution to isolate the underlying client containers mutualized on the same host. Directories such as `/tmp` and `/dev` are stored in memory. This means that any data written to `/tmp` consumes memory directly. For example, writing a 100 MB file to `/tmp` will increase the container's memory usage by the same amount. + +Excessive writes to `/tmp` can fill up the provisioned memory, leading to out-of-memory (OOM) errors, potentially triggering a job restart. + +## Solutions + +- Avoid writing temporary files in `/tmp`. +- Use alternative storage paths within your job +- Increase the job's memory allocation to accommodate for temporary file usage