Memory consumption spikes when an api endpoint with file upload capability is hit.

[athujoshi24]

I have an API application, created using connexion framework, which uses FlaskApp configuration. This app exposes an endpoint where users/consumers can upload artifacts(ranging from 2.5 MB to 400 MB) and they will be then upload to an internal storage platform.
The pain point is, when the user selects a larger data file(~320 MB) to upload, the application often runs into OOM issue and eventually user receives 502 status response.
Running dmesg command inside containers show the gunicorn thread being killed by oom_killer

It would be of great help if someone here could guide in avoiding such OOM issues for Flask Applications.

API Specifaction

openapi: 3.0.0
info:
  description: Description
  version: 1.1.1
  title:  API

servers:
  - url: /api/v1
    description: base path for version 1 of the API

components:

  responses:
    BinaryFile:
      description: Content of stored binary file
      content:
        "*/*":
          schema:
            type: string
            format: binary
    BadArgument:
      description: Request Validation failed
      content:
        application/json:
          schema:
            $ref: "#/components/schemas/Error"
    NotFound:
      description: Not Found (e.g. for a given Id)
      content:
        application/json:
          schema:
            $ref: "#/components/schemas/Error"
    InternalServerError:
      description: Internal Server Error
      content:
        application/json:
          schema:
            $ref: "#/components/schemas/Error"
    ServiceUnavailable:
      description: Service Temporary Unavailable
      content:
        application/json:
          schema:
            $ref: "#/components/schemas/Error"

  parameters:
    OrganizationId:
      name: teamId
      in: path
      required: true
      schema:
        type: string
        format: path
        additionalProperties: false
        anyOf:
          - $ref: "#/components/schemas/Id"
          - $ref: "#/components/schemas/TeamId"
    TicketId:
      name: ticketId
      in: path
      required: true
      schema:
        $ref: "#/components/schemas/Id"
    ArtifactDataFileType:
      name: dataType
      in: path
      required: true
      schema:
        $ref: "#/components/schemas/ArtifactDataFileType"

  schemas:
    Id:
      description: Unique identifier of a Object/Document/Entity etc
      type: string
      # On why UUID is used see https://www.cockroachlabs.com/docs/stable/sql-faqs.html#what-are-the-differences-between-uuid-sequences-and-unique_rowid
      format: string
    SubscriptionType:
      description: Subscription type
      type: string
      enum: [small, medium, large]
      nullable: true
    ArtifactDataFileType:
      type: string
      enum: [ artifact ]
    TeamId:
      type: string
      description: Team's unique identifier
    # According to "Problem Details for HTTP APIs" -> https://tools.ietf.org/html/draft-ietf-appsawg-http-problem-00
    Error:
      type: object
      description: An Error object returned if something went wrong
      additionalProperties: true
      properties:
        detail:
          type: string
        status:
          type: integer
        title:
          type: string
        type:
          type: string
    ArtifactBodyWithSubscriptionType:
      type: object
      additionalProperties: false
      properties:
        data:
          type: string
          format: binary
        subscriptionType:
          $ref: "#/components/schemas/SubscriptionType"


paths:
  '/team/{teamId}/ticket/{ticketId}/{dataType}':
    parameters:
      - $ref: "#/components/parameters/OrganizationId"
      - $ref: "#/components/parameters/TicketId"
      - $ref: "#/components/parameters/ArtifactDataFileType"
    post:
      summary: Upload Artifact data file for this Ticket
      x-openapi-router-controller: app.operations.epics
      operationId: upload_data
      requestBody:
        required: true
        content:
          multipart/form-data:
            schema:
              $ref: '#/components/schemas/ArtifactBodyWithSubscriptionType'
      responses:
        '201':
          description: Uploaded
        '400':
          $ref: "#/components/responses/BadArgument"
        '404':
          $ref: "#/components/responses/NotFound"
        '500':
          $ref: "#/components/responses/InternalServerError"
        '503':
          $ref: "#/components/responses/ServiceUnavailable"

Python function

import logging
from http import HTTPStatus
from typing import BinaryIO

from connexion import problem

from app.operations import _not_found
from app.models.ticket import Ticket
from app.storage import Storages


logger = logging.getLogger(__name__)


class Storages:
    """
    Internal platform to store artifacts.
    """
    def __init__(self, data_path):
        self.data_path: str = data_path

    def save(self, data_path: str, data: BinaryIO):
        """
        API call to internal platform to store artifacts.
        """
        pass

    def ticket_path(self, ticket: Ticket, data_type):
        """
        Returns the path where the ticket data should be stored.
        """
        # This is a placeholder implementation.
        # The actual implementation would return a path based on the ticket and data type.
        return f"{self.data_path}/{ticket.id}/{data_type}"


def _get_ticket(params):
    """
    Helper function to retrieve the ticket from the parameters.
    """
    ...
    return


def upload_data(**params):
    # Checking file
    data = params["data"]
    file_name = data.filename
    if not file_name:
        return problem(HTTPStatus.INTERNAL_SERVER_ERROR, "Cannot upload file", "File is not provided")

    # Checking existence of ticket
    ticket = _get_ticket(params)
    if not ticket:
        return _not_found(Ticket, params["ticketId"])

    # Saving file in storage
    data_type = params["dataType"]

    try:
        storage = Storages(data_path="/path/to/storage/on/internal/platform"),
        data_path = storage.ticket_path(ticket, data_type)
        storage.save(data_path, data)
    except Exception as ex:
        logger.error(f"cannot upload file into storage: {ex}", exc_info=True)
        return problem(HTTPStatus.INTERNAL_SERVER_ERROR, "Cannot upload file", "Problems with storage")

    return None, HTTPStatus.CREATED

Pipfile

[[source]]
url = "https://pypi.org/simple/"
verify_ssl = true
name = "pypi"

[requires]
python_version = "3.11"

[packages]
backoff = "~=2.2.0"
boto3 = "~=1.26.0"
connexion = {extras = ["flask", "uvicorn", "swagger-ui"],version = "~=3.2.0"}
gunicorn = {extras = ["gevent"],version = "~=23.0.0"}
aiohttp = "~=3.10.10"
aiohttp_retry = "~=2.8.3"
greenlet = "==3.1.1"
kubernetes = "~=24.0"
psycopg-c = "~=3.2.2"
psycopg-pool = "~=3.2.3"
rfc3339-validator = "~=0.1.0"
shapely = "~=2.1.0"
jsonschema = "~=4.23.0"

Application startup command

python -m gunicorn --worker-class uvicorn.workers.UvicornWorker --workers 5 --bind 0.0.0.0:$PORT --access-logfile - --access-logformat "%(t)s %(h)s %(l)s %(p)s \"%(m)s %(U)s%(q)s\" %(s)s %(b)s %(L)s \"%(f)s\" \"%(a)s\"" app.wsgi:app

Filtered logs

[ERROR] Worker (pid:62) was sent SIGKILL! Perhaps out of memory?
[INFO] Booting worker with pid: 96

System info:

• Application is deployed on Kubernetes.
• Python version 3.11
• Resource allocation:
  • Memory limit 3GB

In a thread regarding same issue on Connexion forum , I received following feedback: spec-first/connexion#2062 (comment)

FWIW I think this is more a Flask question than a Connexion question. I checked the Flask doc about file upload:

https://flask.palletsprojects.com/en/stable/patterns/fileuploads/

There I found this:

So how exactly does Flask handle uploads? Well it will store them in the webserver’s memory if the files are reasonably small, otherwise in a temporary location (as returned by tempfile.gettempdir()). But how do you specify the maximum file size after which an upload is aborted? By default Flask will happily accept file uploads with an unlimited amount of memory, but you can limit that by setting the MAX_CONTENT_LENGTH config key:

I didn't see a way to specify the threshold that Flask uses to decide whether to use a "temporary location" instead of memory.

Not sure this is really helpful, sorry.

What I'm looking for is recommendations on better implementation to avoid spike in Memory consumption.

[ezsouza]

Hey @athujoshi24, been through this exact thing with Flask + large uploads. The main problem is Flask loads the whole file into memory before your function even runs. A couple things that help:

1. Force Flask to write to disk instead of memory

from flask import Flask, Request
from tempfile import TemporaryFile

class StreamingRequest(Request):
    max_content_length = 16 * 1024
    
    def _get_file_stream(self, total_content_length, content_type, filename=None, content_length=None):
        return TemporaryFile('w+b')

app = Flask(__name__)
app.request_class = StreamingRequest
app.config['MAX_CONTENT_LENGTH'] = 450 * 1024 * 1024

2. Read the file in chunks inside your function

def upload_data(**params):
    data = params["data"]
    ticket = _get_ticket(params)
    storage = Storages(data_path="/path/to/storage")
    data_path = storage.ticket_path(ticket, params["dataType"])
    
    CHUNK_SIZE = 1024 * 1024  # 1MB
    with open(data_path, 'wb') as f:
        while True:
            chunk = data.stream.read(CHUNK_SIZE)
            if not chunk:
                break
            f.write(chunk)
    
    return None, HTTPStatus.CREATED

3. Tune Gunicorn

You're running 5 UvicornWorker workers with a 3GB limit. Each worker could load a big upload and then Kubernetes kills it with the OOM killer. Try:

• Drop to 2 workers (each Uvicorn worker already handles concurrent connections)
• Increase the timeout: --timeout 120
• Or use gevent: gunicorn app.wsgi:app --worker-class gevent --workers 2 --timeout 120

If it still hits the limit, bump the pod memory to like 5GB.

[davidism]

The whole file is not loaded into memory. SpooledTemporaryFile is used, which switches to an on disk file after a cutoff, which defaults to 500KiB and can be configured with MAX_FORM_MEMORY_SIZE.

[ezsouza]

Thanks for the correction @davidism! Good to know about SpooledTemporaryFile and MAX_FORM_MEMORY_SIZE.