Materialize BLOBs in SQLAlchemy fetch path (#58)#90
Open
Conversation
firebird-driver returns BlobReader objects for BLOBs larger than stream_blob_threshold (default 64 KiB), but SQLAlchemy fully consumes a cursor and closes it before its result processors run -- which closes the BlobReader objects too, leaving result rows unreadable. Even on a still-open cursor, SQLAlchemy's LargeBinary result processor calls bytes(value), which iterates a binary BlobReader and crashes with "Can't read line from binary BLOB". Override stream_blob_threshold to sys.maxsize on every cursor used by the dialect so BLOBs are always returned as materialized bytes/str. driver_config.stream_blob_threshold is left untouched; applications that need streaming can still use the raw dbapi cursor directly, as advised by the SQLAlchemy maintainer in sqlalchemy/sqlalchemy#10549.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #58. Large BLOBs (default >64 KiB) were unreadable from SQLAlchemy because firebird-driver returns
BlobReaderobjects paststream_blob_threshold, and SQLAlchemy's fetch lifecycle is incompatible with them:BlobReaderobjects when the cursor closes, so any subsequent.read()fails.LargeBinary.result_processorcallsbytes(value), which iterates a binaryBlobReaderline-by-line and raisesInterfaceError: Can't read line from binary BLOB.The dialect now sets
cursor.stream_blob_threshold = sys.maxsizeindo_execute,do_executemany, anddo_execute_no_params, forcing the driver to materialize every BLOB asbytes/str. The user-leveldriver_config.stream_blob_thresholdis left untouched, so applications that need streaming can still go through the raw DB-API cursor (as recommended by the SQLAlchemy maintainer in sqlalchemy/sqlalchemy#10549).Test plan
test_issue_58_large_blobcovers a 200 KiBBLOB SUB_TYPE 0andSUB_TYPE 1round trip; verified that it fails onmain(Can't read line from binary BLOBandBlobReader != str) and passes with the fix.pytest --dburi $FIREBIRD_DBURIrun (Firebird 5.0.4 / Python 3.14): 939 passed, 529 skipped, 0 failed.ruff checkandruff format --checkclean.