feat(rust): propagate SEA manifest metadata through Arrow schema#353
Open
vikrantpuppala wants to merge 3 commits intomainfrom
Open
feat(rust): propagate SEA manifest metadata through Arrow schema#353vikrantpuppala wants to merge 3 commits intomainfrom
vikrantpuppala wants to merge 3 commits intomainfrom
Conversation
…schema Add design document for fixing result metadata diffs between the OSS ODBC driver and the reference driver. The root cause is information loss at the ADBC Rust layer — the SEA API provides rich column metadata (type_name, type_text, type_precision, type_scale) in the result manifest, but only the Arrow schema crosses the FFI boundary to the C++ ODBC driver. The solution encodes manifest metadata as Arrow field-level key-value metadata (databricks.* keys), applied in ResultReaderAdapter before FFI export. This matches the JDBC driver's approach of using server-provided metadata as the primary source. Co-authored-by: Isaac
Attach databricks.* field-level metadata (type_name, type_text, type_precision, type_scale, type_interval_type) from the SEA result manifest to Arrow fields in ResultReaderAdapter. This preserves server-provided column metadata through the FFI boundary so the ODBC driver can use it instead of reverse-engineering from Arrow type IDs. Co-authored-by: Isaac
vikrantpuppala
commented
Mar 17, 2026
vikrantpuppala
force-pushed
the
design/result-metadata-propagation
branch
from
9612712 to
c1ba98e
Compare
The metadata FFI functions (get_catalogs, get_schemas, get_tables, etc.) were discarding the manifest from ExecuteResult. Now they use export_execute_result() which passes the manifest to ResultReaderAdapter, ensuring databricks.* field metadata is attached to the Arrow schema for all query paths, not just Statement::execute(). Co-authored-by: Isaac
vikrantpuppala
changed the title
vikrantpuppala
force-pushed
the
design/result-metadata-propagation
branch
from
c1ba98e to
4c2e9ed
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
databricks.*field-level key-value metadataChanges
src/types/sea.rstype_precision,type_scale,type_interval_typetoColumnInfosrc/reader/mod.rsmetadata_keysconstants +augment_schema_with_manifest()src/reader/mod.rsResultReaderAdapter::new()to accept optional manifestsrc/client/mod.rsmanifest: Option<ResultManifest>toExecuteResultsrc/client/sea.rsresponse.manifestthrough toExecuteResultsrc/statement.rsResultReaderAdapter::new()src/ffi/metadata.rsexport_reader()for all metadata FFI functionsTest plan
ColumnInfodeserialization with/without optional fieldsaugment_schema_with_manifest(basic types, DECIMAL precision/scale, INTERVAL, missing ColumnInfo, preserves existing metadata)ResultReaderAdapterwith and without manifestmetadata_propagation_testexample) verifying metadata flows through for INT, LONG, STRING, BOOLEAN, DOUBLE, FLOAT, SHORT, BYTE, DECIMAL(10,2), DECIMAL(18,5), DECIMAL(38,0), DATE, TIMESTAMP, ARRAY, MAP, STRUCT, BINARYThis pull request was AI-assisted by Isaac.