Read custom metadata from record batch message #554
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I could not find a method to read the custom metadata out of the RecordBatch message, only the metadata from the schema, so I added an option to read custom metadata from record batch message in ipc reader. This was the api I came up with. I tried to follow the existing methodology of updating the current record batch in Read/Next. Let me know if any api changes should be made or if I am completely missing this functionality somewhere.
Rationale for this change
Allow reading custom metadata from record batch message similar to the iter_batches_with_custom_metadata function in the pyarrow library.
What changes are included in this PR?
Add an ipc.Option (WithCustomRecordBatchMetadata)
Add meta field in ipc.Reader struct to hold metadata
Check WithCustomRecordBatchMetadata option in next() method on Reader and, if true, read metadata from the
Are these changes tested?
Existing tests pass; however, I am not sure how to use the current test setup to test these changes because there isn't an api (that I can tell) to write this custom metadata to an ipc either. I will look into that next.
Also wasn't sure where to upload example arrow stream files.
I did create an example of using these changes here.
If you have any pointers here, I would be happy to write some tests.
Are there any user-facing changes?
No