feat: switch to writeNominalColumnarBatches endpoint#210
Open
gregakinman wants to merge 2 commits intomainfrom
Open
feat: switch to writeNominalColumnarBatches endpoint#210gregakinman wants to merge 2 commits intomainfrom
gregakinman wants to merge 2 commits intomainfrom
Conversation
Switch from the row-oriented writeNominalBatches endpoint to the new columnar writeNominalColumnarBatches endpoint for better compression and performance. The columnar format uses separate timestamp arrays and packed value arrays per channel instead of per-point timestamps. Key changes: - client.rs: POST to /storage/writer/v1/nominal-columnar (no path param) - consumer.rs: WriteRequestConsumer trait now uses WriteBatchesRequest - stream.rs: SeriesBuffer::take() converts row-oriented PointsType to columnar RecordsBatch with separate timestamps and packed values - listener.rs: NominalStreamListener uses WriteBatchesRequest - data_source_rid is now in the proto body, not the URL path Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2 tasks
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Contributor
|
overall, tested this and it works well and works faster too. it is a break from the perspective of a user of the library, if they write their own WriteConsumers. Connect does this, but we can move to something else - will check with customers too. open question if we should do a bigger refactor to construct the columnar format directly |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
writeNominalBatchesendpoint (POST /storage/writer/v1/nominal/{dataSourceRid}) to the new columnarwriteNominalColumnarBatchesendpoint (POST /storage/writer/v1/nominal-columnar)data_source_ridis now included in the protobuf body rather than as a URL path parameterSeriesBuffer::take()Changes
client.rs: Updated endpoint path, removeddata_source_ridpath parameterconsumer.rs:WriteRequestConsumertrait now operates onWriteBatchesRequest(columnar);NominalCoreConsumersetsdata_source_ridin the proto body before encoding;AvroFileConsumerupdated for columnar formatstream.rs: Addedpoints_type_to_columnar()conversion function;SeriesBuffer::take()now producesVec<RecordsBatch>;batch_processor()createsWriteBatchesRequestlistener.rs:NominalStreamListenertrait usesWriteBatchesRequestlib.rs: Updated prelude exports and all integration tests for columnar formatTest plan
cargo clippyis clean🤖 Generated with Claude Code