Skip to content

Comments

feat: add Parquet reader for Iceberg-backed CloudEvent retrieval#56

Merged
zer0stars merged 11 commits intomainfrom
iceberg
Feb 24, 2026
Merged

feat: add Parquet reader for Iceberg-backed CloudEvent retrieval#56
zer0stars merged 11 commits intomainfrom
iceberg

Conversation

@zer0stars
Copy link
Member

Summary

  • Add parquetreader package: reads CloudEvents from Parquet files on S3 using Arrow/Parquet libraries with row-offset support
  • Update eventrepo to resolve parquet_path#row_offset index keys and fetch events from Parquet files instead of individual S3 objects

Companion to DIMO-Network/dps#55

Test plan

  • Unit tests for parquetreader pass
  • Integration test fetching events with parquet-backed index keys
  • Verify backward compatibility with existing S3 individual-object index keys

Made with Cursor

@zer0stars zer0stars force-pushed the iceberg branch 6 times, most recently from ca5e370 to cdac11b Compare February 17, 2026 19:30
Co-authored-by: Cursor <cursoragent@cursor.com>
zer0stars and others added 10 commits February 23, 2026 15:20
- Add RawEvent scalar (scalar_raw_event.go): Marshal+Write for output; fast path
  for []byte/json.RawMessage in input unmarshal
- Resolvers use requireVehicleOptsByDID; add CheckVehicleRawDataByDID and tests
- Remove unused requireVehicleOpts and subjectFromTokenID; drop unused imports

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@zer0stars zer0stars merged commit a06e68f into main Feb 24, 2026
4 checks passed
@zer0stars zer0stars deleted the iceberg branch February 24, 2026 03:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant