Skip to content

Add Dynamic column support (read + pass-through) for Proton native protocol#28

Open
sofiane-soufi wants to merge 1 commit into
timeplus-io:masterfrom
sofiane-soufi:feature/add-dyanmic-column-support
Open

Add Dynamic column support (read + pass-through) for Proton native protocol#28
sofiane-soufi wants to merge 1 commit into
timeplus-io:masterfrom
sofiane-soufi:feature/add-dyanmic-column-support

Conversation

@sofiane-soufi
Copy link
Copy Markdown
Contributor

This PR adds first-class dynamic support to timeplus-cpp for Proton read + pass-through use cases.

Scope

  • Support reading Dynamic columns from Proton over native TCP.
  • Preserve shared-variant payloads as opaque bytes (ColumnString) without lossy decode/re-encode.
  • Keep in-memory behavior stable for append/slice/clone/save/load.
  • Do not add a rich authoring API for arbitrary Dynamic values.
  • Do not add first-class Variant/JSON type support (follow-up PR).

What changed

Type system / parser / factory

  • Added Type::Code::Dynamic.
  • Added DynamicType(max_types) with canonical names:
    • dynamic
    • dynamic(max_types=N)
  • Added Type::CreateDynamic(size_t max_types = 32).
  • Added parser support for:
    • dynamic
    • dynamic(max_types=N)
  • Added argument validation in factory (0..254, malformed forms throw ValidationError).

New column implementation

  • Added ColumnDynamic:
    • files:
      • timeplus/columns/dynamic.h
      • timeplus/columns/dynamic.cpp
  • Implemented native serialization/deserialization:
    • structure versions V1 and V2
    • discriminator modes BASIC and COMPACT
    • implicit shared_variant handling
  • Implemented column operations:
    • Append, Slice, CloneEmpty, Clear, Reserve, Swap, Size
  • Added inspection helpers for tests/diagnostics:
    • row discriminator
    • null/shared checks
    • variant names/columns

Proton protocol compatibility fixes

  • Updated block-info parsing in client to consume Proton fields and keep stream aligned:
    • handled field ids 100, 101, 102
  • Fixed Dynamic discriminator mapping to match Proton Variant global ordering (lexicographic type-name order, including shared_variant), preventing read desync.

Build/docs/wiring

  • Registered new sources/headers in CMake.
  • Installed dynamic.h.
  • Added Dynamic to supported types in README.
  • Added Type::Dynamic to unsupported ItemView wrapper set.

Tests

Added/updated tests for parser, types, factory, column behavior, and integration:

  • ut/type_parser_ut.cpp
    • parse dynamic
    • parse dynamic(max_types=5)
    • reject malformed input
  • ut/CreateColumnByType_ut.cpp
    • create from both forms
    • invalid args throw
  • ut/types_ut.cpp
    • GetName() / IsEqual() coverage for Dynamic variants
  • ut/column_dynamic_ut.cpp
    • LoadV1Basic / LoadV2Basic
    • save-after-load byte preservation
    • slice/append behavior
    • invalid discriminator rejection
    • unsupported nested variant type rejection
  • ut/roundtrip_tests.cpp
    • Proton integration read-path test for Dynamic shared-variant usage
  • ut/CMakeLists.txt
    • added column_dynamic_ut.cpp

Validation run

  • timeplus-cpp-lib builds successfully.
  • Dynamic-focused unit/integration tests pass locally against Proton 3.0.16 (non-compressed path).
  • Compressed variants in Dynamic integration test are currently skipped by design.

@yuzifeng1984 yuzifeng1984 self-assigned this Feb 27, 2026
@gangtao
Copy link
Copy Markdown

gangtao commented Feb 27, 2026

@sofiane-soufi thanks for contributing on this one

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants