Skip to content

Conversation

@sanketjadhavSF
Copy link
Contributor

This pull request refactors how ClickHouse column types are handled in the codebase, with a focus on improving date/time formatting and adding support for array types. The changes make date and datetime values more readable by formatting them as strings, and introduce a new handler for array types. Additionally, the logic for recognizing certain column types has been enhanced for better normalization.

This will fix issues related to date and array column parsing shown below:

Column Datatype: DateTime64(6, UTC)
Example: toDateTime64('2025-12-11 10:30:45.123456', 6, 'UTC') AS datetime64_col
error": "row scan error: clickhouse [ScanRow]: converting Datetime64 to *string is unsupported"

Column Datatype: Array
Example: arrayDistinct(name) AS name_list
error": "row scan error: clickhouse []: column name_list - needs a slice or any",

Date and DateTime formatting improvements:

  • Added handleDate and handleDateTime functions to format Date, Date32, DateTime, and DateTime64 columns as strings ("YYYY-MM-DD" for dates, and ISO format for datetimes), replacing the previous generic handleTime handler. (internal/pkg/object/command/clickhouse/column_types.go) [1] [2]

Support for array column types:

  • Introduced a new handleArray function to properly handle ClickHouse Array columns, supporting both nullable and non-nullable arrays. (internal/pkg/object/command/clickhouse/column_types.go) [1] [2]

Type normalization enhancements:

  • Improved the unwrapCHType function to correctly identify and normalize DateTime64, DateTime, and Array types based on their string representation. (internal/pkg/object/command/clickhouse/column_types.go)

Test Results:
tested the new changes for all the datatype conversion in clickhouse plugin.

SQL:

SELECT CAST(42 AS UInt8) AS uint8_col,
       CAST(1000 AS UInt16) AS uint16_col,
       CAST(100000 AS UInt32) AS uint32_col,
       CAST(10000000000 AS UInt64) AS uint64_col,
       CAST(-42 AS int8) AS int8_col,
       CAST(-1000 AS Int16) AS int16_col,
       CAST(-100000 AS Int32) AS int32_col,
       CAST(-10000000000 AS Int64) AS int64_col,
       CAST(3.14 AS Float32) AS float32_col,
       CAST(2.718281828 AS Float64) AS float64_col,
       CAST(123.456 AS Decimal(10, 3)) AS decimal_col,
       CAST(999.99 AS Decimal32(2)) AS decimal32_col,
       CAST(12345.6789 AS Decimal64(4)) AS decimal64_col,
       'Hello World' AS string_col,
       CAST('Fixed' AS FixedString(10)) AS fixed_string_col,
       TRUE AS bool_col,
       toDate('2024-01-15') AS date_col,
       toDate32('2024-01-15') AS date32_col,
       toDateTime('2024-01-15 10:30:45') AS datetime_col,
       toDateTime64('2024-01-15 10:30:45.123456', 6, 'UTC') AS datetime64_col,
       [1, 2, 3, 4] AS array_uint8,
       [100, 200, 300] AS array_uint16,
       [1000000, 2000000] AS array_uint32,
       [10000000000, 20000000000] AS array_uint64,
       [-1, -2, -3] AS array_int8,
       [-100, -200, -300] AS array_int16,
       [-1000000, -2000000] AS array_int32,
       [1.1, 2.2, 3.3] AS array_float32,
       [3.14159, 2.71828] AS array_float64,
       ['apple', 'banana', 'cherry'] AS array_string,
       [TRUE, FALSE, TRUE] AS array_bool,
       [toDate('2024-01-01'), toDate('2024-12-31')] AS array_date,
       [toDateTime('2024-01-01 00:00:00'), toDateTime('2024-12-31 23:59:59')] AS array_datetime,
       CAST(NULL AS Nullable(Int32)) AS nullable_int,
       CAST(123 AS Nullable(Int32)) AS nullable_int_value,
       CAST(NULL AS Nullable(String)) AS nullable_string,
       CAST('test' AS Nullable(String)) AS nullable_string_value,
       CAST('low_cardinality_test' AS LowCardinality(String)) AS low_cardinality_col,
       CAST([1, 2, 3] AS Array (Nullable(Int32))) AS nullable_array

Result:

  "result": {
    "columns": [
      {
        "name": "uint8_col",
        "type": "int"
      },
      {
        "name": "uint16_col",
        "type": "int"
      },
      {
        "name": "uint32_col",
        "type": "int"
      },
      {
        "name": "uint64_col",
        "type": "long"
      },
      {
        "name": "int8_col",
        "type": "int"
      },
      {
        "name": "int16_col",
        "type": "int"
      },
      {
        "name": "int32_col",
        "type": "int"
      },
      {
        "name": "int64_col",
        "type": "long"
      },
      {
        "name": "float32_col",
        "type": "float"
      },
      {
        "name": "float64_col",
        "type": "double"
      },
      {
        "name": "decimal_col",
        "type": "double"
      },
      {
        "name": "decimal32_col",
        "type": "double"
      },
      {
        "name": "decimal64_col",
        "type": "double"
      },
      {
        "name": "string_col",
        "type": "string"
      },
      {
        "name": "fixed_string_col",
        "type": "FixedString(10)"
      },
      {
        "name": "bool_col",
        "type": "boolean"
      },
      {
        "name": "date_col",
        "type": "int32"
      },
      {
        "name": "date32_col",
        "type": "int32"
      },
      {
        "name": "datetime_col",
        "type": "long"
      },
      {
        "name": "datetime64_col",
        "type": "long"
      },
      {
        "name": "array_uint8",
        "type": "Array(UInt8)"
      },
      {
        "name": "array_uint16",
        "type": "Array(UInt16)"
      },
      {
        "name": "array_uint32",
        "type": "Array(UInt32)"
      },
      {
        "name": "array_uint64",
        "type": "Array(UInt64)"
      },
      {
        "name": "array_int8",
        "type": "Array(Int8)"
      },
      {
        "name": "array_int16",
        "type": "Array(Int16)"
      },
      {
        "name": "array_int32",
        "type": "Array(Int32)"
      },
      {
        "name": "array_float32",
        "type": "Array(Float64)"
      },
      {
        "name": "array_float64",
        "type": "Array(Float64)"
      },
      {
        "name": "array_string",
        "type": "Array(String)"
      },
      {
        "name": "array_bool",
        "type": "Array(Bool)"
      },
      {
        "name": "array_date",
        "type": "Array(Date)"
      },
      {
        "name": "array_datetime",
        "type": "Array(DateTime)"
      },
      {
        "name": "nullable_int",
        "type": "int"
      },
      {
        "name": "nullable_int_value",
        "type": "int"
      },
      {
        "name": "nullable_string",
        "type": "string"
      },
      {
        "name": "nullable_string_value",
        "type": "string"
      },
      {
        "name": "low_cardinality_col",
        "type": "string"
      },
      {
        "name": "nullable_array",
        "type": "Array(Nullable(Int32))"
      }
    ],
    "data": [
      [
        42,
        1000,
        100000,
        10000000000,
        -42,
        -1000,
        -100000,
        -10000000000,
        3.14,
        2.718281828,
        123.456,
        999.99,
        12345.6789,
        "Hello World",
        "Fixed",
        true,
        "2024-01-15",
        "2024-01-15",
        "2024-01-15T10:30:45",
        "2024-01-15T10:30:45.123456",
        [
          1,
          2,
          3,
          4
        ],
        [
          100,
          200,
          300
        ],
        [
          1000000,
          2000000
        ],
        [
          10000000000,
          20000000000
        ],
        [
          -1,
          -2,
          -3
        ],
        [
          -100,
          -200,
          -300
        ],
        [
          -1000000,
          -2000000
        ],
        [
          1.1,
          2.2,
          3.3
        ],
        [
          3.14159,
          2.71828
        ],
        [
          "apple",
          "banana",
          "cherry"
        ],
        [
          true,
          false,
          true
        ],
        [
          "2024-01-01T00:00:00Z",
          "2024-12-31T00:00:00Z"
        ],
        [
          "2024-01-01T00:00:00Z",
          "2024-12-31T23:59:59Z"
        ],
        null,
        123,
        null,
        "test",
        "low_cardinality_test",
        [
          1,
          2,
          3
        ]
      ]
    ]
  }

@sanketjadhavSF sanketjadhavSF merged commit bc09a76 into main Dec 15, 2025
6 checks passed
@sanketjadhavSF sanketjadhavSF deleted the SanketJ-fix-clickhouse-datatypes branch December 15, 2025 08:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants