[parquet] Use MICROS annotation for TIMESTAMP(n<=3) columns (Iceberg v2 compatibility)

[q8webmaster]

Problem

Any Paimon table with a TIMESTAMP or TIMESTAMP_WITH_LOCAL_TIME_ZONE column of precision ≤ 3 cannot be queried by Iceberg-aware engines (Athena, Trino, Spark Iceberg reader). The engine rejects every Parquet file for that column with an error such as:

Field ts's type INT64 in parquet file … is incompatible with type
timestamp(6) with time zone defined in table schema

The job writes data successfully and Paimon itself can read the files back, but any reader that enforces the Iceberg v2 Parquet spec rejects them.

Root cause

1. Wrong Parquet annotation

ParquetSchemaConverter.createTimestampWithLogicalType selects the annotation unit based on precision:

Precision	Annotation emitted	Iceberg v2 valid?
≤ 3	TIMESTAMP(MILLIS)	❌
4–6	TIMESTAMP(MICROS)	✅
> 6	INT96	—

The Iceberg v2 specification requires INT64 MICROS for both timestamp and timestamptz logical types. MILLIS is only permitted under Iceberg v3, which most engines do not yet support.

2. Wrong Parquet footer-stats decoding

ParquetSimpleStatsExtractor.toTimestampStats called Timestamp.fromEpochMillis for precision ≤ 3. After the annotation fix, footer stats for those columns are INT64 microseconds, so the correct call is Timestamp.fromMicros. Using fromEpochMillis inflates the decoded bound by 1000×.

3. Wrong lazy dictionary decoding

VectorizedColumnReader has a lazy dictionary fast path for INT64/LongColumnVector: the raw Parquet dictionary is attached to the vector without going through LongTimestampUpdater.longTimestamp(), which normalises on-disk microseconds to the milliseconds that ParquetTimestampVector.getTimestamp expects. The result is timestamps ~1000× too large for any dictionary-encoded page.

Fix

• ParquetSchemaConverter: emit MICROS for precision <= 3 in createTimestampWithLogicalType.
• ParquetRowDataWriter.TimestampMillsWriter: call value.toMicros() (milliseconds × 1000) so the stored value matches the MICROS annotation.
• ParquetSimpleStatsExtractor: use Timestamp.fromMicros for precision ≤ 3 footer statistics.
• VectorizedColumnReader: exclude precision ≤ 3 timestamp types from lazy dictionary decoding via isLowPrecisionTimestamp helper.

The reader path for existing files (MILLIS annotation → precision=3, MICROS annotation → precision=6) is intentionally left unchanged so that files written by older Paimon versions remain readable.

Backward compatibility

Existing tables whose files carry a MILLIS annotation are still readable. Tables that mix old (MILLIS) and new (MICROS) files should be rebuilt after upgrading, because Iceberg-aware engines enforcing strict annotation checking will still reject the old files.

After the fix, a TIMESTAMP(3) column read back through Paimon's own Parquet reader returns TIMESTAMP(6) (since MICROS maps to precision 6 in the reader). Stored values remain millisecond-granular; only the declared precision widens.

Prior art

PR #8222 fixed a related bug in PostgresRecordParser where a Debezium io.debezium.time.Timestamp (int64 millis) was mapped to BIGINT instead of TIMESTAMP(3). Both affect the same precision ≤ 3 path. The Iceberg manifest decoding fix is in the companion PR #8231.

Changes

• ParquetSchemaConverter.java: emit MICROS for precision <= 3
• ParquetRowDataWriter.java: TimestampMillsWriter.writeTimestamp calls value.toMicros()
• ParquetSimpleStatsExtractor.java: use fromMicros for precision ≤ 3 footer stats
• ParquetTimestampVector.java: update class javadoc to reflect MICROS storage
• VectorizedColumnReader.java: exclude precision ≤ 3 timestamps from lazy dictionary decoding; add isLowPrecisionTimestamp helper
• ParquetSchemaConverterTest.java: new test testLowPrecisionTimestampUseMicrosAnnotation; testPaimonParquetSchemaConvert updated for widened round-trip precision