Skip to content

Port or enable DateTimeUtilsSuite timestamp format tests in Comet #3776

@parthchandra

Description

@parthchandra

While working on #3730, it was discovered that we are not testing various date time formats because the tests in Spark's DateTimeUtilsSuite are never being called.
The test suite is in sql/catalyst and does not extend either SQLTestUtils or CometTestBase. The tests in this subproject are not part of Comet CI.
Also, IgnoreComet is not available in this subproject and it may not be possible to disable the tests selectively.

The following tests are relevant -

Test Format coverage
string to timestamp YYYY-MM-DD, YYYY-MM-DDTHH:mm:ss, space-vs-T separator, time-only (HH:mm:ss, T18:12:15), timezone suffixes (Z, UTC, GMT±hh:mm, ±hhmm, ±h:m, named tz like Europe/Moscow), milliseconds/microseconds, nanosecond truncation, whitespace/control char
permutations, ~20 invalid formats that must return None
SPARK-35780: support full range of timestamp string Negative years (-1969-…), 5-digit years (99999-…), zero-padded years, Long.MaxValue/Long.MinValue boundary timestamps, integer overflow in individual fields
SPARK-37326: stringToTimestampWithoutTimeZone with allowTimeZone TIMESTAMP_NTZ with allowTimeZone=true/false and a +hh:mm suffix
SPARK-15379: special invalid date string Feb 29 on non-leap year, Apr 31 — must be rejected by both toDate and stringToTimestamp
trailing characters while converting string to timestamp Trailing garbage after a valid ISO timestamp (e.g. 2019-10-31T10:59:23Z:::) must return None
SPARK-29012: special timestamp values Keyword strings: Epoch, NOW, Yesterday, Today, tomorrow (with optional trailing timezone)
SPARK-35979: special timestamp without time zone values Same keyword strings for TIMESTAMP_NTZ via convertSpecialTimestampNTZ

It might be easier to copy the format-related tests from DateTimeUtilsSuite into a new or existing Comet test class (e.g. CometDateTimeUtilsSuite), as opposed to trying to enable catalyst tests in Comet CI.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions