-
Notifications
You must be signed in to change notification settings - Fork 296
Open
Description
While working on #3730, it was discovered that we are not testing various date time formats because the tests in Spark's DateTimeUtilsSuite are never being called.
The test suite is in sql/catalyst and does not extend either SQLTestUtils or CometTestBase. The tests in this subproject are not part of Comet CI.
Also, IgnoreComet is not available in this subproject and it may not be possible to disable the tests selectively.
The following tests are relevant -
| Test | Format coverage |
|---|---|
string to timestamp |
YYYY-MM-DD, YYYY-MM-DDTHH:mm:ss, space-vs-T separator, time-only (HH:mm:ss, T18:12:15), timezone suffixes (Z, UTC, GMT±hh:mm, ±hhmm, ±h:m, named tz like Europe/Moscow), milliseconds/microseconds, nanosecond truncation, whitespace/control char |
permutations, ~20 invalid formats that must return None |
|
SPARK-35780: support full range of timestamp string |
Negative years (-1969-…), 5-digit years (99999-…), zero-padded years, Long.MaxValue/Long.MinValue boundary timestamps, integer overflow in individual fields |
SPARK-37326: stringToTimestampWithoutTimeZone with allowTimeZone |
TIMESTAMP_NTZ with allowTimeZone=true/false and a +hh:mm suffix |
SPARK-15379: special invalid date string |
Feb 29 on non-leap year, Apr 31 — must be rejected by both toDate and stringToTimestamp |
trailing characters while converting string to timestamp |
Trailing garbage after a valid ISO timestamp (e.g. 2019-10-31T10:59:23Z:::) must return None |
SPARK-29012: special timestamp values |
Keyword strings: Epoch, NOW, Yesterday, Today, tomorrow (with optional trailing timezone) |
SPARK-35979: special timestamp without time zone values |
Same keyword strings for TIMESTAMP_NTZ via convertSpecialTimestampNTZ |
It might be easier to copy the format-related tests from DateTimeUtilsSuite into a new or existing Comet test class (e.g. CometDateTimeUtilsSuite), as opposed to trying to enable catalyst tests in Comet CI.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels