fields.DateTime - load/dump to timestamp

[vgavro]

I'm currently using this code to parse/dump timestamp, but I believe this should be in standard library. Should I create pull request, or have I missed another way to work with datetime/timestamp format?

class DateTime(ma.fields.DateTime):
    """
    Class extends marshmallow standart DateTime with "timestamp" format.
    """

    DATEFORMAT_SERIALIZATION_FUNCS = \
        ma.fields.DateTime.DATEFORMAT_SERIALIZATION_FUNCS.copy()
    DATEFORMAT_DESERIALIZATION_FUNCS = \
        ma.fields.DateTime.DATEFORMAT_DESERIALIZATION_FUNCS.copy()

    DATEFORMAT_SERIALIZATION_FUNCS['timestamp'] = lambda x: x.timestamp()
    DATEFORMAT_DESERIALIZATION_FUNCS['timestamp'] = datetime.fromtimestamp

[sloria]

@vgavro I think this is a fair use case. I anticipate that users will typically want to deserialize to timezone-aware datetimes, so the API should make this simple. I haven't given to much thought to what it would look like though. Maybe it would make sense as a separate field that receives a tzinfo param,

my_timestamp = fields.Timestamp(tzinfo=pytz.utc)

[disconnect3d]

Bump, this would be nice to have 👍

[stephan-rayner]

Is there a plan to do this thing? I can't tell you how happy this would make me?

[disconnect3d]

@stephan-rayner You can do it even better: work on that and send a PR? :)

Note that you can also attend in https://hacktoberfest.digitalocean.com and win a t-shirt ;D.

[sloria]

I would gladly review and merge a PR adding a Timestamp field.

[vgavro]

I will gladly work on this PR, but maybe it's better to implement this as custom format for AwareDateTime and NaiveDateTime instead of separate field #933 ? Like I see it's good decision to force conversion to aware datetime, but still explicitly allow naive datetimes.

[disconnect3d]

but still explicitly allow naive datetimes

explicitly is the key here. If the user wants it, let him do that.

I am not a maintainer of the project but I am up for this as we wouldn't make the bad thing (naive datetimes) to be the default ones.

[sloria]

Ah, yes, I see your @vgavro . Yes, I suppose it could be a new datetime format. I'm fine with that.

[lig]

@sloria I've tried this as a format for fields.DateTime and it is not quite right. DateTime field produces the value of type str. However, a timestamp has to have type float. Also, there are some edge cases (python type → serialized from):

• float → float (obviously, it could be done via the Float field but it should be positive and it will be handy to have something like from_milliseconds for JavaScript compatibility)
• float → datetime string (rare case but fair)
• datetime or float → float
• datetime or float → datetime string

My idea is that supporting timestamps is a bit more than formatting datetime to float.

[vgavro]

@lig As you can see in implementation of DateTime, it is not forcing str or float, so I don't get the problem you stated, also serialization in one format and deserialization in other not a matter of specific field responsibility (you can use different fields for that). But I really agree with your from_milliseconds usecase.

@sloria @lafrech Here are scenarios with timestamps (+ timezone aware/naive datetimes) we want to cover, I would really love to have feedback before I can work on this, if I haven't anything missed (I hope?)

[1]. By definition timestamp (POSIX time or UNIX Epoch time) should be in UTC.
[2]. Anyway we may see external systems that is using timestamp in timezone-related manner, on working with such systems we should provide timezone-aware datetime with explicitly set timezone to replace/conversion.
[3]. Also we may have python systems that is internally using timezone-naive datetimes, asserting all datetimes are using same timezone (local or utc, for example).

So generally timestamp adding because of [1] can be solved:
Adding timestamp and timestamp_ms to DateTime.format.
By default all timestamps must be converted to utc by default, and we can add this to DateTime formats without extra logic for timezone conversion, it will be fair use.
timestamp_ms stands for "milliseconds" (as stated in wikipedia, where milliseconds marked as "ms", and microseconds marked as "μs", so it's good name to be compatible with javascript-related projects, where timestamp is Number in milliseconds - for example moment.now() and Number(new Date()) returns milliseconds Number - good point from @lig).

As for [2],[3] - one option may be to interpret timestamp timezones (or other datetimes) in custom ways (replace/drop/convert tzinfo) in subclasses, that willl inherit DateTime, but not required to be in it's core functionality. This way problem is partly trying to solve #933 , but in case with timestamp we want to explicitly replace timezone, not just default it (because timestamp default is UTC, like we see in [1]). I think this functionality can be in base DateTime field by extending it with timezone=None, timezone_naive=False, this can be solved in backward-compatible way.

The [2] problem may be solved:
adding timezone=None to DateTime field, to force timestamps + timezone-naive strings in explicitly set timezone. If self.timezone - on deserialization timezone will be replaced for datetime-naive strings and timestamps, on serialization datetime will be converted to self.timezone for timezone-aware (only for timestamps, not to have different input/output for timezone aware strings) or replaced for timezone-naive datetimes, This will also cover AwareDateTime use-case from #933 , which is also enforces only timezone-naive datetimes. timezone will be set to None by default (for backward-compatibility), in this case timestamps always will be in UTC (because of [1]), and other formats can return timezone-naive datetimes. Or we may default it to UTC (which breaks backward-compatibility and does not allow parsing timezone-naive datetimes, which may be interpreted as special use-case by user code for example, but maybe for good reason).

The [3] problem may be solved:
adding timezone_naive=False, which may:
drops tzinfo on deserialization (ony after conversion to timezone from proposition above, if self.timezone is not None), this also will cover NaiveDateTime use-case from #933 . If we have timezone=None by default on serialization timezone-naive datetimes will be anyway converted to UTC (as in current implementation for iso/rfc), and we have consistent behaviour with timestamp also in this case. I know there is discussion around serializing NaiveDateTime to timezone-naive strings, this can be achieved with format_options proposed here #889 (comment)

Cons on this:

• We're may receive timestamps as int, but always serialize as float. Rare case when this can be a probem can be fixed with custom format_options, as proposed here ISO8601 time string with a Z comes out with +00:00 after load and then dump #889 (comment)
• We have different timezone option behaviour for formats - replacing timezone for timestamp and setting default for timezone-naive strings.
  (what else? some things from Add load_as_tz_aware (and dump_as_tz_aware ?) parameter to DateTime #520 The deserialization of DateTime fields doesn't match the documentation #955 ?)

P.S. One more use-case that personally I see not fits in conception "timestamp is just another datetime format" is interpreting 0 as null for timestamps, but it's too easy to implement in user space and really is just bad api design that I've seen - but I'm sure we don't need it in marshmallow code anyway.

I really think this will fix timezones/timestamps issues once for all (at last), sorry for tagging (I know it's hard to give time for such big open source project), and thanks in advance for your feedback. I can use tests and utils.str2tz from #933 if you think this is ok to go.