When pause mode is set to ANY, and when a player enters fishing, time does not freeze. I can see this both via the golden clock and in the SMAPI logs.
I read through the code and don't completely understand how the playerstates are supposed to be transmitted, but since we're using the vanilla shouldTimePass function, I'm not sure why this would fail.