Implement timeout classification and comprehensive health monitoring tests

This commit completes the health monitoring implementation with timeout
vs error differentiation and comprehensive test coverage.

**Driver Improvements:**
- Fix owon_spm: Remove buggy try-except, let exceptions propagate naturally
- Fix owon_oel: Improve empty response handling, remove nested try-except
- Fix owon_xdm: Handle device-off state correctly with empty dict return
- Fix rigol_dho800: Add exception propagation documentation
- Add DriverBase._poll_multi_class() helper method with proper docs

**Critical Bug Fix in poll_worker.py:**
- Fix double failure recording when poll_status() raises exceptions
- Added exception_occurred flag to prevent recording failure twice
- Ensures timeout penalties are correctly preserved (was being overwritten)

**Comprehensive Test Coverage (35 tests, all passing):**

Unit Tests (test_connection_health.py - 26 tests):
- Health state transitions (unknown → healthy → degraded → dead)
- Timeout vs error classification validation
- Quality monitor penalty differentiation (-10 timeout vs -15 error)
- Quality score calculations and tier assignments
- Connection reset functionality
- Integration scenarios for timeout/error patterns

Integration Tests (test_poll_health_integration.py - 9 tests):
- Complete flow: driver → worker → connection → health states
- Mock drivers for various scenarios (healthy, timeout, error, empty, intermittent)
- Validates timeout vs error penalty differentiation end-to-end
- Tests recovery patterns and quality score accumulation
- Verifies consecutive failure tracking

**Test Fixes:**
- Update test_driver_identify.py to use new class names (OwonSPM, OwonXDM)
- Add get_quality_multiplier() method to FakeDeviceConnection mock
- Update test_driver_base.py for new helper method

**Test Results:**
✅ 35/35 health monitoring tests pass (26 unit + 9 integration)
✅ 306/322 total tests pass (16 pre-existing failures unrelated to this work)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

Author: MarkoVcode

benchmesh-serial-service/src/benchmesh_service/drivers/base.py

@@ -61,13 +61,107 @@ def poll_status(self, channel: int) -> Dict[str, Any]:
         Must be implemented by all drivers.
         Called periodically by polling worker thread (every 2-3 seconds typically).
 
+        **RETURN VALUE CONTRACT:**
+
+        - **Success**: Return dictionary with status data
+        - **Device off/not responding**: Return empty dict {}
+        - **Communication error**: Raise exception (TimeoutError, SerialException, etc.)
+
+        **IMPORTANT - Connection Monitoring:**
+
+        This method serves dual purposes:
+        1. **Data collection**: Returns device measurements/status
+        2. **Health monitoring**: Signals connection health to worker
+
+        The worker uses the return value to determine connection health:
+        - Non-empty dict → Connection healthy, data recorded
+        - Empty dict {} → Device not responding, connection dropped
+        - Exception raised → Communication failure, connection dropped
+
+        **DO NOT:**
+        - ❌ Catch exceptions and return fake data (defeats health monitoring)
+        - ❌ Return {"VOUT": None, "IOUT": None} on timeout
+        - ❌ Return {"ERROR": str(e)} on exception
+        - ❌ Return None instead of {} or raising
+
+        **DO:**
+        - ✅ Let transport exceptions propagate naturally
+        - ✅ Return empty dict {} only for valid "device off" states
+        - ✅ Return meaningful data for standby/off states when possible
+
+        **Examples:**
+
+        CORRECT - Let exceptions propagate (tenma_72 pattern):
+            def poll_status(self, channel: int):
+                # No try/except - transport errors bubble up
+                v = self.query_voltage(channel)  # May raise TimeoutError
+                i = self.query_current(channel)  # May raise TimeoutError
+                return {"VOUT": v, "IOUT": i}
+
+        CORRECT - Empty dict for device powered off:
+            def poll_status(self, channel: int):
+                raw = self.query_status(channel)
+                if not raw or raw.strip() == "":
+                    return {}  # Device powered off, no data available
+                return self._parse_status(raw)
+
+        CORRECT - Standby state returns valid data:
+            def poll_status(self, channel: int):
+                v = self.query_voltage(channel)
+                i = self.query_current(channel)
+                output_on = self.query_output_state(channel)
+                return {
+                    "VOUT": v,
+                    "IOUT": i,
+                    "OUTPUT": "ON" if output_on else "OFF",
+                    "SBY": not output_on  # Standby flag
+                }
+
+        WRONG - Fake data on communication error:
+            def poll_status(self, channel: int):
+                try:
+                    v = self.query_voltage(channel)
+                except TimeoutError:
+                    return {"VOUT": None}  # ❌ WRONG! Worker thinks success!
+
+        WRONG - Catching and hiding errors:
+            def poll_status(self, channel: int):
+                try:
+                    return {"DATA": self.query_data(channel)}
+                except Exception as e:
+                    return {"ERROR": str(e)}  # ❌ WRONG! Truthy dict!
+
+        **Multi-Class Devices:**
+
+        Use _poll_multi_class() helper for devices with multiple classes:
+
+            def poll_status(self, channel: int):
+                return self._poll_multi_class(channel, {
+                    "PSU": self.poll_status_psu,
+                    "DMM": self.poll_status_dmm
+                })
+
+        The helper handles partial success gracefully (some classes succeed, some fail).
+
         Args:
             channel: Channel number (1-based, may be ignored for single-channel devices)
 
         Returns:
             Dictionary with device status. Keys depend on device type/class.
             For multi-class devices (e.g., OWONSPM), return nested dict:
                 {"PSU": {...}, "DMM": {...}}
+            Empty dict {} if device powered off or not responding.
+
+        Raises:
+            TimeoutError: Serial read timeout (device not responding to query)
+            SerialException: Physical connection error (cable disconnected, port closed)
+            ValueError: Invalid/unparseable response from device
+
+            Exceptions are caught by worker and trigger health monitoring + reconnection.
+
+        See Also:
+            - tenma_72 driver for reference implementation
+            - _poll_multi_class() helper for multi-class devices
         """
         pass
 
@@ -118,6 +212,85 @@ def is_connected(self) -> bool:
         """
         return self.t.is_open
 
+    def _poll_multi_class(
+        self,
+        channel: int,
+        poll_methods: Dict[str, callable]
+    ) -> Dict[str, Any]:
+        """
+        Helper for multi-class devices with partial success support.
+
+        Simplifies polling for devices that combine multiple instrument classes
+        (e.g., OWON SPM has both PSU and DMM functionality).
+
+        Attempts to poll each class independently. If at least one class
+        succeeds, returns partial data with successful classes. If ALL classes
+        fail, raises exception to trigger connection drop and reconnection.
+
+        This allows graceful degradation - if DMM fails but PSU works, PSU data
+        is still usable while DMM reconnection is attempted.
+
+        Args:
+            channel: Channel number to pass to poll methods
+            poll_methods: Dict mapping class name to poll method callable
+                         e.g., {"PSU": self.poll_status_psu, "DMM": self.poll_status_dmm}
+
+        Returns:
+            Dict with class-keyed data. Successful classes have data,
+            failed classes have None value.
+
+        Raises:
+            RuntimeError: If all classes failed to poll (device completely unresponsive)
+
+        Example:
+            def poll_status(self, channel: int):
+                return self._poll_multi_class(channel, {
+                    "PSU": self.poll_status_psu,
+                    "DMM": self.poll_status_dmm
+                })
+
+        Example output (partial success):
+            {
+                "PSU": {"VOUT": 12.5, "IOUT": 1.2},  # Success
+                "DMM": None                            # Failed
+            }
+
+        Example behavior:
+            - If PSU and DMM both succeed: Returns both data dicts
+            - If PSU succeeds, DMM fails: Returns PSU data, DMM=None (partial success)
+            - If both fail: Raises RuntimeError (triggers reconnection)
+        """
+        import logging
+        logger = logging.getLogger(__name__)
+
+        result = {}
+        any_success = False
+
+        for class_name, poll_method in poll_methods.items():
+            try:
+                data = poll_method(channel)
+                if data:
+                    result[class_name] = data
+                    any_success = True
+                else:
+                    # Poll method returned empty dict
+                    logger.warning(f"{class_name} poll returned empty data for channel {channel}")
+                    result[class_name] = None
+
+            except Exception as e:
+                # Poll method raised exception (timeout, serial error, etc.)
+                logger.warning(f"Failed to poll {class_name} channel {channel}: {e}")
+                result[class_name] = None
+
+        # If all classes failed, raise to trigger reconnection
+        if not any_success:
+            class_names = list(poll_methods.keys())
+            raise RuntimeError(
+                f"All classes {class_names} failed to poll channel {channel} - device not responding"
+            )
+
+        return result
+
     # ===== TRANSPORT DELEGATION METHODS =====
 
     def write(self, data: bytes) -> None:

benchmesh-serial-service/src/benchmesh_service/drivers/owon_oel/driver.py

@@ -7,12 +7,21 @@ def query_status(self, channel: int):
         return self.t.read_until_reol(1024)
 
     def poll_status(self, channel: int):
+        """
+        Poll electronic load status.
+
+        Exceptions propagate naturally for health monitoring.
+        Returns empty dict {} if device is off/not responding.
+        """
         raw = self.query_status(channel) or ""
+
+        # Device off or not responding - return empty dict
         if raw is None or raw == "":
-            # Return a minimal but truthy structure to avoid dropping the connection
-            return {"VIN": si_to_value(0), "IIN": si_to_value(0), "PIN": si_to_value(0), "OVP": "OFF", "OCP": "OFF", "OTP": "OFF", "REMOTE": "OFF", "INPUT": "OFF", "MODE": "CURR"}
+            return {}
+
         if isinstance(raw, bytes):
             raw = raw.decode(errors='ignore')
+
         parts = raw.strip().split(',')
         result = {}
         keys = ["VIN", "IIN", "PIN", "OVP", "OCP", "OTP"]

benchmesh-serial-service/src/benchmesh_service/drivers/owon_spm/driver.py

@@ -56,12 +56,21 @@ def query_function(self, channel: int):
         return self.normalize_spaces(funct)
 
     def poll_status_psu(self, channel: int):
+        """
+        Poll PSU status data.
+
+        Exceptions propagate naturally for health monitoring.
+        Returns empty dict {} if device is off/not responding.
+        """
         raw = self.query_status(channel) or ""
+
+        # Device off or not responding - return empty dict
         if raw is None or raw == "":
-            # Return a minimal but truthy structure to avoid dropping the connection
-            return {"VOUT": None, "IOUT": None, "POUT": None}
+            return {}
+
         if isinstance(raw, bytes):
             raw = raw.decode(errors='ignore')
+
         parts = raw.strip().split(',')
         result = {}
         keys = ["VOUT", "IOUT", "POUT", "OVP", "OCP", "OTP", "OM"]
@@ -109,38 +118,27 @@ def poll_status_dmm(self, channel: int):
     def poll_status(self, channel: int):
         """
         Unified polling method for multi-class device.
-        
+
         Polls both PSU and DMM data in a single operation to avoid
         double serial port access for devices with multiple classes
         on a single serial port.
-        
+
+        Uses _poll_multi_class() helper for partial success handling.
+        Exceptions propagate naturally for health monitoring.
+
         Returns dict with class-keyed data:
         {
             "PSU": {psu status data},
             "DMM": {dmm status data}
         }
+
+        Returns empty dict {} if device is off/not responding.
+        Raises exception on communication errors.
         """
-        result = {}
-        
-        # Get PSU data
-        try:
-            psu_data = self.poll_status_psu(channel)
-            if psu_data:
-                result["PSU"] = psu_data
-        except Exception as e:
-            logger.warning(f"Failed to poll PSU status: {e}")
-            result["PSU"] = {"VOUT": None, "IOUT": None, "POUT": None}
-        
-        # Get DMM data  
-        try:
-            dmm_data = self.poll_status_dmm(channel)
-            if dmm_data:
-                result["DMM"] = dmm_data
-        except Exception as e:
-            logger.warning(f"Failed to poll DMM status: {e}")
-            result["DMM"] = None
-        
-        return result
+        return self._poll_multi_class(channel, {
+            "PSU": self.poll_status_psu,
+            "DMM": self.poll_status_dmm
+        })
 
     def set_output(self, channel: int, value):  # ON / OFF
         self.t.write_line('OUTP ' + str(value))

benchmesh-serial-service/src/benchmesh_service/drivers/owon_xdm/driver.py

@@ -4,11 +4,20 @@
 
 class OwonXDM(DriverBase):
     def poll_status(self, channel: int):
+        """
+        Poll multimeter status.
+
+        Exceptions propagate naturally for health monitoring.
+        Returns empty dict {} if device is off/not responding.
+        """
         query_measurement = self._query_measurement(1)
         func = self._query_function(1)
         raw = self.query_identify() or b""
+
+        # Device off or not responding - return empty dict
         if not raw:
-            return None
+            return {}
+
         return {"MEAS": sci_to_value(query_measurement), "MODE": func, "RANGE": self.cache.get(func + ":RANGE")}
 
     def set_remote(self, channel: int, value):

benchmesh-serial-service/src/benchmesh_service/drivers/rigol_dho800/driver.py

@@ -10,34 +10,38 @@ class RigolDHO800(DriverBase):
     """Driver for RIGOL DHO800 series oscilloscopes"""
 
     def poll_status(self, channel: int = 1):
-        """Poll device status"""
+        """
+        Poll oscilloscope status.
+
+        Exceptions propagate naturally for health monitoring.
+
+        Note: Oscilloscopes don't have an "off" state over USB TMC - they either
+        respond successfully or timeout. No need to check for empty response.
+        """
         # Query basic oscilloscope parameters
-        try:
-            # Get channel scale (V/div)
-            self.t.write_line(f':CHANnel{channel}:SCALe?')
-            scale = self.t.read_until_reol(1024)
-            
-            # Get channel offset
-            self.t.write_line(f':CHANnel{channel}:OFFSet?')
-            offset = self.t.read_until_reol(1024)
-            
-            # Get channel coupling
-            self.t.write_line(f':CHANnel{channel}:COUPling?')
-            coupling = self.t.read_until_reol(1024)
-            
-            # Get timebase scale
-            self.t.write_line(':TIMebase:MAIN:SCALe?')
-            timebase = self.t.read_until_reol(1024)
-            
-            return {
-                "CHANNEL": channel,
-                "SCALE": sci_to_value(scale),
-                "OFFSET": sci_to_value(offset),
-                "COUPLING": coupling,
-                "TIMEBASE": sci_to_value(timebase)
-            }
-        except Exception as e:
-            return {"ERROR": str(e)}
+        # Get channel scale (V/div)
+        self.t.write_line(f':CHANnel{channel}:SCALe?')
+        scale = self.t.read_until_reol(1024)
+
+        # Get channel offset
+        self.t.write_line(f':CHANnel{channel}:OFFSet?')
+        offset = self.t.read_until_reol(1024)
+
+        # Get channel coupling
+        self.t.write_line(f':CHANnel{channel}:COUPling?')
+        coupling = self.t.read_until_reol(1024)
+
+        # Get timebase scale
+        self.t.write_line(':TIMebase:MAIN:SCALe?')
+        timebase = self.t.read_until_reol(1024)
+
+        return {
+            "CHANNEL": channel,
+            "SCALE": sci_to_value(scale),
+            "OFFSET": sci_to_value(offset),
+            "COUPLING": coupling,
+            "TIMEBASE": sci_to_value(timebase)
+        }
 
     def set_output(self, channel: int, state: str):
         """Enable/disable oscilloscope channel display