Integrate TOF lookup system into monitor and detector views

[SimonHeybrock]

Currently monitor_data and detector_data both produce results in time_of_arrival (aka event_time_offset). This is frame-wrapped and w.r.t. the source pulse timestamp. While this may be sufficient initially I think scientists will soon prefer to see time_of_flight or even wavelength (or be able to switch between these three options).

This is not possible without more thought in the current setup. monitor_data and detector_data are not based on the Sciline-based reduction workflows, but perform simpler lower-level (somewhat faster) processing. We need to consider carefully what our options are. Without looking closer into whether it would be possible, I think not using Sciline is likely still what we want, i.e., we would want to use the TOF lookup mechanism from ess.reduce directly.

[nvaytet]

How were you making the live dream dspacing plots? Was that not using a sciling workflow?
I had always assumed that we wanted to use the sciline workflows so as to not re-implement the logic in different places...

[SimonHeybrock]

How were you making the live dream dspacing plots? Was that not using a sciling workflow? I had always assumed that we wanted to use the sciline workflows so as to not re-implement the logic in different places...

Those are two different things:

1. Full reduction workflows. Use technique-packages and their builtin TOF calculation.
2. Raw monitor and detector views. Use some projection utilities from essreduce but not the technique packages, and no Sciline.

[nvaytet]

My opinion would be that if you want a tof coordinate on a monitor or detector view, the simplest would be to treat it as a (small) reduction workflow, in the same way as the full reduction, with Sciline.

[SimonHeybrock]

Investigation: Using TOF Lookup from essreduce Without Sciline for Streaming

Summary: This is feasible with minimal changes

The core TOF lookup system in essreduce is already structured to allow standalone use. The Sciline coupling is primarily in type annotations and wrapper functions, not in the core algorithms.

What Can Be Used Directly (No Changes Needed)

1. TofLookupTable - A plain dataclass with no Sciline dependency (types.py:21-42)

2. load_tof_lookup_table(filename) - Plain function that returns TofLookupTable (workflow.py:17-43)

3. TofInterpolator - Plain class wrapping the interpolation logic (eto_to_tof.py:45-95)

4. _compute_tof_data(da, lookup, ltotal, pulse_stride_offset) - The core conversion function (eto_to_tof.py:396-411)

   • Takes plain sc.DataArray, TofLookupTable, sc.Variable, and int|None
   • Returns plain sc.DataArray

What Requires Sciline (Setup Phase Only)

1. Computing ltotal - The DetectorLtotal computation uses detector_ltotal_from_straight_line_approximation() which depends on detector geometry loaded via NeXus workflows. This is a one-time setup.

2. pulse_stride_offset - See below.

Proposed Pattern (Following RollingDetectorView)

from dataclasses import dataclass
import scipp as sc
from ess.reduce.time_of_flight.types import TofLookupTable
from ess.reduce.time_of_flight.workflow import load_tof_lookup_table
from ess.reduce.time_of_flight.eto_to_tof import _compute_tof_data

@dataclass
class TofConverter:
    """Plain Python wrapper for streaming TOF conversion."""
    lookup: TofLookupTable
    ltotal: sc.Variable
    pulse_stride_offset: int | None = None
    
    @classmethod
    def from_workflow(cls, workflow, detector_key) -> 'TofConverter':
        """Setup via Sciline - one-time expensive computation."""
        # Compute ltotal and load lookup table via Sciline
        lookup = workflow.compute(TofLookupTable)
        ltotal = workflow.compute(DetectorLtotal[SampleRun])
        return cls(lookup=lookup, ltotal=ltotal)
    
    def convert(self, detector_data: sc.DataArray) -> sc.DataArray:
        """Streaming conversion - plain function call."""
        return _compute_tof_data(
            da=detector_data,
            lookup=self.lookup,
            ltotal=self.ltotal,
            pulse_stride_offset=self.pulse_stride_offset,
        )

Avoiding Per-Batch pulse_stride_offset Guessing

The _prepare_tof_interpolation_inputs() function in essreduce will guess pulse_stride_offset on every call if passed None. This can be avoided without modifying essreduce:

• If pulse_stride_offset is provided as a concrete int, no guessing occurs (line 275 check)
• Recommendation: Make pulse_stride_offset a user setting. If plots look wrong, users can adjust it.
• Automatic one-time guessing during setup was considered but rejected: if streaming starts while choppers are not yet in phase, the guess would be wrong and then locked in.

Issues to Consider

1. _compute_tof_data is "private" - The leading underscore suggests it's not part of the public API. Options:

   • Request that essreduce expose it as public (preferred)
   • Accept the coupling to an internal function (pragmatic)

2. ltotal Broadcasting - The _time_of_flight_data_events() function broadcasts ltotal to event shape internally. For streaming, if ltotal is constant, this could potentially be pre-computed once (optimization opportunity).

Effort Estimate

• In esslivedata: Create a TofConverter wrapper class (~50-100 lines)
• In essreduce: Ideally make _compute_tof_data public (or add a thin public wrapper)
• Integration: Similar to how DetectorView uses RollingDetectorView - Sciline for setup, plain methods for streaming

Conclusion

This is very doable. The _compute_tof_data function already does exactly what's needed - it takes plain types and returns plain types. The RollingDetectorView pattern from ess.reduce.live.raw is the exact template to follow. The main question is whether essreduce should expose _compute_tof_data publicly or if we're comfortable depending on the internal function.

[SimonHeybrock]

@nvaytet Considering both alternatives seriously. Using the Sciline-based reduction-workflow mechanism is not totally trivial either, because in particular the DetectorView workflow produces special outputs (current as well as cumulative) and has a fair amount of logic related to ROI handling. This is not a 100% match for how StreamProcessor works (but it could probably be wrapped somehow).

There is also the issue of work-sharing between workflows - for detector views we have the (expensive) GroupIntoPixels preprocessor, which is shared by all detector views. Sharing between different Sciline-based is not possible (there is a shared preprocessor, but the expensive pixel-grouping is in the Sciline workflow).

[nvaytet]

the DetectorView workflow produces special outputs (current as well as cumulative) and has a fair amount of logic related to ROI handling

I probably indeed did not think of everything. What I wrote above was my initial guess. I imagined we already had something in place for reduction workflows, and we could just plug in a different sciline workflow and it would just work. But it sounds like it probably won't "just work".

If you think it's easier/better to do without sciline, then let's go with that. As long as we are not re-implementing any of the code but just using the internals of the time-of-flight module, then I'm happy.

[SimonHeybrock]

As an (slightly simpler than detectors) exercise I refactored the monitor workflow to use the same Sciline+StreamProcessor machinery as the reduction workflows in #655. I could not find a significant slowdown (latency increase), and suspect that may only show up for detectors if we cannot use the simplified GroupIntoPixels but have to use the full pixel-grouping code in the GenericNeXusWorkflow (but as it is unclear whether that could even by used in non-TOA mode that may not be relevant).

This leaves the question of work-sharing (or not) from repeated pixel-grouping, assuming we want to convert all detector_data workflows to use StreamProcessor. Do we need to rethink how we structure things? Make larger workflows with multiple outputs (losing the ability to start/stop/reconfigure subworkflows)? Or allow sharing the "frontend" of the StreamProcessor for shared work?

[SimonHeybrock]

Refactored DetectorView to use Sciline+StreamProcessor in sciline-detector-workflows. It looks a bit verbose and does not support TOF/wavelength yet (but should be ready for it). It works, but I still need to investigate performance more.

Big bonus: This now preserves full history as an accumulated (screen-x,screen-y,TOA) histogram, i.e., we do not lose history when moving a ROI or drawing a new one. The same can also be used to allow selecting a TOA slice for the image view.