Data acquisition on multiple computers

[AustinHouston]

Rethink acquisition dataflow for Gatan EELS and multi-computer setups

As we add EELS acquisition through Gatan DigitalMicrograph, we need to revisit the current acquisition/data workflow.

Current synchronous flow:

1. Acquire with hardware
2. Generate data key/path
3. Save data to disk
4. Register that path with Tiled
5. Return key to client only after registration completes

This becomes problematic when data is produced on multiple computers, e.g. AutoScript PC + Gatan PC. It pushes us toward either:

1. Separate storage: each computer saves locally and has its own Tiled server
2. Unified storage: all data is transferred to one computer before save/register

Both introduce transfer or registration latency into the acquisition loop.

Proposal

Decouple acquisition from data availability.

New flow:

1. Acquisition is called
2. Hardware acquires
3. A data reference/key is generated immediately
4. Key/reference is returned to the client
5. Data save and Tiled registration continue asynchronously

The client is then free to continue the experiment immediately. If the user tries to read before the data is ready, .read() may fail or report unavailable data. To make this robust, we should add a client-side .wait() helper:

ref = microscope.acquire_eels()
data.wait(ref)
spectrum = data.open(ref).read()

[utkarshp1161]

Thanks, this sounds great.

I will add comments if I have more relevant thoughts.

Just for completeness, I'll mention a few additional items:

• Adding the EELS detector (https://github.com/pycroscopy/asyncroscopy/blob/main/asyncroscopy/detectors/EELS.py) would be a great exercise. It should also help verify whether the detector extension guide needs any updates:
  https://github.com/pycroscopy/asyncroscopy/blob/main/docs/Adding_New_Hardware/add_detector.md

• run_servers.py will likely need changes as we add another detector.

• If run_servers.py changes, this documentation should be updated accordingly:
  https://github.com/pycroscopy/asyncroscopy/blob/main/docs/Operation/run-servers.md

• I suspect EELS data handling will require changes in:
  https://github.com/pycroscopy/asyncroscopy/blob/main/asyncroscopy/software/DataWriter.py

• Once DataWriter changes, the corresponding documentation should also be updated:
  https://github.com/pycroscopy/asyncroscopy/blob/main/docs/Tiled_server/data_integration.md

• tests - Need more thought --> maybe test_EELS.py?