[Contribution] TimeImageProcessor for multimodal pipelines

[joshuasteier]

Contributor Information

• Name: Joshua Steier
• Contribution Type: Processor + Documentation + Example

Description

Added TimeImageProcessor, a time-aware image processor for multimodal PyHealth pipelines. This processor pairs image loading with temporal metadata, enabling tasks where patients have multiple images taken at different times (e.g., serial chest X-rays during an ICU stay).

Takes a tuple (List[image_path], List[time_diff_from_admission]) and returns (N×C×H×W image tensor, N timestamp tensor, "image") for the unified multimodal embedding model.

Features:

• Chronological sorting by timestamp
• Optional max_images truncation (keeps most recent)
• fit() infers channel count from mode or data
• size() method matching TimeseriesProcessor pattern
• Registered as "time_image" via @register_processor
• Reuses ImageProcessor's transform pipeline (resize, normalize, mode conversion)

This processor is part of the multimodal embedding pipeline:

TimeImageProcessor → VisionEmbeddingModel → MultimodalEmbeddingModel → BottleneckTransformer

Files to Review

• pyhealth/processors/time_image_processor.py — processor implementation
• pyhealth/processors/__init__.py — import + __all__ entry
• tests/core/test_time_image_processor.py — 22 unit tests
• examples/time_image_processor_tutorial.ipynb — end-to-end tutorial notebook
• docs/api/processors/pyhealth.processors.TimeImageProcessor.rst — API docs
• docs/api/processors.rst — added to processor listing and toctree

Testing

python -m pytest tests/core/test_time_image_processor.py -v
# Result: 22 passed

Usage in task schema

input_schema = {
    "chest_xray": ("time_image", {
        "image_size": 224,
        "mode": "RGB",
        "normalize": True,
        "mean": [0.485, 0.456, 0.406],
        "std": [0.229, 0.224, 0.225],
        "max_images": 8,
    }),
}

Running the example

jupyter notebook examples/time_image_processor_tutorial.ipynb

[jhnwu3]

Lgtm , we can iterate on this later if need be!