tiatoolbox-qupath-extension

A QuPath extension that runs TIAToolbox inference engines from inside QuPath. Inference is performed by tiatoolbox's Python engines. The extension provides the QuPath-side GUI and scripting API, manages a Python sidecar process, and imports the resulting annotations into the open image.

Requirements

• QuPath 0.7.0 or compatible.
• JDK 21+ — only required to build the extension. End users do not need a JDK installed, QuPath ships its own JRE.

Installation

Recommended (via the QuPath extension catalog)

From inside QuPath:

1. Extensions → Manage extension catalogs.
2. Add a catalog, paste the URL of the TIA catalog:

   https://github.com/TissueImageAnalytics/qupath-catalog
   

3. Find TIAToolbox extension in the catalog list and click Install.

Then set up the Python runtime (see Python setup below).

Developers (local build)

From a clone of this repo:

./gradlew clean jar
# → build/libs/qupath-extension-tiatoolbox-<version>.jar

Add the JAR into QuPath using whichever you prefer:

• Drag-and-drop the JAR onto the running QuPath window, or
• Copy the JAR into the QuPath user-extensions folder. By default this is <QuPath dir>/extensions/ (The exact path is shown under Edit → Preferences → User directory). QuPath auto-loads any JAR placed in this folder.

Then, restart QuPath.

Python setup

Inside QuPath, click Extensions → TIAToolbox → Install Python runtime… and press Install.

The wizard sets up an isolated Python environment under your QuPath user directory (<QuPath user dir>/tiatoolbox-runtime/) using uv. It installs Python, TIAToolbox, and the small qupath-tiatoolbox sidecar package.

If setup is interrupted, or if you need to refresh the environment after an extension update, open the same dialog and click Re-install.

Usage

GUI

1. Open a whole-slide image (or a QuPath project).
2. Extensions → TIAToolbox → Run TIAToolbox…
3. Choose a model and device (cpu, cuda, or mps), adjust batch size if needed.
4. Choose the Run on scope:
   • Current image — runs on the active viewer.
   • All project images — iterates every entry in the open project, saving the resulting hierarchy back into each .qpdata so results survive a crash mid-batch.
5. Click Run. Progress is shown per image. Click Cancel to stop at the next image.

The first run for a given model downloads its pretrained weights from the HuggingFace repository TIACentre/TIAToolbox_pretrained_weights.

Scripting

The extension exposes a Groovy-friendly API. Four templates are shipped under Extensions → TIAToolbox → Script templates:

• Patch classification: resnet18-kather100k on the current image.
• Nucleus segmentation: hovernet_fast-pannuke on the current image.
• Batch process project: iterates project.getImageList() and saves each entry's hierarchy.

The underlying API is easy to use:

import qupath.ext.tiatoolbox.TIAToolbox

TIAToolbox.builder()
    .model("resnet18-kather100k")
    .device("cpu")            // "cpu" | "cuda" | "mps"
    .batchSize(8)
    .build()
    .run()                    // active viewer image; returns int (objects added)

run(ImageData) and run(ImageData, ProgressListener) overloads are available for batch loops. The Python path used is the uv-managed runtime installed by the extension, unless overridden with .pythonExecutable(...) on the builder.

Models included

Model	Engine	Output
resnet18-kather100k	PatchPredictor	Patch-level colorectal tissue classification (9 classes).
resnet18-pcam	PatchPredictor	Binary lymph-node metastasis classification (tumor vs. negative).
resnet34-idars-msi	PatchPredictor	IDaRS microsatellite-instability biomarker prediction (MSS / MSI).
fcn-tissue_mask	SemanticSegmentor	Foreground tissue / background mask.
hovernet_fast-pannuke	MultiTaskSegmentor	Per-nucleus polygons with 6 type classes (PanNuke).
hovernetplus-oed	MultiTaskSegmentor	Nuclei + epithelial layer segmentation, OED dataset.
KongNet_CoNIC_1	NucleusDetector	KongNet 6-class nucleus point detection on colorectal H&E (CoNIC).
KongNet_PanNuke_1	NucleusDetector	KongNet 5-class nucleus point detection (PanNuke).
KongNet_MONKEY_1	NucleusDetector	KongNet immune-cell point detection on PAS-stained kidney biopsies (MONKEY challenge).
KongNet_Det_MIDOG_1	NucleusDetector	KongNet mitotic-figure point detection on H&E (MIDOG).
KongNet_PUMA_T1_3	NucleusDetector	KongNet 3-class point detection on melanoma H&E (PUMA Track 1).
KongNet_PUMA_T2_3	NucleusDetector	KongNet 10-class point detection on melanoma H&E (PUMA Track 2).
mapde-conic	NucleusDetector	MapDe single-class nucleus point detection (CoNIC).
mapde-crchisto	NucleusDetector	MapDe single-class nucleus point detection (CRCHisto).

NucleusDetector outputs points (centroids) rather than polygons, so it imports into QuPath as Detection objects at the detected coordinates. This is the faster option when you only need cell counts or density — for full nucleus masks, use a MultiTaskSegmentor model instead.

The list is curated from tiatoolbox's pretrained_model.yaml. To add or remove models, edit src/main/resources/qupath/ext/tiatoolbox/ui/models.json and rebuild the JAR. Any pretrained model accepted by the corresponding tiatoolbox engine works (see PatchPredictor, SemanticSegmentor, MultiTaskSegmentor, NucleusDetector).

Architecture

The project has two halves: a Java extension that runs inside QuPath, and a Python sidecar that wraps the tiatoolbox engines. They communicate over Py4J in ClientServer mode. Python is the server hosting a TIATask entry point, the JVM is the client.

┌──────────── QuPath (JVM) ─────────┐  Py4J   ┌──── Python sidecar ────┐
│  TIAToolboxExtension (menus)      │ ◄─────► │  qupath_tiatoolbox     │
│  TIAToolbox (scripting API)       │         │   bridge.TIATask       │
│  TIAController (FXML dialog)      │         │   runners.run_engine   │
│  BridgeManager (sidecar lifecycle)│         │   tiatoolbox.engine    │
│  ResultImporter (GeoJSON → hier.) │         │                        │
└───────────────────────────────────┘         └────────────────────────┘

The UI and Groovy scripting go through a single code path: TIAToolbox.run(imageData, listener). TIAToolbox owns a process-wide BridgeManager. The first call spawns the Python sidecar. Subsequent calls (including across Groovy scripts and batch images) reuse it. The sidecar is restarted only if the configured Python interpreter changes, and is shut down via a JVM hook on QuPath exit.

Per call, the sidecar invokes the matching tiatoolbox engine with output_type="qupath", which writes a GeoJSON output file. Java reads that file and adds the objects to the QuPath hierarchy.

Repository layout

.
├── build.gradle.kts                         # Java/Gradle build
├── settings.gradle.kts
├── gradle/                                  # Gradle wrapper
├── src/main/
│   ├── java/qupath/ext/tiatoolbox/
│   │   ├── TIAToolboxExtension.java         # entry point: menus + script templates
│   │   ├── TIAToolbox.java                  # public scripting API (builder) + shared run() path
│   │   ├── core/                            # bridge, wire format, and import logic
│   │   │   ├── BridgeManager.java           # owns the Py4J ClientServer + Python subprocess
│   │   │   ├── PythonLauncher.java          # spawns `python -m qupath_tiatoolbox`
│   │   │   ├── TiaRunner.java               # Java view of the Python TIATask interface
│   │   │   ├── InferenceRequest.java        # JSON sent to Python
│   │   │   ├── InferenceResponse.java       # JSON returned by Python
│   │   │   ├── ProgressListener.java        # status / heartbeat callbacks (Python → Java)
│   │   │   └── ResultImporter.java          # GeoJSON → QuPath hierarchy, reapplies PathClass
│   │   ├── install/                         # uv-managed runtime installation
│   │   │   ├── RuntimeInstaller.java         # extracts uv/sidecar and runs `uv sync`
│   │   │   └── RuntimePaths.java             # resolves runtime paths under the QuPath user dir
│   │   └── ui/                              # JavaFX dialog (scope radios, model picker, progress)
│   │       ├── RuntimeInstallCommand.java
│   │       ├── RuntimeInstallController.java
│   │       ├── TIACommand.java
│   │       ├── TIAController.java
│   │       ├── TIAPrefs.java
│   │       └── ModelInfo.java
│   └── resources/
│       ├── META-INF/services/qupath.lib.gui.extensions.QuPathExtension
│       └── qupath/ext/tiatoolbox/
│           ├── scripts/                      # Groovy script templates
│           │   ├── PatchClassification.groovy
│           │   ├── NucleusSegmentation.groovy
│           │   └── BatchProcessProject.groovy
│           └── ui/{tiatoolbox_control.fxml, runtime_install.fxml, strings.properties, models.json}
├── runtime/
│   └── pyproject.toml                       # uv-managed runtime environment
└── python/
    ├── pyproject.toml                       # qupath-tiatoolbox package
    └── src/qupath_tiatoolbox/
        ├── __init__.py
        ├── __main__.py                      # python -m qupath_tiatoolbox
        ├── bridge.py                        # Py4J ClientServer entry point
        └── runners.py                       # engine dispatch

Wire protocol

The Java side sends a JSON request to the Python sidecar and receives a JSON response. The contract is defined by InferenceRequest.java and InferenceResponse.java on the Java side, and consumed by runners.run_engine on the Python side. Both sides share one open Py4J connection across calls.

// request: Java → Python
{
  "engine":      "patch_predictor",       // also: "semantic_segmentor", "multi_task_segmentor", "nucleus_detector"
  "model":       "resnet18-kather100k",   // any model accepted by the engine
  "wsi_path":    "/abs/path/to/slide.svs",
  "save_dir":    "/tmp/tia-{uuid}",       // GeoJSON is written here
  "device":      "cpu",                   // "cpu" | "cuda" | "mps"
  "batch_size":  8,
  "num_workers": 0,
  "classes":     ["Tumor", "Stroma", "Immune cells", "..."]
                                          // remaps numeric labels to QuPath class names
}

// response: Python → Java
{ "status": "ok",    "geojson": ["/tmp/tia-{uuid}/0.geojson"] }
{ "status": "error", "message": "...", "trace": "..." }

While inference is running, the sidecar invokes two callbacks on the Java-side ProgressListener over the same Py4J connection:

• onStatus(String) — at significant transitions (model load, tiling, postprocessing, GeoJSON write).
• onHeartbeat(int elapsedSeconds) — roughly every two seconds with the elapsed running time

For batch runs, the Java side iterates BatchEntrys sequentially and reuses the same connection (one request per image, with save_dir parameterised per call).

Current limitations

• Inference on region of interests is not yet supported.
• The slide(s) must have a real file path on disk.
• Only one inference run at a time per QuPath instance.

License

See LICENSE.