Design: uniform HPC/local segregation across the tool surface

[rajeeja]

Background

Feedback from Peng Ding (toolregistry maintainer) during review:

You have some HPC parts and some non-HPC stuff. So it might be worth thinking
how you want to segregate them. Either as different endpoints / two separated
instances, or some clever way to do the auth depending route folding.

This issue captures that as a design item.

The current situation

• HPC dispatch is centralized in tools/remote_tools.py::_run_with_optional_hpc
  (consistent local / remote / fallback semantics, keyed on use_remote,
  endpoint, and execution_mode + path_prefixes).
• Each remote-capable compute op has a paired self-contained remote_* function
  in remote/compute_functions.py (required because Globus Compute ships
  function source to the worker via AllCodeStrategies, which may import only
  uxarray/xarray, not uxarray_mcp).
• Inconsistency: a subset of compute ops have remote variants and a growing
  subset does not. The newly added zonal_anomaly and remap_to_rectilinear
  are local-only, and there is no uniform way for a user or agent to tell which
  operations are HPC-capable from the tool surface.

This was surfaced while testing the Improv endpoint
(caf37dc0-...) end-to-end: the remote path works well (warm round-trips
~1s, validate/probe pass), but zonal_anomaly on an HPC-resident file fails
with FileNotFoundError because it only runs locally.

Options on the table

1. Two separate instances / endpoints (Peng) — a "local" server and an
   "HPC" server, each with a coherent tool set.
   • Pro: clean separation. Con: fragments UX (user must choose a server),
     doubles deployment/config surface.
2. Auth-based route folding (Peng) — one server; local-vs-HPC decided by a
   policy layer keyed on auth/path rather than per-tool flags.
   • Pro: elegant single decision point. Con: significant auth/routing subsystem
     ahead of demonstrated need.
3. Declared HPC-capability as uniform metadata + the existing single dispatch
   point (proposed alternative) — every compute op declares whether it has a
   remote variant; that flag is surfaced in get_capabilities and policy tags;
   _run_with_optional_hpc stays the single venue-decision point (it nearly is
   already). Ops without a remote variant are explicitly tagged "local-only"
   until one is added.
   • Pro: incremental hardening of what exists; makes the surface honest without
     splitting servers or building auth routing. Con: still requires writing a
     remote_* per op when promoting to HPC.

Scope notes

• Adding a remote variant for zonal_anomaly is low-risk (mirrors the existing
  remote_calculate_zonal_mean) and could be the first concrete step once a
  direction is chosen.
• remap_to_rectilinear needs more thought: its output is a structured array
  that must be transferred back from the worker, so the return-size/serialization
  story matters.

Proposed next step

Pick a direction (lean toward option 3 unless multi-instance deployment is a
near-term requirement), then:

• Define an explicit "HPC-capable" property per compute op.
• Surface it in get_capabilities and tool tags so agents/users can see it.
• Backfill remote_* functions for currently local-only ops where it makes
  sense (zonal_anomaly first).

cc @Oaklight (Peng) for the architecture discussion.