feat: add ort-multimodal evaluator for fast multimodal benchmarks

[justinchuby]

Summary

Add LMEvalORTMultimodalEvaluator (ort-multimodal backend) that uses direct ORT InferenceSession for multimodal GenAI packages. This is 15× faster than the ortgenai backend for text-only benchmarks on multimodal models.

Problem

The ortgenai evaluator loads all sub-models (decoder + vision + audio + embedding) and creates a new og.Generator per model call, adding significant overhead for text-only benchmarks like MMLU Pro.

The built-in ort evaluator can't handle models with heterogeneous KV cache head dimensions (e.g. Gemma4 with head_dim=256 for sliding attention and head_dim=512 for full attention) because its Prefill class allocates uniform IOBinding buffers.

Solution

The new ort-multimodal evaluator:

• Reads genai_config.json to locate decoder and embedding ONNX files
• Loads only the decoder and embedding sessions (skips vision/audio)
• Runs embedding model to convert input_ids → inputs_embeds
• Runs decoder with per-layer empty KV cache buffers (supporting heterogeneous head_dim)
• Returns full logits [batch, seq_len, vocab] in a single forward pass

Benchmark

Gemma4 E2B-IT MMLU Pro (12,032 questions, CUDA EP, H200 GPU):

Backend	Accuracy	Throughput	vs ortgenai
PyTorch HF	22.6%	185 req/s	19.5×
ort-multimodal	22.6%	142 req/s	15×
ortgenai	22.6%	9.5 req/s	1×

Full analysis: onnxruntime/mobius#285

Usage

import olive.evaluator.lmeval_ort  # registers backends

from lm_eval.api.registry import get_model
model = get_model('ort-multimodal')(
    pretrained='/path/to/genai/package',
    ep='CUDAExecutionProvider',
    max_length=2048,
)

Or via Olive config:

{"type": "LMEvaluator", "tasks": ["leaderboard_mmlu_pro"], "model_class": "ort-multimodal"}

[justinchuby]

Not generalized

[Copilot]

Pull request overview

This PR adds a new lm-eval backend (ort-multimodal) to run fast text-only benchmarks on ORT GenAI multimodal packages by directly invoking ONNX Runtime InferenceSession for the decoder (+ embedding), avoiding GenAI generator overhead and supporting heterogeneous KV cache head dimensions.

Changes:

• Add LMEvalORTMultimodalEvaluator registered as ort-multimodal in olive.evaluator.lmeval_ort.
• Wire LMEvaluator to dispatch model_class="ort-multimodal" with appropriate init args.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 5 comments.

File	Description
olive/evaluator/olive_evaluator.py	Adds ort-multimodal dispatch path to construct lm-eval model init args.
olive/evaluator/lmeval_ort.py	Implements the ort-multimodal lm-eval backend using direct ORT sessions for decoder/embedding.