Voxtral ASR Pipeline Integration Broken in 4.57.3

[juzhxng]

System Info

• transformers version: 4.57.3
• Platform: Linux-6.1.0-31-amd64-x86_64-with-glibc2.36
• Python version: 3.12.9
• Huggingface_hub version: 0.36.0
• Safetensors version: 0.7.0
• Accelerate version: 1.12.0
• Accelerate config: not found
• DeepSpeed version: not installed
• PyTorch version (accelerator?): 2.9.1+cu128 (CUDA)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using distributed or parallel set-up in script?: no
• Using GPU in script?: yes
• GPU type: NVIDIA A100 80GB PCIe

Who can help?

@Rocketknight1 @ebezzam

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

from transformers import pipeline

asr = pipeline(
    "automatic-speech-recognition",
    model="mistralai/Voxtral-Mini-3B-2507"
)

from transformers import VoxtralForConditionalGeneration, VoxtralProcessor, pipeline

model = VoxtralForConditionalGeneration.from_pretrained("mistralai/Voxtral-Mini-3B-2507")
processor = VoxtralProcessor.from_pretrained("mistralai/Voxtral-Mini-3B-2507")

asr = pipeline(
    "automatic-speech-recognition",
    model=model,
    tokenizer=processor.tokenizer,
    feature_extractor=processor.feature_extractor,
)

result = asr("example.wav")

Expected behavior

The automatic-speech-recognition pipeline should seamlessly handle mistralai/Voxtral models by:

1. The tokenizer should accept standard pipeline arguments (like _from_pipeline) without raising a ValueError
2. The pipeline should automatically generate the required request and the corresponding input_ids

[ebezzam]

@juzhxng thanks for raising this issue.

While Voxtral can perform transcription like an ASR model, it is actually an audio-text-to-text model and can be used with the new any-to-any pipeline. You can see an example with Voxtral here.

You could do transcription with a script like such:

from transformers import pipeline

pipe = pipeline("any-to-any", model="mistralai/Voxtral-Mini-3B-2507")
messages = [
     {
         "role": "user",
         "content": [
             {
                 "type": "audio",
                 "url": "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/obama.mp3",
             },
             {"type": "text", "text": "Transcribe this text"},
         ],
     },
]


outputs = pipe(text=messages, max_new_tokens=None, return_full_text=False)
print(outputs[0]["generated_text"])

However, for transcription the recommended approach is to directly use the model and its processor (see here under "Transcription"). As far as I know, this isn't supported with pipeline.

cc @zucchini-nlp, @eustlb