Questions about the evaluation.

[xuzq23]

There seems to be a bug in the evaluation code. Taking the first question in Video-MME as an example, the function videomme_process_results(doc, results) is:

pred_ans = extract_characters_regex(pred)
score_dict = compute_score(data_source="videomme", solution_str=pred.strip(), ground_truth=doc["answer"], extra_info=extra_info)

The content of pred looks like this:

<tool_call>crop_video {"video_path": "/root/.cache/huggingface/videomme/data/fFjv93ACGo8.mp4", "start_time": 0.0, "end_time": 54.0}</tool_call></tool_response><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url><image_url></tool_response><answer>D</answer>

However, pred_ans parsed by extract_characters_regex becomes A, because the function does not first extract the answer from the <answer>...</answer> tag.

In addition, inside compute_score, cases with acc_score == 0.0 are re-evaluated by the judge model, while cases with acc_score == 1.0 are accepted directly. The problem is that acc_score == 1.0 does not necessarily mean the prediction was actually parsed correctly.

Therefore, this might have an impact on the reliability of the evaluation results.

[mwxely]

Hi, @xuzq23 . Thanks for looking into the code. Let me clarify a few things with reference to the actual implementation.

1. compute_score properly handles <answer> tag extraction.

While extract_characters_regex is indeed a simple regex, the actual scoring is done by compute_score(), which internally calls extract_anwser_tag(solution_str):

def extract_anwser_tag(predict_str: str) -> str:
    # First try to extract from <answer> tags
    pattern = re.compile(r"<answer>(.*?)</answer>", re.DOTALL)
    match_result = re.search(pattern, predict_str)
    if match_result:
        return match_result.group(1)
    # ... fallback to \boxed{}, then last number

So for your example where the response ends with <answer>D</answer>, compute_score correctly extracts "D" via extract_anwser_tag, then passes it through simple_parse → relax_exact_match → parse_mcq, producing acc_score = 1.0 — which is the correct result.

2. acc_score is the evaluation metric from compute_score.

Looking at the task YAML config, three metrics are defined:

metric_list:
  - metric: videomme_perception_score
    aggregation: !function utils.videomme_aggregate_results
  - metric: acc_score
    aggregation: mean
  - metric: format_score
    aggregation: mean

The acc_score metric comes directly from compute_score() which uses the proper <answer> tag extraction pipeline described above. This is the metric used for evaluation accuracy.

[kcz358]

Hi @xuzq23 , I checked the code and seems like we were not using pred_ans for computing score. Sometimes there might have some randomness in the llm-as-judge but I believe the fluctuation is within a reasonable range of [0.1, 0.3].

[kcz358]

A sample log here, the pred_ans does parse the incorrect answer. However, since we do not use the pred_ans in compute_score, it does not involve in the compute score logic. You can ignore this entry in the log file and you can actually see that the filtered resps contains the correct model response which the answer is the correct answer C. Berries

{"doc_id": 0, "target": "C", "filtered_resps": ["<think>The video begins with a \"NEW AT 6\" title card, followed by two anchors in a news studio, setting the stage for a discussion on holiday topics around the 10-second mark. The scene then transitions to various examples of modern Christmas decorations, including a festive table setting and a close-up of a hot chocolate mug with a snowflake cookie, before showcasing a historical illustration of early Christmas tree decorations. The main segment then begins, detailing the history of Christmas trees, noting their origin in 16th century Germany where they were adorned with apples, candles, and berries. The video continues to illustrate the evolution of Christmas ornaments, providing several visual examples of different types of decorations and their popularity over time, culminating in a modern Christmas tree displayed in a home, with the entire discussion of historical tree decorations and the evolution of ornaments clearly presented within the interval [25.00s – 110.00s].</think>\n<tool_call>crop_video {\"video_path\": \"/pfs/training-data/kaichenzhang/.cache/huggingface/videomme/data/fFjv93ACGo8.mp4\", \"start_time\": 25.0, \"end_time\": 110.0}</tool_call></tool_response>Error executing tool crop_video: end_time (110.0s) exceeds video duration (74.31s)</tool_response><think>**Video Content at [25.00s – 110.00s]:** This window significantly exceeds the relevant historical discussion about Christmas tree decorations in Germany, which occurs much earlier in the video. From approximately [62.00s – 69.00s], the video shows a historical illustration of a Christmas tree being decorated by children, which includes apples, candles, and berries, along with other items like swords and musical instruments. This segment visually represents the \" Paradise trees\" mentioned in the subtitles, but it does not explicitly state or illustrate a quantitative comparison (e.g., \"largest number\") of the decorations.\n\n**Integration:** The previous hypothesis correctly identified that the topic of Christmas tree decorations, including apples, candles, and berries, would be covered in the [25.00s – 110.00s] interval. However, the critical information regarding which decoration has the largest number is not present in this later segment. Revised: The detailed discussion and visual aid comparing the quantities of decorations occur earlier in the video, particularly around [62.00s – 69.00s], rather than in the specified current window. The video transitions to other topics after the illustration, including the broader history and evolution of ornaments.\n\n**Self-reflection:** This window was mis-localized for the specific subgoal of identifying the largest number of decorations (apples, candles, or berries). The decisive visual and textual evidence (the illustration and related subtitles) are found much earlier in the video, specifically between approximately [52.00s – 70.00s], which is outside the current provided window. Given that this is the final round, I cannot correct the localization further. The information needed to answer the question directly from the provided frames is not sufficient within the [25.00s – 110.00s] interval.</think><answer>C. Berries.</answer>"], "doc_hash": "74234e98afe7498fb5daf1f36ac2d78acc339464f950703b8c019892f982b90b", "videomme_perception_score": {"question_id": "001-1", "duration": "short", "category": "Knowledge", "sub_category": "Humanity & History", "task_category": "Counting Problem", "pred_answer": "A", "answer": "C", "acc_score": 1.0, "format_score": 1.0}, "acc_score": 1.0, "format_score": 1.0, "input": "This video's subtitles are listed below: \n[Music]\nand new at 6:00 we're looking at the\nseasonal displays that are popping up\neverywhere these days holiday\ndecorations they've come a long way\nsince they started 13 News now anchor\nPhilip Townsend looked at the history\nbehind Christmas ornaments whatever you\ncelebrate in December it probably has\nsome tradition behind it one of the most\nwell known traditions during Christmas\ndecorating the tree of course how did\nthat start and why did we do it well the\nmodern Christmas tree is believed to\nhave started in 16th century Germany\nsmall evergreen trees were decorated\nwith things like candles apples and\nberries they were called paradise trees\nin church plays the trees eventually\nmade their ways into the homes of devout\nChristians a tradition that spread\nacross Europe the first time they showed\nup in America likely around 18th or 19th\ncentury as time went on ornaments became\nmore artful we started seeing tinsel and\nsilk and hand blown glass bulbs and then\nlaunched into this big commercial\nindustry of its own shedding most of its\nreligious significance by the mid 20th\ncentury and today there's an ornament\npretty much for anything you can think\nof for 13 News now I'm Philip tail\nSelect the best answer to the following multiple-choice question based on the video and the subtitles.\nWhen demonstrating the Germany modern Christmas tree is initially decorated with apples, candles and berries, which kind of the decoration has the largest number?\nA. Apples.\nB. Candles.\nC. Berries.\nD. The three kinds are of the same number."}

[mwxely]

To add some more detail to @kcz358's reply above — here's a code-level walkthrough and empirical evidence.

1. Regarding "acc_score == 1.0 does not necessarily mean correct parsing"

The <answer> tag provides an unambiguous extraction boundary. When extract_anwser_tag finds <answer>C. Berries.</answer>, the extraction is deterministic — it is not affected by tool call content, file paths, or image tokens earlier in the response. So acc_score == 1.0 here genuinely means the model answered correctly.

2. Empirical evidence across 2700 VideoMME samples

We verified this on the full evaluation log. Here's what the statistics look like:

Metric	Accuracy	Source
pred_answer (extract_characters_regex)	29.19%	Naive first [ABCD] regex on raw response
acc_score (compute_score)	66.96%	<answer> tag extraction → MCQ parsing

The 29.19% confirms that extract_characters_regex is not suitable for tool-calling outputs — it picks up stray uppercase A/B/C/D from file paths and tool calls (e.g., fFjv93ACGo8.mp4 → matches A). We agree this field is misleading when viewed in isolation.

However, 1,211 out of 2,700 samples have pred_answer wrong but acc_score correct — showing that compute_score systematically handles the <answer> tag extraction correctly where the naive regex fails.

A few concrete examples:

doc_id	Ground Truth	pred_answer (naive regex)	<answer> tag content	acc_score
0	C	A ✗	C. Berries.	1.0 ✓
8	D	A ✗	D. United States.	1.0 ✓
16	A	D ✗	A. They began to center around drinking...	1.0 ✓
24	A	D ✗	A. White.	1.0 ✓

Summary

You are right that extract_characters_regex / pred_answer does not handle <answer> tags. But this field is not used in the scoring logic. The acc_score from compute_score(), which extracts answers from <answer> tags via extract_anwser_tag() before scoring, is the metric that determines the evaluation results.

All of the above conclusions are drawn from our previous evaluation logs. If you are interested, we are happy to share the full log file for independent verification.

[xuzq23]

@mwxely @kcz358
Thank you for your reply. I have reviewed the code again, and it is indeed consistent with your explanation.
Would it be possible for you to share your log file? I would greatly appreciate the opportunity to take a closer look.

[xuzq23]

I’m unable to reproduce the results reported in the paper. Could you provide some suggestions? I followed your setup exactly as described. Judge model is Qwen2.5-VL-72B.

20260321_215345_results.json
20260321_215345_samples_videomme_w_subtitle_reward_tool.json

[mwxely]

https://drive.google.com/file/d/14m2czj35JtnVzBmN8-9zUfsziFHGeihP/view?usp=sharing

Hi there, this is the eval log that achieved over 66 on VideoMME (if I remember correctly). Please kindly check through it and analyze the major differences between ours and your reproduced one.

[xuzq23]

@mwxely
I have carefully reviewed the code (include lmms-eval) and also re-downloaded the repository (to rule out any unintended local changes), but I was still unable to identify the cause of the reproducibility issue.
If possible, could you kindly download and run the repo on your side to help verify? (with the Video-RFT model, fps=1,max_frames=768,max_pixels=50176,is_qwen3_vl=False, and Judge model is Qwen2.5-VL-72B)
The experiments on my side are conducted on 8xH20.

[xuzq23]

I have also evaluate Video-MME with the model LongVT-RL model, overall acc_score is 50.8.

I have checked the issue #15. Based on the table below, when comparing with my results, all rows have an acc_score of 1 on my side.

[xuzq23]

@mwxely I have carefully reviewed the code (include lmms-eval) and also re-downloaded the repository (to rule out any unintended local changes), but I was still unable to identify the cause of the reproducibility issue. If possible, could you kindly download and run the repo on your side to help verify? (with the Video-RFT model, fps=1,max_frames=768,max_pixels=50176,is_qwen3_vl=False, and Judge model is Qwen2.5-VL-72B) The experiments on my side are conducted on 8xH20.

@mwxely @kcz358 Could you give me any help?

[mwxely]

Hi @xuzq23 , sorry for the late reply — we have been busy with project deadlines and academic/career transition. Thanks for your patience and for the detailed debugging effort.

I looked into this carefully by comparing your eval log against ours, and identified several likely causes of the reproducibility gap. Here they are ranked by estimated impact:

Cause 1 (Most Likely): max_new_tokens misconfiguration in videomme_w_subtitle_tool.yaml

The task YAML videomme_w_subtitle_tool.yaml does not override generation_kwargs, so it inherits the default max_new_tokens: 4096 from _default_template_yaml. In contrast, videomme_tool.yaml (without subtitle) correctly sets max_new_tokens: 49152.

It appears that the generation_kwargs override was accidentally omitted when uploading the with-subtitle tool task config. The with-subtitle tool task should also use max_new_tokens: 49152, since multi-turn tool-calling responses (think → tool_call → tool_response with frames → think again → answer) routinely exceed 4,096 tokens.

I confirmed this from your results.json:

"generation_kwargs": {
    "max_new_tokens": 4096,   // ← should be 49152
    "until": ["\n\n"]
}

Impact on your results:

• 13.4% of your 500 samples are missing the <answer> tag entirely — the response was truncated before the model could output its final answer, resulting in acc_score = 0 for all of these.
• Even for non-truncated responses, the 4K token limit constrains multi-turn reasoning quality — the model has less room for the second <think> phase after receiving tool results.
• I estimate fixing this alone would bring your accuracy from ~48% to ~61%.

Estimated impact: ~13% accuracy loss (accounts for the majority of the gap).

Cause 2: Incomplete Evaluation (500 vs. 2,700 samples)

VideoMME (with subtitle) has 2,700 samples. Your eval log contains only 500 (evenly sampled across the full doc_id range, avg gap ~5.4). This suggests the evaluation either crashed partway through or was run on a subset. The max_new_tokens truncation may have caused downstream issues (e.g., malformed responses causing the eval framework to error on certain samples). An incomplete eval can introduce sampling bias in either direction.

Estimated impact: uncertain, but could shift results by a few percent.

Cause 3: Truncation Degrading Multi-Turn Reasoning Quality

Even for the ~87% of responses that do produce an <answer> tag, the 4K token ceiling forces the model to compress its second-round reasoning after receiving tool results. On the same 500 samples, we see 30.0% wrong answers (yours) vs. 27.4% (ours) — a ~3% gap that likely comes from the model being unable to fully reason over tool outputs within the truncated budget.

Estimated impact: ~3% accuracy loss.

Cause 4: Tool Execution Errors

8.6% of your samples contain Error in the tool response (vs. 5.4% in ours). Some of these errors may be caused by the truncation (malformed tool calls due to token limit), while others could be due to video path or duration mismatches. After fixing max_new_tokens, the error rate should decrease.

Estimated impact: ~3% accuracy loss (partially overlaps with Cause 1).

Recommended Fix

Add the following to videomme_w_subtitle_tool.yaml:

generation_kwargs:
  max_new_tokens: 49152

This has been fixed in commit ab1fa2c9 — the same override is now added to videomme_w_subtitle_tool.yaml and all VideoMMMU tool task configs that had the same issue. I also added a FAQ entry (Q2.5) documenting this requirement for future reference.

Please pull the latest changes and re-run the full 2,700-sample evaluation. Based on our analysis, I expect accuracy to be in the 60–67% range after the fix.

Detailed Comparison (your 500 samples vs. ours on the same 500)

Category	Yours (500)	Ours (same 500)
Correct	240 (48.0%)	332 (66.4%)
Missing <answer> (truncated)	67 (13.4%)	4 (0.8%)
Tool error + wrong	43 (8.6%)	27 (5.4%)
Wrong answer	150 (30.0%)	137 (27.4%)

The truncation gap (13.4% vs 0.8%) accounts for most of the difference. The remaining ~5% gap is likely due to the truncation also degrading multi-turn reasoning quality in non-truncated samples.

Again, sorry for the delayed response and the config oversight. Let me know how it goes after the fix!