Question about reproducing WritingBench leaderboard scores for Qwen2.5-7B-Instruct

[kcc273619-lab]

Hi WritingBench authors,

Thank you for open-sourcing WritingBench and the evaluation scripts.

We are trying to reproduce the length subset result for Qwen2.5-7B-Instruct, and we would like to confirm whether our current setting is comparable to the leaderboard / paper setting.

What we have done:

• We aligned the generation configuration with the official recommendation as closely as possible:
  • temperature = 0.7
  • top_p = 0.8
  • top_k = 20
  • max_tokens = 8192 (limited by our serving platform)
• We also verified the scoring / aggregation logic carefully:
  official generate_response.py + evaluate_benchmark.py + calculate_scores.py

All of these routes consistently give us a result around:

• Overall ≈ 5.50
• length_R ≈ 5.50

The only unavoidable difference we found is in the Claude judge call:

• we use claude_sonnet-4-5-20250929
• for Claude Sonnet 4.5, the provider does not allow temperature and top_p to be specified at the same time
• so for evaluation we used the closest compatible setting:
  • top_p = 0.95
  • max_tokens = 2048
  • temperature omitted, letting the provider default apply

Given that the rest of the pipeline seems aligned, we would like to ask:

1. Is the public leaderboard / paper result for Qwen2.5-7B-Instruct directly comparable to the current open-source evaluation scripts?
2. Was the reported result produced with the released Claude-based evaluation pipeline, or with the WritingBench critic model?
3. Could the difference come from a different judge deployment environment or parameter handling?

We would really appreciate any clarification. Thank you very much!

[A-Quarter-Mile]

Hi,

Thank you for your interest in and use of WritingBench! I also received your email last week and have already replied (it might have ended up in your spam/junk folder).

• The latest leaderboard is available on HuggingFace or ModelScope. The leaderboard results are obtained using Claude. The results reported in the paper are from an earlier version scored by a critic model, and therefore may differ from the current eaderboard. Please refer to the open-source leaderboard as the up-to-date reference.

• Evaluation results may be affected by both the inference configuration used to generate responses and the LLM used for scoring. For example, setting max_tokens to 8192 during response generation may cause responses to be truncated prematurely. In particular, qwen-2.5-7b-instruct tends to produce repetitive content at the end of its responses, which can interfere with the evaluation of the length dimension. If your experiments are constrained by your serving platform, it is sufficient to keep the inference parameters consistent across all models in your evaluation.

Best regards,