📊 RefChartQA: Grounding Visual Answer on Chart Images

Alexander Vogel · Omar Moured · Yufan Chen · Jiaming Zhang · Rainer Stiefelhagen

 

Our paper is accepted at ICDAR 2025 !

🧠 About

RefChartQA is a large-scale benchmark for visual grounding in chart-based question answering. It extends the ChartQA and TinyChart-PoT datasets by adding explicit bounding box annotations that link each answer to supporting visual elements in the chart. RefChartQA contains 73,702 annotated samples, including:

• 55,789 training,
• 6,223 validation,
• 11,690 testing instances.

✅ To-Do

• [March 28, 2025] Upload dataset to Hugging Face 🤗
• [July 09, 2025] Released evaluation scripts.
• Add visualization tools for bounding box inspection.

🛠️ Usage

📦 Environment Setup

Make sure you have Python installed, then install the required libraries:

pip install datasets pillow

📂 Load the Dataset

from datasets import load_dataset

dataset = load_dataset("omoured/RefChartQA")

# Print dataset splits and one example
print(dataset)
print(dataset["train"][0])

Evaluation Script

Note: in the evaluation folder, you can find an example "filtered_results.jsonl" file showing the appropriate format.

python evaluate.py --result_file filtered_results.jsonl

📚 Citation

@misc{vogel2025refchartqagroundingvisualanswer,
      title={RefChartQA: Grounding Visual Answer on Chart Images through Instruction Tuning}, 
      author={Alexander Vogel and Omar Moured and Yufan Chen and Jiaming Zhang and Rainer Stiefelhagen},
      year={2025},
      eprint={2503.23131},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2503.23131}, 
}