Request for intermediate data in the dataset construction

[Su-my]

Thanks for your excellent work!

I noticed that the dataset released on Hugging Face mainly contains the final training annotations (e.g., reasoning traces and tool-augmented sequences). I am particularly interested in better understanding the data construction pipeline, especially the intermediate stages.

Specifically, I am wondering:

1. Are the video clip-level captions used during dataset construction available?
2. Is there access to the ground-truth temporal annotations (i.e., start/end timestamps of relevant segments)?

[mwxely]

Hi @Su-my , thanks for your interest in our work!

Regarding Q1 (video clip-level captions from the data construction pipeline):

Unfortunately, we do not have these intermediate artifacts readily available. During this project, we migrated GPU clusters, and some intermediate data products (including clip-level captions from the construction pipeline) were not carried over. Additionally, prior to the public release, we made a deliberate decision not to include certain intermediate outputs, as some of the data cleaning and processing stages produced artifacts that were not sufficiently polished for release.

That said, this part of the data pipeline was a collaborative effort — cc @XIAO4579 and @kcz358 who were closely involved in the data construction. They may have additional records or backups of these intermediate stages. We will follow up here if anything turns up.

From our observation, not many repos in the video understanding realm have fully open-sourced their data construction pipelines — we hope that our released pipeline and code can serve as a good foundation for building precisely annotated video datasets for your own use cases. In the meantime, the paper (Section 3.1 & Appendix) describes the clip-level captioning procedure in detail, which should be sufficient to reproduce the pipeline if needed.

Regarding Q2 (ground-truth temporal annotations):

The RL training split longvt_rl_selfqa_1k6.parquet does contain ground-truth start/end timestamps for relevant segments. These temporal annotations are used during training to compute the joint temporal grounding reward. You can inspect the messages field in the parquet file to find the annotated timestamps.

[Su-my]

Thank you very much for your detailed response! I got another question. In the paper, you mentioned that the SFT split contains 228,835 non-tool samples and 19,161 tool-augmented samples. During evaluation/inference, is the model always expected to perform tool-augmented reasoning?

[mwxely]

Good question! No, the model does not always invoke tool-augmented reasoning during inference — whether and how frequently the model calls tools varies depending on the benchmark and the nature of the queries.

As expected, we observe that benchmarks involving longer videos or requiring finer temporal localization tend to elicit more frequent tool calls, as the model learns that the initial visual context is insufficient and actively requests additional temporal segments. On benchmarks with shorter videos or more straightforward questions, the model may choose to answer directly without tool use.

The specific tool-calling patterns generalize differently across benchmarks — the model is making its own decisions about when tools are beneficial based on the input context, rather than following a fixed protocol. That said, across all our evaluated benchmarks, enabling tool-augmented inference consistently provides a net positive gain over the no-tool baseline, as shown in our paper (Table 2).