CareQA (Closed and Open-Ended English)

[warner-benjamin]

Implement verifiers env for CareQA (Closed and Open-Ended English)

Implement a verifiers environment for the CareQA (Closed and Open-Ended English) datasets. Use the authors’ question, system, and LLM-as-judge prompts if provided. Only create a training dataset if a training split exists.

Paper: https://arxiv.org/abs/2502.06666
Dataset / Project: https://huggingface.co/datasets/HPAI-BSC/CareQA

Format & Env

• Medical QA with multiple-choice answers:
  • Use vf.SingleTurnEnv + MCQ accuracy
• Single-turn open-ended clinical Q&A:
  • Use vf.SingleTurnEnv + vf.JudgeRubric()

[Arya-Hari]

@warner-benjamin I can take this up. #28 is already being handled by @mkieffer1107 I think.

[Arya-Hari]

@warner-benjamin just a clarification, won't the MCQ subset from this dataset be a single-turn env? There's just four choices and one fixed answer. Same for the open-ended as well

[warner-benjamin]

@warner-benjamin just a clarification, won't the MCQ subset from this dataset be a single-turn env? There's just four choices and one fixed answer. Same for the open-ended as well

Yes, I fixed the issue

[warner-benjamin]

Resolved by #48