Retrieval corpus?

[okhat]

Thanks for the awesome work! I'm not sure what retrieval corpus is used in the DPR results. The paper mentions 2.9M entities and I can see that they seem to be in the intersection of Wikipedia and Wikidata, but what precise corpus is indexed and searched?

[vzhong]

Hi @okhat , the retrieval corpus are the documents from T-REX: https://hadyelsahar.github.io/t-rex/

[okhat]

I’m thinking about tackling your task in a more realistic context by doing retrieval from all of Wikipedia. Is there any reason that sounds like a bad idea? What Wikipedia year would be most appropriate? 2016, 2018, 2023?

[vzhong]

I think we used a Wikidata release from August 2022, so anything near that time period would be preferable. What you’re trying to do sounds reasonable to me - doesn’t sound like a bad idea. On Jan 6, 2023, at 2:49 PM, Omar Khattab ***@***.***> wrote: I’m thinking about tackling your task in a more realistic context by doing retrieval from all of Wikipedia. Is there any reason that sounds like a bad idea? What Wikipedia year would be most appropriate? 2016, 2018, 2023? —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: ***@***.***>

[okhat]

Thank you! Closing then.

[okhat]

By the way, is the T-REX corpus you used part of the downloadable (or easy to recreate) steps? Or do we need to download all of Wikidata, etc.?

[vzhong]

It is part of the downloadable steps. To access this data you either

• download preprocessed files
• reconstruct the dataset by downloading its raw components (wikidata, t-rex)
  If you do the latter, we do not have guarantees that the reconstruction will be identical to the released dataset because Wikidata is constantly changing (and they do not store all old dumps)