A question about Persona Chat datatset.

[StevenTang1998]

Hello, I have downloaded the Persona Chat dataset in the glge website and used the easy version (full data). When testing my own BART-base code following basic seq2seq settings, the BLEU result of Persona Chat is far below the one of BART-large provided in your paper.
BLEU-1: 12.79, BLEU-2: 7.81

While I use the same code to test dataset GigaWord, the result is expected.
ROUGE-1: 37.98, ROUGE-2: 19.18

So I am confused about the evaluation of Persona Chat, and I get that this is a multi-turn dialog dataset. Hence, I want to know if there is any difference in the training or evaluation settings.

[dayihengliu]

Hello, I have downloaded the Persona Chat dataset in the glge website and used the easy version (full data). When testing my own BART-base code following basic seq2seq settings, the BLEU result of Persona Chat is far below the one of BART-large provided in your paper.
BLEU-1: 12.79, BLEU-2: 7.81

While I use the same code to test dataset GigaWord, the result is expected.
ROUGE-1: 37.98, ROUGE-2: 19.18

So I am confused about the evaluation of Persona Chat, and I get that this is a multi-turn dialog dataset. Hence, I want to know if there is any difference in the training or evaluation settings.

You can try the script in https://github.com/microsoft/ProphetNet/blob/master/GLGE_baselines/script/script/evaluate/personachat/eval.py to evaluate the results of Persona Chat. This evaluation script is also used in https://github.com/PaddlePaddle/Research/tree/master/NLP/Dialogue-PLATO.

[StevenTang1998]

Thanks for your solution, the difference is the smooth function of BLEU. I use method0, however the script uses method7.

[StevenTang1998]

And I also have tested in the cnndm dataset. I use the script in https://github.com/microsoft/ProphetNet/blob/master/GLGE_baselines/script/eval.py.
However, I find the ROUGE-1 and ROUGE-2 score are expected, while the ROUGE-L is lower than expected with a large margin. The script result of gigaword dataset is normal and the result of files2rouge in cnndm is also normal, so I guess the preprocess the cnndm may have some mistakes about eos?

[yuyan2do]

@StevenTang1998 Could you share the generated data and golden file which you observed a low ROUGE-L.

[StevenTang1998]

OK, I sended them to glge@microsoft.com two days ago.
The result evaluated by me shows as follows:

• cnndm1
  rouge1/rouge2/rougeL 30.80/7.70/23.77 (file2rouge)
  rouge1/rouge2/rougeL 30.80/7.70/19.62 (your eval.py)

• cnndm2
  rouge1/rouge2/rougeL 31.43/8.08/24.18 (file2rouge)
  rouge1/rouge2/rougeL 31.43/8.07/19.96 (your eval.py)

• cnndm3
  rouge1/rouge2/rougeL 42.39/19.94/34.62 (file2rouge)
  rouge1/rouge2/rougeL 42.39/19.94/29.87 (your eval.py)

• bart-large (your paper and BART paper)
  rouge1/rouge2/rougeL 44.16/21.28/40.90

• t5-base (T5 paper)
  rouge1/rouge2/rougeL 42.05/20.34/39.40

You can see that the results of rouge1/2 are the same and expected, however, the results of rougeL are different and not expected, even with files2rouge. So I want to know how to get a correct rougeL score as the your paper and other papers like T5.

[yuyan2do]

Please use our data and evaluation script together. Mix data and evaluation method (inconsistence of line separator) causes the ROUGE-L difference.

[StevenTang1998]

I have tried in many ways, however, I still cannot get the consistent results.
I have sent the output of CNN_DM to your email.

rouge1/rouge2/rougeL 36.36/20.87/33.32 (file2rouge)
rouge1/rouge2/rougeL 28.25/14.18/26.17 (your eval.py)

[StevenTang1998]

I'm sorry to bother, but could you give me some guidance?

[dayihengliu]

I'm sorry to bother, but could you give me some guidance?

@yuyan2do Maybe you can provide some guidance?