From bad9ec6a9e45e30854ea4e597a56997e333d305b Mon Sep 17 00:00:00 2001 From: fossabot Date: Wed, 22 Sep 2021 11:20:52 -0700 Subject: [PATCH 1/3] Add license scan report and status Signed off by: fossabot --- README.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/README.md b/README.md index 7fd0f91..4dcce83 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,6 @@ # BDG(Distractor Generation) +[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Fvoidful%2FBDG.svg?type=shield)](https://app.fossa.com/projects/git%2Bgithub.com%2Fvoidful%2FBDG?ref=badge_shield) + Code for "A BERT-based Distractor Generation Scheme with Multi-tasking and Negative Answer Training Strategies." [Paper](https://www.aclweb.org/anthology/2020.findings-emnlp.393/) @@ -157,3 +159,7 @@ python ./examples/run_multiple_choice.py \ ``` + + +## License +[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Fvoidful%2FBDG.svg?type=large)](https://app.fossa.com/projects/git%2Bgithub.com%2Fvoidful%2FBDG?ref=badge_large) \ No newline at end of file From 41fd3f1a48535d72c0c6b0b5f47237167a9d3b1a Mon Sep 17 00:00:00 2001 From: "Restyled.io" Date: Wed, 22 Sep 2021 18:21:01 +0000 Subject: [PATCH 2/3] Restyled by prettier-markdown --- README.md | 98 +++++++++++++++++++++++++++++++++++-------------------- 1 file changed, 63 insertions(+), 35 deletions(-) diff --git a/README.md b/README.md index 4dcce83..b795030 100644 --- a/README.md +++ b/README.md @@ -1,23 +1,25 @@ # BDG(Distractor Generation) + [![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Fvoidful%2FBDG.svg?type=shield)](https://app.fossa.com/projects/git%2Bgithub.com%2Fvoidful%2FBDG?ref=badge_shield) -Code for "A BERT-based Distractor Generation Scheme with Multi-tasking and Negative Answer Training Strategies." +Code for "A BERT-based Distractor Generation Scheme with Multi-tasking and +Negative Answer Training Strategies." [Paper](https://www.aclweb.org/anthology/2020.findings-emnlp.393/) ## V2 -Updated result using BART. BART model is uploaded in HuggingFace model hub. -| model | BLEU1 | BLEU2 | BLEU3 | BLEU4 | ROUGEL | -|---------------|-------|-------|-------|-------|--------| -| BERT DG | 35.30 | 20.65 | 13.66 | 9.53 | 31.11 | -| BERT DG pm | 39.81 | 24.81 | 17.66 | 13.56 | 34.01 | -| BERT DG an+pm | 39.52 | 24.29 | 17.28 | 13.28 | 33.40 | -| BART DG | 40.76 | 26.40 | 19.14 | 14.65 | 35.53 | -| BART DG pm | 41.85 | 27.45 | 20.47 | 16.33 | 37.15 | -| BART DG an+pm | 40.26 | 25.86 | 18.85 | 14.65 | 35.64 | -* higher is better + +Updated result using BART. BART model is uploaded in HuggingFace model hub. | +model | BLEU1 | BLEU2 | BLEU3 | BLEU4 | ROUGEL | +|---------------|-------|-------|-------|-------|--------| | BERT DG | 35.30 | +20.65 | 13.66 | 9.53 | 31.11 | | BERT DG pm | 39.81 | 24.81 | 17.66 | 13.56 | +34.01 | | BERT DG an+pm | 39.52 | 24.29 | 17.28 | 13.28 | 33.40 | | BART DG | +40.76 | 26.40 | 19.14 | 14.65 | 35.53 | | BART DG pm | 41.85 | 27.45 | 20.47 | +16.33 | 37.15 | | BART DG an+pm | 40.26 | 25.86 | 18.85 | 14.65 | 35.64 | + +- higher is better | model | Count BLEU1 > 0.95 | -|---------------|--------------------| +| ------------- | ------------------ | | BERT DG | 115 | | BERT DG pm | 57 | | BERT DG an+pm | 43 | @@ -25,15 +27,19 @@ Updated result using BART. BART model is uploaded in HuggingFace model hub. | BART DG pm | 60 | | BART DG an+pm | 23 | | Gold | 12 | -* lower is better + +- lower is better ## Trained Model and Code Example + ### BART + Distractor: https://huggingface.co/voidful/bart-distractor-generation Distractor PM: https://huggingface.co/voidful/bart-distractor-generation-pm -Distractor AN+PM: https://huggingface.co/voidful/bart-distractor-generation-both +Distractor AN+PM: https://huggingface.co/voidful/bart-distractor-generation-both + +### BERT -### BERT Trained model available on release: https://github.com/voidful/BDG/releases/tag/v1.0 @@ -42,7 +48,8 @@ https://colab.research.google.com/drive/1yA3Rex9JHKJmc52E3YdsBQ4eQ_R6kEZB?usp=sh ## Citation -If you make use of the code in this repository, please cite the following papers: +If you make use of the code in this repository, please cite the following +papers: @inproceedings{chung-etal-2020-BERT, title = "A {BERT}-based Distractor Generation Scheme with Multi-tasking and Negative Answer Training Strategies.", @@ -59,21 +66,27 @@ If you make use of the code in this repository, please cite the following papers abstract = "In this paper, we investigate the following two limitations for the existing distractor generation (DG) methods. First, the quality of the existing DG methods are still far from practical use. There are still room for DG quality improvement. Second, the existing DG designs are mainly for single distractor generation. However, for practical MCQ preparation, multiple distractors are desired. Aiming at these goals, in this paper, we present a new distractor generation scheme with multi-tasking and negative answer training strategies for effectively generating \textit{multiple} distractors. The experimental results show that (1) our model advances the state-of-the-art result from 28.65 to 39.81 (BLEU 1 score) and (2) the generated multiple distractors are diverse and shows strong distracting power for multiple choice question.", } - ## Environment Setup + ```bash pip install -r requirement.txt ``` -## Data Preprocessing +## Data Preprocessing + Inside `data_preprocessing` folder. -Download dataset [here](https://github.com/Yifan-Gao/Distractor-Generation-RACE), put it into `distractor` folder. +Download dataset +[here](https://github.com/Yifan-Gao/Distractor-Generation-RACE), put it into +`distractor` folder. run `convert_data.py` to do preprocessing. -run `dataset_stat.py` for dataset statistics. +run `dataset_stat.py` for dataset statistics. ## Train Distractor Generator + ### BART -using tfkit==0.7.0 and transformers==4.4.2 + +using tfkit==0.7.0 and transformers==4.4.2 + ```bash tfkit-train --savedir ./race_cqa_gen_d_bart/ --train ./race_train_updated_cqa_dsep_a_bart.csv --test ./race_test_updated_cqa_dsep_a_bart.csv --model seq2seq --config facebook/bart-base --batch 9 --epoch 10 --grad_accum 2 --no_eval; tfkit-train --savedir ./race_cqa_gen_d_bart_pm/ --train ./race_train_updated_cqa_dsep_a_bart.csv --test ./race_test_updated_cqa_dsep_a_bart.csv --model seq2seq --config facebook/bart-base --batch 9 --epoch 10 --grad_accum 2 --no_eval --likelihood pos; @@ -81,46 +94,62 @@ tfkit-train --savedir ./race_cqa_gen_d_bart_both/ --train ./race_train_updated_c ``` ### BERT -using environment from `requirement.txt` -run the following in main dir: + +using environment from `requirement.txt` +run the following in main dir: + ### Train BDG Model + ```bash tfkit-train --maxlen 512 --savedir ./race_cqa_gen_d/ --train ./data_preprocessing/processed_data/race_train_updated_cqa_dsep_a.csv --test ./data_preprocessing/processed_data/race_test_updated_cqa_dsep_a.csv --model onebyone --tensorboard --config bert-base-cased --batch 30 --epoch 6; ``` + ### Train BDG AN model + ```bash tfkit-train --maxlen 512 --savedir ./race_cqa_gen_d_an/ --train ./data_preprocessing/processed_data/race_train_updated_cqa_dsep_a.csv --test ./data_preprocessing/processed_data/race_test_updated_cqa_dsep_a.csv --model onebyone-neg --tensorboard --config bert-base-cased --batch 30 --epoch 6; ``` + ### Train BDG PM model + ```bash tfkit-train --maxlen 512 --savedir ./race_cqa_gen_d_pm/ --train ./data_preprocessing/processed_data/race_train_updated_cqa_dsep_a.csv --test ./data_preprocessing/processed_data/race_test_updated_cqa_dsep_a.csv --model onebyone-pos --tensorboard --config bert-base-cased --batch 30 --epoch 6; ``` + ### Train BDG AN+PM model + ```bash tfkit-train --maxlen 512 --savedir ./race_cqa_gen_d_both/ --train ./data_preprocessing/processed_data/race_train_updated_cqa_dsep_a.csv --test ./data_preprocessing/processed_data/race_test_updated_cqa_dsep_a.csv --model onebyone-both --tensorboard --config bert-base-cased --batch 30 --epoch 6; ``` -### Eval generator + +### Eval generator + ```bash tfkit-eval --model model_path --valid ./data_preprocessing/processed_data/race_test_updated_cqa_dall.csv --metric nlg ``` ## Distractor Analysis + Inside `distractor analysis` folder -- `preprocess_model_result.py` for result preprocessing and statistics. -- `normalize_jsonl_file.py` merge different model result with same question and context. -- `create_rank_dataset.py` prepare data for Entropy Maximization. + +- `preprocess_model_result.py` for result preprocessing and statistics. +- `normalize_jsonl_file.py` merge different model result with same question and + context. +- `create_rank_dataset.py` prepare data for Entropy Maximization. ## RACE MRC + ### Preparation + ```bash git clone https://github.com/huggingface/transformers cp our transformer file into huggingface/transformers ``` ### Training Multiple Choice Question Answering Model -Based on the script [`run_multiple_choice.py`](). -Download race data -Train + +Based on the script [`run_multiple_choice.py`](). Download race data Train + ```bash #training on 4 tesla V100(16GB) GPUS export RACE_DIR=../RACE @@ -140,10 +169,11 @@ python ./examples/run_multiple_choice.py \ --gradient_accumulation_steps 2 \ --save_steps 5000 \ --eval_all_checkpoints \ ---seed 77 +--seed 77 ``` ### Eval QA & Get entropy ensemble result + ```bash export RACE_DIR=../multi_dist_normalized_jsonl/xxx.jsonl python ./examples/run_multiple_choice.py \ @@ -158,8 +188,6 @@ python ./examples/run_multiple_choice.py \ --overwrite_cache ``` - - - ## License -[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Fvoidful%2FBDG.svg?type=large)](https://app.fossa.com/projects/git%2Bgithub.com%2Fvoidful%2FBDG?ref=badge_large) \ No newline at end of file + +[![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Fvoidful%2FBDG.svg?type=large)](https://app.fossa.com/projects/git%2Bgithub.com%2Fvoidful%2FBDG?ref=badge_large) From 89c35782378c88a8972a0dd788551d7cbbcf9f7c Mon Sep 17 00:00:00 2001 From: "Restyled.io" Date: Wed, 22 Sep 2021 18:21:02 +0000 Subject: [PATCH 3/3] Restyled by whitespace --- README.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index b795030..acca0a8 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,7 @@ [![FOSSA Status](https://app.fossa.com/api/projects/git%2Bgithub.com%2Fvoidful%2FBDG.svg?type=shield)](https://app.fossa.com/projects/git%2Bgithub.com%2Fvoidful%2FBDG?ref=badge_shield) Code for "A BERT-based Distractor Generation Scheme with Multi-tasking and -Negative Answer Training Strategies." +Negative Answer Training Strategies." [Paper](https://www.aclweb.org/anthology/2020.findings-emnlp.393/) ## V2 @@ -34,16 +34,16 @@ model | BLEU1 | BLEU2 | BLEU3 | BLEU4 | ROUGEL | ### BART -Distractor: https://huggingface.co/voidful/bart-distractor-generation -Distractor PM: https://huggingface.co/voidful/bart-distractor-generation-pm +Distractor: https://huggingface.co/voidful/bart-distractor-generation +Distractor PM: https://huggingface.co/voidful/bart-distractor-generation-pm Distractor AN+PM: https://huggingface.co/voidful/bart-distractor-generation-both ### BERT -Trained model available on release: +Trained model available on release: https://github.com/voidful/BDG/releases/tag/v1.0 -Colab notebook for using pre trained model: +Colab notebook for using pre trained model: https://colab.research.google.com/drive/1yA3Rex9JHKJmc52E3YdsBQ4eQ_R6kEZB?usp=sharing ## Citation @@ -74,11 +74,11 @@ pip install -r requirement.txt ## Data Preprocessing -Inside `data_preprocessing` folder. +Inside `data_preprocessing` folder. Download dataset [here](https://github.com/Yifan-Gao/Distractor-Generation-RACE), put it into -`distractor` folder. -run `convert_data.py` to do preprocessing. +`distractor` folder. +run `convert_data.py` to do preprocessing. run `dataset_stat.py` for dataset statistics. ## Train Distractor Generator @@ -95,7 +95,7 @@ tfkit-train --savedir ./race_cqa_gen_d_bart_both/ --train ./race_train_updated_c ### BERT -using environment from `requirement.txt` +using environment from `requirement.txt` run the following in main dir: ### Train BDG Model