valhalla/distilbart-mnli-12-1 is a forked repo from huggingface. License: None

Go to file

patil-suraj 506336d421 add flax model		2021-06-14 10:27:55 +00:00
.gitattributes	add flax model	2021-06-14 10:27:55 +00:00
README.md	Migrate model card from transformers-repo	2020-12-11 23:03:21 +01:00
config.json	add flax model	2021-06-14 10:27:55 +00:00
flax_model.msgpack	add flax model	2021-06-14 10:27:55 +00:00
merges.txt	Update merges.txt	2020-09-20 13:43:25 +00:00
pytorch_model.bin	Update pytorch_model.bin	2020-09-20 13:42:36 +00:00
special_tokens_map.json	Update special_tokens_map.json	2020-09-20 13:42:33 +00:00
tokenizer_config.json	Update tokenizer_config.json	2020-09-20 13:43:29 +00:00
vocab.json	Update vocab.json	2020-09-20 13:43:27 +00:00

README.md

datasets

DistilBart-MNLI

distilbart-mnli is the distilled version of bart-large-mnli created using the No Teacher Distillation technique proposed for BART summarisation by Huggingface, here.

We just copy alternating layers from bart-large-mnli and finetune more on the same data.

	matched acc	mismatched acc
bart-large-mnli (baseline, 12-12)	89.9	90.01
distilbart-mnli-12-1	87.08	87.5
distilbart-mnli-12-3	88.1	88.19
distilbart-mnli-12-6	89.19	89.01
distilbart-mnli-12-9	89.56	89.52

This is a very simple and effective technique, as we can see the performance drop is very little.

Detailed performace trade-offs will be posted in this sheet.

Fine-tuning

If you want to train these models yourself, clone the distillbart-mnli repo and follow the steps below

Clone and install transformers from source

git clone https://github.com/huggingface/transformers.git
pip install -qqq -U ./transformers

Download MNLI data

python transformers/utils/download_glue_data.py --data_dir glue_data --tasks MNLI

Create student model

python create_student.py \
  --teacher_model_name_or_path facebook/bart-large-mnli \
  --student_encoder_layers 12 \
  --student_decoder_layers 6 \
  --save_path student-bart-mnli-12-6 \

Start fine-tuning

python run_glue.py args.json

You can find the logs of these trained models in this wandb project.