Adding `safetensors` variant of this model (#3)

- Adding `safetensors` variant of this model (ab6d5e4057481583df31a2abba35e78e027dd838)


Co-authored-by: Safetensors convertbot <SFconvertbot@users.noreply.huggingface.co>

.gitattributes

@@ -14,3 +14,4 @@
 *.pb filter=lfs diff=lfs merge=lfs -text
 *.pt filter=lfs diff=lfs merge=lfs -text
 *.pth filter=lfs diff=lfs merge=lfs -text
+model.safetensors filter=lfs diff=lfs merge=lfs -text

README.md

@@ -4,11 +4,116 @@ pipeline_tag: zero-shot-classification
 tags:
 - distilbert
 datasets:
-- mulit_nli
+- multi_nli
 metrics:
 - accuracy
 ---
 
 # DistilBERT base model (uncased)
 
-This model is the Multi-Genre Natural Language Inference (MNLI) fine-turned version of the [uncased DistilBERT model](https://huggingface.co/distilbert-base-uncased).
+
+## Table of Contents
+- [Model Details](#model-details)
+- [How to Get Started With the Model](#how-to-get-started-with-the-model)
+- [Uses](#uses)
+- [Risks, Limitations and Biases](#risks-limitations-and-biases)
+- [Training](#training)
+- [Evaluation](#evaluation)
+- [Environmental Impact](#environmental-impact)
+
+
+
+## Model Details
+**Model Description:**  This is the [uncased DistilBERT model](https://huggingface.co/distilbert-base-uncased) fine-tuned on [Multi-Genre Natural Language Inference](https://huggingface.co/datasets/multi_nli) (MNLI) dataset for the zero-shot classification task. 
+- **Developed by:** The [Typeform](https://www.typeform.com/) team.
+- **Model Type:** Zero-Shot Classification
+- **Language(s):** English
+- **License:** Unknown
+- **Parent Model:** See the [distilbert base uncased model](https://huggingface.co/distilbert-base-uncased) for more information about the Distilled-BERT base model.
+    
+   
+ ## How to Get Started with the Model
+
+```python
+from transformers import AutoTokenizer, AutoModelForSequenceClassification
+
+tokenizer = AutoTokenizer.from_pretrained("typeform/distilbert-base-uncased-mnli")
+
+model = AutoModelForSequenceClassification.from_pretrained("typeform/distilbert-base-uncased-mnli")
+
+```
+
+## Uses
+This model can be used for text classification tasks.
+
+
+## Risks, Limitations and Biases
+**CONTENT WARNING: Readers should be aware this section contains content that is disturbing, offensive, and can propagate historical and current stereotypes.**
+
+Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)).
+
+
+## Training
+
+#### Training Data
+
+
+This model of DistilBERT-uncased is pretrained on the Multi-Genre Natural Language Inference [(MultiNLI)](https://huggingface.co/datasets/multi_nli) corpus. It is a crowd-sourced collection of 433k sentence pairs annotated with textual entailment information. The corpus covers a range of genres of spoken and written text, and supports a distinctive cross-genre generalization evaluation.
+
+This model is also **not** case-sensitive, i.e., it does not make a difference between "english" and "English".
+
+
+#### Training Procedure
+
+Training is done on a [p3.2xlarge](https://aws.amazon.com/ec2/instance-types/p3/) AWS EC2 with the following hyperparameters:
+
+```
+$ run_glue.py \
+    --model_name_or_path distilbert-base-uncased \
+    --task_name mnli \
+    --do_train \
+    --do_eval \
+    --max_seq_length 128 \
+    --per_device_train_batch_size 16 \
+    --learning_rate 2e-5 \
+    --num_train_epochs 5 \
+    --output_dir /tmp/distilbert-base-uncased_mnli/
+```
+
+## Evaluation
+
+
+#### Evaluation Results
+When fine-tuned on downstream tasks, this model achieves the following results:
+- **Epoch = ** 5.0
+- **Evaluation Accuracy =**  0.8206875508543532
+- **Evaluation Loss =** 0.8706700205802917
+- ** Evaluation Runtime = ** 17.8278
+- ** Evaluation Samples per second = ** 551.498
+
+MNLI and MNLI-mm results:
+
+| Task | MNLI | MNLI-mm |
+|:----:|:----:|:----:|
+|      | 82.0 | 82.0 |
+
+
+
+## Environmental Impact
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700). We present the hardware type based on the [associated paper](https://arxiv.org/pdf/2105.09680.pdf).
+
+
+**Hardware Type:** 1 NVIDIA Tesla V100 GPUs
+
+**Hours used:**  Unknown
+
+**Cloud Provider:** AWS  EC2 P3
+
+
+**Compute Region:** Unknown
+
+
+
+**Carbon Emitted:** (Power consumption x Time x Carbon produced based on location of power grid): Unknown
+

config.json

@@ -0,0 +1,35 @@
+{
+  "_name_or_path": "/tmp/mnli_distil_output/checkpoint-6000",
+  "activation": "gelu",
+  "architectures": [
+    "DistilBertForSequenceClassification"
+  ],
+  "attention_dropout": 0.1,
+  "dim": 768,
+  "dropout": 0.1,
+  "finetuning_task": "mnli",
+  "hidden_dim": 3072,
+  "id2label": {
+    "0": "ENTAILMENT",
+    "1": "NEUTRAL",
+    "2": "CONTRADICTION"
+  },
+  "initializer_range": 0.02,
+  "label2id": {
+    "ENTAILMENT": 0,
+    "NEUTRAL": 1,
+    "CONTRADICTION": 2
+  },
+  "max_position_embeddings": 512,
+  "model_type": "distilbert",
+  "n_heads": 12,
+  "n_layers": 6,
+  "pad_token_id": 0,
+  "qa_dropout": 0.1,
+  "seq_classif_dropout": 0.2,
+  "sinusoidal_pos_embds": false,
+  "tie_weights_": true,
+  "transformers_version": "4.3.2",
+  "vocab_size": 30522,
+  "xla_device": true
+}

eval_results_mnli-mm.txt

@@ -0,0 +1,5 @@
+epoch = 5.0
+eval_accuracy = 0.8206875508543532
+eval_loss = 0.8706700205802917
+eval_runtime = 17.8278
+eval_samples_per_second = 551.498

eval_results_mnli.txt

@@ -0,0 +1,5 @@
+epoch = 5.0
+eval_accuracy = 0.8221090168110036
+eval_loss = 0.8824708461761475
+eval_runtime = 18.1167
+eval_samples_per_second = 541.765

special_tokens_map.json

@@ -0,0 +1 @@
+{"unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]"}

tokenizer_config.json

@@ -0,0 +1 @@
+{"do_lower_case": true, "unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]", "tokenize_chinese_chars": true, "strip_accents": null, "model_max_length": 512, "name_or_path": "distilbert-base-uncased"}

train_results.txt

@@ -0,0 +1,3 @@
+epoch = 5.0
+train_runtime = 14248.9446
+train_samples_per_second = 8.613