Compare commits
10 Commits
c1f27fe3bd
...
95c62bc0d4
Author | SHA1 | Date |
---|---|---|
|
95c62bc0d4 | |
|
75e45e37b8 | |
|
aa84615d8f | |
|
de53cbc407 | |
|
86eea0ee28 | |
|
09838ca216 | |
|
f4d98ed686 | |
|
4d4ccab781 | |
|
7a5d44c282 | |
|
f9f2a5f6f1 |
|
@ -6,3 +6,4 @@
|
|||
*.tar.gz filter=lfs diff=lfs merge=lfs -text
|
||||
*.ot filter=lfs diff=lfs merge=lfs -text
|
||||
*.onnx filter=lfs diff=lfs merge=lfs -text
|
||||
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
||||
|
|
|
@ -0,0 +1,120 @@
|
|||
---
|
||||
language: en
|
||||
datasets:
|
||||
- conll2003
|
||||
license: mit
|
||||
---
|
||||
# bert-base-NER
|
||||
|
||||
## Model description
|
||||
|
||||
**bert-large-NER** is a fine-tuned BERT model that is ready to use for **Named Entity Recognition** and achieves **state-of-the-art performance** for the NER task. It has been trained to recognize four types of entities: location (LOC), organizations (ORG), person (PER) and Miscellaneous (MISC).
|
||||
|
||||
Specifically, this model is a *bert-large-cased* model that was fine-tuned on the English version of the standard [CoNLL-2003 Named Entity Recognition](https://www.aclweb.org/anthology/W03-0419.pdf) dataset.
|
||||
|
||||
If you'd like to use a smaller BERT model fine-tuned on the same dataset, a [**bert-base-NER**](https://huggingface.co/dslim/bert-base-NER/) version is also available.
|
||||
|
||||
|
||||
## Intended uses & limitations
|
||||
|
||||
#### How to use
|
||||
|
||||
You can use this model with Transformers *pipeline* for NER.
|
||||
|
||||
```python
|
||||
from transformers import AutoTokenizer, AutoModelForTokenClassification
|
||||
from transformers import pipeline
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained("dslim/bert-base-NER")
|
||||
model = AutoModelForTokenClassification.from_pretrained("dslim/bert-base-NER")
|
||||
|
||||
nlp = pipeline("ner", model=model, tokenizer=tokenizer)
|
||||
example = "My name is Wolfgang and I live in Berlin"
|
||||
|
||||
ner_results = nlp(example)
|
||||
print(ner_results)
|
||||
```
|
||||
|
||||
#### Limitations and bias
|
||||
|
||||
This model is limited by its training dataset of entity-annotated news articles from a specific span of time. This may not generalize well for all use cases in different domains. Furthermore, the model occassionally tags subword tokens as entities and post-processing of results may be necessary to handle those cases.
|
||||
|
||||
## Training data
|
||||
|
||||
This model was fine-tuned on English version of the standard [CoNLL-2003 Named Entity Recognition](https://www.aclweb.org/anthology/W03-0419.pdf) dataset.
|
||||
|
||||
The training dataset distinguishes between the beginning and continuation of an entity so that if there are back-to-back entities of the same type, the model can output where the second entity begins. As in the dataset, each token will be classified as one of the following classes:
|
||||
|
||||
Abbreviation|Description
|
||||
-|-
|
||||
O|Outside of a named entity
|
||||
B-MIS |Beginning of a miscellaneous entity right after another miscellaneous entity
|
||||
I-MIS | Miscellaneous entity
|
||||
B-PER |Beginning of a person’s name right after another person’s name
|
||||
I-PER |Person’s name
|
||||
B-ORG |Beginning of an organization right after another organization
|
||||
I-ORG |organization
|
||||
B-LOC |Beginning of a location right after another location
|
||||
I-LOC |Location
|
||||
|
||||
|
||||
### CoNLL-2003 English Dataset Statistics
|
||||
This dataset was derived from the Reuters corpus which consists of Reuters news stories. You can read more about how this dataset was created in the CoNLL-2003 paper.
|
||||
#### # of training examples per entity type
|
||||
Dataset|LOC|MISC|ORG|PER
|
||||
-|-|-|-|-
|
||||
Train|7140|3438|6321|6600
|
||||
Dev|1837|922|1341|1842
|
||||
Test|1668|702|1661|1617
|
||||
#### # of articles/sentences/tokens per dataset
|
||||
Dataset |Articles |Sentences |Tokens
|
||||
-|-|-|-
|
||||
Train |946 |14,987 |203,621
|
||||
Dev |216 |3,466 |51,362
|
||||
Test |231 |3,684 |46,435
|
||||
|
||||
## Training procedure
|
||||
|
||||
This model was trained on a single NVIDIA V100 GPU with recommended hyperparameters from the [original BERT paper](https://arxiv.org/pdf/1810.04805) which trained & evaluated the model on CoNLL-2003 NER task.
|
||||
|
||||
## Eval results
|
||||
metric|dev|test
|
||||
-|-|-
|
||||
f1 |95.7 |91.7
|
||||
precision |95.3 |91.2
|
||||
recall |96.1 |92.3
|
||||
|
||||
The test metrics are a little lower than the official Google BERT results which encoded document context & experimented with CRF. More on replicating the original results [here](https://github.com/google-research/bert/issues/223).
|
||||
|
||||
### BibTeX entry and citation info
|
||||
|
||||
```
|
||||
@article{DBLP:journals/corr/abs-1810-04805,
|
||||
author = {Jacob Devlin and
|
||||
Ming{-}Wei Chang and
|
||||
Kenton Lee and
|
||||
Kristina Toutanova},
|
||||
title = {{BERT:} Pre-training of Deep Bidirectional Transformers for Language
|
||||
Understanding},
|
||||
journal = {CoRR},
|
||||
volume = {abs/1810.04805},
|
||||
year = {2018},
|
||||
url = {http://arxiv.org/abs/1810.04805},
|
||||
archivePrefix = {arXiv},
|
||||
eprint = {1810.04805},
|
||||
timestamp = {Tue, 30 Oct 2018 20:39:56 +0100},
|
||||
biburl = {https://dblp.org/rec/journals/corr/abs-1810-04805.bib},
|
||||
bibsource = {dblp computer science bibliography, https://dblp.org}
|
||||
}
|
||||
```
|
||||
```
|
||||
@inproceedings{tjong-kim-sang-de-meulder-2003-introduction,
|
||||
title = "Introduction to the {C}o{NLL}-2003 Shared Task: Language-Independent Named Entity Recognition",
|
||||
author = "Tjong Kim Sang, Erik F. and
|
||||
De Meulder, Fien",
|
||||
booktitle = "Proceedings of the Seventh Conference on Natural Language Learning at {HLT}-{NAACL} 2003",
|
||||
year = "2003",
|
||||
url = "https://www.aclweb.org/anthology/W03-0419",
|
||||
pages = "142--147",
|
||||
}
|
||||
```
|
|
@ -0,0 +1,67 @@
|
|||
{
|
||||
"architectures": [
|
||||
"BertForTokenClassification"
|
||||
],
|
||||
"attention_probs_dropout_prob": 0.1,
|
||||
"bos_token_id": null,
|
||||
"directionality": "bidi",
|
||||
"do_sample": false,
|
||||
"eos_token_ids": null,
|
||||
"finetuning_task": null,
|
||||
"hidden_act": "gelu",
|
||||
"hidden_dropout_prob": 0.1,
|
||||
"hidden_size": 1024,
|
||||
"id2label": {
|
||||
"0": "O",
|
||||
"1": "B-MISC",
|
||||
"2": "I-MISC",
|
||||
"3": "B-PER",
|
||||
"4": "I-PER",
|
||||
"5": "B-ORG",
|
||||
"6": "I-ORG",
|
||||
"7": "B-LOC",
|
||||
"8": "I-LOC"
|
||||
},
|
||||
"initializer_range": 0.02,
|
||||
"intermediate_size": 4096,
|
||||
"is_decoder": false,
|
||||
"label2id": {
|
||||
"B-LOC": 7,
|
||||
"B-MISC": 1,
|
||||
"B-ORG": 5,
|
||||
"B-PER": 3,
|
||||
"I-LOC": 8,
|
||||
"I-MISC": 2,
|
||||
"I-ORG": 6,
|
||||
"I-PER": 4,
|
||||
"O": 0
|
||||
},
|
||||
"layer_norm_eps": 1e-12,
|
||||
"length_penalty": 1.0,
|
||||
"max_length": 20,
|
||||
"max_position_embeddings": 512,
|
||||
"model_type": "bert",
|
||||
"num_attention_heads": 16,
|
||||
"num_beams": 1,
|
||||
"num_hidden_layers": 24,
|
||||
"num_labels": 9,
|
||||
"num_return_sequences": 1,
|
||||
"output_attentions": false,
|
||||
"output_hidden_states": false,
|
||||
"output_past": true,
|
||||
"pad_token_id": 0,
|
||||
"pooler_fc_size": 768,
|
||||
"pooler_num_attention_heads": 12,
|
||||
"pooler_num_fc_layers": 3,
|
||||
"pooler_size_per_head": 128,
|
||||
"pooler_type": "first_token_transform",
|
||||
"pruned_heads": {},
|
||||
"repetition_penalty": 1.0,
|
||||
"temperature": 1.0,
|
||||
"top_k": 50,
|
||||
"top_p": 1.0,
|
||||
"torchscript": false,
|
||||
"type_vocab_size": 2,
|
||||
"use_bfloat16": false,
|
||||
"vocab_size": 28996
|
||||
}
|
Binary file not shown.
Binary file not shown.
|
@ -0,0 +1 @@
|
|||
{"unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]"}
|
Binary file not shown.
Loading…
Reference in New Issue