Compare commits
10 Commits
da1702f12c
...
d0a1f6ab88
Author | SHA1 | Date |
---|---|---|
|
d0a1f6ab88 | |
|
a08ecc1591 | |
|
62cb70a235 | |
|
4355f59b0b | |
|
1d0521195d | |
|
18d30d29a8 | |
|
dcf3033e95 | |
|
2cc62bf3d7 | |
|
4bba75a1b0 | |
|
a5faec5a6d |
36
README.md
36
README.md
|
@ -1,5 +1,12 @@
|
||||||
---
|
---
|
||||||
|
tags:
|
||||||
|
- visual-question-answering
|
||||||
license: apache-2.0
|
license: apache-2.0
|
||||||
|
widget:
|
||||||
|
- text: "What's the animal doing?"
|
||||||
|
src: "https://huggingface.co/datasets/mishig/sample_images/resolve/main/tiger.jpg"
|
||||||
|
- text: "What is on top of the building?"
|
||||||
|
src: "https://huggingface.co/datasets/mishig/sample_images/resolve/main/palace.jpg"
|
||||||
---
|
---
|
||||||
|
|
||||||
# Vision-and-Language Transformer (ViLT), fine-tuned on VQAv2
|
# Vision-and-Language Transformer (ViLT), fine-tuned on VQAv2
|
||||||
|
@ -9,17 +16,36 @@ Without Convolution or Region Supervision](https://arxiv.org/abs/2102.03334) by
|
||||||
|
|
||||||
Disclaimer: The team releasing ViLT did not write a model card for this model so this model card has been written by the Hugging Face team.
|
Disclaimer: The team releasing ViLT did not write a model card for this model so this model card has been written by the Hugging Face team.
|
||||||
|
|
||||||
## Model description
|
|
||||||
|
|
||||||
(to do)
|
|
||||||
|
|
||||||
## Intended uses & limitations
|
## Intended uses & limitations
|
||||||
|
|
||||||
You can use the raw model for visual question answering.
|
You can use the raw model for visual question answering.
|
||||||
|
|
||||||
### How to use
|
### How to use
|
||||||
|
|
||||||
(to do)
|
Here is how to use this model in PyTorch:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from transformers import ViltProcessor, ViltForQuestionAnswering
|
||||||
|
import requests
|
||||||
|
from PIL import Image
|
||||||
|
|
||||||
|
# prepare image + question
|
||||||
|
url = "http://images.cocodataset.org/val2017/000000039769.jpg"
|
||||||
|
image = Image.open(requests.get(url, stream=True).raw)
|
||||||
|
text = "How many cats are there?"
|
||||||
|
|
||||||
|
processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-vqa")
|
||||||
|
model = ViltForQuestionAnswering.from_pretrained("dandelin/vilt-b32-finetuned-vqa")
|
||||||
|
|
||||||
|
# prepare inputs
|
||||||
|
encoding = processor(image, text, return_tensors="pt")
|
||||||
|
|
||||||
|
# forward pass
|
||||||
|
outputs = model(**encoding)
|
||||||
|
logits = outputs.logits
|
||||||
|
idx = logits.argmax(-1).item()
|
||||||
|
print("Predicted answer:", model.config.id2label[idx])
|
||||||
|
```
|
||||||
|
|
||||||
## Training data
|
## Training data
|
||||||
|
|
||||||
|
|
|
@ -0,0 +1,18 @@
|
||||||
|
{
|
||||||
|
"do_normalize": true,
|
||||||
|
"do_resize": true,
|
||||||
|
"feature_extractor_type": "ViltFeatureExtractor",
|
||||||
|
"image_mean": [
|
||||||
|
0.5,
|
||||||
|
0.5,
|
||||||
|
0.5
|
||||||
|
],
|
||||||
|
"image_std": [
|
||||||
|
0.5,
|
||||||
|
0.5,
|
||||||
|
0.5
|
||||||
|
],
|
||||||
|
"resample": 3,
|
||||||
|
"size": 384,
|
||||||
|
"size_divisor": 32
|
||||||
|
}
|
|
@ -0,0 +1 @@
|
||||||
|
{"unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]"}
|
File diff suppressed because one or more lines are too long
|
@ -1 +1 @@
|
||||||
{"do_lower_case": true, "do_basic_tokenize": true, "never_split": null, "unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]", "tokenize_chinese_chars": true, "strip_accents": null, "model_max_length": 512, "special_tokens_map_file": null, "tokenizer_file": "/root/.cache/huggingface/transformers/534479488c54aeaf9c3406f647aa2ec13648c06771ffe269edabebd4c412da1d.7f2721073f19841be16f41b0a70b600ca6b880c8f3df6f3535cbc704371bdfa4", "name_or_path": "bert-base-uncased", "tokenizer_class": "BertTokenizer"}
|
{"do_lower_case": true, "unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]", "tokenize_chinese_chars": true, "strip_accents": null, "model_max_length": 40, "special_tokens_map_file": null, "name_or_path": "bert-base-uncased", "tokenizer_class": "BertTokenizer"}
|
Loading…
Reference in New Issue