Compare commits
No commits in common. "d0a1f6ab88522427a7ae76ceb6e1e1e7b68a1d08" and "da1702f12c4479f781ab0764175bc3c93227b63d" have entirely different histories.
d0a1f6ab88
...
da1702f12c
36
README.md
36
README.md
|
@ -1,12 +1,5 @@
|
||||||
---
|
---
|
||||||
tags:
|
|
||||||
- visual-question-answering
|
|
||||||
license: apache-2.0
|
license: apache-2.0
|
||||||
widget:
|
|
||||||
- text: "What's the animal doing?"
|
|
||||||
src: "https://huggingface.co/datasets/mishig/sample_images/resolve/main/tiger.jpg"
|
|
||||||
- text: "What is on top of the building?"
|
|
||||||
src: "https://huggingface.co/datasets/mishig/sample_images/resolve/main/palace.jpg"
|
|
||||||
---
|
---
|
||||||
|
|
||||||
# Vision-and-Language Transformer (ViLT), fine-tuned on VQAv2
|
# Vision-and-Language Transformer (ViLT), fine-tuned on VQAv2
|
||||||
|
@ -16,36 +9,17 @@ Without Convolution or Region Supervision](https://arxiv.org/abs/2102.03334) by
|
||||||
|
|
||||||
Disclaimer: The team releasing ViLT did not write a model card for this model so this model card has been written by the Hugging Face team.
|
Disclaimer: The team releasing ViLT did not write a model card for this model so this model card has been written by the Hugging Face team.
|
||||||
|
|
||||||
|
## Model description
|
||||||
|
|
||||||
|
(to do)
|
||||||
|
|
||||||
## Intended uses & limitations
|
## Intended uses & limitations
|
||||||
|
|
||||||
You can use the raw model for visual question answering.
|
You can use the raw model for visual question answering.
|
||||||
|
|
||||||
### How to use
|
### How to use
|
||||||
|
|
||||||
Here is how to use this model in PyTorch:
|
(to do)
|
||||||
|
|
||||||
```python
|
|
||||||
from transformers import ViltProcessor, ViltForQuestionAnswering
|
|
||||||
import requests
|
|
||||||
from PIL import Image
|
|
||||||
|
|
||||||
# prepare image + question
|
|
||||||
url = "http://images.cocodataset.org/val2017/000000039769.jpg"
|
|
||||||
image = Image.open(requests.get(url, stream=True).raw)
|
|
||||||
text = "How many cats are there?"
|
|
||||||
|
|
||||||
processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-vqa")
|
|
||||||
model = ViltForQuestionAnswering.from_pretrained("dandelin/vilt-b32-finetuned-vqa")
|
|
||||||
|
|
||||||
# prepare inputs
|
|
||||||
encoding = processor(image, text, return_tensors="pt")
|
|
||||||
|
|
||||||
# forward pass
|
|
||||||
outputs = model(**encoding)
|
|
||||||
logits = outputs.logits
|
|
||||||
idx = logits.argmax(-1).item()
|
|
||||||
print("Predicted answer:", model.config.id2label[idx])
|
|
||||||
```
|
|
||||||
|
|
||||||
## Training data
|
## Training data
|
||||||
|
|
||||||
|
|
|
@ -1,18 +0,0 @@
|
||||||
{
|
|
||||||
"do_normalize": true,
|
|
||||||
"do_resize": true,
|
|
||||||
"feature_extractor_type": "ViltFeatureExtractor",
|
|
||||||
"image_mean": [
|
|
||||||
0.5,
|
|
||||||
0.5,
|
|
||||||
0.5
|
|
||||||
],
|
|
||||||
"image_std": [
|
|
||||||
0.5,
|
|
||||||
0.5,
|
|
||||||
0.5
|
|
||||||
],
|
|
||||||
"resample": 3,
|
|
||||||
"size": 384,
|
|
||||||
"size_divisor": 32
|
|
||||||
}
|
|
|
@ -1 +0,0 @@
|
||||||
{"unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]"}
|
|
File diff suppressed because one or more lines are too long
|
@ -1 +1 @@
|
||||||
{"do_lower_case": true, "unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]", "tokenize_chinese_chars": true, "strip_accents": null, "model_max_length": 40, "special_tokens_map_file": null, "name_or_path": "bert-base-uncased", "tokenizer_class": "BertTokenizer"}
|
{"do_lower_case": true, "do_basic_tokenize": true, "never_split": null, "unk_token": "[UNK]", "sep_token": "[SEP]", "pad_token": "[PAD]", "cls_token": "[CLS]", "mask_token": "[MASK]", "tokenize_chinese_chars": true, "strip_accents": null, "model_max_length": 512, "special_tokens_map_file": null, "tokenizer_file": "/root/.cache/huggingface/transformers/534479488c54aeaf9c3406f647aa2ec13648c06771ffe269edabebd4c412da1d.7f2721073f19841be16f41b0a70b600ca6b880c8f3df6f3535cbc704371bdfa4", "name_or_path": "bert-base-uncased", "tokenizer_class": "BertTokenizer"}
|
Loading…
Reference in New Issue