Update info
This commit is contained in:
anton-l
parent
8cf46f207c
commit
10e57bf5c9
74
README.md
74
README.md
|
|
@ -6,64 +6,90 @@ tags:
|
|||
- speech
|
||||
- audio
|
||||
- hubert
|
||||
- s3prl
|
||||
- audio-classification
|
||||
license: apache-2.0
|
||||
---
|
||||
|
||||
# Hubert-Base for Keyword Spotting
|
||||
|
||||
[S3PRL speech toolkit](https://github.com/s3prl/s3prl)
|
||||
## Model description
|
||||
|
||||
[Facebook's Hubert](https://ai.facebook.com/blog/hubert-self-supervised-representation-learning-for-speech-recognition-generation-and-compression)
|
||||
This is a ported version of [S3PRL's Hubert for the SUPERB Keyword Spotting task](https://github.com/s3prl/s3prl/tree/master/s3prl/downstream/speech_commands).
|
||||
|
||||
The base model is pretrained on 16kHz sampled speech audio. When using the model make sure that your speech input is also sampled at 16Khz.
|
||||
The classification head is trained using the Keyword Spotting part of the SUPERB dataset.
|
||||
The base model is [hubert-base-ls960](https://huggingface.co/facebook/hubert-base-ls960), which is pretrained on 16kHz
|
||||
sampled speech audio. When using the model make sure that your speech input is also sampled at 16Khz.
|
||||
|
||||
[Paper](https://arxiv.org/abs/2105.01051)
|
||||
For more information refer to [SUPERB: Speech processing Universal PERformance Benchmark](https://arxiv.org/abs/2105.01051)
|
||||
|
||||
Authors: Shu-wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y. Lin, Andy T. Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, Tzu-Hsien Huang, Wei-Cheng Tseng, Ko-tik Lee, Da-Rong Liu, Zili Huang, Shuyan Dong, Shang-Wen Li, Shinji Watanabe, Abdelrahman Mohamed, Hung-yi Lee
|
||||
## Task and dataset description
|
||||
|
||||
**Abstract**
|
||||
Self-supervised learning (SSL) has proven vital for advancing research in natural language processing (NLP) and computer vision (CV). The paradigm pretrains a shared model on large volumes of unlabeled data and achieves state-of-the-art (SOTA) for various tasks with minimal adaptation. However, the speech processing community lacks a similar setup to systematically explore the paradigm. To bridge this gap, we introduce Speech processing Universal PERformance Benchmark (SUPERB). SUPERB is a leaderboard to benchmark the performance of a shared model across a wide range of speech processing tasks with minimal architecture changes and labeled data. Among multiple usages of the shared model, we especially focus on extracting the representation learned from SSL due to its preferable re-usability. We present a simple framework to solve SUPERB tasks by learning task-specialized lightweight prediction heads on top of the frozen shared model. Our results demonstrate that the framework is promising as SSL representations show competitive generalizability and accessibility across SUPERB tasks. We release SUPERB as a challenge with a leaderboard and a benchmark toolkit to fuel the research in representation learning and general speech processing.
|
||||
Keyword Spotting (KS) detects preregistered keywords by classifying utterances into a predefined set of
|
||||
words. The task is usually performed on-device for the fast response time. Thus, accuracy, model size, and
|
||||
inference time are all crucial. SUPERB uses the widely used
|
||||
[Speech Commands dataset v1.0](https://www.tensorflow.org/datasets/catalog/speech_commands) for the task.
|
||||
The dataset consists of ten classes of keywords, a class for silence, and an unknown class to include the
|
||||
false positive.
|
||||
|
||||
The original model can be found under https://github.com/s3prl/s3prl/tree/master/s3prl/downstream/speech_commands.
|
||||
For the original model's training and evaluation instructions refer to the
|
||||
[S3PRL downstream task README](https://github.com/s3prl/s3prl/tree/master/s3prl/downstream#ks-keyword-spotting).
|
||||
|
||||
The base model is [hubert-base-ls960](https://huggingface.co/facebook/hubert-base-ls960)
|
||||
|
||||
# Usage examples
|
||||
## Usage examples
|
||||
|
||||
You can use the model via the Audio Classification pipeline:
|
||||
```python
|
||||
import numpy as np
|
||||
from datasets import load_dataset
|
||||
from transformers import pipeline, PreTrainedTokenizer
|
||||
from transformers import pipeline
|
||||
|
||||
superb_ks = load_dataset("anton-l/superb_dummy", "ks", split="test")
|
||||
model = "superb/hubert-base-superb-ks"
|
||||
tokenizer = PreTrainedTokenizer() # a dummy tokenizer, since the classifier doesn't need a real one
|
||||
classifier = pipeline("audio-classification", model=model, feature_extractor=model, tokenizer=tokenizer)
|
||||
dataset = load_dataset("anton-l/superb_demo", "ks", split="test")
|
||||
|
||||
audio = np.array(superb_ks[0]["speech"])
|
||||
labels = classifier(audio, top_k=5)
|
||||
classifier = pipeline("audio-classification", model="superb/hubert-base-superb-ks")
|
||||
labels = classifier(dataset[0]["file"], top_k=5)
|
||||
```
|
||||
|
||||
Or use the model directly:
|
||||
```python
|
||||
import torch
|
||||
import numpy as np
|
||||
from datasets import load_dataset
|
||||
from transformers import HubertForSequenceClassification, Wav2Vec2FeatureExtractor
|
||||
from torchaudio.sox_effects import apply_effects_file
|
||||
|
||||
effects = [["channels", "1"], ["rate", "16000"], ["gain", "-3.0"]]
|
||||
def map_to_array(example):
|
||||
speech, _ = apply_effects_file(example["file"], effects)
|
||||
example["speech"] = speech.squeeze(0).numpy()
|
||||
return example
|
||||
|
||||
# load a demo dataset and read audio files
|
||||
dataset = load_dataset("anton-l/superb_demo", "ks", split="test")
|
||||
dataset = dataset.map(map_to_array)
|
||||
|
||||
superb_ks = load_dataset("anton-l/superb_dummy", "ks", split="test")
|
||||
model = HubertForSequenceClassification.from_pretrained("superb/hubert-base-superb-ks")
|
||||
feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained("superb/hubert-base-superb-ks")
|
||||
|
||||
audio = np.array(superb_ks[0]["speech"])
|
||||
# compute attention masks and normalize the waveform if needed
|
||||
inputs = feature_extractor(audio, sampling_rate=16_000, return_tensors="pt")
|
||||
inputs = feature_extractor(dataset[:4]["speech"], sampling_rate=16000, padding=True, return_tensors="pt")
|
||||
|
||||
logits = model(**inputs).logits
|
||||
predicted_ids = torch.argmax(logits, dim=-1)
|
||||
labels = [model.config.id2label[_id] for _id in predicted_ids.tolist()]
|
||||
```
|
||||
|
||||
## Eval results
|
||||
|
||||
The evaluation metric is accuracy.
|
||||
|
||||
| | **s3prl** | **transformers** |
|
||||
|--------|-----------|------------------|
|
||||
|**test**| `0.9630` | `0.9672` |
|
||||
|
||||
### BibTeX entry and citation info
|
||||
|
||||
```bibtex
|
||||
@article{yang2021superb,
|
||||
title={SUPERB: Speech processing Universal PERformance Benchmark},
|
||||
author={Yang, Shu-wen and Chi, Po-Han and Chuang, Yung-Sung and Lai, Cheng-I Jeff and Lakhotia, Kushal and Lin, Yist Y and Liu, Andy T and Shi, Jiatong and Chang, Xuankai and Lin, Guan-Ting and others},
|
||||
journal={arXiv preprint arXiv:2105.01051},
|
||||
year={2021}
|
||||
}
|
||||
```
|
||||
Loading…
Reference in New Issue