Hubert-Base for Keyword Spotting

The base model is pretrained on 16kHz sampled speech audio. When using the model make sure that your speech input is also sampled at 16Khz. The classification head is trained using the Keyword Spotting part of the SUPERB dataset.

Paper

Authors: Shu-wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y. Lin, Andy T. Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, Tzu-Hsien Huang, Wei-Cheng Tseng, Ko-tik Lee, Da-Rong Liu, Zili Huang, Shuyan Dong, Shang-Wen Li, Shinji Watanabe, Abdelrahman Mohamed, Hung-yi Lee

Abstract Self-supervised learning (SSL) has proven vital for advancing research in natural language processing (NLP) and computer vision (CV). The paradigm pretrains a shared model on large volumes of unlabeled data and achieves state-of-the-art (SOTA) for various tasks with minimal adaptation. However, the speech processing community lacks a similar setup to systematically explore the paradigm. To bridge this gap, we introduce Speech processing Universal PERformance Benchmark (SUPERB). SUPERB is a leaderboard to benchmark the performance of a shared model across a wide range of speech processing tasks with minimal architecture changes and labeled data. Among multiple usages of the shared model, we especially focus on extracting the representation learned from SSL due to its preferable re-usability. We present a simple framework to solve SUPERB tasks by learning task-specialized lightweight prediction heads on top of the frozen shared model. Our results demonstrate that the framework is promising as SSL representations show competitive generalizability and accessibility across SUPERB tasks. We release SUPERB as a challenge with a leaderboard and a benchmark toolkit to fuel the research in representation learning and general speech processing.

The original model can be found under https://github.com/s3prl/s3prl/tree/master/s3prl/downstream/speech_commands.

The base model is hubert-base-ls960

Usage examples

You can use the model via the Audio Classification pipeline:

import numpy as np
from datasets import load_dataset
from transformers import pipeline, PreTrainedTokenizer

superb_ks = load_dataset("anton-l/superb_dummy", "ks", split="test")
model = "superb/hubert-base-superb-ks"
tokenizer = PreTrainedTokenizer()  # a dummy tokenizer, since the classifier doesn't need a real one
classifier = pipeline("audio-classification", model=model, feature_extractor=model, tokenizer=tokenizer)

audio = np.array(superb_ks[0]["speech"])
labels = classifier(audio, top_k=5)

Or use the model directly:

import torch
import numpy as np
from datasets import load_dataset
from transformers import HubertForSequenceClassification, Wav2Vec2FeatureExtractor

superb_ks = load_dataset("anton-l/superb_dummy", "ks", split="test")
model = HubertForSequenceClassification.from_pretrained("superb/hubert-base-superb-ks")
feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained("superb/hubert-base-superb-ks")

audio = np.array(superb_ks[0]["speech"])
# compute attention masks and normalize the waveform if needed
inputs = feature_extractor(audio, sampling_rate=16_000, return_tensors="pt")

logits = model(**inputs).logits
predicted_ids = torch.argmax(logits, dim=-1)
labels = [model.config.id2label[_id] for _id in predicted_ids.tolist()]

3.7 KiB Raw Blame History

Hubert-Base for Keyword Spotting

Usage examples

3.7 KiB

Raw Blame History