Wav2Vec 2.0: Self-Supervised Speech Recognition for Low-Resource Languages

Self-Supervised Pre-Training

The fundamental problem with ASR for low-resource languages: collecting transcribed speech is expensive. Wav2Vec 2.0 solves this by separating pre-training (unlabeled audio only) from fine-tuning (small labeled set).

Pre-training works through contrastive learning:

1. Raw audio is processed through a convolutional feature extractor
2. Features are quantized into a discrete codebook of speech units
3. A transformer context network predicts quantized representations for masked time steps
4. The model learns to distinguish true masked tokens from distractors

This self-supervised objective learns phonetic and acoustic structure from unlabeled speech - recordings of any kind, without transcripts.

Fine-Tuning With 10 Minutes of Labeled Data

After pre-training on unlabeled audio, fine-tuning with CTC (Connectionist Temporal Classification) requires remarkably little labeled data. The fine-tuning guide demonstrates fine-tuning on 960 hours for English, but researchers have achieved workable WER with 10 minutes for low-resource languages.

from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
import torch
import librosa

processor = Wav2Vec2Processor.from_pretrained("facebook/wav2vec2-large-960h-lv60-self")
model = Wav2Vec2ForCTC.from_pretrained("facebook/wav2vec2-large-960h-lv60-self")

# Load audio (must be 16kHz)
audio, sr = librosa.load("speech.wav", sr=16000)
input_values = processor(audio, sampling_rate=16000, return_tensors="pt").input_values

with torch.no_grad():
    logits = model(input_values).logits

predicted_ids = torch.argmax(logits, dim=-1)
transcription = processor.batch_decode(predicted_ids)
print(transcription[0])

The HuggingFace Wav2Vec2 page hosts the 960h-lv60-self model - the strongest self-supervised English variant.

CTC Decoder With Language Model Integration

Pure CTC decoding outputs the most likely character sequence independently at each timestep. Adding a language model (kenlm) significantly reduces WER by incorporating word-level probability:

from pyctcdecode import build_ctcdecoder
import kenlm

vocab = list(processor.tokenizer.get_vocab().keys())
language_model = kenlm.Model("language_model.arpa")

decoder = build_ctcdecoder(
    vocab,
    kenlm_model=language_model,
    alpha=0.5,  # LM weight
    beta=1.5,   # Word insertion bonus
)

# Decode logits with LM
logits_np = logits[0].numpy()
transcription_lm = decoder.decode(logits_np)

XLSR-53: Multilingual Variant

XLSR-53 pre-trains on 53 languages simultaneously, enabling cross-lingual transfer. Fine-tune XLSR-53 on a low-resource language and it leverages phonetic knowledge from related languages. This is particularly effective for language families - fine-tuning for Catalan benefits from Spanish pre-training data.

Custom Dataset Fine-Tuning

Prepare data as HuggingFace Dataset with audio (16kHz) and text columns. The fine-tuning guide walks through vocabulary building, CTC tokenizer setup, and DataCollatorCTCWithPadding. Training a language-specific model from XLSR-53 requires approximately 4-8 GPU hours on an A10G for a 100-hour dataset.