DistilBERT vs BERT
The DistilBERT paper introduced knowledge distillation for transformer compression. A large BERT teacher trains a 6-layer DistilBERT student to mimic its output distributions, not just match labels. The result: 66M parameters (vs BERT-base's 110M), 60% faster inference, 40% smaller, with 97% of BERT's GLUE performance.
For production classification (sentiment, intent, topic), that 3% gap rarely matters. DistilBERT's latency advantage often matters more than marginal accuracy when serving thousands of requests per second.
The HuggingFace DistilBERT page provides cased and uncased variants.
Fine-Tuning for Classification With Trainer API
from transformers import (
DistilBertForSequenceClassification,
DistilBertTokenizerFast,
Trainer,
TrainingArguments,
)
from datasets import load_dataset
import numpy as np
tokenizer = DistilBertTokenizerFast.from_pretrained("distilbert-base-uncased")
model = DistilBertForSequenceClassification.from_pretrained(
"distilbert-base-uncased",
num_labels=2
)
dataset = load_dataset("imdb")
def tokenize(batch):
return tokenizer(batch["text"], truncation=True, padding=True, max_length=512)
dataset = dataset.map(tokenize, batched=True, batch_size=1000)
training_args = TrainingArguments(
output_dir="./distilbert-sentiment",
num_train_epochs=3,
per_device_train_batch_size=32,
per_device_eval_batch_size=64,
evaluation_strategy="epoch",
learning_rate=2e-5,
)
trainer = Trainer(
model=model,
args=training_args,
train_dataset=dataset["train"],
eval_dataset=dataset["test"],
)
trainer.train()
