RoBERTa: BERT Done Right - When and How to Use It for Classification

Updated
7 min read
RoBERTa: BERT Done Right - When and How to Use It for Classification
TL;DR

RoBERTa improves on BERT through better pre-training - dynamic masking, no next-sentence prediction, larger batches, and more data - delivering consistent GLUE leaderboard advantages for classification tasks.

What RoBERTa Fixed in BERT

BERT's pre-training had three weaknesses that RoBERTa corrected:

  1. Static masking: BERT masks the same tokens every epoch. RoBERTa uses dynamic masking - different tokens masked each epoch, forcing more robust representations.
  2. Next Sentence Prediction (NSP): BERT's NSP objective was shown to hurt performance. RoBERTa removes it entirely, training only on masked language modeling.
  3. Training scale: BERT trained on 16GB for 1M steps. RoBERTa trained on 160GB for 500K steps with larger batch sizes.

The result: RoBERTa-base consistently outperforms BERT-base on GLUE tasks by 3-7 points without architectural changes. The HuggingFace RoBERTa page provides base and large variants.

Fine-Tuning in Under 30 Lines

python
from transformers import RobertaForSequenceClassification, RobertaTokenizer, Trainer, TrainingArguments
from datasets import load_dataset

tokenizer = RobertaTokenizer.from_pretrained("roberta-base")
model = RobertaForSequenceClassification.from_pretrained("roberta-base", num_labels=4)

dataset = load_dataset("ag_news")

def tokenize_fn(batch):
    return tokenizer(batch["text"], truncation=True, padding="max_length", max_length=256)

tokenized = dataset.map(tokenize_fn, batched=True)
tokenized = tokenized.rename_column("label", "labels")

args = TrainingArguments(
    output_dir="roberta-ag-news",
    num_train_epochs=3,
    per_device_train_batch_size=32,
    evaluation_strategy="epoch",
    load_best_model_at_end=True,
)

trainer = Trainer(model=model, args=args, train_dataset=tokenized["train"], eval_dataset=tokenized["test"])
trainer.train()

Multi-Label Classification

For multi-label tasks (a text belonging to multiple categories simultaneously), change the loss function:

python
import torch
from transformers import RobertaForSequenceClassification

model = RobertaForSequenceClassification.from_pretrained("roberta-base", num_labels=10, problem_type="multi_label_classification")
# Trainer handles BCEWithLogitsLoss automatically when labels are float tensors
# Labels should be float tensors: [0.0, 1.0, 0.0, 1.0, ...]

Few-Shot Classification With SetFit

For low-data scenarios (<50 labeled examples), SetFit outperforms RoBERTa fine-tuning by using contrastive sentence transformer training:

python
from setfit import SetFitModel, SetFitTrainer
from datasets import Dataset

train_data = Dataset.from_dict({
    "text": ["positive example 1", "negative example 1", ...],  # 8-16 examples per class
    "label": [1, 0, ...]
})

model = SetFitModel.from_pretrained("sentence-transformers/paraphrase-mpnet-base-v2")
trainer = SetFitTrainer(model=model, train_dataset=train_data)
trainer.train()

SetFit requires no GPU for fine-tuning with small datasets and reaches full RoBERTa fine-tune quality with 50x less data.

Production ONNX Export

Export fine-tuned RoBERTa to ONNX following the same pattern as DistilBERT. RoBERTa-base in ONNX on CPU achieves 15-25ms latency per request - acceptable for most classification use cases without GPU infrastructure.

#roberta#bert#text-classification#nlp#fine-tuning

One AI engineering post, weekly

LLM benchmarks, prompt techniques, and token-cost breakdowns, delivered to your inbox.

What should we test or explain next?

Comments

Have a question or something to add? Sign in to join the discussion.

Mahmudul Haque Qudrati — CEO & ML Engineer at Pristren

Mahmudul Haque Qudrati

CEO & ML Engineer

Software architect, full-stack engineer, and machine learning practitioner. Founder & CEO of Pristren, CTO at Venture Kick-backed PIN Sport Application GmbH (Basel, Switzerland), Software Development Lead at DailyFrame GmbH (building Dimetra patient transport software), and creator of Zlyqor. Focused on custom workflow automation, scalable web and mobile applications, and production AI/ML systems.