The Problem With Dense Retrieval
Dense retrieval models (DPR, ColBERT, nomic-embed) encode queries and documents into dense vectors and retrieve via approximate nearest neighbor search. They handle semantic similarity well but struggle with exact term matching and require specialized vector databases. BM25, the classic TF-IDF variant, does the opposite: exact term matching with fast inverted index lookup, no semantic generalization.
SPLADE (Sparse Lexical And Expansion model) occupies the best of both worlds: sparse representations that work with standard inverted indexes, but with learned weights and vocabulary expansion that capture semantics BM25 misses.
How SPLADE Produces Sparse Representations
SPLADE takes a BERT model and repurposes its masked language model (MLM) head. Instead of predicting masked tokens, it uses the MLM head to score every token in the vocabulary for each input position, then aggregates across positions via max-pooling and applies a log(1 + ReLU(x)) transformation to produce sparse, non-negative weights:
from transformers import AutoTokenizer, AutoModelForMaskedLM
import torch
model_name = "naver/splade-cocondenser-ensembledistil"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForMaskedLM.from_pretrained(model_name)
def encode_splade(text):
inputs = tokenizer(text, return_tensors="pt", max_length=512, truncation=True)
with torch.no_grad():
output = model(**inputs)
logits = output.logits # [1, seq_len, vocab_size]
# Max-pool across sequence positions, apply log(1 + ReLU)
vec = torch.log(1 + torch.relu(logits)).max(dim=1).values.squeeze()
return vec
query_vec = encode_splade("transformer self-attention mechanism explained")
# Most weights are 0; non-zero entries correspond to vocabulary terms
nonzero = (query_vec > 0).sum().item()
print(f"Non-zero terms: {nonzero} out of {len(query_vec)}") # typically 20-200
# See which terms were expanded
nonzero_indices = query_vec.nonzero(as_tuple=True)[0]
terms = [(tokenizer.decode([idx.item()]), query_vec[idx.item()].item())
for idx in nonzero_indices]
print(sorted(terms, key=lambda x: -x[1])[:10])
