The BERT 512-Token Problem
Standard transformer attention is quadratic in sequence length - doubling the sequence length quadruples computation. BERT's 512-token limit means long documents must be chunked, losing inter-chunk context. A question that spans two chunks, or a document summary that requires reading the conclusion before understanding the introduction, breaks with naive chunking.
Longformer from AllenAI solves this with two attention mechanisms applied together:
- Sliding window attention: Each token attends to
window_sizeneighbors on each side. Local context is preserved efficiently at O(n × w) cost. - Global attention: Selected tokens (like
[CLS]or question tokens) attend to all other tokens and are attended to by all other tokens. Global tokens gather document-level context.
This combination maintains O(n) complexity while preserving the ability to reason across the full document.
When to Use Longformer vs Chunking Strategies
Use Longformer when:
- Questions or labels depend on evidence scattered across the document
- You need document-level representations (not sentence-level)
- Documents are 600-4000 tokens consistently
Use chunking when:
- Documents are 4000+ tokens (Longformer's 4096 limit still applies)
- Questions can always be answered from a local passage
- Throughput matters more than accuracy (chunking + retrieval is faster)
The sweet spot is 800-2500 token documents - long enough that BERT fails, short enough that Longformer handles cleanly.
