Documents Are Not Plain Text
A PDF invoice has amounts in the top-right, addresses in the top-left, and line items in a table. A standard language model that strips formatting sees "Invoice #1234 Acme Corp 123 Main St Total $5,000" with no spatial context. LayoutLMv3 understands that "Total" in the bottom-right with "$5,000" immediately to its right - within the same bounding box row - means something very different from "Total" appearing in a header.
Architecture: Text + Layout + Image Together
LayoutLMv3 uses a unified multimodal transformer that processes three input streams jointly:
- Text tokens - from an OCR engine (Tesseract, Azure Form Recognizer, or any OCR output)
- Layout tokens - 2D bounding box coordinates (x1, y1, x2, y2, width, height) for each text token, normalized to [0, 1000]
- Image patches - the document image divided into 16×16 patches, processed like ViT
The model is pretrained with three objectives: Masked Language Modeling (MLM) on text, Masked Image Modeling (MIM) on image patches, and Word-Patch Alignment (WPA) - predicting whether a text token and an image patch are aligned.
from transformers import LayoutLMv3Processor, LayoutLMv3ForTokenClassification
from PIL import Image
import torch
processor = LayoutLMv3Processor.from_pretrained("microsoft/layoutlmv3-base", apply_ocr=True)
model = LayoutLMv3ForTokenClassification.from_pretrained("microsoft/layoutlmv3-base")
# apply_ocr=True runs Tesseract internally
image = Image.open("receipt.png").convert("RGB")
encoding = processor(image, return_tensors="pt")
with torch.no_grad():
outputs = model(**encoding)
predictions = outputs.logits.argmax(-1).squeeze().tolist()
token_boxes = encoding.bbox.squeeze().tolist()
for token_id, box, pred in zip(encoding.input_ids.squeeze().tolist(), token_boxes, predictions):
token = processor.tokenizer.decode([token_id])
print(f"Token: {token:15} Box: {box} Label: {pred}")
