LayoutLMv3: Understanding PDFs, Forms, and Documents With Layout Awareness

Documents Are Not Plain Text

A PDF invoice has amounts in the top-right, addresses in the top-left, and line items in a table. A standard language model that strips formatting sees "Invoice #1234 Acme Corp 123 Main St Total $5,000" with no spatial context. LayoutLMv3 understands that "Total" in the bottom-right with "$5,000" immediately to its right - within the same bounding box row - means something very different from "Total" appearing in a header.

Architecture: Text + Layout + Image Together

LayoutLMv3 uses a unified multimodal transformer that processes three input streams jointly:

1. Text tokens - from an OCR engine (Tesseract, Azure Form Recognizer, or any OCR output)
2. Layout tokens - 2D bounding box coordinates (x1, y1, x2, y2, width, height) for each text token, normalized to [0, 1000]
3. Image patches - the document image divided into 16×16 patches, processed like ViT

The model is pretrained with three objectives: Masked Language Modeling (MLM) on text, Masked Image Modeling (MIM) on image patches, and Word-Patch Alignment (WPA) - predicting whether a text token and an image patch are aligned.

from transformers import LayoutLMv3Processor, LayoutLMv3ForTokenClassification
from PIL import Image
import torch

processor = LayoutLMv3Processor.from_pretrained("microsoft/layoutlmv3-base", apply_ocr=True)
model = LayoutLMv3ForTokenClassification.from_pretrained("microsoft/layoutlmv3-base")

# apply_ocr=True runs Tesseract internally
image = Image.open("receipt.png").convert("RGB")
encoding = processor(image, return_tensors="pt")

with torch.no_grad():
    outputs = model(**encoding)

predictions = outputs.logits.argmax(-1).squeeze().tolist()
token_boxes = encoding.bbox.squeeze().tolist()

for token_id, box, pred in zip(encoding.input_ids.squeeze().tolist(), token_boxes, predictions):
    token = processor.tokenizer.decode([token_id])
    print(f"Token: {token:15} Box: {box} Label: {pred}")

Fine-Tuning on Custom Documents

For domain-specific forms, fine-tuning LayoutLMv3 on labeled examples consistently outperforms prompting a general VLM:

from transformers import LayoutLMv3ForTokenClassification, TrainingArguments, Trainer

label2id = {"O": 0, "B-DATE": 1, "B-TOTAL": 2, "B-VENDOR": 3, "B-ADDRESS": 4}
id2label = {v: k for k, v in label2id.items()}

model = LayoutLMv3ForTokenClassification.from_pretrained(
    "microsoft/layoutlmv3-base",
    num_labels=len(label2id),
    id2label=id2label,
    label2id=label2id,
)

training_args = TrainingArguments(
    output_dir="layoutlmv3-receipts",
    per_device_train_batch_size=4,
    num_train_epochs=10,
    learning_rate=5e-5,
    fp16=True,
    evaluation_strategy="epoch",
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    tokenizer=processor,
)
trainer.train()

Benchmark Datasets for Document AI

• FUNSD (Form Understanding in Noisy Scanned Documents) - 199 forms with semantic entity labels and entity linking; F1 benchmark for form NER
• CORD (Consolidated Receipt Dataset) - 11,000 receipt images with 30 semantic label categories; used for receipt parsing
• DocVQA - 50,000 QA pairs over scanned documents; requires reading comprehension from layout-aware context

LayoutLMv3 achieves F1 > 90% on FUNSD and CORD with standard fine-tuning recipes.

Integration With Azure Form Recognizer

Azure Form Recognizer uses LayoutLM variants as its underlying model for structured extraction. If you need managed infrastructure with SLA guarantees and pre-built models for invoices, receipts, ID cards, and W-2s, Azure Form Recognizer is the production-ready path that avoids managing model serving yourself.

Links

• HuggingFace model page
• LayoutLMv3 paper
• HuggingFace fine-tune guide