LoRA Explained: How 0.1% of Parameters Can Match Full Fine-Tuning

The Problem: Full Fine-Tuning Is Prohibitively Expensive

Fine-tuning a 175B parameter model like GPT-3 requires storing the full parameter gradients and optimizer states. With Adam, that is roughly 175B parameters × 4 bytes (fp32) × 3 (params + two Adam states) = over 2TB of GPU memory. Even with mixed precision, you need hundreds of GB across dozens of A100s. This is not accessible for most teams.

The LoRA Insight: Weight Updates Are Low-Rank

The key observation in the LoRA paper (arXiv:2106.09685) by Hu et al. is that the weight update matrix during fine-tuning has a low intrinsic rank. Rather than learning a full d×d update, you can decompose it as:

W = W_0 + BA

where W_0 is the frozen pretrained weight, B is d×r, and A is r×d, with rank r much smaller than d. For a 4096×4096 weight matrix with r=8, that is 4096×4096 = 16.7M parameters reduced to 2×(4096×8) = 65K parameters - a 256x reduction.

Rank and Alpha Hyperparameters

The rank r controls the expressivity of the adaptation. Higher rank captures more complex fine-tuning signals but uses more memory. The alpha hyperparameter scales the LoRA output: the effective update is (alpha/r) × BA. Setting alpha = 2r means the scaling is constant at 2 regardless of rank choice, which simplifies hyperparameter search. Common settings: r=4 or r=8 for most tasks, r=16 or r=64 for complex domain adaptation.

Memory Savings in Practice

LoRA is typically applied to the attention weight matrices (Q, K, V, and output projection). Applying it to all linear layers including MLP further improves quality.

Zero Inference Latency

Because W_0 + BA can be pre-merged before serving, LoRA adds exactly zero latency at inference time. This is a critical advantage over adapters and prefix tuning, which add extra computation on every forward pass.

LoRA vs Adapters vs Prefix Tuning

Adapters insert small bottleneck layers between Transformer blocks - they add inference latency. Prefix tuning prepends trainable soft tokens to the input - it reduces the effective context window. LoRA modifies the weight matrices directly with no architectural change and no runtime overhead.

import torch
import torch.nn as nn

class LoRALinear(nn.Module):
    def __init__(self, in_features, out_features, rank=8, alpha=16):
        super().__init__()
        self.weight = nn.Parameter(
            torch.randn(out_features, in_features), requires_grad=False
        )
        self.lora_A = nn.Parameter(torch.randn(rank, in_features) * 0.01)
        self.lora_B = nn.Parameter(torch.zeros(out_features, rank))
        self.scale = alpha / rank

    def forward(self, x):
        base = x @ self.weight.T
        lora = x @ self.lora_A.T @ self.lora_B.T
        return base + self.scale * lora

Further Reading

• LoRA paper arXiv:2106.09685
• HuggingFace PEFT library