QLoRA: Fine-Tune a 65B LLM on a Single 48GB GPU

The Goal: 65B Parameters on One GPU

Even with LoRA, fine-tuning a 65B LLM requires storing the frozen base model in memory - roughly 130GB in fp16. The QLoRA paper (arXiv:2305.14314) by Dettmers et al. at the University of Washington solved this with three innovations that together compress the base model to fit on a single 48GB A40 or A100 GPU.

Innovation 1: 4-bit NormalFloat (NF4) Quantization

Standard 4-bit integers allocate bit ranges uniformly, but neural network weights are not uniformly distributed - they follow a roughly normal (Gaussian) distribution. NF4 is an information-theoretically optimal quantization for normally distributed data. It places quantization levels at equal quantiles of the normal distribution, meaning more levels near zero (where weights cluster) and fewer at the extremes.

The result is that NF4 quantization incurs less rounding error than INT4 for neural network weights. Each weight is stored in 4 bits but dequantized to bf16 for computation. The base model goes from ~130GB (fp16) to ~32.5GB (4-bit) - a 4x compression.

Innovation 2: Double Quantization

The quantization constants (the scale factors used to dequantize 4-bit weights back to bf16) are themselves fp32 values. QLoRA quantizes these constants too - quantizing the quantization - reducing the memory overhead of quantization metadata from about 0.5 bits per parameter to about 0.2 bits per parameter.

Innovation 3: Paged Optimizers

GPU memory usage spikes during certain operations (gradient checkpointing, long sequences). These spikes can cause out-of-memory errors even when average memory use fits. Paged optimizers use NVIDIA's unified memory to automatically page optimizer states from GPU to CPU RAM during spikes, and back when needed. This prevents crashes during training without significant throughput loss.

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import get_peft_model, LoraConfig
import torch

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",          # NF4 quantization
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,      # Double quantization
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-65b-hf",
    quantization_config=bnb_config,
    device_map="auto",
)

lora_config = LoraConfig(r=64, lora_alpha=16, target_modules=["q_proj", "v_proj"])
model = get_peft_model(model, lora_config)

The Guanaco Models

The QLoRA paper trained the Guanaco family of models by fine-tuning LLaMA on 9,000 samples of the OASST1 dataset for roughly 24 hours on a single GPU. Guanaco 65B, trained for $300 in cloud GPU costs, reached 99.3% of ChatGPT performance on the Vicuna benchmark - demonstrating that high-quality instruction tuning did not require massive compute budgets.

GPU Memory Requirements

Further Reading

• QLoRA paper arXiv:2305.14314
• BitsAndBytes library