PEFT and LoRA: Fine-Tune a 7B LLM on a Single GPU in 2026

PEFT Library Overview

The PEFT (Parameter-Efficient Fine-Tuning) library from HuggingFace implements four main techniques:

• LoRA: Low-rank adaptation - adds trainable rank decomposition matrices to attention layers
• QLoRA: LoRA on a 4-bit quantized base model - the memory breakthrough
• Prefix tuning: Prepends learned virtual tokens to the input
• IA3: Infused Adapter by Inhibiting and Amplifying - even smaller than LoRA

LoRA and QLoRA dominate production use because they're compatible with most architectures and merge back into the base model at inference time.

LoRA: How It Works

LoRA decomposes weight updates into low-rank matrices. For a weight matrix W (d × k), instead of updating W directly, LoRA learns two matrices: A (d × r) and B (r × k), where r << min(d, k). The effective weight update is BA, but only A and B are trained.

Key hyperparameters:

• r (rank): 8-64 typically. Higher r = more capacity but more memory. Start with r=16.
• lora_alpha: Scaling factor, usually 2× the rank value (alpha=32 for r=16)
• target_modules: Which layers to apply LoRA to. For LLaMA: ["q_proj", "v_proj"] or all attention layers

Memory Usage: 7B fp16 vs QLoRA

QLoRA makes 7B fine-tuning possible on a single RTX 4090 (24GB) or even an RTX 3090/4080 (16-24GB).

Full QLoRA Training Script

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, TrainingArguments
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from trl import SFTTrainer
from datasets import load_dataset

# 4-bit quantization config
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    quantization_config=bnb_config,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")
tokenizer.pad_token = tokenizer.eos_token

model = prepare_model_for_kbit_training(model)

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: 33,554,432 || all params: 6,771,970,048 || trainable%: 0.4955

dataset = load_dataset("timdettmers/openassistant-guanaco", split="train")

training_args = TrainingArguments(
    output_dir="./llama2-7b-finetuned",
    num_train_epochs=1,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,  # Effective batch size = 16
    gradient_checkpointing=True,
    optim="paged_adamw_32bit",
    learning_rate=2e-4,
    bf16=True,
    logging_steps=25,
    save_strategy="epoch",
)

trainer = SFTTrainer(
    model=model,
    train_dataset=dataset,
    args=training_args,
    dataset_text_field="text",
    max_seq_length=2048,
)
trainer.train()

Merging and Pushing to HuggingFace Hub

After training, merge the LoRA weights back into the base model for standard inference:

from peft import PeftModel
from transformers import AutoModelForCausalLM

base_model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf", torch_dtype=torch.float16)
model = PeftModel.from_pretrained(base_model, "./llama2-7b-finetuned/checkpoint-final")
merged_model = model.merge_and_unload()

merged_model.push_to_hub("your-username/llama2-7b-custom")
tokenizer.push_to_hub("your-username/llama2-7b-custom")

Dataset Formats

The TRL SFTTrainer accepts two common formats:

Alpaca format (instruction/input/output fields):

{"instruction": "Summarize this text", "input": "Long text here...", "output": "Short summary."}

ShareGPT format (conversations array):

{"conversations": [{"from": "human", "value": "Hello"}, {"from": "gpt", "value": "Hi there!"}]}

The QLoRA paper demonstrates that QLoRA fine-tuned Guanaco-65B matches ChatGPT on the Vicuna benchmark - proving that 4-bit quantization with LoRA does not meaningfully degrade fine-tune quality versus full precision training.

Best Practices for PEFT and LoRA in 2026

Choosing the Right Rank

Start with r=16 for most tasks. For domain adaptation (e.g., legal, medical), r=32 may capture more nuanced patterns. For simple instruction tuning, r=8 is often sufficient. Monitor validation loss to avoid overfitting.

Target Modules Selection

For decoder-only models like LLaMA, target ["q_proj", "v_proj"] is a safe default. Adding k_proj and o_proj increases capacity but also memory. For encoder-decoder models (T5, Flan-T5), target attention and feed-forward layers.

Learning Rate Scheduling

Use a cosine schedule with warmup (10% of steps). QLoRA typically works well with learning rates between 1e-4 and 3e-4. For full LoRA (no quantization), lower to 1e-5 to 5e-5.

Gradient Checkpointing

Always enable gradient checkpointing when VRAM is tight. It trades compute for memory, reducing VRAM by ~30% with minimal slowdown.

Mixed Precision Training

Use bf16 if your GPU supports it (Ampere and later). Otherwise, fp16 with gradient scaling. This halves memory usage compared to fp32.

Cost Analysis: PEFT vs Full Fine-Tuning

QLoRA reduces costs by 10-20x, making fine-tuning accessible to individuals and small teams.

Common Pitfalls and How to Avoid Them

Overfitting on Small Datasets

If your dataset has <1000 examples, use higher dropout (0.1-0.2) and lower rank (r=8). Monitor training loss vs eval loss.

Tokenizer Mismatch

Ensure the tokenizer matches the base model. Adding new tokens requires resizing embeddings and may degrade performance.

Forgetting to Merge Weights

For deployment, always merge LoRA weights into the base model. Loading separate adapters adds latency and complexity.

Future Trends: PEFT in 2026

By 2026, PEFT techniques have evolved:

• DoRA: Weight-decomposed low-rank adaptation, outperforming LoRA on several benchmarks.
• LoRA-FA: Freezes the random projection matrix A, reducing memory further.
• Multi-task LoRA: Shares adapters across tasks with task-specific scaling.

Despite these innovations, LoRA and QLoRA remain the go-to choices due to their simplicity and broad support.

Conclusion

PEFT and LoRA have democratized LLM fine-tuning. With QLoRA, anyone with a consumer GPU can fine-tune a 7B model for under $20. The key is understanding hyperparameters, quantization trade-offs, and proper training setup. Start with the provided script, experiment with rank and target modules, and iterate from there.