PEFT Library Overview
The PEFT (Parameter-Efficient Fine-Tuning) library from HuggingFace implements four main techniques:
- LoRA: Low-rank adaptation - adds trainable rank decomposition matrices to attention layers
- QLoRA: LoRA on a 4-bit quantized base model - the memory breakthrough
- Prefix tuning: Prepends learned virtual tokens to the input
- IA3: Infused Adapter by Inhibiting and Amplifying - even smaller than LoRA
LoRA and QLoRA dominate production use because they're compatible with most architectures and merge back into the base model at inference time.
LoRA: How It Works
LoRA decomposes weight updates into low-rank matrices. For a weight matrix W (d × k), instead of updating W directly, LoRA learns two matrices: A (d × r) and B (r × k), where r << min(d, k). The effective weight update is BA, but only A and B are trained.
Key hyperparameters:
- r (rank): 8-64 typically. Higher r = more capacity but more memory. Start with r=16.
- lora_alpha: Scaling factor, usually 2× the rank value (alpha=32 for r=16)
- target_modules: Which layers to apply LoRA to. For LLaMA:
["q_proj", "v_proj"]or all attention layers
Memory Usage: 7B fp16 vs QLoRA
| 7B fp16 (full fine-tune) | 28GB+ |
| 7B fp16 (inference only) | 14GB |
| 7B int8 + LoRA | 10GB |
| 7B int4 (QLoRA) | 4-5GB |
| 7B int4 + gradient checkpointing | 3.5-4GB |
QLoRA makes 7B fine-tuning possible on a single RTX 4090 (24GB) or even an RTX 3090/4080 (16-24GB).
Full QLoRA Training Script
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, TrainingArguments
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from trl import SFTTrainer
from datasets import load_dataset
# 4-bit quantization config
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True,
)
model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-2-7b-hf",
quantization_config=bnb_config,
device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")
tokenizer.pad_token = tokenizer.eos_token
model = prepare_model_for_kbit_training(model)
lora_config = LoraConfig(
r=16,
lora_alpha=32,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: 33,554,432 || all params: 6,771,970,048 || trainable%: 0.4955
dataset = load_dataset("timdettmers/openassistant-guanaco", split="train")
training_args = TrainingArguments(
output_dir="./llama2-7b-finetuned",
num_train_epochs=1,
per_device_train_batch_size=2,
gradient_accumulation_steps=8, # Effective batch size = 16
gradient_checkpointing=True,
optim="paged_adamw_32bit",
learning_rate=2e-4,
bf16=True,
logging_steps=25,
save_strategy="epoch",
)
trainer = SFTTrainer(
model=model,
train_dataset=dataset,
args=training_args,
dataset_text_field="text",
max_seq_length=2048,
)
trainer.train()Merging and Pushing to HuggingFace Hub
After training, merge the LoRA weights back into the base model for standard inference:
from peft import PeftModel
from transformers import AutoModelForCausalLM
base_model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf", torch_dtype=torch.float16)
model = PeftModel.from_pretrained(base_model, "./llama2-7b-finetuned/checkpoint-final")
merged_model = model.merge_and_unload()
merged_model.push_to_hub("your-username/llama2-7b-custom")
tokenizer.push_to_hub("your-username/llama2-7b-custom")Dataset Formats
The TRL SFTTrainer accepts two common formats:
Alpaca format (instruction/input/output fields):
{"instruction": "Summarize this text", "input": "Long text here...", "output": "Short summary."}ShareGPT format (conversations array):
{"conversations": [{"from": "human", "value": "Hello"}, {"from": "gpt", "value": "Hi there!"}]}The QLoRA paper demonstrates that QLoRA fine-tuned Guanaco-65B matches ChatGPT on the Vicuna benchmark - proving that 4-bit quantization with LoRA does not meaningfully degrade fine-tune quality versus full precision training.
Best Practices for PEFT and LoRA in 2026
Choosing the Right Rank
Start with r=16 for most tasks. For domain adaptation (e.g., legal, medical), r=32 may capture more nuanced patterns. For simple instruction tuning, r=8 is often sufficient. Monitor validation loss to avoid overfitting.
Target Modules Selection
For decoder-only models like LLaMA, target ["q_proj", "v_proj"] is a safe default. Adding k_proj and o_proj increases capacity but also memory. For encoder-decoder models (T5, Flan-T5), target attention and feed-forward layers.
Learning Rate Scheduling
Use a cosine schedule with warmup (10% of steps). QLoRA typically works well with learning rates between 1e-4 and 3e-4. For full LoRA (no quantization), lower to 1e-5 to 5e-5.
Gradient Checkpointing
Always enable gradient checkpointing when VRAM is tight. It trades compute for memory, reducing VRAM by ~30% with minimal slowdown.
Mixed Precision Training
Use bf16 if your GPU supports it (Ampere and later). Otherwise, fp16 with gradient scaling. This halves memory usage compared to fp32.
Cost Analysis: PEFT vs Full Fine-Tuning
| Method | GPU Hours (7B) | Cloud Cost (approx) |
|---|---|---|
| Full fine-tune (fp16) | 48 hrs on A100 | $200-400 |
| QLoRA (int4) | 12 hrs on RTX 4090 | $10-20 (spot) |
| LoRA (int8) | 20 hrs on RTX 4090 | $15-30 (spot) |
QLoRA reduces costs by 10-20x, making fine-tuning accessible to individuals and small teams.
Common Pitfalls and How to Avoid Them
Overfitting on Small Datasets
If your dataset has <1000 examples, use higher dropout (0.1-0.2) and lower rank (r=8). Monitor training loss vs eval loss.
Tokenizer Mismatch
Ensure the tokenizer matches the base model. Adding new tokens requires resizing embeddings and may degrade performance.
Forgetting to Merge Weights
For deployment, always merge LoRA weights into the base model. Loading separate adapters adds latency and complexity.
Future Trends: PEFT in 2026
By 2026, PEFT techniques have evolved:
- DoRA: Weight-decomposed low-rank adaptation, outperforming LoRA on several benchmarks.
- LoRA-FA: Freezes the random projection matrix A, reducing memory further.
- Multi-task LoRA: Shares adapters across tasks with task-specific scaling.
Despite these innovations, LoRA and QLoRA remain the go-to choices due to their simplicity and broad support.
Conclusion
PEFT and LoRA have democratized LLM fine-tuning. With QLoRA, anyone with a consumer GPU can fine-tune a 7B model for under $20. The key is understanding hyperparameters, quantization trade-offs, and proper training setup. Start with the provided script, experiment with rank and target modules, and iterate from there.


Comments
Have a question or something to add? Sign in to join the discussion.