Fine-tuning large language models (LLMs) has become essential for adapting general-purpose models to specific domains and tasks. This guide covers modern fine-tuning techniques that balance performance with computational efficiency.
LoRA vs Full Fine-Tuning Comparison
Why Fine-Tune LLMs?
While pre-trained models like GPT-4, Claude, or Llama 2 are powerful, fine-tuning offers several advantages:
- Domain Adaptation: Specialize models for specific industries or use cases
- Improved Accuracy: Better performance on task-specific benchmarks
- Cost Efficiency: Smaller fine-tuned models can outperform larger general models
- Custom Behavior: Control model outputs and align with brand voice
- Data Privacy: Keep sensitive data in-house
Fine-Tuning Approaches
1. Full Fine-Tuning
Update all model parameters:
from transformers import AutoModelForCausalLM, AutoTokenizer, Trainer
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")
trainer = Trainer(
model=model,
train_dataset=train_dataset,
eval_dataset=eval_dataset,
args=training_args,
)
trainer.train()
Pros: Maximum flexibility, best performance Cons: High memory usage, expensive, slow
2. LoRA (Low-Rank Adaptation)
Fine-tune small adapter matrices instead of full weights:
from peft import get_peft_model, LoraConfig, TaskType
lora_config = LoraConfig(
task_type=TaskType.CAUSAL_LM,
r=16, # Rank
lora_alpha=32,
lora_dropout=0.05,
target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
)
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")
model = get_peft_model(model, lora_config)
print(f"Trainable params: {model.print_trainable_parameters()}")
# Trainable params: 4,194,304 || all params: 6,742,609,920 || trainable%: 0.06%
Pros: 100x fewer parameters, faster training, minimal memory Cons: Slightly lower performance than full fine-tuning
3. QLoRA (Quantized LoRA)
Combine LoRA with 4-bit quantization:
from transformers import BitsAndBytesConfig
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_use_double_quant=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
)
model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-2-7b-hf",
quantization_config=bnb_config,
device_map="auto",
)
model = get_peft_model(model, lora_config)
Pros: Fine-tune 65B models on single consumer GPU Cons: Quantization may impact quality slightly
4. Prefix Tuning
Add trainable prompt embeddings:
from peft import PrefixTuningConfig
prefix_config = PrefixTuningConfig(
task_type=TaskType.CAUSAL_LM,
num_virtual_tokens=30,
encoder_hidden_size=4096,
)
model = get_peft_model(model, prefix_config)
Data Preparation
Quality Over Quantity
# Example: High-quality training data format
training_data = [
{
"instruction": "Explain quantum entanglement",
"input": "",
"output": "Quantum entanglement is a phenomenon where two or more particles become correlated in such a way that the quantum state of one particle cannot be described independently..."
},
{
"instruction": "Write a Python function to",
"input": "calculate fibonacci numbers",
"output": "def fibonacci(n):
if n <= 1:
return n
return fibonacci(n-1) + fibonacci(n-2)"
}
]
Data Formatting
def format_instruction(sample):
"""Format samples for instruction tuning"""
instruction = sample["instruction"]
input_text = sample.get("input", "")
output = sample["output"]
if input_text:
prompt = f"""Below is an instruction with additional context. Write a response that appropriately completes the request.
### Instruction:
{instruction}
### Input:
{input_text}
### Response:
{output}"""
else:
prompt = f"""Below is an instruction. Write a response that appropriately completes the request.
### Instruction:
{instruction}
### Response:
{output}"""
return tokenizer(
prompt,
truncation=True,
max_length=2048,
padding="max_length",
)
tokenized_dataset = dataset.map(format_instruction)
Training Configuration
Optimal Hyperparameters
from transformers import TrainingArguments
training_args = TrainingArguments(
output_dir="./results",
num_train_epochs=3,
per_device_train_batch_size=4,
per_device_eval_batch_size=4,
gradient_accumulation_steps=4, # Effective batch size = 16
learning_rate=2e-4,
lr_scheduler_type="cosine",
warmup_ratio=0.03,
weight_decay=0.01,
fp16=True, # Mixed precision training
logging_steps=10,
eval_steps=100,
save_steps=100,
save_total_limit=2,
load_best_model_at_end=True,
metric_for_best_model="eval_loss",
greater_is_better=False,
report_to="wandb", # Experiment tracking
)
Gradient Checkpointing
Reduce memory usage:
model.gradient_checkpointing_enable()
model.config.use_cache = False # Incompatible with checkpointing
DeepSpeed Integration
For multi-GPU training:
// ds_config.json
{
"fp16": {
"enabled": true
},
"zero_optimization": {
"stage": 2,
"offload_optimizer": {
"device": "cpu"
}
},
"gradient_accumulation_steps": 4,
"train_micro_batch_size_per_gpu": 4
}
training_args = TrainingArguments(
...
deepspeed="ds_config.json"
)
Evaluation
Perplexity
import torch
from torch.nn import CrossEntropyLoss
def calculate_perplexity(model, eval_dataloader):
model.eval()
losses = []
for batch in eval_dataloader:
with torch.no_grad():
outputs = model(**batch)
losses.append(outputs.loss.item())
return torch.exp(torch.tensor(losses).mean())
perplexity = calculate_perplexity(model, eval_dataloader)
print(f"Perplexity: {perplexity:.2f}")
Task-Specific Metrics
from datasets import load_metric
# For classification tasks
metric = load_metric("accuracy")
def compute_metrics(eval_pred):
logits, labels = eval_pred
predictions = np.argmax(logits, axis=-1)
return metric.compute(predictions=predictions, references=labels)
Human Evaluation
Create evaluation prompts:
test_prompts = [
"Explain how transformers work",
"Write a sorting algorithm in Python",
"What are the benefits of exercise?",
]
for prompt in test_prompts:
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(
**inputs,
max_length=200,
temperature=0.7,
top_p=0.9,
do_sample=True,
)
print(tokenizer.decode(outputs[0]))
Common Pitfalls
1. Overfitting
Symptoms: Low training loss, high validation loss
Solutions:
- Increase dropout
- Use more training data
- Early stopping
- Data augmentation
2. Catastrophic Forgetting
Symptoms: Model forgets general knowledge
Solutions:
- Mix general and domain-specific data (80/20 ratio)
- Use instruction tuning format
- Lower learning rate
- Replay mechanism
3. Mode Collapse
Symptoms: Model generates repetitive outputs
Solutions:
- Diverse training data
- Temperature sampling
- Nucleus (top-p) sampling
- Repetition penalty
Advanced Techniques
Instruction Tuning
# Use instruction-following datasets
from datasets import load_dataset
dataset = load_dataset("tatsu-lab/alpaca")
# Contains 52K instruction-following examples
RLHF (Reinforcement Learning from Human Feedback)
from trl import PPOTrainer, PPOConfig
ppo_config = PPOConfig(
model_name="fine-tuned-model",
learning_rate=1.41e-5,
batch_size=16,
)
ppo_trainer = PPOTrainer(
config=ppo_config,
model=model,
ref_model=ref_model,
tokenizer=tokenizer,
dataset=dataset,
data_collator=collator,
)
Multi-Task Learning
Train on multiple tasks simultaneously:
# Combine different task datasets
mixed_dataset = concatenate_datasets([
qa_dataset,
summarization_dataset,
classification_dataset,
])
Deployment
Model Merging
Merge LoRA weights back into base model:
from peft import PeftModel
base_model = AutoModelForCausalLM.from_pretrained("base-model")
peft_model = PeftModel.from_pretrained(base_model, "lora-adapter")
merged_model = peft_model.merge_and_unload()
merged_model.save_pretrained("merged-model")
Quantization for Inference
# Convert to GGUF for llama.cpp deployment
!python convert-to-gguf.py merged-model
!./quantize merged-model.gguf merged-model-q4.gguf q4_0
Cost Optimization
- Use QLoRA for large models
- Gradient accumulation instead of large batch sizes
- Mixed precision (FP16/BF16)
- Spot instances for training
- Start with smaller models (7B before 70B)
Conclusion
Fine-tuning LLMs has become accessible through techniques like LoRA and QLoRA. Success requires:
- High-quality training data
- Appropriate fine-tuning method
- Careful hyperparameter selection
- Comprehensive evaluation
- Monitoring for common issues
With modern tools and techniques, you can fine-tune state-of-the-art models on consumer hardware and achieve impressive results for specialized tasks.
