Fine-Tuning Large Language Models: Strategies and Best Practices

Learn effective strategies for fine-tuning LLMs including LoRA, QLoRA, and PEFT techniques. Covers data preparation, training optimization, and evaluation.

Mahmudul Haque Qudrati

CEO & ML Engineer

November 15, 2024

16 min read

Fine-Tuning Large Language Models: Strategies and Best Practices

Contents

Why Fine-Tune LLMs?
Fine-Tuning Approaches
Data Preparation
Training Configuration
Evaluation
Common Pitfalls
Advanced Techniques
Deployment
Cost Optimization
Conclusion

Fine-tuning large language models (LLMs) has become essential for adapting general-purpose models to specific domains and tasks. This guide covers modern fine-tuning techniques that balance performance with computational efficiency.

LoRA vs Full Fine-Tuning Comparison

Why Fine-Tune LLMs?

One AI engineering post, weekly

LLM benchmarks, prompt techniques, and token-cost breakdowns — not another AI news roundup.

While pre-trained models like GPT-4, Claude, or Llama 2 are powerful, fine-tuning offers several advantages:

Domain Adaptation: Specialize models for specific industries or use cases
Improved Accuracy: Better performance on task-specific benchmarks
Cost Efficiency: Smaller fine-tuned models can outperform larger general models
Custom Behavior: Control model outputs and align with brand voice
Data Privacy: Keep sensitive data in-house

Fine-Tuning Approaches

1. Full Fine-Tuning

Update all model parameters:

from transformers import AutoModelForCausalLM, AutoTokenizer, Trainer

model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")

trainer = Trainer(
    model=model,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    args=training_args,
)

trainer.train()

Pros: Maximum flexibility, best performance Cons: High memory usage, expensive, slow

2. LoRA (Low-Rank Adaptation)

Fine-tune small adapter matrices instead of full weights:

from peft import get_peft_model, LoraConfig, TaskType

lora_config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=16,  # Rank
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
)

model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")
model = get_peft_model(model, lora_config)

print(f"Trainable params: {model.print_trainable_parameters()}")
# Trainable params: 4,194,304 || all params: 6,742,609,920 || trainable%: 0.06%

Pros: 100x fewer parameters, faster training, minimal memory Cons: Slightly lower performance than full fine-tuning

3. QLoRA (Quantized LoRA)

Combine LoRA with 4-bit quantization:

from transformers import BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    quantization_config=bnb_config,
    device_map="auto",
)

model = get_peft_model(model, lora_config)

Pros: Fine-tune 65B models on single consumer GPU Cons: Quantization may impact quality slightly

4. Prefix Tuning

Add trainable prompt embeddings:

from peft import PrefixTuningConfig

prefix_config = PrefixTuningConfig(
    task_type=TaskType.CAUSAL_LM,
    num_virtual_tokens=30,
    encoder_hidden_size=4096,
)

model = get_peft_model(model, prefix_config)

Team workspace

Ship faster with chat, meetings, and projects in one place — Zlyqor.

Start free

Data Preparation

Quality Over Quantity

# Example: High-quality training data format
training_data = [
    {
        "instruction": "Explain quantum entanglement",
        "input": "",
        "output": "Quantum entanglement is a phenomenon where two or more particles become correlated in such a way that the quantum state of one particle cannot be described independently..."
    },
    {
        "instruction": "Write a Python function to",
        "input": "calculate fibonacci numbers",
        "output": "def fibonacci(n):
    if n <= 1:
        return n
    return fibonacci(n-1) + fibonacci(n-2)"
    }
]

Data Formatting

def format_instruction(sample):
    """Format samples for instruction tuning"""
    instruction = sample["instruction"]
    input_text = sample.get("input", "")
    output = sample["output"]

    if input_text:
        prompt = f"""Below is an instruction with additional context. Write a response that appropriately completes the request.

### Instruction:
{instruction}

### Input:
{input_text}

### Response:
{output}"""
    else:
        prompt = f"""Below is an instruction. Write a response that appropriately completes the request.

### Instruction:
{instruction}

### Response:
{output}"""

    return tokenizer(
        prompt,
        truncation=True,
        max_length=2048,
        padding="max_length",
    )

tokenized_dataset = dataset.map(format_instruction)

Training Configuration

Optimal Hyperparameters

from transformers import TrainingArguments

training_args = TrainingArguments(
    output_dir="./results",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    per_device_eval_batch_size=4,
    gradient_accumulation_steps=4,  # Effective batch size = 16
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    weight_decay=0.01,
    fp16=True,  # Mixed precision training
    logging_steps=10,
    eval_steps=100,
    save_steps=100,
    save_total_limit=2,
    load_best_model_at_end=True,
    metric_for_best_model="eval_loss",
    greater_is_better=False,
    report_to="wandb",  # Experiment tracking
)

Gradient Checkpointing

Reduce memory usage:

model.gradient_checkpointing_enable()
model.config.use_cache = False  # Incompatible with checkpointing

DeepSpeed Integration

For multi-GPU training:

// ds_config.json
{
  "fp16": {
    "enabled": true
  },
  "zero_optimization": {
    "stage": 2,
    "offload_optimizer": {
      "device": "cpu"
    }
  },
  "gradient_accumulation_steps": 4,
  "train_micro_batch_size_per_gpu": 4
}

training_args = TrainingArguments(
    ...
    deepspeed="ds_config.json"
)

Evaluation

Perplexity

import torch
from torch.nn import CrossEntropyLoss

def calculate_perplexity(model, eval_dataloader):
    model.eval()
    losses = []

    for batch in eval_dataloader:
        with torch.no_grad():
            outputs = model(**batch)
            losses.append(outputs.loss.item())

    return torch.exp(torch.tensor(losses).mean())

perplexity = calculate_perplexity(model, eval_dataloader)
print(f"Perplexity: {perplexity:.2f}")

Task-Specific Metrics

from datasets import load_metric

# For classification tasks
metric = load_metric("accuracy")

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predictions = np.argmax(logits, axis=-1)
    return metric.compute(predictions=predictions, references=labels)

Human Evaluation

Create evaluation prompts:

test_prompts = [
    "Explain how transformers work",
    "Write a sorting algorithm in Python",
    "What are the benefits of exercise?",
]

for prompt in test_prompts:
    inputs = tokenizer(prompt, return_tensors="pt")
    outputs = model.generate(
        **inputs,
        max_length=200,
        temperature=0.7,
        top_p=0.9,
        do_sample=True,
    )
    print(tokenizer.decode(outputs[0]))

Common Pitfalls

1. Overfitting

Symptoms: Low training loss, high validation loss

Solutions:

Increase dropout
Use more training data
Early stopping
Data augmentation

2. Catastrophic Forgetting

Symptoms: Model forgets general knowledge

Solutions:

Mix general and domain-specific data (80/20 ratio)
Use instruction tuning format
Lower learning rate
Replay mechanism

3. Mode Collapse

Symptoms: Model generates repetitive outputs

Solutions:

Diverse training data
Temperature sampling
Nucleus (top-p) sampling
Repetition penalty

Advanced Techniques

Instruction Tuning

# Use instruction-following datasets
from datasets import load_dataset

dataset = load_dataset("tatsu-lab/alpaca")
# Contains 52K instruction-following examples

RLHF (Reinforcement Learning from Human Feedback)

from trl import PPOTrainer, PPOConfig

ppo_config = PPOConfig(
    model_name="fine-tuned-model",
    learning_rate=1.41e-5,
    batch_size=16,
)

ppo_trainer = PPOTrainer(
    config=ppo_config,
    model=model,
    ref_model=ref_model,
    tokenizer=tokenizer,
    dataset=dataset,
    data_collator=collator,
)

Multi-Task Learning

Train on multiple tasks simultaneously:

# Combine different task datasets
mixed_dataset = concatenate_datasets([
    qa_dataset,
    summarization_dataset,
    classification_dataset,
])

Deployment

Model Merging

Merge LoRA weights back into base model:

from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained("base-model")
peft_model = PeftModel.from_pretrained(base_model, "lora-adapter")

merged_model = peft_model.merge_and_unload()
merged_model.save_pretrained("merged-model")

Quantization for Inference

# Convert to GGUF for llama.cpp deployment
!python convert-to-gguf.py merged-model
!./quantize merged-model.gguf merged-model-q4.gguf q4_0

Cost Optimization

Use QLoRA for large models
Gradient accumulation instead of large batch sizes
Mixed precision (FP16/BF16)
Spot instances for training
Start with smaller models (7B before 70B)

Conclusion

Fine-tuning LLMs has become accessible through techniques like LoRA and QLoRA. Success requires:

High-quality training data
Appropriate fine-tuning method
Careful hyperparameter selection
Comprehensive evaluation
Monitoring for common issues

With modern tools and techniques, you can fine-tune state-of-the-art models on consumer hardware and achieve impressive results for specialized tasks.

#llm#fine-tuning#lora#machine-learning#nlp

// discussion

Comments

Mahmudul Haque Qudrati

CEO & ML Engineer

Visionary technologist, software engineer, and machine learning specialist. Founder and CEO of Pristren, directing engineering teams that ship production-grade AI/ML pipelines, mission-critical full-stack applications, and developer tooling. Creator of Zlyqor, the unified team workspace platform. Author of 540+ technical guides and benchmark research reports on large language models, agentic workflows, Model Context Protocol (MCP), and modern web stacks.

Continue reading

Next.js 14 Server Actions: A Complete Guide

14 min read

11 min read

Machine Learning: Complete Guide for Software Developers

Learn machine learning as a software developer with this complete guide covering Python, algorithms, mathematics, projects, deep learning, and a practical roadmap.

5 min read

GPT-6 Astra Benchmarks, Pricing, and Safety: What Developers Need to Know

OpenAI's GPT-6 Astra delivers state-of-the-art coding and agentic performance, but costs 50% more than GPT-4o. This guide covers benchmarks, pricing, safety, and practical advice for developers deciding whether to upgrade.

5 min read

What Is GPT-5.6 Sol Ultra Will Be in Codex? A Practical Overview

GPT-5.6 Sol Ultra is a rumored model optimized for code generation, integrated into Codex. We analyze the claims, potential capabilities, and what developers should expect.

Back to blog

Artificial Intelligence

Fine-Tuning Large Language Models: Strategies and Best Practices

Learn effective strategies for fine-tuning LLMs including LoRA, QLoRA, and PEFT techniques. Covers data preparation, training optimization, and evaluation.

Mahmudul Haque Qudrati

CEO & ML Engineer

November 15, 2024

16 min read

Contents

Why Fine-Tune LLMs?
Fine-Tuning Approaches
Data Preparation
Training Configuration
Evaluation
Common Pitfalls
Advanced Techniques
Deployment
Cost Optimization
Conclusion

LoRA vs Full Fine-Tuning Comparison

Why Fine-Tune LLMs?

One AI engineering post, weekly

LLM benchmarks, prompt techniques, and token-cost breakdowns — not another AI news roundup.

While pre-trained models like GPT-4, Claude, or Llama 2 are powerful, fine-tuning offers several advantages:

Domain Adaptation: Specialize models for specific industries or use cases
Improved Accuracy: Better performance on task-specific benchmarks
Cost Efficiency: Smaller fine-tuned models can outperform larger general models
Custom Behavior: Control model outputs and align with brand voice
Data Privacy: Keep sensitive data in-house

Fine-Tuning Approaches

1. Full Fine-Tuning

Update all model parameters:

from transformers import AutoModelForCausalLM, AutoTokenizer, Trainer

model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")

trainer = Trainer(
    model=model,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    args=training_args,
)

trainer.train()

Pros: Maximum flexibility, best performance Cons: High memory usage, expensive, slow

2. LoRA (Low-Rank Adaptation)

Fine-tune small adapter matrices instead of full weights:

from peft import get_peft_model, LoraConfig, TaskType

lora_config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=16,  # Rank
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
)

model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")
model = get_peft_model(model, lora_config)

print(f"Trainable params: {model.print_trainable_parameters()}")
# Trainable params: 4,194,304 || all params: 6,742,609,920 || trainable%: 0.06%

Pros: 100x fewer parameters, faster training, minimal memory Cons: Slightly lower performance than full fine-tuning

3. QLoRA (Quantized LoRA)

Combine LoRA with 4-bit quantization:

from transformers import BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    quantization_config=bnb_config,
    device_map="auto",
)

model = get_peft_model(model, lora_config)

Pros: Fine-tune 65B models on single consumer GPU Cons: Quantization may impact quality slightly

4. Prefix Tuning

Add trainable prompt embeddings:

from peft import PrefixTuningConfig

prefix_config = PrefixTuningConfig(
    task_type=TaskType.CAUSAL_LM,
    num_virtual_tokens=30,
    encoder_hidden_size=4096,
)

model = get_peft_model(model, prefix_config)

Team workspace

Ship faster with chat, meetings, and projects in one place — Zlyqor.

Start free

Data Preparation

Quality Over Quantity

# Example: High-quality training data format
training_data = [
    {
        "instruction": "Explain quantum entanglement",
        "input": "",
        "output": "Quantum entanglement is a phenomenon where two or more particles become correlated in such a way that the quantum state of one particle cannot be described independently..."
    },
    {
        "instruction": "Write a Python function to",
        "input": "calculate fibonacci numbers",
        "output": "def fibonacci(n):
    if n <= 1:
        return n
    return fibonacci(n-1) + fibonacci(n-2)"
    }
]

Data Formatting

def format_instruction(sample):
    """Format samples for instruction tuning"""
    instruction = sample["instruction"]
    input_text = sample.get("input", "")
    output = sample["output"]

    if input_text:
        prompt = f"""Below is an instruction with additional context. Write a response that appropriately completes the request.

### Instruction:
{instruction}

### Input:
{input_text}

### Response:
{output}"""
    else:
        prompt = f"""Below is an instruction. Write a response that appropriately completes the request.

### Instruction:
{instruction}

### Response:
{output}"""

    return tokenizer(
        prompt,
        truncation=True,
        max_length=2048,
        padding="max_length",
    )

tokenized_dataset = dataset.map(format_instruction)

Training Configuration

Optimal Hyperparameters

from transformers import TrainingArguments

training_args = TrainingArguments(
    output_dir="./results",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    per_device_eval_batch_size=4,
    gradient_accumulation_steps=4,  # Effective batch size = 16
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    weight_decay=0.01,
    fp16=True,  # Mixed precision training
    logging_steps=10,
    eval_steps=100,
    save_steps=100,
    save_total_limit=2,
    load_best_model_at_end=True,
    metric_for_best_model="eval_loss",
    greater_is_better=False,
    report_to="wandb",  # Experiment tracking
)

Gradient Checkpointing

Reduce memory usage:

model.gradient_checkpointing_enable()
model.config.use_cache = False  # Incompatible with checkpointing

DeepSpeed Integration

For multi-GPU training:

// ds_config.json
{
  "fp16": {
    "enabled": true
  },
  "zero_optimization": {
    "stage": 2,
    "offload_optimizer": {
      "device": "cpu"
    }
  },
  "gradient_accumulation_steps": 4,
  "train_micro_batch_size_per_gpu": 4
}

training_args = TrainingArguments(
    ...
    deepspeed="ds_config.json"
)

Evaluation

Perplexity

import torch
from torch.nn import CrossEntropyLoss

def calculate_perplexity(model, eval_dataloader):
    model.eval()
    losses = []

    for batch in eval_dataloader:
        with torch.no_grad():
            outputs = model(**batch)
            losses.append(outputs.loss.item())

    return torch.exp(torch.tensor(losses).mean())

perplexity = calculate_perplexity(model, eval_dataloader)
print(f"Perplexity: {perplexity:.2f}")

Task-Specific Metrics

from datasets import load_metric

# For classification tasks
metric = load_metric("accuracy")

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predictions = np.argmax(logits, axis=-1)
    return metric.compute(predictions=predictions, references=labels)

Human Evaluation

Create evaluation prompts:

test_prompts = [
    "Explain how transformers work",
    "Write a sorting algorithm in Python",
    "What are the benefits of exercise?",
]

for prompt in test_prompts:
    inputs = tokenizer(prompt, return_tensors="pt")
    outputs = model.generate(
        **inputs,
        max_length=200,
        temperature=0.7,
        top_p=0.9,
        do_sample=True,
    )
    print(tokenizer.decode(outputs[0]))

Common Pitfalls

1. Overfitting

Symptoms: Low training loss, high validation loss

Solutions:

Increase dropout
Use more training data
Early stopping
Data augmentation

2. Catastrophic Forgetting

Symptoms: Model forgets general knowledge

Solutions:

Mix general and domain-specific data (80/20 ratio)
Use instruction tuning format
Lower learning rate
Replay mechanism

3. Mode Collapse

Symptoms: Model generates repetitive outputs

Solutions:

Diverse training data
Temperature sampling
Nucleus (top-p) sampling
Repetition penalty

Advanced Techniques

Instruction Tuning

# Use instruction-following datasets
from datasets import load_dataset

dataset = load_dataset("tatsu-lab/alpaca")
# Contains 52K instruction-following examples

RLHF (Reinforcement Learning from Human Feedback)

from trl import PPOTrainer, PPOConfig

ppo_config = PPOConfig(
    model_name="fine-tuned-model",
    learning_rate=1.41e-5,
    batch_size=16,
)

ppo_trainer = PPOTrainer(
    config=ppo_config,
    model=model,
    ref_model=ref_model,
    tokenizer=tokenizer,
    dataset=dataset,
    data_collator=collator,
)

Multi-Task Learning

Train on multiple tasks simultaneously:

# Combine different task datasets
mixed_dataset = concatenate_datasets([
    qa_dataset,
    summarization_dataset,
    classification_dataset,
])

Deployment

Model Merging

Merge LoRA weights back into base model:

from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained("base-model")
peft_model = PeftModel.from_pretrained(base_model, "lora-adapter")

merged_model = peft_model.merge_and_unload()
merged_model.save_pretrained("merged-model")

Quantization for Inference

# Convert to GGUF for llama.cpp deployment
!python convert-to-gguf.py merged-model
!./quantize merged-model.gguf merged-model-q4.gguf q4_0

Cost Optimization

Use QLoRA for large models
Gradient accumulation instead of large batch sizes
Mixed precision (FP16/BF16)
Spot instances for training
Start with smaller models (7B before 70B)

Conclusion

Fine-tuning LLMs has become accessible through techniques like LoRA and QLoRA. Success requires:

High-quality training data
Appropriate fine-tuning method
Careful hyperparameter selection
Comprehensive evaluation
Monitoring for common issues

With modern tools and techniques, you can fine-tune state-of-the-art models on consumer hardware and achieve impressive results for specialized tasks.

#llm#fine-tuning#lora#machine-learning#nlp

// discussion

Comments

Mahmudul Haque Qudrati

CEO & ML Engineer

Continue reading

Next.js 14 Server Actions: A Complete Guide

14 min read

11 min read

Fine-Tuning Large Language Models: Strategies and Best Practices

Why Fine-Tune LLMs?

One AI engineering post, weekly

Fine-Tuning Approaches

1. Full Fine-Tuning

2. LoRA (Low-Rank Adaptation)

3. QLoRA (Quantized LoRA)

4. Prefix Tuning

Data Preparation

Quality Over Quantity

Data Formatting

Training Configuration

Optimal Hyperparameters

Gradient Checkpointing

DeepSpeed Integration

Evaluation

Perplexity

Task-Specific Metrics

Human Evaluation

Common Pitfalls

1. Overfitting

2. Catastrophic Forgetting

3. Mode Collapse

Advanced Techniques

Instruction Tuning

RLHF (Reinforcement Learning from Human Feedback)

Multi-Task Learning

Deployment

Model Merging

Quantization for Inference

Cost Optimization

Conclusion

Comments

Mahmudul Haque Qudrati

Continue reading

Related Articles

Machine Learning: Complete Guide for Software Developers

GPT-6 Astra Benchmarks, Pricing, and Safety: What Developers Need to Know

What Is GPT-5.6 Sol Ultra Will Be in Codex? A Practical Overview

Fine-Tuning Large Language Models: Strategies and Best Practices

Why Fine-Tune LLMs?

One AI engineering post, weekly

Fine-Tuning Approaches

1. Full Fine-Tuning

2. LoRA (Low-Rank Adaptation)

3. QLoRA (Quantized LoRA)

4. Prefix Tuning

Data Preparation

Quality Over Quantity

Data Formatting

Training Configuration

Optimal Hyperparameters

Gradient Checkpointing

DeepSpeed Integration

Evaluation

Perplexity

Task-Specific Metrics

Human Evaluation

Common Pitfalls

1. Overfitting

2. Catastrophic Forgetting

3. Mode Collapse

Advanced Techniques

Instruction Tuning

RLHF (Reinforcement Learning from Human Feedback)

Multi-Task Learning

Deployment

Model Merging

Quantization for Inference

Cost Optimization

Conclusion

Comments

Mahmudul Haque Qudrati

Continue reading

Related Articles

Machine Learning: Complete Guide for Software Developers

GPT-6 Astra Benchmarks, Pricing, and Safety: What Developers Need to Know

What Is GPT-5.6 Sol Ultra Will Be in Codex? A Practical Overview