Weights & Biases for LLM Fine-Tuning: Track Every Run and Compare Results
W&B provides experiment tracking for fine-tuning runs and LLM tracing via Weave, letting you compare models, trace agent calls, and manage models in a production registry.
Fine-tuning is iterative. You run with lr=2e-4, then try 5e-4. You experiment with 100 examples, then 500. Without tracking, you lose the context of what you tried and why something worked. Weights & Biases (W&B) captures every hyperparameter, metric, and artifact automatically.
Integrating with HuggingFace Trainer
The simplest integration requires one line:
from transformers import TrainingArguments
training_args = TrainingArguments(
output_dir="./outputs",
num_train_epochs=3,
per_device_train_batch_size=4,
learning_rate=2e-4,
report_to="wandb", # this one line
run_name="llama3-qlora-v2",
)
W&B automatically logs training loss, validation loss, learning rate schedule, gradient norms, and system metrics (GPU utilization, memory). No other code changes needed.
Team workspace
Ship faster with chat, meetings, and projects in one place — Zlyqor.
For custom training loops or logging additional metrics:
import wandb
wandb.init(
project="llm-fine-tuning",
config={
"model": "meta-llama/Meta-Llama-3-8B",
"lora_r": 16,
"dataset_size": 500,
"learning_rate": 2e-4,
}
)
for step, batch in enumerate(train_loader):
loss = train_step(batch)
wandb.log({"train/loss": loss, "train/step": step})
wandb.finish()
W&B Weave for LLM Tracing
Weave is W&B's LLM observability layer. It automatically instruments OpenAI and Anthropic API calls when you add two lines:
import weave
from openai import OpenAI
weave.init("my-llm-app") # that's it - OpenAI calls are now traced
client = OpenAI()
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Hello"}]
)
Every call is logged with model, input, output, latency, token counts, and cost. For agent workflows, you see the full call tree showing how sub-calls compose.
Comparing Fine-Tuned vs Base Model
Weave's Evaluation feature lets you run both models on the same dataset and compare outputs side-by-side with automatic metrics:
import weave
@weave.op()
def evaluate_model(model_name: str, prompt: str) -> str:
# call your model and return the output
...
evaluation = weave.Evaluation(
dataset=test_cases,
scorers=[accuracy_scorer, format_scorer],
)
asyncio.run(evaluation.evaluate(base_model_predict))
asyncio.run(evaluation.evaluate(finetuned_model_predict))
Model Registry
After fine-tuning, register your model artifact for production:
artifact = wandb.Artifact("llama3-customer-support", type="model")
artifact.add_dir("./outputs/final_model")
wandb.log_artifact(artifact)
# Link to registry for deployment tracking
artifact.link("my-org/model-registry/llama3-customer-support", aliases=["v2", "production"])
Practical deep-dives on LLMs, developer tools, and AI engineering. No filler. Unsubscribe any time.
// written byFIG. AUTH-01
538
Mahmudul Haque Qudrati
CEO & ML Engineer
CEO and ML Engineer at Pristren. Builds AI-powered software for teams and writes about machine learning, LLMs, developer tools, and practical AI applications.
What Is Alibaba Banning Claude Code Over Backdoor Risks? A Practical Overview
Alibaba is reportedly banning Claude Code from its workplace due to alleged backdoor risks. This post explains the incident, the technical concerns, and the broader implications for AI coding assistants in regulated environments.
