Why Experiment Tracking Matters for Fine-Tuning

One AI engineering post, weekly

LLM benchmarks, prompt techniques, and token-cost breakdowns — not another AI news roundup.

Fine-tuning is iterative. You run with lr=2e-4, then try 5e-4. You experiment with 100 examples, then 500. Without tracking, you lose the context of what you tried and why something worked. Weights & Biases (W&B) captures every hyperparameter, metric, and artifact automatically.

Integrating with HuggingFace Trainer

The simplest integration requires one line:

python

from transformers import TrainingArguments

training_args = TrainingArguments(
    output_dir="./outputs",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    learning_rate=2e-4,
    report_to="wandb",          # this one line
    run_name="llama3-qlora-v2",
)

W&B automatically logs training loss, validation loss, learning rate schedule, gradient norms, and system metrics (GPU utilization, memory). No other code changes needed.

Team workspace

Ship faster with chat, meetings, and projects in one place — Zlyqor.

Start free

Direct Logging

For custom training loops or logging additional metrics:

python

import wandb

wandb.init(
    project="llm-fine-tuning",
    config={
        "model": "meta-llama/Meta-Llama-3-8B",
        "lora_r": 16,
        "dataset_size": 500,
        "learning_rate": 2e-4,
    }
)

for step, batch in enumerate(train_loader):
    loss = train_step(batch)
    wandb.log({"train/loss": loss, "train/step": step})

wandb.finish()

W&B Weave for LLM Tracing

Weave is W&B's LLM observability layer. It automatically instruments OpenAI and Anthropic API calls when you add two lines:

python

import weave
from openai import OpenAI

weave.init("my-llm-app")   # that's it  -  OpenAI calls are now traced

client = OpenAI()
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello"}]
)

Every call is logged with model, input, output, latency, token counts, and cost. For agent workflows, you see the full call tree showing how sub-calls compose.

Comparing Fine-Tuned vs Base Model

Weave's Evaluation feature lets you run both models on the same dataset and compare outputs side-by-side with automatic metrics:

python

import weave

@weave.op()
def evaluate_model(model_name: str, prompt: str) -> str:
    # call your model and return the output
    ...

evaluation = weave.Evaluation(
    dataset=test_cases,
    scorers=[accuracy_scorer, format_scorer],
)
asyncio.run(evaluation.evaluate(base_model_predict))
asyncio.run(evaluation.evaluate(finetuned_model_predict))

Model Registry

After fine-tuning, register your model artifact for production:

python

artifact = wandb.Artifact("llama3-customer-support", type="model")
artifact.add_dir("./outputs/final_model")
wandb.log_artifact(artifact)

# Link to registry for deployment tracking
artifact.link("my-org/model-registry/llama3-customer-support", aliases=["v2", "production"])

Resources

#weights-&-biases#mlops#fine-tuning#experiment-tracking#weave

Mahmudul Haque Qudrati

CEO & ML Engineer

Visionary technologist, software engineer, and machine learning specialist. Founder and CEO of Pristren, directing engineering teams that ship production-grade AI/ML pipelines, mission-critical full-stack applications, and developer tooling. Creator of Zlyqor, the unified team workspace platform. Author of 540+ technical guides and benchmark research reports on large language models, agentic workflows, Model Context Protocol (MCP), and modern web stacks.