Why Grid Search Is Insufficient

One AI engineering post, weekly

LLM benchmarks, prompt techniques, and token-cost breakdowns — not another AI news roundup.

GridSearchCV exhaustively tries every combination. With 5 hyperparameters and 5 values each, that is 5^5 = 3,125 trials. Most of those are wasted on obviously bad combinations.

Optuna uses Bayesian optimization with Tree-structured Parzen Estimators (TPE). It builds a probabilistic model of which hyperparameters lead to good results and samples more from promising regions. You get better results with fewer trials.

Basic Optuna Setup

python

import optuna
import lightgbm as lgb
from sklearn.model_selection import cross_val_score
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)

def objective(trial: optuna.Trial) -> float:
    params = {
        "n_estimators": trial.suggest_int("n_estimators", 100, 2000),
        "num_leaves": trial.suggest_int("num_leaves", 20, 300),
        "learning_rate": trial.suggest_float("learning_rate", 1e-4, 0.3, log=True),
        "min_child_samples": trial.suggest_int("min_child_samples", 5, 100),
        "subsample": trial.suggest_float("subsample", 0.5, 1.0),
        "colsample_bytree": trial.suggest_float("colsample_bytree", 0.5, 1.0),
    }
    model = lgb.LGBMClassifier(**params, random_state=42)
    scores = cross_val_score(model, X, y, cv=5, scoring="roc_auc", n_jobs=-1)
    return scores.mean()

study = optuna.create_study(direction="maximize")
study.optimize(objective, n_trials=100, n_jobs=1)

print(f"Best AUC: {study.best_value:.4f}")
print(f"Best params: {study.best_params}")

Team workspace

Ship faster with chat, meetings, and projects in one place — Zlyqor.

Start free

Pruning: Stop Bad Trials Early

Optuna can prune trials that are clearly worse than the best so far, saving significant compute:

python

import optuna
import pytorch_lightning as pl
from optuna.integration import PyTorchLightningPruningCallback

def objective(trial: optuna.Trial) -> float:
    lr = trial.suggest_float("lr", 1e-5, 1e-2, log=True)
    hidden_size = trial.suggest_categorical("hidden_size", [64, 128, 256, 512])

    model = MyModel(lr=lr, hidden_size=hidden_size)
    trainer = pl.Trainer(
        max_epochs=50,
        callbacks=[PyTorchLightningPruningCallback(trial, monitor="val_loss")],
    )
    trainer.fit(model, train_loader, val_loader)
    return trainer.callback_metrics["val_loss"].item()

study = optuna.create_study(
    direction="minimize",
    pruner=optuna.pruners.MedianPruner(n_startup_trials=5, n_warmup_steps=10),
)
study.optimize(objective, n_trials=100)

The MedianPruner stops a trial if its intermediate value is worse than the median of completed trials at the same step.

Distributed Optimization

Run Optuna across multiple machines using a shared database:

python

import optuna

# All workers share this study via PostgreSQL
study = optuna.create_study(
    study_name="distributed_lgbm",
    storage="postgresql://user:pass@db-host/optuna",
    direction="maximize",
    load_if_exists=True,
)
study.optimize(objective, n_trials=50)  # run on each machine

RDB storage also enables persistent studies - restart a study without losing previous results.

Visualization

python

from optuna.visualization import (
    plot_optimization_history,
    plot_param_importances,
    plot_parallel_coordinate,
)

# Which trials converged to best values
fig = plot_optimization_history(study)
fig.show()

# Which hyperparameters matter most (Fanova-based)
fig = plot_param_importances(study)
fig.show()

# Parallel coordinates  -  see correlations between params and objective
fig = plot_parallel_coordinate(study)
fig.show()

Optuna Dashboard

bash

pip install optuna-dashboard
optuna-dashboard postgresql://user:pass@db-host/optuna

Real-time dashboard showing trial progress, parameter importances, and contour plots - accessible to the whole team.

Resources: Optuna, GitHub, docs.

#optuna#hyperparameter-optimization#bayesian#tpe#automl

Mahmudul Haque Qudrati

CEO & ML Engineer

Visionary technologist, software engineer, and machine learning specialist. Founder and CEO of Pristren, directing engineering teams that ship production-grade AI/ML pipelines, mission-critical full-stack applications, and developer tooling. Creator of Zlyqor, the unified team workspace platform. Author of 540+ technical guides and benchmark research reports on large language models, agentic workflows, Model Context Protocol (MCP), and modern web stacks.