Optuna: Modern Hyperparameter Optimization That Beats Grid Search

Optuna uses Tree-structured Parzen Estimators to learn from previous trials and focus on promising regions - finding better hyperparameters in fewer trials than grid or random search.

Mahmudul Haque Qudrati

CEO & ML Engineer

April 23, 2026

7 min read

// tags

#optuna#hyperparameter-optimization#bayesian#tpe#automl

FIG. ART-28

7 min read

“

// reading plan

sections

358

words

min read

// Machine Learning

ONNX: Export Any ML Model and Run It Anywhere

ONNX (Open Neural Network Exchange) is the universal model format - export from PyTorch, scikit-learn, or HuggingFace and run 3x faster inference with ONNX Runtime on CPU or GPU.

7 min read

// Machine Learning

Gradient Descent Explained: How Machine Learning Models Actually Learn

Pruning: Stop Bad Trials Early

Optuna can prune trials that are clearly worse than the best so far, saving significant compute:

import optuna
import pytorch_lightning as pl
from optuna.integration import PyTorchLightningPruningCallback

def objective(trial: optuna.Trial) -> float:
    lr = trial.suggest_float("lr", 1e-5, 1e-2, log=True)
    hidden_size = trial.suggest_categorical("hidden_size", [64, 128, 256, 512])

    model = MyModel(lr=lr, hidden_size=hidden_size)
    trainer = pl.Trainer(
        max_epochs=50,
        callbacks=[PyTorchLightningPruningCallback(trial, monitor="val_loss")],
    )
    trainer.fit(model, train_loader, val_loader)
    return trainer.callback_metrics["val_loss"].item()

study = optuna.create_study(
    direction="minimize",
    pruner=optuna.pruners.MedianPruner(n_startup_trials=5, n_warmup_steps=10),
)
study.optimize(objective, n_trials=100)

The MedianPruner stops a trial if its intermediate value is worse than the median of completed trials at the same step.

Distributed Optimization

Run Optuna across multiple machines using a shared database:

import optuna

# All workers share this study via PostgreSQL
study = optuna.create_study(
    study_name="distributed_lgbm",
    storage="postgresql://user:pass@db-host/optuna",
    direction="maximize",
    load_if_exists=True,
)
study.optimize(objective, n_trials=50)  # run on each machine

RDB storage also enables persistent studies - restart a study without losing previous results.

Visualization

from optuna.visualization import (
    plot_optimization_history,
    plot_param_importances,
    plot_parallel_coordinate,
)

# Which trials converged to best values
fig = plot_optimization_history(study)
fig.show()

# Which hyperparameters matter most (Fanova-based)
fig = plot_param_importances(study)
fig.show()

# Parallel coordinates  -  see correlations between params and objective
fig = plot_parallel_coordinate(study)
fig.show()

Optuna Dashboard

pip install optuna-dashboard
optuna-dashboard postgresql://user:pass@db-host/optuna

Real-time dashboard showing trial progress, parameter importances, and contour plots - accessible to the whole team.

Resources: Optuna, GitHub, docs.

Optuna: Modern Hyperparameter Optimization That Beats Grid Search

Related Articles

ONNX: Export Any ML Model and Run It Anywhere

Gradient Descent Explained: How Machine Learning Models Actually Learn

Why Grid Search Is Insufficient

Basic Optuna Setup

Pruning: Stop Bad Trials Early

Distributed Optimization

Visualization

Optuna Dashboard

The workspace your team
actually needs

AI & ML insights, weekly

Mahmudul Haque Qudrati

Supervised Learning Explained: How Models Learn from Labeled Examples

Optuna: Modern Hyperparameter Optimization That Beats Grid Search

Related Articles

ONNX: Export Any ML Model and Run It Anywhere

Gradient Descent Explained: How Machine Learning Models Actually Learn

Why Grid Search Is Insufficient

Basic Optuna Setup

Pruning: Stop Bad Trials Early

Distributed Optimization

Visualization

Optuna Dashboard

The workspace your teamactually needs

AI & ML insights, weekly

Mahmudul Haque Qudrati

Supervised Learning Explained: How Models Learn from Labeled Examples

The workspace your team
actually needs