Optuna: Modern Hyperparameter Optimization That Beats Grid Search
Optuna uses Tree-structured Parzen Estimators to learn from previous trials and focus on promising regions - finding better hyperparameters in fewer trials than grid or random search.
GridSearchCV exhaustively tries every combination. With 5 hyperparameters and 5 values each, that is 5^5 = 3,125 trials. Most of those are wasted on obviously bad combinations.
Optuna uses Bayesian optimization with Tree-structured Parzen Estimators (TPE). It builds a probabilistic model of which hyperparameters lead to good results and samples more from promising regions. You get better results with fewer trials.
The MedianPruner stops a trial if its intermediate value is worse than the median of completed trials at the same step.
Distributed Optimization
Run Optuna across multiple machines using a shared database:
import optuna
# All workers share this study via PostgreSQL
study = optuna.create_study(
study_name="distributed_lgbm",
storage="postgresql://user:pass@db-host/optuna",
direction="maximize",
load_if_exists=True,
)
study.optimize(objective, n_trials=50) # run on each machine
RDB storage also enables persistent studies - restart a study without losing previous results.
Visualization
from optuna.visualization import (
plot_optimization_history,
plot_param_importances,
plot_parallel_coordinate,
)
# Which trials converged to best values
fig = plot_optimization_history(study)
fig.show()
# Which hyperparameters matter most (Fanova-based)
fig = plot_param_importances(study)
fig.show()
# Parallel coordinates - see correlations between params and objective
fig = plot_parallel_coordinate(study)
fig.show()
Practical deep-dives on LLMs, developer tools, and AI engineering. No filler. Unsubscribe any time.
// written byFIG. AUTH-01
530
Mahmudul Haque Qudrati
CEO & ML Engineer
CEO and ML Engineer at Pristren. Builds AI-powered software for teams and writes about machine learning, LLMs, developer tools, and practical AI applications.
Supervised learning is the most widely used ML paradigm. Here is exactly how the train-measure-adjust loop works, where labels come from, and when the approach breaks down.
