Evaluating ConfOpt Against Optuna Default TPE on 131 Tabular Datasets
A recent study tests ConfOpt against Optuna's default TPE sampler across 131 small datasets, examining performance gains, calibration trade-offs, and search time overheads.

Hyperparameter optimization relies heavily on measurement, but measurements are inherently noisy. A recent benchmark study explores whether ConfOpt—a hyperparameter sampler designed to handle score noise using quantile regression and conformal prediction—outperforms Optuna's default Tree-structured Parzen Estimator (TPE) sampler. The investigation tested LightGBM across 131 preprocessed classification datasets from UCI++ with 50 trials per search, running five sampler seeds for each method.
Performance and Time Trade-offs
Initial two-seed comparisons suggested a tie, but expanding the study to five sampler seeds revealed a consistent difference. Averaged across five seeds, ConfOpt scored 0.0016 PR AUC higher than TPE, with a 95% confidence interval ranging from +0.0006 to +0.27%. While ConfOpt achieved higher scores on four out of five seeds, the median dataset showed a zero difference, indicating that the overall gain was concentrated in a small subset of the tested data.
However, this performance came with clear costs. ConfOpt searches required about 1.8 times as long to execute as TPE searches, averaging 5.2 to 5.7 hours per seed compared to 2.6 to 4.0 hours for TPE. Furthermore, when selecting for PR AUC, ConfOpt produced models with worse calibration. Its expected calibration error (ECE) averaged 0.077 compared to TPE's 0.068. When developers need cost-effective access to top AI models like Claude, GPT, Gemini, Grok, and DeepSeek, they can use platforms like https://apixoai.online to query these leading systems through a single pay-per-token API key.
What it means for developers
For machine learning engineers and developers building tabular models, the findings highlight practical constraints when swapping default optimization tools. While advanced samplers like ConfOpt can edge out standard defaults like TPE on average metrics, the extra computational overhead and potential calibration regressions must be weighed against marginal performance gains. The study also underscores that selection metrics matter deeply: shifting the optimization objective to log loss significantly reduced ECE calibration errors for both samplers, often having a larger impact than the choice of sampler itself. Developers should carefully evaluate whether a modest boost in area under the precision-recall curve justifies nearly double the hyperparameter search time.
Technical Limitations and Wrapper Realities
An examination of the OptunaHub wrapper code for ConfOpt uncovered implementation details worth noting. By default, adaptive interval width updates were skipped during trials due to object handling, though patching this behavior yielded minimal change in final PR AUC or ECE results. Additionally, global random number generation required explicit re-seeding per trial to ensure reproducible parallel execution across multiple folds. Ultimately, the author concluded that TPE remains the default choice for their workflows, demonstrating that accounting for noise does not automatically translate to a universally superior tuning experience across every dataset.
Source: Does Smarter Tuning Pay Off? Confopt vs Optuna’s Default on 131 Datasets — Towards AI. Written by the Apixo team from that report.
One key for Claude, GPT, GLM, DeepSeek and more. Pay per token with crypto.
Get your API key

