Practical early stopping for adaptive barrier SGD: balancing stochastic speed with validation accuracy

Articles in Press

Document Type : Research Article

Authors

Abdelouakil Fillali

Smail Addoune

Rebiha Zeghdane

Faculty of Mathematics and Informatics, Department of Mathematics, University Mohamed El Bachir El Ibrahimi of Bordj Bou-Arreridj,Algeria.

10.22067/ijnao.2026.96710.1788

Abstract

Stochastic Gradient Descent (SGD) algorithms based on adaptive barrier functions are highly efficient for large-scale constrained optimization. However, their inherent stochastic nature leads to the critical ”last- iterate problem”, where update noise causes significant oscillations, making reliance on the final solution a high-stakes gamble. To address this instability, we introduce the Early-Stopped Barrier SGD (EB-SGD) algorithm, a practical approach that replaces reliance on the noisy last iterate with a robust ”best-iterate tracking” mechanism. Our method employs periodic validation, where the cheap stochastic process is temporarily halted to compute a high-quality, deterministic ”Validation Score.” This score is defined as a combination of the full-batch objective function and a large penalty for constraint violation. We maintain the iterate that achieves the best score recorded so far and utilize a ”patience”-based stopping criterion to filter out minor stochastic oscillations and halt the algorithm only upon persistent stagnation. This strategy introduces a deliberate computational trade-off: exchanging many inexpensive stochastic steps for fewer, more expensive validation steps. We prove theoretically that the sequence of best iterates returned by EB-SGD converges almost surely to the unique optimal solution. Furthermore, our numerical experiments confirm the practical superiority of this approach, demonstrating significantly reduced variance (higher stability) and finding a final solution that is substantially more accurate than the baseline last-iterate method.

Keywords

Constrained Optimization

Stochastic Gradient Descent

Early Stopping

Best-Iterate Tracking

Subjects

• Optimization

---