Stochastic Optimization

ICLR 2024arXiv:2309.06657

#2

Statistical Rejection Sampling Improves Preference Optimization

Tianqi Liu, Yao Zhao, Rishabh Joshi et al.

329

ICLR 2024arXiv:2305.14342

#3

Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training

Hong Liu, Zhiyuan Li, David Hall et al.

241

ICML 2024arXiv:2402.05749

#4

Generalized Preference Optimization: A Unified Approach to Offline Alignment

Yunhao Tang, Zhaohan Guo, Zeyu Zheng et al.

preference optimizationoffline alignmentkl divergence regularizationconvex loss functions+4

150

ICML 2025arXiv:2502.07529

#5

Training Deep Learning Models with Norm-Constrained LMOs

Thomas Pethick, Wanyun Xie, Kimon Antonakopoulos et al.

72

ICLR 2024arXiv:2309.01753

#6

On Penalty Methods for Nonconvex Bilevel Optimization and First-Order Stochastic Approximation

Jeongyeol Kwon, Dohyun Kwon, Stephen Wright et al.

56

ICLR 2025arXiv:2501.05803

#7

Test-time Alignment of Diffusion Models without Reward Over-optimization

Sunwoo Kim, Minkyu Kim, Dongmin Park

47

ICLR 2024arXiv:2303.03751

#8

Zeroth-Order Optimization Meets Human Feedback: Provable Learning via Ranking Oracles

Zhiwei Tang, Dmitry Rybin, Tsung-Hui Chang

42

optimization algorithmsautoregressive language modelingadam optimizerlayerwise adaptation+4

#9

Deconstructing What Makes a Good Optimizer for Autoregressive Language Models

Rosie Zhao, Depen Morwani, David Brandfonbrener et al.

ICLR 2025

37

ICML 2024arXiv:1809.04564

#10

On the Generalization of Stochastic Gradient Descent with Momentum

Ali Ramezani-Kebrya, Kimon Antonakopoulos, Volkan Cevher et al.

stochastic gradient descentmomentum methodsgeneralization errorstability gap+3

36

ICLR 2024arXiv:2211.09359

#11

How to Fine-Tune Vision Models with SGD

Ananya Kumar, Ruoqi Shen, Sebastien Bubeck et al.

36

ICLR 2025arXiv:2409.14989

#12

Methods for Convex $(L_0,L_1)$-Smooth Optimization: Clipping, Acceleration, and Adaptivity

Eduard Gorbunov, Nazarii Tupitsa, Sayantan Choudhury et al.

31

NEURIPS 2025arXiv:2503.20762

#13

ASGO: Adaptive Structured Gradient Optimization

Kang An, Yuxing Liu, Rui Pan et al.

structured optimization problemlow-rank gradientsblock diagonal hessianspreconditioner design+2

31

ICLR 2025arXiv:2403.15180

#14

Self-Improvement for Neural Combinatorial Optimization: Sample Without Replacement, but Improvement

Dominik Grimm, Jonathan Pirnay

neural combinatorial optimizationbehavior cloningpolicy gradient methodsstochastic beam search+4

28

ICML 2025arXiv:2410.11289

#15

Subspace Optimization for Large Language Models with Convergence Guarantees

Yutong He, Pengrui Li, Yipeng Hu et al.

28

ICLR 2025arXiv:2412.19529

#16

Nonconvex Stochastic Optimization under Heavy-Tailed Noises: Optimal Convergence without Gradient Clipping

Zijian Liu, Zhengyuan Zhou

stochastic optimizationheavy-tailed noisenonconvex optimizationgradient clipping+4

28

ICLR 2024arXiv:2312.08531

#17

Revisiting the Last-Iterate Convergence of Stochastic Gradient Methods

Zijian Liu, Zhengyuan Zhou

27

ICLR 2025arXiv:2410.16103

#18

LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics

Thomas Robert, Mher Safaryan, Ionut-Vlad Modoranu et al.

26

ICML 2024arXiv:2402.10758

#19

Stochastic Localization via Iterative Posterior Sampling

Louis Grenioux, Maxence Noble, Marylou Gabrié et al.

stochastic localizationposterior samplingscore-based learningdenoising schedules+3

25

ICML 2024arXiv:2310.01860

#20

High-Probability Convergence for Composite and Distributed Stochastic Minimization and Variational Inequalities with Heavy-Tailed Noise

Eduard Gorbunov, Abdurakhmon Sadiev, Marina Danilova et al.

stochastic optimizationheavy-tailed noisegradient clippingcomposite optimization+4

25

ICML 2024arXiv:2306.04815

#21

Catapults in SGD: spikes in the training loss and their impact on generalization through feature learning

Libin Zhu, Chaoyue Liu, Adityanarayanan Radhakrishnan et al.

training loss spikesstochastic gradient descentoptimization phenomenonfeature learning+4

25

ICLR 2025arXiv:2410.10800

#22

Optimizing $(L_0, L_1)$-Smooth Functions by Gradient Methods

Daniil Vankov, Anton Rodomanov, Angelia Nedich et al.

gradient methodssmooth function optimizationconvex optimizationnonconvex optimization+4

24

ICML 2024arXiv:2402.01567

#23

Understanding Adam Optimizer via Online Learning of Updates: Adam is FTRL in Disguise

Kwangjun Ahn, Zhiyu Zhang, Yunbum Kook et al.

adam optimizeronline learningfollow-the-regularized-leaderoptimizer design+2

NEURIPS 2025arXiv:2507.07101

#24

Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation is Wasteful

Martin Marek, Sanae Lotfi, Aditya Somasundaram et al.

language model trainingbatch size scalingadam hyperparametersvanilla sgd+4

AAAI 2025arXiv:2304.11787

#25

B2Opt: Learning to Optimize Black-box Optimization with Little Budget

Xiaobin Li, Kai Wu, Xiaoyu Zhang et al.

black-box optimizationgenetic algorithmsurrogate functionshigh-dimensional optimization+2

ICLR 2025arXiv:2410.24206

#26

Understanding Optimization in Deep Learning with Central Flows

Jeremy Cohen, Alex Damian, Ameet Talwalkar et al.

optimization dynamicsedge of stabilitycentral flow analysistime-averaged trajectories+2

ICLR 2025arXiv:2405.16002

#27

Does SGD really happen in tiny subspaces?

Minhak Song, Kwangjun Ahn, Chulhee Yun

21

ICLR 2025arXiv:2402.02746

#28

Standard Gaussian Process is All You Need for High-Dimensional Bayesian Optimization

Zhitong Xu, Haitao Wang, Jeff Phillips et al.

21

AAAI 2024arXiv:2312.15184

#29

ZO-AdaMU Optimizer: Adapting Perturbation by the Momentum and Uncertainty in Zeroth-Order Optimization

Shuoran Jiang, Qingcai Chen, Yang Xiang et al.

zeroth-order optimizationmemory-efficient traininglarge language modelsmomentum adaptation+3

21

AAAI 2024arXiv:2405.05349

#30

Offline Model-Based Optimization via Policy-Guided Gradient Search

Yassine Chemingui, Aryan Deshwal, Nghia Hoang et al.

20

ICML 2024arXiv:2402.07318

#31

Position: Optimization in SciML Should Employ the Function Space Geometry

Johannes Müller, Marius Zeinhofer

function space geometryinfinite-dimensional optimizationscientific machine learningoptimize then discretize+1

19

ICLR 2025arXiv:2409.19605

#32

The Crucial Role of Samplers in Online Direct Preference Optimization

Ruizhe Shi, Runlong Zhou, Simon Du

19

NEURIPS 2025arXiv:2504.19983

#33

Emergence and scaling laws in SGD learning of shallow neural networks

Yunwei Ren, Eshaan Nichani, Denny Wu et al.

stochastic gradient descentshallow neural networksscaling lawsextensive-width regime+4

17

ICLR 2025arXiv:2502.08696

#34

Scalable Discrete Diffusion Samplers: Combinatorial Optimization and Statistical Physics

Sebastian Sanokowski, Wilhelm Berghammer, Haoyu Wang et al.

discrete diffusion modelscombinatorial optimizationstatistical physicspolicy gradient theorem+4

17

ICLR 2025arXiv:2411.15958

#35

Adaptive Methods through the Lens of SDEs: Theoretical Insights on the Role of Noise

Enea Monzio Compagnoni, Tianlin Liu, Rustem Islamov et al.

adaptive optimization methodsstochastic differential equationsgradient noise analysiscurvature adaptation+4

17

NEURIPS 2025arXiv:2504.19599

#36

GVPO: Group Variance Policy Optimization for Large Language Model Post-Training

Kaichen Zhang, Yuzhong Hong, Junwei Bao et al.

16

ICLR 2024arXiv:2310.20581

#37

Stochastic Gradient Descent for Gaussian Processes Done Right

Jihao Andreas Lin, Shreyas Padhy, Javier Antorán et al.

15

ICML 2024arXiv:2212.04486

#38

A New Linear Scaling Rule for Private Adaptive Hyperparameter Optimization

Ashwinee Panda, Xinyu Tang, Saeed Mahloujifar et al.

differential privacyhyperparameter optimizationprivate deep learningdp-sgd+3

15

ICLR 2025arXiv:2206.00395

#39

Optimization with Access to Auxiliary Information

EL MAHDI CHAYTI, Sai Karimireddy

gradient computation optimizationauxiliary function utilizationhessian similarity assumptionstochastic optimization+4

ICML 2024arXiv:2305.15659

#40

How to Escape Sharp Minima with Random Perturbations

Kwangjun Ahn, Ali Jadbabaie, Suvrit Sra

flat minimahessian tracesharpness-aware minimizationoptimization algorithms+3

ICLR 2025arXiv:2406.15244

#41

AdaGrad under Anisotropic Smoothness

Yuxing Liu, Rui Pan, Tong Zhang

adaptive gradient methodsanisotropic smoothnessconvergence guaranteeslarge batch training+3

AAAI 2024arXiv:2310.17319

#42

Trust Region Methods for Nonconvex Stochastic Optimization beyond Lipschitz Smoothness

Chenghan Xie, Chenxi Li, Chuwen Zhang et al.

trust region methodsnonconvex stochastic optimizationgeneralized smoothnessdistributionally robust optimization+4

ICLR 2024arXiv:2310.06689

#43

Approximating Nash Equilibria in Normal-Form Games via Stochastic Optimization

Ian Gemp, Luke Marris, Georgios Piliouras

ICML 2024arXiv:2402.07101

#44

On The Complexity of First-Order Methods in Stochastic Bilevel Optimization

Jeongyeol Kwon, Dohyun Kwon, Hanbaek Lyu

bilevel optimizationstochastic optimizationfirst-order methodscomplexity analysis+4

ICML 2024arXiv:2406.09241

#45

What is the Long-Run Distribution of Stochastic Gradient Descent? A Large Deviations Analysis

Waïss Azizian, Franck Iutzeler, Jérôme Malick et al.

stochastic gradient descentnon-convex optimizationlarge deviations theoryboltzmann-gibbs distribution+4

ICML 2024arXiv:2105.13937

#46

Polygonal Unadjusted Langevin Algorithms: Creating stable and efficient adaptive algorithms for neural networks

Dongyoung Lim, Sotirios Sabanis

langevin algorithmsadaptive optimization algorithmsstochastic differential equationsmonotone coefficients+4

ICLR 2024arXiv:2312.01127

#47

Symmetric Mean-field Langevin Dynamics for Distributional Minimax Problems

Juno Kim, Kakei Yamamoto, Kazusato Oko et al.

NEURIPS 2025arXiv:2401.07844

#48

The ODE Method for Stochastic Approximation and Reinforcement Learning with Markovian Noise

Shuze Daniel Liu, Shuhang Chen, Shangtong Zhang

stochastic approximationmarkovian noisereinforcement learningtemporal difference learning+4

ICML 2024arXiv:2402.07793

#49

Tuning-Free Stochastic Optimization

Ahmed Khaled, Chi Jin

stochastic gradient descenthyperparameter tuningconvex optimizationnonconvex optimization+4

ICML 2024arXiv:2403.02967

#50

Non-convex Stochastic Composite Optimization with Polyak Momentum

Yuan Gao, Anton Rodomanov, Sebastian Stich

stochastic composite optimizationproximal gradient methodpolyak momentumnon-convex optimization+3

ICLR 2024arXiv:2309.01570

#51

Advancing the Lower Bounds: an Accelerated, Stochastic, Second-order Method with Optimal Adaptation to Inexactness

Artem Agafonov, Dmitry Kamzolov, Alexander Gasnikov et al.

ICLR 2025arXiv:2510.19975

#52

Revisiting Zeroth-Order Optimization: Minimum-Variance Two-Point Estimators and Directionally Aligned Perturbations

Shaocong Ma, Heng Huang

zeroth-order optimizationgradient estimationperturbation distributionsasymptotic variance minimization+3

ICLR 2024arXiv:2310.02671

#53

Beyond Stationarity: Convergence Analysis of Stochastic Softmax Policy Gradient Methods

Sara Klein, Simon Weissmann, Leif Döring

AAAI 2024arXiv:2307.09421

#54

Jointly Improving the Sample and Communication Complexities in Decentralized Stochastic Minimax Optimization

Xuan Zhang, Gabriel Mancino-Ball, Necdet Serhat Aybat et al.

ICLR 2025arXiv:2410.04642

#55

The Optimization Landscape of SGD Across the Feature Learning Strength

Alexander Atanasov, Alexandru Meterez, James Simon et al.

feature learning strengthscaling regimeslearning rate scalingultra-rich regime+3

ICML 2025arXiv:2504.03461

#56

Conditioning Diffusions Using Malliavin Calculus

Jakiw Pidstrigach, Elizabeth Baker, Carles Domingo i Enrich et al.

ICML 2024arXiv:2402.02720

#57

Discounted Adaptive Online Learning: Towards Better Regularization

Zhiyu Zhang, David Bombara, Heng Yang

online convex optimizationdiscounted regretadaptive online learningfollow-the-regularized-leader+4

ICML 2024arXiv:2310.15351

#58

Random Exploration in Bayesian Optimization: Order-Optimal Regret and Computational Efficiency

Sudeep Salgia, Sattar Vakili, Qing Zhao

bayesian optimizationgaussian process modelskernel-based bandit optimizationrandom exploration+4

ICLR 2024arXiv:2405.20439

#59

Sharpness-Aware Minimization Enhances Feature Quality via Balanced Learning

Jacob Springer, Vaishnavh Nagarajan, Aditi Raghunathan

ICML 2024arXiv:2402.11173

#60

How to Make the Gradients Small Privately: Improved Rates for Differentially Private Non-Convex Optimization

Andrew Lowy, Jonathan Ullman, Stephen Wright

differentially private optimizationnon-convex optimizationstationary pointsquasar-convex functions+3

ICLR 2025arXiv:2406.04142

#61

Stochastic Polyak Step-sizes and Momentum: Convergence Guarantees and Practical Performance

Dimitris Oikonomou, Nicolas Loizou

ICML 2024arXiv:2402.10475

#62

Fundamental Benefit of Alternating Updates in Minimax Optimization

Jaewook Lee, Hanseul Cho, Chulhee Yun

minimax optimizationgradient descent ascentalternating updatesglobal convergence rates+4

ICML 2024arXiv:2402.03126

#63

How Free is Parameter-Free Stochastic Optimization?

Amit Attia, Tomer Koren

parameter-free optimizationstochastic optimizationnon-convex optimizationconvex optimization+4

ICML 2024arXiv:2306.02939

#64

Improved Stability and Generalization Guarantees of the Decentralized SGD Algorithm

Batiste Le Bars, Aurélien Bellet, Marc Tommasi et al.

decentralized optimizationstochastic gradient descentalgorithmic stabilitygeneralization error analysis+4

multi-objective optimizationpareto-optimal solutionspreference optimizationconstrained optimization+3

#65

Preference Optimization on Pareto Sets: On a Theory of Multi-Objective Optimization

Abhishek Roy, Geelon So, Yian Ma

NEURIPS 2025

ICLR 2024arXiv:2310.11428

#66

Butterfly Effects of SGD Noise: Error Amplification in Behavior Cloning and Autoregression

Adam Block, Dylan Foster, Akshay Krishnamurthy et al.

NEURIPS 2025arXiv:2401.12033

#67

Momentum-SAM: Sharpness Aware Minimization without Computational Overhead

Marlon Becker, Frederick Altrock, Benjamin Risse

ICLR 2024arXiv:2307.15196

#68

The Marginal Value of Momentum for Small Learning Rate SGD

Runzhe Wang, Sadhika Malladi, Tianhao Wang et al.

ICML 2025arXiv:2412.18164

#69

Stochastic Control for Fine-tuning Diffusion Models: Optimality, Regularity, and Convergence

Yinbin Han, Meisam Razaviyayn, Renyuan Xu

ICLR 2024arXiv:2305.19838

#70

Relaxing the Additivity Constraints in Decentralized No-Regret High-Dimensional Bayesian Optimization

Anthony Bardou, Patrick Thiran, Thomas Begin

ICLR 2025arXiv:2409.08469

#71

Improved Finite-Particle Convergence Rates for Stein Variational Gradient Descent

Sayan Banerjee, Krishna Balasubramanian, PROMIT GHOSAL

stein variational gradient descentfinite-particle convergence rateskernelized stein discrepancywasserstein-2 metrics+3

ICML 2024arXiv:2406.01484

#72

An Online Optimization Perspective on First-Order and Zero-Order Decentralized Nonsmooth Nonconvex Stochastic Optimization

Emre Sahinoglu, Shahin Shahrampour

decentralized stochastic optimizationnonsmooth nonconvex optimizationonline-to-nonconvex techniquerandomized smoothing+4

NEURIPS 2025arXiv:2508.12511

#73

Trust Region Constrained Measure Transport in Path Space for Stochastic Optimal Control and Inference

Denis Blessing, Julius Berner, Lorenz Richter et al.

stochastic optimal controlpath space measuretrust region optimizationmeasure transport+4

ICLR 2024arXiv:2308.06703

#74

Understanding the robustness difference between stochastic gradient descent and adaptive gradient methods

Avery Ma, Yangchen Pan, Amir-massoud Farahmand

ICML 2025arXiv:2412.11773

#75

Toward a Unified Theory of Gradient Descent under Generalized Smoothness

Alexander Tyurin

ICML 2025arXiv:2501.16168

#76

Ringmaster ASGD: The First Asynchronous SGD with Optimal Time Complexity

Artavazd Maranjyan, Alexander Tyurin, Peter Richtarik

AAAI 2024arXiv:2207.05631

#77

DGPO: Discovering Multiple Strategies with Diversity-Guided Policy Optimization

Wenze Chen, Shiyu Huang, Yuan Chiang et al.

reinforcement learningdiverse strategy discoverypolicy optimizationinformation-theoretic diversity+3

ICLR 2024arXiv:2403.11844

#78

Near-Optimal Solutions of Constrained Learning Problems

Juan Elenter, Luiz Chamon, Alejandro Ribeiro

NEURIPS 2025arXiv:2502.04664

#79

Implicit Bias of Spectral Descent and Muon on Multiclass Separable Data

Chen Fan, Mark Schmidt, Christos Thrampoulidis

implicit optimization biasmulticlass linear classificationp-norm normalized descentmomentum steepest descent+4

ICLR 2025arXiv:2407.02772

#80

Gradient descent with generalized Newton’s method

Zhiqi Bu, Shiyun Xu

gradient descent optimizationhessian-informed methodslearning rate selectionoptimizer acceleration+4

ICML 2024arXiv:2405.04795

#81

Variational Schrödinger Diffusion Models

Wei Deng, Weijian Luo, Yixin Tan et al.

schrödinger bridgediffusion modelsvariational inferencescore matching+3

ICML 2024arXiv:2412.20017

#82

A Nearly Optimal Single Loop Algorithm for Stochastic Bilevel Optimization under Unbounded Smoothness

Xiaochuan Gong, Jie Hao, Mingrui Liu

stochastic bilevel optimizationunbounded smoothnessmeta-learningsingle loop algorithm+4

ICML 2024arXiv:2304.11737

#83

Sarah Frank-Wolfe: Methods for Constrained Optimization with Best Rates and Practical Features

Aleksandr Beznosikov, David Dobre, Gauthier Gidel

frank-wolfe methodconstrained optimizationstochastic optimizationfinite-sum minimization+3

ICML 2024arXiv:2405.08424

#84

Tackling Prevalent Conditions in Unsupervised Combinatorial Optimization: Cardinality, Minimum, Covering, and More

Fanchen Bu, Hyeonsoo Jo, Soo Yong Lee et al.

combinatorial optimizationprobabilistic methodunsupervised learningderandomization techniques+4

ICML 2024arXiv:2403.07723

#85

On the Last-Iterate Convergence of Shuffling Gradient Methods

Zijian Liu, Zhengyuan Zhou

shuffling gradient methodslast-iterate convergencerandom reshuffleincremental gradient+4

ICML 2024arXiv:2403.04154

#86

Stabilizing Policy Gradients for Stochastic Differential Equations via Consistency with Perturbation Process

Xiangxin Zhou, Liang Wang, Yichi Zhou

stochastic differential equationspolicy gradient methodsgenerative modelsstructure-based drug design+2

ICML 2024arXiv:2402.04843

#87

Spectral Preconditioning for Gradient Methods on Graded Non-convex Functions

Nikita Doikov, Sebastian Stich, Martin Jaggi

spectral preconditioninggradient methodsgraded non-convexityhessian spectrum+4

ICML 2025arXiv:2502.09198

#88

Understanding High-Dimensional Bayesian Optimization

Leonard Papenmeier, Matthias Poloczek, Luigi Nardi

ICML 2024arXiv:2405.09742

#89

Random Scaling and Momentum for Non-smooth Non-convex Optimization

Qinzi Zhang, Ashok Cutkosky

non-convex optimizationstochastic gradient descentmomentum algorithmsnon-smooth optimization+3

ICML 2024arXiv:2402.03210

#90

Universal Gradient Methods for Stochastic Convex Optimization

Anton Rodomanov, Ali Kavis, Yongtao Wu et al.

stochastic convex optimizationgradient methodshölder smoothnessstep-size adaptation+2

ICML 2024arXiv:2306.08590

#91

Beyond Implicit Bias: The Insignificance of SGD Noise in Online Learning

Nikhil Vyas, Depen Morwani, Rosie Zhao et al.

implicit biassgd noiseonline learningoffline learning+3

NEURIPS 2025arXiv:2505.24749

#92

SUMO: Subspace-Aware Moment-Orthogonalization for Accelerating Memory-Efficient LLM Training

Yehonathan Refael, Guy Smorodinsky, Tom Tirer et al.

low-rank optimizationmemory-efficient traininglarge language modelsgradient-based optimization+4

ICML 2024arXiv:2406.01870

#93

Understanding Stochastic Natural Gradient Variational Inference

Kaiwen Wu, Jacob Gardner

variational inferencenatural gradient optimizationstochastic optimizationconjugate likelihoods+4

ICML 2024arXiv:2310.13033

#94

LASER: Linear Compression in Wireless Distributed Optimization

Ashok Vardhan Makkuva, Marco Bondaschi, Thijs Vogels et al.

distributed optimizationgradient compressionwireless communicationlow-rank gradients+3

ICML 2025arXiv:2502.11229

#95

Provable and Practical Online Learning Rate Adaptation with Hypergradient Descent

Ya-Chi Chu, Wenzhi Gao, Yinyu Ye et al.

ICML 2025arXiv:2502.04164

#96

Efficient Distributed Optimization under Heavy-Tailed Noise

Su Hyeong Lee, Manzil Zaheer, Tian Li

ICML 2025arXiv:2406.04443

#97

Clipping Improves Adam-Norm and AdaGrad-Norm when the Noise Is Heavy-Tailed

Savelii Chezhegov, Klyukin Yaroslav, Andrei Semenov et al.

NEURIPS 2025arXiv:2506.01913

#98

Generalized Gradient Norm Clipping & Non-Euclidean $(L_0,L_1)$-Smoothness

Thomas Pethick, Wanyun Xie, Mete Erdogan et al.

gradient norm clippingnon-euclidean optimizationsteepest descentconditional gradient+4

ICML 2024arXiv:2402.07070

#99

Efficient Algorithms for Sum-Of-Minimum Optimization

Lisang Ding, Ziang Chen, Xinshang Wang et al.

sum-of-minimum optimizationclustering applicationsgeneralized principal component analysismixed linear regression+4

ICML 2025arXiv:2502.13283

#100

Benefits of Early Stopping in Gradient Descent for Overparameterized Logistic Regression

Jingfeng Wu, Peter Bartlett, Matus Telgarsky et al.