Policy Optimization

ICLR 2024arXiv:2309.03409

#2

Large Language Models as Optimizers

Chengrun Yang, Xuezhi Wang, Yifeng Lu et al.

683

ICLR 2024arXiv:2309.06657

#3

Statistical Rejection Sampling Improves Preference Optimization

Tianqi Liu, Yao Zhao, Rishabh Joshi et al.

329

ICCV 2025arXiv:2503.12937

#4

R1-VL: Learning to Reason with Multimodal Large Language Models via Step-wise Group Relative Policy Optimization

Jingyi Zhang, Jiaxing Huang, Huanjin Yao et al.

220

ICLR 2024arXiv:2310.02743

#5

Reward Model Ensembles Help Mitigate Overoptimization

Thomas Coste, Usman Anwar, Robert Kirk et al.

188

ICML 2024arXiv:2404.14367

#6

Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data

Fahim Tajwar, Anikait Singh, Archit Sharma et al.

preference fine-tuninglarge language modelson-policy reinforcement learningcontrastive learning+4

179

ICML 2024arXiv:2402.05749

#7

Generalized Preference Optimization: A Unified Approach to Offline Alignment

Yunhao Tang, Zhaohan Guo, Zeyu Zheng et al.

preference optimizationoffline alignmentkl divergence regularizationconvex loss functions+4

150

ICLR 2025arXiv:2409.00588

#8

Diffusion Policy Policy Optimization

Allen Ren, Justin Lidard, Lars Ankile et al.

diffusion-based policiespolicy gradient methodscontinuous controlrobot learning+4

146

ICML 2024arXiv:2401.04056

#9

A Minimaximalist Approach to Reinforcement Learning from Human Feedback

Gokul Swamy, Christoph Dann, Rahul Kidambi et al.

reinforcement learning from human feedbackpreference optimizationself-play algorithmsminimax winner+4

139

ICML 2024arXiv:2404.11999

#10

Token-level Direct Preference Optimization

Yongcheng Zeng, Guoqing Liu, Weiyu Ma et al.

direct preference optimizationtoken-level optimizationkl divergence constraintsbradley-terry model+4

120

ICLR 2024arXiv:2308.02151

#11

Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization

Weiran Yao, Shelby Heinecke, Juan Carlos Niebles et al.

108

ICML 2025arXiv:2404.18922

#12

DPO Meets PPO: Reinforced Token Optimization for RLHF

Han Zhong, Zikang Shan, Guhao Feng et al.

106

ICML 2024arXiv:2403.00409

#13

Provably Robust DPO: Aligning Language Models with Noisy Feedback

Sayak Ray Chowdhury, Anush Kini, Nagarajan Natarajan

preference-based feedbackdirect preference optimizationnoisy preference pairspolicy optimization+4

103

ICML 2024arXiv:2403.08635

#14

Human Alignment of Large Language Models through Online Preference Optimisation

Daniele Calandriello, Zhaohan Guo, REMI MUNOS et al.

human alignmentpreference optimizationreinforcement learning from human feedbackdirect policy optimization+3

88

NEURIPS 2025arXiv:2410.07163

#15

Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning

Chongyu Fan, Jiancheng Liu, Licong Lin et al.

llm unlearningnegative preference optimizationreference model biasgradient ascent methods+4

81

ICLR 2025arXiv:2405.19320

#16

Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF

Shicong Cen, Jincheng Mei, Katayoon Goshvadi et al.

62

ICLR 2025arXiv:2410.13213

#17

LLMOPT: Learning to Define and Solve General Optimization Problems from Scratch

caigao jiang, Xiang Shu, Hong Qian et al.

57

ICLR 2024arXiv:2309.01753

#18

On Penalty Methods for Nonconvex Bilevel Optimization and First-Order Stochastic Approximation

Jeongyeol Kwon, Dohyun Kwon, Stephen Wright et al.

56

NEURIPS 2025arXiv:2503.22342

#19

CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Models

Zhihang Lin, Mingbao Lin, Yuan Xie et al.

policy optimizationtraining accelerationreasoning modelscompletion pruning+2

56

ICLR 2025arXiv:2407.13399

#20

Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization

Audrey Huang, Wenhao Zhan, Tengyang Xie et al.

54

ICLR 2024arXiv:2310.07297

#21

Score Regularized Policy Optimization through Diffusion Behavior

Huayu Chen, Cheng Lu, Zhengyi Wang et al.

51

ICML 2024arXiv:2407.05872

#22

Scaling Exponents Across Parameterizations and Optimizers

Katie Everett, Lechao Xiao, Mitchell Wortsman et al.

scaling exponentsparameterization methodsoptimizer designlearning rate scaling+4

51

ICLR 2024arXiv:2303.03751

#23

Zeroth-Order Optimization Meets Human Feedback: Provable Learning via Ranking Oracles

Zhiwei Tang, Dmitry Rybin, Tsung-Hui Chang

42

ICML 2024arXiv:2402.19078

#24

Smooth Tchebycheff Scalarization for Multi-Objective Optimization

Xi Lin, Xiaoyuan Zhang, Zhiyuan Yang et al.

multi-objective optimizationtchebycheff scalarizationgradient-based optimizationpareto solutions+2

38

NEURIPS 2025arXiv:2502.00604

#25

Gradient Alignment in Physics-informed Neural Networks: A Second-Order Optimization Perspective

Sifan Wang, Ananyae bhartari, Bowen Li et al.

gradient alignmentphysics-informed neural networkssecond-order optimizationmulti-task learning+4

38

ICLR 2024arXiv:2308.02585

#26

PARL: A Unified Framework for Policy Alignment in Reinforcement Learning from Human Feedback

Souradip Chakraborty, Amrit Bedi, Alec Koppel et al.

38

ICLR 2025arXiv:2410.11820

#27

Adaptive Data Optimization: Dynamic Sample Selection with Scaling Laws

Yiding Jiang, Allan Zhou, Zhili Feng et al.

pretraining data compositiondata distribution optimizationscaling lawsfoundation models+3

36

ICLR 2025arXiv:2407.00617

#28

Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learning

Yuheng Zhang, Dian Yu, Baolin Peng et al.

reinforcement learning human feedbackpreference modelinggame-theoretic alignmentnash policy optimization+4

34

ICML 2024arXiv:2402.00856

#29

CPPO: Continual Learning for Reinforcement Learning with Human Feedback

Han Zhang, Yu Lei, Lin Gui et al.

Towards Efficient Exact Optimization of Language Model Alignment

Haozhe Ji, Cheng Lu, Yilin Niu et al.

language model alignmenthuman preference optimizationreinforcement learning variancedirect preference optimization+3

32

ICML 2025arXiv:2502.00361

#31

Efficient Online Reinforcement Learning for Diffusion Policy

Haitong Ma, Tianyi Chen, Kai Wang et al.

31

ICLR 2025arXiv:2409.14989

#32

Methods for Convex $(L_0,L_1)$-Smooth Optimization: Clipping, Acceleration, and Adaptivity

Eduard Gorbunov, Nazarii Tupitsa, Sayantan Choudhury et al.

31

NEURIPS 2025arXiv:2212.09900

#33

Policy learning “without” overlap: Pessimism and generalized empirical Bernstein’s inequality

Ying Jin, Zhimei Ren, Zhuoran Yang et al.

offline policy learninguniform overlap assumptionpessimistic policy learninglower confidence bounds+4

31

AAAI 2025arXiv:2402.17564

#34

Unleashing the Potential of Large Language Models as Prompt Optimizers: Analogical Analysis with Gradient-based Model Optimizers

Xinyu Tang, Xiaolei Wang, Wayne Xin Zhao et al.

30

ICML 2024arXiv:2402.06886

#35

Principled Penalty-based Methods for Bilevel Reinforcement Learning and RLHF

Han Shen, Zhuoran Yang, Tianyi Chen

bilevel optimizationreinforcement learninginverse reinforcement learningrl from human feedback+4

30

NEURIPS 2025arXiv:2505.15201

#36

Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problems

Christian Walder, Deep Tejas Karkhanis

policy optimizationreward transformationpass@k optimizationunbiased gradient estimation+3

28

ICLR 2025arXiv:2403.15180

#37

Self-Improvement for Neural Combinatorial Optimization: Sample Without Replacement, but Improvement

Dominik Grimm, Jonathan Pirnay

neural combinatorial optimizationbehavior cloningpolicy gradient methodsstochastic beam search+4

28

ICLR 2025arXiv:2410.10800

#38

Optimizing $(L_0, L_1)$-Smooth Functions by Gradient Methods

Daniil Vankov, Anton Rodomanov, Angelia Nedich et al.

gradient methodssmooth function optimizationconvex optimizationnonconvex optimization+4

ICML 2024arXiv:2402.08552

#39

Confronting Reward Overoptimization for Diffusion Models: A Perspective of Inductive and Primacy Biases

Ziyi Zhang, Sen Zhang, Yibing Zhan et al.

reward overoptimizationdiffusion model alignmentpolicy gradient algorithmtemporal inductive bias+4

NEURIPS 2025arXiv:2502.01384

#40

Fine-Tuning Discrete Diffusion Models with Policy Gradient Methods

Oussama Zekri, Nicolas Boulle

ICML 2024arXiv:2407.01606

#41

On Discrete Prompt Optimization for Diffusion Models

Ruochen Wang, Ting Liu, Cho-Jui Hsieh et al.

prompt optimizationdiffusion modelstext-to-image generationdiscrete optimization+3

ICLR 2025arXiv:2406.01660

#42

Self-Improving Robust Preference Optimization

Eugene Choi, Arash Ahmadian, Matthieu Geist et al.

ICLR 2025arXiv:2406.07327

#43

3D-Properties: Identifying Challenges in DPO and Charting a Path Forward

Yuzi Yan, Yibo Miao, Jialian Li et al.

AAAI 2025arXiv:2304.11787

#44

B2Opt: Learning to Optimize Black-box Optimization with Little Budget

Xiaobin Li, Kai Wu, Xiaoyu Zhang et al.

black-box optimizationgenetic algorithmsurrogate functionshigh-dimensional optimization+2

ICLR 2025arXiv:2410.24206

#45

Understanding Optimization in Deep Learning with Central Flows

Jeremy Cohen, Alex Damian, Ameet Talwalkar et al.

optimization dynamicsedge of stabilitycentral flow analysistime-averaged trajectories+2

ICML 2024arXiv:2403.01857

#46

Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferences

Andi Nika, Debmalya Mandal, Parameswaran Kamalaruban et al.

reinforcement learning from human feedbackdirect preference optimizationloglinear policy parametrizationlinear reward functions+4

20

AAAI 2024arXiv:2405.05349

#47

Offline Model-Based Optimization via Policy-Guided Gradient Search

Yassine Chemingui, Aryan Deshwal, Nghia Hoang et al.

20

AAAI 2024arXiv:2305.16830

#48

Leaving the Nest: Going beyond Local Loss Functions for Predict-Then-Optimize

Sanket Shah, Bryan Wilder, Andrew Perrault et al.

predict-then-optimizedecision-making under uncertaintytask-specific loss functionssample efficiency+1

20

NEURIPS 2025arXiv:2505.13438

#49

Optimizing Anytime Reasoning via Budget Relative Policy Optimization

Penghui Qi, Zichen Liu, Tianyu Pang et al.

19

ICLR 2025arXiv:2410.04203

#50

RainbowPO: A Unified Framework for Combining Improvements in Preference Optimization

Hanyang Zhao, Genta Winata, Anirban Das et al.

preference optimizationdirect preference optimizationhuman preference alignmentmodel alignment+2

19

NEURIPS 2025arXiv:2505.23564

#51

Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Models

Yiran Guo, Lijie Xu, Jie Liu et al.

reinforcement learningcredit assignmentadvantage estimationpolicy optimization+4

18

ICML 2024arXiv:2305.18728

#52

Plug-in Performative Optimization

Licong Lin, Tijana Zrnic

performative predictionperformative risk optimizationstrategic classificationbest-response models+4

18

ICML 2024arXiv:2106.08414

#53

On the Sample Complexity and Metastability of Heavy-tailed Policy Search in Continuous Control

Amrit Singh Bedi, Anjaly Parayil, Junyu Zhang et al.

policy searchcontinuous controlstochastic policy gradientheavy-tailed distributions+4

18

ICML 2024arXiv:2402.10342

#54

Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilization

Yihan Du, Anna Winnicki, Gal Dalal et al.

reinforcement learning from human feedbackpolicy optimizationtrajectory-based comparison feedbackelliptical potential analysis+3

17

ICML 2024arXiv:2402.10119

#55

Physics-Informed Neural Network Policy Iteration: Algorithms, Convergence, and Verification

Yiming Meng, Ruikun Zhou, Amartya Mukherjee et al.

physics-informed neural networksoptimal control problemspolicy iteration algorithmspartial differential equations+4

17

NEURIPS 2025arXiv:2504.19599

#56

GVPO: Group Variance Policy Optimization for Large Language Model Post-Training

Kaichen Zhang, Yuzhong Hong, Junwei Bao et al.

16

ICML 2024arXiv:2212.04486

#57

A New Linear Scaling Rule for Private Adaptive Hyperparameter Optimization

Ashwinee Panda, Xinyu Tang, Saeed Mahloujifar et al.

differential privacyhyperparameter optimizationprivate deep learningdp-sgd+3

15

ICML 2024arXiv:2402.05367

#58

Principled Preferential Bayesian Optimization

Wenjie Xu, Wenbin Wang, Yuning Jiang et al.

preferential bayesian optimizationblack-box function optimizationconfidence set constructionlikelihood ratio methods+3

ICLR 2025arXiv:2206.00395

#59

Optimization with Access to Auxiliary Information

EL MAHDI CHAYTI, Sai Karimireddy

gradient computation optimizationauxiliary function utilizationhessian similarity assumptionstochastic optimization+4

ICML 2024arXiv:2405.03329

#60

Policy Learning for Balancing Short-Term and Long-Term Rewards

Peng Wu, Ziyu Shen, Feng Xie et al.

policy learningshort-term rewardslong-term rewardssemiparametric efficiency bounds+4

ICML 2025arXiv:2410.10148

#61

AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization

Junkang Wu, xue wang, Zhengyi Yang et al.

ICLR 2024arXiv:2311.18207

#62

Towards Assessing and Benchmarking Risk-Return Tradeoff of Off-Policy Evaluation

Haruka Kiyohara, Ren Kishimoto, Kosuke Kawakami et al.

AAAI 2024arXiv:2311.13460

#63

Multi-Objective Bayesian Optimization with Active Preference Learning

Ryota Ozaki, Kazuki Ishikawa, Youhei Kanzaki et al.

bayesian optimizationmulti-objective optimizationpreference learningpareto front identification+4

ICML 2024arXiv:2407.20230

#64

SAPG: Split and Aggregate Policy Gradients

Jayesh Singla, Ananye Agarwal, Deepak Pathak

policy gradient methodssample efficiencyparallelized environmentsimportance sampling+2

COLM 2025arXiv:2503.08942

#65

Extragradient Preference Optimization (EGPO): Beyond Last-Iterate Convergence for Nash Learning from Human Feedback

Runlong Zhou, Maryam Fazel, Simon Shaolei Du

reinforcement learningreinforcement learning from human feedbacklarge language models

ICML 2024arXiv:2105.13937

#66

Polygonal Unadjusted Langevin Algorithms: Creating stable and efficient adaptive algorithms for neural networks

Dongyoung Lim, Sotirios Sabanis

langevin algorithmsadaptive optimization algorithmsstochastic differential equationsmonotone coefficients+4

ICML 2024arXiv:2402.07793

#67

Tuning-Free Stochastic Optimization

Ahmed Khaled, Chi Jin

stochastic gradient descenthyperparameter tuningconvex optimizationnonconvex optimization+4

ICML 2025arXiv:2409.06957

#68

Policy Filtration for RLHF to Mitigate Noise in Reward Models

Chuheng Zhang, Wei Shen, Li Zhao et al.

ICLR 2024arXiv:2310.02671

#69

Beyond Stationarity: Convergence Analysis of Stochastic Softmax Policy Gradient Methods

Sara Klein, Simon Weissmann, Leif Döring

NEURIPS 2025arXiv:2505.20686

#70

Accelerating RL for LLM Reasoning with Optimal Advantage Regression

Kianté Brantley, Mingyu Chen, Zhaolin Gao et al.

reinforcement learninglarge language modelsreasoning taskspolicy optimization+4

NEURIPS 2025arXiv:2505.18830

#71

On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimization

wenlong deng, Yi Ren, Muchen Li et al.

ICLR 2024arXiv:2401.09587

#72

Bilevel Optimization under Unbounded Smoothness: A New Algorithm and Convergence Analysis

Jie Hao, Xiaochuan Gong, Mingrui Liu

multi-objective optimizationpareto-optimal solutionspreference optimizationconstrained optimization+3

#73

Preference Optimization on Pareto Sets: On a Theory of Multi-Objective Optimization

Abhishek Roy, Geelon So, Yian Ma

NEURIPS 2025

11

ICML 2024arXiv:2402.11173

#74

How to Make the Gradients Small Privately: Improved Rates for Differentially Private Non-Convex Optimization

Andrew Lowy, Jonathan Ullman, Stephen Wright

differentially private optimizationnon-convex optimizationstationary pointsquasar-convex functions+3

11

ICML 2025arXiv:2412.03092

#75

Revolve: Optimizing AI Systems by Tracking Response Evolution in Textual Optimization

Peiyan Zhang, Haibo Jin, Leyang Hu et al.

11

NEURIPS 2025arXiv:2507.05197

#76

Pre-Trained Policy Discriminators are General Reward Models

Shihan Dou, Shichun Liu, Yuming Yang et al.

ICML 2025arXiv:2412.18164

#77

Stochastic Control for Fine-tuning Diffusion Models: Optimality, Regularity, and Convergence

Yinbin Han, Meisam Razaviyayn, Renyuan Xu

ICML 2025arXiv:2401.16025

#78

Simple Policy Optimization

Zhengpeng Xie, Qiang Zhang, Fan Yang et al.

ICLR 2025arXiv:2412.09544

#79

Sail into the Headwind: Alignment via Robust Rewards and Dynamic Labels against Reward Hacking

Paria Rashidinejad, Yuandong Tian

NEURIPS 2025arXiv:2502.01930

#80

Robust LLM Alignment via Distributionally Robust Direct Preference Optimization

Zaiyan Xu, Sushil Vemuri, Kishan Panaganti et al.

NEURIPS 2025arXiv:2404.13831

#81

Data-Driven Performance Guarantees for Classical and Learned Optimizers

Rajiv Sambharya, Bartolomeo Stellato

optimization algorithmsgeneralization guaranteesstatistical learning theoryparametric optimization problems+4

ICLR 2025arXiv:2409.17401

#82

Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference

Qining Zhang, Lei Ying

reinforcement learning from human feedbackzeroth-order policy gradientdirect preference optimizationreward model inference+4

NEURIPS 2025arXiv:2505.23749

#83

Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?

Paul Gölz, Nika Haghtalab, Kunhe Yang

NEURIPS 2025arXiv:2503.17299

#84

Preference-Guided Diffusion for Multi-Objective Offline Optimization

Yashas Annadani, Syrine Belakaria, Stefano Ermon et al.

AAAI 2024arXiv:2207.05631

#85

DGPO: Discovering Multiple Strategies with Diversity-Guided Policy Optimization

Wenze Chen, Shiyu Huang, Yuan Chiang et al.

reinforcement learningdiverse strategy discoverypolicy optimizationinformation-theoretic diversity+3

ICLR 2025arXiv:2407.00898

#86

Residual-MPPI: Online Policy Customization for Continuous Control

Pengcheng Wang, Chenran Li, Catherine Weaver et al.

ICML 2024arXiv:2308.14642

#87

Rate-Optimal Policy Optimization for Linear Markov Decision Processes

Uri Sherman, Alon Cohen, Tomer Koren et al.

regret minimizationlinear markov decision processespolicy optimizationonline episodic learning+3

ICLR 2024arXiv:2312.12226

#88

On the Parameterization of Second-Order Optimization Effective towards the Infinite Width

Satoki Ishikawa, Ryo Karakida

ICLR 2024arXiv:2311.08384

#89

Offline Data Enhanced On-Policy Policy Gradient with Provable Guarantees

Yifei Zhou, Ayush Sekhari, Yuda Song et al.

ICML 2024arXiv:2403.04154

#90

Stabilizing Policy Gradients for Stochastic Differential Equations via Consistency with Perturbation Process

Xiangxin Zhou, Liang Wang, Yichi Zhou

stochastic differential equationspolicy gradient methodsgenerative modelsstructure-based drug design+2

ICLR 2025arXiv:2408.16286

#91

Near-Optimal Policy Identification in Robust Constrained Markov Decision Processes via Epigraph Form

Toshinori Kitamura, Tadashi Kozuno, Wataru Kumagai et al.

NEURIPS 2025arXiv:2509.17940

#92

DriveDPO: Policy Learning via Safety DPO For End-to-End Autonomous Driving

Shuyao Shang, Yuntao Chen, Yuqi Wang et al.

end-to-end autonomous drivingpolicy learningsafety preference optimizationtrajectory prediction+3

ICML 2024arXiv:2402.04843

#93

Spectral Preconditioning for Gradient Methods on Graded Non-convex Functions

Nikita Doikov, Sebastian Stich, Martin Jaggi

spectral preconditioninggradient methodsgraded non-convexityhessian spectrum+4

AAAI 2025arXiv:2409.05622

#94

Forward KL Regularized Preference Optimization for Aligning Diffusion Policies

Zhao Shan, Chenyou Fan, Shuang Qiu et al.

ICML 2024arXiv:2412.20017

#95

A Nearly Optimal Single Loop Algorithm for Stochastic Bilevel Optimization under Unbounded Smoothness

Xiaochuan Gong, Jie Hao, Mingrui Liu

stochastic bilevel optimizationunbounded smoothnessmeta-learningsingle loop algorithm+4

ICML 2025arXiv:2512.23133

#96

Principled Algorithms for Optimizing Generalized Metrics in Binary Classification

Anqi Mao, Mehryar Mohri, Yutao Zhong

ICML 2024arXiv:2310.05792

#97

Two-timescale Derivative Free Optimization for Performative Prediction with Markovian Data

Haitong LIU, Qiang Li, Hoi To Wai

performative predictionderivative free optimizationmarkovian datacontrolled markov chain+4

AAAI 2025arXiv:2411.16506

#98

Online Guidance Graph Optimization for Lifelong Multi-Agent Path Finding

Hongzhi Zang, Yulun Zhang, He Jiang et al.

ICLR 2024arXiv:2401.14758

#99

Off-Policy Primal-Dual Safe Reinforcement Learning

Zifan Wu, Bo Tang, Qian Lin et al.

AAAI 2024arXiv:2309.03581

#100

Interactive Hyperparameter Optimization in Multi-Objective Problems via Preference Learning

Joseph Giovanelli, Alexander Tornede, Tanja Tornede et al.

hyperparameter optimizationmulti-objective optimizationpreference learningpareto front quality+3