Multimodal Learning

ICML 2024arXiv:2308.02490

#2

MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities

Weihao Yu, Zhengyuan Yang, Linjie Li et al.

large multimodal modelsmultimodal evaluation benchmarkvision-language capabilitiesllm-based evaluation+2

1,066

ICLR 2024arXiv:2306.14824

#3

Grounding Multimodal Large Language Models to the World

Zhiliang Peng, Wenhui Wang, Li Dong et al.

1,059

ICML 2024arXiv:2309.05519

#4

NExT-GPT: Any-to-Any Multimodal LLM

Shengqiong Wu, Hao Fei, Leigang Qu et al.

multimodal large language modelsany-to-any generationmodality-switching instruction tuningdiffusion decoders+3

726

CVPR 2024arXiv:2311.04257

#5

mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration

Qinghao Ye, Haiyang Xu, Jiabo Ye et al.

multi-modal large language modelsmodality collaborationmodularized network designuniversal interface decoder+3

614

CVPR 2024arXiv:2401.06209

#6

Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs

Shengbang Tong, Zhuang Liu, Yuexiang Zhai et al.

multimodal llmsvisual representation learningclip-blind pairsvision-language models+4

593

CVPR 2024arXiv:2312.13286

#7

Generative Multimodal Models are In-Context Learners

Quan Sun, Yufeng Cui, Xiaosong Zhang et al.

in-context learningmultimodal understandingvisual promptingobject-grounded generation+4

438

CVPR 2024arXiv:2311.03356

#8

GLaMM: Pixel Grounding Large Multimodal Model

Hanoona Rasheed, Muhammad Maaz, Sahal Shaji Mullappilly et al.

411

ICLR 2024arXiv:2309.11499

#9

DreamLLM: Synergistic Multimodal Comprehension and Creation

Runpei Dong, chunrui han, Yuang Peng et al.

287

CVPR 2024arXiv:2312.17172

#10

Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision Language Audio and Action

Jiasen Lu, Christopher Clark, Sangho Lee et al.

autoregressive multimodal modelsmultimodal tokenizationencoder-decoder transformermultimodal mixture of denoisers+4

280

ICCV 2025arXiv:2503.10615

#11

R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalization

yi yang, Xiaoxuan He, Hongkun Pan et al.

265

ECCV 2024arXiv:2403.09611

#12

MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training

Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier et al.

250

CVPR 2024arXiv:2312.03700

#13

OneLLM: One Framework to Align All Modalities with Language

Jiaming Han, Kaixiong Gong, Yiyuan Zhang et al.

201

ECCV 2024arXiv:2311.05437

#14

LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents

Shilong Liu, Hao Cheng, Haotian Liu et al.

200

ICLR 2024arXiv:2309.07915

#15

MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning

Haozhe Zhao, Zefan Cai, Shuzheng Si et al.

191

CVPR 2024arXiv:2404.05726

#16

MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding

Bo He, Hengduo Li, Young Kyun Jang et al.

188

CVPR 2024arXiv:2305.11443

#17

Equivariant Multi-Modality Image Fusion

Zixiang Zhao, Haowen Bai, Jiangshe Zhang et al.

164

CVPR 2024arXiv:2311.06783

#18

Q-Instruct: Improving Low-level Visual Abilities for Multi-modality Foundation Models

Haoning Wu, Zicheng Zhang, Erli Zhang et al.

158

ICLR 2024arXiv:2307.05222

#19

Emu: Generative Pretraining in Multimodality

Quan Sun, Qiying Yu, Yufeng Cui et al.

158

CVPR 2024arXiv:2312.00784

#20

ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts

Mu Cai, Haotian Liu, Siva Mustikovela et al.

155

CVPR 2024arXiv:2312.06968

#21

Hallucination Augmented Contrastive Learning for Multimodal Large Language Model

Chaoya Jiang, Haiyang Xu, Mengfan Dong et al.

121

AAAI 2024arXiv:2312.16400

#22

LGMRec: Local and Global Graph Learning for Multimodal Recommendation

Zhiqiang Guo, Jianjun Li, Guohui Li et al.

multimodal recommendationgraph learninglocal topological relationsglobal hypergraph embedding+4

120

ICLR 2025arXiv:2410.05160

#23

VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks

Ziyan Jiang, Rui Meng, Xinyi Yang et al.

117

ICLR 2025arXiv:2408.15998

#24

Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders

Min Shi, Fuxiao Liu, Shihao Wang et al.

multimodal large language modelsvision encodersoptical character recognitiondocument analysis+4

116

CVPR 2024arXiv:2401.18084

#25

Binding Touch to Everything: Learning Unified Multimodal Tactile Representations

Fengyu Yang, Chao Feng, Ziyang Chen et al.

tactile representation learningmultimodal alignmentvision-based touch sensorszero-shot learning+4

112

NEURIPS 2025arXiv:2506.15564

#26

Show-o2: Improved Native Unified Multimodal Models

Jinheng Xie, Zhenheng Yang, Mike Zheng Shou

autoregressive modelingflow matchingmultimodal understandingvideo generation+4

106

CVPR 2024arXiv:2311.15851

#27

Single-Model and Any-Modality for Video Object Tracking

Zongwei Wu, Jilai Zheng, Xiangxuan Ren et al.

100

ICML 2025arXiv:2501.05444

#28

Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark

Yunzhuo Hao, Jiawei Gu, Huichen Wang et al.

100

CVPR 2025arXiv:2411.14432

#29

Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models

Yuhao Dong, Zuyan Liu, Hai-Long Sun et al.

96

CVPR 2024arXiv:2403.16002

#30

SDSTrack: Self-Distillation Symmetric Adapter Learning for Multi-Modal Visual Object Tracking

Xiaojun Hou, Jiazheng Xing, Yijie Qian et al.

95

ICLR 2024arXiv:2308.04152

#31

Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructions

Juncheng Li, Kaihang Pan, Zhiqi Ge et al.

90

CVPR 2024arXiv:2311.17049

#32

MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training

Pavan Kumar Anasosalu Vasu, Hadi Pouransari, Fartash Faghri et al.

contrastive pretrainingimage-text modelszero-shot classificationknowledge transfer+4

90

AAAI 2024arXiv:2312.08636

#33

MmAP: Multi-Modal Alignment Prompt for Cross-Domain Multi-Task Learning

Yi Xin, Junlong Du, Qiang Wang et al.

88

ICLR 2025arXiv:2411.02571

#34

MM-EMBED: UNIVERSAL MULTIMODAL RETRIEVAL WITH MULTIMODAL LLMS

Sheng-Chieh Lin, Chankyu Lee, Mohammad Shoeybi et al.

multimodal retrievalbi-encoder retrievermodality biashard negative mining+4

86

CVPR 2024arXiv:2311.11860

#35

LION: Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge

Gongwei Chen, Leyang Shen, Rui Shao et al.

84

CVPR 2024arXiv:2311.10707

#36

Multimodal Representation Learning by Alternating Unimodal Adaptation

Xiaohui Zhang, Jaehong Yoon, Mohit Bansal et al.

80

ECCV 2024arXiv:2403.08730

#37

Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimization

Renjie Pi, Tianyang Han, Wei Xiong et al.

78

ICLR 2024arXiv:2308.12038

#38

Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages

Jinyi Hu, Yuan Yao, Chongyi Wang et al.

77

CVPR 2025arXiv:2411.14402

#39

Multimodal Autoregressive Pre-training of Large Vision Encoders

Enrico Fini, Mustafa Shukor, Xiujun Li et al.

autoregressive pre-trainingvision encodersmultimodal learningimage patch generation+3

77

CVPR 2024arXiv:2311.16714

#40

Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorld

Yijun Yang, Tianyi Zhou, kanxue Li et al.

76

CVPR 2025arXiv:2410.08202

#41

Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training

Luo, Xue Yang, Wenhan Dou et al.

monolithic multimodal modelsendogenous visual pre-trainingmultimodal mixture-of-expertsvisual parameter space+3

68

ICLR 2025arXiv:2409.20566

#42

MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning

Haotian Zhang, Mingfei Gao, Zhe Gan et al.

67

ICML 2024arXiv:2405.17730

#43

MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistance

Yake Wei, Di Hu

multimodal learningunimodal learning objectivesgradient conflictpareto optimization+3

64

ICLR 2025arXiv:2405.17430

#44

Matryoshka Multimodal Models

Mu Cai, Jianwei Yang, Jianfeng Gao et al.

multimodal modelsvisual token granularitytoken efficiency optimizationhigh-resolution image processing+3

63

AAAI 2025arXiv:2408.08640

#45

Math-PUMA: Progressive Upward Multimodal Alignment to Enhance Mathematical Reasoning

Wenwen Zhuang, Xin Huang, Xiantao Zhang et al.

multimodal large language modelsmathematical reasoningmultimodal alignmentkullback-leibler divergence+2

60

AAAI 2024arXiv:2309.08912

#46

Delving into Multimodal Prompting for Fine-Grained Visual Classification

Xin Jiang, Hao Tang, Junyao Gao et al.

fine-grained visual classificationvision-language modelsmultimodal promptingcontrastive language-image pre-training+4

59

AAAI 2024arXiv:2403.06197

#47

DrFuse: Learning Disentangled Representation for Clinical Multi-Modal Fusion with Missing Modality and Modal Inconsistency

Wenfang Yao, Kejing Yin, William Cheung et al.

multi-modal fusionmissing modalitydisentangled representationelectronic health records+4

56

ECCV 2024arXiv:2404.08351

#48

OmniSat: Self-Supervised Modality Fusion for Earth Observation

Guillaume Astruc, Nicolas Gonthier, Clement Mallet et al.

56

ICLR 2024arXiv:2401.01646

#49

Prototypical Information Bottlenecking and Disentangling for Multimodal Cancer Survival Prediction

Yilan Zhang, Yingxue XU, Jianqi Chen et al.

56

NEURIPS 2025arXiv:2409.15272

#50

OmniBench: Towards The Future of Universal Omni-Language Models

Yizhi Li, Ge Zhang, Yinghao Ma et al.

multimodal large language modelstri-modal processingomni-language modelscross-modal reasoning+3

53

CVPR 2025arXiv:2412.01720

#51

LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant

Yikun Liu, Yajie Zhang, jiayin cai et al.

53

CVPR 2024arXiv:2309.06255

#52

Enhancing Multimodal Cooperation via Sample-level Modality Valuation

Yake Wei, Ruoxuan Feng, Zihe Wang et al.

multimodal learningmodality valuationsample-level analysismodality discrepancy+2

52

CVPR 2024arXiv:2310.00582

#53

Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs

shiyu xuan, Qingpei Guo, Ming Yang et al.

52

CVPR 2024arXiv:2401.08577

#54

MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D World

Yining Hong, Zishuo Zheng, Peihao Chen et al.

embodied large language modelsmultisensory interaction dataobject-centric representations3d scene encoding+4

52

ICML 2024arXiv:2406.01210

#55

GeminiFusion: Efficient Pixel-wise Multimodal Fusion for Vision Transformer

Ding Jia, Jianyuan Guo, Kai Han et al.

vision transformermultimodal fusioncross-modal attentionpixel-wise fusion+4

51

AAAI 2025arXiv:2402.19407

#56

MENTOR: Multi-level Self-supervised Learning for Multimodal Recommendation

Jinfeng Xu, Zheyu Chen, Shuo Yang et al.

multimodal recommendationself-supervised learningmodality alignmentgraph convolutional network+3

50

AAAI 2025arXiv:2408.09698

#57

Harnessing Multimodal Large Language Models for Multimodal Sequential Recommendation

Yuyang Ye, Zhi Zheng, Yishan Shen et al.

49

CVPR 2024arXiv:2403.10254

#58

Magic Tokens: Select Diverse Tokens for Multi-modal Object Re-Identification

Pingping Zhang, Yuhao Wang, Yang Liu et al.

multi-modal object re-identificationvision transformertoken selectionfeature aggregation+2

49

ICLR 2025arXiv:2408.02718

#59

Reinforced Adaptive Knowledge Learning for Multimodal Fake News Detection

Litian Zhang, Xiaoming Zhang, Chaozhuo Li et al.

MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models

Fanqing Meng, Jin Wang, Chuanhao Li et al.

multimodal multi-image understandingvision-language modelsmulti-image relationshipsspatial understanding+4

48

ICML 2024arXiv:2402.18819

#61

Dual Operating Modes of In-Context Learning

Ziqian Lin, Kangwook Lee

in-context learningtask retrievaltask learningprobabilistic modeling+3

46

CVPR 2024arXiv:2312.04521

#62

Multimodal Industrial Anomaly Detection by Crossmodal Feature Mapping

Alex Costanzino, Pierluigi Zama Ramirez, Giuseppe Lisanti et al.

multimodal anomaly detectioncrossmodal feature mappingindustrial inspectionpoint cloud processing+3

46

ECCV 2024arXiv:2312.02896

#63

BenchLMM: Benchmarking Cross-style Visual Capability of Large Multimodal Models

Rizhao Cai, Zirui Song, DAYAN GUAN et al.

45

ICLR 2025arXiv:2411.04986

#64

The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalities

Zhaofeng Wu, Xinyan Yu, Dani Yogatama et al.

43

CVPR 2024arXiv:2311.17597

#65

Continual Self-supervised Learning: Towards Universal Multi-modal Medical Data Representation Learning

Yiwen Ye, Yutong Xie, Jianpeng Zhang et al.

43

ICLR 2024arXiv:2403.10667

#66

Towards Unified Multi-Modal Personalization: Large Vision-Language Models for Generative Recommendation and Beyond

Tianxin Wei, Bowen Jin, Ruirui Li et al.

43

AAAI 2025arXiv:2412.15691

#67

Exploiting Multimodal Spatial-temporal Patterns for Video Object Tracking

Xiantao Hu, Ying Tai, Xu Zhao et al.

42

ICML 2024arXiv:2405.09321

#68

ReconBoost: Boosting Can Achieve Modality Reconcilement

Cong Hua, Qianqian Xu, Shilong Bao et al.

modality reconcilementmulti-modal learningmodality competitionalternating learning+4

41

ICLR 2024arXiv:2311.11385

#69

Multi-Task Reinforcement Learning with Mixture of Orthogonal Experts

Ahmed Hendawy, Jan Peters, Carlo D'Eramo

41

CVPR 2024arXiv:2507.13364

#70

OmniVec2 - A Novel Transformer based Network for Large Scale Multimodal and Multitask Learning

Siddharth Srivastava, Gaurav Sharma

39

CVPR 2024arXiv:2403.02969

#71

Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception

Junwen He, Yifan Wang, Lijun Wang et al.

39

ECCV 2024arXiv:2309.05300

#72

Decoupling Common and Unique Representations for Multimodal Self-supervised Learning

Yi Wang, Conrad M Albrecht, Nassim Ait Ali Braham et al.

39

ECCV 2024arXiv:2407.09705

#73

Rethinking Reverse Distillation for Multi-Modal Anomaly Detection

Zhihao Gu, Jiangning Zhang, Liang Liu et al.

Diagnosing and Re-learning for Balanced Multimodal Learning

Yake Wei, Siwei Li, Ruoxuan Feng et al.

multimodal learningmodality imbalancerepresentation separabilityencoder re-initialization+3

38

AAAI 2024arXiv:2312.08762

#75

Multi-Modal Latent Space Learning for Chain-of-Thought Reasoning in Language Models

Liqi He, Zuchao Li, Xiantao Cai et al.

chain-of-thought reasoningmulti-modal reasoninglatent space learningdiffusion processes+4

37

CVPR 2024arXiv:2403.02628

#76

Interactive Continual Learning: Fast and Slow Thinking

Biqing Qi, Xinquan Chen, Junqi Gao et al.

continual learningcomplementary learning systemsmultimodal large language modelsvision transformer models+4

36

AAAI 2024arXiv:2401.08206

#77

Generative Multi-Modal Knowledge Retrieval with Large Language Models

Xinwei Long, Jiali Zeng, Fandong Meng et al.

35

AAAI 2024arXiv:2312.14667

#78

Token-Level Contrastive Learning with Modality-Aware Prompting for Multimodal Intent Recognition

Qianrui Zhou, Hua Xu, Hao Li et al.

multimodal intent recognitioncontrastive learningmodality-aware promptingcross-modality attention+4

35

CVPR 2024arXiv:2303.15230

#79

Troika: Multi-Path Cross-Modal Traction for Compositional Zero-Shot Learning

Siteng Huang, Biao Gong, Yutong Feng et al.

CVPR 2024arXiv:2312.00968

#80

Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Experts

Jialin Wu, Xia Hu, Yaqing Wang et al.

ICLR 2025arXiv:2409.07402

#81

What to align in multimodal contrastive learning?

Benoit Dufumier, Javiera Castillo Navarro, Devis Tuia et al.

multimodal contrastive learningmultimodal self-supervised learningmultimodal representation alignmentmultimodal interactions+3

CVPR 2024arXiv:2403.07241

#82

Calibrating Multi-modal Representations: A Pursuit of Group Robustness without Annotations

Chenyu You, Yifei Min, Weicheng Dai et al.

ICLR 2024arXiv:2309.15564

#83

Jointly Training Large Autoregressive Multimodal Models

Emanuele Aiello, Lili Yu, Yixin Nie et al.

ICLR 2025arXiv:2412.11959

#84

Gramian Multimodal Representation Learning and Alignment

Giordano Cicchetti, Eleonora Grassucci, Luigi Sigillo et al.

multimodal representation learningcontrastive learningmodality alignmentgramian volume minimization+3

33

ECCV 2024arXiv:2407.05540

#85

GTP-4o: Modality-prompted Heterogeneous Graph Learning for Omni-modal Biomedical Representation

Chenxin Li, Xinyu Liu, Cheng Wang et al.

heterogeneous graph learningmulti-modal representationmodality-prompted completiongraph embedding+4

33

CVPR 2025arXiv:2502.20808

#86

MV-MATH: Evaluating Multimodal Math Reasoning in Multi-Visual Contexts

Peijie Wang, Zhong-Zhi Li, Fei Yin et al.

33

ICML 2024arXiv:2405.01926

#87

Auto-Encoding Morph-Tokens for Multimodal LLM

Kaihang Pan, Siliang Tang, Juncheng Li et al.

morph-tokensmultimodal llmsvisual comprehensionvisual generation+3

32

CVPR 2024arXiv:2311.16081

#88

ViT-Lens: Towards Omni-modal Representations

Stan Weixian Lei, Yixiao Ge, Kun Yi et al.

32

CVPR 2025arXiv:2411.18620

#89

Cross-modal Information Flow in Multimodal Large Language Models

Zhi Zhang, Srishti Yadav, Fengze Han et al.

32

ICLR 2025arXiv:2407.11895

#90

OmniBind: Large-scale Omni Multimodal Representation via Binding Spaces

zehan wang, Ziang Zhang, Minjie Hong et al.

CVPR 2025arXiv:2503.00513

#91

Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuning

Hanxun Yu, Wentong Li, Song Wang et al.

CVPR 2025arXiv:2412.19412

#92

MINIMA: Modality Invariant Image Matching

Jiangwei Ren, Xingyu Jiang, Zizhuo Li et al.

NEURIPS 2025arXiv:2410.12787

#93

The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audio

Sicong Leng, Yun Xing, Zesen Cheng et al.

large multimodal modelshallucination evaluationmultimodal integrationcross-modal learning+4

CVPR 2024arXiv:2306.08832

#94

Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understanding

Le Zhang, Rabiul Awal, Aishwarya Agrawal

multimodal representation learningmissing modalitiesmissing labelsgraph contrastive learning+4

#95

Multimodal Patient Representation Learning with Missing Modalities and Labels

Zhenbang Wu, Anant Dadu, Nicholas Tustison et al.

ICLR 2024

ICLR 2025arXiv:2410.06172

#96

Multimodal Situational Safety

Kaiwen Zhou, Chengzhi Liu, Xuandong Zhao et al.

AAAI 2025arXiv:2404.14368

#97

Graphic Design with Large Multimodal Model

Yutao Cheng, Zhao Zhang, Maoke Yang et al.

graphic layout generationhierarchical layout generationmultimodal sequence generationlayout generation model+4

ICML 2025arXiv:2412.01441

#98

LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations

Anian Ruoss, Fabio Pardo, Harris Chan et al.

ICML 2025arXiv:2412.17451

#99

Diving into Self-Evolving Training for Multimodal Reasoning

Wei Liu, Junlong Li, Xiwen Zhang et al.

ECCV 2024arXiv:2407.11351

#100

Learning Modality-agnostic Representation for Semantic Segmentation from Any Modalities

Xu Zheng, Yuanhuiyi Lyu, LIN WANG

modality-agnostic representationsemantic segmentationknowledge distillationmulti-modal vision-language models+3

28