Transformer Architecture

ICLR 2024arXiv:2310.06625

#2

iTransformer: Inverted Transformers Are Effective for Time Series Forecasting

Yong Liu, Tengge Hu, Haoran Zhang et al.

1,356

ICML 2024arXiv:2405.21060

#3

Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality

Tri Dao, Albert Gu

state-space modelsattention mechanismtransformer architecturestructured matrices+4

1,146

ICLR 2024arXiv:2309.16588

#4

Vision Transformers Need Registers

Timothée Darcet, Maxime Oquab, Julien Mairal et al.

649

ICML 2024arXiv:2312.06635

#5

Gated Linear Attention Transformers with Hardware-Efficient Training

Songlin Yang, Bailin Wang, Yikang Shen et al.

linear attentionhardware-efficient traininggated linear attentionlength generalization+4

329

CVPR 2024arXiv:2311.17132

#6

TransNeXt: Robust Foveal Visual Perception for Vision Transformers

Dai Shi

279

ECCV 2024arXiv:2312.08874

#7

Agent Attention: On the Integration of Softmax and Linear Attention

Dongchen Han, Tianzhu Ye, Yizeng Han et al.

attention mechanismlinear attentioncomputational efficiencyvision transformers+3

212

CVPR 2024arXiv:2309.11523

#8

RMT: Retentive Networks Meet Vision Transformers

Qihang Fan, Huaibo Huang, Mingrui Chen et al.

184

ICML 2024arXiv:2402.01032

#9

Repeat After Me: Transformers are Better than State Space Models at Copying

Samy Jelassi, David Brandfonbrener, Sham Kakade et al.

sequence modelingstate space modelsattention mechanismtransformer architecture+4

162

CVPR 2024arXiv:2306.05427

#10

Grounded Text-to-Image Synthesis with Attention Refocusing

Quynh Phung, Songwei Ge, Jia-Bin Huang

text-to-image synthesisdiffusion modelsattention mechanismcross-attention layers+4

162

ICLR 2024arXiv:2307.03576

#11

One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attention

Arvind Mahankali, Tatsunori Hashimoto, Tengyu Ma

150

NEURIPS 2025arXiv:2501.00663

#12

Titans: Learning to Memorize at Test Time

Ali Behrouz, Peilin Zhong, Vahab Mirrokni

147

ICML 2024arXiv:2402.18668

#13

Simple linear attention language models balance the recall-throughput tradeoff

Simran Arora, Sabri Eyuboglu, Michael Zhang et al.

attention mechanismlinear attentionlanguage modelsrecall ability+4

140

ICLR 2024arXiv:2404.03663

#14

Spike-driven Transformer V2: Meta Spiking Neural Network Architecture Inspiring the Design of Next-generation Neuromorphic Chips

Man Yao, Jiakui Hu, Tianxiang Hu et al.

130

ICML 2024arXiv:2404.08472

#15

TSLANet: Rethinking Transformers for Time Series Representation Learning

Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen et al.

time series representation learningadaptive spectral blockfourier analysisinteractive convolution block+3

127

ICLR 2024arXiv:2312.04927

#16

Zoology: Measuring and Improving Recall in Efficient Language Models

Simran Arora, Sabri Eyuboglu, Aman Timalsina et al.

123

ECCV 2024arXiv:2402.05892

#17

Mamba-ND: Selective State Space Modeling for Multi-Dimensional Data

Shufan Li, Aditya Grover, Harkanwar Singh

selective state space modelingmulti-dimensional datasequence modelingself-attention complexity+3

106

ICML 2024arXiv:2310.05249

#18

In-context Convergence of Transformers

Yu Huang, Yuan Cheng, Yingbin LIANG

in-context learningtransformer dynamicsattention mechanismgradient descent analysis+4

106

AAAI 2024arXiv:2312.10325

#19

An Attentive Inductive Bias for Sequential Recommendation beyond the Self-Attention

Yehjin Shin, Jeongwhan Choi, Hyowon Wi et al.

sequential recommendationself-attention mechanismoversmoothing problemtransformer-based models+4

104

CVPR 2025arXiv:2503.10622

#20

Transformers without Normalization

Jiachen Zhu, Xinlei Chen, Kaiming He et al.

transformer architecturenormalization layersdynamic tanhself-supervised learning+3

103

CVPR 2024arXiv:2401.16456

#21

SHViT: Single-Head Vision Transformer with Memory Efficient Macro Design

Seokju Yun, Youngmin Ro

vision transformersingle-head attentionpatch embeddingsmemory efficient design+4

102

ICML 2024arXiv:2402.14735

#22

How Transformers Learn Causal Structure with Gradient Descent

Eshaan Nichani, Alex Damian, Jason Lee

attention mechanismcausal structure learningin-context learninggradient descent analysis+4

102

ICLR 2024arXiv:2310.08744

#23

Circuit Component Reuse Across Tasks in Transformer Language Models

Jack Merullo, Carsten Eickhoff, Ellie Pavlick

99

ICLR 2025arXiv:2410.02367

#24

SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration

Jintao Zhang, Jia wei, Pengle Zhang et al.

95

ICML 2024arXiv:2402.05602

#25

AttnLRP: Attention-Aware Layer-Wise Relevance Propagation for Transformers

Reduan Achtibat, Sayed Mohammad Vakilzadeh Hatefi, Maximilian Dreyer et al.

attention mechanismlayer-wise relevance propagationtransformer interpretabilityfaithful attribution+4

92

ICLR 2025arXiv:2409.10594

#26

Kolmogorov-Arnold Transformer

Xingyi Yang, Xinchao Wang

kolmogorov-arnold networktransformer architecturerational basis functionsparameter efficiency+4

92

ICLR 2024arXiv:2402.04347

#27

The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicry

Michael Zhang, Kush Bhatia, Hermann Kumbong et al.

91

NEURIPS 2025arXiv:2409.10516

#28

RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval

Di Liu, Meng Chen, Baotong Lu et al.

attention mechanismlong-context inferencevector retrievalapproximate nearest neighbor search+3

90

ICLR 2024arXiv:2305.12095

#29

CARD: Channel Aligned Robust Blend Transformer for Time Series Forecasting

xue wang, Tian Zhou, Qingsong Wen et al.

83

CVPR 2024arXiv:2403.16131

#30

Salience DETR: Enhancing Detection Transformer with Hierarchical Salience Filtering Refinement

Xiuquan Hou, Meiqin Liu, Senlin Zhang et al.

detection transformerquery selectionhierarchical salience filteringscale bias+4

82

ICLR 2024arXiv:2310.01082

#31

Linear attention is (maybe) all you need (to understand Transformer optimization)

Kwangjun Ahn, Xiang Cheng, Minhak Song et al.

79

ICLR 2024arXiv:2310.10616

#32

How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representations

Tianyu Guo, Wei Hu, Song Mei et al.

77

CVPR 2024arXiv:2401.08209

#33

Transcending the Limit of Local Window: Advanced Super-Resolution Transformer with Adaptive Token Dictionary

Leheng Zhang, Yawei Li, Xingyu Zhou et al.

single image super-resolutionwindow-based self-attentionadaptive token dictionarycategory-based self-attention+3

76

ICCV 2025arXiv:2503.07027

#34

EasyControl: Adding Efficient and Flexible Control for Diffusion Transformer

Yuxuan Zhang, Yirui Yuan, Yiren Song et al.

diffusion transformercondition injectionspatial controlsubject control+4

75

ICLR 2024arXiv:2310.03016

#35

Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functions

Satwik Bhattamishra, Arkil Patel, Phil Blunsom et al.

75

ICML 2025arXiv:2503.16428

#36

XAttention: Block Sparse Attention with Antidiagonal Scoring

Ruyi Xu, Guangxuan Xiao, Haofeng Huang et al.

71

ICLR 2025arXiv:2503.07656

#37

DriveTransformer: Unified Transformer for Scalable End-to-End Autonomous Driving

Xiaosong Jia, Junqi You, Zhiyuan Zhang et al.

end-to-end autonomous drivingtask parallelismsparse representationstreaming processing+4

70

ICLR 2024arXiv:2310.08566

#38

Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretraining

Licong Lin, Yu Bai, Song Mei

70

ICLR 2024arXiv:2303.06373

#39

Recursive Generalization Transformer for Image Super-Resolution

Zheng Chen, Yulun Zhang, Jinjin Gu et al.

68

ICLR 2024arXiv:2311.12424

#40

Looped Transformers are Better at Learning Learning Algorithms

Liu Yang, Kangwook Lee, Robert Nowak et al.

67

CVPR 2024arXiv:2403.14302

#41

SpikingResformer: Bridging ResNet and Vision Transformer in Spiking Neural Networks

Xinyu Shi, Zecheng Hao, Zhaofei Yu

67

CVPR 2025arXiv:2504.05298

#42

One-Minute Video Generation with Test-Time Training

Jiarui Xu, Shihao Han, Karan Dalal et al.

video generationtest-time traininglong-context modelingtransformer architecture+3

67

ICML 2025arXiv:2502.04507

#43

Fast Video Generation with Sliding Tile Attention

Peiyuan Zhang, Yongqi Chen, Runlong Su et al.

64

ICML 2024arXiv:2312.06528

#44

Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Context

Xiang Cheng, Yuxin Chen, Suvrit Sra

transformers architecturegradient descentin-context learningnon-linear functions+2

63

ICLR 2025arXiv:2503.03321

#45

See What You Are Told: Visual Attention Sink in Large Multimodal Models

Seil Kang, Jinyeong Kim, Junhyeok Kim et al.

attention mechanismvisual attention sinkmultimodal modelsvision-language tasks+4

61

ICML 2024arXiv:2402.09268

#46

Transformers, parallel computation, and logarithmic depth

Clayton Sanford, Daniel Hsu, Matus Telgarsky

self-attention layersparallel computationlogarithmic-depth transformersneural sequence models+2

60

ICLR 2025arXiv:2411.13676

#47

Hymba: A Hybrid-head Architecture for Small Language Models

Xin Dong, Yonggan Fu, Shizhe Diao et al.

small language modelshybrid-head architecturetransformer attention mechanismsstate space models+3

58

ECCV 2024arXiv:2407.05878

#48

HiT-SR: Hierarchical Transformer for Efficient Image Super-Resolution

Xiang Zhang, Yulun Zhang, Fisher Yu

56

ICML 2024arXiv:2402.10198

#49

SAMformer: Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attention

Romain Ilbert, Ambroise Odonnat, Vasilii Feofanov et al.

time series forecastingtransformer architecturesharpness-aware minimizationchannel-wise attention+4

56

ICLR 2024arXiv:2401.11673

#50

MVSFormer++: Revealing the Devil in Transformer's Details for Multi-View Stereo

chenjie cao, xinlin ren, Yanwei Fu

54

AAAI 2024arXiv:2210.12381

#51

S2WAT: Image Style Transfer via Hierarchical Vision Transformer Using Strips Window Attention

Chiyu Zhang, Xiaogang Xu, Lei Wang et al.

style transfervision transformerattention mechanismwindow attention+4

52

ICLR 2025arXiv:2406.09415

#52

An Image is Worth More Than 16x16 Patches: Exploring Transformers on Individual Pixels

Duy-Kien Nguyen, Mahmoud Assran, Unnat Jain et al.

51

ICLR 2024arXiv:2310.04064

#53

How to Capture Higher-order Correlations? Generalizing Matrix Softmax Attention to Kronecker Computation

Josh Alman, Zhao Song

49

ICLR 2024arXiv:2311.01906

#54

Simplifying Transformer Blocks

Bobby He, Thomas Hofmann

49

AAAI 2024arXiv:2401.03870

#55

Gramformer: Learning Crowd Counting via Graph-Modulated Transformer

Hui LIN, Zhiheng Ma, Xiaopeng Hong et al.

49

ICLR 2024arXiv:2310.00535

#56

JoMA: Demystifying Multilayer Transformers via Joint Dynamics of MLP and Attention

Yuandong Tian, Yiping Wang, Zhenyu Zhang et al.

48

CVPR 2024arXiv:2404.02117

#57

Pre-trained Vision and Language Transformers Are Few-Shot Incremental Learners

Keon Hee Park, Kyungwoo Song, Gyeong-Moon Park

48

ICLR 2024arXiv:2303.06440

#58

Xformer: Hybrid X-Shaped Transformer for Image Denoising

Jiale Zhang, Yulun Zhang, Jinjin Gu et al.

47

ICML 2024arXiv:2406.15765

#59

Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration

Zhongzhi Yu, Zheng Wang, Yonggan Fu et al.

attention mechanismattention sinkslarge language modelsattention calibration+2

47

ICLR 2025arXiv:2310.12680

#60

On the Optimization and Generalization of Multi-head Attention

Christos Thrampoulidis, Rouzbeh Ghaderi, Hossein Taheri et al.

attention mechanismmulti-head attentionoptimization guaranteesgeneralization guarantees+4

44

CVPR 2024arXiv:2404.01882

#61

Scene Adaptive Sparse Transformer for Event-based Object Detection

Yansong Peng, Li Hebei, Yueyi Zhang et al.

43

ICML 2024arXiv:2202.13060

#62

Graph Attention Retrospective

Kimon Fountoulakis, Amit Levi, Shenghao Yang et al.

graph attention networksnode classificationstochastic block modelmixture of gaussians+3

43

ICML 2025arXiv:2502.18137

#63

SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inference

Jintao Zhang, Chendong Xiang, Haofeng Huang et al.

43

NEURIPS 2025arXiv:2505.13389

#64

Faster Video Diffusion with Trainable Sparse Attention

Peiyuan Zhang, Yongqi Chen, Haofeng Huang et al.

AAAI 2025arXiv:2407.09298

#65

Transformer Layers as Painters

Qi Sun, Marc Pickett, Aakash Kumar Nain et al.

transformer interpretabilitylayer ablation analysismodel robustnessparallel layer execution+3

ECCV 2024arXiv:2403.13042

#66

TAPTR: Tracking Any Point with Transformers as Detection

Hongyang Li, Hao Zhang, Shilong Liu et al.

ICLR 2025arXiv:2501.15061

#67

PolaFormer: Polarity-aware Linear Attention for Vision Transformers

Weikang Meng, Yadan Luo, Xin Li et al.

linear attentionvision transformersattention mechanismquery-key interactions+2

ICLR 2024arXiv:2403.16030

#68

VCR-Graphormer: A Mini-batch Graph Transformer via Virtual Connections

Dongqi Fu, Zhigang Hua, Yan Xie et al.

41

ICML 2025arXiv:2410.11842

#69

MoH: Multi-Head Attention as Mixture-of-Head Attention

Peng Jin, Bo Zhu, Li Yuan et al.

40

ICLR 2025arXiv:2402.04161

#70

Attention with Markov: A Curious Case of Single-layer Transformers

Ashok Makkuva, Marco Bondaschi, Adway Girish et al.

attention mechanismmarkov input processesinduction head mechanismsingle-layer transformers+3

39

ICLR 2025arXiv:2409.04431

#71

Theory, Analysis, and Best Practices for Sigmoid Self-Attention

Jason Ramapuram, Federico Danieli, Eeshan Gunesh Dhekane et al.

sigmoid attentiontransformer architectureuniversal function approximationattention mechanism alternatives+2

39

CVPR 2024arXiv:2507.13364

#72

OmniVec2 - A Novel Transformer based Network for Large Scale Multimodal and Multitask Learning

Siddharth Srivastava, Gaurav Sharma

39

ICML 2024arXiv:2402.01258

#73

Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscape

Juno Kim, Taiji Suzuki

attention mechanismmean-field dynamicsin-context learningnonconvex optimization+4

38

ICLR 2024arXiv:2301.02240

#74

Skip-Attention: Improving Vision Transformers by Paying Less Attention

Shashank Venkataramanan, Amir Ghodrati, Yuki Asano et al.

37

ICML 2024arXiv:2402.13512

#75

From Self-Attention to Markov Models: Unveiling the Dynamics of Generative Transformers

Muhammed Emrullah Ildiz, Yixiao HUANG, Yingcong Li et al.

self-attention mechanismmarkov modelsgenerative transformersteacher-student setting+4

36

ICML 2024arXiv:2405.11582

#76

SLAB: Efficient Transformers with Simplified Linear Attention and Progressive Re-parameterized Batch Normalization

Jialong Guo, Xinghao Chen, Yehui Tang et al.

linear attentiontransformer efficiencybatch normalization reparameterizationimage classification+4

NEURIPS 2025arXiv:2501.06425

#77

Tensor Product Attention Is All You Need

Yifan Zhang, Yifeng Liu, Huizhuo Yuan et al.

attention mechanismtensor decompositionskv cache compressionsequence modeling+4

ICML 2024arXiv:2402.15607

#78

How Do Nonlinear Transformers Learn and Generalize in In-Context Learning?

Hongkang Li, Meng Wang, Songtao Lu et al.

in-context learningtransformer training dynamicsnonlinear self-attentiongeneralization analysis+3

CVPR 2024arXiv:2401.06312

#79

Video Super-Resolution Transformer with Masked Inter&Intra-Frame Attention

Xingyu Zhou, Leheng Zhang, Xiaorui Zhao et al.

video super-resolutionvision transformerattention mechanismcomputational efficiency+4

ICML 2025arXiv:2502.01951

#80

On the Emergence of Position Bias in Transformers

Xinyi Wu, Yifei Wang, Stefanie Jegelka et al.

ICLR 2024arXiv:2306.02010

#81

Memorization Capacity of Multi-Head Attention in Transformers

Sadegh Mahdavi, Renjie Liao, Christos Thrampoulidis

33

ICLR 2024arXiv:2310.03817

#82

Logical Languages Accepted by Transformer Encoders with Hard Attention

Pablo Barcelo, Alexander Kozachinskiy, Anthony W. Lin et al.

33

ICLR 2024arXiv:2310.10375

#83

GTA: A Geometry-Aware Attention Mechanism for Multi-View Transformers

Takeru Miyato, Bernhard Jaeger, Max Welling et al.

32

AAAI 2024arXiv:2402.04563

#84

Attention Guided CAM: Visual Explanations of Vision Transformer Guided by Self-Attention

Saebom Leem, Hyunseok Seo

vision transformerattention mechanismvisual explanationsweakly-supervised localization+4

32

ICLR 2025arXiv:2410.11261

#85

Beyond Linear Approximations: A Novel Pruning Approach for Attention Matrix

Yingyu Liang, Jiangxuan Long, Zhenmei Shi et al.

32

NEURIPS 2025arXiv:2505.11594

#86

SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Training

Jintao Zhang, Jia wei, Haoxu Wang et al.

ICLR 2025arXiv:2503.02130

#87

Forgetting Transformer: Softmax Attention with a Forget Gate

Zhixuan Lin, Evgenii Nikishin, Xu He et al.

ICLR 2024arXiv:2310.05719

#88

Transformer Fusion with Optimal Transport

Moritz Imfeld, Jacopo Graldi, Marco Giordano et al.

ICLR 2025arXiv:2405.13998

#89

CViT: Continuous Vision Transformer for Operator Learning

Sifan Wang, Jacob Seidman, Shyam Sankaran et al.

operator learningvision transformer encodercross-attention mechanismpartial differential equations+4

ICLR 2024arXiv:2310.03022

#90

Decision ConvFormer: Local Filtering in MetaFormer is Sufficient for Decision Making

Jeonghye Kim, Su Young Lee, Woojun Kim et al.

30

ICLR 2024arXiv:2309.16354

#91

Transformer-VQ: Linear-Time Transformers via Vector Quantization

Lucas D. Lingle

30

ICLR 2024arXiv:2310.13121

#92

Understanding Addition in Transformers

Philip Quirke, Fazl Barez

30

AAAI 2024arXiv:2305.04440

#93

Vision Transformer Off-the-Shelf: A Surprising Baseline for Few-Shot Class-Agnostic Counting

Zhicheng Wang, Liwen Xiao, Zhiguo Cao et al.

class-agnostic countingvision transformerfew-shot learningself-attention mechanism+3

29

ICML 2024arXiv:2312.14820

#94

How Smooth Is Attention?

Valérie Castin, Pierre Ablin, Gabriel Peyré

attention mechanismlipschitz propertiesself-attention analysistransformer architecture+3

29

ICML 2024arXiv:2402.13934

#95

Do Efficient Transformers Really Save Computation?

Kai Yang, Jan Ackermann, Zhenyu He et al.

efficient transformerssparse transformerlinear transformerchain-of-thought reasoning+3

29

ICLR 2025arXiv:2412.17810

#96

Token Statistics Transformer: Linear-Time Attention via Variational Rate Reduction

Ziyang Wu, Tianjiao Ding, Yifu Lu et al.

attention mechanismlinear-time attentionmaximal coding rate reductionvariational objective+3

ICLR 2024arXiv:2307.14023

#97

Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators?

Tokio Kajitsuka, Issei Sato

ICLR 2024arXiv:2302.00456

#98

Analyzing Feed-Forward Blocks in Transformers through the Lens of Attention Maps

Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi et al.

ICML 2025arXiv:2411.17116

#99

Star Attention: Efficient LLM Inference over Long Sequences

Shantanu Acharya, Fei Jia, Boris Ginsburg

ICLR 2025arXiv:2412.06538

#100

Understanding Factual Recall in Transformers via Associative Memories

Eshaan Nichani, Jason Lee, Alberto Bietti