Aussie AI

Transformer Architectures

  • Last Updated 30 August, 2025
  • by David Spuler, Ph.D.

The Transformer was itself a major architectural advance in 2017. Since then, numerous modified Transformer architectures have been tested, and many ways to optimize Transformers have been found. Discussion of the various major architectural changes is given below; see also Transformer code optimizations, inference optimization techniques, and a very long list of Transformer optimizations

Global Transformer Architecture Changes

Since the introduction of the vanilla Transformer in 2017, researchers have been searching for optimizations up and down the Transformer's tech stack.

Global Transformer optimizations: Some of the architectural-level optimizations to Transformer inference engines include:

Depth-wise optimizations (layers):

  • Layer pruning / early exit: cutting short the layers in the encoder and/or decoder is a successful optimization strategy; see layer pruning and early exit inference.
  • Shallow decoder architecture. This idea for modifying the Transformer's decoder architecture uses layer pruning in the decoder to achieve a "deep encoder/shallow decoder" architecture, as reported in several research papers, such as Kasai et al. (2021) and Hsu et al. (2020); see shallow decoder architectures, and also depth pruning.

Width-wise optimizations (attention heads):

  • Attention head pruning: not all of the attention heads are important, esp. in the decoder (numerous research papers; see head pruning). There's some irony here when you consider the title of the original 2017 Transformer paper!
  • Flash Attention: Of all the various attention optimizations, Flash Attention (Dao et al., June 2022), and particularly Flash Attention 2 (Dao, July 2023), seems to have emerged as the most popular. See attention optimization methods.
  • Smaller Attention Head Components. It is possible to use more simplified attention head components, esp. in the decoder, as in Kasai et al. (2021); see approximate attention heads and head pruning. Another method is "weight sharing" for attention heads (or "fused" heads), such as in Zhai et al. (2023).

Lengthwise optimizations (input sequences):

Too Much of a Smart Thing: Just like in Highlander, there can be only one. No, wait, that's incorrect! It's called "multi-AI" or "ensemble" AI:

  • Ensemble architectures. Most architectures with two or more Transformers are aiming to achieve more advanced reasoning (usually at a worse speed), but the "big-small" dual architecture aims to improve inference speed by sending common queries to the smaller model. See enemble architectures.

Component-Level Transformer Architecture Changes

Attention heads are addressed above under width pruning, and layers are depth pruning, but various other Transformer components can be optimized:

Normalization optimizations:

Activation function optimizations:

Decoder algorithms:

  • Faster decoding algorithms. Research in Transformers includes beam search decoding and greedy decoding. There's also aggressive decoding, speculative decoding and collaborative decoding.
  • Speculative decoding (supervised dual decoding). A parallelization method whereby the decoding occurs in the small model to generate possible tokens. A larger model has the smaller model running ahead, and it confirms or vetos the suggested tokens, which is basically the same plot as Terminator II. No, but, I'm just checking if you're reading this stuff like an AI, rather than scanning and skipping like a human. If the small model is usually correct, this speeds up the overall process compared to only running a large model. This is similar to Big-Little architectures, but differs because both models are still running. See speculative decoding.

Feed-Fordware Network optimizations:

  • FFN Pruning. Simplified decoders, with FFN removed, as in Kasai et al. (2021), although this may be dependent on the use case; see "FFN pruning" section.

MatMul optimizations: Also known as GEMM and various other dumb names. It's matrix multiplication and vector dot product like you did in High School.

Softmax optimizations: Occurs less frequently than MatMul, but Softmax can still be optimized:

Positional encoding optimizations: Not usually considered a bottleneck, but even the PE can be optimized:

But wait, there's more. And there are more ways to optimize. Refer to the complete list of Transformer optimizations.

Survey Papers on Transformer Architectures

Several papers have surveyed the literature for the latest Transformer ideas:

Decoder-Only Architectures

Decoder-only architectures are the modern version of Transformers, such as GPT. It was discovered that the encoder in the older encoder-decoder Transformers from 2017 was not needed, and was actually an inefficiency. Decoder-only models are faster, and need fewer weights.

Research on the decoder-only transformer architectures:

  • Sathya Krishnan Suresh, Shunmugapriya P, 24 Apr 2024 (v2), Towards smaller, faster decoder-only transformers: Architectural variants and their implications, https://arxiv.org/abs/2404.14462 Code: https://github.com/SkAndMl/gpt-variations (Focuses on three new variants of decoder-only Transformer architectures: ParallelGPT (p-gpt), LinearlyCompressedGPT (lc-gpt), and ConvCompressedGPT (cc-gpt).)
  • Sachin Mehta, Mohammad Hossein Sekhavat, Qingqing Cao, Maxwell Horton, Yanzi Jin, Chenfan Sun, Iman Mirzadeh, Mahyar Najibi, Dmitry Belenko, Peter Zatloukal, Mohammad Rastegari, 22 Apr 2024, OpenELM: An Efficient Language Model Family with Open-source Training and Inference Framework, Apple Research, https://arxiv.org/abs/2404.14619 Code: https://huggingface.co/apple/OpenELM
  • Jesse Roberts, 2 Feb 2024 (v3), How Powerful are Decoder-Only Transformer Neural Models? https://arxiv.org/abs/2305.17026
  • Georgy Tyukin, 2 Apr 2024, Enhancing Inference Efficiency of Large Language Models: Investigating Optimization Strategies and Architectural Innovations, Masters Thesis, Data Science and Machine Learning, University College London., https://arxiv.org/abs/2404.05741 (Reviews various model compression and inference optimization techniques, and specifically analyzes layer skipping and sublayer skipping, such as attention head pruning and FFN/MLP pruning.)
  • Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy, Julien Launay, Colin Raffel, Apr 2022, What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization? https://arxiv.org/abs/2204.05832
  • Urvashi Khandelwal, Kevin Clark, Dan Jurafsky, Lukasz Kaiser, 21 May 2019, Sample Efficient Text Summarization Using a Single Pre-Trained Transformer, https://arxiv.org/abs/1905.08836
  • Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Łukasz Kaiser, Noam Shazeer, Jan 2018, GENERATING WIKIPEDIA BY SUMMARIZING LONG SEQUENCES, ICLR 2018 https://arxiv.org/pdf/1801.10198.pdf
  • Peter J Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating Wikipedia by Summarizing Long Sequences. In Proceedings of the 6th International Conference on Learning Representations (ICLR). https://arxiv.org/abs/1801.10198
  • Yumo Bai, Feb 3, 2024 Why are most LLMs decoder-only? Dive into the rabbit hole of recent advancement in Large Language Models, https://medium.com/@yumo-bai/why-are-most-llms-decoder-only-590c903e4789
  • M Fujitake, 2023 DTrOCR: Decoder-only Transformer for Optical Character Recognition, https://arxiv.org/pdf/2308.15996.pdf
  • Benjamin Bergner, Andrii Skliar, Amelie Royer, Tijmen Blankevoort, Yuki Asano, Babak Ehteshami Bejnordi, 26 Feb 2024, Think Big, Generate Quick: LLM-to-SLM for Fast Autoregressive Decoding, https://arxiv.org/abs/2402.16844
  • Mengwei Xu, Wangsong Yin, Dongqi Cai, Rongjie Yi, Daliang Xu, Qipeng Wang, Bingyang Wu, Yihao Zhao, Chen Yang, Shihe Wang, Qiyang Zhang, Zhenyan Lu, Li Zhang, Shangguang Wang, Yuanchun Li, Yunxin Liu, Xin Jin, Xuanzhe Liu, 16 Jan 2024, A Survey of Resource-efficient LLM and Multimodal Foundation Models, https://arxiv.org/abs/2401.08092 Project: https://github.com/UbiquitousLearning/Efficient_Foundation_Model_Survey
  • Meta, July 23, 2024, Introducing Llama 3.1: Our most capable models to date, https://ai.meta.com/blog/meta-llama-3-1/
  • Yiheng Liu, Hao He, Tianle Han, Xu Zhang, Mengyuan Liu, Jiaming Tian, Yutong Zhang, Jiaqi Wang, Xiaohui Gao, Tianyang Zhong, Yi Pan, Shaochen Xu, Zihao Wu, Zhengliang Liu, Xin Zhang, Shu Zhang, Xintao Hu, Tuo Zhang, Ning Qiang, Tianming Liu, Bao Ge, 6 Jan 2024 (v2), Understanding LLMs: A Comprehensive Overview from Training to Inference, https://arxiv.org/abs/2401.02038
  • Hao Zhou, Chengming Hu, Ye Yuan, Yufei Cui, Yili Jin, Can Chen, Haolun Wu, Dun Yuan, Li Jiang, Di Wu, Xue Liu, Charlie Zhang, Xianbin Wang, Jiangchuan Liu, 17 May 2024, Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunities, https://arxiv.org/abs/2405.10825
  • Andrea Matarazzo, Riccardo Torlone, 3 Jan 2025, A Survey on Large Language Models with some Insights on their Capabilities and Limitations, https://arxiv.org/abs/2501.04040 (Broad survey with many LLM topics covered from history to architectures to optimizations.)
  • Tong Xiao, Jingbo Zhu, 16 Jan 2025, Foundations of Large Language Models, https://arxiv.org/abs/2501.09223 (Huge 230 page paper on many topics such as training, prompting, alignment, and long context.)
  • Ailiang Lin, Zhuoyun Li, Kotaro Funakoshi, 31 Jul 2025, Causal2Vec: Improving Decoder-only LLMs as Versatile Embedding Models, https://arxiv.org/abs/2507.23386
  • Beilong Tang, Bang Zeng, Ming Li, 16 Aug 2025, LauraTSE: Target Speaker Extraction using Auto-Regressive Decoder-Only Language Models, https://arxiv.org/abs/2504.07402
  • Hamed Firooz, Maziar Sanjabi, Adrian Englhardt, Aman Gupta, Ben Levine, Dre Olgiati, Gungor Polatkan, Iuliia Melnychuk, Karthik Ramgopal, Kirill Talanine, Kutta Srinivasan, Luke Simon, Natesh Sivasubramoniapillai, Necip Fazil Ayan, Qingquan Song, Samira Sriram, Souvik Ghosh, Tao Song, Vignesh Kothapalli, Xiaoling Zhai, Ya Xu, Yu Wang, and Yun Dai, 23 Aug 2025, 360Brew: A Decoder-only Foundation Model for Personalized Ranking and Recommendation, https://arxiv.org/abs/2501.16450

Encoder-Decoder Architectures

Encoder-decoder Transformers are the older architecture from 2017. Decoder-only architectures have largely superceded this version, but it is still used in some use cases such as machine translation (foreign language translation).

Research on encoder-decoder architectures:

Encoder-Only Architectures

Encoder-only architectures lack a decoder, and are only used where the output is not a full text sequence. This makes sense in models where the output can be an embedding vector. Most modern LLMs are not using this architecture.

Research on encoder-only architectures:

  • Ting Hu, Christoph Meinel, Haojin Yang, 2024, A flexible BERT model enabling width- and depth-dynamic inference, Computer Speech & Language 4 April 2024, 101646, https://www.sciencedirect.com/science/article/pii/S0885230824000299 (Dual pruning method with layerwise "neural grafting" that gives dynamic width models, and combined with early exit on the depth dimension.)
  • Yehui Tang, Yunhe Wang, Jianyuan Guo, Zhijun Tu, Kai Han, Hailin Hu, Dacheng Tao, 5 Feb 2024. A Survey on Transformer Compression. https://arxiv.org/abs/2402.05964 (Model compression survey paper with focus on pruning, quantization, knowledge distillation, and efficient architecture design.)
  • Mengwei Xu, Wangsong Yin, Dongqi Cai, Rongjie Yi, Daliang Xu, Qipeng Wang, Bingyang Wu, Yihao Zhao, Chen Yang, Shihe Wang, Qiyang Zhang, Zhenyan Lu, Li Zhang, Shangguang Wang, Yuanchun Li, Yunxin Liu, Xin Jin, Xuanzhe Liu, 16 Jan 2024, A Survey of Resource-efficient LLM and Multimodal Foundation Models, https://arxiv.org/abs/2401.08092 Project: https://github.com/UbiquitousLearning/Efficient_Foundation_Model_Survey
  • Yiheng Liu, Hao He, Tianle Han, Xu Zhang, Mengyuan Liu, Jiaming Tian, Yutong Zhang, Jiaqi Wang, Xiaohui Gao, Tianyang Zhong, Yi Pan, Shaochen Xu, Zihao Wu, Zhengliang Liu, Xin Zhang, Shu Zhang, Xintao Hu, Tuo Zhang, Ning Qiang, Tianming Liu, Bao Ge, 6 Jan 2024 (v2), Understanding LLMs: A Comprehensive Overview from Training to Inference, https://arxiv.org/abs/2401.02038
  • Hao Zhou, Chengming Hu, Ye Yuan, Yufei Cui, Yili Jin, Can Chen, Haolun Wu, Dun Yuan, Li Jiang, Di Wu, Xue Liu, Charlie Zhang, Xianbin Wang, Jiangchuan Liu, 17 May 2024, Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunities, https://arxiv.org/abs/2405.10825
  • Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, Iacopo Poli, 19 Dec 2024 (v2)], Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference, https://arxiv.org/abs/2412.13663 (Encoder-only BERT model updated with modern optimizations including Flash attention, bias removal, RoPE, pre-norm, and GeGLU, a GELU varaint, hybrid local-global attention, and zero padding removal.)
  • Andrea Matarazzo, Riccardo Torlone, 3 Jan 2025, A Survey on Large Language Models with some Insights on their Capabilities and Limitations, https://arxiv.org/abs/2501.04040 (Broad survey with many LLM topics covered from history to architectures to optimizations.)
  • Tong Xiao, Jingbo Zhu, 16 Jan 2025, Foundations of Large Language Models, https://arxiv.org/abs/2501.09223 (Huge 230 page paper on many topics such as training, prompting, alignment, and long context.)

Hybrid Transformer Architectures

Transformer architectures have been merged with aspects of previous neural network theory to create hybrid architectures. Examples include:

  • Vision Transformer (ViT)
  • Transformer-RNN hybrid architectures
  • Transformer-CNN hybrid architectures

Research papers on hybrid transformer architectures:

Innovative New Transformer Architecture Research Papers

Since the original Transformer paper in 2017, and various other Transformer milestone papers, there have been numerous architectural variations proposed to alleviate efficiency or accuracy concerns. Research papers on specific modifications to the Transformer architecture include:

That's a Lot of BERTs!

BERT was an early 2019 Transformer architecture that was significantly innovative. Since then, there have been a great many variants of "BERT" (e.g. FastBERT, MobileBERT, DistilBERT, etc.). Research papers on variants of BERT include:

  • Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, May 2019, https://arxiv.org/abs/1810.04805, Code: https://github.com/google-research/bert (The one BERT to rule them all.)
  • Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. MobileBERT: a compact task-agnostic BERT for resource-limited devices. arXiv preprint arXiv:2004.02984, 2020. https://arxiv.org/abs/2004.02984
  • Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Haotang Deng, and Qi Ju. FastBERT: a self-distilling BERT with adaptive inference time. arXiv preprint arXiv:2004.02178, 2020. https://arxiv.org/abs/2004.02178
  • Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin. DeeBERT: Dynamic early exiting for accelerating BERT inference. arXiv preprint arXiv:2004.12993, 2020. https://arxiv.org/abs/2004.12993
  • Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019, https://arxiv.org/abs/1910.01108
  • Forrest N Iandola, Albert E Shaw, Ravi Krishna, and Kurt W Keutzer. SqueezeBERT: What can computer vision teach NLP about efficient neural networks? arXiv preprint arXiv:2006.11316, 2020. https://arxiv.org/abs/2006.11316
  • Daoyuan Chen, Yaliang Li, Minghui Qiu, Zhen Wang, Bofang Li, Bolin Ding, Hongbo Deng, Jun Huang, Wei Lin, and Jingren Zhou. AdaBERT: Task-adaptive BERT compression with differentiable neural architecture search. arXiv preprint arXiv:2001.04246, 2020. https://arxiv.org/abs/2001.04246
  • Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu. Dynabert: Dynamic bert with adaptive width and depth. arXiv preprint arXiv:2004.04037, 2020. https://arxiv.org/abs/2004.04037
  • Zejian Liu, Fanrong Li, Gang Li, and Jian Cheng. EBERT: Efficient BERT Inference with Dynamic Structured Pruning. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4814– 4823, 2021. https://aclanthology.org/2021.findings-acl.425/
  • Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized BERT pretraining approach,” CoRR, 2019. https://arxiv.org/abs/1907.11692
  • Z. Jiang, W. Yu, D. Zhou, Y. Chen, J. Feng, and S. Yan, “Convbert: Improving BERT with span-based dynamic convolution,” in NeurIPS, 2020, https://arxiv.org/abs/2008.02496
  • H. Bao, L. Dong, S. Piao, and F. Wei, “BEit: BERT pre-training of image transformers,” in International Conference on Learning Representations, 2022. https://arxiv.org/abs/2106.08254
  • Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, Qun Liu, Oct 2020, TinyBERT: Distilling BERT for Natural Language Understanding, https://arxiv.org/abs/1909.10351
  • Mengwei Xu, Wangsong Yin, Dongqi Cai, Rongjie Yi, Daliang Xu, Qipeng Wang, Bingyang Wu, Yihao Zhao, Chen Yang, Shihe Wang, Qiyang Zhang, Zhenyan Lu, Li Zhang, Shangguang Wang, Yuanchun Li, Yunxin Liu, Xin Jin, Xuanzhe Liu, 16 Jan 2024, A Survey of Resource-efficient LLM and Multimodal Foundation Models, https://arxiv.org/abs/2401.08092 Project: https://github.com/UbiquitousLearning/Efficient_Foundation_Model_Survey
  • Tong Xiao, Jingbo Zhu, 16 Jan 2025, Foundations of Large Language Models, https://arxiv.org/abs/2501.09223 (Huge 230 page paper on many topics such as training, prompting, alignment, and long context.)
  • Kester Wong, Sahan Bulathwela and Mutlu Cukurova, 19 Jul 2025, Exploring Human-AI Complementarity in CPS Diagnosis Using Unimodal and Multimodal BERT Models, https://arxiv.org/abs/2507.14579
  • Kester Wong, Sahan Bulathwela and Mutlu Cukurova, 19 Jul 2025, Explainable Collaborative Problem Solving Diagnosis with BERT using SHAP and its Implications for Teacher Adoption, https://arxiv.org/abs/2507.14584
  • Qiyao Xue, Yuchen Dou, Ryan Shi, Xiang Lorraine Li, Wei Gao, 1 Aug 2025, MMBERT: Scaled Mixture-of-Experts Multimodal BERT for Robust Chinese Hate Speech Detection under Cloaking Perturbations, https://arxiv.org/abs/2508.00760
  • Tianpei Lu, Bingsheng Zhang, Lekun Peng, Bowen Zheng, Lichun Li, Kui Ren, 3 Aug 2025, Privacy-Preserving Inference for Quantized BERT Models, https://arxiv.org/abs/2508.01636
  • Zijian Zhao, Fanyi Meng, Zhonghao Lyu, Hang Li, Xiaoyang Li, Guangxu Zhu, 3 Aug 2025, CSI-BERT2: A BERT-inspired Framework for Efficient CSI Prediction and Classification in Wireless Communication and Sensing, https://arxiv.org/abs/2412.06861
  • Tai Vu, Robert Yang, 14 Aug 2025, BERT-VQA: Visual Question Answering on Plots, https://arxiv.org/abs/2508.13184
  • Minh Tran, Jeffery C. Chan, Min Li Huang, Maya Kansara, John P. Grady, Christine E. Napier, Subotheni Thavaneswaran, Mandy L. Ballinger, David M. Thomas, Frank P. Lin, 21 Aug 2025, A Robust BERT-Based Deep Learning Model for Automated Cancer Type Extraction from Unstructured Pathology Reports, https://arxiv.org/abs/2508.15149
  • Kun Liu, Tuozhen Liu, Feifei Wang, and Rui Pan, 13 Aug 2025, A BERT-based Hierarchical Classification Model with Applications in Chinese Commodity Classification, https://arxiv.org/abs/2508.15800
  • Daniel Frees, Aditri Bhagirath, Moritz Bolling, 25 Aug 2025, Exploring Efficient Learning of Small BERT Networks with LoRA and DoRA, https://arxiv.org/abs/2508.17586
  • Suramya Jadhav, Abhay Shanbhag, Amogh Thakurdesai, Ridhima Sinare, Ananya Joshi, Raviraj Joshi, 24 Aug 2025, MahaParaphrase: A Marathi Paraphrase Detection Corpus and BERT-based Models, https://arxiv.org/abs/2508.17444

Next-Generation Architectures

What comes after Transformers? Maybe the answer is: more Transformers! Certainly, the newer multi-modal Transformers are gaining momentum, and there are other advanced Transformers:

  • Vision Transformer (ViT)
  • Multimodal transformer
  • Ensemble architectures (multi-AI, such as MoE)
  • Agent architectures (e.g., function calling, autonomous agents)
  • Advanced RAG architectures
  • Tool Augmented Language Models (TALM)
  • Retrieval Augment Language Models (RALM)
  • Compound AI architectures

However, there are some alternatives to Transformers that have been gathering steam. Here are a few newer architectures already being worked on:

  • State Space Models (SSMs)
  • RWKV (Transformer-RNN hybrid)
  • Mamba (a type of SSM)
  • Graph Neural Networks and Knowledge Graph extensions
  • S4 Hyena architecture
  • Spiking Neural Networks (SNNs) and Spiking Transformers
  • Weightless Neural Networks (WNNs)
  • Liquid Neural Networks (LNNs)
  • Hybrid Transformer-RNN architectures
  • Hybrid Transformer-CNN architectures

Research papers on next-gen architectures:

RWKV Architecture

The RWKV architecture is a hybrid Transformer-RNN architecture. Research papers on RWKV include:

  • Jean Mercat, Igor Vasiljevic, Sedrick Keh, Kushal Arora, Achal Dave, Adrien Gaidon, Thomas Kollar, 10 May 2024, Linearizing Large Language Models, https://arxiv.org/abs/2405.06640 Code: https://github.com/TRI-ML/linear_open_lm
  • Yehui Tang, Yunhe Wang, Jianyuan Guo, Zhijun Tu, Kai Han, Hailin Hu, Dacheng Tao, 5 Feb 2024. A Survey on Transformer Compression. https://arxiv.org/abs/2402.05964 (Model compression survey paper with focus on pruning, quantization, knowledge distillation, and efficient architecture design.)
  • Tiancheng Gu, Kaicheng Yang, Xiang An, Ziyong Feng, Dongnan Liu, Weidong Cai, Jiankang Deng, 11 Jun 2024. RWKV-CLIP: A Robust Vision-Language Representation Learner, https://arxiv.org/abs/2406.06973 Code: https://github.com/deepglint/RWKV-CLIP
  • Xinji Mai, Zeng Tao, Junxiong Lin, Haoran Wang, Yang Chang, Yanlan Kang, Yan Wang, Wenqiang Zhang, 27 Jun 2024, From Efficient Multimodal Models to World Models: A Survey, https://arxiv.org/abs/2407.00118 (A survey of multimodal models with coverage of many optimization techniques.)
  • 18 Apr 2024 (v2), The Efficiency Spectrum of Large Language Models: An Algorithmic Survey, Tianyu Ding, Tianyi Chen, Haidong Zhu, Jiachen Jiang, Yiqi Zhong, Jinxin Zhou, Guangzhi Wang, Zhihui Zhu, Ilya Zharkov, Luming Liang, https://arxiv.org/abs/2312.00678
  • Joanne Chen, July 23, 2024, What’s Next After Transformers, https://foundationcapital.com/whats-next-after-transformers/
  • Théodor Lemerle, Harrison Vanderbyl, Vaibhav Srivastav, Nicolas Obin, Axel Roebel, 30 Oct 2024, Lina-Speech: Gated Linear Attention is a Fast and Parameter-Efficient Learner for text-to-speech synthesis, https://arxiv.org/abs/2410.23320 https://theodorblackbird.github.io/blog/demo_lina/
  • Akul Datta, 5 Nov 2024, The Evolution of RWKV: Advancements in Efficient Language Modeling, https://arxiv.org/abs/2411.02795
  • From Transformers to the Future: An In-Depth Exploration of Modern Language Model Architectures H Xu, Z Bi, H Tseng, X Song, P Feng, https://osf.io/n8r5j/download
  • Wonkyo Choe, Yangfeng Ji, Felix Lin, 14 Dec 2024, RWKV-edge: Deeply Compressed RWKV for Resource-Constrained Devices, https://arxiv.org/abs/2412.10856
  • Haoyang Li, Yiming Li, Anxin Tian, Tianhao Tang, Zhanchao Xu, Xuejia Chen, Nicole Hu, Wei Dong, Qing Li, Lei Chen, 27 Dec 2024, A Survey on Large Language Model Acceleration based on KV Cache Management, https://arxiv.org/abs/2412.19442 (Huge survey of all KV cache optimization methods.)
  • Xiaoran Liu, Ruixiao Li, Mianqiu Huang, Zhigeng Liu, Yuerong Song, Qipeng Guo, Siyang He, Qiqi Wang, Linlin Li, Qun Liu, Yaqian Zhou, Xuanjing Huang, Xipeng Qiu, 24 Feb 2025, Thus Spake Long-Context Large Language Model, https://arxiv.org/abs/2502.17129 (Impressive survey of many techniques to improve efficiency and accuracy of long context processing in both inference and training, covering text, video and multimodal models.)
  • Sicheng Chen, Tianyi Zhang, Dankai Liao, Dandan Li, Low Chang Han, Yanqin Jiang, Yueming Jin, Shangqing Lyu, 5 Mar 2025, PathRWKV: Enabling Whole Slide Prediction with Recurrent-Transformer, https://arxiv.org/abs/2503.03199
  • Liu Xiao, Li Zhiyuan, Lin Yueyu, 27 Apr 2025, WuNeng: Hybrid State with Attention, https://arxiv.org/abs/2504.19191
  • Xiao Wang, Haiyang Wang, Shiao Wang, Qiang Chen, Jiandong Jin, Haoyu Song, Bo Jiang, Chenglong Li, 6 Aug 2025, RGB-Event based Pedestrian Attribute Recognition: A Benchmark Dataset and An Asymmetric RWKV Fusion Framework, https://arxiv.org/abs/2504.10018

State Space Models (SSMs)

  • Badri Narayana Patro, Vijay Srinivas Agneeswaran, 24 Apr 2024, Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challenges, https://arxiv.org/abs/2404.16112
  • 8 Jun 2024 (v2), A Survey on Efficient Inference for Large Language Models, Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, Shengen Yan, Guohao Dai, Xiao-Ping Zhang, Yuhan Dong, Yu Wang, https://arxiv.org/abs/2404.14294
  • Karan Goel, August 27, 2024, The On‑Device Intelligence Update https://cartesia.ai/blog/2024-08-27-on-device (On-device state space models.)
  • Nicolas Stellwag, 2024, Structured State Space Models, https://nicolasstellwag.com/download/structured_SSMs.pdf
  • Jinhao Li, Jiaming Xu, Shan Huang, Yonghua Chen, Wen Li, Jun Liu, Yaoxiu Lian, Jiayi Pan, Li Ding, Hao Zhou, Guohao Dai, 6 Oct 2024, Large Language Model Inference Acceleration: A Comprehensive Hardware Perspective, https://arxiv.org/abs/2410.04466
  • Cong Guo, Feng Cheng, Zhixu Du, James Kiessling, Jonathan Ku, Shiyu Li, Ziru Li, Mingyuan Ma, Tergel Molom-Ochir, Benjamin Morris, Haoxuan Shan, Jingwei Sun, Yitu Wang, Chiyue Wei, Xueying Wu, Yuhao Wu, Hao Frank Yang, Jingyang Zhang, Junyao Zhang, Qilin Zheng, Guanglei Zhou, Hai (Helen)Li, Yiran Chen, 8 Oct 2024. A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models, https://arxiv.org/abs/2410.07265
  • Xin Dong, Yonggan Fu, Shizhe Diao, Wonmin Byeon, Zijia Chen, Ameya Sunil Mahabaleshwarkar, Shih-Yang Liu, Matthijs Van Keirsbilck, Min-Hung Chen, Yoshi Suhara, Yingyan Lin, Jan Kautz, Pavlo Molchanov, 20 Nov 2024, Hymba: A Hybrid-head Architecture for Small Language Models, https://arxiv.org/abs/2411.13676
  • Yash Akhauri, Safeen Huda, Mohamed S. Abdelfattah, 26 Nov 2024, Attamba: Attending To Multi-Token States, https://arxiv.org/abs/2411.17685
  • Rui Pan, Zhuang Wang, Zhen Jia, Can Karakus, Luca Zancato, Tri Dao, Ravi Netravali, Yida Wang, 28 Nov 2024, Marconi: Prefix Caching for the Era of Hybrid LLMs, https://arxiv.org/abs/2411.19379 (Prefix caching applied to hybrid SSM-Transformer LLMs.)
  • Haoyang Li, Yiming Li, Anxin Tian, Tianhao Tang, Zhanchao Xu, Xuejia Chen, Nicole Hu, Wei Dong, Qing Li, Lei Chen, 27 Dec 2024, A Survey on Large Language Model Acceleration based on KV Cache Management, https://arxiv.org/abs/2412.19442 (Huge survey of all KV cache optimization methods.)
  • Jonas Ulmen, Ganesh Sundaram, and Daniel G\"orges, 14 Aug 2025, Learning State-Space Models of Dynamic Systems from Arbitrary Data using Joint Embedding Predictive Architectures, https://arxiv.org/abs/2508.10489
  • Xiaochun Lei, Siqi Wu, Weilin Wu, Zetao Jiang, 24 Jul 2025, MambaNeXt-YOLO: A Hybrid State Space Model for Real-time Object Detection, https://arxiv.org/abs/2506.03654
  • Sen Lu, Xiaoyu Zhang, Mingtao Hu, Eric Yeu-Jer Lee, Soohyeon Kim, Wei D. Lu, 18 Jul 2025, State Space Models Naturally Produce Traveling Waves, Time Cells, and Scale to Abstract Cognitive Functions, https://arxiv.org/abs/2507.13638
  • Saptarshi Mitra, Rachid Karami, Haocheng Xu, Sitao Huang, Hyoukjun Kwon, 19 Jul 2025, Characterizing State Space Model (SSM) and SSM-Transformer Hybrid Language Model Performance with Long Context Length, https://arxiv.org/abs/2507.12442
  • A. Quadir, M. Tanveer, 8 Aug 2025, Hypergraph Neural Network with State Space Models for Node Classification, https://arxiv.org/abs/2508.06587
  • Yizhuo Wu, Francesco Fioranelli, Chang Gao, 27 Jul 2025, RadMamba: Efficient Human Activity Recognition through Radar-based Micro-Doppler-Oriented Mamba State-Space Model, https://arxiv.org/abs/2504.12039
  • Shiva Raja, Cansu Demirkiran, Aakash Sarkar, Milos Popovic, Ajay Joshi, 29 Jul 2025, Systolic Array-based Accelerator for State-Space Models, https://arxiv.org/abs/2507.21394
  • Yifan Yu, Shengjie Xiu, Daniel P. Palomar, 30 Jul 2025, Robust Filtering and Learning in State-Space Models: Skewness and Heavy Tails Via Asymmetric Laplace Distribution, https://arxiv.org/abs/2507.22343
  • Hiroki Sakamoto and Kazuhiro Sato, 30 Jul 2025, Compression Method for Deep Diagonal State Space Model Based on $H^2$ Optimal Reduction, https://arxiv.org/abs/2507.10078
  • Julian Lemmel, Manuel Kranzl, Adam Lamine, Philipp Neubauer, Radu Grosu, Sophie Neubauer, 1 Aug 2025, Online Fine-Tuning of Carbon Emission Predictions using Real-Time Recurrent Learning for State Space Models, https://arxiv.org/abs/2508.00804
  • Joshua Dimasaka, Christian Gei{\ss}, Emily So, 2 Aug 2025, GraphVSSM: Graph Variational State-Space Model for Probabilistic Spatiotemporal Inference of Dynamic Exposure and Vulnerability for Regional Disaster Resilience Assessment, https://arxiv.org/abs/2508.01310
  • Federico Arangath Joseph, Kilian Konstantin Haefeli, Noah Liniger and Caglar Gulcehre, 3 Aug 2025, HiPPO-Prophecy: State-Space Models can Provably Learn Dynamical Systems in Context, https://arxiv.org/abs/2407.09375
  • Yiyi Wang, Jian'an Zhang, Hongyi Duan, Haoyang Liu, Qingyang Li, 5 Aug 2025, Rethinking Selectivity in State Space Models: A Minimal Predictive Sufficiency Approach, https://arxiv.org/abs/2508.03158
  • Leon G\"otz, Marcel Kollovieh, Stephan G\"unnemann, Leo Schwinn, 5 Aug 2025, Efficient Time Series Processing for Transformers and State-Space Models through Token Merging, https://arxiv.org/abs/2405.17951
  • Yuannuo Feng, Wenyong Zhou, Yuexi Lyu, Hanjie Liu, Zhengwu Liu, Ngai Wong, Wang Kang, 16 Aug 2025, HPD: Hybrid Projection Decomposition for Robust State Space Models on Analog CIM Hardware, https://arxiv.org/abs/2508.11935
  • Chenhui Xu, Dancheng Liu, Yuting Hu, Jiajie Li, Ruiyang Qin, Qingxiao Zheng, Jinjun Xiong, 16 Aug 2025, Sub-Sequential Physics-Informed Learning with State Space Model, https://arxiv.org/abs/2502.00318
  • Zhihao Zhan, Jianan Zhao, Zhaocheng Zhu, Jian Tang, 16 Aug 2025, Overcoming Long-Context Limitations of State-Space Models via Context-Dependent Sparse Attention, https://arxiv.org/abs/2507.00449
  • Hongfan Gao, Wangmeng Shen, Xiangfei Qiu, Ronghui Xu, Jilin Hu and Bin Yang, 19 Aug 2025, SSD-TS: Exploring the Potential of Linear State Space Models for Diffusion Models in Time Series Imputation, https://arxiv.org/abs/2410.13338
  • Peiming Li, Ziyi Wang, Yulin Yuan, Hong Liu, Xiangming Meng, Junsong Yuan, Mengyuan Liu, 20 Aug 2025, UST-SSM: Unified Spatio-Temporal State Space Models for Point Cloud Video Modeling, https://arxiv.org/abs/2508.14604
  • Trinayan Baruah, Kaustubh Shivdikar, Sara Prescott, and David Kaeli, 25 Aug 2025, Characterizing the Behavior of Training Mamba-based State Space Models on GPUs, https://arxiv.org/abs/2508.17679
  • Eric Alsmann, Martin Lange, 25 Aug 2025, The Computational Complexity of Satisfiability in State Space Models, https://arxiv.org/abs/2508.18162
  • Xavier Gonzalez, Leo Kozachkov, David M. Zoltowski, Kenneth L. Clarkson, Scott W. Linderman, 22 Aug 2025, Predictability Enables Parallelization of Nonlinear State Space Models, https://arxiv.org/abs/2508.16817
  • Siddharth Chaudhary, Bennett Browning, 20 Aug 2025, Hydra: A 1.6B-Parameter State-Space Language Model with Sparse Attention, Mixture-of-Experts, and Memory, https://arxiv.org/abs/2508.15099
  • Behnoush Khavari, Mehran Shakerinava, Jayesh Khullar, Jerry Huang, Fran\c{c}ois Rivest, Siamak Ravanbakhsh, Sarath Chandar, 10 Aug 2025, Parity Requires Unified Input Dependence and Negative Eigenvalues in SSMs, https://arxiv.org/abs/2508.07395

Hyena Architecture

  • Pierre-David Letourneau, Manish Kumar Singh, Hsin-Pai Cheng, Shizhong Han, Yunxiao Shi, Dalton Jones, Matthew Harper Langston, Hong Cai, Fatih Porikli, 16 Jul 2024, PADRe: A Unifying Polynomial Attention Drop-in Replacement for Efficient Vision Transformer, https://arxiv.org/abs/2407.11306
  • 18 Apr 2024 (v2), The Efficiency Spectrum of Large Language Models: An Algorithmic Survey, Tianyu Ding, Tianyi Chen, Haidong Zhu, Jiachen Jiang, Yiqi Zhong, Jinxin Zhou, Guangzhi Wang, Zhihui Zhu, Ilya Zharkov, Luming Liang, https://arxiv.org/abs/2312.00678
  • From Transformers to the Future: An In-Depth Exploration of Modern Language Model Architectures H Xu, Z Bi, H Tseng, X Song, P Feng, https://osf.io/n8r5j/download

Mamba Architecture

The Mamba architecture is an advanced AI architecture based on the State Space Model (SSM) architecture. Research papers on Mamba include:

  • Jean Mercat, Igor Vasiljevic, Sedrick Keh, Kushal Arora, Achal Dave, Adrien Gaidon, Thomas Kollar, 10 May 2024, Linearizing Large Language Models, https://arxiv.org/abs/2405.06640 Code: https://github.com/TRI-ML/linear_open_lm
  • Badri Narayana Patro, Vijay Srinivas Agneeswaran, 24 Apr 2024, Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challenges, https://arxiv.org/abs/2404.16112
  • Zeyu Wang, Chen Li, Huiying Xu, Xinzhong Zhu, 9 Jun 2024, Mamba YOLO: SSMs-Based YOLO For Object Detection, https://arxiv.org/abs/2406.05835
  • Mehmet Hamza Erol, Arda Senocak, Jiu Feng, Joon Son Chung, 5 Jun 2024, Audio Mamba: Bidirectional State Space Model for Audio Representation Learning, https://arxiv.org/abs/2406.03344
  • Zhengcong Fei, Mingyuan Fan, Changqian Yu, Debang Li, Youqiang Zhang, Junshi Huang, 3 Jun 2024, Dimba: Transformer-Mamba Diffusion Models, https://arxiv.org/abs/2406.01159
  • Radar AI, Mar 2024, An Introduction to the Mamba LLM Architecture: A New Paradigm in Machine Learning, https://www.datacamp.com/tutorial/introduction-to-the-mamba-llm-architecture
  • Albert Gu, Tri Dao, 31 May 2024 (v2), Mamba: Linear-Time Sequence Modeling with Selective State Spaces, https://arxiv.org/abs/2312.00752
  • Xiaogang Jia, Qian Wang, Atalay Donat, Bowen Xing, Ge Li, Hongyi Zhou, Onur Celik, Denis Blessing, Rudolf Lioutikov, Gerhard Neumann, 12 Jun 2024, MaIL: Improving Imitation Learning with Mamba, https://arxiv.org/abs/2406.08234
  • Marko Vidrih, Jun 7, 2024, Mamba-2 is Out: Can it replace Transformers? https://vidrihmarko.medium.com/mamba-2-is-out-can-it-replace-transformers-6cfb3372ea39
  • Albert Gu, Tri Dao, State Space Duality (Mamba-2) Part I - The Model, May 31, 2024, https://goombalab.github.io/blog/2024/mamba2-part1-model/
  • azhar, Dec 29, 2023, Decoding Mamba: The Next Big Leap in AI Sequence Modeling, https://medium.com/ai-insights-cobet/decoding-mamba-the-next-big-leap-in-ai-sequence-modeling-ef3908060cb8
  • Waleffe, Roger ; Byeon, Wonmin ; Riach, Duncan ; Norick, Brandon ; Korthikanti, Vijay ; Dao, Tri ; Gu, Albert ; Hatamizadeh, Ali ; Singh, Sudhakar ; Narayanan, Deepak ; Kulshreshtha, Garvit ; Singh, Vartika ; Casper, Jared ; Kautz, Jan ; Shoeybi, Mohammad ; Catanzaro, Bryan, June 2024, An Empirical Study of Mamba-based Language Models, https://arxiv.org/abs/2406.07887 https://ui.adsabs.harvard.edu/abs/2024arXiv240607887W/abstract
  • 8 Jun 2024 (v2), A Survey on Efficient Inference for Large Language Models, Zixuan Zhou, Xuefei Ning, Ke Hong, Tianyu Fu, Jiaming Xu, Shiyao Li, Yuming Lou, Luning Wang, Zhihang Yuan, Xiuhong Li, Shengen Yan, Guohao Dai, Xiao-Ping Zhang, Yuhan Dong, Yu Wang, https://arxiv.org/abs/2404.14294
  • Xinji Mai, Zeng Tao, Junxiong Lin, Haoran Wang, Yang Chang, Yanlan Kang, Yan Wang, Wenqiang Zhang, 27 Jun 2024, From Efficient Multimodal Models to World Models: A Survey, https://arxiv.org/abs/2407.00118 (A survey of multimodal models with coverage of many optimization techniques.)
  • Pierre-David Letourneau, Manish Kumar Singh, Hsin-Pai Cheng, Shizhong Han, Yunxiao Shi, Dalton Jones, Matthew Harper Langston, Hong Cai, Fatih Porikli, 16 Jul 2024, PADRe: A Unifying Polynomial Attention Drop-in Replacement for Efficient Vision Transformer, https://arxiv.org/abs/2407.11306
  • 18 Apr 2024 (v2), The Efficiency Spectrum of Large Language Models: An Algorithmic Survey, Tianyu Ding, Tianyi Chen, Haidong Zhu, Jiachen Jiang, Yiqi Zhong, Jinxin Zhou, Guangzhi Wang, Zhihui Zhu, Ilya Zharkov, Luming Liang, https://arxiv.org/abs/2312.00678
  • Haohao Qu, Liangbo Ning, Rui An, Wenqi Fan, Tyler Derr, Xin Xu, Qing Li, 2 Aug 2024, A Survey of Mamba, https://arxiv.org/abs/2408.01129
  • Jingwei Zuo, Maksim Velikanov, Dhiya Eddine, Ilyas Chahed, Younes Belkada, Guillaume Kunsch, August 12, 2024, Welcome FalconMamba: The first strong attention-free 7B model, https://huggingface.co/blog/falconmamba
  • Jamba Team, 22 Aug 2024, Jamba-1.5: Hybrid Transformer-Mamba Models at Scale, https://arxiv.org/abs/2408.12570
  • Danlong Yuan, Jiahao Liu, Bei Li, Huishuai Zhang, Jingang Wang, Xunliang Cai, Dongyan Zhao, 9 Aug 2024 (v2), ReMamba: Equip Mamba with Effective Long-Sequence Modeling, https://arxiv.org/abs/2408.15496
  • From Transformers to the Future: An In-Depth Exploration of Modern Language Model Architectures H Xu, Z Bi, H Tseng, X Song, P Feng, https://osf.io/n8r5j/download
  • Shengkun Tang, Liqun Ma, Haonan Li, Mingjie Sun, Zhiqiang Shen, 18 Nov 2024, Bi-Mamba: Towards Accurate 1-Bit State Space Models, https://arxiv.org/abs/2411.11843
  • Thanaphon Suwannaphong, Ferdian Jovan, Ian Craddock, Ryan McConville, 12 Dec 2024, Optimising TinyML with Quantization and Distillation of Transformer and Mamba Models for Indoor Localisation on Edge Devices, https://arxiv.org/abs/2412.09289
  • Mingjia Shi, Yuhao Zhou, Ruiji Yu, Zekai Li, Zhiyuan Liang, Xuanlei Zhao, Xiaojiang Peng, Tanmay Rajpurohit, Shanmukha Ramakrishna Vedantam, Wangbo Zhao, Kai Wang, Yang You, 17 Dec 2024, Faster Vision Mamba is Rebuilt in Minutes via Merged Token Re-training, https://arxiv.org/abs/2412.12496
  • HF, December 18, 2024, Bamba: Inference-Efficient Hybrid Mamba2 Model, https://huggingface.co/blog/bamba
  • Haoyang Li, Yiming Li, Anxin Tian, Tianhao Tang, Zhanchao Xu, Xuejia Chen, Nicole Hu, Wei Dong, Qing Li, Lei Chen, 27 Dec 2024, A Survey on Large Language Model Acceleration based on KV Cache Management, https://arxiv.org/abs/2412.19442 (Huge survey of all KV cache optimization methods.)
  • Zhenxuan Yu, Yutaka Matsuo, Takeshi Kojima, Yusuke Iwasawa, Jan 2025, Slender-Mamba: Fully Quantized Mamba in 1.58 Bits From Head to Toe, Proceedings of the 31st International Conference on Computational Linguistics, pages 4715–4724, January 19–24, 2025, Association for Computational Linguistics, https://aclanthology.org/2025.coling-main.316.pdf
  • Zukang Xu, Yuxuan Yue, Xing Hu, Zhihang Yuan, Zixu Jiang, Zhixuan Chen, Jiangyong Yu, Chen Xu, Sifan Zhou, Dawei Yang, 23 Jan 2025, MambaQuant: Quantizing the Mamba Family with Variance Aligned Rotation Methods, https://arxiv.org/abs/2501.13484
  • Xiaoran Liu, Ruixiao Li, Mianqiu Huang, Zhigeng Liu, Yuerong Song, Qipeng Guo, Siyang He, Qiqi Wang, Linlin Li, Qun Liu, Yaqian Zhou, Xuanjing Huang, Xipeng Qiu, 24 Feb 2025, Thus Spake Long-Context Large Language Model, https://arxiv.org/abs/2502.17129 (Impressive survey of many techniques to improve efficiency and accuracy of long context processing in both inference and training, covering text, video and multimodal models.)
  • Jiyong Kim, Jaeho Lee, Jiahao Lin, Alish Kanani, Miao Sun, Umit Y. Ogras, and Jaehyun Park, 14 Aug 2025, eMamba: Efficient Acceleration Framework for Mamba Models in Edge Computing, https://arxiv.org/abs/2508.10370
  • Farnoush Bayatmakou, Reza Taleei, Nicole Simone, Arash Mohammadi, 23 Jul 2025, Mammo-Mamba: A Hybrid State-Space and Transformer Architecture with Sequential Mixture of Experts for Multi-View Mammography, https://arxiv.org/abs/2507.17662
  • Hanwen Liu, Yifeng Gong, Zuwei Yan, Zeheng Zhuang, Jiaxuan Lu, 21 Jul 2025, MSGM: A Multi-Scale Spatiotemporal Graph Mamba for EEG Emotion Recognition, https://arxiv.org/abs/2507.15914
  • Osama Hardan, Omar Elshenhabi, Tamer Khattab, Mohamed Mabrok, 15 Jul 2025, Flatten Wisely: How Patch Order Shapes Mamba-Powered Vision for MRI Segmentation, https://arxiv.org/abs/2507.13384
  • Andrew H. Zhang, Alex He-Mo, Richard Fei Yin, Chunlin Li, Yuzhi Tang, Dharmendra Gurve, Veronique van der Horst, Aron S. Buchman, Nasim Montazeri Ghahjaverestan, Maged Goubran, Bo Wang, Andrew S. P. Lim, 9 Aug 2025, Mamba-based Deep Learning Approach for Sleep Staging on a Wireless Multimodal Wearable System without Electroencephalography, https://arxiv.org/abs/2412.15947
  • Ze Rong, ZiYue Zhao, Zhaoxin Wang, Lei Ma, 26 Jul 2025, FaRMamba: Frequency-based learning and Reconstruction aided Mamba for Medical Segmentation, https://arxiv.org/abs/2507.20056
  • Baijiong Lin, Weisen Jiang, Pengguang Chen, Shu Liu, and Ying-Cong Chen, 26 Jul 2025, MTMamba++: Enhancing Multi-Task Dense Scene Understanding via Mamba-Based Decoders, https://arxiv.org/abs/2408.15101
  • Yizhuo Wu, Francesco Fioranelli, Chang Gao, 27 Jul 2025, RadMamba: Efficient Human Activity Recognition through Radar-based Micro-Doppler-Oriented Mamba State-Space Model, https://arxiv.org/abs/2504.12039
  • Aotao Wang, Haikuo Shao, Shaobo Ma, Zhongfeng Wang, 28 Jul 2025, FastMamba: A High-Speed and Efficient Mamba Accelerator on FPGA with Accurate Quantization, https://arxiv.org/abs/2505.18975
  • Hui Liu, Chen Jia, Fan Shi, Xu Cheng, Mengfei Shi, Xia Xie, Shengyong Chen, 31 Jul 2025, LIDAR: Lightweight Adaptive Cue-Aware Fusion Vision Mamba for Multimodal Segmentation of Structural Cracks, https://arxiv.org/abs/2507.22477
  • Alice Zhang, Chao Li, 29 Jul 2025, Adaptive State-Space Mamba for Real-Time Sensor Data Anomaly Detection, https://arxiv.org/abs/2503.22743
  • Jiaxuan Lu, Yuhui Lin, Junyan Shi, Fang Yan, Dongzhan Zhou, Yue Gao, Xiaosong Wang, 4 Aug 2025, Hypergraph Mamba for Efficient Whole Slide Image Understanding, https://arxiv.org/abs/2505.17457
  • Meng Zhou, Farzad Khalvati, 5 Aug 2025, ClinicalFMamba: Advancing Clinical Assessment using Mamba-based Multimodal Neuroimaging Fusion, https://arxiv.org/abs/2508.03008
  • Siyi Lu, Run Liu, Dongsheng Yang, Lei He, 8 Aug 2025, ME$^3$-BEV: Mamba-Enhanced Deep Reinforcement Learning for End-to-End Autonomous Driving with BEV-Perception, https://arxiv.org/abs/2508.06074
  • Kaichuan Kong, Dongjie Liu, Xiaobo Jin, Zhiying Li, Guanggang Geng, Jian Weng, 6 Aug 2025, MambaITD: An Efficient Cross-Modal Mamba Network for Insider Threat Detection, https://arxiv.org/abs/2508.05695
  • Zineddine Bettouche, Khalid Ali, Andreas Fischer, Andreas Kassler, 7 Aug 2025, HiSTM: Hierarchical Spatiotemporal Mamba for Cellular Traffic Forecasting, https://arxiv.org/abs/2508.09184
  • Xi Xuan, Zimo Zhu, Wenxin Zhang, Yi-Cheng Lin, Tomi Kinnunen, 12 Aug 2025, Fake-Mamba: Real-Time Speech Deepfake Detection Using Bidirectional Mamba as Self-Attention's Alternative, https://arxiv.org/abs/2508.09294
  • Honggang Jia, Nan Cheng, Xiucheng Wang, Conghao Zhou, Ruijin Sun, Xuemin (Sherman) Shen, 28 Jul 2025, RadioMamba: Breaking the Accuracy-Efficiency Trade-off in Radio Map Construction via a Hybrid Mamba-UNet, https://arxiv.org/abs/2508.09140
  • Haolong Chen, Liang Zhang, Zhengyuan Xin, Guangxu Zhu, 17 Aug 2025, STM3: Mixture of Multiscale Mamba for Long-Term Spatio-Temporal Time-Series Prediction, https://arxiv.org/abs/2508.12247
  • Jun Zeng, Yannan Huang, Elif Keles, Halil Ertugrul Aktas, Gorkem Durak, Nikhil Kumar Tomar, Quoc-Huy Trinh, Deepak Ranjan Nayak, Ulas Bagci, Debesh Jha, 17 Aug 2025, SRMA-Mamba: Spatial Reverse Mamba Attention Network for Pathological Liver Segmentation in MRI Volumes, https://arxiv.org/abs/2508.12410
  • NVIDIA: Aarti Basant, Abhijit Khairnar, Abhijit Paithankar, Abhinav Khattar, Adi Renduchintala, Adithya Renduchintala, Aditya Malte, Akhiad Bercovich, Akshay Hazare, Alejandra Rico, Aleksander Ficek, Alex Kondratenko, Alex Shaposhnikov, Ali Taghibakhshi, Amelia Barton, Ameya Sunil Mahabaleshwarkar, Amy Shen, Andrew Tao, Ann Guan, Anna Shors, Anubhav Mandarwal, Arham Mehta, Arun Venkatesan, Ashton Sharabiani, Ashwath Aithal, Ashwin Poojary, Ayush Dattagupta, Balaram Buddharaju, Banghua Zhu, Barnaby Simkin, Bilal Kartal, Bita Darvish Rouhani, Bobby Chen, Boris Ginsburg, Brandon Norick, Brian Yu, Bryan Catanzaro, Charles Wang, Charlie Truong, Chetan Mungekar, Chintan Patel, Chris Alexiuk, Christian Munley, Christopher Parisien, Dan Su, Daniel Afrimi, Daniel Korzekwa, Daniel Rohrer, Daria Gitman, et al. (161 additional authors not shown), 20 Aug 2025, NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Model, https://arxiv.org/abs/2508.14444
  • Trinayan Baruah, Kaustubh Shivdikar, Sara Prescott, and David Kaeli, 25 Aug 2025, Characterizing the Behavior of Training Mamba-based State Space Models on GPUs, https://arxiv.org/abs/2508.17679

Knowledge Graph AI Architectures

Knowledge graphs represent structured information as a graph, usually a Directed Acyclic Graph (DAG). This additional structural information can improve LLM results, but it is not easy to integrate graph-structured data into the sequential text sequences expected by an LLM. One particular usage of Knowledge Graphs is to extend RAG architectures, called a "RAG Graph" architecture.

Research papers on Knowledge Graphs in AI include:

  • Shenzhe Zhu, 6 May 2024, Exploring knowledge graph-based neural-symbolic system from application perspective, https://arxiv.org/abs/2405.03524 (Integrate knowledge graph and symbolic reasoning into neural networks.)
  • GG Klager, March 12, 2024, Is GPT fit for KGQA? Masters Thesis, Department of Information Systems & Operations Management, Vienna University of Economics and Business, https://aic.ai.wu.ac.at/~polleres/supervised_theses/Gerhard_Klager_MSc_2024.pdf
  • Louis-François Bouchard, Aug 12, 2024, When to Use GraphRAG, https://louisbouchard.substack.com/p/when-to-use-graphrag
  • Bhaskarjit Sarmah, Benika Hall, Rohan Rao, Sunil Patel, Stefano Pasquali, Dhagash Mehta, 9 Aug 2024, HybridRAG: Integrating Knowledge Graphs and Vector Retrieval Augmented Generation for Efficient Information Extraction, https://arxiv.org/abs/2408.04948
  • Dr. Ashish Bamania, Aug 2024, ‘MedGraphRAG’ Is A Complete Game Changer For AI In Medicine A deep-dive into how RAG, GraphRAG, and MedGraphRAG work and how they significantly improve the performance of LLM responses in Medicine, https://levelup.gitconnected.com/medgraphrag-is-a-complete-game-changer-for-ai-in-medicine-c6b41b0effd6
  • Junde Wu, Jiayuan Zhu, Yunli Qi, 8 Aug 2024, Medical Graph RAG: Towards Safe Medical Large Language Model via Graph Retrieval-Augmented Generation, https://arxiv.org/abs/2408.04187 Code: https://github.com/MedicineToken/Medical-Graph-RAG/tree/main
  • Yuntong Hu, Zhihan Lei, Zheng Zhang, Bo Pan, Chen Ling, Liang Zhao, 26 May 2024, GRAG: Graph Retrieval-Augmented Generation, https://arxiv.org/abs/2405.16506
  • Philip Rathle, Jul 11, 2024, The GraphRAG Manifesto: Adding Knowledge to GenAI, https://neo4j.com/blog/graphrag-manifesto/
  • Microsoft, Aug 2024 (accessed), GraphRAG: A modular graph-based Retrieval-Augmented Generation (RAG) system, https://github.com/microsoft/graphrag
  • Harry Li, Gabriel Appleby, Ashley Suh, 7 Jun 2024, LinkQ: An LLM-Assisted Visual Interface for Knowledge Graph Question-Answering, https://arxiv.org/abs/2406.06621
  • Xuan Chen, Tong Lu, Zhichun Wang, 6 Dec 2024, LLM-Align: Utilizing Large Language Models for Entity Alignment in Knowledge Graphs, https://arxiv.org/abs/2412.04690
  • Lei Liang, Mengshu Sun, Zhengke Gui, Zhongshu Zhu, Zhouyu Jiang, Ling Zhong, Yuan Qu, Peilong Zhao, Zhongpu Bo, Jin Yang, Huaidong Xiong, Lin Yuan, Jun Xu, Zaoyang Wang, Zhiqiang Zhang, Wen Zhang, Huajun Chen, Wenguang Chen, Jun Zhou, 26 Sep 2024 (v3), KAG: Boosting LLMs in Professional Domains via Knowledge Augmented Generation, https://arxiv.org/abs/2409.13731
  • Mayi Xu, Yunfeng Ning, Yongqi Li, Jianhao Chen, Jintao Wen, Yao Xiao, Shen Zhou, Birong Pan, Zepeng Bao, Xin Miao, Hankun Kang, Ke Sun, Tieyun Qian, 2 Jan 2025, Reasoning based on symbolic and parametric knowledge bases: a survey, https://arxiv.org/abs/2501.01030 (Extensive survey of reasoning from CoT to knowledge graphs to table-based reasoning.)
  • Alhassan Mumuni, Fuseini Mumuni, 6 Jan 2025, Large language models for artificial general intelligence (AGI): A survey of foundational principles and approaches, https://arxiv.org/abs/2501.03151
  • Aidan Hogan, Xin Luna Dong, Denny Vrandečić, Gerhard Weikum, 12 Jan 2025, Large Language Models, Knowledge Graphs and Search Engines: A Crossroads for Answering Users' Questions, https://arxiv.org/abs/2501.06699 (Classic search engines versus LLMs with knowledge graphs with a categorization of search use cases.)
  • Tiesunlong Shen, Jin Wang1, Xuejie Zhang, Erik Cambria, Jan 2025, Reasoning with Trees: Faithful Question Answering over Knowledge Graph, Proceedings of the 31st International Conference on Computational Linguistics, pages 3138–3157 January 19–24, 2025, Association for Computational Linguistics, https://aclanthology.org/2025.coling-main.211.pdf
  • Yuxing Lu, Sin Yee Goi, Xukai Zhao, Jinzhuo Wang, 22 Jan 2025 (v2), Biomedical Knowledge Graph: A Survey of Domains, Tasks, and Real-World Applications, https://arxiv.org/abs/2501.11632
  • Maria Korolov, 29 Jan 2025, Knowledge graphs: the missing link in enterprise AI, CIO, https://www.cio.com/article/3808569/knowledge-graphs-the-missing-link-in-enterprise-ai.html
  • Junde Wu, Jiayuan Zhu, Yuyuan Liu, 7 Feb 2025, Agentic Reasoning: Reasoning LLMs with Tools for the Deep Research, https://arxiv.org/abs/2502.04644 https://github.com/theworldofagents/Agentic-Reasoning
  • Pengcheng Huang, Zhenghao Liu, Yukun Yan, Xiaoyuan Yi, Hao Chen, Zhiyuan Liu, Maosong Sun, Tong Xiao, Ge Yu, Chenyan Xiong, 21 Feb 2025, PIP-KAG: Mitigating Knowledge Conflicts in Knowledge-Augmented Generation via Parametric Pruning, https://arxiv.org/abs/2502.15543
  • Han Zhang, Langshi Zhou, Hanfang Yang, 20 Feb 2025, Learning to Retrieve and Reason on Knowledge Graph through Active Self-Reflection, https://arxiv.org/abs/2502.14932
  • Anastasios Nentidis, Charilaos Akasiadis, Angelos Charalambidis, Alexander Artikis, 26 Feb 2025, Dealing with Inconsistency for Reasoning over Knowledge Graphs: A Survey, https://arxiv.org/abs/2502.19023
  • R Chen, Mar 2025, Retrieval-Augmented Generation with Knowledge Graphs: A Survey Computer Science Undergradaute Conference 2025, https://openreview.net/pdf?id=ZikTuGY28C
  • Khorashadizadeh Hanieh, Amara Fatima Zahra, Ezzabady Morteza, Ieng Frédéric, Tiwari Sanju, et al.. Research Trends for the Interplay between Large Language Models and Knowledge Graphs. 1st International Workshop on Data Management Opportunities in Unifying Large Language Models + Knowledge Graph. Workshop at the 50th International Conference on Very Large Data Bases (VLDB 2024), Aug 2024, Guangzhou, China. hal-04770598 https://hal.science/hal-04770598/document
  • Ziheng Zhang, Zhenxi Lin, Yefeng Zheng, and Xian Wu. 2025. How much Medical Knowledge do LLMs have? An Evaluation of Medical Knowledge Coverage for LLMs. In Proceedings of the ACM on Web Conference 2025 (WWW '25). Association for Computing Machinery, New York, NY, USA, 5330–5341. https://doi.org/10.1145/3696410.3714535 https://dl.acm.org/doi/abs/10.1145/3696410.3714535 https://dl.acm.org/doi/pdf/10.1145/3696410.3714535
  • Chuzhan Hao, Wenfeng Feng, Yuewei Zhang, Hao Wang, 23 Jul 2025, DynaSearcher: Dynamic Knowledge Graph Augmented Search Agent via Multi-Reward Reinforcement Learning, https://arxiv.org/abs/2507.17365
  • Qikai Wei and Huansheng Ning and Chunlong Han and Jianguo Ding, 7 Jul 2025, A Query-Aware Multi-Path Knowledge Graph Fusion Approach for Enhancing Retrieval-Augmented Generation in Large Language Models, https://arxiv.org/abs/2507.16826
  • Mingda Zhang, Na Zhao, Jianglong Qin, Guoyu Ye, Ruixiang Tang, 22 Jul 2025, A Multi-granularity Concept Sparse Activation and Hierarchical Knowledge Graph Fusion Framework for Rare Disease Diagnosis, https://arxiv.org/abs/2507.08529
  • Junming Liu, Siyuan Meng, Yanting Gao, Song Mao, Pinlong Cai, Guohang Yan, Yirong Chen, Zilin Bian, Ding Wang, Botian Shi, 24 Jul 2025, Aligning Vision to Language: Annotation-Free Multimodal Knowledge Graph Construction for Enhanced LLMs Reasoning, https://arxiv.org/abs/2503.12972
  • Bhishma Dedhia, Yuval Kansal, Niraj K. Jha, 18 Jul 2025, Bottom-up Domain-specific Superintelligence: A Reliable Knowledge Graph is What We Need, https://arxiv.org/abs/2507.13966
  • Arief Purnama Muharram and Ayu Purwarianti, 21 Jul 2025, Enhancing Natural Language Inference Performance with Knowledge Graph for COVID-19 Automated Fact-Checking in Indonesian Language, https://arxiv.org/abs/2409.00061
  • Xueli Pan, Victor de Boer, Jacco van Ossenbruggen, 14 Aug 2025, FIRESPARQL: A LLM-based Framework for SPARQL Query Generation over Scholarly Knowledge Graphs, https://arxiv.org/abs/2508.10467
  • Rishi Parekh, Saisubramaniam Gopalakrishnan, Zishan Ahmad, Anirudh Deodhar, 23 Jul 2025, Leveraging Knowledge Graphs and LLM Reasoning to Identify Operational Bottlenecks for Warehouse Planning Assistance, https://arxiv.org/abs/2507.17273
  • Aleksandr Perevalov, Andreas Both, 22 Jul 2025, Text-to-SPARQL Goes Beyond English: Multilingual Question Answering Over Knowledge Graphs through Human-Inspired Reasoning, https://arxiv.org/abs/2507.16971
  • Haoran Jiang, Shaohan Shi, Yunjie Yao, Chang Jiang, Quan Li, 23 Jul 2025, HypoChainer: A Collaborative System Combining LLMs and Knowledge Graphs for Hypothesis-Driven Scientific Discovery, https://arxiv.org/abs/2507.17209
  • Jianhao Chen, Junyang Ren, Wentao Ding, Haoyuan Ouyang, Wei Hu, Yuzhong Qu, 23 Jul 2025, Conflict Detection for Temporal Knowledge Graphs:A Fast Constraint Mining Algorithm and New Benchmarks, https://arxiv.org/abs/2312.11053
  • Adrian Kaiser and Claudiu Leoveanu-Condrei and Ryan Gold and Marius-Constantin Dinu and Markus Hofmarcher, 23 Jul 2025, HyDRA: A Hybrid-Driven Reasoning Architecture for Verifiable Knowledge Graphs, https://arxiv.org/abs/2507.15917
  • Jean Lelong, Adnane Errazine and Annabelle Blangero, 22 Jul 2025, Agentic RAG with Knowledge Graphs for Complex Multi-Hop Reasoning in Real-World Applications, https://arxiv.org/abs/2507.16507
  • Mingda Zhang, Na Zhao, Jianglong Qing, Qing xu, Kaiwen Pan, Ting luo, 22 Jul 2025, An Integrated Framework of Prompt Engineering and Multidimensional Knowledge Graphs for Legal Dispute Analysis, https://arxiv.org/abs/2507.07893
  • Yuxin Zhang (1), Xi Wang (1), Mo Hu (1), Zhenyu Zhang (1) ((1) Department of Construction Science, College of Architecture, Texas A&M University, College Station, USA), 18 Jul 2025, BifrostRAG: Bridging Dual Knowledge Graphs for Multi-Hop Question Answering in Construction Safety, https://arxiv.org/abs/2507.13625
  • Nur A Zarin Nishat, Andrea Coletta, Luigi Bellomarini, Kossi Amouzouvi, Jens Lehmann, Sahar Vahdati, 17 Jul 2025, Aligning Knowledge Graphs and Language Models for Factual Accuracy, https://arxiv.org/abs/2507.13411
  • Hosein Azarbonyad, Zi Long Zhu, Georgios Cheirmpos, Zubair Afzal, Vikrant Yadav, Georgios Tsatsaronis, 18 Jul 2025, Question-Answer Extraction from Scientific Articles Using Knowledge Graphs and Large Language Models, https://arxiv.org/abs/2507.13827
  • Akash Bajwa and Chia Jeng Yang, May 27, 2024, The RAG Stack: Featuring Knowledge Graphs: Reducing Hallucinations To Make LLMs Production-Grade With Complex RAG, https://akashbajwa.substack.com/p/the-rag-stack-featuring-knowledge
  • Igor Novikov, Jul 23, 2024, RAG Architecture: Advanced RAG, https://pub.towardsai.net/rag-architecture-advanced-rag-3fea83e0d189
  • Junda Wu, Xintong Li, Ruoyu Wang, Yu Xia, Yuxin Xiong, Jianing Wang, Tong Yu, Xiang Chen, Branislav Kveton, Lina Yao, Jingbo Shang, Julian McAuley, 31 Oct 2024, OCEAN: Offline Chain-of-thought Evaluation and Alignment in Large Language Models, https://arxiv.org/abs/2410.23703
  • Jaikrishna Manojkumar Patil, Nathaniel Lee, Al Mehdi Saadat Chowdhury, YooJung Choi, Paulo Shakarian, 8 Aug 2025, Probabilistic Circuits for Knowledge Graph Completion with Reduced Rule Sets, https://arxiv.org/abs/2508.06706
  • Yongkang Xiao, Rui Zhang, 8 Aug 2025, HERGC: Heterogeneous Experts Representation and Generative Completion for Multimodal Knowledge Graphs, https://arxiv.org/abs/2506.00826
  • Yuzhang Xie, Xu Han, Ran Xu, Xiao Hu, Jiaying Lu, Carl Yang, 26 Jul 2025, HypKG: Hypergraph-based Knowledge Graph Contextualization for Precision Healthcare, https://arxiv.org/abs/2507.19726
  • Alec Scully, Cameron Stockton, and Forrest Hare, 26 Jul 2025, Integrating Activity Predictions in Knowledge Graphs, https://arxiv.org/abs/2507.19733
  • Keyan Ding, Jing Yu, Junjie Huang, Yuchen Yang, Qiang Zhang, Huajun Chen, 27 Jul 2025, SciToolAgent: A Knowledge Graph-Driven Scientific Agent for Multi-Tool Integration, https://arxiv.org/abs/2507.20280
  • Jiajun Liu, Wenjun Ke, Peng Wang, Yao He, Ziyu Shang, Guozheng Li, Zijie Xu, and Ke Ji, 28 Jul 2025, Unlearning of Knowledge Graph Embedding via Preference Optimization, https://arxiv.org/abs/2507.20566
  • Lijian Li, 28 Jul 2025, Complementarity-driven Representation Learning for Multi-modal Knowledge Graph Completion, https://arxiv.org/abs/2507.20620
  • Xueyao Wan, Hang Yu, 28 Jul 2025, MMGraphRAG: Bridging Vision and Language with Interpretable Multimodal Knowledge Graphs, https://arxiv.org/abs/2507.20804
  • Enjun Du, Siyi Liu, Yongqi Zhang, 28 Jul 2025, Mixture of Length and Pruning Experts for Knowledge Graphs Reasoning, https://arxiv.org/abs/2507.20498
  • Wenbin Guo, Xin Wang, Jiaoyan Chen, Zhao Li and Zirui Chen, 28 Jul 2025, Ontology-Enhanced Knowledge Graph Completion using Large Language Models, https://arxiv.org/abs/2507.20643
  • Muhammad Tayyab Khan, Lequn Chen, Wenhe Feng and Seung Ki Moon, 28 Jul 2025, Large Language Model Powered Decision Support for a Metal Additive Manufacturing Knowledge Graph, https://arxiv.org/abs/2505.20308
  • Hao Ye, Mengshi Qi, Zhaohong Liu, Liang Liu and Huadong Ma, 29 Jul 2025, SafeDriveRAG: Towards Safe Autonomous Driving with Knowledge Graph-based Retrieval-Augmented Generation, https://arxiv.org/abs/2507.21585
  • Alessandro Lonardi and Samy Badreddine and Tarek R. Besold and Pablo Sanchez Martin, 29 Jul 2025, Unifying Post-hoc Explanations of Knowledge Graph Completions, https://arxiv.org/abs/2507.22951
  • Nasim Shirvani-Mahdavi, Devin Wingfield, Amin Ghasemi, Chengkai Li, 31 Jul 2025, Rule2Text: Natural Language Explanation of Logical Rules in Knowledge Graphs, https://arxiv.org/abs/2507.23740
  • Jiaxin Bai, Wei Fan, Qi Hu, Qing Zong, Chunyang Li, Hong Ting Tsang, Hongyu Luo, Yauwai Yim, Haoyu Huang, Xiao Zhou, Feng Qin, Tianshi Zheng, Xi Peng, Xin Yao, Huiwen Yang, Leijie Wu, Yi Ji, Gong Zhang, Renhai Chen, Yangqiu Song, 31 Jul 2025, AutoSchemaKG: Autonomous Knowledge Graph Construction through Dynamic Schema Induction from Web-Scale Corpora, https://arxiv.org/abs/2505.23628
  • Tung-Wei Lin, Gabe Fierro, Han Li, Tianzhen Hong, Pierluigi Nuzzo, Alberto Sangiovanni-Vinentelli, 30 Jul 2025, Systematic Evaluation of Knowledge Graph Repair with Large Language Models, https://arxiv.org/abs/2507.22419
  • Thanh Hoang-Minh, 30 Jul 2025, Graph Collaborative Attention Network for Link Prediction in Knowledge Graphs, https://arxiv.org/abs/2507.03947
  • Antonis Klironomos, Baifan Zhou, Zhipeng Tan, Zhuoxun Zheng, Mohamed H. Gad-Elrab, Heiko Paulheim, Evgeny Kharlamov, 1 Aug 2025, ExeKGLib: A Platform for Machine Learning Analytics based on Knowledge Graphs, https://arxiv.org/abs/2508.00394
  • Yuanyuan Liang, Xiaoman Wang, Tingyu Xie, and Lei Pan, 3 Aug 2025, ProKG-Dial: Progressive Multi-Turn Dialogue Construction with Domain Knowledge Graphs, https://arxiv.org/abs/2508.01869
  • Hanchen Yang, Jiaqi Wang, Jiannong Cao, Wengen Li, Jialun Zheng, Yangning Li, Chunyu Miao, Jihong Guan, Shuigeng Zhou, and Philip S. Yu, 31 Jul 2025, OKG-LLM: Aligning Ocean Knowledge Graph with Observation Data via LLMs for Global Sea Surface Temperature Prediction, https://arxiv.org/abs/2508.00933
  • Xiang Li, Penglei Sun, Wanyun Zhou, Zikai Wei, Yongqi Zhang, Xiaowen Chu, 1 Aug 2025, FinKario: Event-Enhanced Automated Construction of Financial Knowledge Graph, https://arxiv.org/abs/2508.00961
  • Wei Zhou, Peng Sun, Xuanhe Zhou, Qianglei Zang, Ji Xu, Tieying Zhang, Guoliang Li, Fan Wu, 2 Aug 2025, DBAIOps: A Reasoning LLM-Enhanced Database Operation and Maintenance System using Knowledge Graphs, https://arxiv.org/abs/2508.01136
  • Yang Zhao, Chengxiao Dai, Wei Zhuo, Tan Chuan Fu, Yue Xiu, Dusit Niyato, Jonathan Z. Low, Eugene Ho Hong Zhuang, Daren Zong Loong Tan, 3 Aug 2025, AGENTICT$^2$S:Robust Text-to-SPARQL via Agentic Collaborative Reasoning over Heterogeneous Knowledge Graphs for the Circular Economy, https://arxiv.org/abs/2508.01815
  • Linyu Li, Zhi Jin, Yuanpeng He, Dongming Jin, Yichi Zhang, Haoran Duan, Nyima Tash, 4 Aug 2025, Learning to Evolve: Bayesian-Guided Continual Knowledge Graph Embedding, https://arxiv.org/abs/2508.02426
  • Xinjie Zhao, Moritz Blum, Fan Gao, Yingjian Chen, Boming Yang, Luis Marquez-Carpintero, M\'onica Pina-Navarro, Yanran Fu, So Morikawa, Yusuke Iwasawa, Yutaka Matsuo, Chanjun Park, Irene Li, 5 Aug 2025, AGENTiGraph: A Multi-Agent Knowledge Graph Framework for Interactive, Domain-Specific LLM Chatbots, https://arxiv.org/abs/2508.02999
  • Taine J. Elliott, Stephen P. Levitt, Ken Nixon and Martin Bekker, 5 Aug 2025, Data Overdose? Time for a Quadruple Shot: Knowledge Graph Construction using Enhanced Triple Extraction, https://arxiv.org/abs/2508.03438
  • Yubo Wang, Shimin Di, Zhili Wang, Haoyang Li, Fei Teng, Hao Xin and Lei Chen, 5 Aug 2025, Understanding the Embedding Models on Hyper-relational Knowledge Graph, https://arxiv.org/abs/2508.03280
  • Ge Shi, Kaiyu Huang, Guochen Feng, 5 Aug 2025, Long Story Generation via Knowledge Graph and Literary Theory, https://arxiv.org/abs/2508.03137
  • Futian Wang, Yuhan Qiao, Xiao Wang, Fuling Wang, Yuxiang Zhang, Dengdi Sun, 5 Aug 2025, R2GenKG: Hierarchical Multi-modal Knowledge Graph for LLM-based Radiology Report Generation, https://arxiv.org/abs/2508.03426
  • Nandana Mihindukulasooriya, Niharika S. D'Souza, Faisal Chowdhury, Horst Samulowitz, 4 Aug 2025, Automatic Prompt Optimization for Knowledge Graph Construction: Insights from an Empirical Study, https://arxiv.org/abs/2506.19773
  • Hudson de Martim, 5 Aug 2025, A Foundational Schema.org Mapping for a Legal Knowledge Graph: Representing Brazilian Legal Norms as FRBR Works, https://arxiv.org/abs/2508.00827
  • Ruochen Zhao, Simone Conia, Eric Peng, Min Li, Saloni Potdar, 6 Aug 2025, AgREE: Agentic Reasoning for Knowledge Graph Completion on Emerging Entities, https://arxiv.org/abs/2508.04118
  • Qian Yong, Yanhui Li, Jialiang Shi, Yaguang Dou, Tian Qi, 6 Aug 2025, Enhancing Serendipity Recommendation System by Constructing Dynamic User Knowledge Graphs with Large Language Models, https://arxiv.org/abs/2508.04032
  • Krzysztof Olejniczak, Xingyue Huang, Mikhail Galkin, \.Ismail \.Ilkan Ceylan, 5 Aug 2025, One Model, Any Conjunctive Query: Graph Neural Networks for Answering Queries over Incomplete Knowledge Graphs, https://arxiv.org/abs/2409.13959
  • Ge Chang, Jinbo Su, Jiacheng Liu, Pengfei Yang, Yuhao Shang, Huiwen Zheng, Hongli Ma, Yan Liang, Yuanchun Li, Yunxin Liu, 7 Aug 2025, GRAIL:Learning to Interact with Large Knowledge Graphs for Retrieval Augmented Reasoning, https://arxiv.org/abs/2508.05498
  • Claudia d'Amato, Ivan Diliso, Nicola Fanizzi, Zafar Saeed, 7 Aug 2025, Enhancing PyKEEN with Multiple Negative Sampling Solutions for Knowledge Graph Embedding Models, https://arxiv.org/abs/2508.05587
  • Xu Yuan, Liangbo Ning, Wenqi Fan, Qing Li, 7 Aug 2025, mKG-RAG: Multimodal Knowledge Graph-Enhanced RAG for Visual Question Answering, https://arxiv.org/abs/2508.05318
  • Claudia dAmato, Giuseppe Rubini, Francesco Didio, Donato Francioso, Fatima Zahra Amara, Nicola Fanizzi, 8 Aug 2025, Automated Creation of the Legal Knowledge Graph Addressing Legislation on Violence Against Women: Resource, Methodology and Lessons Learned, https://arxiv.org/abs/2508.06368
  • Siamak Farshidi and Amir Saberhabibi and Behbod Eskafi and Niloofar Nikfarjam and Sadegh Eskandari and Slinger Jansen and Michel Chaudron and Bedir Tekinerdogan, 6 Aug 2025, Empirical Evaluation of AI-Assisted Software Package Selection: A Knowledge Graph Approach, https://arxiv.org/abs/2508.05693
  • Congmin Min, Rhea Mathew, Joyce Pan, Sahil Bansal, Abbas Keshavarzi, Amar Viswanathan Kannan, 7 Aug 2025, Efficient Knowledge Graph Construction and Retrieval from Unstructured Text for Large-Scale RAG Systems, https://arxiv.org/abs/2507.03226
  • Dongzhuoran Zhou, Yuqicheng Zhu, Xiaxia Wang, Hongkuan Zhou, Yuan He, Jiaoyan Chen, Evgeny Kharlamov, Steffen Staab, 11 Aug 2025, What Breaks Knowledge Graph based RAG? Empirical Insights into Reasoning under Incomplete Knowledge, https://arxiv.org/abs/2508.08344
  • Roberto Barile, Claudia d'Amato, Nicola Fanizzi, 12 Aug 2025, GRainsaCK: a Comprehensive Software Library for Benchmarking Explanations of Link Prediction Tasks on Knowledge Graphs, https://arxiv.org/abs/2508.08815
  • Bhavik Agarwal, Hemant Sunil Jomraj, Simone Kaplunov, Jack Krolick, Viktoria Rojkova, 13 Aug 2025, RAGulating Compliance: A Multi-Agent Knowledge Graph for Regulatory QA, https://arxiv.org/abs/2508.09893
  • Yuheng Wang, Tianze Yu, Jiayue Cai, Sunil Kalia, Harvey Lui, Z. Jane Wang, Tim K. Lee, 13 Aug 2025, Integrating Clinical Knowledge Graphs and Gradient-Based Neural Systems for Enhanced Melanoma Diagnosis via the 7-Point Checklist, https://arxiv.org/abs/2407.16822
  • Yifei Li, Lingling Zhang, Hang Yan, Tianzhe Zhao, Zihan Ma, Muye Huang, Jun Liu, 15 Aug 2025, SAGE: Scale-Aware Gradual Evolution for Continual Knowledge Graph Embedding, https://arxiv.org/abs/2508.11347
  • Nasim Shirvani-Mahdavi, Chengkai Li, 14 Aug 2025, Rule2Text: A Framework for Generating and Evaluating Natural Language Explanations of Knowledge Graph Rules, https://arxiv.org/abs/2508.10971
  • Duzhen Zhang, Zixiao Wang, Zhong-Zhi Li, Yahan Yu, Shuncheng Jia, Jiahua Dong, Haotian Xu, Xing Wu, Yingying Zhang, Tielin Zhang, Jie Yang, Xiuying Chen, Le Song, 17 Aug 2025, MedKGent: A Large Language Model Agent Framework for Constructing Temporally Evolving Medical Knowledge Graph, https://arxiv.org/abs/2508.12393
  • Ziteng Hu, Yingjie Xia, Xiyuan Chen, Li Kuang, 18 Aug 2025, SecFSM: Knowledge Graph-Guided Verilog Code Generation for Secure Finite State Machines in Systems-on-Chip, https://arxiv.org/abs/2508.12910
  • Hung Nghiep Tran, Atsuhiro Takasu, 15 Aug 2025, Exploring Scholarly Data by Semantic Query on Knowledge Graph Embedding Space, https://arxiv.org/abs/1909.08191
  • Daniel Daza, Alberto Bernardi, Luca Costabello, Christophe Gueret, Masoud Mansoury, Michael Cochez, Martijn Schut, 19 Aug 2025, Interactive Query Answering on Knowledge Graphs with Soft Entity Constraints, https://arxiv.org/abs/2508.13663
  • Mariam Arustashvili, J\"org Deigm\"oller, Heiko Paulheim, 19 Aug 2025, Knowledge Graph Completion for Action Prediction on Situational Graphs -- A Case Study on Household Tasks, https://arxiv.org/abs/2508.13675
  • Yang Xiao, Ruimeng Ye, Bohan Liu, Xiaolong Ma, Bo Hui, 19 Aug 2025, Efficient Knowledge Graph Unlearning with Zeroth-order Information, https://arxiv.org/abs/2508.14013
  • Peilin Ji, Xiao Xue, Simeng Wang, Wenhao Yan, 20 Aug 2025, Entropy-Constrained Strategy Optimization in Urban Floods: A Multi-Agent Framework with LLM and Knowledge Graph Integration, https://arxiv.org/abs/2508.14654
  • Dennis Schiese, Aleksandr Perevalov, Andreas Both, 20 Aug 2025, Towards LLM-generated explanations for Component-based Knowledge Graph Question Answering Systems, https://arxiv.org/abs/2508.14553
  • Haji Gul, Abul Ghani Naim, Ajaz Ahmad Bhat, 21 Aug 2025, Evaluating Knowledge Graph Complexity via Semantic, Spectral, and Structural Metrics for Link Prediction, https://arxiv.org/abs/2508.15291
  • Runxuan Liu, Bei Luo, Jiaqi Li, Baoxin Wang, Ming Liu, Dayong Wu, Shijin Wang, Bing Qin, 21 Aug 2025, Ontology-Guided Reverse Thinking Makes Large Language Models Stronger on Knowledge Graph Question Answering, https://arxiv.org/abs/2502.11491
  • Nan Wang, Yongqi Fan, yansha zhu, ZongYu Wang, Xuezhi Cao, Xinyan He, Haiyun Jiang, Tong Ruan, Jingping Liu, 12 Aug 2025, KG-o1: Enhancing Multi-hop Question Answering in Large Language Models via Knowledge Graph Integration, https://arxiv.org/abs/2508.15790
  • Ryoma Kondo, Riona Matsuoka, Takahiro Yoshida, Kazuyuki Yamasawa, Ryohei Hisano, 24 Aug 2025, Capturing Legal Reasoning Paths from Facts to Law in Court Judgments using Knowledge Graphs, https://arxiv.org/abs/2508.17340
  • Yitong Lin, Jiaying He, Jiahe Chen, Xinnan Zhu, Jianwei Zheng, Tao Bo, 22 Jul 2025, BioGraphFusion: Graph Knowledge Embedding for Biological Completion and Reasoning, https://arxiv.org/abs/2507.14468
  • Jialiang Wang, Hanmo Liu, Shimin Di, Zhili Wang, Jiachuan Wang, Lei Chen, Xiaofang Zhou, 21 Jul 2025, Proficient Graph Neural Network Design by Accumulating Knowledge on Large Language Models, https://arxiv.org/abs/2408.06717
  • Mubaris Nadeem, Johannes Zenkert, Lisa Bender, Christian Weber, Madjid Fathi, 11 Aug 2025, KIRETT: Knowledge-Graph-Based Smart Treatment Assistant for Intelligent Rescue Operations, https://arxiv.org/abs/2508.07834
  • Yaoze Zhang, Rong Wu, Pinlong Cai, Xiaoman Wang, Guohang Yan, Song Mao, Ding Wang, Botian Shi, 14 Aug 2025, LeanRAG: Knowledge-Graph-Based Generation with Semantic Aggregation and Hierarchical Retrieval, https://arxiv.org/abs/2508.10391
  • Robert Frenken, Sidra Ghayour Bhatti, Hanqin Zhang, Qadeer Ahmed, 25 Jul 2025, KD-GAT: Combining Knowledge Distillation and Graph Attention Transformer for a Controller Area Network Intrusion Detection System, https://arxiv.org/abs/2507.19686
  • Zhaoyan Wang, Hyunjun Ahn, In-Young Ko, 28 Jul 2025, Beyond Interactions: Node-Level Graph Generation for Knowledge-Free Augmentation in Recommender Systems, https://arxiv.org/abs/2507.20578
  • Zhen Wu, Ritam Dutt, Luke M. Breitfeller, Armineh Nourbakhsh, Siddharth Parekh, Carolyn Ros\'e, 2 Aug 2025, $R^2$-CoD: Understanding Text-Graph Complementarity in Relational Reasoning via Knowledge Co-Distillation, https://arxiv.org/abs/2508.01475
  • Jiayi Wen, Tianxin Chen, Zhirun Zheng, Cheng Huang, 6 Aug 2025, A Few Words Can Distort Graphs: Knowledge Poisoning Attacks on Graph-based Retrieval-Augmented Generation of Large Language Models, https://arxiv.org/abs/2508.04276
  • Zhu Xu, Ting Lei, Zhimin Li, Guan Wang, Qingchao Chen, Yuxin Peng, Yang liu, 7 Aug 2025, TRKT: Weakly Supervised Dynamic Scene Graph Generation with Temporal-enhanced Relation-aware Knowledge Transferring, https://arxiv.org/abs/2508.04943
  • Daniel Airinei, Elena Burceanu, Marius Leordeanu, 15 Aug 2025, Inside Knowledge: Graph-based Path Generation with Explainable Data Augmentation and Curriculum Learning for Visual Indoor Navigation, https://arxiv.org/abs/2508.11446
  • Bowen Wang, Zhouqiang Jiang, Yasuaki Susumu, Shotaro Miwa, Tianwei Chen, Yuta Nakashima, 25 Aug 2025, Taming the Untamed: Graph-Based Knowledge Retrieval and Reasoning for MLLMs to Conquer the Unknown, https://arxiv.org/abs/2506.17589

Graph Neural Networks

Research papers on Graph Neural Networks (GNNs):

  • Zhichun Guo, April 2024, Empowering Graph Neural Networks for Real-World Tasks, Ph.D. Thesis, Computer Science and Engineering, University of Notre Dame, Indiana, https://doi.org/10.7274/25608504.v1 https://curate.nd.edu/articles/dataset/Empowering_Graph_Neural_Networks_for_Real-World_Tasks/25608504/1 PDF: https://curate.nd.edu/ndownloader/files/46035312/1
  • Lu Ma, Zeang Sheng, Xunkai Li, Xinyi Gao, Zhezheng Hao, Ling Yang, Wentao Zhang, Bin Cui, 7 May 2024, Acceleration Algorithms in GNNs: A Survey, https://arxiv.org/abs/2405.04114
  • Yun Zhu, Yaoke Wang, Haizhou Shi, Siliang Tang, 28 Jan 2024, Efficient Tuning and Inference for Large Language Models on Textual Graphs, https://arxiv.org/abs/2401.15569 (Optimizing Graph Neural Networks on textual graphs using caching and early exit inference.)
  • Sebastian Eliassen, Raghavendra Selvan, 16 Jan 2024 (v2), Activation Compression of Graph Neural Networks using Block-wise Quantization with Improved Variance Minimization, https://arxiv.org/abs/2309.11856
  • Weishu Deng, Jia Rao, 2024, Mega: More Efficient Graph Attention for GNNs, 2024 IEEE 44th International Conference on Distributed Computing Systems (ICDCS), Year: 2024, Pages: 71-81, DOI Bookmark: 10.1109/ICDCS60910.2024.00016, https://www.computer.org/csdl/proceedings-article/icdcs/2024/860500a071/1ZCgMaVLfRm
  • Hongrong Cheng, Miao Zhang, Javen Qinfeng Shi, 9 Aug 2024 (v2), A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations, IEEE Transactions on Pattern Analysis and Machine Intelligence, doi: 10.1109/TPAMI.2024.3447085, https://arxiv.org/abs/2308.06767 https://ieeexplore.ieee.org/abstract/document/10643325
  • Lena Sasal, Daniel Busby, Abdenour Hadid, 29 Aug 2024, TempoKGAT: A Novel Graph Attention Network Approach for Temporal Graph Analysis, https://arxiv.org/abs/2408.16391
  • Xinke Jiang, Rihong Qiu, Yongxin Xu, Wentao Zhang, Yichen Zhu, Ruizhe Zhang, Yuchen Fang, Xu Chu, Junfeng Zhao, Yasha Wang, 31 Oct 2024, RAGraph: A General Retrieval-Augmented Graph Learning Framework, https://arxiv.org/abs/2410.23855
  • V. Slavin, O. Kryvchikov, D. Laptev, 23 Jul 2025, Graph Neural Network Approach to Predicting Magnetization in Quasi-One-Dimensional Ising Systems, https://arxiv.org/abs/2507.17509
  • Yuelin Wang, Kai Yi, Xinliang Liu, Yu Guang Wang, Shi Jin, 23 Jul 2025, ACMP: Allen-Cahn Message Passing with Attractive and Repulsive Forces for Graph Neural Networks, https://arxiv.org/abs/2206.05437
  • Ana Gonzalez Bermudez, Miquel Farreras, Milan Groshev, Jos\'e Antonio Trujillo, Isabel de la Bandera and Raquel Barco, 23 Jul 2025, Graph Neural Networks for O-RAN Mobility Management: A Link Prediction Approach, https://arxiv.org/abs/2502.02170
  • Yumeng Wang, Zengyi Wo, Wenjun Wang, Xingcheng Fu, Minglai Shao, 22 Jul 2025, Leveraging Personalized PageRank and Higher-Order Topological Structures for Heterophily Mitigation in Graph Neural Networks, https://arxiv.org/abs/2507.16347
  • Olga Solodova, Nick Richardson, Deniz Oktay, Ryan P. Adams, 22 Jul 2025, Graph Neural Networks Gone Hogwild, https://arxiv.org/abs/2407.00494
  • Zihao Song, Shirantha Welikala, Panos J. Antsaklis and Hai Lin, 22 Jul 2025, Graph Neural Network-Based Distributed Optimal Control for Linear Networked Systems: An Online Distributed Training Approach, https://arxiv.org/abs/2504.06439
  • Jai Bardhan, Tanumoy Mandal, Subhadip Mitra, Cyrin Neeraj, Mihir Rawat, 22 Jul 2025, Tagging fully hadronic exotic decays of the vectorlike $\mathbf{B}$ quark using a graph neural network, https://arxiv.org/abs/2505.07769
  • Lijun Wu, Dong Hao, Zhiyi Fan, 19 Jul 2025, Explainable Graph Neural Networks via Structural Externalities, https://arxiv.org/abs/2507.17848
  • Ahmad ALBarqawi, Mahmoud Nazzal, Issa Khalil, Abdallah Khreishah, NhatHai Phan, 24 Jul 2025, ViGText: Deepfake Image Detection with Vision-Language Model Explanations and Graph Neural Networks, https://arxiv.org/abs/2507.18031
  • Xinran Li, Xiujuan Xu, Jiaqi Qiao, 24 Jul 2025, Long-Short Distance Graph Neural Networks and Improved Curriculum Learning for Emotion Recognition in Conversation, https://arxiv.org/abs/2507.15205
  • Hao Ai and Yu-xi Liu, 24 Jul 2025, Scalable Parameter Design for Superconducting Quantum Circuits with Graph Neural Networks, https://arxiv.org/abs/2411.16354
  • Guanyuan Pan, Tiansheng Zhou, Bingtao Ma, Yaqi Wang, Jianxiang Zhao, Zhi Li, Yugui Lin, Pietro Lio, Shuai Wang, 24 Jul 2025, GNN-ACLP: Graph Neural Networks Based Analog Circuit Link Prediction, https://arxiv.org/abs/2504.10240
  • Edward Henderson, Dewi Gould, Richard Everson, George De Ath, Nick Pepper, 17 Jul 2025, Air Traffic Controller Task Demand via Graph Neural Networks: An Interpretable Approach to Airspace Complexity, https://arxiv.org/abs/2507.13423
  • Yifan Wei, Anwar Said, Waseem Abbas, Xenofon Koutsoukos, 18 Jul 2025, Robust Anomaly Detection with Graph Neural Networks using Controllability, https://arxiv.org/abs/2507.13954
  • Jagruti Patel, Thomas A. W. Bolton, Mikkel Sch\"ottner, Anjali Tarun, Sebastien Tourbier, Yasser Alem\`an-G\`omez, Jonas Richiardi, Patric Hagmann, 18 Jul 2025, Structural Connectome Harmonization Using Deep Learning: The Strength of Graph Neural Networks, https://arxiv.org/abs/2507.13992
  • Yeming Cai, Zhenglin Li, Yang Wang, 11 Jul 2025, Enhancing Breast Cancer Detection with Vision Transformers and Graph Neural Networks, https://arxiv.org/abs/2507.13372
  • Vijay K. Dubey (1), Collin E. Haese (1), Osman G\"ultekin (1), David Dalton (2), Manuel K. Rausch (1), Jan N. Fuhg (1) ((1) The University of Texas at Austin, (2) University of Glasgow), 17 Jul 2025, Graph Neural Network Surrogates for Contacting Deformable Bodies with Necessary and Sufficient Contact Detection, https://arxiv.org/abs/2507.13459
  • Srinitish Srinivasan and Omkumar CU, 18 Jul 2025, Can we ease the Injectivity Bottleneck on Lorentzian Manifolds for Graph Neural Networks?, https://arxiv.org/abs/2504.00142
  • Xu Cheng, Liang Yao, Feng He, Yukuo Cen, Yufei He, Chenhui Zhang, Wenzheng Feng, Hongyun Cai, Jie Tang, 19 Jul 2025, LPS-GNN : Deploying Graph Neural Networks on Graphs with 100-Billion Edges, https://arxiv.org/abs/2507.14570
  • Rabia Latief Bhat and Iqra Altaf Gillani, 21 Jul 2025, Spatio-Temporal Demand Prediction for Food Delivery Using Attention-Driven Graph Neural Networks, https://arxiv.org/abs/2507.15246
  • Yufei Jin and Xingquan Zhu, 18 Jul 2025, Oversmoothing Alleviation in Graph Neural Networks: A Survey and Unified View, https://arxiv.org/abs/2405.01663
  • Jialiang Wang, Hanmo Liu, Shimin Di, Zhili Wang, Jiachuan Wang, Lei Chen, Xiaofang Zhou, 21 Jul 2025, Proficient Graph Neural Network Design by Accumulating Knowledge on Large Language Models, https://arxiv.org/abs/2408.06717
  • Zizhou Zhang, Qinyan Shen, Zhuohuan Hu, Qianying Liu, Huijie Shen, 20 Jul 2025, Credit Risk Analysis for SMEs Using Graph Neural Networks in Supply Chain, https://arxiv.org/abs/2507.07854
  • Peyman Baghershahi, Gregoire Fournier, Pranav Nyati, Sourav Medya, 9 Aug 2025, From Nodes to Narratives: Explaining Graph Neural Networks with LLMs and Graph Context, https://arxiv.org/abs/2508.07117
  • Zhihao Xue, Yun Zi, Nia Qi, Ming Gong, Yujun Zou, 9 Aug 2025, Multi-Level Service Performance Forecasting via Spatiotemporal Graph Neural Networks, https://arxiv.org/abs/2508.07122
  • Tiantian Yang, Zhiqian Chen, 10 Aug 2025, MOTGNN: Interpretable Graph Neural Networks for Multi-Omics Disease Classification, https://arxiv.org/abs/2508.07465
  • Rahul Khorana, 11 Aug 2025, Topological Feature Compression for Molecular Graph Neural Networks, https://arxiv.org/abs/2508.07807
  • Bowen Zhang, Genan Dai, Hu Huang, Long Lan, 9 Aug 2025, Geometry-Aware Spiking Graph Neural Network, https://arxiv.org/abs/2508.06793
  • Morteza Ziabakhsh, Kiyan Rezaee, Sadegh Eskandari, Seyed Amir Hossein Tabatabaei, Mohammad M. Ghassemi, 10 Aug 2025, Extracting Overlapping Microservices from Monolithic Code via Deep Semantic Embeddings and Graph Neural Network-Based Soft Clustering, https://arxiv.org/abs/2508.07486
  • Sujia Huang, Lele Fu, Zhen Cui, Tong Zhang, Na Song, Bo Huang, 29 Jul 2025, Torque-based Graph Surgery:Enhancing Graph Neural Networks with Hierarchical Rewiring, https://arxiv.org/abs/2507.21422
  • Mustapha Hemis, Hamza Kheddar, Mohamed Chahine Ghanem, Bachir Boudraa, 29 Jul 2025, Hierarchical Graph Neural Network for Compressed Speech Steganalysis, https://arxiv.org/abs/2507.21591
  • Zhanhong Cheng, Lingqian Hu, Yuheng Bu, Yuqi Zhou, Shenhao Wang, 28 Jul 2025, Graph neural networks for residential location choice: connection to classical logit models, https://arxiv.org/abs/2507.21334
  • Haolin Li, Haoyu Wang, Luana Ruiz, 30 Jul 2025, Graph Sampling for Scalable and Expressive Graph Neural Networks on Homophilic Graphs, https://arxiv.org/abs/2410.16593
  • Shuyang Guo, Wenjin Xie, Ping Lu, Ting Deng, Richong Zhang, Jianxin Li, Xiangping Huang, Zhongyi Liu, 27 Jul 2025, Improving Subgraph Matching by Combining Algorithms and Graph Neural Networks, https://arxiv.org/abs/2507.20226
  • Mohit Gupta, Debjit Bhowmick, Ben Beck, 18 Jul 2025, BikeVAE-GNN: A Variational Autoencoder-Augmented Hybrid Graph Neural Network for Sparse Bicycle Volume Estimation, https://arxiv.org/abs/2507.19517
  • Yihan Wang, Jianing Zhao, 20 Jul 2025, Research on the application of graph data structure and graph neural network in node classification/clustering tasks, https://arxiv.org/abs/2507.19527
  • Yazeed Alrubyli, Omar Alomeir, Abrar Wafa, Di\'ana Hidv\'egi, Hend Alrasheed, Mohsen Bahrami, 25 Jul 2025, NAICS-Aware Graph Neural Networks for Large-Scale POI Co-visitation Prediction: A Multi-Modal Dataset and Methodology, https://arxiv.org/abs/2507.19697
  • Kunhao Li, Di Wu, Jun Bai, Jing Xu, Lei Yang, Ziyi Zhang, Yiliao Song, Wencheng Yang, Taotao Cai, Yan Li, 26 Jul 2025, Who Owns This Sample: Cross-Client Membership Inference Attack in Federated Graph Neural Networks, https://arxiv.org/abs/2507.19964
  • Vicente Ramos (1), Sundous Hussein (1), Mohamed Abdel-Hafiz (1), Arunangshu Sarkar (2), Weixuan Liu (2), Katerina J. Kechris (2), Russell P. Bowler (3), Leslie Lange (4), Farnoush Banaei-Kashani (1) ((1) Department of Computer Science and Engineering, University of Colorado Denver, Denver, USA, (2) Department of Biostatistics and Informatics, University of Colorado Anschutz Medical Campus, Aurora, USA, (3) Genomic Medicine Institute, Cleveland Clinic, Cleveland, USA, (4) Division of Biomedical Informatics and Personalized Medicine, University of Colorado Anschutz Medical Campus, Aurora, USA), 27 Jul 2025, BioNeuralNet: A Graph Neural Network based Multi-Omics Network Data Analysis Tool, https://arxiv.org/abs/2507.20440
  • Yongzheng Liu, Yiming Wang, Po Xu, Yingjie Xu, Yuntian Chen, Dongxiao Zhang, 28 Jul 2025, BuildSTG: A Multi-building Energy Load Forecasting Method using Spatio-Temporal Graph Neural Network, https://arxiv.org/abs/2507.20838
  • Bernardo Cuenca Grau, Eva Feng, Przemys{\l}aw A. Wa{\l}\k{e}ga, 26 Jul 2025, The Correspondence Between Bounded Graph Neural Networks and Fragments of First-Order Logic, https://arxiv.org/abs/2505.08021
  • Miguel Lopez-Duran, Julian Fierrez, Aythami Morales, Ruben Tolosana, Oscar Delgado-Mohatar, Alvaro Ortigosa, 28 Jul 2025, Benchmarking Graph Neural Networks for Document Layout Analysis in Public Affairs, https://arxiv.org/abs/2505.14699
  • Eran Rosenbluth and Martin Grohe, 30 Jul 2025, Repetition Makes Perfect: Recurrent Graph Neural Networks Match Message Passing Limit, https://arxiv.org/abs/2505.00291
  • Tong Nie, Jian Sun, Wei Ma, 31 Jul 2025, Predicting Large-scale Urban Network Dynamics with Energy-informed Graph Neural Diffusion, https://arxiv.org/abs/2508.00037
  • Mohit Gupta, Debjit Bhowmick, Rhys Newbury, Meead Saberi, Shirui Pan and Ben Beck, 31 Jul 2025, INSPIRE-GNN: Intelligent Sensor Placement to Improve Sparse Bicycling Network Prediction via Reinforcement Learning Boosted Graph Neural Networks, https://arxiv.org/abs/2508.00141
  • Yoonhyuk Choi, Jiho Choi, Chong-Kwon Kim, 1 Aug 2025, Sheaf Graph Neural Networks via PAC-Bayes Spectral Optimization, https://arxiv.org/abs/2508.00357
  • Mukesh Kumar Sahu and Pinki Roy, 1 Aug 2025, Similarity-Based Self-Construct Graph Model for Predicting Patient Criticalness Using Graph Neural Networks and EHR Data, https://arxiv.org/abs/2508.00615
  • Molly Noel, Gabriel Mancino-Ball, Yangyang Xu, 1 Aug 2025, Neighbor-Sampling Based Momentum Stochastic Methods for Training Graph Neural Networks, https://arxiv.org/abs/2508.00267
  • Gaotang Li, Danai Koutra, Yujun Yan, 1 Aug 2025, Tackling Size Generalization of Graph Neural Networks on Biological Data from a Spectral Perspective, https://arxiv.org/abs/2305.15611
  • Yoonhyuk Choi, Jiho Choi, Chong-Kwon Kim, 1 Aug 2025, Adaptive Branch Specialization in Spectral-Spatial Graph Neural Networks for Certified Robustness, https://arxiv.org/abs/2505.08320
  • Trung Nguyen, Md Masud Rana, Farjana Tasnim Mukta, Chang-Guo Zhan, Duc Duy Nguyen, 1 Aug 2025, Geometric Multi-color Message Passing Graph Neural Networks for Blood-brain Barrier Permeability Prediction, https://arxiv.org/abs/2507.18926
  • Xudong Wang, Tongxin Li, Chris Ding, Jicong Fan, 4 Aug 2025, Adaptive Riemannian Graph Neural Networks, https://arxiv.org/abs/2508.02600
  • Yanmei Hu, Siyuan Yin, Yihang Wu, Xue Yue and Yue Liu, 2 Aug 2025, A graph neural network based on feature network for identifying influential nodes, https://arxiv.org/abs/2508.01278
  • Divya Anand Sinha, Ruijie Du, Yezi Liu, Athina Markopolou, Yanning Shen, 3 Aug 2025, Gradient Inversion Attack on Graph Neural Networks, https://arxiv.org/abs/2411.19440
  • Antonio Tudisco, Deborah Volpe, Giacomo Orlandi, Giovanna Turvani, 4 Aug 2025, Graph Neural Network-Based Predictor for Optimal Quantum Hardware Selection, https://arxiv.org/abs/2507.19093
  • Kangkang Lu, Yanhua Yu, Zhiyong Huang, Tat-Seng Chua, 5 Aug 2025, Enhancing Spectral Graph Neural Networks with LLM-Predicted Homophily, https://arxiv.org/abs/2506.14220
  • Santhoshkumar Peddi, Sadhvik Bathini, Arun Balasubramanian, Monalisa Sarma, Debasis Samanta, 6 Aug 2025, ProtoN: Prototype Node Graph Neural Network for Unconstrained Multi-Impression Ear Recognition, https://arxiv.org/abs/2508.04381
  • Zhihao Wen, Yuan Fang, Pengcheng Wei, Fayao Liu, Zhenghua Chen, Min Wu, 6 Aug 2025, Temporal and Heterogeneous Graph Neural Network for Remaining Useful Life Prediction, https://arxiv.org/abs/2405.04336
  • Krzysztof Olejniczak, Xingyue Huang, Mikhail Galkin, \.Ismail \.Ilkan Ceylan, 5 Aug 2025, One Model, Any Conjunctive Query: Graph Neural Networks for Answering Queries over Incomplete Knowledge Graphs, https://arxiv.org/abs/2409.13959
  • Moshe Eliasof, Eldad Haber, Carola-Bibiane Sch\"onlieb, 7 Aug 2025, TANGO: Graph Neural Dynamics via Learned Energy and Tangential Flows, https://arxiv.org/abs/2508.05070
  • Massimiliano Romiti, 7 Aug 2025, A Graph Neural Network Approach for Mapping the Conceptual Structure and Inter-Branch Connectivity of Physics, https://arxiv.org/abs/2508.05724
  • Qin Chen, Guojie Song, 8 Aug 2025, Adaptive Heterogeneous Graph Neural Networks: Bridging Heterophily and Heterogeneity, https://arxiv.org/abs/2508.06034
  • Vibhor Agrawal, Fay Wang, Rishi Puri, 25 Jul 2025, Query-Aware Graph Neural Networks for Enhanced Retrieval-Augmented Generation, https://arxiv.org/abs/2508.05647
  • Dahai Yu, Dingyi Zhuang, Lin Jiang, Rongchao Xu, Xinyue Ye, Yuheng Bu, Shenhao Wang, Guang Wang, 12 Aug 2025, UQGNN: Uncertainty Quantification of Graph Neural Networks for Multivariate Spatiotemporal Prediction, https://arxiv.org/abs/2508.08551
  • Luigi D'Amico, Daniel De Rosso, Ninad Dixit, Raul Salles de Padua, Samuel Palmer, Samuel Mugel, Rom\'an Or\'us, Holger Eble, and Ali Abedi, 12 Aug 2025, Blockchain Network Analysis using Quantum Inspired Graph Neural Networks & Ensemble Models, https://arxiv.org/abs/2508.09237
  • Minghao Liu, Chia-Hsuan Lu, Marta Kwiatkowska, 12 Aug 2025, Exact Verification of Graph Neural Networks with Incremental Constraint Solving, https://arxiv.org/abs/2508.09320
  • Yun Zi, Ming Gong, Zhihao Xue, Yujun Zou, Nia Qi, Yingnan Deng, 13 Aug 2025, Graph Neural Network and Transformer Integration for Unsupervised System Anomaly Discovery, https://arxiv.org/abs/2508.09401
  • Fang Wang and Ernesto Damiani, 13 Aug 2025, Time-Aware and Transition-Semantic Graph Neural Networks for Interpretable Predictive Business Process Monitoring, https://arxiv.org/abs/2508.09527
  • Subhankar Sarkar and Souvik Chakraborty, 13 Aug 2025, Physics- and geometry-aware spatio-spectral graph neural operator for time-independent and time-dependent PDEs, https://arxiv.org/abs/2508.09627
  • Mohammad Zia Ur Rehman, Sufyaan Zahoor, Areeb Manzoor, Musharaf Maqbool, Nagendra Kumar, 7 Aug 2025, A Context-aware Attention and Graph Neural Network-based Multimodal Framework for Misogyny Detection, https://arxiv.org/abs/2508.09175
  • Marco S\"alzer, Fran\c{c}ois Schwarzentruber, Nicolas Troquard, 13 Aug 2025, Verifying Quantized Graph Neural Networks is PSPACE-complete, https://arxiv.org/abs/2502.16244
  • Asela Hevapathige, Asiri Wijesinghe, Ahad N. Zehmakan, 15 Aug 2025, Graph Neural Diffusion via Generalized Opinion Dynamics, https://arxiv.org/abs/2508.11249
  • Fanzhen Liu, Xiaoxiao Ma, Jian Yang, Alsharif Abuadbba, Kristen Moore, Surya Nepal, Cecile Paris, Quan Z. Sheng, Jia Wu, 15 Aug 2025, Towards Faithful Class-level Self-explainability in Graph Neural Networks by Subgraph Dependencies, https://arxiv.org/abs/2508.11513
  • Hossein Shokouhinejad, Roozbeh Razavi-Far, Griffin Higgins, Ali A Ghorbani, 14 Aug 2025, Explainable Attention-Guided Stacked Graph Neural Networks for Malware Detection, https://arxiv.org/abs/2508.09801
  • BG Tong, 10 Aug 2025, A Graph Neural Network based on a Functional Topology Model: Unveiling the Dynamic Mechanisms of Non-Suicidal Self-Injury in Single-Channel EEG, https://arxiv.org/abs/2508.11684
  • Anshul Ahluwalia, Payman Behnam, Rohit Das, Alind Khare, Biswadeep Chakraborty, Pan Li, Alexey Tumanov, 16 Aug 2025, STRIDE: Structure and Embedding Distillation with Attention for Graph Neural Networks, https://arxiv.org/abs/2310.15938
  • Ningyi Liao, Haoyu Liu, Zulun Zhu, Siqiang Luo, Laks V.S. Lakshmanan, 18 Aug 2025, Benchmarking Spectral Graph Neural Networks: A Comprehensive Study on Effectiveness and Efficiency, https://arxiv.org/abs/2406.09675
  • Su Chen, Xiaohua Qi, Xixun Lin, Yanmin Shang, Xiaolin Xu and Yangxi Li, 17 Aug 2025, Deep Graph Neural Point Process For Learning Temporal Interactive Networks, https://arxiv.org/abs/2508.13219
  • Junwei Su, Chuan Wu, 20 Aug 2025, On the Interplay between Graph Structure and Learning Algorithms in Graph Neural Networks, https://arxiv.org/abs/2508.14338
  • Zengyi Wo, Chang Liu, Yumeng Wang, Minglai Shao, Wenjun Wang, 20 Aug 2025, Improving Fairness in Graph Neural Networks via Counterfactual Debiasing, https://arxiv.org/abs/2508.14683
  • Mengyang Cao, Frank F. Yang, Yi Jin, Yijun Yan, 10 Aug 2025, Graph Neural Network for Product Recommendation on the Amazon Co-purchase Graph, https://arxiv.org/abs/2508.14059
  • Sebastian Musia{\l}, Bartosz Zieli\'nski, Tomasz Danel, 20 Aug 2025, Fragment-Wise Interpretability in Graph Neural Networks via Molecule Decomposition and Contribution Analysis, https://arxiv.org/abs/2508.15015
  • Mustafa Mohammadi Gharasuie and Luis Rueda, 20 Aug 2025, Fast Graph Neural Network for Image Classification, https://arxiv.org/abs/2508.14958
  • Zhiqiang Que, Chang Sun, Sudarshan Paramesvaran, Emyr Clement, Katerina Karakoulaki, Christopher Brown, Lauri Laatu, Arianna Cox, Alexander Tapper, Wayne Luk, Maria Spiropulu, 21 Aug 2025, JEDI-linear: Fast and Efficient Graph Neural Networks for Jet Tagging on FPGAs, https://arxiv.org/abs/2508.15468
  • Anahita Asadi, Leonid Popryho, Inna Partin-Vaisband, 22 Aug 2025, Fast and Accurate RFIC Performance Prediction via Pin Level Graph Neural Networks and Probabilistic Flow, https://arxiv.org/abs/2508.16403
  • Circe Hsu, Claire Schlesinger, Karan Mudaliar, Jordan Leung, Robin Walters, Peter Schindler, 22 Aug 2025, FIRE-GNN: Force-informed, Relaxed Equivariance Graph Neural Network for Rapid and Accurate Prediction of Surface Properties, https://arxiv.org/abs/2508.16012
  • Yuebo Luo, Shiyang Li, Junran Tao, Kiran Thorat, Xi Xie, Hongwu Peng, Nuo Xu, Caiwen Ding, Shaoyi Huang, 22 Aug 2025, DR-CircuitGNN: Training Acceleration of Heterogeneous Circuit Graph Neural Network on GPUs, https://arxiv.org/abs/2508.16769
  • Junhyun Lee, Veronika Thost, Bumsoo Kim, Jaewoo Kang, Tengfei Ma, 22 Aug 2025, Understanding and Tackling Over-Dilution in Graph Neural Networks, https://arxiv.org/abs/2508.16829
  • Bicheng Wang and Junping Wang and Yibo Xue, 22 Aug 2025, Physics-Inspired Spatial Temporal Graph Neural Networks for Predicting Industrial Chain Resilience, https://arxiv.org/abs/2508.16836
  • Silvia Beddar-Wiesing and Alice Moallemy-Oureh, 25 Aug 2025, Weisfeiler-Lehman meets Events: An Expressivity Analysis for Continuous-Time Dynamic Graph Neural Networks, https://arxiv.org/abs/2508.18052
  • Menglin Yang, Min Zhou, Tong Zhang, Jiahong Liu, Zhihao Li, Lujia Pan, Hui Xiong, Irwin King, 22 Aug 2025, Hyperbolic Graph Neural Networks: A Review of Methods and Applications, https://arxiv.org/abs/2202.13852
  • Moshe Eliasof, Eldad Haber, 23 Aug 2025, Quadratic Binary Optimization with Graph Neural Networks, https://arxiv.org/abs/2404.04874
  • XiaYu Liu, Chao Fan, Yang Liu, Hou-biao Li, 24 Aug 2025, Multi-Level Fusion Graph Neural Network for Molecule Property Prediction, https://arxiv.org/abs/2507.03430
  • Michela Lapenna, Caterina De Bacco, 22 Aug 2025, How do Probabilistic Graphical Models and Graph Neural Networks Look at Network Data?, https://arxiv.org/abs/2506.11869
  • Kevin Monteiro, Sam Nallaperuma-Herzberg, Martina Mason, Steve Niederer, 2 Jul 2025, Graph Convolutional Neural Networks to Model the Brain for Insomnia, https://arxiv.org/abs/2507.14147
  • Ganesh Sundaram, Jonas Ulmen, and Daniel G\"orges, 20 Jul 2025, Enhanced Pruning Strategy for Multi-Component Neural Architectures Using Component-Aware Graph Analysis, https://arxiv.org/abs/2504.13296
  • Andrew Kiruluta, Andreas Lemos, and Priscilla Burity, 27 Jul 2025, Beyond Neural Networks: Symbolic Reasoning over Wavelet Logic Graph Signals, https://arxiv.org/abs/2507.21190
  • Cencheng Shen, Yuexiao Dong, 8 Aug 2025, A Graph Sufficiency Perspective for Neural Networks, https://arxiv.org/abs/2507.10215
  • Mustafa Mohammadi Gharasuie and Luis Rueda, 19 Aug 2025, Accelerating Image Classification with Graph Convolutional Neural Networks using Voronoi Diagrams, https://arxiv.org/abs/2508.14218
  • Nathan X. Kodama and Kenneth A. Loparo, 22 Aug 2025, Latent Graph Learning in Generative Models of Neural Signals, https://arxiv.org/abs/2508.16776
  • Lingkai Kong, Haotian Sun, Yuchen Zhuang, Haorui Wang, Wenhao Mu, Chao Zhang, 23 Aug 2025, Two Birds with One Stone: Enhancing Uncertainty Quantification and Interpretability with Graph Functional Neural Process, https://arxiv.org/abs/2508.17097
  • Riccardo Cappi, Paolo Frazzetto, Nicol\`o Navarin, Alessandro Sperduti, 25 Aug 2025, Unveiling the Actual Performance of Neural-based Models for Equation Discovery on Graph Dynamical Systems, https://arxiv.org/abs/2508.18173

Compound AI Architectures

Compound AI architectures are a new category that generalizes both RAG and multi-AI ensemble architectures. The general idea is that various components can be placed around an LLM, or multiple LLM queries can be used, and this can be done in a variety of ways. RAG is a well-known subcategory in this vein, as are extensions using Knowledge Graphs.

Research on Compound AI architectures:

Research on Efficient Architectures

AI Books from Aussie AI



The Sweetest Lesson: Your Brain Versus AI The Sweetest Lesson: Your Brain Versus AI: new book on AI intelligence theory:
  • Your brain is 50 times bigger than the best AI engines.
  • Truly intelligent AI will require more compute!
  • Another case of the bitter lesson?
  • Maybe it's the opposite of that: the sweetest lesson.

Get your copy from Amazon: The Sweetest Lesson



RAG Optimization RAG Optimization: Accurate and Efficient LLM Applications: new book on RAG architectures:
  • Smarter RAG
  • Faster RAG
  • Cheaper RAG
  • Agentic RAG
  • RAG reasoning

Get your copy from Amazon: RAG Optimization



Generative AI in C++ Generative AI Applications book:
  • Deciding on your AI project
  • Planning for success and safety
  • Designs and LLM architectures
  • Expediting development
  • Implementation and deployment

Get your copy from Amazon: Generative AI Applications



Generative AI in C++ Generative AI programming book:
  • Generative AI coding in C++
  • Transformer engine speedups
  • LLM models
  • Phone and desktop AI
  • Code examples
  • Research citations

Get your copy from Amazon: Generative AI in C++



CUDA C++ Optimization CUDA C++ Optimization book:
  • Faster CUDA C++ kernels
  • Optimization tools & techniques
  • Compute optimization
  • Memory optimization

Get your copy from Amazon: CUDA C++ Optimization



CUDA C++ Optimization CUDA C++ Debugging book:
  • Debugging CUDA C++ kernels
  • Tools & techniques
  • Self-testing & reliability
  • Common GPU kernel bugs

Get your copy from Amazon: CUDA C++ Debugging

More AI Research

Read more about: